# HG changeset patch # User Thierry Florac # Date 1520330805 -3600 # Node ID e1cb9c60685966258c60e91e97dfb6052a287f9f # Parent 8c0c5400e770b6a2e1c727410993cfc7e1c70183 Reorganized index diff -r 8c0c5400e770 -r e1cb9c606859 src/source/appinstall.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/appinstall.rst Tue Mar 06 11:06:45 2018 +0100 @@ -0,0 +1,381 @@ +.. _appinstall: + +Installing PyAMS application +============================ + +PyAMS default installation is based on `Buildout `_ utility. It's not mandatory to use a +virtual environment, but it allows you to have a better control over your Python resources. + +Current PyAMS version is based and validated for Python 3.5; your Python environment must also include a C +compiler as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and +eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. *Cython* compiler can also be useful to optimize +several packages. + +PyAMS default components configuration also pre-suppose that the following external tools are available: + +- a *Memcached* or *Redis* server, to store sessions and cache (can be changed through Beaker configuration) + +Optional tools also include: + +- an *LDAP* server for authentication + +- an *ElasticSearch* server for full text indexing (see *PyAMS_content_es* package) + +- a *WebSockets* server using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* + packages). An *out of the box* environment can be built using *pyams_notify* scaffold. + + +PyAMS also needs that you use a ZODB shared server, as several background processes needing a concurrent access to ZODB +are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db. +See :ref:`zodb` to know how to initialize database with the help of PyAMS tools. + + +Creating initial buildout +------------------------- + +PyAMS provides a new Pyramid scaffold, called *pyams*, generated via a *cookiecutter* template. + +A simple option to install PyAMS is to create a buildout environment including *Pyramid* and all *PyAMS* packages: + +.. code-block:: bash + + # mkdir /var/local/ + # pip3 install virtualenv + # virtualenv --python=python3.5 env + # cd env + # . bin/activate + (env) # pip3.5 install cookiecutter + (env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams + +*CookieCutter* will ask you for a small set of input variables that you can change or not: + +- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release; + you can also choose to point to a given release ("0.1.4" for example) + +- **project_name**: current environment name in "human form" + +- **project_slug**: "technical" package name, based on project name + +- **virtual_hostname**: Apache virtual-host name + +- **webapp_name**: web application package name ("webapp" as default) + +- **webapp_port**: TCP/IP port to use when running application outside Apache ("6543" as default) + +- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be + shared with other projects ("eggs" as default) + +- **logs_directory**: absolute path to directory containing Apache's log files + +- **run_user**: user name under which Apache process will run ("www-data" as default) + +- **run_group**: group name under which Apache process will run ("www-data" as default) + +- **beaker_backend**: name of Beaker backend to use to store sessions and cache data ("redis" as default) + +- **beaker_server**: IP address and port of Beaker backend server ("127.0.0.1:6379" as default) + +- **db_type**: ZODB database storage; available options include ZEO, RelStorage and NewtDB + +- **db_host**: IP address of database server ("127.0.0.1" as default); WARNING: database server installation + is not part of application installation; another "zeo_server" cookiecutter recipe is available for ZEO + +- **db_port**: listening port of database server ("8100" is given as default for ZEO) + +- **db_name**: database or ZEO storage name to use + +- **db_username**: database user name + +- **db_password**: database password + +- **zeo_realm**: ZEO authentication realm + +- **blobs_dir**: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default + +- **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files + are available to compile PsycoPG2 extension + +- **use_oracle**: specify if Oracle access is required; if so, please check that Oracle development files are + available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below) + +- **use_ldap**: specify if LDAP access will be required for authentication + +- **use_elasticsearch**: specify if an ElasticSearch server will be used for indexation + +- **elasticsearch_server**: URL used to access Elasticsearch server ("http://127.0.0.1:9200" as default); this URL can + include login and password ("http://login:password@127.0.0.1:9200"), if required... + +- **elasticsearch_index**: name of Elasticsearch index to use ("pyams" as default) + +- **create_elasticsearch_index**: specify if Elasticsearch index should be created after installation is complete + +- **define_elasticsearch_mappings** : specify if Elasticsearch mappings should be defined after installation is complete + +- **smtp_server**: DNS name of SMTP server ("localhost" as default) + +- **smtp_server_name**: "human" name given to SMTP server ("pyams" as default) + +- **pyams_scheduler**: TCP/IP address and port to use to access PyAMS tasks scheduler process ("127.0.0.1:5555" as + default); see :ref:`pyams_scheduler` + +- **start_scheduler**: boolean value to indicate if scheduler process is started by this application instance + +- **pyams_medias_converter**: TCP/IP address and port to use to access PyAMS medias converter process ("127.0.0.1:5556" + as default); see :ref:`pyams_medias` + +- **start_medias_converter**: boolean value to indicate if medias converter process is started by this application + instance + +- **pyams_es_indexer**: TCP/IP address and port to use to access PyAMS Elasticsearch indexer process ("127.0.0.1:5557" + as default); see :ref:`pyams_content_es` + +- **start_es_indexer** boolean value to indicate if Elasticsearch indexer process is started by this application + instance + +- **use_notifications**: specify if PyAMS notifications services are to be used (see :ref:`pyams_notify`) + +- **pyams_ws_notify**: TCP/IP address and port of PyAMS websockets server managing notifications service + ("127.0.0.1:8081" as default) + +- **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default) + +- **extension_package**: name of a PyAMS extension package to include in environment configuration + +- **need_pyams_gis**: specify if PyAMS GIS features are to be used by given extension package; if so, please check + that *libgdal* development files are available; on Debian (and maybe others), you have to specify environment + variables (see below). + + +Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*; setting +any option to empty string will keep the default proposed value. + +You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove +packages or update settings to your needs. Then finalize Bootstrap initialization: + +.. code-block:: bash + + (env) # python3.5 bootstrap.py + (env) # ./bin/buildout + +This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual +environment. If you encounter any compile error, just install the required dependencies and restart the buildout. + +Some dependencies can require the definition of custom environment variables before running *buildout*, like: + +- for *libgdal*, which is required by **PyAMS_gis** package, use: + +.. code-block:: bash + + (env) # export C_INCLUDE_PATH=/usr/include/gdal + (env) # export CPLUS_INCLUDE_PATH=/usr/include/gdal + +**WARNING**: you have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS +configuration file (actually 2.1.0). + +- for *cx_Oracle*, which is required if you use Oracle database connections, use: + +.. code-block:: bash + + (env) # export ORACLE_HOME=/usr/lib/oracle/12.1/client64 + +These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux +distribution and packages versions. + + +Environment settings +-------------------- + +The project generated from *pyams* scaffold is based on default Pyramid's *zodb* scaffold, but it adds: + +- a custom application factory, in the *webapp* directory (see :ref:`site`) + +- a set of directories to store runtime data, in the *var* directory; each directory contains a *README.txt* file + which should be self-explanatory to indicate what this directory should contain, including a ZEO cache + +- a set of configuration files, in the *etc* directory; here are standard *development.ini* and *production.ini* + configuration files, a ZODB configuration files (*zodb-zeo.conf*) for a ZEO client storage and two Apache + configurations (for Apache 2.2 and 2.4) using *mod_wsgi*. + +Once the project have been created from the scaffold, you are free to update all the configuration files. + +If you need to add packages to the environment, you have to add them to the *buildout.cfg* file **AND** to the INI +file (in the *pyramid.includes* section) before running the *buildout* another time; don't forget to add the +requested version at the end of *buildout.cfg* file, as Buildout is not configured by default to automatically +download the last release of a given unknown package. + +*development.ini* and *production.ini* files contain many commented directives related to PyAMS components. Read and +update them carefully before initializing your application database! + + +Initializing the database +------------------------- + +When you have downloaded and installed all required packages, you have to initialize the database so that all +required components are available. + +From a shell, just type: + +.. code-block:: bash + + (env) # ./bin/pyams_upgrade etc/development.ini + +This process requires that every package is correctly included into *pyramid.includes* directive from selected +configuration file. + + +Initializing Elasticsearch index +-------------------------------- + +If you want to use an Elasticsearch index, you have to initialize index settings and mappings; the Ingest attachment +plug-in is also required to handle attachments correctly. + +Elasticsearch integration is defined through the *PyAMS_content_es* package. Configuration files are available in this +package, for attachment pipeline, index settings and mappings: + +.. code-block:: bash + + (env) # cd /var/local/src/pyams/pyams_content_es + (env) # curl --noproxy localhost -XDELETE http://localhost:9200/pyams (1) + (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams -d @index-settings.json + + (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json + (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json + (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json + +(1) If 'pyams' is defined as Elasticsearch index name. + + +NLTK initialization +------------------- + +Some NLTK (Natural Language Toolkit) tokenizers and stopwords utilities are used to index fulltext contents elements. +This package requires downloading and configuration of several elements which are done as follow: + +.. code-block:: bash + + (end) # ./bin/py + >>> import nltk + >>> nltk.download() + NLTK Downloader + --------------------------------------------------------------------------- + d) Download l) List u) Update c) Config h) Help q) Quit + --------------------------------------------------------------------------- + Downloader> c + + Data Server: + - URL: + - 6 Package Collections Available + - 107 Individual Packages Available + + Local Machine: + - Data directory: /home/tflorac/nltk_data + + --------------------------------------------------------------------------- + s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu + --------------------------------------------------------------------------- + Config> d + New directory> /usr/local/lib/nltk_data (1) + Config> m + + --------------------------------------------------------------------------- + d) Download l) List u) Update c) Config h) Help q) Quit + --------------------------------------------------------------------------- + Downloader> d + + Download which package (l=list; x=cancel)? + Identifier> punkt + Downloading package punkt to /usr/local/lib/nltk_data... + + Downloader> d + + Download which package (l=list; x=cancel)? + Identifier> stopwords + Downloading package stopwords to /usr/local/lib/nltk_data... + + +(1) On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running +Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and +'*/usr/local/lib/nltk_data*'. + + +Starting the application +------------------------ + +When database upgrade process has ended, you can start the web application process with the standard Pyramid's +*pserve* command line tool: + +.. code-block:: bash + + (env) # ./bin/pserve etc/development.ini + +In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO): + +.. code-block:: bash + + 2018-01-14 11:37:54,339 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=28695) created RW/normal for storage: 'pyams' + 2018-01-14 11:37:54,340 INFO [ZEO.cache][MainThread] created temporary cache file 3 + 2018-01-14 11:37:54,345 INFO [ZODB.blob][MainThread] (28695) Blob directory `/var/local/env/pyams/var/db/blobs` is used but has no layout marker set. Selected `lawn` layout. + 2018-01-14 11:37:54,345 WARNI [ZODB.blob][MainThread] (28695) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout. + 2018-01-14 11:37:54,346 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector + 2018-01-14 11:37:54,347 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None + 2018-01-14 11:37:54,348 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting + 2018-01-14 11:37:54,349 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) + 2018-01-14 11:37:54,355 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) + 2018-01-14 11:37:54,358 DEBUG [txn.140663320073984][MainThread] new transaction + 2018-01-14 11:37:54,360 DEBUG [txn.140663320073984][MainThread] commit + 2018-01-14 11:37:54,484 DEBUG [config][MainThread] include /home/tflorac/Dropbox/src/PyAMS/pyams_template/src/pyams_template/configure.zcml + 2018-01-14 11:37:54,485 DEBUG [config][MainThread] include /var/local/env/pycharm/lib/python3.5/site-packages/pyramid_zcml/configure.zcml + ... + 2018-01-14 11:37:54,833 DEBUG [PyAMS (utils)][MainThread] Registering utility named 'PyAMS timezone' providing + 2018-01-14 11:37:54,834 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS timezones" + 2018-01-14 11:37:54,835 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing + 2018-01-14 11:37:54,839 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2018-01-14 11:37:54,847 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing + 2018-01-14 11:37:54,942 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing + 2018-01-14 11:37:54,943 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () + 2018-01-14 11:37:54,949 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () + 2018-01-14 11:37:54,980 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZEO connections" + 2018-01-14 11:37:54,981 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZODB connections" + 2018-01-14 11:37:55,015 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "add-zeo-connection.html" for () + 2018-01-14 11:37:55,016 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing + 2018-01-14 11:37:55,017 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () + ... + 2018-01-14 11:41:13,214 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2018-01-14 11:43:36,665 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' + 2018-01-14 11:43:36,665 INFO [ZEO.cache][MainThread] created temporary cache file 9 + 2018-01-14 11:43:36,673 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector + 2018-01-14 11:43:36,674 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) + 2018-01-14 11:43:36,674 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None + 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 + 2018-01-14 11:43:36,675 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting + 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> + 2018-01-14 11:43:36,677 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) + 2018-01-14 11:43:36,679 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) + 2018-01-14 11:43:36,682 DEBUG [txn.140713340237568][MainThread] new transaction + 2018-01-14 11:43:36,683 DEBUG [txn.140713340237568][MainThread] commit + 2018-01-14 11:43:36,690 INFO [PyAMS (scheduler][MainThread] Starting tasks scheduler ... + 2018-01-14 11:43:36,698 INFO [PyAMS (scheduler][MainThread] Started tasks scheduler with PID 29361. + 2018-01-14 11:43:36,701 INFO [apscheduler.scheduler][MainThread] Scheduler started + 2018-01-14 11:43:36,702 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run + 2018-01-14 11:43:36,704 DEBUG [apscheduler.scheduler][APScheduler] No jobs; waiting until a job is added + 2018-01-14 11:43:36,719 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' + 2018-01-14 11:43:36,720 INFO [ZEO.cache][MainThread] created temporary cache file 15 + 2018-01-14 11:43:36,724 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector + 2018-01-14 11:43:36,725 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None + 2018-01-14 11:43:36,726 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting + 2018-01-14 11:43:36,727 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) + 2018-01-14 11:43:36,728 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) + 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 + 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> + 2018-01-14 11:43:36,732 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) + 2018-01-14 11:43:36,735 DEBUG [txn.140713340237568][MainThread] new transaction + 2018-01-14 11:43:36,736 DEBUG [txn.140713340237568][MainThread] commit + 2018-01-14 11:43:36,743 INFO [PyAMS (media)][MainThread] Starting medias converter ... + 2018-01-14 11:43:36,751 INFO [PyAMS (media)][MainThread] Started medias converter with PID 29367. + Starting server in PID 29335. + Serving on http://0.0.0.0:6543 + + +From this point, you can launch a browser and open URL *http://127.0.0.1:6543/admin* to get access to PyAMS +management interface; default login is "admin/admin", that you may change as soon as possible (see +:ref:`pyams_security`)!!. diff -r 8c0c5400e770 -r e1cb9c606859 src/source/architecture.rst --- a/src/source/architecture.rst Mon Mar 05 10:40:40 2018 +0100 +++ b/src/source/architecture.rst Tue Mar 06 11:06:45 2018 +0100 @@ -2,3 +2,8 @@ PyAMS application architecture ============================== + +.. toctree:: + :maxdepth: 2 + + site diff -r 8c0c5400e770 -r e1cb9c606859 src/source/components.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/components.rst Tue Mar 06 11:06:45 2018 +0100 @@ -0,0 +1,10 @@ +.. _components: + +Custom PyAMS components +======================= + +.. toctree:: + :maxdepth: 2 + + traverser + tales diff -r 8c0c5400e770 -r e1cb9c606859 src/source/extend.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/extend.rst Tue Mar 06 11:06:45 2018 +0100 @@ -0,0 +1,10 @@ +.. _extending: + +Extending PyAMS +=============== + +.. toctree:: + :maxdepth: 2 + + zca + components diff -r 8c0c5400e770 -r e1cb9c606859 src/source/index.rst --- a/src/source/index.rst Mon Mar 05 10:40:40 2018 +0100 +++ b/src/source/index.rst Tue Mar 06 11:06:45 2018 +0100 @@ -17,22 +17,22 @@ PyAMS is built on top of MyAMS (My Application Management Skin), a small web client framework built on top of JQuery and Bootstrap, which was developed by the french national forestry office (ONF -- Office national des forêts -- -http://www.onf.fr) to build Java and Python web applications. The new ONF website is now completely handled with -PyAMS framework. +http://www.onf.fr) to build web applications in several languages (actually java, Python and PHP). The new ONF website +is now completely handled with PyAMS framework. PyAMS is a multipurpose set of packages, providing tools including: * custom interfaces -* custom ZCA registry annotations +* custom registry annotations +* custom security policy * local registry support * network protocols utilities (for HTTP and XML-RPC) * command line scripts -* a custom security policy -* custom utilities... +* custom utilities. If you just want to know how easy it is to create a PyAMS application instance, just read the :ref:`quickstart`! -To know how PyAMS management interface looks like, just see :ref:`screenshots`! +To know how PyAMS management interface looks like, look at :ref:`screenshots`! .. toctree:: @@ -40,13 +40,10 @@ quickstart architecture - zodb install - zca - site - traverser - tales - utilities + config + manage + extend PyAMS applications architecture diff -r 8c0c5400e770 -r e1cb9c606859 src/source/install.rst --- a/src/source/install.rst Mon Mar 05 10:40:40 2018 +0100 +++ b/src/source/install.rst Tue Mar 06 11:06:45 2018 +0100 @@ -3,379 +3,8 @@ Installing PyAMS ================ -PyAMS default installation is based on `Buildout `_ utility. It's not mandatory to use a -virtual environment, but it allows you to have a better control over your Python resources. - -Current PyAMS version is based and validated for Python 3.5; your Python environment must also include a C -compiler as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and -eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. *Cython* compiler can also be useful to optimize -several packages. - -PyAMS default components configuration also pre-suppose that the following external tools are available: - -- a *Memcached* or *Redis* server, to store sessions and cache (can be changed through Beaker configuration) - -Optional tools also include: - -- an *LDAP* server for authentication - -- an *ElasticSearch* server for full text indexing (see *PyAMS_content_es* package) - -- a *WebSockets* server using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* - packages). An *out of the box* environment can be built using *pyams_notify* scaffold. - - -PyAMS also needs that you use a ZODB shared server, as several background processes needing a concurrent access to ZODB -are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db. -See :ref:`zodb` to know how to initialize database with the help of PyAMS tools. - - -Creating initial buildout -------------------------- - -PyAMS provides a new Pyramid scaffold, called *pyams*, generated via a *cookiecutter* template. - -A simple option to install PyAMS is to create a buildout environment including *Pyramid* and all *PyAMS* packages: - -.. code-block:: bash - - # mkdir /var/local/ - # pip3 install virtualenv - # virtualenv --python=python3.5 env - # cd env - # . bin/activate - (env) # pip3.5 install cookiecutter - (env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams - -*CookieCutter* will ask you for a small set of input variables that you can change or not: - -- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release; - you can also choose to point to a given release ("0.1.4" for example) - -- **project_name**: current environment name in "human form" - -- **project_slug**: "technical" package name, based on project name - -- **virtual_hostname**: Apache virtual-host name - -- **webapp_name**: web application package name ("webapp" as default) - -- **webapp_port**: TCP/IP port to use when running application outside Apache ("6543" as default) - -- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be - shared with other projects ("eggs" as default) - -- **logs_directory**: absolute path to directory containing Apache's log files - -- **run_user**: user name under which Apache process will run ("www-data" as default) - -- **run_group**: group name under which Apache process will run ("www-data" as default) - -- **beaker_backend**: name of Beaker backend to use to store sessions and cache data ("redis" as default) - -- **beaker_server**: IP address and port of Beaker backend server ("127.0.0.1:6379" as default) - -- **db_type**: ZODB database storage; available options include ZEO, RelStorage and NewtDB - -- **db_host**: IP address of database server ("127.0.0.1" as default); WARNING: database server installation - is not part of application installation; another "zeo_server" cookiecutter recipe is available for ZEO - -- **db_port**: listening port of database server ("8100" is given as default for ZEO) - -- **db_name**: database or ZEO storage name to use - -- **db_username**: database user name - -- **db_password**: database password - -- **zeo_realm**: ZEO authentication realm - -- **blobs_dir**: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default - -- **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files - are available to compile PsycoPG2 extension - -- **use_oracle**: specify if Oracle access is required; if so, please check that Oracle development files are - available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below) - -- **use_ldap**: specify if LDAP access will be required for authentication - -- **use_elasticsearch**: specify if an ElasticSearch server will be used for indexation - -- **elasticsearch_server**: URL used to access Elasticsearch server ("http://127.0.0.1:9200" as default); this URL can - include login and password ("http://login:password@127.0.0.1:9200"), if required... - -- **elasticsearch_index**: name of Elasticsearch index to use ("pyams" as default) - -- **create_elasticsearch_index**: specify if Elasticsearch index should be created after installation is complete - -- **define_elasticsearch_mappings** : specify if Elasticsearch mappings should be defined after installation is complete - -- **smtp_server**: DNS name of SMTP server ("localhost" as default) - -- **smtp_server_name**: "human" name given to SMTP server ("pyams" as default) - -- **pyams_scheduler**: TCP/IP address and port to use to access PyAMS tasks scheduler process ("127.0.0.1:5555" as - default); see :ref:`pyams_scheduler` - -- **start_scheduler**: boolean value to indicate if scheduler process is started by this application instance - -- **pyams_medias_converter**: TCP/IP address and port to use to access PyAMS medias converter process ("127.0.0.1:5556" - as default); see :ref:`pyams_medias` - -- **start_medias_converter**: boolean value to indicate if medias converter process is started by this application - instance - -- **pyams_es_indexer**: TCP/IP address and port to use to access PyAMS Elasticsearch indexer process ("127.0.0.1:5557" - as default); see :ref:`pyams_content_es` - -- **start_es_indexer** boolean value to indicate if Elasticsearch indexer process is started by this application - instance - -- **use_notifications**: specify if PyAMS notifications services are to be used (see :ref:`pyams_notify`) - -- **pyams_ws_notify**: TCP/IP address and port of PyAMS websockets server managing notifications service - ("127.0.0.1:8081" as default) - -- **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default) - -- **extension_package**: name of a PyAMS extension package to include in environment configuration - -- **need_pyams_gis**: specify if PyAMS GIS features are to be used by given extension package; if so, please check - that *libgdal* development files are available; on Debian (and maybe others), you have to specify environment - variables (see below). - - -Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*; setting -any option to empty string will keep the default proposed value. - -You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove -packages or update settings to your needs. Then finalize Bootstrap initialization: - -.. code-block:: bash - - (env) # python3.5 bootstrap.py - (env) # ./bin/buildout - -This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual -environment. If you encounter any compile error, just install the required dependencies and restart the buildout. - -Some dependencies can require the definition of custom environment variables before running *buildout*, like: - -- for *libgdal*, which is required by **PyAMS_gis** package, use: - -.. code-block:: bash - - (env) # export C_INCLUDE_PATH=/usr/include/gdal - (env) # export CPLUS_INCLUDE_PATH=/usr/include/gdal - -**WARNING**: you have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS -configuration file (actually 2.1.0). - -- for *cx_Oracle*, which is required if you use Oracle database connections, use: - -.. code-block:: bash - - (env) # export ORACLE_HOME=/usr/lib/oracle/12.1/client64 - -These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux -distribution and packages versions. - - -Environment settings --------------------- - -The project generated from *pyams* scaffold is based on default Pyramid's *zodb* scaffold, but it adds: - -- a custom application factory, in the *webapp* directory (see :ref:`site`) +.. toctree:: + :maxdepth: 2 -- a set of directories to store runtime data, in the *var* directory; each directory contains a *README.txt* file - which should be self-explanatory to indicate what this directory should contain, including a ZEO cache - -- a set of configuration files, in the *etc* directory; here are standard *development.ini* and *production.ini* - configuration files, a ZODB configuration files (*zodb-zeo.conf*) for a ZEO client storage and two Apache - configurations (for Apache 2.2 and 2.4) using *mod_wsgi*. - -Once the project have been created from the scaffold, you are free to update all the configuration files. - -If you need to add packages to the environment, you have to add them to the *buildout.cfg* file **AND** to the INI -file (in the *pyramid.includes* section) before running the *buildout* another time; don't forget to add the -requested version at the end of *buildout.cfg* file, as Buildout is not configured by default to automatically -download the last release of a given unknown package. - -*development.ini* and *production.ini* files contain many commented directives related to PyAMS components. Read and -update them carefully before initializing your application database! - - -Initializing the database -------------------------- - -When you have downloaded and installed all required packages, you have to initialize the database so that all -required components are available. - -From a shell, just type: - -.. code-block:: bash - - (env) # ./bin/pyams_upgrade etc/development.ini - -This process requires that every package is correctly included into *pyramid.includes* directive from selected -configuration file. - - -Initializing Elasticsearch index --------------------------------- - -If you want to use an Elasticsearch index, you have to initialize index settings and mappings; the Ingest attachment -plug-in is also required to handle attachments correctly. - -Elasticsearch integration is defined through the *PyAMS_content_es* package. Configuration files are available in this -package, for attachment pipeline, index settings and mappings: - -.. code-block:: bash - - (env) # cd /var/local/src/pyams/pyams_content_es - (env) # curl --noproxy localhost -XDELETE http://localhost:9200/pyams (1) - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams -d @index-settings.json - - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json - -(1) If 'pyams' is defined as Elasticsearch index name. - - -NLTK initialization -------------------- - -Some NLTK (Natural Language Toolkit) tokenizers and stopwords utilities are used to index fulltext contents elements. -This package requires downloading and configuration of several elements which are done as follow: - -.. code-block:: bash - - (end) # ./bin/py - >>> import nltk - >>> nltk.download() - NLTK Downloader - --------------------------------------------------------------------------- - d) Download l) List u) Update c) Config h) Help q) Quit - --------------------------------------------------------------------------- - Downloader> c - - Data Server: - - URL: - - 6 Package Collections Available - - 107 Individual Packages Available - - Local Machine: - - Data directory: /home/tflorac/nltk_data - - --------------------------------------------------------------------------- - s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu - --------------------------------------------------------------------------- - Config> d - New directory> /usr/local/lib/nltk_data (1) - Config> m - - --------------------------------------------------------------------------- - d) Download l) List u) Update c) Config h) Help q) Quit - --------------------------------------------------------------------------- - Downloader> d - - Download which package (l=list; x=cancel)? - Identifier> punkt - Downloading package punkt to /usr/local/lib/nltk_data... - - Downloader> d - - Download which package (l=list; x=cancel)? - Identifier> stopwords - Downloading package stopwords to /usr/local/lib/nltk_data... - - -(1) On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running -Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and -'*/usr/local/lib/nltk_data*'. - - -Starting the application ------------------------- - -When database upgrade process has ended, you can start the web application process with the standard Pyramid's -*pserve* command line tool: - -.. code-block:: bash - - (env) # ./bin/pserve etc/development.ini - -In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO): - -.. code-block:: bash - - 2018-01-14 11:37:54,339 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=28695) created RW/normal for storage: 'pyams' - 2018-01-14 11:37:54,340 INFO [ZEO.cache][MainThread] created temporary cache file 3 - 2018-01-14 11:37:54,345 INFO [ZODB.blob][MainThread] (28695) Blob directory `/var/local/env/pyams/var/db/blobs` is used but has no layout marker set. Selected `lawn` layout. - 2018-01-14 11:37:54,345 WARNI [ZODB.blob][MainThread] (28695) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout. - 2018-01-14 11:37:54,346 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector - 2018-01-14 11:37:54,347 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None - 2018-01-14 11:37:54,348 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting - 2018-01-14 11:37:54,349 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) - 2018-01-14 11:37:54,355 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) - 2018-01-14 11:37:54,358 DEBUG [txn.140663320073984][MainThread] new transaction - 2018-01-14 11:37:54,360 DEBUG [txn.140663320073984][MainThread] commit - 2018-01-14 11:37:54,484 DEBUG [config][MainThread] include /home/tflorac/Dropbox/src/PyAMS/pyams_template/src/pyams_template/configure.zcml - 2018-01-14 11:37:54,485 DEBUG [config][MainThread] include /var/local/env/pycharm/lib/python3.5/site-packages/pyramid_zcml/configure.zcml - ... - 2018-01-14 11:37:54,833 DEBUG [PyAMS (utils)][MainThread] Registering utility named 'PyAMS timezone' providing - 2018-01-14 11:37:54,834 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS timezones" - 2018-01-14 11:37:54,835 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing - 2018-01-14 11:37:54,839 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing - 2018-01-14 11:37:54,847 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing - 2018-01-14 11:37:54,942 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing - 2018-01-14 11:37:54,943 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () - 2018-01-14 11:37:54,949 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () - 2018-01-14 11:37:54,980 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZEO connections" - 2018-01-14 11:37:54,981 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZODB connections" - 2018-01-14 11:37:55,015 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "add-zeo-connection.html" for () - 2018-01-14 11:37:55,016 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing - 2018-01-14 11:37:55,017 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () - ... - 2018-01-14 11:41:13,214 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing - 2018-01-14 11:43:36,665 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' - 2018-01-14 11:43:36,665 INFO [ZEO.cache][MainThread] created temporary cache file 9 - 2018-01-14 11:43:36,673 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector - 2018-01-14 11:43:36,674 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) - 2018-01-14 11:43:36,674 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None - 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 - 2018-01-14 11:43:36,675 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting - 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> - 2018-01-14 11:43:36,677 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) - 2018-01-14 11:43:36,679 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) - 2018-01-14 11:43:36,682 DEBUG [txn.140713340237568][MainThread] new transaction - 2018-01-14 11:43:36,683 DEBUG [txn.140713340237568][MainThread] commit - 2018-01-14 11:43:36,690 INFO [PyAMS (scheduler][MainThread] Starting tasks scheduler ... - 2018-01-14 11:43:36,698 INFO [PyAMS (scheduler][MainThread] Started tasks scheduler with PID 29361. - 2018-01-14 11:43:36,701 INFO [apscheduler.scheduler][MainThread] Scheduler started - 2018-01-14 11:43:36,702 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run - 2018-01-14 11:43:36,704 DEBUG [apscheduler.scheduler][APScheduler] No jobs; waiting until a job is added - 2018-01-14 11:43:36,719 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' - 2018-01-14 11:43:36,720 INFO [ZEO.cache][MainThread] created temporary cache file 15 - 2018-01-14 11:43:36,724 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector - 2018-01-14 11:43:36,725 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None - 2018-01-14 11:43:36,726 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting - 2018-01-14 11:43:36,727 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) - 2018-01-14 11:43:36,728 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) - 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 - 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> - 2018-01-14 11:43:36,732 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) - 2018-01-14 11:43:36,735 DEBUG [txn.140713340237568][MainThread] new transaction - 2018-01-14 11:43:36,736 DEBUG [txn.140713340237568][MainThread] commit - 2018-01-14 11:43:36,743 INFO [PyAMS (media)][MainThread] Starting medias converter ... - 2018-01-14 11:43:36,751 INFO [PyAMS (media)][MainThread] Started medias converter with PID 29367. - Starting server in PID 29335. - Serving on http://0.0.0.0:6543 - - -From this point, you can launch a browser and open URL *http://127.0.0.1:6543/admin* to get access to PyAMS -management interface; default login is "admin/admin", that you may change as soon as possible (see -:ref:`pyams_security`)!!. + zodb + appinstall diff -r 8c0c5400e770 -r e1cb9c606859 src/source/manage.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/manage.rst Tue Mar 06 11:06:45 2018 +0100 @@ -0,0 +1,10 @@ +.. _manage: + +Managing PyAMS environment +========================== + +.. toctree:: + :maxdepth: 2 + + utilities + diff -r 8c0c5400e770 -r e1cb9c606859 src/source/quickstart.rst --- a/src/source/quickstart.rst Mon Mar 05 10:40:40 2018 +0100 +++ b/src/source/quickstart.rst Tue Mar 06 11:06:45 2018 +0100 @@ -10,6 +10,10 @@ See :ref:`install` to get a full description of installation process. +.. contents:: + :local: + + To do first +++++++++++ @@ -107,6 +111,9 @@ daemon process started, pid=26230 +See :ref:`zodb` to know more about ZODB storage alternatives. + + Create application instance +++++++++++++++++++++++++++ @@ -187,6 +194,8 @@ Packages download, compilation and installation can easily take several minutes. Install errors are generally due to missing libraries development headers. +See :ref:`appinstall` to get more detailed information about installation process. + Start application +++++++++++++++++ diff -r 8c0c5400e770 -r e1cb9c606859 src/source/scripts.rst --- a/src/source/scripts.rst Mon Mar 05 10:40:40 2018 +0100 +++ b/src/source/scripts.rst Tue Mar 06 11:06:45 2018 +0100 @@ -2,3 +2,4 @@ PyAMS maintenance scripts ========================= + diff -r 8c0c5400e770 -r e1cb9c606859 src/source/zodb.rst --- a/src/source/zodb.rst Mon Mar 05 10:40:40 2018 +0100 +++ b/src/source/zodb.rst Tue Mar 06 11:06:45 2018 +0100 @@ -148,3 +148,48 @@ Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available if you need to switch from a classic RelStorage database to a Newt database. + + +.. _ZODB.init: + +Initializing ZODB ++++++++++++++++++ + +When your ZODB is created, you have to initialize it! + +After installing PyAMS application (see :ref:`appinstall`), a database upgrade script is available which allows you +to initialize the database. See :ref:`config` and :ref:`scripts` to know more. + + +.. _ZODB.migration: + +ZODB migration +++++++++++++++ + +After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command +line script provided by RelStorage. + +This Python script is using a configuration file containing directives of both source and target databases, which can +be any storage described in the previous sections (or can even use the same storage). + +Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage: + +.. code-block:: + + %import relstorage + + + server zeo-server.mydomain:8100 + storage pyams + blob-dir /var/local/env/zeo/var/zeoclient/blobs + shared-blob-dir false + + + + keep-history false + blob-dir /var/local/env/pyams/var/relstorage/blobs + shared-blob-dir false + + dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx' + +