# HG changeset patch # User Damien Correia # Date 1544694713 -3600 # Node ID 49e4432c0a1dcea39547d8772ea296affd66cf78 # Parent 097b0c025eec9a9c1ba1d360b8323476c508dc13 Updated admin guide diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/app-install.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/app-install.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,227 @@ +.. _appinstall: + +.. _install: + +Installing PyAMS application +============================ + +**PyAMS** default installation is based on `Buildout `_ utility. + +.. tip:: + + Take a look at virtualenv_. This tool provide isolated Python environments, which are more practical than installing packages systemwide. + They also allow installing packages without administrator privileges. + +.. _virtualenv: https://virtualenv.pypa.io/en/stable/ + + +Current PyAMS version is based and validated for **Python 3.5**; your Python environment must also include a **C compiler** +as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and +eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. **Cython** compiler can also be useful to optimize +several packages. + + +.. note:: + + PyAMS default components configuration also pre-suppose that the following external services are available. + + * Mandatory services: + + - a **ZODB shared server** (see :ref:`zodb` before). Several background processes needing a concurrent access to ZODB are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db. + + - a **Memcached** or **Redis*** server, to store sessions and cache + + * Optional services: + + - an **LDAP server** for authentication + + - an **ElasticSearch server** for full text indexing (see *PyAMS_content_es* package) + + - a **WebSockets server** using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* packages). An *out of the box* environment can be built using *pyams_notify* scaffold. + + +Creating initial buildout +------------------------- + +PyAMS provides a new Pyramid scaffold, called *pyams*, generated via a *cookiecutter* template. + +A simple option to install PyAMS is to create a buildout environment including *Pyramid* and all *PyAMS* packages: + +.. code-block:: bash + + $ mkdir /var/local/ + $ pip3 install virtualenv + $ virtualenv --python=python3.5 env + $ cd env + $ . bin/activate + (env) $ pip3.5 install cookiecutter + (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams + +*CookieCutter* will ask you for a small set of input variables that you can change or not (**parameter** [*default value*]:) + +- **pyams_release** [*latest*] : version of PyAMS configuration file to use. "latest" will point to last release; + you can also choose to point to a given release ("0.1.4" for example) + +- **project_name** [*PyAMS*]: current environment name in "human form" + +- **project_slug**: [*pyams*] "technical" package name, based on project name + +- **virtual_hostname** [*pyams.mydomain.com*]: Apache virtual-host name + +- **webapp_name** [*webapps*]: web application package name + +- **webapp_port** [*6543*]: TCP/IP port to use when running application outside Apache. + +- **eggs_directory** [*eggs*]: relative or absolute path to directory containing downloaded eggs; this directory can be + shared with other projects. + +- **logs_directory** [*/var/log*]: absolute path to directory containing Apache's log files + +- **run_user** [*www-data*]: user name under which Apache process will run + +- **run_group** [*www-data*]: group name under which Apache process will run + +- **beaker_backend** : name of Beaker backend to use to store sessions and cache data ("redis" as default) + +- **beaker_server** [*127.0.0.1:6379*]: IP address and port of Beaker backend server + +- **db_type** [*zeo*]: ZODB database storage; available options include ZEO, RelStorage and NewtDB + +- **db_host** [*127.0.0.1*]: IP address of database server ("127.0.0.1" as default); WARNING: database server installation + is not part of application installation; another "zeo_server" cookiecutter recipe is available for ZEO + +- **db_port** [*8100*]: listening port of database server ("8100" is given as default for ZEO) + +- **db_name** [*pyams*]: database or ZEO storage name to use + +- **db_username**: database user name + +- **db_password**: database password + +- **zeo_realm**: ZEO authentication realm + +- **blobs_dir** [*/var/db/blobs*]: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default + +- **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files + are available to compile PsycoPG2 extension + +- **use_oracle**: specify if Oracle access is required; if so, please check that Oracle development files are + available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below) + +- **use_ldap**: specify if LDAP access will be required for authentication + +- **use_elasticsearch**: specify if an ElasticSearch server will be used for indexation + +- **elasticsearch_server**: URL used to access Elasticsearch server ("http://127.0.0.1:9200" as default); this URL can + include login and password ("http://login:password@127.0.0.1:9200"), if required... + +- **elasticsearch_index**: name of Elasticsearch index to use ("pyams" as default) + +- **create_elasticsearch_index**: specify if Elasticsearch index should be created after installation is complete + +- **define_elasticsearch_mappings** : specify if Elasticsearch mappings should be defined after installation is complete + +- **smtp_server**: DNS name of SMTP server ("localhost" as default) + +- **smtp_server_name**: "human" name given to SMTP server ("pyams" as default) + +- **pyams_scheduler** [*127.0.0.1:5555*]: TCP/IP address and port to use to access PyAMS tasks scheduler process (see :ref:`pyams_scheduler`) + +- **start_scheduler** [*True*]: boolean value to indicate if scheduler process is started by this application instance + +- **pyams_medias_converter** [*127.0.0.1:5556*]: TCP/IP address and port to use to access PyAMS medias converter process (see :ref:`pyams_media`) + +- **start_medias_converter** [*True*]: boolean value to indicate if medias converter process is started by this application + instance + +- **pyams_es_indexer** [*127.0.0.1:5557*] : TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (see :ref:`pyams_content_es`) + +- **start_es_indexer** boolean value to indicate if Elasticsearch indexer process is started by this application + instance + +- **use_notifications**: specify if PyAMS notifications services are to be used (see :ref:`pyams_notify`) + +- **pyams_ws_notify** [*127.0.0.1:8081*]: TCP/IP address and port of PyAMS websockets server managing notifications service + +- **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default) + +- **extension_package**: name of a PyAMS extension package to include in environment configuration + +- **need_pyams_gis**: specify if PyAMS GIS features are to be used by given extension package; if so, please check + that *libgdal* development files are available; on Debian (and maybe others), you have to specify environment + variables (see below). + +.. tip:: + + 1. Setting any option to empty string will keep the default proposed value. + 2. Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*; + +You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove +packages or update settings to your needs. + +This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual +environment. If you encounter any compile error, just install the required dependencies and restart the buildout. + +Some dependencies can require the definition of custom environment variables before running *buildout*, like: + +- for *libgdal*, which is required by **PyAMS_gis** package, use: + +.. code-block:: bash + + (env) $ export C_INCLUDE_PATH=/usr/include/gdal + (env) $ export CPLUS_INCLUDE_PATH=/usr/include/gdal + + +.. WARNING:: + You have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS + configuration file (actually 2.1.0). + +- for *cx_Oracle*, which is required if you use Oracle database connections, use: + +.. code-block:: bash + + (env) $ export ORACLE_HOME=/usr/lib/oracle/12.1/client64 + +These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux +distribution and packages versions. + + +**Then finalize Bootstrap initialization:** + +.. code-block:: bash + + (env) $ python3.5 bootstrap.py + (env) $ ./bin/buildout + + +.. seealso:: + + `zc.buildout `_ is a powerful + tool for creating repeatable builds of a given software configuration + and environment. + + +Environment settings +-------------------- + +The project generated from *pyams* scaffold is based on default Pyramid's *zodb* scaffold, but it adds: + +- a custom application factory, in the *webapp* directory (see :ref:`site`) + +- a set of directories to store runtime data, in the *var* directory; each directory contains a *README.txt* file + which should be self-explanatory to indicate what this directory should contain, including a ZEO cache + +- a set of configuration files, in the *etc* directory; here are standard *development.ini* and *production.ini* + configuration files, a ZODB configuration files (*zodb-zeo.conf*) for a ZEO client storage and two Apache + configurations (for Apache 2.2 and 2.4) using *mod_wsgi*. + +Once the project have been created from the scaffold, you are free to update all the configuration files. + +If you need to add packages to the environment, you have to add them to the *buildout.cfg* file **AND** to the INI +file (in the *pyramid.includes* section) before running the *buildout* another time; don't forget to add the +requested version at the end of *buildout.cfg* file, as Buildout is not configured by default to automatically +download the last release of a given unknown package. + +*development.ini* and *production.ini* files contain many commented directives related to PyAMS components. Read and +update them carefully before initializing your application database! + diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/app-notify.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/app-notify.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,4 @@ +.. _notify: + +Installing PyAMS notification services +====================================== diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/app-plugins.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/app-plugins.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,209 @@ +.. _plugins: + +PyAMS additional features and services +====================================== + + +Elasticsearch ++++++++++++++ + +At first you need to install ElasticSearch (ES); PyAMS is actually compatible with version 5.4. The Ingest attachment +plug-in is also required to handle attachments correctly. + +Visit https://www.elastic.co/ to learn how to install Elasticsearch Server and `ingest-attachment` plug-in + + +.. tip:: Documentation for installing ElasticSearch 6.4 + + - https://www.elastic.co/guide/en/elasticsearch/reference/6.4/gs-installation.html + - https://www.elastic.co/guide/en/elasticsearch/plugins/4.4/ingest-attachment.html + + +After Elasticsearch installation, following steps describe how to configure ES with PyAMS. + + +Initializing Elasticsearch index +-------------------------------- + +If you want to use an Elasticsearch index, you have to initialize index settings and mappings; +Elasticsearch integration is defined through the *PyAMS_content_es* package. + + +1. Enable service +''''''''''''''''' + +In Pyramid INI application files (*etc/development.ini* and *etc/production.ini*): + +.. code-block:: ini + + # Elasticsearch server settings + elastic.server = http://127.0.0.1:9200 + elastic.index = pyams + +Where: + - **elastic.server**: address of Elasticsearch server; you can include authentication arguments in the form + *http://login:password@w.x.y.z:9200* + - **elastic.index**: name of Elasticsearch index. + + +On startup, main PyAMS application process can start in *indexer* process which will handle indexing requests in +asynchronous mode; this process settings are defined like this: + +.. code-block:: ini + + # PyAMS content Elasticsearch indexer process settings + pyams_content.es.tcp_handler = 127.0.0.1:5557 + pyams_content.es.start_handler = false + pyams_content.es.allow_auth = admin:admin + pyams_content.es.allow_clients = 127.0.0.1 + +Where: + - **pyams_content.es.tcp_handler**: IP address and listening port of PyAMS indexer process + - **pyams_content.es.start_handler**: if *true*, the indexer process is started on PyAMS startup; otherwise (typically + in a cluster configuration), the process is supposed to be started from another *master* server + - **pyams_content.es.allow_auth**: login and password to be used to connect to indexer process (settings are defined + in the same way on indexer process and on all it's clients) + - **pyams_content.es.allow_clients**: list of IP addresses allowed to connect to indexer process. + + +2. Initialize Elasticsearch database +'''''''''''''''''''''''''''''''''''' + +Configuration files for attachment pipeline, index and mappings settings are available into `pyams_content_es` source +package or in PyAMS installation folder: + + +.. code-block:: bash + + (env) $ cd docs/elasticsearch + (env) $ curl --noproxy localhost -XPUT http://localhost:9200/_ingest/pipeline/attachment -d @attachment-pipeline.json + + +And with ``elastic.index = pyams`` defined as Elasticsearch index name: *"http://localhost:9200/pyams"*: + +.. code-block:: shell + + (env) $ curl -XDELETE http://localhost:9200/pyams + + (env) $ curl -XPUT http://localhost:9200/pyams -d @index-settings.json + + (env) $ curl -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json + (env) $ curl -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json + (env) $ curl -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json + + +*Troubleshooting*: If you have a 406 error try to add ``-H 'Content-Type: application/json'`` in Curl command lines. + + +3. Update index contents +'''''''''''''''''''''''' + +If your ZODB database already store contents, you can update ElasticSearch indexes with all these contents with +``pymas_es_index`` command line script. From a shell: + +.. code-block:: bash + + (env) $ ./bin/pyams_es_index ../etc/development.ini + + + +Natural Language Toolkit - NLTK ++++++++++++++++++++++++++++++++ + +PyAMS is using NLTK features through the *PyAMS_calalog*. + +.. seealso:: + + Visit https://www.nltk.org/ to learn more about NLTK + + +Initializing NLTK (Natural Language ToolKit) +-------------------------------------------- + +Some NLTK collections like **tokenizers** and **stopwords** utilities are used to index fulltext contents +elements. You can enhance NLTK indexation according to your own needs. This package requires downloading and +configuration of several elements which are done as follow: + + +*1. Run the Python shell into PyAMS environment:* + +.. code-block:: bash + + (env) $ ./bin/py + + +*2. In the Python shell:* + +.. code-block:: pycon + + >>> import nltk + >>> nltk.download() + + +*3. Configuration installation directory:* + +.. tip:: + + On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running + Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and + '*/usr/local/lib/nltk_data*' + + Please check if you have permission to write to this directory! + + +.. code-block:: shell + + NLTK Downloader + --------------------------------------------------------------------------- + d) Download l) List u) Update c) Config h) Help q) Quit + --------------------------------------------------------------------------- + Downloader> c + + Data Server: + - URL: + - 6 Package Collections Available + - 107 Individual Packages Available + + Local Machine: + - Data directory: /home/tflorac/nltk_data + + Config> d + New directory> /usr/local/lib/nltk_data + + +*4. Return to the main menu:* + +.. code-block:: shell + + --------------------------------------------------------------------------- + s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu + --------------------------------------------------------------------------- + Config> m + + +*5. Download utilities:* + + punkt + Punkt Tokenizer Models + stopwords + Stopwords Corpus + + +.. code-block:: shell + + --------------------------------------------------------------------------- + d) Download l) List u) Update c) Config h) Help q) Quit + --------------------------------------------------------------------------- + Downloader> d + Download which package (l=list; x=cancel)? + Identifier> punkt + Downloading package punkt to /usr/local/lib/nltk_data... + Downloader> d + Download which package (l=list; x=cancel)? + Identifier> stopwords + Downloading package stopwords to /usr/local/lib/nltk_data... + + +.. tip:: + + The full list of NTLK Collection can be displayed with the ``l) list`` option. diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/app-start.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/app-start.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,95 @@ +.. _appstart: + +Starting the application +======================== + +When database upgrade process has ended ( :ref:`zodbinit` ) , you can start the web application process with the standard Pyramid's +*pserve* command line tool: + +.. code-block:: bash + + (env) $ ./bin/pserve etc/development.ini + +In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO): + +.. code-block:: bash + + 2018-01-14 11:37:54,339 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=28695) created RW/normal for storage: 'pyams' + 2018-01-14 11:37:54,340 INFO [ZEO.cache][MainThread] created temporary cache file 3 + 2018-01-14 11:37:54,345 INFO [ZODB.blob][MainThread] (28695) Blob directory `/var/local/env/pyams/var/db/blobs` is used but has no layout marker set. Selected `lawn` layout. + 2018-01-14 11:37:54,345 WARNI [ZODB.blob][MainThread] (28695) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout. + 2018-01-14 11:37:54,346 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector + 2018-01-14 11:37:54,347 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None + 2018-01-14 11:37:54,348 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting + 2018-01-14 11:37:54,349 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) + 2018-01-14 11:37:54,355 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) + 2018-01-14 11:37:54,358 DEBUG [txn.140663320073984][MainThread] new transaction + 2018-01-14 11:37:54,360 DEBUG [txn.140663320073984][MainThread] commit + 2018-01-14 11:37:54,484 DEBUG [config][MainThread] include /home/tflorac/Dropbox/src/PyAMS/pyams_template/src/pyams_template/configure.zcml + 2018-01-14 11:37:54,485 DEBUG [config][MainThread] include /var/local/env/pycharm/lib/python3.5/site-packages/pyramid_zcml/configure.zcml + ... + 2018-01-14 11:37:54,833 DEBUG [PyAMS (utils)][MainThread] Registering utility named 'PyAMS timezone' providing + 2018-01-14 11:37:54,834 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS timezones" + 2018-01-14 11:37:54,835 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing + 2018-01-14 11:37:54,839 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2018-01-14 11:37:54,847 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing + 2018-01-14 11:37:54,942 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing + 2018-01-14 11:37:54,943 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () + 2018-01-14 11:37:54,949 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () + 2018-01-14 11:37:54,980 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZEO connections" + 2018-01-14 11:37:54,981 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZODB connections" + 2018-01-14 11:37:55,015 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "add-zeo-connection.html" for () + 2018-01-14 11:37:55,016 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing + 2018-01-14 11:37:55,017 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () + ... + 2018-01-14 11:41:13,214 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2018-01-14 11:43:36,665 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' + 2018-01-14 11:43:36,665 INFO [ZEO.cache][MainThread] created temporary cache file 9 + 2018-01-14 11:43:36,673 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector + 2018-01-14 11:43:36,674 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) + 2018-01-14 11:43:36,674 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None + 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 + 2018-01-14 11:43:36,675 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting + 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> + 2018-01-14 11:43:36,677 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) + 2018-01-14 11:43:36,679 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) + 2018-01-14 11:43:36,682 DEBUG [txn.140713340237568][MainThread] new transaction + 2018-01-14 11:43:36,683 DEBUG [txn.140713340237568][MainThread] commit + 2018-01-14 11:43:36,690 INFO [PyAMS (scheduler][MainThread] Starting tasks scheduler ... + 2018-01-14 11:43:36,698 INFO [PyAMS (scheduler][MainThread] Started tasks scheduler with PID 29361. + 2018-01-14 11:43:36,701 INFO [apscheduler.scheduler][MainThread] Scheduler started + 2018-01-14 11:43:36,702 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run + 2018-01-14 11:43:36,704 DEBUG [apscheduler.scheduler][APScheduler] No jobs; waiting until a job is added + 2018-01-14 11:43:36,719 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' + 2018-01-14 11:43:36,720 INFO [ZEO.cache][MainThread] created temporary cache file 15 + 2018-01-14 11:43:36,724 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector + 2018-01-14 11:43:36,725 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None + 2018-01-14 11:43:36,726 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting + 2018-01-14 11:43:36,727 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) + 2018-01-14 11:43:36,728 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) + 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 + 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> + 2018-01-14 11:43:36,732 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) + 2018-01-14 11:43:36,735 DEBUG [txn.140713340237568][MainThread] new transaction + 2018-01-14 11:43:36,736 DEBUG [txn.140713340237568][MainThread] commit + 2018-01-14 11:43:36,743 INFO [PyAMS (media)][MainThread] Starting medias converter ... + 2018-01-14 11:43:36,751 INFO [PyAMS (media)][MainThread] Started medias converter with PID 29367. + Starting server in PID 29335. + Serving on http://0.0.0.0:6543 + + +Congrats: PyAMS is running ! + +Launch a browser and open URL http://127.0.0.1:6543/admin to get access to PyAMS management interface. + +.. warning:: + + By default: + + - Login: admin + + - Password: admin + + **You must change it as soon as possible!** + (see :ref:`pyams_security`)!!. + diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/app-tuning.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/app-tuning.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,80 @@ +.. _tuning: + +Application tuning +================== + +.. Note:: + + PyAMS application configuration is done via several complementary ways: + + - via **Pyramid's configuration** files located in 'etc/' folder + + - via **ZCML files** . + + - via **Site Management** to setup Local utilities defined into ZODB (see :ref:`appmanage`) + + +Pyramid's configuration files +----------------------------- + +PyAMS is using default Pyramid's ini_ configuration files to define settings. When using default PyAMS scaffold, these +INI files are stored into *etc* folder. Two files are created by scaffold: a development file (*development.ini*) and +a production file (*production.ini*) which contain nearly the same directives, except for debug toolbar (activated only +in development), Fanstatic configuration and logging. + +.. _ini: https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/project.html#development-ini + +These files are actually documented, but here are a few important directives: + +- **zodbconn.uri**: (ex: zconfig:/zodb-relstorage.conf) + +- **pyams.application_name**: Name of your own application +- **pyams.application_factory**: Configuration module path to your custom application. + +- **pyams_xxx.tcp_handler** : Adress of the tcp handler for the service associated to the pyams package +- **pyams_xxx.start_handler**: True / False + +- **pyams_content.config.xxx_tool_factory** and **pyams_content.config.xxx_tool_name**: You can override default factory by defining these settings or by providing your own adapters. + + + +Pyramid's configuration files powered by PyAMS_utils +---------------------------------------------------- + +On application startup, if PyAMS_utils package is included into Pyramid configuration, several operations take +place: + + - a custom **site factory** is defined + + - custom request methods are defined + + - a custom **traverser** handling **namespaces** is defined + + - a custom subscribers predicate based on interfaces support is defined + + - several adapters are registered, to handle annotations and key references + + - custom TALES extensions are registered. + +The site factory is an important component in this process. It is this factory which will define the application root +and create a **local site manager**. + +Pyramid application is loaded from ZODB's root via a key defined in Pyramid's configuration file; the key is named +*pyams.application_name* and it's default value is *application*. + +If the application can't be found, PyAMS is looking for an application class name in Pyramid's configuration file; the +class name configuration key is called *pyams.application_factory* and defined by default as +*pyams_utils.site.BaseSiteRoot*. PyAMS default site factory will then create the application, and add a local site +manager to it (see :ref:`zca`). + +After application creation, a :py:class:`NewLocalSiteCreatedEvent ` is +notified. Custom packages can subscribe to this event to register custom components. + + + +ZCML configuration files +------------------------ + +Adding more ZCML configuration directives is not mandatory in any way but can be useful in several use cases. For +example, templates can't be overriden via Python annotations without creating a new subclass; this can be done +easily via ZCML. diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/architecture.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/architecture.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,47 @@ +.. _architecture: + +PyAMS application architecture +============================== + +PyAMS applications are built on a small set of prerequisites and external components which *can* or, for some of them, +*must* be included, the minimal configuration is: + +* a mandatory shared ZODB; :ref:`zodb` chapter + +* a Redis or Memcached server, to handle cache and sessions; + :ref:`pyams_cache` chapter + + +Single server deployment +------------------------ + +Overview of the environment system after the :ref:`quickstart` installation. +It's a very lite configuration, with only mandatory resources. +Obviously you can extend this system by connecting more resources like is described in the :ref:`config` chapiter + + +.. image:: ../_static/architecture_quickstart.png + + + +Development deployment +------------------------ + +In additon of mandatory resource **PyAMS** could work with: + +* a websockets server (GuniCorn), used to handle desktop notifications; + you can find information about this component in the :ref:`pyams_notify` chapter; + +* an Elasticsearch index, used to handle quick and optimized searching. + + +.. image:: ../_static/architecture_dev.png + + +Advanced deployment +------------------- + +Use case in an advanced environment. + +.. image:: ../_static/architecture_prod.png + diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/pyams-install.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/pyams-install.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,14 @@ +Installing PyAMS +================ + + +.. toctree:: + + Installing database + app-install + app-notify + app-plugins + app-start + + + diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/scripts.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/scripts.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,5 @@ +.. _scripts: + +Command line scripts +==================== + diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/security.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/security.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,4 @@ +.. _security: + +PyAMS Security model +==================== diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/tasks.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/tasks.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,4 @@ +.. _tasks: + +Scheduling tasks +================ diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/troubleshooting.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/troubleshooting.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,4 @@ +.. _troubleshooting: + +Troubleshooting +=============== diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/zodb-init.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/zodb-init.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,17 @@ + +.. _zodbinit: + +Initializing the ZODB database +------------------------------ + +After, you have downloaded and installed all required packages, you have to initialize the database so that all +required components are available. + +From a shell, just type: + +.. code-block:: bash + + (env) $ ./bin/pyams_upgrade etc/development.ini + +This process requires that every package is correctly included into *pyramid.includes* directive from selected +configuration file. diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/zodb-support.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/zodb-support.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,49 @@ +.. _dbsupport: + +Database maintenance tasks +========================== + + +pyams_upgrade +------------- + + +Pyramid allows to define custom command line scripts for application management. A script called *pyams_upgrade* is +provided by PyAMS_utils package; this script apply the same process as PyAMS site factory, but can also be used to +manage **database generations**. The idea behind this is just to allow custom packages to provide a way to check and +upgrade database configuration away from application startup process: + +.. code-block:: bash + + # ./bin/pyams_upgrade webapp/development.ini + + +A **site generation checker** is just a named utility providing :py:class:`pyams_utils.interfaces.site.ISiteGenerations` +interface. For example, **pyams_security** package provides such utility, to make sure that local site manager +contains a PyAMS security manager and a principal annotation utility: + +.. code-block:: python + + from pyams_utils.site import check_required_utilities + + REQUIRED_UTILITIES = ((ISecurityManager, '', SecurityManager, 'Security manager'), + (IPrincipalAnnotationUtility, '', PrincipalAnnotationUtility, 'User profiles')) + + @utility_config(name='PyAMS security', provides=ISiteGenerations) + class SecurityGenerationsChecker(object): + """I18n generations checker""" + + generation = 1 + + def evolve(self, site, current=None): + """Check for required utilities""" + check_required_utilities(site, REQUIRED_UTILITIES) + +:py:func:`check_required_utilities ` is a PyAMS_utils utility function which +can to used to verify that a set of local utilities are correctly registered with the given names and interfaces. + + +zodbupdate +---------- + +:ref:`renamehowto` diff -r 097b0c025eec -r 49e4432c0a1d src/source/admin_guide/zodb.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/admin_guide/zodb.rst Thu Dec 13 10:51:53 2018 +0100 @@ -0,0 +1,209 @@ +.. _zodb: + + +ZODB server +============ + + +Installation +'''''''''''' + +PyAMS relies mainly on a ZODB (**\Z**\ope **\O**\bjects **\D**\ata\ **\B**\ase) to store it's configuration and for exemple *PyAMS_content*, but other packages may +rely on other database(s). + +Concurrent accesses are **required** on the ZODB, to fully enjoy PyAMS. +Several ZODB storages implementations providing a shared access are available and the installation are describe below: + + * `ZEO Server`_ + * `RelStorage`_ + * `Newt.DB`_ + + +.. note:: + + When you start your application in "*single process*" mode, PyAMS packages start several processes + and the "*synchronization*" is done via **ØMQ**. + + + +.. _`ZEO Server`: + +1. ZEO server ++++++++++++++ + +ZEO (**\Z**\ope **\E**\nterprise **\O**\bjects) is the first available implementation available for concurrent access to a +FileStorage, provided through the ZEO package. + +.. seealso:: + + ZEO package documentation and complete configuration settings are available on PyPI_ (cf: `ZOE 5.2.0`_). + +.. _PyPI: https://pypi.python.org/pypi/ZEO +.. _`ZOE 5.2.0`: https://pypi.python.org/pypi/ZEO/5.2.0 + + +a) Installation with initial buildout +------------------------------------- + +PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template. + +A simple option to create a ZEO server is to create a buildout environment including *ZEO* and *ZODB* packages: + +.. code-block:: bash + + $ mkdir /var/local/ + $ pip3 install virtualenv + $ virtualenv --python=python3.5 env + $ cd env + $ . bin/activate + (env) $ pip3.5 install cookiecutter + (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server + +*CookieCutter* will ask you for a small set of input variables that you can change or not: + +- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release; + you can also choose to point to a given release ("0.1.4" for example) + +- **project_name**: current environment name in "human form" + +- **project_slug**: "technical" package name, based on project name + +- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be + shared with other projects ("eggs" as default) + +- **run_user**: user name under which ZEO process will run ("zeo" as default) + +- **run_group**: group name under which ZEO process will run ("zeo" as default) + +- **zeo_server_port**: listening port of ZEO server ("8100" as default) + +- **zeo_storage**: name of first ZEO storage ("pyams" as default) + +- **zeo_pack_report**: email address to which pack reports should be sent + +- **logs_directory**: absolute path to directory containing ZEO's log files. + +A message is displayed after initialization to finalize environment creation: + +.. code:: + + Your ZEO environment is initialized. + To finalize it''s creation, just type: + - cd zeo_server + - python3.5 bootstrap.py + - ./bin/buildout + + To initialize authentication database, please run following command after buildout: + ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx + +b) ZEO server configuration +--------------------------- + +All ZEO configuration files are generated in "etc" subdirectory. These includes: + +- **etc/zeo_server-zdaemon.conf**: ZDaemon configuration file + +- **etc/zeo_server-zeo.conf**: ZEO server configuration file + +- **etc/auth.db**: ZEO authentication file; WARNING: this file is not created automatically, you have to create it + after buildout. + +In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during +*CookieCutter* template creation. + + +c) ZEO server tools +------------------- + +A set of system configuration files are produced to handle your ZEO environment. These includes: + +- **etc/init.d/zeo-zeo_server**: ZEO server start/stop script in Init-D format. Create a link to this file in + */etc/init.d* and update Init.d scripts (*update-rc.d zeo-zeo_server defaults*) to include ZEO in server start/stop + process. You can also use this script to start/stop ZEO by hand with *start* and *stop* arguments. + +- **etc/systemd/zeo-zeo_server.service**: SystemD service configuration file for ZEO server. Create a link to this + file in */etc/systemd/system* and reload SystemD daemon (*systemctl daemon-reload*) before activating ZEO service + (*systemctl enable zeo-zeo_server.service* and *systemctl start zeo-zeo_server.service*). + +- **etc/logrotate.d/zeo-zeo_server**: LogRotate configuration file for ZEO log files. Create a link to this file in + */etc/logrotate.d* to activate log rotation for ZEO server. + +- **etc/cron.d/pack-zeo-zeo_server**: Cron configuration file for ZEO database packing. Just create a link to this + file in */etc/cron.d* directory to enable ZODB packing on a weekly basis (by default). + +In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during +*CookieCutter* template creation. All directory names are those used on a Debian GNU/Linux distribution and may have +to be changed on other distributions. + + +.. _RelStorage: + +2. RelStorage server +++++++++++++++++++++ + +RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores +Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are +supported. + +To create a database compatible with RelStorage, you just have to install the database server and create a database +dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup. + +RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get +benefit from many extensions (clustering, fail-over, hot-standby...) provided by these databases. + + +.. _Newt.DB: + +3. NewtDB server +++++++++++++++++ + +NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is +dedicated to PostgreSQL (>= 9.5). + +NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB +column. The JSON data supplements the native data to support indexing, search, and access from non-Python application. +Because the JSON format is lossy, compared to the native format, the native format is still used for loading +objects from the database. For this reason, the JSON data are read-only. + +Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a +convenient API for raw data searches. + +Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available +if you need to switch from a classic RelStorage database to a Newt database. + + +.. _ZODB.migration: + +Migration +''''''''' + +After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command +line script provided by RelStorage. + +This Python script is using a configuration file containing directives of both source and target databases, which can +be any storage described in the previous sections (or can even use the same storage). + +Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage: + + +.. code-block:: html + + %import relstorage + + + server zeo-server.mydomain:8100 + storage pyams + blob-dir /var/local/env/zeo/var/zeoclient/blobs + shared-blob-dir false + + + + keep-history false + blob-dir /var/local/env/pyams/var/relstorage/blobs + shared-blob-dir false + + dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx' + + + + diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/appconfig.rst --- a/src/source/admln_guide/appconfig.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,80 +0,0 @@ -.. _config: - -PyAMS configuration -=================== - -.. Note:: - - PyAMS application configuration is done via several complementary ways: - - - via **Pyramid's configuration** files located in 'etc/' folder - - - via **ZCML files** . - - - via **Site Management** to setup Local utilities defined into ZODB (see :ref:`appmanage`) - - -Pyramid's configuration files ------------------------------ - -PyAMS is using default Pyramid's ini_ configuration files to define settings. When using default PyAMS scaffold, these -INI files are stored into *etc* folder. Two files are created by scaffold: a development file (*development.ini*) and -a production file (*production.ini*) which contain nearly the same directives, except for debug toolbar (activated only -in development), Fanstatic configuration and logging. - -.. _ini: https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/project.html#development-ini - -These files are actually documented, but here are a few important directives: - -- **zodbconn.uri**: (ex: zconfig:/zodb-relstorage.conf) - -- **pyams.application_name**: Name of your own application -- **pyams.application_factory**: Configuration module path to your custom application. - -- **pyams_xxx.tcp_handler** : Adress of the tcp handler for the service associated to the pyams package -- **pyams_xxx.start_handler**: True / False - -- **pyams_content.config.xxx_tool_factory** and **pyams_content.config.xxx_tool_name**: You can override default factory by defining these settings or by providing your own adapters. - - - -Pyramid's configuration files powered by PyAMS_utils ----------------------------------------------------- - -On application startup, if PyAMS_utils package is included into Pyramid configuration, several operations take -place: - - - a custom **site factory** is defined - - - custom request methods are defined - - - a custom **traverser** handling **namespaces** is defined - - - a custom subscribers predicate based on interfaces support is defined - - - several adapters are registered, to handle annotations and key references - - - custom TALES extensions are registered. - -The site factory is an important component in this process. It is this factory which will define the application root -and create a **local site manager**. - -Pyramid application is loaded from ZODB's root via a key defined in Pyramid's configuration file; the key is named -*pyams.application_name* and it's default value is *application*. - -If the application can't be found, PyAMS is looking for an application class name in Pyramid's configuration file; the -class name configuration key is called *pyams.application_factory* and defined by default as -*pyams_utils.site.BaseSiteRoot*. PyAMS default site factory will then create the application, and add a local site -manager to it (see :ref:`zca`). - -After application creation, a :py:class:`NewLocalSiteCreatedEvent ` is -notified. Custom packages can subscribe to this event to register custom components. - - - -ZCML configuration files ------------------------- - -Adding more ZCML configuration directives is not mandatory in any way but can be useful in several use cases. For -example, templates can't be overriden via Python annotations without creating a new subclass; this can be done -easily via ZCML. diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/appinstall.rst --- a/src/source/admln_guide/appinstall.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,244 +0,0 @@ -.. _appinstall: - -.. _install: - -Installing PyAMS application -============================ - -**PyAMS** default installation is based on `Buildout `_ utility. - -.. tip:: - - Take a look at virtualenv_. This tool provide isolated Python environments, which are more practical than installing packages systemwide. - They also allow installing packages without administrator privileges. - -.. _virtualenv: https://virtualenv.pypa.io/en/stable/ - - -Current PyAMS version is based and validated for **Python 3.5**; your Python environment must also include a **C compiler** -as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and -eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. **Cython** compiler can also be useful to optimize -several packages. - - -.. note:: - - PyAMS default components configuration also pre-suppose that the following external services are available. - - * Mandatory services: - - - a **ZODB shared server** (see :ref:`zodb` before). Several background processes needing a concurrent access to ZODB are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db. - - - a **Memcached** or **Redis*** server, to store sessions and cache - - * Optional services: - - - an **LDAP server** for authentication - - - an **ElasticSearch server** for full text indexing (see *PyAMS_content_es* package) - - - a **WebSockets server** using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* packages). An *out of the box* environment can be built using *pyams_notify* scaffold. - - -Creating initial buildout -------------------------- - -PyAMS provides a new Pyramid scaffold, called *pyams*, generated via a *cookiecutter* template. - -A simple option to install PyAMS is to create a buildout environment including *Pyramid* and all *PyAMS* packages: - -.. code-block:: bash - - $ mkdir /var/local/ - $ pip3 install virtualenv - $ virtualenv --python=python3.5 env - $ cd env - $ . bin/activate - (env) $ pip3.5 install cookiecutter - (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams - -*CookieCutter* will ask you for a small set of input variables that you can change or not (**parameter** [*default value*]:) - -- **pyams_release** [*latest*] : version of PyAMS configuration file to use. "latest" will point to last release; - you can also choose to point to a given release ("0.1.4" for example) - -- **project_name** [*PyAMS*]: current environment name in "human form" - -- **project_slug**: [*pyams*] "technical" package name, based on project name - -- **virtual_hostname** [*pyams.mydomain.com*]: Apache virtual-host name - -- **webapp_name** [*webapps*]: web application package name - -- **webapp_port** [*6543*]: TCP/IP port to use when running application outside Apache. - -- **eggs_directory** [*eggs*]: relative or absolute path to directory containing downloaded eggs; this directory can be - shared with other projects. - -- **logs_directory** [*/var/log*]: absolute path to directory containing Apache's log files - -- **run_user** [*www-data*]: user name under which Apache process will run - -- **run_group** [*www-data*]: group name under which Apache process will run - -- **beaker_backend** : name of Beaker backend to use to store sessions and cache data ("redis" as default) - -- **beaker_server** [*127.0.0.1:6379*]: IP address and port of Beaker backend server - -- **db_type** [*zeo*]: ZODB database storage; available options include ZEO, RelStorage and NewtDB - -- **db_host** [*127.0.0.1*]: IP address of database server ("127.0.0.1" as default); WARNING: database server installation - is not part of application installation; another "zeo_server" cookiecutter recipe is available for ZEO - -- **db_port** [*8100*]: listening port of database server ("8100" is given as default for ZEO) - -- **db_name** [*pyams*]: database or ZEO storage name to use - -- **db_username**: database user name - -- **db_password**: database password - -- **zeo_realm**: ZEO authentication realm - -- **blobs_dir** [*/var/db/blobs*]: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default - -- **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files - are available to compile PsycoPG2 extension - -- **use_oracle**: specify if Oracle access is required; if so, please check that Oracle development files are - available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below) - -- **use_ldap**: specify if LDAP access will be required for authentication - -- **use_elasticsearch**: specify if an ElasticSearch server will be used for indexation - -- **elasticsearch_server**: URL used to access Elasticsearch server ("http://127.0.0.1:9200" as default); this URL can - include login and password ("http://login:password@127.0.0.1:9200"), if required... - -- **elasticsearch_index**: name of Elasticsearch index to use ("pyams" as default) - -- **create_elasticsearch_index**: specify if Elasticsearch index should be created after installation is complete - -- **define_elasticsearch_mappings** : specify if Elasticsearch mappings should be defined after installation is complete - -- **smtp_server**: DNS name of SMTP server ("localhost" as default) - -- **smtp_server_name**: "human" name given to SMTP server ("pyams" as default) - -- **pyams_scheduler** [*127.0.0.1:5555*]: TCP/IP address and port to use to access PyAMS tasks scheduler process (see :ref:`pyams_scheduler`) - -- **start_scheduler** [*True*]: boolean value to indicate if scheduler process is started by this application instance - -- **pyams_medias_converter** [*127.0.0.1:5556*]: TCP/IP address and port to use to access PyAMS medias converter process (see :ref:`pyams_media`) - -- **start_medias_converter** [*True*]: boolean value to indicate if medias converter process is started by this application - instance - -- **pyams_es_indexer** [*127.0.0.1:5557*] : TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (see :ref:`pyams_content_es`) - -- **start_es_indexer** boolean value to indicate if Elasticsearch indexer process is started by this application - instance - -- **use_notifications**: specify if PyAMS notifications services are to be used (see :ref:`pyams_notify`) - -- **pyams_ws_notify** [*127.0.0.1:8081*]: TCP/IP address and port of PyAMS websockets server managing notifications service - -- **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default) - -- **extension_package**: name of a PyAMS extension package to include in environment configuration - -- **need_pyams_gis**: specify if PyAMS GIS features are to be used by given extension package; if so, please check - that *libgdal* development files are available; on Debian (and maybe others), you have to specify environment - variables (see below). - -.. tip:: - - 1. Setting any option to empty string will keep the default proposed value. - 2. Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*; - -You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove -packages or update settings to your needs. - -This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual -environment. If you encounter any compile error, just install the required dependencies and restart the buildout. - -Some dependencies can require the definition of custom environment variables before running *buildout*, like: - -- for *libgdal*, which is required by **PyAMS_gis** package, use: - -.. code-block:: bash - - (env) $ export C_INCLUDE_PATH=/usr/include/gdal - (env) $ export CPLUS_INCLUDE_PATH=/usr/include/gdal - - -.. WARNING:: - You have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS - configuration file (actually 2.1.0). - -- for *cx_Oracle*, which is required if you use Oracle database connections, use: - -.. code-block:: bash - - (env) $ export ORACLE_HOME=/usr/lib/oracle/12.1/client64 - -These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux -distribution and packages versions. - - -**Then finalize Bootstrap initialization:** - -.. code-block:: bash - - (env) $ python3.5 bootstrap.py - (env) $ ./bin/buildout - - -.. seealso:: - - `zc.buildout `_ is a powerful - tool for creating repeatable builds of a given software configuration - and environment. - - -Environment settings --------------------- - -The project generated from *pyams* scaffold is based on default Pyramid's *zodb* scaffold, but it adds: - -- a custom application factory, in the *webapp* directory (see :ref:`site`) - -- a set of directories to store runtime data, in the *var* directory; each directory contains a *README.txt* file - which should be self-explanatory to indicate what this directory should contain, including a ZEO cache - -- a set of configuration files, in the *etc* directory; here are standard *development.ini* and *production.ini* - configuration files, a ZODB configuration files (*zodb-zeo.conf*) for a ZEO client storage and two Apache - configurations (for Apache 2.2 and 2.4) using *mod_wsgi*. - -Once the project have been created from the scaffold, you are free to update all the configuration files. - -If you need to add packages to the environment, you have to add them to the *buildout.cfg* file **AND** to the INI -file (in the *pyramid.includes* section) before running the *buildout* another time; don't forget to add the -requested version at the end of *buildout.cfg* file, as Buildout is not configured by default to automatically -download the last release of a given unknown package. - -*development.ini* and *production.ini* files contain many commented directives related to PyAMS components. Read and -update them carefully before initializing your application database! - - -.. _zodbinit: - -Initializing the ZODB database ------------------------------- - -After, you have downloaded and installed all required packages, you have to initialize the database so that all -required components are available. - -From a shell, just type: - -.. code-block:: bash - - (env) $ ./bin/pyams_upgrade etc/development.ini - -This process requires that every package is correctly included into *pyramid.includes* directive from selected -configuration file. diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/appstart.rst --- a/src/source/admln_guide/appstart.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,95 +0,0 @@ -.. _appstart: - -Starting the application -======================== - -When database upgrade process has ended ( :ref:`zodbinit` ) , you can start the web application process with the standard Pyramid's -*pserve* command line tool: - -.. code-block:: bash - - (env) $ ./bin/pserve etc/development.ini - -In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO): - -.. code-block:: bash - - 2018-01-14 11:37:54,339 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=28695) created RW/normal for storage: 'pyams' - 2018-01-14 11:37:54,340 INFO [ZEO.cache][MainThread] created temporary cache file 3 - 2018-01-14 11:37:54,345 INFO [ZODB.blob][MainThread] (28695) Blob directory `/var/local/env/pyams/var/db/blobs` is used but has no layout marker set. Selected `lawn` layout. - 2018-01-14 11:37:54,345 WARNI [ZODB.blob][MainThread] (28695) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout. - 2018-01-14 11:37:54,346 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector - 2018-01-14 11:37:54,347 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None - 2018-01-14 11:37:54,348 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting - 2018-01-14 11:37:54,349 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) - 2018-01-14 11:37:54,355 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) - 2018-01-14 11:37:54,358 DEBUG [txn.140663320073984][MainThread] new transaction - 2018-01-14 11:37:54,360 DEBUG [txn.140663320073984][MainThread] commit - 2018-01-14 11:37:54,484 DEBUG [config][MainThread] include /home/tflorac/Dropbox/src/PyAMS/pyams_template/src/pyams_template/configure.zcml - 2018-01-14 11:37:54,485 DEBUG [config][MainThread] include /var/local/env/pycharm/lib/python3.5/site-packages/pyramid_zcml/configure.zcml - ... - 2018-01-14 11:37:54,833 DEBUG [PyAMS (utils)][MainThread] Registering utility named 'PyAMS timezone' providing - 2018-01-14 11:37:54,834 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS timezones" - 2018-01-14 11:37:54,835 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing - 2018-01-14 11:37:54,839 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing - 2018-01-14 11:37:54,847 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing - 2018-01-14 11:37:54,942 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing - 2018-01-14 11:37:54,943 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () - 2018-01-14 11:37:54,949 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () - 2018-01-14 11:37:54,980 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZEO connections" - 2018-01-14 11:37:54,981 DEBUG [PyAMS (utils)][MainThread] Registering class as vocabulary with name "PyAMS ZODB connections" - 2018-01-14 11:37:55,015 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "add-zeo-connection.html" for () - 2018-01-14 11:37:55,016 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, ) providing - 2018-01-14 11:37:55,017 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () - ... - 2018-01-14 11:41:13,214 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing - 2018-01-14 11:43:36,665 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' - 2018-01-14 11:43:36,665 INFO [ZEO.cache][MainThread] created temporary cache file 9 - 2018-01-14 11:43:36,673 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector - 2018-01-14 11:43:36,674 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) - 2018-01-14 11:43:36,674 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None - 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 - 2018-01-14 11:43:36,675 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting - 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> - 2018-01-14 11:43:36,677 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) - 2018-01-14 11:43:36,679 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) - 2018-01-14 11:43:36,682 DEBUG [txn.140713340237568][MainThread] new transaction - 2018-01-14 11:43:36,683 DEBUG [txn.140713340237568][MainThread] commit - 2018-01-14 11:43:36,690 INFO [PyAMS (scheduler][MainThread] Starting tasks scheduler ... - 2018-01-14 11:43:36,698 INFO [PyAMS (scheduler][MainThread] Started tasks scheduler with PID 29361. - 2018-01-14 11:43:36,701 INFO [apscheduler.scheduler][MainThread] Scheduler started - 2018-01-14 11:43:36,702 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run - 2018-01-14 11:43:36,704 DEBUG [apscheduler.scheduler][APScheduler] No jobs; waiting until a job is added - 2018-01-14 11:43:36,719 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams' - 2018-01-14 11:43:36,720 INFO [ZEO.cache][MainThread] created temporary cache file 15 - 2018-01-14 11:43:36,724 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector - 2018-01-14 11:43:36,725 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected None - 2018-01-14 11:43:36,726 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting - 2018-01-14 11:43:36,727 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642) - 2018-01-14 11:43:36,728 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False) - 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0 - 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 --> - 2018-01-14 11:43:36,732 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100) - 2018-01-14 11:43:36,735 DEBUG [txn.140713340237568][MainThread] new transaction - 2018-01-14 11:43:36,736 DEBUG [txn.140713340237568][MainThread] commit - 2018-01-14 11:43:36,743 INFO [PyAMS (media)][MainThread] Starting medias converter ... - 2018-01-14 11:43:36,751 INFO [PyAMS (media)][MainThread] Started medias converter with PID 29367. - Starting server in PID 29335. - Serving on http://0.0.0.0:6543 - - -Congrats: PyAMS is running ! - -Launch a browser and open URL http://127.0.0.1:6543/admin to get access to PyAMS management interface. - -.. warning:: - - By default: - - - Login: admin - - - Password: admin - - **You must change it as soon as possible!** - (see :ref:`pyams_security`)!!. - diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/architecture.rst --- a/src/source/admln_guide/architecture.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,47 +0,0 @@ -.. _architecture: - -PyAMS application architecture -============================== - -PyAMS applications are built on a small set of prerequisites and external components which *can* or, for some of them, -*must* be included, the minimal configuration is: - -* a mandatory shared ZODB; :ref:`zodb` chapter - -* a Redis or Memcached server, to handle cache and sessions; - :ref:`pyams_cache` chapter - - -Quickstart deployment ---------------------- - -Overview of the environement system after the :ref:`quickstart` installation. -It's a very lite configuration, with only mandatory resourses. -Obviously you can extend this system by connecting more resources like is described in the :ref:`config` chapiter - - -.. image:: _static/architecture_quickstart.png - - - -Developpement deployment ------------------------- - -In additon of mandatory resource **PyAMS** could work with: - -* a websockets server (GuniCorn), used to handle desktop notifications; - you can find information about this component in the :ref:`pyams_notify` chapter; - -* an Elasticsearch index, used to handle quick and optimized searching. - - -.. image:: _static/architecture_dev.png - - -Production deployment ---------------------- - -Use case in a more complete prodution environement. - -.. image:: _static/architecture_prod.png - diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/plugins.rst --- a/src/source/admln_guide/plugins.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,209 +0,0 @@ -.. _plugins: - -PyAMS additional features and services -====================================== - - -Elasticsearch -+++++++++++++ - -At first you need to install ElasticSearch (ES); PyAMS is actually compatible with version 5.4. The Ingest attachment -plug-in is also required to handle attachments correctly. - -Visit https://www.elastic.co/ to learn how to install Elasticsearch Server and `ingest-attachment` plug-in - - -.. tip:: Documentation for installing ElasticSearch 5.4 - - - https://www.elastic.co/guide/en/elasticsearch/reference/5.4/gs-installation.html - - https://www.elastic.co/guide/en/elasticsearch/plugins/5.4/ingest-attachment.html - - -After Elasticsearch installation, following steps describe how to configure ES with PyAMS. - - -Initializing Elasticsearch index --------------------------------- - -If you want to use an Elasticsearch index, you have to initialize index settings and mappings; -Elasticsearch integration is defined through the *PyAMS_content_es* package. - - -1. Enable service -''''''''''''''''' - -In Pyramid INI application files (*etc/development.ini* and *etc/production.ini*): - -.. code-block:: ini - - # Elasticsearch server settings - elastic.server = http://127.0.0.1:9200 - elastic.index = pyams - -Where: - - **elastic.server**: address of Elasticsearch server; you can include authentication arguments in the form - *http://login:password@w.x.y.z:9200* - - **elastic.index**: name of Elasticsearch index. - - -On startup, main PyAMS application process can start in *indexer* process which will handle indexing requests in -asynchronous mode; this process settings are defined like this: - -.. code-block:: ini - - # PyAMS content Elasticsearch indexer process settings - pyams_content.es.tcp_handler = 127.0.0.1:5557 - pyams_content.es.start_handler = false - pyams_content.es.allow_auth = admin:admin - pyams_content.es.allow_clients = 127.0.0.1 - -Where: - - **pyams_content.es.tcp_handler**: IP address and listening port of PyAMS indexer process - - **pyams_content.es.start_handler**: if *true*, the indexer process is started on PyAMS startup; otherwise (typically - in a cluster configuration), the process is supposed to be started from another *master* server - - **pyams_content.es.allow_auth**: login and password to be used to connect to indexer process (settings are defined - in the same way on indexer process and on all it's clients) - - **pyams_content.es.allow_clients**: list of IP addresses allowed to connect to indexer process. - - -2. Initialize Elasticsearch database -'''''''''''''''''''''''''''''''''''' - -Configuration files for attachment pipeline, index and mappings settings are available into `pyams_content_es` source -package or in PyAMS installation folder: - - -.. code-block:: bash - - (env) $ cd docs/elasticsearch - (env) $ curl --noproxy localhost -XPUT http://localhost:9200/_ingest/pipeline/attachment -d @attachment-pipeline.json - - -And with ``elastic.index = pyams`` defined as Elasticsearch index name: *"http://localhost:9200/pyams"*: - -.. code-block:: shell - - (env) $ curl -XDELETE http://localhost:9200/pyams - - (env) $ curl -XPUT http://localhost:9200/pyams -d @index-settings.json - - (env) $ curl -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json - (env) $ curl -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json - (env) $ curl -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json - - -*Troubleshooting*: If you have a 406 error try to add ``-H 'Content-Type: application/json'`` in Curl command lines. - - -3. Update index contents -'''''''''''''''''''''''' - -If your ZODB database already store contents, you can update ElasticSearch indexes with all these contents with -``pymas_es_index`` command line script. From a shell: - -.. code-block:: bash - - (env) $ ./bin/pyams_es_index ../etc/development.ini - - - -Natural Language Toolkit - NLTK -+++++++++++++++++++++++++++++++ - -PyAMS is using NLTK features through the *PyAMS_calalog*. - -.. seealso:: - - Visit https://www.nltk.org/ to learn more about NLTK - - -Initializing NLTK (Natural Language ToolKit) --------------------------------------------- - -Some NLTK collections like **tokenizers** and **stopwords** utilities are used to index fulltext contents -elements. You can enhance NLTK indexation according to your own needs. This package requires downloading and -configuration of several elements which are done as follow: - - -*1. Run the Python shell into PyAMS environment:* - -.. code-block:: bash - - (env) $ ./bin/py - - -*2. In the Python shell:* - -.. code-block:: pycon - - >>> import nltk - >>> nltk.download() - - -*3. Configuration installation directory:* - -.. tip:: - - On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running - Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and - '*/usr/local/lib/nltk_data*' - - Please check if you have permission to write to this directory! - - -.. code-block:: shell - - NLTK Downloader - --------------------------------------------------------------------------- - d) Download l) List u) Update c) Config h) Help q) Quit - --------------------------------------------------------------------------- - Downloader> c - - Data Server: - - URL: - - 6 Package Collections Available - - 107 Individual Packages Available - - Local Machine: - - Data directory: /home/tflorac/nltk_data - - Config> d - New directory> /usr/local/lib/nltk_data - - -*4. Return to the main menu:* - -.. code-block:: shell - - --------------------------------------------------------------------------- - s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu - --------------------------------------------------------------------------- - Config> m - - -*5. Download utilities:* - - punkt - Punkt Tokenizer Models - stopwords - Stopwords Corpus - - -.. code-block:: shell - - --------------------------------------------------------------------------- - d) Download l) List u) Update c) Config h) Help q) Quit - --------------------------------------------------------------------------- - Downloader> d - Download which package (l=list; x=cancel)? - Identifier> punkt - Downloading package punkt to /usr/local/lib/nltk_data... - Downloader> d - Download which package (l=list; x=cancel)? - Identifier> stopwords - Downloading package stopwords to /usr/local/lib/nltk_data... - - -.. tip:: - - The full list of NTLK Collection can be displayed with the ``l) list`` option. diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/scripts.rst --- a/src/source/admln_guide/scripts.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,49 +0,0 @@ -.. _scripts: - -PyAMS maintenance scripts -========================= - - -pyams_upgrade -------------- - - -Pyramid allows to define custom command line scripts for application management. A script called *pyams_upgrade* is -provided by PyAMS_utils package; this script apply the same process as PyAMS site factory, but can also be used to -manage **database generations**. The idea behind this is just to allow custom packages to provide a way to check and -upgrade database configuration away from application startup process: - -.. code-block:: bash - - # ./bin/pyams_upgrade webapp/development.ini - - -A **site generation checker** is just a named utility providing :py:class:`pyams_utils.interfaces.site.ISiteGenerations` -interface. For example, **pyams_security** package provides such utility, to make sure that local site manager -contains a PyAMS security manager and a principal annotation utility: - -.. code-block:: python - - from pyams_utils.site import check_required_utilities - - REQUIRED_UTILITIES = ((ISecurityManager, '', SecurityManager, 'Security manager'), - (IPrincipalAnnotationUtility, '', PrincipalAnnotationUtility, 'User profiles')) - - @utility_config(name='PyAMS security', provides=ISiteGenerations) - class SecurityGenerationsChecker(object): - """I18n generations checker""" - - generation = 1 - - def evolve(self, site, current=None): - """Check for required utilities""" - check_required_utilities(site, REQUIRED_UTILITIES) - -:py:func:`check_required_utilities ` is a PyAMS_utils utility function which -can to used to verify that a set of local utilities are correctly registered with the given names and interfaces. - - -zodbupdate ----------- - -:ref:`renamehowto` diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/security.rst --- a/src/source/admln_guide/security.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,1 +0,0 @@ -.. _security: diff -r 097b0c025eec -r 49e4432c0a1d src/source/admln_guide/zodb.rst --- a/src/source/admln_guide/zodb.rst Tue Dec 11 17:00:42 2018 +0100 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,209 +0,0 @@ -.. _zodb: - - -ZODB server -============ - - -Installation -'''''''''''' - -PyAMS relies mainly on a ZODB (**\Z**\ope **\O**\bjects **\D**\ata\ **\B**\ase) to store it's configuration and for exemple *PyAMS_content*, but other packages may -rely on other database(s). - -Concurrent accesses are **required** on the ZODB, to fully enjoy PyAMS. -Several ZODB storages implementations providing a shared access are available and the installation are describe below: - - * `ZEO Server`_ - * `RelStorage`_ - * `Newt.DB`_ - - -.. note:: - - When you start your application in "*single process*" mode, PyAMS packages start several processes - and the "*synchronization*" is done via **ØMQ**. - - - -.. _`ZEO Server`: - -1. ZEO server -+++++++++++++ - -ZEO (**\Z**\ope **\E**\nterprise **\O**\bjects) is the first available implementation available for concurrent access to a -FileStorage, provided through the ZEO package. - -.. seealso:: - - ZEO package documentation and complete configuration settings are available on PyPI_ (cf: `ZOE 5.2.0`_). - -.. _PyPI: https://pypi.python.org/pypi/ZEO -.. _`ZOE 5.2.0`: https://pypi.python.org/pypi/ZEO/5.2.0 - - -a) Installation with initial buildout -------------------------------------- - -PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template. - -A simple option to create a ZEO server is to create a buildout environment including *ZEO* and *ZODB* packages: - -.. code-block:: bash - - $ mkdir /var/local/ - $ pip3 install virtualenv - $ virtualenv --python=python3.5 env - $ cd env - $ . bin/activate - (env) $ pip3.5 install cookiecutter - (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server - -*CookieCutter* will ask you for a small set of input variables that you can change or not: - -- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release; - you can also choose to point to a given release ("0.1.4" for example) - -- **project_name**: current environment name in "human form" - -- **project_slug**: "technical" package name, based on project name - -- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be - shared with other projects ("eggs" as default) - -- **run_user**: user name under which ZEO process will run ("zeo" as default) - -- **run_group**: group name under which ZEO process will run ("zeo" as default) - -- **zeo_server_port**: listening port of ZEO server ("8100" as default) - -- **zeo_storage**: name of first ZEO storage ("pyams" as default) - -- **zeo_pack_report**: email address to which pack reports should be sent - -- **logs_directory**: absolute path to directory containing ZEO's log files. - -A message is displayed after initialization to finalize environment creation: - -.. code:: - - Your ZEO environment is initialized. - To finalize it''s creation, just type: - - cd zeo_server - - python3.5 bootstrap.py - - ./bin/buildout - - To initialize authentication database, please run following command after buildout: - ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx - -b) ZEO server configuration ---------------------------- - -All ZEO configuration files are generated in "etc" subdirectory. These includes: - -- **etc/zeo_server-zdaemon.conf**: ZDaemon configuration file - -- **etc/zeo_server-zeo.conf**: ZEO server configuration file - -- **etc/auth.db**: ZEO authentication file; WARNING: this file is not created automatically, you have to create it - after buildout. - -In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during -*CookieCutter* template creation. - - -c) ZEO server tools -------------------- - -A set of system configuration files are produced to handle your ZEO environment. These includes: - -- **etc/init.d/zeo-zeo_server**: ZEO server start/stop script in Init-D format. Create a link to this file in - */etc/init.d* and update Init.d scripts (*update-rc.d zeo-zeo_server defaults*) to include ZEO in server start/stop - process. You can also use this script to start/stop ZEO by hand with *start* and *stop* arguments. - -- **etc/systemd/zeo-zeo_server.service**: SystemD service configuration file for ZEO server. Create a link to this - file in */etc/systemd/system* and reload SystemD daemon (*systemctl daemon-reload*) before activating ZEO service - (*systemctl enable zeo-zeo_server.service* and *systemctl start zeo-zeo_server.service*). - -- **etc/logrotate.d/zeo-zeo_server**: LogRotate configuration file for ZEO log files. Create a link to this file in - */etc/logrotate.d* to activate log rotation for ZEO server. - -- **etc/cron.d/pack-zeo-zeo_server**: Cron configuration file for ZEO database packing. Just create a link to this - file in */etc/cron.d* directory to enable ZODB packing on a weekly basis (by default). - -In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during -*CookieCutter* template creation. All directory names are those used on a Debian GNU/Linux distribution and may have -to be changed on other distributions. - - -.. _RelStorage: - -2. RelStorage server -++++++++++++++++++++ - -RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores -Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are -supported. - -To create a database compatible with RelStorage, you just have to install the database server and create a database -dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup. - -RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get -benefit from many extensions (clustering, fail-over, hot-standby...) provided by these databases. - - -.. _Newt.DB: - -3. NewtDB server -++++++++++++++++ - -NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is -dedicated to PostgreSQL (>= 9.5). - -NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB -column. The JSON data supplements the native data to support indexing, search, and access from non-Python application. -Because the JSON format is lossy, compared to the native format, the native format is still used for loading -objects from the database. For this reason, the JSON data are read-only. - -Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a -convenient API for raw data searches. - -Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available -if you need to switch from a classic RelStorage database to a Newt database. - - -.. _ZODB.migration: - -Migration -''''''''' - -After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command -line script provided by RelStorage. - -This Python script is using a configuration file containing directives of both source and target databases, which can -be any storage described in the previous sections (or can even use the same storage). - -Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage: - - -.. code-block:: html - - %import relstorage - - - server zeo-server.mydomain:8100 - storage pyams - blob-dir /var/local/env/zeo/var/zeoclient/blobs - shared-blob-dir false - - - - keep-history false - blob-dir /var/local/env/pyams/var/relstorage/blobs - shared-blob-dir false - - dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx' - - - - diff -r 097b0c025eec -r 49e4432c0a1d src/source/index.rst --- a/src/source/index.rst Tue Dec 11 17:00:42 2018 +0100 +++ b/src/source/index.rst Thu Dec 13 10:51:53 2018 +0100 @@ -20,6 +20,8 @@ Overview Quickstart + Glossary + .. toctree:: :maxdepth: 2 @@ -27,13 +29,13 @@ PyAMS Architectures PyAMS Security model - Installing PyAMS - Database initialization - Database maintenance tasks - Scheduling tasks - Command line scripts - Application tuning - Troubleshooting + Installing PyAMS + Database initialization + Database maintenance tasks + Scheduling tasks + Command line scripts + Application tuning + Troubleshooting Configuring and Running PyAMS Managing PyAMS environment