--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/app-install.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,227 @@
+.. _appinstall:
+
+.. _install:
+
+Installing PyAMS application
+============================
+
+**PyAMS** default installation is based on `Buildout <http://www.buildout.org>`_ utility.
+
+.. tip::
+
+ Take a look at virtualenv_. This tool provide isolated Python environments, which are more practical than installing packages systemwide.
+ They also allow installing packages without administrator privileges.
+
+.. _virtualenv: https://virtualenv.pypa.io/en/stable/
+
+
+Current PyAMS version is based and validated for **Python 3.5**; your Python environment must also include a **C compiler**
+as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and
+eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. **Cython** compiler can also be useful to optimize
+several packages.
+
+
+.. note::
+
+ PyAMS default components configuration also pre-suppose that the following external services are available.
+
+ * Mandatory services:
+
+ - a **ZODB shared server** (see :ref:`zodb` before). Several background processes needing a concurrent access to ZODB are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db.
+
+ - a **Memcached** or **Redis*** server, to store sessions and cache
+
+ * Optional services:
+
+ - an **LDAP server** for authentication
+
+ - an **ElasticSearch server** for full text indexing (see *PyAMS_content_es* package)
+
+ - a **WebSockets server** using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* packages). An *out of the box* environment can be built using *pyams_notify* scaffold.
+
+
+Creating initial buildout
+-------------------------
+
+PyAMS provides a new Pyramid scaffold, called *pyams*, generated via a *cookiecutter* template.
+
+A simple option to install PyAMS is to create a buildout environment including *Pyramid* and all *PyAMS* packages:
+
+.. code-block:: bash
+
+ $ mkdir /var/local/
+ $ pip3 install virtualenv
+ $ virtualenv --python=python3.5 env
+ $ cd env
+ $ . bin/activate
+ (env) $ pip3.5 install cookiecutter
+ (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams
+
+*CookieCutter* will ask you for a small set of input variables that you can change or not (**parameter** [*default value*]:)
+
+- **pyams_release** [*latest*] : version of PyAMS configuration file to use. "latest" will point to last release;
+ you can also choose to point to a given release ("0.1.4" for example)
+
+- **project_name** [*PyAMS*]: current environment name in "human form"
+
+- **project_slug**: [*pyams*] "technical" package name, based on project name
+
+- **virtual_hostname** [*pyams.mydomain.com*]: Apache virtual-host name
+
+- **webapp_name** [*webapps*]: web application package name
+
+- **webapp_port** [*6543*]: TCP/IP port to use when running application outside Apache.
+
+- **eggs_directory** [*eggs*]: relative or absolute path to directory containing downloaded eggs; this directory can be
+ shared with other projects.
+
+- **logs_directory** [*/var/log*]: absolute path to directory containing Apache's log files
+
+- **run_user** [*www-data*]: user name under which Apache process will run
+
+- **run_group** [*www-data*]: group name under which Apache process will run
+
+- **beaker_backend** : name of Beaker backend to use to store sessions and cache data ("redis" as default)
+
+- **beaker_server** [*127.0.0.1:6379*]: IP address and port of Beaker backend server
+
+- **db_type** [*zeo*]: ZODB database storage; available options include ZEO, RelStorage and NewtDB
+
+- **db_host** [*127.0.0.1*]: IP address of database server ("127.0.0.1" as default); WARNING: database server installation
+ is not part of application installation; another "zeo_server" cookiecutter recipe is available for ZEO
+
+- **db_port** [*8100*]: listening port of database server ("8100" is given as default for ZEO)
+
+- **db_name** [*pyams*]: database or ZEO storage name to use
+
+- **db_username**: database user name
+
+- **db_password**: database password
+
+- **zeo_realm**: ZEO authentication realm
+
+- **blobs_dir** [*/var/db/blobs*]: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default
+
+- **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files
+ are available to compile PsycoPG2 extension
+
+- **use_oracle**: specify if Oracle access is required; if so, please check that Oracle development files are
+ available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below)
+
+- **use_ldap**: specify if LDAP access will be required for authentication
+
+- **use_elasticsearch**: specify if an ElasticSearch server will be used for indexation
+
+- **elasticsearch_server**: URL used to access Elasticsearch server ("http://127.0.0.1:9200" as default); this URL can
+ include login and password ("http://login:password@127.0.0.1:9200"), if required...
+
+- **elasticsearch_index**: name of Elasticsearch index to use ("pyams" as default)
+
+- **create_elasticsearch_index**: specify if Elasticsearch index should be created after installation is complete
+
+- **define_elasticsearch_mappings** : specify if Elasticsearch mappings should be defined after installation is complete
+
+- **smtp_server**: DNS name of SMTP server ("localhost" as default)
+
+- **smtp_server_name**: "human" name given to SMTP server ("pyams" as default)
+
+- **pyams_scheduler** [*127.0.0.1:5555*]: TCP/IP address and port to use to access PyAMS tasks scheduler process (see :ref:`pyams_scheduler`)
+
+- **start_scheduler** [*True*]: boolean value to indicate if scheduler process is started by this application instance
+
+- **pyams_medias_converter** [*127.0.0.1:5556*]: TCP/IP address and port to use to access PyAMS medias converter process (see :ref:`pyams_media`)
+
+- **start_medias_converter** [*True*]: boolean value to indicate if medias converter process is started by this application
+ instance
+
+- **pyams_es_indexer** [*127.0.0.1:5557*] : TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (see :ref:`pyams_content_es`)
+
+- **start_es_indexer** boolean value to indicate if Elasticsearch indexer process is started by this application
+ instance
+
+- **use_notifications**: specify if PyAMS notifications services are to be used (see :ref:`pyams_notify`)
+
+- **pyams_ws_notify** [*127.0.0.1:8081*]: TCP/IP address and port of PyAMS websockets server managing notifications service
+
+- **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default)
+
+- **extension_package**: name of a PyAMS extension package to include in environment configuration
+
+- **need_pyams_gis**: specify if PyAMS GIS features are to be used by given extension package; if so, please check
+ that *libgdal* development files are available; on Debian (and maybe others), you have to specify environment
+ variables (see below).
+
+.. tip::
+
+ 1. Setting any option to empty string will keep the default proposed value.
+ 2. Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*;
+
+You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove
+packages or update settings to your needs.
+
+This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual
+environment. If you encounter any compile error, just install the required dependencies and restart the buildout.
+
+Some dependencies can require the definition of custom environment variables before running *buildout*, like:
+
+- for *libgdal*, which is required by **PyAMS_gis** package, use:
+
+.. code-block:: bash
+
+ (env) $ export C_INCLUDE_PATH=/usr/include/gdal
+ (env) $ export CPLUS_INCLUDE_PATH=/usr/include/gdal
+
+
+.. WARNING::
+ You have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS
+ configuration file (actually 2.1.0).
+
+- for *cx_Oracle*, which is required if you use Oracle database connections, use:
+
+.. code-block:: bash
+
+ (env) $ export ORACLE_HOME=/usr/lib/oracle/12.1/client64
+
+These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux
+distribution and packages versions.
+
+
+**Then finalize Bootstrap initialization:**
+
+.. code-block:: bash
+
+ (env) $ python3.5 bootstrap.py
+ (env) $ ./bin/buildout
+
+
+.. seealso::
+
+ `zc.buildout <https://pypi.python.org/pypi/zc.buildout>`_ is a powerful
+ tool for creating repeatable builds of a given software configuration
+ and environment.
+
+
+Environment settings
+--------------------
+
+The project generated from *pyams* scaffold is based on default Pyramid's *zodb* scaffold, but it adds:
+
+- a custom application factory, in the *webapp* directory (see :ref:`site`)
+
+- a set of directories to store runtime data, in the *var* directory; each directory contains a *README.txt* file
+ which should be self-explanatory to indicate what this directory should contain, including a ZEO cache
+
+- a set of configuration files, in the *etc* directory; here are standard *development.ini* and *production.ini*
+ configuration files, a ZODB configuration files (*zodb-zeo.conf*) for a ZEO client storage and two Apache
+ configurations (for Apache 2.2 and 2.4) using *mod_wsgi*.
+
+Once the project have been created from the scaffold, you are free to update all the configuration files.
+
+If you need to add packages to the environment, you have to add them to the *buildout.cfg* file **AND** to the INI
+file (in the *pyramid.includes* section) before running the *buildout* another time; don't forget to add the
+requested version at the end of *buildout.cfg* file, as Buildout is not configured by default to automatically
+download the last release of a given unknown package.
+
+*development.ini* and *production.ini* files contain many commented directives related to PyAMS components. Read and
+update them carefully before initializing your application database!
+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/app-notify.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,4 @@
+.. _notify:
+
+Installing PyAMS notification services
+======================================
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/app-plugins.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,209 @@
+.. _plugins:
+
+PyAMS additional features and services
+======================================
+
+
+Elasticsearch
++++++++++++++
+
+At first you need to install ElasticSearch (ES); PyAMS is actually compatible with version 5.4. The Ingest attachment
+plug-in is also required to handle attachments correctly.
+
+Visit https://www.elastic.co/ to learn how to install Elasticsearch Server and `ingest-attachment` plug-in
+
+
+.. tip:: Documentation for installing ElasticSearch 6.4
+
+ - https://www.elastic.co/guide/en/elasticsearch/reference/6.4/gs-installation.html
+ - https://www.elastic.co/guide/en/elasticsearch/plugins/4.4/ingest-attachment.html
+
+
+After Elasticsearch installation, following steps describe how to configure ES with PyAMS.
+
+
+Initializing Elasticsearch index
+--------------------------------
+
+If you want to use an Elasticsearch index, you have to initialize index settings and mappings;
+Elasticsearch integration is defined through the *PyAMS_content_es* package.
+
+
+1. Enable service
+'''''''''''''''''
+
+In Pyramid INI application files (*etc/development.ini* and *etc/production.ini*):
+
+.. code-block:: ini
+
+ # Elasticsearch server settings
+ elastic.server = http://127.0.0.1:9200
+ elastic.index = pyams
+
+Where:
+ - **elastic.server**: address of Elasticsearch server; you can include authentication arguments in the form
+ *http://login:password@w.x.y.z:9200*
+ - **elastic.index**: name of Elasticsearch index.
+
+
+On startup, main PyAMS application process can start in *indexer* process which will handle indexing requests in
+asynchronous mode; this process settings are defined like this:
+
+.. code-block:: ini
+
+ # PyAMS content Elasticsearch indexer process settings
+ pyams_content.es.tcp_handler = 127.0.0.1:5557
+ pyams_content.es.start_handler = false
+ pyams_content.es.allow_auth = admin:admin
+ pyams_content.es.allow_clients = 127.0.0.1
+
+Where:
+ - **pyams_content.es.tcp_handler**: IP address and listening port of PyAMS indexer process
+ - **pyams_content.es.start_handler**: if *true*, the indexer process is started on PyAMS startup; otherwise (typically
+ in a cluster configuration), the process is supposed to be started from another *master* server
+ - **pyams_content.es.allow_auth**: login and password to be used to connect to indexer process (settings are defined
+ in the same way on indexer process and on all it's clients)
+ - **pyams_content.es.allow_clients**: list of IP addresses allowed to connect to indexer process.
+
+
+2. Initialize Elasticsearch database
+''''''''''''''''''''''''''''''''''''
+
+Configuration files for attachment pipeline, index and mappings settings are available into `pyams_content_es` source
+package or in PyAMS installation folder:
+
+
+.. code-block:: bash
+
+ (env) $ cd docs/elasticsearch
+ (env) $ curl --noproxy localhost -XPUT http://localhost:9200/_ingest/pipeline/attachment -d @attachment-pipeline.json
+
+
+And with ``elastic.index = pyams`` defined as Elasticsearch index name: *"http://localhost:9200/pyams"*:
+
+.. code-block:: shell
+
+ (env) $ curl -XDELETE http://localhost:9200/pyams
+
+ (env) $ curl -XPUT http://localhost:9200/pyams -d @index-settings.json
+
+ (env) $ curl -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json
+ (env) $ curl -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json
+ (env) $ curl -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json
+
+
+*Troubleshooting*: If you have a 406 error try to add ``-H 'Content-Type: application/json'`` in Curl command lines.
+
+
+3. Update index contents
+''''''''''''''''''''''''
+
+If your ZODB database already store contents, you can update ElasticSearch indexes with all these contents with
+``pymas_es_index`` command line script. From a shell:
+
+.. code-block:: bash
+
+ (env) $ ./bin/pyams_es_index ../etc/development.ini
+
+
+
+Natural Language Toolkit - NLTK
++++++++++++++++++++++++++++++++
+
+PyAMS is using NLTK features through the *PyAMS_calalog*.
+
+.. seealso::
+
+ Visit https://www.nltk.org/ to learn more about NLTK
+
+
+Initializing NLTK (Natural Language ToolKit)
+--------------------------------------------
+
+Some NLTK collections like **tokenizers** and **stopwords** utilities are used to index fulltext contents
+elements. You can enhance NLTK indexation according to your own needs. This package requires downloading and
+configuration of several elements which are done as follow:
+
+
+*1. Run the Python shell into PyAMS environment:*
+
+.. code-block:: bash
+
+ (env) $ ./bin/py
+
+
+*2. In the Python shell:*
+
+.. code-block:: pycon
+
+ >>> import nltk
+ >>> nltk.download()
+
+
+*3. Configuration installation directory:*
+
+.. tip::
+
+ On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running
+ Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and
+ '*/usr/local/lib/nltk_data*'
+
+ Please check if you have permission to write to this directory!
+
+
+.. code-block:: shell
+
+ NLTK Downloader
+ ---------------------------------------------------------------------------
+ d) Download l) List u) Update c) Config h) Help q) Quit
+ ---------------------------------------------------------------------------
+ Downloader> c
+
+ Data Server:
+ - URL: <https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml>
+ - 6 Package Collections Available
+ - 107 Individual Packages Available
+
+ Local Machine:
+ - Data directory: /home/tflorac/nltk_data
+
+ Config> d
+ New directory> /usr/local/lib/nltk_data
+
+
+*4. Return to the main menu:*
+
+.. code-block:: shell
+
+ ---------------------------------------------------------------------------
+ s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu
+ ---------------------------------------------------------------------------
+ Config> m
+
+
+*5. Download utilities:*
+
+ punkt
+ Punkt Tokenizer Models
+ stopwords
+ Stopwords Corpus
+
+
+.. code-block:: shell
+
+ ---------------------------------------------------------------------------
+ d) Download l) List u) Update c) Config h) Help q) Quit
+ ---------------------------------------------------------------------------
+ Downloader> d
+ Download which package (l=list; x=cancel)?
+ Identifier> punkt
+ Downloading package punkt to /usr/local/lib/nltk_data...
+ Downloader> d
+ Download which package (l=list; x=cancel)?
+ Identifier> stopwords
+ Downloading package stopwords to /usr/local/lib/nltk_data...
+
+
+.. tip::
+
+ The full list of NTLK Collection can be displayed with the ``l) list`` option.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/app-start.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,95 @@
+.. _appstart:
+
+Starting the application
+========================
+
+When database upgrade process has ended ( :ref:`zodbinit` ) , you can start the web application process with the standard Pyramid's
+*pserve* command line tool:
+
+.. code-block:: bash
+
+ (env) $ ./bin/pserve etc/development.ini
+
+In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO):
+
+.. code-block:: bash
+
+ 2018-01-14 11:37:54,339 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=28695) created RW/normal for storage: 'pyams'
+ 2018-01-14 11:37:54,340 INFO [ZEO.cache][MainThread] created temporary cache file 3
+ 2018-01-14 11:37:54,345 INFO [ZODB.blob][MainThread] (28695) Blob directory `/var/local/env/pyams/var/db/blobs` is used but has no layout marker set. Selected `lawn` layout.
+ 2018-01-14 11:37:54,345 WARNI [ZODB.blob][MainThread] (28695) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout.
+ 2018-01-14 11:37:54,346 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
+ 2018-01-14 11:37:54,347 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7feeb1de7390> None
+ 2018-01-14 11:37:54,348 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
+ 2018-01-14 11:37:54,349 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
+ 2018-01-14 11:37:54,355 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
+ 2018-01-14 11:37:54,358 DEBUG [txn.140663320073984][MainThread] new transaction
+ 2018-01-14 11:37:54,360 DEBUG [txn.140663320073984][MainThread] commit
+ 2018-01-14 11:37:54,484 DEBUG [config][MainThread] include /home/tflorac/Dropbox/src/PyAMS/pyams_template/src/pyams_template/configure.zcml
+ 2018-01-14 11:37:54,485 DEBUG [config][MainThread] include /var/local/env/pycharm/lib/python3.5/site-packages/pyramid_zcml/configure.zcml
+ ...
+ 2018-01-14 11:37:54,833 DEBUG [PyAMS (utils)][MainThread] Registering utility <class 'pyams_utils.timezone.utility.TimezoneGenerationsChecker'> named 'PyAMS timezone' providing <InterfaceClass pyams_utils.interfaces.site.ISiteGenerations>
+ 2018-01-14 11:37:54,834 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.timezone.vocabulary.TimezonesVocabulary'> as vocabulary with name "PyAMS timezones"
+ 2018-01-14 11:37:54,835 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.traversing.PathElementsAdapter'> for (<InterfaceClass zope.location.interfaces.IContained>,) providing <InterfaceClass pyams_utils.interfaces.traversing.IPathElements>
+ 2018-01-14 11:37:54,839 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.url.AbsoluteUrlTalesExtension'> for (<InterfaceClass zope.interface.Interface>, <InterfaceClass zope.interface.Interface>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_utils.interfaces.tales.ITALESExtension>
+ 2018-01-14 11:37:54,847 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.widget.decimal.DottedDecimalDataConverter'> for (<InterfaceClass pyams_utils.schema.IDottedDecimalField>, <InterfaceClass z3c.form.interfaces.IWidget>) providing <InterfaceClass z3c.form.interfaces.IDataConverter>
+ 2018-01-14 11:37:54,942 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.zmi.intids.IntIdsLengthAdapter'> for (<InterfaceClass zope.intid.interfaces.IIntIds>,) providing <InterfaceClass pyams_utils.interfaces.intids.IIndexLength>
+ 2018-01-14 11:37:54,943 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass zope.intid.interfaces.IIntIds> (<class 'pyams_utils.zmi.intids.IntIdsPropertiesDisplayForm'>)
+ 2018-01-14 11:37:54,949 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass pyams_utils.interfaces.timezone.IServerTimezone> (<class 'pyams_utils.zmi.timezone.ServerTimezonePropertiesEditForm'>)
+ 2018-01-14 11:37:54,980 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.zodb.ZEOConnectionVocabulary'> as vocabulary with name "PyAMS ZEO connections"
+ 2018-01-14 11:37:54,981 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.zodb.ZODBConnectionVocabulary'> as vocabulary with name "PyAMS ZODB connections"
+ 2018-01-14 11:37:55,015 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "add-zeo-connection.html" for <InterfaceClass zope.component.interfaces.ISite> (<class 'pyams_utils.zmi.zeo.ZEOConnectionAddForm'>)
+ 2018-01-14 11:37:55,016 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.zmi.zeo.ZEOConnectionNameAdapter'> for (<InterfaceClass pyams_utils.interfaces.zeo.IZEOConnection>, <InterfaceClass pyams_zmi.layer.IAdminLayer>) providing <InterfaceClass pyams_skin.interfaces.container.ITableElementName>
+ 2018-01-14 11:37:55,017 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass pyams_utils.interfaces.zeo.IZEOConnection> (<class 'pyams_utils.zmi.zeo.ZEOConnectionPropertiesEditForm'>)
+ ...
+ 2018-01-14 11:41:13,214 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_default_theme.skin.ResourcesAdapter'> for (<InterfaceClass zope.interface.Interface>, <InterfaceClass pyams_default_theme.layer.IPyAMSDefaultLayer>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_skin.interfaces.resources.IResources>
+ 2018-01-14 11:43:36,665 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams'
+ 2018-01-14 11:43:36,665 INFO [ZEO.cache][MainThread] created temporary cache file 9
+ 2018-01-14 11:43:36,673 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
+ 2018-01-14 11:43:36,674 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642)
+ 2018-01-14 11:43:36,674 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7ffa54058860> None
+ 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0
+ 2018-01-14 11:43:36,675 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
+ 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 -->
+ 2018-01-14 11:43:36,677 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
+ 2018-01-14 11:43:36,679 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
+ 2018-01-14 11:43:36,682 DEBUG [txn.140713340237568][MainThread] new transaction
+ 2018-01-14 11:43:36,683 DEBUG [txn.140713340237568][MainThread] commit
+ 2018-01-14 11:43:36,690 INFO [PyAMS (scheduler][MainThread] Starting tasks scheduler <SchedulerProcess(SchedulerProcess-1, initial)>...
+ 2018-01-14 11:43:36,698 INFO [PyAMS (scheduler][MainThread] Started tasks scheduler with PID 29361.
+ 2018-01-14 11:43:36,701 INFO [apscheduler.scheduler][MainThread] Scheduler started
+ 2018-01-14 11:43:36,702 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run
+ 2018-01-14 11:43:36,704 DEBUG [apscheduler.scheduler][APScheduler] No jobs; waiting until a job is added
+ 2018-01-14 11:43:36,719 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams'
+ 2018-01-14 11:43:36,720 INFO [ZEO.cache][MainThread] created temporary cache file 15
+ 2018-01-14 11:43:36,724 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
+ 2018-01-14 11:43:36,725 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7ffa557e8b00> None
+ 2018-01-14 11:43:36,726 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
+ 2018-01-14 11:43:36,727 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642)
+ 2018-01-14 11:43:36,728 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
+ 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0
+ 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 -->
+ 2018-01-14 11:43:36,732 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
+ 2018-01-14 11:43:36,735 DEBUG [txn.140713340237568][MainThread] new transaction
+ 2018-01-14 11:43:36,736 DEBUG [txn.140713340237568][MainThread] commit
+ 2018-01-14 11:43:36,743 INFO [PyAMS (media)][MainThread] Starting medias converter <MediaConversionProcess(MediaConversionProcess-2, initial)>...
+ 2018-01-14 11:43:36,751 INFO [PyAMS (media)][MainThread] Started medias converter with PID 29367.
+ Starting server in PID 29335.
+ Serving on http://0.0.0.0:6543
+
+
+Congrats: PyAMS is running !
+
+Launch a browser and open URL http://127.0.0.1:6543/admin to get access to PyAMS management interface.
+
+.. warning::
+
+ By default:
+
+ - Login: admin
+
+ - Password: admin
+
+ **You must change it as soon as possible!**
+ (see :ref:`pyams_security`)!!.
+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/app-tuning.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,80 @@
+.. _tuning:
+
+Application tuning
+==================
+
+.. Note::
+
+ PyAMS application configuration is done via several complementary ways:
+
+ - via **Pyramid's configuration** files located in 'etc/' folder
+
+ - via **ZCML files** .
+
+ - via **Site Management** to setup Local utilities defined into ZODB (see :ref:`appmanage`)
+
+
+Pyramid's configuration files
+-----------------------------
+
+PyAMS is using default Pyramid's ini_ configuration files to define settings. When using default PyAMS scaffold, these
+INI files are stored into *etc* folder. Two files are created by scaffold: a development file (*development.ini*) and
+a production file (*production.ini*) which contain nearly the same directives, except for debug toolbar (activated only
+in development), Fanstatic configuration and logging.
+
+.. _ini: https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/project.html#development-ini
+
+These files are actually documented, but here are a few important directives:
+
+- **zodbconn.uri**: (ex: zconfig:/zodb-relstorage.conf)
+
+- **pyams.application_name**: Name of your own application
+- **pyams.application_factory**: Configuration module path to your custom application.
+
+- **pyams_xxx.tcp_handler** : Adress of the tcp handler for the service associated to the pyams package
+- **pyams_xxx.start_handler**: True / False
+
+- **pyams_content.config.xxx_tool_factory** and **pyams_content.config.xxx_tool_name**: You can override default factory by defining these settings or by providing your own adapters.
+
+
+
+Pyramid's configuration files powered by PyAMS_utils
+----------------------------------------------------
+
+On application startup, if PyAMS_utils package is included into Pyramid configuration, several operations take
+place:
+
+ - a custom **site factory** is defined
+
+ - custom request methods are defined
+
+ - a custom **traverser** handling **namespaces** is defined
+
+ - a custom subscribers predicate based on interfaces support is defined
+
+ - several adapters are registered, to handle annotations and key references
+
+ - custom TALES extensions are registered.
+
+The site factory is an important component in this process. It is this factory which will define the application root
+and create a **local site manager**.
+
+Pyramid application is loaded from ZODB's root via a key defined in Pyramid's configuration file; the key is named
+*pyams.application_name* and it's default value is *application*.
+
+If the application can't be found, PyAMS is looking for an application class name in Pyramid's configuration file; the
+class name configuration key is called *pyams.application_factory* and defined by default as
+*pyams_utils.site.BaseSiteRoot*. PyAMS default site factory will then create the application, and add a local site
+manager to it (see :ref:`zca`).
+
+After application creation, a :py:class:`NewLocalSiteCreatedEvent <pyams_utils.site.NewLocalSiteCreatedEvent>` is
+notified. Custom packages can subscribe to this event to register custom components.
+
+
+
+ZCML configuration files
+------------------------
+
+Adding more ZCML configuration directives is not mandatory in any way but can be useful in several use cases. For
+example, templates can't be overriden via Python annotations without creating a new subclass; this can be done
+easily via ZCML.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/architecture.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,47 @@
+.. _architecture:
+
+PyAMS application architecture
+==============================
+
+PyAMS applications are built on a small set of prerequisites and external components which *can* or, for some of them,
+*must* be included, the minimal configuration is:
+
+* a mandatory shared ZODB; :ref:`zodb` chapter
+
+* a Redis or Memcached server, to handle cache and sessions;
+ :ref:`pyams_cache` chapter
+
+
+Single server deployment
+------------------------
+
+Overview of the environment system after the :ref:`quickstart` installation.
+It's a very lite configuration, with only mandatory resources.
+Obviously you can extend this system by connecting more resources like is described in the :ref:`config` chapiter
+
+
+.. image:: ../_static/architecture_quickstart.png
+
+
+
+Development deployment
+------------------------
+
+In additon of mandatory resource **PyAMS** could work with:
+
+* a websockets server (GuniCorn), used to handle desktop notifications;
+ you can find information about this component in the :ref:`pyams_notify` chapter;
+
+* an Elasticsearch index, used to handle quick and optimized searching.
+
+
+.. image:: ../_static/architecture_dev.png
+
+
+Advanced deployment
+-------------------
+
+Use case in an advanced environment.
+
+.. image:: ../_static/architecture_prod.png
+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/pyams-install.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,14 @@
+Installing PyAMS
+================
+
+
+.. toctree::
+
+ Installing database <zodb>
+ app-install
+ app-notify
+ app-plugins
+ app-start
+
+
+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/scripts.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,5 @@
+.. _scripts:
+
+Command line scripts
+====================
+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/security.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,4 @@
+.. _security:
+
+PyAMS Security model
+====================
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/tasks.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,4 @@
+.. _tasks:
+
+Scheduling tasks
+================
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/troubleshooting.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,4 @@
+.. _troubleshooting:
+
+Troubleshooting
+===============
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/zodb-init.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,17 @@
+
+.. _zodbinit:
+
+Initializing the ZODB database
+------------------------------
+
+After, you have downloaded and installed all required packages, you have to initialize the database so that all
+required components are available.
+
+From a shell, just type:
+
+.. code-block:: bash
+
+ (env) $ ./bin/pyams_upgrade etc/development.ini
+
+This process requires that every package is correctly included into *pyramid.includes* directive from selected
+configuration file.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/zodb-support.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,49 @@
+.. _dbsupport:
+
+Database maintenance tasks
+==========================
+
+
+pyams_upgrade
+-------------
+
+
+Pyramid allows to define custom command line scripts for application management. A script called *pyams_upgrade* is
+provided by PyAMS_utils package; this script apply the same process as PyAMS site factory, but can also be used to
+manage **database generations**. The idea behind this is just to allow custom packages to provide a way to check and
+upgrade database configuration away from application startup process:
+
+.. code-block:: bash
+
+ # ./bin/pyams_upgrade webapp/development.ini
+
+
+A **site generation checker** is just a named utility providing :py:class:`pyams_utils.interfaces.site.ISiteGenerations`
+interface. For example, **pyams_security** package provides such utility, to make sure that local site manager
+contains a PyAMS security manager and a principal annotation utility:
+
+.. code-block:: python
+
+ from pyams_utils.site import check_required_utilities
+
+ REQUIRED_UTILITIES = ((ISecurityManager, '', SecurityManager, 'Security manager'),
+ (IPrincipalAnnotationUtility, '', PrincipalAnnotationUtility, 'User profiles'))
+
+ @utility_config(name='PyAMS security', provides=ISiteGenerations)
+ class SecurityGenerationsChecker(object):
+ """I18n generations checker"""
+
+ generation = 1
+
+ def evolve(self, site, current=None):
+ """Check for required utilities"""
+ check_required_utilities(site, REQUIRED_UTILITIES)
+
+:py:func:`check_required_utilities <pyams_utils.site.check_required_utilities>` is a PyAMS_utils utility function which
+can to used to verify that a set of local utilities are correctly registered with the given names and interfaces.
+
+
+zodbupdate
+----------
+
+:ref:`renamehowto`
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/src/source/admin_guide/zodb.rst Thu Dec 13 10:51:53 2018 +0100
@@ -0,0 +1,209 @@
+.. _zodb:
+
+
+ZODB server
+============
+
+
+Installation
+''''''''''''
+
+PyAMS relies mainly on a ZODB (**\Z**\ope **\O**\bjects **\D**\ata\ **\B**\ase) to store it's configuration and for exemple *PyAMS_content*, but other packages may
+rely on other database(s).
+
+Concurrent accesses are **required** on the ZODB, to fully enjoy PyAMS.
+Several ZODB storages implementations providing a shared access are available and the installation are describe below:
+
+ * `ZEO Server`_
+ * `RelStorage`_
+ * `Newt.DB`_
+
+
+.. note::
+
+ When you start your application in "*single process*" mode, PyAMS packages start several processes
+ and the "*synchronization*" is done via **ØMQ**.
+
+
+
+.. _`ZEO Server`:
+
+1. ZEO server
++++++++++++++
+
+ZEO (**\Z**\ope **\E**\nterprise **\O**\bjects) is the first available implementation available for concurrent access to a
+FileStorage, provided through the ZEO package.
+
+.. seealso::
+
+ ZEO package documentation and complete configuration settings are available on PyPI_ (cf: `ZOE 5.2.0`_).
+
+.. _PyPI: https://pypi.python.org/pypi/ZEO
+.. _`ZOE 5.2.0`: https://pypi.python.org/pypi/ZEO/5.2.0
+
+
+a) Installation with initial buildout
+-------------------------------------
+
+PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template.
+
+A simple option to create a ZEO server is to create a buildout environment including *ZEO* and *ZODB* packages:
+
+.. code-block:: bash
+
+ $ mkdir /var/local/
+ $ pip3 install virtualenv
+ $ virtualenv --python=python3.5 env
+ $ cd env
+ $ . bin/activate
+ (env) $ pip3.5 install cookiecutter
+ (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server
+
+*CookieCutter* will ask you for a small set of input variables that you can change or not:
+
+- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release;
+ you can also choose to point to a given release ("0.1.4" for example)
+
+- **project_name**: current environment name in "human form"
+
+- **project_slug**: "technical" package name, based on project name
+
+- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be
+ shared with other projects ("eggs" as default)
+
+- **run_user**: user name under which ZEO process will run ("zeo" as default)
+
+- **run_group**: group name under which ZEO process will run ("zeo" as default)
+
+- **zeo_server_port**: listening port of ZEO server ("8100" as default)
+
+- **zeo_storage**: name of first ZEO storage ("pyams" as default)
+
+- **zeo_pack_report**: email address to which pack reports should be sent
+
+- **logs_directory**: absolute path to directory containing ZEO's log files.
+
+A message is displayed after initialization to finalize environment creation:
+
+.. code::
+
+ Your ZEO environment is initialized.
+ To finalize it''s creation, just type:
+ - cd zeo_server
+ - python3.5 bootstrap.py
+ - ./bin/buildout
+
+ To initialize authentication database, please run following command after buildout:
+ ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx
+
+b) ZEO server configuration
+---------------------------
+
+All ZEO configuration files are generated in "etc" subdirectory. These includes:
+
+- **etc/zeo_server-zdaemon.conf**: ZDaemon configuration file
+
+- **etc/zeo_server-zeo.conf**: ZEO server configuration file
+
+- **etc/auth.db**: ZEO authentication file; WARNING: this file is not created automatically, you have to create it
+ after buildout.
+
+In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
+*CookieCutter* template creation.
+
+
+c) ZEO server tools
+-------------------
+
+A set of system configuration files are produced to handle your ZEO environment. These includes:
+
+- **etc/init.d/zeo-zeo_server**: ZEO server start/stop script in Init-D format. Create a link to this file in
+ */etc/init.d* and update Init.d scripts (*update-rc.d zeo-zeo_server defaults*) to include ZEO in server start/stop
+ process. You can also use this script to start/stop ZEO by hand with *start* and *stop* arguments.
+
+- **etc/systemd/zeo-zeo_server.service**: SystemD service configuration file for ZEO server. Create a link to this
+ file in */etc/systemd/system* and reload SystemD daemon (*systemctl daemon-reload*) before activating ZEO service
+ (*systemctl enable zeo-zeo_server.service* and *systemctl start zeo-zeo_server.service*).
+
+- **etc/logrotate.d/zeo-zeo_server**: LogRotate configuration file for ZEO log files. Create a link to this file in
+ */etc/logrotate.d* to activate log rotation for ZEO server.
+
+- **etc/cron.d/pack-zeo-zeo_server**: Cron configuration file for ZEO database packing. Just create a link to this
+ file in */etc/cron.d* directory to enable ZODB packing on a weekly basis (by default).
+
+In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
+*CookieCutter* template creation. All directory names are those used on a Debian GNU/Linux distribution and may have
+to be changed on other distributions.
+
+
+.. _RelStorage:
+
+2. RelStorage server
+++++++++++++++++++++
+
+RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores
+Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are
+supported.
+
+To create a database compatible with RelStorage, you just have to install the database server and create a database
+dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup.
+
+RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get
+benefit from many extensions (clustering, fail-over, hot-standby...) provided by these databases.
+
+
+.. _Newt.DB:
+
+3. NewtDB server
+++++++++++++++++
+
+NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is
+dedicated to PostgreSQL (>= 9.5).
+
+NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB
+column. The JSON data supplements the native data to support indexing, search, and access from non-Python application.
+Because the JSON format is lossy, compared to the native format, the native format is still used for loading
+objects from the database. For this reason, the JSON data are read-only.
+
+Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a
+convenient API for raw data searches.
+
+Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available
+if you need to switch from a classic RelStorage database to a Newt database.
+
+
+.. _ZODB.migration:
+
+Migration
+'''''''''
+
+After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command
+line script provided by RelStorage.
+
+This Python script is using a configuration file containing directives of both source and target databases, which can
+be any storage described in the previous sections (or can even use the same storage).
+
+Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage:
+
+
+.. code-block:: html
+
+ %import relstorage
+
+ <zeoclient source>
+ server zeo-server.mydomain:8100
+ storage pyams
+ blob-dir /var/local/env/zeo/var/zeoclient/blobs
+ shared-blob-dir false
+ </zeoclient>
+
+ <relstorage destination>
+ keep-history false
+ blob-dir /var/local/env/pyams/var/relstorage/blobs
+ shared-blob-dir false
+ <postgresql>
+ dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx'
+ </postgresql>
+ </relstorage>
+
+
--- a/src/source/admln_guide/appconfig.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,80 +0,0 @@
-.. _config:
-
-PyAMS configuration
-===================
-
-.. Note::
-
- PyAMS application configuration is done via several complementary ways:
-
- - via **Pyramid's configuration** files located in 'etc/' folder
-
- - via **ZCML files** .
-
- - via **Site Management** to setup Local utilities defined into ZODB (see :ref:`appmanage`)
-
-
-Pyramid's configuration files
------------------------------
-
-PyAMS is using default Pyramid's ini_ configuration files to define settings. When using default PyAMS scaffold, these
-INI files are stored into *etc* folder. Two files are created by scaffold: a development file (*development.ini*) and
-a production file (*production.ini*) which contain nearly the same directives, except for debug toolbar (activated only
-in development), Fanstatic configuration and logging.
-
-.. _ini: https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/project.html#development-ini
-
-These files are actually documented, but here are a few important directives:
-
-- **zodbconn.uri**: (ex: zconfig:/zodb-relstorage.conf)
-
-- **pyams.application_name**: Name of your own application
-- **pyams.application_factory**: Configuration module path to your custom application.
-
-- **pyams_xxx.tcp_handler** : Adress of the tcp handler for the service associated to the pyams package
-- **pyams_xxx.start_handler**: True / False
-
-- **pyams_content.config.xxx_tool_factory** and **pyams_content.config.xxx_tool_name**: You can override default factory by defining these settings or by providing your own adapters.
-
-
-
-Pyramid's configuration files powered by PyAMS_utils
-----------------------------------------------------
-
-On application startup, if PyAMS_utils package is included into Pyramid configuration, several operations take
-place:
-
- - a custom **site factory** is defined
-
- - custom request methods are defined
-
- - a custom **traverser** handling **namespaces** is defined
-
- - a custom subscribers predicate based on interfaces support is defined
-
- - several adapters are registered, to handle annotations and key references
-
- - custom TALES extensions are registered.
-
-The site factory is an important component in this process. It is this factory which will define the application root
-and create a **local site manager**.
-
-Pyramid application is loaded from ZODB's root via a key defined in Pyramid's configuration file; the key is named
-*pyams.application_name* and it's default value is *application*.
-
-If the application can't be found, PyAMS is looking for an application class name in Pyramid's configuration file; the
-class name configuration key is called *pyams.application_factory* and defined by default as
-*pyams_utils.site.BaseSiteRoot*. PyAMS default site factory will then create the application, and add a local site
-manager to it (see :ref:`zca`).
-
-After application creation, a :py:class:`NewLocalSiteCreatedEvent <pyams_utils.site.NewLocalSiteCreatedEvent>` is
-notified. Custom packages can subscribe to this event to register custom components.
-
-
-
-ZCML configuration files
-------------------------
-
-Adding more ZCML configuration directives is not mandatory in any way but can be useful in several use cases. For
-example, templates can't be overriden via Python annotations without creating a new subclass; this can be done
-easily via ZCML.
--- a/src/source/admln_guide/appinstall.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,244 +0,0 @@
-.. _appinstall:
-
-.. _install:
-
-Installing PyAMS application
-============================
-
-**PyAMS** default installation is based on `Buildout <http://www.buildout.org>`_ utility.
-
-.. tip::
-
- Take a look at virtualenv_. This tool provide isolated Python environments, which are more practical than installing packages systemwide.
- They also allow installing packages without administrator privileges.
-
-.. _virtualenv: https://virtualenv.pypa.io/en/stable/
-
-
-Current PyAMS version is based and validated for **Python 3.5**; your Python environment must also include a **C compiler**
-as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and
-eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. **Cython** compiler can also be useful to optimize
-several packages.
-
-
-.. note::
-
- PyAMS default components configuration also pre-suppose that the following external services are available.
-
- * Mandatory services:
-
- - a **ZODB shared server** (see :ref:`zodb` before). Several background processes needing a concurrent access to ZODB are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db.
-
- - a **Memcached** or **Redis*** server, to store sessions and cache
-
- * Optional services:
-
- - an **LDAP server** for authentication
-
- - an **ElasticSearch server** for full text indexing (see *PyAMS_content_es* package)
-
- - a **WebSockets server** using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* packages). An *out of the box* environment can be built using *pyams_notify* scaffold.
-
-
-Creating initial buildout
--------------------------
-
-PyAMS provides a new Pyramid scaffold, called *pyams*, generated via a *cookiecutter* template.
-
-A simple option to install PyAMS is to create a buildout environment including *Pyramid* and all *PyAMS* packages:
-
-.. code-block:: bash
-
- $ mkdir /var/local/
- $ pip3 install virtualenv
- $ virtualenv --python=python3.5 env
- $ cd env
- $ . bin/activate
- (env) $ pip3.5 install cookiecutter
- (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams
-
-*CookieCutter* will ask you for a small set of input variables that you can change or not (**parameter** [*default value*]:)
-
-- **pyams_release** [*latest*] : version of PyAMS configuration file to use. "latest" will point to last release;
- you can also choose to point to a given release ("0.1.4" for example)
-
-- **project_name** [*PyAMS*]: current environment name in "human form"
-
-- **project_slug**: [*pyams*] "technical" package name, based on project name
-
-- **virtual_hostname** [*pyams.mydomain.com*]: Apache virtual-host name
-
-- **webapp_name** [*webapps*]: web application package name
-
-- **webapp_port** [*6543*]: TCP/IP port to use when running application outside Apache.
-
-- **eggs_directory** [*eggs*]: relative or absolute path to directory containing downloaded eggs; this directory can be
- shared with other projects.
-
-- **logs_directory** [*/var/log*]: absolute path to directory containing Apache's log files
-
-- **run_user** [*www-data*]: user name under which Apache process will run
-
-- **run_group** [*www-data*]: group name under which Apache process will run
-
-- **beaker_backend** : name of Beaker backend to use to store sessions and cache data ("redis" as default)
-
-- **beaker_server** [*127.0.0.1:6379*]: IP address and port of Beaker backend server
-
-- **db_type** [*zeo*]: ZODB database storage; available options include ZEO, RelStorage and NewtDB
-
-- **db_host** [*127.0.0.1*]: IP address of database server ("127.0.0.1" as default); WARNING: database server installation
- is not part of application installation; another "zeo_server" cookiecutter recipe is available for ZEO
-
-- **db_port** [*8100*]: listening port of database server ("8100" is given as default for ZEO)
-
-- **db_name** [*pyams*]: database or ZEO storage name to use
-
-- **db_username**: database user name
-
-- **db_password**: database password
-
-- **zeo_realm**: ZEO authentication realm
-
-- **blobs_dir** [*/var/db/blobs*]: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default
-
-- **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files
- are available to compile PsycoPG2 extension
-
-- **use_oracle**: specify if Oracle access is required; if so, please check that Oracle development files are
- available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below)
-
-- **use_ldap**: specify if LDAP access will be required for authentication
-
-- **use_elasticsearch**: specify if an ElasticSearch server will be used for indexation
-
-- **elasticsearch_server**: URL used to access Elasticsearch server ("http://127.0.0.1:9200" as default); this URL can
- include login and password ("http://login:password@127.0.0.1:9200"), if required...
-
-- **elasticsearch_index**: name of Elasticsearch index to use ("pyams" as default)
-
-- **create_elasticsearch_index**: specify if Elasticsearch index should be created after installation is complete
-
-- **define_elasticsearch_mappings** : specify if Elasticsearch mappings should be defined after installation is complete
-
-- **smtp_server**: DNS name of SMTP server ("localhost" as default)
-
-- **smtp_server_name**: "human" name given to SMTP server ("pyams" as default)
-
-- **pyams_scheduler** [*127.0.0.1:5555*]: TCP/IP address and port to use to access PyAMS tasks scheduler process (see :ref:`pyams_scheduler`)
-
-- **start_scheduler** [*True*]: boolean value to indicate if scheduler process is started by this application instance
-
-- **pyams_medias_converter** [*127.0.0.1:5556*]: TCP/IP address and port to use to access PyAMS medias converter process (see :ref:`pyams_media`)
-
-- **start_medias_converter** [*True*]: boolean value to indicate if medias converter process is started by this application
- instance
-
-- **pyams_es_indexer** [*127.0.0.1:5557*] : TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (see :ref:`pyams_content_es`)
-
-- **start_es_indexer** boolean value to indicate if Elasticsearch indexer process is started by this application
- instance
-
-- **use_notifications**: specify if PyAMS notifications services are to be used (see :ref:`pyams_notify`)
-
-- **pyams_ws_notify** [*127.0.0.1:8081*]: TCP/IP address and port of PyAMS websockets server managing notifications service
-
-- **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default)
-
-- **extension_package**: name of a PyAMS extension package to include in environment configuration
-
-- **need_pyams_gis**: specify if PyAMS GIS features are to be used by given extension package; if so, please check
- that *libgdal* development files are available; on Debian (and maybe others), you have to specify environment
- variables (see below).
-
-.. tip::
-
- 1. Setting any option to empty string will keep the default proposed value.
- 2. Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*;
-
-You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove
-packages or update settings to your needs.
-
-This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual
-environment. If you encounter any compile error, just install the required dependencies and restart the buildout.
-
-Some dependencies can require the definition of custom environment variables before running *buildout*, like:
-
-- for *libgdal*, which is required by **PyAMS_gis** package, use:
-
-.. code-block:: bash
-
- (env) $ export C_INCLUDE_PATH=/usr/include/gdal
- (env) $ export CPLUS_INCLUDE_PATH=/usr/include/gdal
-
-
-.. WARNING::
- You have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS
- configuration file (actually 2.1.0).
-
-- for *cx_Oracle*, which is required if you use Oracle database connections, use:
-
-.. code-block:: bash
-
- (env) $ export ORACLE_HOME=/usr/lib/oracle/12.1/client64
-
-These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux
-distribution and packages versions.
-
-
-**Then finalize Bootstrap initialization:**
-
-.. code-block:: bash
-
- (env) $ python3.5 bootstrap.py
- (env) $ ./bin/buildout
-
-
-.. seealso::
-
- `zc.buildout <https://pypi.python.org/pypi/zc.buildout>`_ is a powerful
- tool for creating repeatable builds of a given software configuration
- and environment.
-
-
-Environment settings
---------------------
-
-The project generated from *pyams* scaffold is based on default Pyramid's *zodb* scaffold, but it adds:
-
-- a custom application factory, in the *webapp* directory (see :ref:`site`)
-
-- a set of directories to store runtime data, in the *var* directory; each directory contains a *README.txt* file
- which should be self-explanatory to indicate what this directory should contain, including a ZEO cache
-
-- a set of configuration files, in the *etc* directory; here are standard *development.ini* and *production.ini*
- configuration files, a ZODB configuration files (*zodb-zeo.conf*) for a ZEO client storage and two Apache
- configurations (for Apache 2.2 and 2.4) using *mod_wsgi*.
-
-Once the project have been created from the scaffold, you are free to update all the configuration files.
-
-If you need to add packages to the environment, you have to add them to the *buildout.cfg* file **AND** to the INI
-file (in the *pyramid.includes* section) before running the *buildout* another time; don't forget to add the
-requested version at the end of *buildout.cfg* file, as Buildout is not configured by default to automatically
-download the last release of a given unknown package.
-
-*development.ini* and *production.ini* files contain many commented directives related to PyAMS components. Read and
-update them carefully before initializing your application database!
-
-
-.. _zodbinit:
-
-Initializing the ZODB database
-------------------------------
-
-After, you have downloaded and installed all required packages, you have to initialize the database so that all
-required components are available.
-
-From a shell, just type:
-
-.. code-block:: bash
-
- (env) $ ./bin/pyams_upgrade etc/development.ini
-
-This process requires that every package is correctly included into *pyramid.includes* directive from selected
-configuration file.
--- a/src/source/admln_guide/appstart.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,95 +0,0 @@
-.. _appstart:
-
-Starting the application
-========================
-
-When database upgrade process has ended ( :ref:`zodbinit` ) , you can start the web application process with the standard Pyramid's
-*pserve* command line tool:
-
-.. code-block:: bash
-
- (env) $ ./bin/pserve etc/development.ini
-
-In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO):
-
-.. code-block:: bash
-
- 2018-01-14 11:37:54,339 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=28695) created RW/normal for storage: 'pyams'
- 2018-01-14 11:37:54,340 INFO [ZEO.cache][MainThread] created temporary cache file 3
- 2018-01-14 11:37:54,345 INFO [ZODB.blob][MainThread] (28695) Blob directory `/var/local/env/pyams/var/db/blobs` is used but has no layout marker set. Selected `lawn` layout.
- 2018-01-14 11:37:54,345 WARNI [ZODB.blob][MainThread] (28695) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout.
- 2018-01-14 11:37:54,346 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
- 2018-01-14 11:37:54,347 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7feeb1de7390> None
- 2018-01-14 11:37:54,348 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
- 2018-01-14 11:37:54,349 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
- 2018-01-14 11:37:54,355 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
- 2018-01-14 11:37:54,358 DEBUG [txn.140663320073984][MainThread] new transaction
- 2018-01-14 11:37:54,360 DEBUG [txn.140663320073984][MainThread] commit
- 2018-01-14 11:37:54,484 DEBUG [config][MainThread] include /home/tflorac/Dropbox/src/PyAMS/pyams_template/src/pyams_template/configure.zcml
- 2018-01-14 11:37:54,485 DEBUG [config][MainThread] include /var/local/env/pycharm/lib/python3.5/site-packages/pyramid_zcml/configure.zcml
- ...
- 2018-01-14 11:37:54,833 DEBUG [PyAMS (utils)][MainThread] Registering utility <class 'pyams_utils.timezone.utility.TimezoneGenerationsChecker'> named 'PyAMS timezone' providing <InterfaceClass pyams_utils.interfaces.site.ISiteGenerations>
- 2018-01-14 11:37:54,834 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.timezone.vocabulary.TimezonesVocabulary'> as vocabulary with name "PyAMS timezones"
- 2018-01-14 11:37:54,835 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.traversing.PathElementsAdapter'> for (<InterfaceClass zope.location.interfaces.IContained>,) providing <InterfaceClass pyams_utils.interfaces.traversing.IPathElements>
- 2018-01-14 11:37:54,839 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.url.AbsoluteUrlTalesExtension'> for (<InterfaceClass zope.interface.Interface>, <InterfaceClass zope.interface.Interface>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_utils.interfaces.tales.ITALESExtension>
- 2018-01-14 11:37:54,847 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.widget.decimal.DottedDecimalDataConverter'> for (<InterfaceClass pyams_utils.schema.IDottedDecimalField>, <InterfaceClass z3c.form.interfaces.IWidget>) providing <InterfaceClass z3c.form.interfaces.IDataConverter>
- 2018-01-14 11:37:54,942 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.zmi.intids.IntIdsLengthAdapter'> for (<InterfaceClass zope.intid.interfaces.IIntIds>,) providing <InterfaceClass pyams_utils.interfaces.intids.IIndexLength>
- 2018-01-14 11:37:54,943 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass zope.intid.interfaces.IIntIds> (<class 'pyams_utils.zmi.intids.IntIdsPropertiesDisplayForm'>)
- 2018-01-14 11:37:54,949 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass pyams_utils.interfaces.timezone.IServerTimezone> (<class 'pyams_utils.zmi.timezone.ServerTimezonePropertiesEditForm'>)
- 2018-01-14 11:37:54,980 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.zodb.ZEOConnectionVocabulary'> as vocabulary with name "PyAMS ZEO connections"
- 2018-01-14 11:37:54,981 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.zodb.ZODBConnectionVocabulary'> as vocabulary with name "PyAMS ZODB connections"
- 2018-01-14 11:37:55,015 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "add-zeo-connection.html" for <InterfaceClass zope.component.interfaces.ISite> (<class 'pyams_utils.zmi.zeo.ZEOConnectionAddForm'>)
- 2018-01-14 11:37:55,016 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.zmi.zeo.ZEOConnectionNameAdapter'> for (<InterfaceClass pyams_utils.interfaces.zeo.IZEOConnection>, <InterfaceClass pyams_zmi.layer.IAdminLayer>) providing <InterfaceClass pyams_skin.interfaces.container.ITableElementName>
- 2018-01-14 11:37:55,017 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass pyams_utils.interfaces.zeo.IZEOConnection> (<class 'pyams_utils.zmi.zeo.ZEOConnectionPropertiesEditForm'>)
- ...
- 2018-01-14 11:41:13,214 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_default_theme.skin.ResourcesAdapter'> for (<InterfaceClass zope.interface.Interface>, <InterfaceClass pyams_default_theme.layer.IPyAMSDefaultLayer>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_skin.interfaces.resources.IResources>
- 2018-01-14 11:43:36,665 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams'
- 2018-01-14 11:43:36,665 INFO [ZEO.cache][MainThread] created temporary cache file 9
- 2018-01-14 11:43:36,673 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
- 2018-01-14 11:43:36,674 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642)
- 2018-01-14 11:43:36,674 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7ffa54058860> None
- 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0
- 2018-01-14 11:43:36,675 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
- 2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 -->
- 2018-01-14 11:43:36,677 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
- 2018-01-14 11:43:36,679 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
- 2018-01-14 11:43:36,682 DEBUG [txn.140713340237568][MainThread] new transaction
- 2018-01-14 11:43:36,683 DEBUG [txn.140713340237568][MainThread] commit
- 2018-01-14 11:43:36,690 INFO [PyAMS (scheduler][MainThread] Starting tasks scheduler <SchedulerProcess(SchedulerProcess-1, initial)>...
- 2018-01-14 11:43:36,698 INFO [PyAMS (scheduler][MainThread] Started tasks scheduler with PID 29361.
- 2018-01-14 11:43:36,701 INFO [apscheduler.scheduler][MainThread] Scheduler started
- 2018-01-14 11:43:36,702 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run
- 2018-01-14 11:43:36,704 DEBUG [apscheduler.scheduler][APScheduler] No jobs; waiting until a job is added
- 2018-01-14 11:43:36,719 INFO [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams'
- 2018-01-14 11:43:36,720 INFO [ZEO.cache][MainThread] created temporary cache file 15
- 2018-01-14 11:43:36,724 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
- 2018-01-14 11:43:36,725 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7ffa557e8b00> None
- 2018-01-14 11:43:36,726 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
- 2018-01-14 11:43:36,727 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642)
- 2018-01-14 11:43:36,728 INFO [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
- 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 blob cache size: 0
- 2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 -->
- 2018-01-14 11:43:36,732 INFO [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
- 2018-01-14 11:43:36,735 DEBUG [txn.140713340237568][MainThread] new transaction
- 2018-01-14 11:43:36,736 DEBUG [txn.140713340237568][MainThread] commit
- 2018-01-14 11:43:36,743 INFO [PyAMS (media)][MainThread] Starting medias converter <MediaConversionProcess(MediaConversionProcess-2, initial)>...
- 2018-01-14 11:43:36,751 INFO [PyAMS (media)][MainThread] Started medias converter with PID 29367.
- Starting server in PID 29335.
- Serving on http://0.0.0.0:6543
-
-
-Congrats: PyAMS is running !
-
-Launch a browser and open URL http://127.0.0.1:6543/admin to get access to PyAMS management interface.
-
-.. warning::
-
- By default:
-
- - Login: admin
-
- - Password: admin
-
- **You must change it as soon as possible!**
- (see :ref:`pyams_security`)!!.
-
--- a/src/source/admln_guide/architecture.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,47 +0,0 @@
-.. _architecture:
-
-PyAMS application architecture
-==============================
-
-PyAMS applications are built on a small set of prerequisites and external components which *can* or, for some of them,
-*must* be included, the minimal configuration is:
-
-* a mandatory shared ZODB; :ref:`zodb` chapter
-
-* a Redis or Memcached server, to handle cache and sessions;
- :ref:`pyams_cache` chapter
-
-
-Quickstart deployment
----------------------
-
-Overview of the environement system after the :ref:`quickstart` installation.
-It's a very lite configuration, with only mandatory resourses.
-Obviously you can extend this system by connecting more resources like is described in the :ref:`config` chapiter
-
-
-.. image:: _static/architecture_quickstart.png
-
-
-
-Developpement deployment
-------------------------
-
-In additon of mandatory resource **PyAMS** could work with:
-
-* a websockets server (GuniCorn), used to handle desktop notifications;
- you can find information about this component in the :ref:`pyams_notify` chapter;
-
-* an Elasticsearch index, used to handle quick and optimized searching.
-
-
-.. image:: _static/architecture_dev.png
-
-
-Production deployment
----------------------
-
-Use case in a more complete prodution environement.
-
-.. image:: _static/architecture_prod.png
-
--- a/src/source/admln_guide/plugins.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,209 +0,0 @@
-.. _plugins:
-
-PyAMS additional features and services
-======================================
-
-
-Elasticsearch
-+++++++++++++
-
-At first you need to install ElasticSearch (ES); PyAMS is actually compatible with version 5.4. The Ingest attachment
-plug-in is also required to handle attachments correctly.
-
-Visit https://www.elastic.co/ to learn how to install Elasticsearch Server and `ingest-attachment` plug-in
-
-
-.. tip:: Documentation for installing ElasticSearch 5.4
-
- - https://www.elastic.co/guide/en/elasticsearch/reference/5.4/gs-installation.html
- - https://www.elastic.co/guide/en/elasticsearch/plugins/5.4/ingest-attachment.html
-
-
-After Elasticsearch installation, following steps describe how to configure ES with PyAMS.
-
-
-Initializing Elasticsearch index
---------------------------------
-
-If you want to use an Elasticsearch index, you have to initialize index settings and mappings;
-Elasticsearch integration is defined through the *PyAMS_content_es* package.
-
-
-1. Enable service
-'''''''''''''''''
-
-In Pyramid INI application files (*etc/development.ini* and *etc/production.ini*):
-
-.. code-block:: ini
-
- # Elasticsearch server settings
- elastic.server = http://127.0.0.1:9200
- elastic.index = pyams
-
-Where:
- - **elastic.server**: address of Elasticsearch server; you can include authentication arguments in the form
- *http://login:password@w.x.y.z:9200*
- - **elastic.index**: name of Elasticsearch index.
-
-
-On startup, main PyAMS application process can start in *indexer* process which will handle indexing requests in
-asynchronous mode; this process settings are defined like this:
-
-.. code-block:: ini
-
- # PyAMS content Elasticsearch indexer process settings
- pyams_content.es.tcp_handler = 127.0.0.1:5557
- pyams_content.es.start_handler = false
- pyams_content.es.allow_auth = admin:admin
- pyams_content.es.allow_clients = 127.0.0.1
-
-Where:
- - **pyams_content.es.tcp_handler**: IP address and listening port of PyAMS indexer process
- - **pyams_content.es.start_handler**: if *true*, the indexer process is started on PyAMS startup; otherwise (typically
- in a cluster configuration), the process is supposed to be started from another *master* server
- - **pyams_content.es.allow_auth**: login and password to be used to connect to indexer process (settings are defined
- in the same way on indexer process and on all it's clients)
- - **pyams_content.es.allow_clients**: list of IP addresses allowed to connect to indexer process.
-
-
-2. Initialize Elasticsearch database
-''''''''''''''''''''''''''''''''''''
-
-Configuration files for attachment pipeline, index and mappings settings are available into `pyams_content_es` source
-package or in PyAMS installation folder:
-
-
-.. code-block:: bash
-
- (env) $ cd docs/elasticsearch
- (env) $ curl --noproxy localhost -XPUT http://localhost:9200/_ingest/pipeline/attachment -d @attachment-pipeline.json
-
-
-And with ``elastic.index = pyams`` defined as Elasticsearch index name: *"http://localhost:9200/pyams"*:
-
-.. code-block:: shell
-
- (env) $ curl -XDELETE http://localhost:9200/pyams
-
- (env) $ curl -XPUT http://localhost:9200/pyams -d @index-settings.json
-
- (env) $ curl -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json
- (env) $ curl -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json
- (env) $ curl -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json
-
-
-*Troubleshooting*: If you have a 406 error try to add ``-H 'Content-Type: application/json'`` in Curl command lines.
-
-
-3. Update index contents
-''''''''''''''''''''''''
-
-If your ZODB database already store contents, you can update ElasticSearch indexes with all these contents with
-``pymas_es_index`` command line script. From a shell:
-
-.. code-block:: bash
-
- (env) $ ./bin/pyams_es_index ../etc/development.ini
-
-
-
-Natural Language Toolkit - NLTK
-+++++++++++++++++++++++++++++++
-
-PyAMS is using NLTK features through the *PyAMS_calalog*.
-
-.. seealso::
-
- Visit https://www.nltk.org/ to learn more about NLTK
-
-
-Initializing NLTK (Natural Language ToolKit)
---------------------------------------------
-
-Some NLTK collections like **tokenizers** and **stopwords** utilities are used to index fulltext contents
-elements. You can enhance NLTK indexation according to your own needs. This package requires downloading and
-configuration of several elements which are done as follow:
-
-
-*1. Run the Python shell into PyAMS environment:*
-
-.. code-block:: bash
-
- (env) $ ./bin/py
-
-
-*2. In the Python shell:*
-
-.. code-block:: pycon
-
- >>> import nltk
- >>> nltk.download()
-
-
-*3. Configuration installation directory:*
-
-.. tip::
-
- On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running
- Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and
- '*/usr/local/lib/nltk_data*'
-
- Please check if you have permission to write to this directory!
-
-
-.. code-block:: shell
-
- NLTK Downloader
- ---------------------------------------------------------------------------
- d) Download l) List u) Update c) Config h) Help q) Quit
- ---------------------------------------------------------------------------
- Downloader> c
-
- Data Server:
- - URL: <https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml>
- - 6 Package Collections Available
- - 107 Individual Packages Available
-
- Local Machine:
- - Data directory: /home/tflorac/nltk_data
-
- Config> d
- New directory> /usr/local/lib/nltk_data
-
-
-*4. Return to the main menu:*
-
-.. code-block:: shell
-
- ---------------------------------------------------------------------------
- s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu
- ---------------------------------------------------------------------------
- Config> m
-
-
-*5. Download utilities:*
-
- punkt
- Punkt Tokenizer Models
- stopwords
- Stopwords Corpus
-
-
-.. code-block:: shell
-
- ---------------------------------------------------------------------------
- d) Download l) List u) Update c) Config h) Help q) Quit
- ---------------------------------------------------------------------------
- Downloader> d
- Download which package (l=list; x=cancel)?
- Identifier> punkt
- Downloading package punkt to /usr/local/lib/nltk_data...
- Downloader> d
- Download which package (l=list; x=cancel)?
- Identifier> stopwords
- Downloading package stopwords to /usr/local/lib/nltk_data...
-
-
-.. tip::
-
- The full list of NTLK Collection can be displayed with the ``l) list`` option.
--- a/src/source/admln_guide/scripts.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,49 +0,0 @@
-.. _scripts:
-
-PyAMS maintenance scripts
-=========================
-
-
-pyams_upgrade
--------------
-
-
-Pyramid allows to define custom command line scripts for application management. A script called *pyams_upgrade* is
-provided by PyAMS_utils package; this script apply the same process as PyAMS site factory, but can also be used to
-manage **database generations**. The idea behind this is just to allow custom packages to provide a way to check and
-upgrade database configuration away from application startup process:
-
-.. code-block:: bash
-
- # ./bin/pyams_upgrade webapp/development.ini
-
-
-A **site generation checker** is just a named utility providing :py:class:`pyams_utils.interfaces.site.ISiteGenerations`
-interface. For example, **pyams_security** package provides such utility, to make sure that local site manager
-contains a PyAMS security manager and a principal annotation utility:
-
-.. code-block:: python
-
- from pyams_utils.site import check_required_utilities
-
- REQUIRED_UTILITIES = ((ISecurityManager, '', SecurityManager, 'Security manager'),
- (IPrincipalAnnotationUtility, '', PrincipalAnnotationUtility, 'User profiles'))
-
- @utility_config(name='PyAMS security', provides=ISiteGenerations)
- class SecurityGenerationsChecker(object):
- """I18n generations checker"""
-
- generation = 1
-
- def evolve(self, site, current=None):
- """Check for required utilities"""
- check_required_utilities(site, REQUIRED_UTILITIES)
-
-:py:func:`check_required_utilities <pyams_utils.site.check_required_utilities>` is a PyAMS_utils utility function which
-can to used to verify that a set of local utilities are correctly registered with the given names and interfaces.
-
-
-zodbupdate
-----------
-
-:ref:`renamehowto`
--- a/src/source/admln_guide/security.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,1 +0,0 @@
-.. _security:
--- a/src/source/admln_guide/zodb.rst Tue Dec 11 17:00:42 2018 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,209 +0,0 @@
-.. _zodb:
-
-
-ZODB server
-============
-
-
-Installation
-''''''''''''
-
-PyAMS relies mainly on a ZODB (**\Z**\ope **\O**\bjects **\D**\ata\ **\B**\ase) to store it's configuration and for exemple *PyAMS_content*, but other packages may
-rely on other database(s).
-
-Concurrent accesses are **required** on the ZODB, to fully enjoy PyAMS.
-Several ZODB storages implementations providing a shared access are available and the installation are describe below:
-
- * `ZEO Server`_
- * `RelStorage`_
- * `Newt.DB`_
-
-
-.. note::
-
- When you start your application in "*single process*" mode, PyAMS packages start several processes
- and the "*synchronization*" is done via **ØMQ**.
-
-
-
-.. _`ZEO Server`:
-
-1. ZEO server
-+++++++++++++
-
-ZEO (**\Z**\ope **\E**\nterprise **\O**\bjects) is the first available implementation available for concurrent access to a
-FileStorage, provided through the ZEO package.
-
-.. seealso::
-
- ZEO package documentation and complete configuration settings are available on PyPI_ (cf: `ZOE 5.2.0`_).
-
-.. _PyPI: https://pypi.python.org/pypi/ZEO
-.. _`ZOE 5.2.0`: https://pypi.python.org/pypi/ZEO/5.2.0
-
-
-a) Installation with initial buildout
--------------------------------------
-
-PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template.
-
-A simple option to create a ZEO server is to create a buildout environment including *ZEO* and *ZODB* packages:
-
-.. code-block:: bash
-
- $ mkdir /var/local/
- $ pip3 install virtualenv
- $ virtualenv --python=python3.5 env
- $ cd env
- $ . bin/activate
- (env) $ pip3.5 install cookiecutter
- (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server
-
-*CookieCutter* will ask you for a small set of input variables that you can change or not:
-
-- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release;
- you can also choose to point to a given release ("0.1.4" for example)
-
-- **project_name**: current environment name in "human form"
-
-- **project_slug**: "technical" package name, based on project name
-
-- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be
- shared with other projects ("eggs" as default)
-
-- **run_user**: user name under which ZEO process will run ("zeo" as default)
-
-- **run_group**: group name under which ZEO process will run ("zeo" as default)
-
-- **zeo_server_port**: listening port of ZEO server ("8100" as default)
-
-- **zeo_storage**: name of first ZEO storage ("pyams" as default)
-
-- **zeo_pack_report**: email address to which pack reports should be sent
-
-- **logs_directory**: absolute path to directory containing ZEO's log files.
-
-A message is displayed after initialization to finalize environment creation:
-
-.. code::
-
- Your ZEO environment is initialized.
- To finalize it''s creation, just type:
- - cd zeo_server
- - python3.5 bootstrap.py
- - ./bin/buildout
-
- To initialize authentication database, please run following command after buildout:
- ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx
-
-b) ZEO server configuration
----------------------------
-
-All ZEO configuration files are generated in "etc" subdirectory. These includes:
-
-- **etc/zeo_server-zdaemon.conf**: ZDaemon configuration file
-
-- **etc/zeo_server-zeo.conf**: ZEO server configuration file
-
-- **etc/auth.db**: ZEO authentication file; WARNING: this file is not created automatically, you have to create it
- after buildout.
-
-In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
-*CookieCutter* template creation.
-
-
-c) ZEO server tools
--------------------
-
-A set of system configuration files are produced to handle your ZEO environment. These includes:
-
-- **etc/init.d/zeo-zeo_server**: ZEO server start/stop script in Init-D format. Create a link to this file in
- */etc/init.d* and update Init.d scripts (*update-rc.d zeo-zeo_server defaults*) to include ZEO in server start/stop
- process. You can also use this script to start/stop ZEO by hand with *start* and *stop* arguments.
-
-- **etc/systemd/zeo-zeo_server.service**: SystemD service configuration file for ZEO server. Create a link to this
- file in */etc/systemd/system* and reload SystemD daemon (*systemctl daemon-reload*) before activating ZEO service
- (*systemctl enable zeo-zeo_server.service* and *systemctl start zeo-zeo_server.service*).
-
-- **etc/logrotate.d/zeo-zeo_server**: LogRotate configuration file for ZEO log files. Create a link to this file in
- */etc/logrotate.d* to activate log rotation for ZEO server.
-
-- **etc/cron.d/pack-zeo-zeo_server**: Cron configuration file for ZEO database packing. Just create a link to this
- file in */etc/cron.d* directory to enable ZODB packing on a weekly basis (by default).
-
-In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
-*CookieCutter* template creation. All directory names are those used on a Debian GNU/Linux distribution and may have
-to be changed on other distributions.
-
-
-.. _RelStorage:
-
-2. RelStorage server
-++++++++++++++++++++
-
-RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores
-Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are
-supported.
-
-To create a database compatible with RelStorage, you just have to install the database server and create a database
-dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup.
-
-RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get
-benefit from many extensions (clustering, fail-over, hot-standby...) provided by these databases.
-
-
-.. _Newt.DB:
-
-3. NewtDB server
-++++++++++++++++
-
-NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is
-dedicated to PostgreSQL (>= 9.5).
-
-NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB
-column. The JSON data supplements the native data to support indexing, search, and access from non-Python application.
-Because the JSON format is lossy, compared to the native format, the native format is still used for loading
-objects from the database. For this reason, the JSON data are read-only.
-
-Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a
-convenient API for raw data searches.
-
-Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available
-if you need to switch from a classic RelStorage database to a Newt database.
-
-
-.. _ZODB.migration:
-
-Migration
-'''''''''
-
-After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command
-line script provided by RelStorage.
-
-This Python script is using a configuration file containing directives of both source and target databases, which can
-be any storage described in the previous sections (or can even use the same storage).
-
-Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage:
-
-
-.. code-block:: html
-
- %import relstorage
-
- <zeoclient source>
- server zeo-server.mydomain:8100
- storage pyams
- blob-dir /var/local/env/zeo/var/zeoclient/blobs
- shared-blob-dir false
- </zeoclient>
-
- <relstorage destination>
- keep-history false
- blob-dir /var/local/env/pyams/var/relstorage/blobs
- shared-blob-dir false
- <postgresql>
- dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx'
- </postgresql>
- </relstorage>
-
-
--- a/src/source/index.rst Tue Dec 11 17:00:42 2018 +0100
+++ b/src/source/index.rst Thu Dec 13 10:51:53 2018 +0100
@@ -20,6 +20,8 @@
Overview <overview>
Quickstart <quickstart>
+ Glossary <glossary>
+
.. toctree::
:maxdepth: 2
@@ -27,13 +29,13 @@
PyAMS Architectures <admin_guide/architecture>
PyAMS Security model <admin_guide/security>
- Installing PyAMS <admin_guide/appinstall>
- Database initialization <admin_guide/appinstall>
- Database maintenance tasks
- Scheduling tasks
- Command line scripts
- Application tuning
- Troubleshooting
+ Installing PyAMS <admin_guide/pyams-install>
+ Database initialization <admin_guide/zodb-init>
+ Database maintenance tasks <admin_guide/zodb-support>
+ Scheduling tasks <admin_guide/tasks>
+ Command line scripts <admin_guide/scripts>
+ Application tuning <admin_guide/app-tuning>
+ Troubleshooting <admin_guide/troubleshooting>
Configuring and Running PyAMS <app>
Managing PyAMS environment <manage>