# HG changeset patch # User Damien Correia # Date 1522772268 -7200 # Node ID f234f01d54f9a9b891112f0b9ebb0a85958e1cd0 # Parent eeef4052fcd16030c41eb4afe363f62730649375 Updated doc diff -r eeef4052fcd1 -r f234f01d54f9 src/source/appinstall.rst --- a/src/source/appinstall.rst Thu Mar 29 13:42:57 2018 +0200 +++ b/src/source/appinstall.rst Tue Apr 03 18:17:48 2018 +0200 @@ -3,31 +3,39 @@ Installing PyAMS application ============================ -PyAMS default installation is based on `Buildout `_ utility. It's not mandatory to use a -virtual environment, but it allows you to have a better control over your Python resources. +**PyAMS** default installation is based on `Buildout `_ utility. + +.. tip:: + + Take a look at virtualenv_. This tool provide isolated Python environments, which are more practical than installing packages systemwide. + They also allow installing packages without administrator privileges. -Current PyAMS version is based and validated for Python 3.5; your Python environment must also include a C -compiler as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and -eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. *Cython* compiler can also be useful to optimize +.. _virtualenv: https://virtualenv.pypa.io/en/stable/ + + +Current PyAMS version is based and validated for **Python 3.5**; your Python environment must also include a **C compiler** +as well as development headers for Python, *libjpeg*, *libpng*, *libfreetype*, *libxml2*, *libxslt* and +eventually *libldap*, *libpq*, *libffi*, *libgdal* or *libzmq*. **Cython** compiler can also be useful to optimize several packages. -PyAMS default components configuration also pre-suppose that the following external tools are available: -- a *Memcached* or *Redis* server, to store sessions and cache (can be changed through Beaker configuration) +.. note:: -Optional tools also include: + PyAMS default components configuration also pre-suppose that the following external services are available. -- an *LDAP* server for authentication + * Mandatory services: + + - a **ZODB shared server** (see :ref:`zodb` before). Several background processes needing a concurrent access to ZODB are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db. -- an *ElasticSearch* server for full text indexing (see *PyAMS_content_es* package) + - a **Memcached** or **Redis*** server, to store sessions and cache -- a *WebSockets* server using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* - packages). An *out of the box* environment can be built using *pyams_notify* scaffold. + * Optional services: + - an **LDAP server** for authentication -PyAMS also needs that you use a ZODB shared server, as several background processes needing a concurrent access to ZODB -are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db. -See :ref:`zodb` to know how to initialize database with the help of PyAMS tools. + - an **ElasticSearch server** for full text indexing (see *PyAMS_content_es* package) + + - a **WebSockets server** using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* packages). An *out of the box* environment can be built using *pyams_notify* scaffold. Creating initial buildout @@ -39,50 +47,50 @@ .. code-block:: bash - # mkdir /var/local/ - # pip3 install virtualenv - # virtualenv --python=python3.5 env - # cd env - # . bin/activate - (env) # pip3.5 install cookiecutter - (env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams + $ mkdir /var/local/ + $ pip3 install virtualenv + $ virtualenv --python=python3.5 env + $ cd env + $ . bin/activate + (env) $ pip3.5 install cookiecutter + (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams -*CookieCutter* will ask you for a small set of input variables that you can change or not: +*CookieCutter* will ask you for a small set of input variables that you can change or not (**parameter** [*default value*]:) -- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release; +- **pyams_release** [*latest*] : version of PyAMS configuration file to use. "latest" will point to last release; you can also choose to point to a given release ("0.1.4" for example) -- **project_name**: current environment name in "human form" +- **project_name** [*PyAMS*]: current environment name in "human form" -- **project_slug**: "technical" package name, based on project name +- **project_slug**: [*pyams*] "technical" package name, based on project name -- **virtual_hostname**: Apache virtual-host name +- **virtual_hostname** [*pyams.mydomain.com*]: Apache virtual-host name -- **webapp_name**: web application package name ("webapp" as default) +- **webapp_name** [*webapps*]: web application package name -- **webapp_port**: TCP/IP port to use when running application outside Apache ("6543" as default) +- **webapp_port** [*6543*]: TCP/IP port to use when running application outside Apache. -- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be - shared with other projects ("eggs" as default) +- **eggs_directory** [*eggs*]: relative or absolute path to directory containing downloaded eggs; this directory can be + shared with other projects. -- **logs_directory**: absolute path to directory containing Apache's log files +- **logs_directory** [*/var/log*]: absolute path to directory containing Apache's log files -- **run_user**: user name under which Apache process will run ("www-data" as default) +- **run_user** [*www-data*]: user name under which Apache process will run -- **run_group**: group name under which Apache process will run ("www-data" as default) +- **run_group** [*www-data*]: group name under which Apache process will run -- **beaker_backend**: name of Beaker backend to use to store sessions and cache data ("redis" as default) +- **beaker_backend** : name of Beaker backend to use to store sessions and cache data ("redis" as default) -- **beaker_server**: IP address and port of Beaker backend server ("127.0.0.1:6379" as default) +- **beaker_server** [*127.0.0.1:6379*]: IP address and port of Beaker backend server -- **db_type**: ZODB database storage; available options include ZEO, RelStorage and NewtDB +- **db_type** [*zeo*]: ZODB database storage; available options include ZEO, RelStorage and NewtDB -- **db_host**: IP address of database server ("127.0.0.1" as default); WARNING: database server installation +- **db_host** [*127.0.0.1*]: IP address of database server ("127.0.0.1" as default); WARNING: database server installation is not part of application installation; another "zeo_server" cookiecutter recipe is available for ZEO -- **db_port**: listening port of database server ("8100" is given as default for ZEO) +- **db_port** [*8100*]: listening port of database server ("8100" is given as default for ZEO) -- **db_name**: database or ZEO storage name to use +- **db_name** [*pyams*]: database or ZEO storage name to use - **db_username**: database user name @@ -90,7 +98,7 @@ - **zeo_realm**: ZEO authentication realm -- **blobs_dir**: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default +- **blobs_dir** [*/var/db/blobs*]: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default - **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files are available to compile PsycoPG2 extension @@ -115,27 +123,23 @@ - **smtp_server_name**: "human" name given to SMTP server ("pyams" as default) -- **pyams_scheduler**: TCP/IP address and port to use to access PyAMS tasks scheduler process ("127.0.0.1:5555" as - default); see :ref:`pyams_scheduler` +- **pyams_scheduler** [*127.0.0.1:5555*]: TCP/IP address and port to use to access PyAMS tasks scheduler process (see :ref:`pyams_scheduler`) -- **start_scheduler**: boolean value to indicate if scheduler process is started by this application instance +- **start_scheduler** [*True*]: boolean value to indicate if scheduler process is started by this application instance -- **pyams_medias_converter**: TCP/IP address and port to use to access PyAMS medias converter process ("127.0.0.1:5556" - as default); see :ref:`pyams_medias` +- **pyams_medias_converter** [*127.0.0.1:5556*]: TCP/IP address and port to use to access PyAMS medias converter process (see :ref:`pyams_media`) -- **start_medias_converter**: boolean value to indicate if medias converter process is started by this application +- **start_medias_converter** [*True*]: boolean value to indicate if medias converter process is started by this application instance -- **pyams_es_indexer**: TCP/IP address and port to use to access PyAMS Elasticsearch indexer process ("127.0.0.1:5557" - as default); see :ref:`pyams_content_es` +- **pyams_es_indexer** [*127.0.0.1:5557*] : TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (see :ref:`pyams_content_es`) - **start_es_indexer** boolean value to indicate if Elasticsearch indexer process is started by this application instance - **use_notifications**: specify if PyAMS notifications services are to be used (see :ref:`pyams_notify`) -- **pyams_ws_notify**: TCP/IP address and port of PyAMS websockets server managing notifications service - ("127.0.0.1:8081" as default) +- **pyams_ws_notify** [*127.0.0.1:8081*]: TCP/IP address and port of PyAMS websockets server managing notifications service - **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default) @@ -145,17 +149,13 @@ that *libgdal* development files are available; on Debian (and maybe others), you have to specify environment variables (see below). +.. tip:: -Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*; setting -any option to empty string will keep the default proposed value. + 1. Setting any option to empty string will keep the default proposed value. + 2. Boolean values accept "true", "yes", "on" or "1" (in any case) as *true* values, and anything else as *false*; You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove -packages or update settings to your needs. Then finalize Bootstrap initialization: - -.. code-block:: bash - - (env) # python3.5 bootstrap.py - (env) # ./bin/buildout +packages or update settings to your needs. This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual environment. If you encounter any compile error, just install the required dependencies and restart the buildout. @@ -166,22 +166,32 @@ .. code-block:: bash - (env) # export C_INCLUDE_PATH=/usr/include/gdal - (env) # export CPLUS_INCLUDE_PATH=/usr/include/gdal + (env) $ export C_INCLUDE_PATH=/usr/include/gdal + (env) $ export CPLUS_INCLUDE_PATH=/usr/include/gdal -**WARNING**: you have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS + +.. WARNING:: + You have to check also that your *libgdal* release is matching "GDAL" release given in PyAMS configuration file (actually 2.1.0). - for *cx_Oracle*, which is required if you use Oracle database connections, use: .. code-block:: bash - (env) # export ORACLE_HOME=/usr/lib/oracle/12.1/client64 + (env) $ export ORACLE_HOME=/usr/lib/oracle/12.1/client64 These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux distribution and packages versions. +**Then finalize Bootstrap initialization:** + + +.. code-block:: bash + + (env) $ python3.5 bootstrap.py + (env) $ ./bin/buildout + Environment settings -------------------- @@ -207,24 +217,24 @@ update them carefully before initializing your application database! -Initializing the database -------------------------- +Initializing the ZODB database +========================= -When you have downloaded and installed all required packages, you have to initialize the database so that all +After, you have downloaded and installed all required packages, you have to initialize the database so that all required components are available. From a shell, just type: .. code-block:: bash - (env) # ./bin/pyams_upgrade etc/development.ini + (env) $ ./bin/pyams_upgrade etc/development.ini This process requires that every package is correctly included into *pyramid.includes* directive from selected configuration file. Initializing Elasticsearch index --------------------------------- +================================ If you want to use an Elasticsearch index, you have to initialize index settings and mappings; the Ingest attachment plug-in is also required to handle attachments correctly. @@ -234,26 +244,26 @@ .. code-block:: bash - (env) # cd /var/local/src/pyams/pyams_content_es - (env) # curl --noproxy localhost -XDELETE http://localhost:9200/pyams (1) - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams -d @index-settings.json + (env) $ cd /var/local/src/pyams/pyams_content_es + (env) $ curl --noproxy localhost -XDELETE http://localhost:9200/pyams (1) + (env) $ curl --noproxy localhost -XPUT http://localhost:9200/pyams -d @index-settings.json - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json - (env) # curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json + (env) $ curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json + (env) $ curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json + (env) $ curl --noproxy localhost -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json (1) If 'pyams' is defined as Elasticsearch index name. -NLTK initialization -------------------- +Initializing NLTK +================= Some NLTK (Natural Language Toolkit) tokenizers and stopwords utilities are used to index fulltext contents elements. This package requires downloading and configuration of several elements which are done as follow: .. code-block:: bash - (end) # ./bin/py + (end) $ ./bin/py >>> import nltk >>> nltk.download() NLTK Downloader @@ -299,14 +309,14 @@ Starting the application ------------------------- +======================== When database upgrade process has ended, you can start the web application process with the standard Pyramid's *pserve* command line tool: .. code-block:: bash - (env) # ./bin/pserve etc/development.ini + (env) $ ./bin/pserve etc/development.ini In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO): @@ -379,3 +389,4 @@ From this point, you can launch a browser and open URL *http://127.0.0.1:6543/admin* to get access to PyAMS management interface; default login is "admin/admin", that you may change as soon as possible (see :ref:`pyams_security`)!!. + diff -r eeef4052fcd1 -r f234f01d54f9 src/source/index.rst --- a/src/source/index.rst Thu Mar 29 13:42:57 2018 +0200 +++ b/src/source/index.rst Tue Apr 03 18:17:48 2018 +0200 @@ -10,19 +10,14 @@ Contents --------- -.. toctree:: - :maxdepth: 2 - :caption: GENERAL - Introduction - Application architecture .. toctree:: :maxdepth: 2 - :caption: GET STARTED + :caption: GETTING STARTED + Introduction Quickstart - Installation .. toctree:: :maxdepth: 2 @@ -32,10 +27,14 @@ .. toctree:: :maxdepth: 2 - :caption: DEVELOPMENT + :caption: DEVELOPER DOCUMENTATION - Admin's Guide - Developer’s Guide + Application architecture + ZODB Server + PyAMS Installation + Settings + API documentation + PyAMS external packages diff -r eeef4052fcd1 -r f234f01d54f9 src/source/install.rst --- a/src/source/install.rst Thu Mar 29 13:42:57 2018 +0200 +++ b/src/source/install.rst Tue Apr 03 18:17:48 2018 +0200 @@ -6,6 +6,5 @@ .. toctree:: :maxdepth: 2 - zodb appinstall diff -r eeef4052fcd1 -r f234f01d54f9 src/source/quickstart.rst --- a/src/source/quickstart.rst Thu Mar 29 13:42:57 2018 +0200 +++ b/src/source/quickstart.rst Tue Apr 03 18:17:48 2018 +0200 @@ -243,14 +243,25 @@ See :ref:`appinstall` to get more detailed information about installation process. -3. Start application -+++++++++++++++++ +After *buildout*, just check INI files in *etc* directory (see :ref:`config`) + -Application is ready to run! Just check INI files in *etc* directory (see :ref:`config`) and start the application: +3. Initializing the database +++++++++++++++++++++++++++++ + +You have to initialize the database to populate **PyAMS** objects into database. From a shell: .. code-block:: bash - (env)$ ./bin/pyams_upgrade etc/development.ini + (env) $ ./bin/pyams_upgrade etc/development.ini + + +4. Start application +++++++++++++++++++++ + +Application is ready to run! Start the application: + +.. code-block:: bash (env)$ ./bin/pserve etc/development.ini ... diff -r eeef4052fcd1 -r f234f01d54f9 src/source/zodb.rst --- a/src/source/zodb.rst Thu Mar 29 13:42:57 2018 +0200 +++ b/src/source/zodb.rst Tue Apr 03 18:17:48 2018 +0200 @@ -1,28 +1,49 @@ .. _zodb: -Creating ZODB -============= + +ZODB server +============ + + +Installation +'''''''''''' -PyAMS primarily relies on a ZODB (Zope Objects DataBase) to store it's configuration. Other packages may -rely on other database(s), but *PyAMS_content* package also stores it's contents in a ZODB. +PyAMS relies mainly on a ZODB (**\Z**\ope **\O**\bjects **\D**\ata\ **\B**\ase) to store it's configuration and for exemple *PyAMS_content*, but other packages may +rely on other database(s). -As some PyAMS packages start several processes ("synchronization" is done via **ØMQ**), concurrent accesses are -required on the ZODB (even when you start your application in "single process" mode); several ZODB storages -implementations providing a shared access are available: `ZEO`_, `RelStorage`_ and `Newt.DB`_. +Concurrent accesses are **required** on the ZODB, to fully enjoy PyAMS. +Several ZODB storages implementations providing a shared access are available and the installation are describe below: + + * `ZEO Server`_ + * `RelStorage`_ + * `Newt.DB`_ -.. _ZEO: +.. note:: + + When you start your application in "*single process*" mode, PyAMS packages start several processes + and the "*synchronization*" is done via **ØMQ**. + + -Installing a ZEO server -+++++++++++++++++++++++ +.. _`ZEO Server`: -ZEO (Zope Enterprise Objects) is the first available implementation available for concurrent access to a +1. ZEO server ++++++++++++++ + +ZEO (**\Z**\ope **\E**\nterprise **\O**\bjects) is the first available implementation available for concurrent access to a FileStorage, provided through the ZEO package. -ZEO package documentation and complete configuration settings are available on PyPI. +.. seealso:: + + ZEO package documentation and complete configuration settings are available on PyPI_ (cf: `ZOE 5.2.0`_). -Creating initial buildout -------------------------- +.. _PyPI: https://pypi.python.org/pypi/ZEO +.. _`ZOE 5.2.0`: https://pypi.python.org/pypi/ZEO/5.2.0 + + +a) Installation with initial buildout +------------------------------------- PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template. @@ -30,13 +51,13 @@ .. code-block:: bash - # mkdir /var/local/ - # pip3 install virtualenv - # virtualenv --python=python3.5 env - # cd env - # . bin/activate - (env) # pip3.5 install cookiecutter - (env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server + $ mkdir /var/local/ + $ pip3 install virtualenv + $ virtualenv --python=python3.5 env + $ cd env + $ . bin/activate + (env) $ pip3.5 install cookiecutter + (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server *CookieCutter* will ask you for a small set of input variables that you can change or not: @@ -75,8 +96,8 @@ To initialize authentication database, please run following command after buildout: ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx -ZEO server configuration ------------------------- +b) ZEO server configuration +--------------------------- All ZEO configuration files are generated in "etc" subdirectory. These includes: @@ -90,8 +111,9 @@ In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during *CookieCutter* template creation. -ZEO server tools ----------------- + +c) ZEO server tools +------------------- A set of system configuration files are produced to handle your ZEO environment. These includes: @@ -116,8 +138,8 @@ .. _RelStorage: -Installing a RelStorage server -++++++++++++++++++++++++++++++ +2. RelStorage server +++++++++++++++++++++ RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are @@ -132,8 +154,8 @@ .. _Newt.DB: -Installing a NewtDB server -++++++++++++++++++++++++++ +3. NewtDB server +++++++++++++++++ NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is dedicated to PostgreSQL (>= 9.5). @@ -150,21 +172,10 @@ if you need to switch from a classic RelStorage database to a Newt database. -.. _ZODB.init: - -Initializing ZODB -+++++++++++++++++ - -When your ZODB is created, you have to initialize it! - -After installing PyAMS application (see :ref:`appinstall`), a database upgrade script is available which allows you -to initialize the database. See :ref:`config` and :ref:`scripts` to know more. - - .. _ZODB.migration: -ZODB migration -++++++++++++++ +Migration +''''''''' After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command line script provided by RelStorage. @@ -174,7 +185,8 @@ Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage: -.. code-block:: + +.. code-block:: html %import relstorage @@ -193,3 +205,5 @@ dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx' + +