diff -r 000000000000 -r d153941bb745 src/source/zodb.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/zodb.rst Sun Jan 14 11:48:51 2018 +0100 @@ -0,0 +1,158 @@ +.. _zodb: + +Creating ZODB +============= + +PyAMS primarily relies on a ZODB (Zope Objects DataBase) to store it's configuration. Other packages may +rely on other database(s), but *PyAMS_content* package also stores it's contents in a ZODB. + +As some PyAMS packages start several processes ("synchronization" is done via **ØMQ**), concurrent accesses are +required on the ZODB (even when you start your application in "single process" mode); several ZODB storages +implementations providing a shared access are available: `ZEO`_, `RelStorage`_ and `Newt.DB`_. + + +.. _ZEO: + +Installing a ZEO server ++++++++++++++++++++++++ + +ZEO (Zope Enterprise Objects) is the first available implementation available for concurrent access to a +FileStorage, provided through the ZEO package. + +ZEO package documentation and complete configuration settings are available on PyPI. + +Creating initial buildout +------------------------- + +PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template. + +A simple option to create a ZEO server is to create a buildout environment including *ZEO* and *ZODB* packages: + +.. code-block:: bash + + # mkdir /var/local/ + # pip3 install virtualenv + # virtualenv --python=python3.5 env + # cd env + # . bin/activate + (env) # pip3.5 install cookiecutter + (env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server + +*CookieCutter* will ask you for a small set of input variables that you can change or not: + +- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release; + you can also choose to point to a given release ("0.1.4" for example) + +- **project_name**: current environment name in "human form" + +- **project_slug**: "technical" package name, based on project name + +- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be + shared with other projects ("eggs" as default) + +- **run_user**: user name under which ZEO process will run ("zeoadm" as default) + +- **run_group**: group name under which ZEO process will run ("zeo" as default) + +- **zeo_server_port**: listening port of ZEO server ("8100" as default) + +- **zeo_monitor_port**: listening port of ZEO monitor ("8101" as default) + +- **zeo_storage**: name of first ZEO storage; default value is based on project name + +- **use_zeo_auth**: specify if ZEO authentication should be used + +- **zeo_auth_user**: name of ZEO authenticated user (if ZEO authentication is used) + +- **zeo_auth_password**: password of ZEO authenticated user (if ZEO authentication is used) + +- **zeo_pack_report**: email address to which pack reports should be sent + +- **logs_directory**: absolute path to directory containing ZEO's log files. + +A message is displayed after initialization to finalize environment creation: + +.. code:: + + Your ZEO environment is initialized. + To finalize it''s creation, just type: + - cd zeo_server + - python3.5 bootstrap.py + - ./bin/buildout + + To initialize authentication database, please run following command after buildout: + ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx + +ZEO server configuration +------------------------ + +All ZEO configuration files are generated in "etc" subdirectory. These includes: + +- **etc/zeo_server-zdaemon.conf**: ZDaemon configuration file + +- **etc/zeo_server-zeo.conf**: ZEO server configuration file + +- **etc/auth.db**: ZEO authentication file; WARNING: this file is not created automatically, you have to create it + after buildout. + +In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during +*CookieCutter* template creation. + +ZEO server tools +---------------- + +A set of system configuration files are produced to handle your ZEO environment. These includes: + +- **etc/init.d/zeo-zeo_server**: ZEO server start/stop script in Init-D format. Create a link to this file in + */etc/init.d* and update Init.d scripts (*update-rc.d zeo-zeo_server defaults*) to include ZEO in server start/stop + process. You can also use this script to start/stop ZEO by hand with *start* and *stop* arguments. + +- **etc/systemd/zeo-zeo_server.service**: SystemD service configuration file for ZEO server. Create a link to this + file in */etc/systemd/system* and reload SystemD daemon (*systemctl daemon-reload*) before activating ZEO service + (*systemctl enable zeo-zeo_server.service* and *systemctl start zeo-zeo_server.service*). + +- **etc/logrotate.d/zeo-zeo_server**: LogRotate configuration file for ZEO log files. Create a link to this file in + */etc/logrotate.d* to activate log rotation for ZEO server. + +- **etc/cron.d/pack-zeo-zeo_server**: Cron configuration file for ZEO database packing. Just create a link to this + file in */etc/cron.d* directory to enable ZODB packing on a weekly basis (by default). + +In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during +*CookieCutter* template creation. All directory names are those used on a Debian GNU/Linux distribution and may have +to be changed on other distributions. + + +.. _RelStorage: + +Installing a RelStorage server +++++++++++++++++++++++++++++++ + +RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores +Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are +supported. + +To create a database compatible with RelStorage, you just have to install the database server and create a database +dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup. + +RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get +benefit from many extensions (clustering, fail-over, hot-standby...) provided by these databases. + + +.. _Newt.DB: + +Installing a NewtDB server +++++++++++++++++++++++++++ + +NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is +dedicated to PostgreSQL (>= 9.5). + +NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB +column. The JSON data supplements the native data to support indexing, search, and access from non-Python application. +Because the JSON format is lossy, compared to the native format, the native format is still used for loading +objects from the database. For this reason, the JSON data are read-only. + +Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a +convenient API for raw data searches. + +Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available +if you need to switch from a classic RelStorage database to a Newt database.