+ +
+

Creating ZODB

+

PyAMS primarily relies on a ZODB (Zope Objects DataBase) to store it’s configuration. Other packages may +rely on other database(s), but PyAMS_content package also stores it’s contents in a ZODB.

+

As some PyAMS packages start several processes (“synchronization” is done via ØMQ), concurrent accesses are +required on the ZODB (even when you start your application in “single process” mode); several ZODB storages +implementations providing a shared access are available: ZEO, RelStorage and Newt.DB.

+
+

Installing a ZEO server

+

ZEO (Zope Enterprise Objects) is the first available implementation available for concurrent access to a +FileStorage, provided through the ZEO package.

+

ZEO package documentation and complete configuration settings are available on PyPI.

+
+

Creating initial buildout

+

PyAMS provides a ZEO server scaffold, called zeo_server, generated via a cookiecutter template.

+

A simple option to create a ZEO server is to create a buildout environment including ZEO and ZODB packages:

+
# mkdir /var/local/
+# pip3 install virtualenv
+# virtualenv --python=python3.5 env
+# cd env
+# . bin/activate
+(env) # pip3.5 install cookiecutter
+(env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server
+
+
+

CookieCutter will ask you for a small set of input variables that you can change or not:

+
    +
  • pyams_release: version of PyAMS configuration file to use. “latest” (default value) will point to last release; +you can also choose to point to a given release (“0.1.4” for example)
    • +
  • project_name: current environment name in “human form”
    • +
  • project_slug: “technical” package name, based on project name
    • +
  • eggs_directory: relative or absolute path to directory containing downloaded eggs; this directory can be +shared with other projects (“eggs” as default)
    • +
  • run_user: user name under which ZEO process will run (“zeoadm” as default)
    • +
  • run_group: group name under which ZEO process will run (“zeo” as default)
    • +
  • zeo_server_port: listening port of ZEO server (“8100” as default)
    • +
  • zeo_monitor_port: listening port of ZEO monitor (“8101” as default)
    • +
  • zeo_storage: name of first ZEO storage; default value is based on project name
    • +
  • use_zeo_auth: specify if ZEO authentication should be used
    • +
  • zeo_auth_user: name of ZEO authenticated user (if ZEO authentication is used)
    • +
  • zeo_auth_password: password of ZEO authenticated user (if ZEO authentication is used)
    • +
  • zeo_pack_report: email address to which pack reports should be sent
    • +
  • logs_directory: absolute path to directory containing ZEO’s log files.
    • +
+

A message is displayed after initialization to finalize environment creation:

+
Your ZEO environment is initialized.
+To finalize it''s creation, just type:
+- cd zeo_server
+- python3.5 bootstrap.py
+- ./bin/buildout
+
+To initialize authentication database, please run following command after buildout:
+./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx
+
+
+
+
+

ZEO server configuration

+

All ZEO configuration files are generated in “etc” subdirectory. These includes:

+
    +
  • etc/zeo_server-zdaemon.conf: ZDaemon configuration file
    • +
  • etc/zeo_server-zeo.conf: ZEO server configuration file
    • +
  • etc/auth.db: ZEO authentication file; WARNING: this file is not created automatically, you have to create it +after buildout.
    • +
+

In these file names, always replace “zeo_server” with the value which was given to “project_slug” variable during +CookieCutter template creation.

+
+
+

ZEO server tools

+

A set of system configuration files are produced to handle your ZEO environment. These includes:

+
    +
  • etc/init.d/zeo-zeo_server: ZEO server start/stop script in Init-D format. Create a link to this file in +/etc/init.d and update Init.d scripts (update-rc.d zeo-zeo_server defaults) to include ZEO in server start/stop +process. You can also use this script to start/stop ZEO by hand with start and stop arguments.
    • +
  • etc/systemd/zeo-zeo_server.service: SystemD service configuration file for ZEO server. Create a link to this +file in /etc/systemd/system and reload SystemD daemon (systemctl daemon-reload) before activating ZEO service +(systemctl enable zeo-zeo_server.service and systemctl start zeo-zeo_server.service).
    • +
  • etc/logrotate.d/zeo-zeo_server: LogRotate configuration file for ZEO log files. Create a link to this file in +/etc/logrotate.d to activate log rotation for ZEO server.
    • +
  • etc/cron.d/pack-zeo-zeo_server: Cron configuration file for ZEO database packing. Just create a link to this +file in /etc/cron.d directory to enable ZODB packing on a weekly basis (by default).
    • +
+

In these file names, always replace “zeo_server” with the value which was given to “project_slug” variable during +CookieCutter template creation. All directory names are those used on a Debian GNU/Linux distribution and may have +to be changed on other distributions.

+
+
+
+

Installing a RelStorage server

+

RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores +Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are +supported.

+

To create a database compatible with RelStorage, you just have to install the database server and create a database +dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup.

+

RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get +benefit from many extensions (clustering, fail-over, hot-standby…) provided by these databases.

+
+
+

Installing a NewtDB server

+

NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It’s using RelStorage but is +dedicated to PostgreSQL (>= 9.5).

+

NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB +column. The JSON data supplements the native data to support indexing, search, and access from non-Python application. +Because the JSON format is lossy, compared to the native format, the native format is still used for loading +objects from the database. For this reason, the JSON data are read-only.

+

Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a +convenient API for raw data searches.

+

Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available +if you need to switch from a classic RelStorage database to a Newt database.

+
+
+ + +