src/source/admln_guide/zodb.rst
branchdoc-dc
changeset 112 49e4432c0a1d
parent 111 097b0c025eec
child 113 5108336d3a4c
equal deleted inserted replaced
111:097b0c025eec 112:49e4432c0a1d
     1 .. _zodb:
       
     2 
       
     3 
       
     4 ZODB server
       
     5 ============
       
     6 
       
     7 
       
     8 Installation
       
     9 ''''''''''''
       
    10 
       
    11 PyAMS relies mainly on a ZODB (**\Z**\ope **\O**\bjects **\D**\ata\ **\B**\ase) to store it's configuration and for exemple *PyAMS_content*, but other packages may
       
    12 rely on other database(s).
       
    13 
       
    14 Concurrent accesses are **required** on the ZODB, to fully enjoy PyAMS.
       
    15 Several ZODB storages implementations providing a shared access are available and the installation are describe below:
       
    16 
       
    17     * `ZEO Server`_
       
    18     * `RelStorage`_
       
    19     * `Newt.DB`_
       
    20 
       
    21 
       
    22 .. note::
       
    23 
       
    24     When you start your application in "*single process*" mode, PyAMS packages start several processes
       
    25     and the "*synchronization*" is done via **ØMQ**.
       
    26 
       
    27 
       
    28 
       
    29 .. _`ZEO Server`:
       
    30 
       
    31 1. ZEO server
       
    32 +++++++++++++
       
    33 
       
    34 ZEO (**\Z**\ope **\E**\nterprise **\O**\bjects) is the first available implementation available for concurrent access to a
       
    35 FileStorage, provided through the ZEO package.
       
    36 
       
    37 .. seealso::
       
    38 
       
    39     ZEO package documentation and complete configuration settings are available on PyPI_ (cf: `ZOE 5.2.0`_).
       
    40 
       
    41 .. _PyPI: https://pypi.python.org/pypi/ZEO
       
    42 .. _`ZOE 5.2.0`: https://pypi.python.org/pypi/ZEO/5.2.0
       
    43 
       
    44 
       
    45 a) Installation with initial buildout
       
    46 -------------------------------------
       
    47 
       
    48 PyAMS provides a ZEO server scaffold, called *zeo_server*, generated via a *cookiecutter* template.
       
    49 
       
    50 A simple option to create a ZEO server is to create a buildout environment including *ZEO* and *ZODB* packages:
       
    51 
       
    52 .. code-block:: bash
       
    53 
       
    54     $ mkdir /var/local/
       
    55     $ pip3 install virtualenv
       
    56     $ virtualenv --python=python3.5 env
       
    57     $ cd env
       
    58     $ . bin/activate
       
    59     (env) $ pip3.5 install cookiecutter
       
    60     (env) $ cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/zeo_server
       
    61 
       
    62 *CookieCutter* will ask you for a small set of input variables that you can change or not:
       
    63 
       
    64 - **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release;
       
    65   you can also choose to point to a given release ("0.1.4" for example)
       
    66 
       
    67 - **project_name**: current environment name in "human form"
       
    68 
       
    69 - **project_slug**: "technical" package name, based on project name
       
    70 
       
    71 - **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be
       
    72   shared with other projects ("eggs" as default)
       
    73 
       
    74 - **run_user**: user name under which ZEO process will run ("zeo" as default)
       
    75 
       
    76 - **run_group**: group name under which ZEO process will run ("zeo" as default)
       
    77 
       
    78 - **zeo_server_port**: listening port of ZEO server ("8100" as default)
       
    79 
       
    80 - **zeo_storage**: name of first ZEO storage ("pyams" as default)
       
    81 
       
    82 - **zeo_pack_report**: email address to which pack reports should be sent
       
    83 
       
    84 - **logs_directory**: absolute path to directory containing ZEO's log files.
       
    85 
       
    86 A message is displayed after initialization to finalize environment creation:
       
    87 
       
    88 .. code::
       
    89 
       
    90     Your ZEO environment is initialized.
       
    91     To finalize it''s creation, just type:
       
    92     - cd zeo_server
       
    93     - python3.5 bootstrap.py
       
    94     - ./bin/buildout
       
    95 
       
    96     To initialize authentication database, please run following command after buildout:
       
    97     ./bin/zeopasswd -f etc/auth.db -p digest -r "ZEO_server" zeouser xxxx
       
    98 
       
    99 b) ZEO server configuration
       
   100 ---------------------------
       
   101 
       
   102 All ZEO configuration files are generated in "etc" subdirectory. These includes:
       
   103 
       
   104 - **etc/zeo_server-zdaemon.conf**: ZDaemon configuration file
       
   105 
       
   106 - **etc/zeo_server-zeo.conf**: ZEO server configuration file
       
   107 
       
   108 - **etc/auth.db**: ZEO authentication file; WARNING: this file is not created automatically, you have to create it
       
   109   after buildout.
       
   110 
       
   111 In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
       
   112 *CookieCutter* template creation.
       
   113 
       
   114 
       
   115 c) ZEO server tools
       
   116 -------------------
       
   117 
       
   118 A set of system configuration files are produced to handle your ZEO environment. These includes:
       
   119 
       
   120 - **etc/init.d/zeo-zeo_server**: ZEO server start/stop script in Init-D format. Create a link to this file in
       
   121   */etc/init.d* and update Init.d scripts (*update-rc.d zeo-zeo_server defaults*) to include ZEO in server start/stop
       
   122   process. You can also use this script to start/stop ZEO by hand with *start* and *stop* arguments.
       
   123 
       
   124 - **etc/systemd/zeo-zeo_server.service**: SystemD service configuration file for ZEO server. Create a link to this
       
   125   file in */etc/systemd/system* and reload SystemD daemon (*systemctl daemon-reload*) before activating ZEO service
       
   126   (*systemctl enable zeo-zeo_server.service* and *systemctl start zeo-zeo_server.service*).
       
   127 
       
   128 - **etc/logrotate.d/zeo-zeo_server**: LogRotate configuration file for ZEO log files. Create a link to this file in
       
   129   */etc/logrotate.d* to activate log rotation for ZEO server.
       
   130 
       
   131 - **etc/cron.d/pack-zeo-zeo_server**: Cron configuration file for ZEO database packing. Just create a link to this
       
   132   file in */etc/cron.d* directory to enable ZODB packing on a weekly basis (by default).
       
   133 
       
   134 In these file names, always replace "zeo_server" with the value which was given to "project_slug" variable during
       
   135 *CookieCutter* template creation. All directory names are those used on a Debian GNU/Linux distribution and may have
       
   136 to be changed on other distributions.
       
   137 
       
   138 
       
   139 .. _RelStorage:
       
   140 
       
   141 2. RelStorage server
       
   142 ++++++++++++++++++++
       
   143 
       
   144 RelStorage (http://relstorage.readthedocs.io/en/latest) is an alternate ZODB storage implementation, that stores
       
   145 Python pickles in a relational database; PostgreSQL (>= 9.0), MySQL (>= 5.0.32) and Oracle (> 10g) databases are
       
   146 supported.
       
   147 
       
   148 To create a database compatible with RelStorage, you just have to install the database server and create a database
       
   149 dedicated to RelStorage; schema initialization is then completely done by RelStorage on application startup.
       
   150 
       
   151 RelStorage is supposed to provide better performances than ZEO, notably under high load. RelStorage can also get
       
   152 benefit from many extensions (clustering, fail-over, hot-standby...) provided by these databases.
       
   153 
       
   154 
       
   155 .. _Newt.DB:
       
   156 
       
   157 3. NewtDB server
       
   158 ++++++++++++++++
       
   159 
       
   160 NewtDB (http://www.newtdb.org/en/latest) is another ZODB storage implementation. It's using RelStorage but is
       
   161 dedicated to PostgreSQL (>= 9.5).
       
   162 
       
   163 NewtDB adds conversion of data from the native serialization used by ZODB to JSON, stored in a PostgreSQL JSONB
       
   164 column. The JSON data supplements the native data to support indexing, search, and access from non-Python application.
       
   165 Because the JSON format is lossy, compared to the native format, the native format is still used for loading
       
   166 objects from the database. For this reason, the JSON data are read-only.
       
   167 
       
   168 Newt adds a search API for searching the Postgres JSON data and returning persistent objects. It also provides a
       
   169 convenient API for raw data searches.
       
   170 
       
   171 Database creation is done as with RelStorage, but NewtDB add several schema objects. Migration scripts are available
       
   172 if you need to switch from a classic RelStorage database to a Newt database.
       
   173 
       
   174 
       
   175 .. _ZODB.migration:
       
   176 
       
   177 Migration
       
   178 '''''''''
       
   179 
       
   180 After installation, you can switch from a given storage to another one with the help of the *zodbconvert* command
       
   181 line script provided by RelStorage.
       
   182 
       
   183 This Python script is using a configuration file containing directives of both source and target databases, which can
       
   184 be any storage described in the previous sections (or can even use the same storage).
       
   185 
       
   186 Here is a sample configuration file to convert a ZODB from a ZEO to RelStorage:
       
   187 
       
   188 
       
   189 .. code-block:: html
       
   190 
       
   191     %import relstorage
       
   192 
       
   193     <zeoclient source>
       
   194         server zeo-server.mydomain:8100
       
   195         storage pyams
       
   196         blob-dir /var/local/env/zeo/var/zeoclient/blobs
       
   197         shared-blob-dir false
       
   198     </zeoclient>
       
   199 
       
   200     <relstorage destination>
       
   201         keep-history false
       
   202         blob-dir /var/local/env/pyams/var/relstorage/blobs
       
   203         shared-blob-dir false
       
   204         <postgresql>
       
   205             dsn host='postgresql-server.mydomain' dbname='pyams_rs' user='pyams' password='xxxxxxxx'
       
   206         </postgresql>
       
   207     </relstorage>
       
   208 
       
   209