diff -r 119b9c2f3022 -r 8b2786685e8b docs/source/install.rst --- a/docs/source/install.rst Wed Jul 12 13:56:41 2017 +0200 +++ b/docs/source/install.rst Wed Jul 12 13:57:06 2017 +0200 @@ -16,21 +16,22 @@ Optional tools also include: +- an *LDAP* server for authentication + - an *ElasticSearch* server for full text indexing (see *PyAMS_content_es* package) - a *WebSockets* server using AsyncIO. This is used to manage notifications (see *PyAMS_notify* and *PyAMS_notify_ws* - packages). An *out of the box* environment can be built using *pyams_asyncio* scaffold provided by *pyams_base* - package. + packages). An *out of the box* environment can be built using *pyams_notify* scaffold. -You can also choose to use a local ZODB instance, or a ZEO server (which can be local or remote, but is required if -you want to use PyAMS in a muti-processes configuration). +PyAMS also needs that you use a ZEO server, as several background processes needing a concurrent access to ZODB are +started by PyAMS main process. See :ref:`zeo` to know how to create a ZEO server with the help of PyAMS tools. Creating initial buildout ------------------------- -PyAMS provides a new Pyramid scaffold, called *pyams*, provided by the *pyams_base* package. +PyAMS provides a new Pyramid scaffold, called *pyams*, generated via a *cookiecutter* template. A simple option to install PyAMS is to create a buildout environment including *Pyramid* and all *PyAMS* packages: @@ -41,18 +42,104 @@ # virtualenv --python=python3.5 env # cd env # . bin/activate - (pyams) # pip3.5 install cookiecutter - (pyams) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams + (env) # pip3.5 install cookiecutter + (env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams + +*CookieCutter* will ask you for a small set of input variables that you can change or not: + +- **pyams_release**: version of PyAMS configuration file to use. "latest" (default value) will point to last release; + you can also choose to point to a given release ("0.1.4" for example) + +- **project_name**: current environment name in "human form" + +- **project_slug**: "technical" package name, based on project name + +- **virtual_hostname**: Apache virtual-host name + +- **webapp_name**: web application package name ("webapp" as default) + +- **webapp_port**: TCP/IP port to use when running application outside Apache ("6543" as default) + +- **eggs_directory**: relative or absolute path to directory containing downloaded eggs; this directory can be + shared with other projects ("eggs" as default) + +- **logs_directory**: absolute path to directory containing Apache's log files + +- **run_user**: user name under which Apache process will run ("www-data" as default) + +- **run_group**: group name under which Apache process will run ("www-data" as default) + +- **beaker_backend**: name of Beaker backend to use to store sessions and cache data ("redis" as default) + +- **beaker_server**: IP address and port of Beaker backend server ("127.0.0.1:6379" as default) + +- **zeo_server**: IP address and port of ZEO server ("127.0.0.1:8100" as default); WARNING: ZEO server installation + is not part of application installation; another "zeo_server" cookiecutter receipe is available for that. + +- **zeo_storage**: name of ZEO storage to use + +- **zeo_username**: ZEO user name + +- **zeo_password**: ZEO password + +- **zeo_realm**: ZEO authentication realm + +- **zeo_blobs_dir**: local directory to use to store cache of ZODB blobs; cache size is limited to 1GB as default -CookieCutter will ask for for a small set of input variables that you can change or not. +- **use_postgresql**: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files + are available to compile PsycoPG2 extension + +- **use_oracle**: specify if Oracle access is available; if so, please check that Oracle development files are + available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined + +- **use_ldap**: specify if LDAP access will be required for authentication + +- **use_elasticsearch**: specify if an ElasticSearch server will be used for indexation + +- **elasticsearch_server**: URL used to access Elasticsearch server ("http://127.0.0.1:9200" as default); this URL can + include login and password ("http://login:password@127.0.0.1:9200"), if required... + +- **elasticsearch_index**: name of Eslasticsearch index to use ("pyams" as default) + +- **create_elasticsearch_index**: specify if Elasticsearch index should be created after installation is complete + +- **define_elasticsearch_mappings** : specify if Elasticsearch mappings should be defined after installation is complete + +- **smtp_server**: DNS name of SMTP server ("localhost" as default) + +- **smtp_server_name**: "human" name given to SMTP server ("pyams" as default) + +- **pyams_scheduler**: TCP/IP address and port to use to access PyAMS tasks scheduler process ("127.0.0.1:5555" as + default) + +- **pyams_medias_converter**: TCP/IP address and port to use to access PyAMS medias converter process ("127.0.0.1:5556" + as default) + +- **pyams_es_indexer**: TCP/IP address and port to use to access PyAMS Elasticsearch indexer process ("127.0.0.1:5557" + as default) + +- **use_notifications**: specify if PyAMS notifications services are to be used + +- **pyams_ws_notify**: TCP/IP address and port of PyAMS websockets server managing notifications service + ("127.0.0.1:8081" as default) + +- **lexicon_languages**: NLTK lexicon languages to use ("en:english fr:french" as default) + +- **extension_package**: name of a PyAMS extension package to include in environment configuration + +- **need_pyams_gis**: specify if PyAMS GIS features are to be used by given extension package; if so, please check + that *libgdal* development files are available and on Debian, you have to specify environment variables for + C_INCLUDE_PATH and CPLUS_INCLUDE_PATH pointing to */usr/include/gdal* directory. WARNING: you have to check also + that your *libgdal* release is matching "GDAL" release given in PyAMS configuration file (actually 2.1.0). + You can then check, and eventually update, the proposed Buildout configuration file *buildout.cfg*, to add or remove packages or update settings to your needs. Then finalize Bootstrap initialization: .. code-block:: bash - (pyams) # python3.5 bootstrap.py - (pyams) # ./bin/buildout + (env) # python3.5 bootstrap.py + (env) # ./bin/buildout This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual environment. If you encounter any compile error, just install the required dependencies and restart the buildout. @@ -66,12 +153,11 @@ - a custom application factory, in the *webapp* directory (see :ref:`site`) - a set of directories to store runtime data, in the *var* directory; each directory contains a *README.txt* file - which should be self-explanatory to indicate what this directory should contain, including a ZODB or a ZEO cache + which should be self-explanatory to indicate what this directory should contain, including a ZEO cache - a set of configuration files, in the *etc* directory; here are standard *development.ini* and *production.ini* - configuration files, two ZODB configuration files (*zodb.conf.fs* for a single FileStorage application process, and - *zodb.conf.zeo* for a ZEO client storage; default configuration defined in INI files is based on a single file - storage) and two Apache configurations (for Apache 2.2 and 2.4) using *mod_wsgi*. + configuration files, a ZODB configuration files (*zodb-zeo.conf*) for a ZEO client storage and two Apache + configurations (for Apache 2.2 and 2.4) using *mod_wsgi*. Once the project have been created from the scaffold, you are free to update all the configuration files. @@ -94,7 +180,7 @@ .. code-block:: bash - (pyams) # ./bin/pyams_upgrade etc/development.ini + (env) # ./bin/pyams_upgrade etc/development.ini This process requires that every package is correctly included into *pyramid.includes* directive from selected configuration file. @@ -108,50 +194,78 @@ .. code-block:: bash - (pyams) # ./bin/pserve etc/development.ini + (env) # ./bin/pserve etc/development.ini In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO): .. code-block:: bash - 2016-12-28 10:58:46,347 INFO [ZEO.ClientStorage][MainThread] [('localhost', 8100)] ClientStorage (pid=30133) created RW/normal for storage: 'pyams' - 2016-12-28 10:58:46,349 DEBUG [ZODB.blob][MainThread] (30133) Blob directory `/var/local/env/pyams/db/blobs` has layout marker set. Selected `bushy` layout. - 2016-12-28 10:58:46,349 WARNI [ZODB.blob][MainThread] (30133) Blob dir /var/local/env/pyams/db/blobs/ has insecure mode setting - 2016-12-28 10:58:46,349 INFO [ZEO.cache][MainThread] created temporary cache file 5 - 2016-12-28 10:58:46,373 BLATH [ZEO.zrpc][MainThread] (30133) CM.connect(): starting ConnectThread - 2016-12-28 10:58:46,374 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (30133) CT: attempting to connect on 1 sockets - 2016-12-28 10:58:46,380 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (30133) CW: attempt to connect to ('127.0.0.1', 8100) - 2016-12-28 10:58:46,380 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (30133) CW: connect_ex(('127.0.0.1', 8100)) returned EINPROGRESS - 2016-12-28 10:58:46,381 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (30133) CT: select() 0, 1, 0 - 2016-12-28 10:58:46,381 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (30133) CW: connect_ex(('127.0.0.1', 8100)) returned 0 - 2016-12-28 10:58:46,382 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Testing connection - 2016-12-28 10:58:46,383 INFO [ZEO.zrpc.Connection(b'C')][[('localhost', 8100)] zeo client networking thread] (127.0.0.1:8100) received handshake b'Z3101' - 2016-12-28 10:58:46,483 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Server authentication protocol None - 2016-12-28 10:58:46,484 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Connected to storage: ('localhost', 8100) - 2016-12-28 10:58:46,485 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] No verification necessary -- empty cache - 2016-12-28 10:58:46,486 DEBUG [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] pickled inval None b'\x03\xbc>\x1a\x166f"' - 2016-12-28 10:58:46,488 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (30133) CM.connect_done(preferred=1) - 2016-12-28 10:58:46,489 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (30133) CT: exiting thread: Connect([(, ('localhost', 8100))]) - 2016-12-28 10:58:46,676 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_template-0.1.0-py3.4.egg/pyams_template/configure.zcml - 2016-12-28 10:58:46,677 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/configure.zcml - 2016-12-28 10:58:46,678 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/meta.zcml - 2016-12-28 10:58:46,684 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_template-0.1.0-py3.4.egg/pyams_template/meta.zcml - 2016-12-28 10:58:46,710 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_viewlet-0.1.0-py3.4.egg/pyams_viewlet/configure.zcml - 2016-12-28 10:58:46,711 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/configure.zcml - 2016-12-28 10:58:46,711 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyramid_zcml-1.0.0-py3.4.egg/pyramid_zcml/meta.zcml - 2016-12-28 10:58:46,746 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/zope.browserpage-4.1.0-py3.4.egg/zope/browserpage/meta.zcml - 2016-12-28 10:58:46,825 DEBUG [config][MainThread] include /var/local/env/pyams/eggs/pyams_template-0.1.0-py3.4.egg/pyams_template/meta.zcml + 2017-06-19 17:21:40,495 INFO [ZEO.ClientStorage][MainThread] [('localhost', 8100)] ClientStorage (pid=23253) created RW/normal for storage: 'pyams' + 2017-06-19 17:21:40,497 DEBUG [ZODB.blob][MainThread] (23253) Blob directory `var/db/blobs` has layout marker set. Selected `bushy` layout. + 2017-06-19 17:21:40,497 WARNI [ZODB.blob][MainThread] (23253) Blob dir var/db/blobs/ has insecure mode setting + 2017-06-19 17:21:40,498 INFO [ZEO.cache][MainThread] created temporary cache file 4 + 2017-06-19 17:21:40,517 BLATH [ZEO.zrpc][MainThread] (23253) CM.connect(): starting ConnectThread + 2017-06-19 17:21:40,518 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CT: attempting to connect on 1 sockets + 2017-06-19 17:21:40,524 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CW: attempt to connect to ('127.0.0.1', 8100) + 2017-06-19 17:21:40,525 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CW: connect_ex(('127.0.0.1', 8100)) returned EINPROGRESS + 2017-06-19 17:21:40,525 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CT: select() 0, 1, 0 + 2017-06-19 17:21:40,526 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CW: connect_ex(('127.0.0.1', 8100)) returned 0 + 2017-06-19 17:21:40,526 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Testing connection + 2017-06-19 17:21:40,527 INFO [ZEO.zrpc.Connection(b'C')][[('localhost', 8100)] zeo client networking thread] (127.0.0.1:8100) received handshake b'Z4' + 2017-06-19 17:21:40,628 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Server authentication protocol None + 2017-06-19 17:21:40,629 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Connected to storage: ('localhost', 8100) + 2017-06-19 17:21:40,630 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] No verification necessary -- empty cache + 2017-06-19 17:21:40,631 DEBUG [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] pickled inval None b'\x03\xc0\n*g\xcf\xcd\xbb' + 2017-06-19 17:21:40,633 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CM.connect_done(preferred=1) + ... + 2017-06-19 17:22:39,342 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2017-06-19 17:22:39,342 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2017-06-19 17:22:39,343 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2017-06-19 17:22:39,344 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2017-06-19 17:22:39,344 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2017-06-19 17:22:39,345 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing ... - 2016-12-28 11:02:01,354 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing - 2016-12-28 11:02:01,370 DEBUG [PyAMS (utils)][MainThread] Registering utility named 'PyAMS content indexer' providing - 2016-12-28 11:02:01,383 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "test-indexer-process.html" for () - 2016-12-28 11:02:01,387 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for () - 2016-12-28 11:02:01,400 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing - 2016-12-28 11:02:01,403 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "" for () - 2016-12-28 11:02:01,410 DEBUG [PyAMS (utils)][MainThread] Registering utility named 'PyAMS default skin' providing - 2016-12-28 11:02:01,411 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing - Starting server in PID 30235. - serving on http://0.0.0.0:6543 + 2017-06-19 17:22:40,221 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (, , ) providing + 2017-06-19 17:22:40,222 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing + 2017-06-19 17:22:40,233 DEBUG [PyAMS (utils)][MainThread] Registering adapter for (,) providing + 2017-06-19 17:22:40,244 DEBUG [config][MainThread] include onf_website/configure.zcml + 2017-06-19 17:22:40,245 DEBUG [config][MainThread] include eggs/pyramid_zcml-1.1.0-py3.5.egg/pyramid_zcml/configure.zcml + 2017-06-19 17:22:40,246 DEBUG [config][MainThread] include eggs/pyramid_zcml-1.1.0-py3.5.egg/pyramid_zcml/meta.zcml + 2017-06-19 17:22:40,252 DEBUG [config][MainThread] include eggs/zope.i18n-4.1.0-py3.5.egg/zope/i18n/meta.zcml + 2017-06-19 17:22:40,253 DEBUG [zope.i18n][MainThread] register directory onf_website/src/onf_website/locales + 2017-06-19 17:22:40,258 DEBUG [config][MainThread] include webapp/webapp/configure.zcml + ... + 2017-06-19 17:22:41,288 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CM.connect_done(preferred=1) + 2017-06-19 17:22:41,288 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CT: exiting thread: Connect([(, ('localhost', 8100))]) + 2017-06-19 17:22:41,303 DEBUG [PyAMS (scheduler][MainThread] Adding scheduler job for task 'Compactage ZEO' + 2017-06-19 17:22:41,304 INFO [apscheduler.scheduler][MainThread] Adding job tentatively -- it will be properly scheduled when the scheduler starts + 2017-06-19 17:22:41,305 DEBUG [PyAMS (scheduler][MainThread] Starting tasks scheduler ... + 2017-06-19 17:22:41,313 INFO [apscheduler.scheduler][MainThread] Added job "Compactage ZEO" to job store "default" + 2017-06-19 17:22:41,314 INFO [apscheduler.scheduler][MainThread] Scheduler started + 2017-06-19 17:22:41,314 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run + 2017-06-19 17:22:41,315 DEBUG [apscheduler.scheduler][APScheduler] Next wakeup is due at 2017-06-20 03:00:00+02:00 (in 34638.684629 seconds) + 2017-06-19 17:22:41,326 INFO [ZEO.ClientStorage][MainThread] [('localhost', 8100)] ClientStorage (pid=23253) created RW/normal for storage: 'pyams' + 2017-06-19 17:22:41,327 DEBUG [ZODB.blob][MainThread] (23253) Blob directory `var/db/blobs` has layout marker set. Selected `bushy` layout. + 2017-06-19 17:22:41,328 WARNI [ZODB.blob][MainThread] (23253) Blob dir var/db/blobs/ has insecure mode setting + 2017-06-19 17:22:41,329 INFO [ZEO.cache][MainThread] created temporary cache file 15 + 2017-06-19 17:22:41,382 BLATH [ZEO.zrpc][MainThread] (23253) CM.connect(): starting ConnectThread + 2017-06-19 17:22:41,383 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CT: attempting to connect on 1 sockets + 2017-06-19 17:22:41,383 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CW: attempt to connect to ('127.0.0.1', 8100) + 2017-06-19 17:22:41,384 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CW: connect_ex(('127.0.0.1', 8100)) returned EINPROGRESS + 2017-06-19 17:22:41,385 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CT: select() 0, 1, 0 + 2017-06-19 17:22:41,385 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CW: connect_ex(('127.0.0.1', 8100)) returned 0 + 2017-06-19 17:22:41,386 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Testing connection + 2017-06-19 17:22:41,386 INFO [ZEO.zrpc.Connection(b'C')][[('localhost', 8100)] zeo client networking thread] (127.0.0.1:8100) received handshake b'Z4' + 2017-06-19 17:22:41,487 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Server authentication protocol None + 2017-06-19 17:22:41,488 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] Connected to storage: ('localhost', 8100) + 2017-06-19 17:22:41,489 INFO [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] [('localhost', 8100)] No verification necessary -- empty cache + 2017-06-19 17:22:41,490 DEBUG [ZEO.ClientStorage][Connect([(, ('localhost', 8100))])] pickled inval None b'\x03\xc0\n*g\xcf\xcd\xbb' + 2017-06-19 17:22:41,492 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CM.connect_done(preferred=1) + 2017-06-19 17:22:41,493 BLATH [ZEO.zrpc][Connect([(, ('localhost', 8100))])] (23253) CT: exiting thread: Connect([(, ('localhost', 8100))]) + 2017-06-19 17:22:41,498 DEBUG [PyAMS (media)][MainThread] Starting medias conversion process ... + 2017-06-19 17:22:41,503 DEBUG [PyAMS (content.es)][MainThread] Starting content indexer process ... + Starting server in PID 23253. + Serving on http://0.0.0.0:6543 From this point, you can launch a browser and open URL *http://127.0.0.1:6543/admin* to get access to PyAMS management interface; default login is "admin/admin", that you may change as soon as possible...