+ +
+

Installing PyAMS

+

PyAMS default installation is based on Buildout utility. It’s not mandatory to use a +virtual environment, but it allows you to have a better control over your Python resources.

+

Current PyAMS version is based and validated for Python 3.5; your Python environment must also include a C +compiler as well as development headers for Python, libjpeg, libpng, libfreetype, libxml2, libxslt and +eventually libldap, libffi, libgdal or libzmq.

+

PyAMS default components configuration also pre-suppose that the following external tools are available:

+
    +
  • a Memcached or Redis server, to store sessions and cache (can be changed through Beaker configuration)
    • +
+

Optional tools also include:

+
    +
  • an LDAP server for authentication
    • +
  • an ElasticSearch server for full text indexing (see PyAMS_content_es package)
    • +
  • a WebSockets server using AsyncIO. This is used to manage notifications (see PyAMS_notify and PyAMS_notify_ws +packages). An out of the box environment can be built using pyams_notify scaffold.
    • +
+

PyAMS also needs that you use a ZODB remote server, as several background processes needing a concurrent access to ZODB +are started by PyAMS main process. Three ZODB storages are already provided through PyAMS: ZEO, RelStorage or Newt.db. +See Creating ZODB to know how to initialize database with the help of PyAMS tools.

+
+

Creating initial buildout

+

PyAMS provides a new Pyramid scaffold, called pyams, generated via a cookiecutter template.

+

A simple option to install PyAMS is to create a buildout environment including Pyramid and all PyAMS packages:

+
# mkdir /var/local/
+# pip3 install virtualenv
+# virtualenv --python=python3.5 env
+# cd env
+# . bin/activate
+(env) # pip3.5 install cookiecutter
+(env) # cookiecutter hg+http://hg.ztfy.org/pyams/scaffolds/pyams
+
+
+

CookieCutter will ask you for a small set of input variables that you can change or not:

+
    +
  • pyams_release: version of PyAMS configuration file to use. “latest” (default value) will point to last release; +you can also choose to point to a given release (“0.1.4” for example)
    • +
  • project_name: current environment name in “human form”
    • +
  • project_slug: “technical” package name, based on project name
    • +
  • virtual_hostname: Apache virtual-host name
    • +
  • webapp_name: web application package name (“webapp” as default)
    • +
  • webapp_port: TCP/IP port to use when running application outside Apache (“6543” as default)
    • +
  • eggs_directory: relative or absolute path to directory containing downloaded eggs; this directory can be +shared with other projects (“eggs” as default)
    • +
  • logs_directory: absolute path to directory containing Apache’s log files
    • +
  • run_user: user name under which Apache process will run (“www-data” as default)
    • +
  • run_group: group name under which Apache process will run (“www-data” as default)
    • +
  • beaker_backend: name of Beaker backend to use to store sessions and cache data (“redis” as default)
    • +
  • beaker_server: IP address and port of Beaker backend server (“127.0.0.1:6379” as default)
    • +
  • db_type: ZODB database storage; available options include ZEO, RelStorage and NewtDB
    • +
  • db_host: IP address of database server (“127.0.0.1” as default); WARNING: database server installation +is not part of application installation; another “zeo_server” cookiecutter recipe is available for ZEO
    • +
  • db_port: listening port of database server (“8100” is given as default for ZEO)
    • +
  • db_name: database or ZEO storage name to use
    • +
  • db_username: database user name
    • +
  • db_password: database password
    • +
  • zeo_realm: ZEO authentication realm
    • +
  • blobs_dir: local directory to use to store cache of ZODB blobs; cache size is limited to 10GB as default
    • +
  • use_postgresql: specify if PostgreSQL access is required; if so, please check that PostgreSQL development files +are available to compile PsycoPG2 extension
    • +
  • use_oracle: specify if Oracle access is required; if so, please check that Oracle development files are +available to compile cx_Oracle extension, and that ORACLE_HOME environment variable is correctly defined (see below)
    • +
  • use_ldap: specify if LDAP access will be required for authentication
    • +
  • use_elasticsearch: specify if an ElasticSearch server will be used for indexation
    • +
  • elasticsearch_server: URL used to access Elasticsearch server (“http://127.0.0.1:9200” as default); this URL can +include login and password (“http://login:password@127.0.0.1:9200”), if required…
    • +
  • elasticsearch_index: name of Elasticsearch index to use (“pyams” as default)
    • +
  • create_elasticsearch_index: specify if Elasticsearch index should be created after installation is complete
    • +
  • define_elasticsearch_mappings : specify if Elasticsearch mappings should be defined after installation is complete
    • +
  • smtp_server: DNS name of SMTP server (“localhost” as default)
    • +
  • smtp_server_name: “human” name given to SMTP server (“pyams” as default)
    • +
  • pyams_scheduler: TCP/IP address and port to use to access PyAMS tasks scheduler process (“127.0.0.1:5555” as +default); see pyams_scheduler
    • +
  • start_scheduler: boolean value to indicate if scheduler process is started by this application instance
    • +
  • pyams_medias_converter: TCP/IP address and port to use to access PyAMS medias converter process (“127.0.0.1:5556” +as default); see pyams_medias
    • +
  • start_medias_converter: boolean value to indicate if medias converter process is started by this application +instance
    • +
  • pyams_es_indexer: TCP/IP address and port to use to access PyAMS Elasticsearch indexer process (“127.0.0.1:5557” +as default); see pyams_content_es
    • +
  • start_es_indexer boolean value to indicate if Elasticsearch indexer process is started by this application +instance
    • +
  • use_notifications: specify if PyAMS notifications services are to be used (see PyAMS notification services)
    • +
  • pyams_ws_notify: TCP/IP address and port of PyAMS websockets server managing notifications service +(“127.0.0.1:8081” as default)
    • +
  • lexicon_languages: NLTK lexicon languages to use (“en:english fr:french” as default)
    • +
  • extension_package: name of a PyAMS extension package to include in environment configuration
    • +
  • need_pyams_gis: specify if PyAMS GIS features are to be used by given extension package; if so, please check +that libgdal development files are available; on Debian (and maybe others), you have to specify environment +variables (see below).
    • +
+

You can then check, and eventually update, the proposed Buildout configuration file buildout.cfg, to add or remove +packages or update settings to your needs. Then finalize Bootstrap initialization:

+
(env) # python3.5 bootstrap.py
+(env) # ./bin/buildout
+
+
+

This last operation can be quite long, as many packages have to downloaded, compiled and installed in the virtual +environment. If you encounter any compile error, just install the required dependencies and restart the buildout.

+

Some dependencies can require the definition of custom environment variables before running buildout, like:

+
    +
  • for libgdal, which is required by PyAMS_gis package, use:
    • +
+
(env) # export C_INCLUDE_PATH=/usr/include/gdal
+(env) # export CPLUS_INCLUDE_PATH=/usr/include/gdal
+
+
+

WARNING: you have to check also that your libgdal release is matching “GDAL” release given in PyAMS +configuration file (actually 2.1.0).

+
    +
  • for cx_Oracle, which is required if you use Oracle database connections, use:
    • +
+
(env) # export ORACLE_HOME=/usr/lib/oracle/12.1/client64
+
+
+

These examples are given for Debian GNU/Linux. You may have to adapt configuration based on your own Linux +distribution and packages versions.

+
+
+

Environment settings

+

The project generated from pyams scaffold is based on default Pyramid’s zodb scaffold, but it adds:

+
    +
  • a custom application factory, in the webapp directory (see PyAMS site management)
    • +
  • a set of directories to store runtime data, in the var directory; each directory contains a README.txt file +which should be self-explanatory to indicate what this directory should contain, including a ZEO cache
    • +
  • a set of configuration files, in the etc directory; here are standard development.ini and production.ini +configuration files, a ZODB configuration files (zodb-zeo.conf) for a ZEO client storage and two Apache +configurations (for Apache 2.2 and 2.4) using mod_wsgi.
    • +
+

Once the project have been created from the scaffold, you are free to update all the configuration files.

+

If you need to add packages to the environment, you have to add them to the buildout.cfg file AND to the INI +file (in the pyramid.includes section) before running the buildout another time; don’t forget to add the +requested version at the end of buildout.cfg file, as Buildout is not configured by default to automatically +download the last release of a given unknown package.

+

development.ini and production.ini files contain many commented directives related to PyAMS components. Read and +update them carefully before initializing your application database!

+
+
+

Initializing the database

+

When you have downloaded and installed all required packages, you have to initialize the database so that all +required components are available.

+

From a shell, just type:

+
(env) # ./bin/pyams_upgrade etc/development.ini
+
+
+

This process requires that every package is correctly included into pyramid.includes directive from selected +configuration file.

+
+
+

Initializing Elasticsearch index

+

If you want to use an Elasticsearch index, you have to initialize index settings and mappings; the Ingest attachment +plug-in is also required to handle attachments correctly.

+

Elasticsearch integration is defined through the PyAMS_content_es package. Configuration files are available in this +package, for attachment pipeline, index settings and mappings:

+
(env) # cd /var/local/src/pyams/pyams_content_es
+(env) # curl --noproxy localhost -XDELETE http://localhost:9200/pyams (1)
+(env) # curl --noproxy localhost -XPUT    http://localhost:9200/pyams -d @index-settings.json
+
+(env) # curl --noproxy localhost -XPUT    http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json
+(env) # curl --noproxy localhost -XPUT    http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json
+(env) # curl --noproxy localhost -XPUT    http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json
+
+
+
    +
  1. If ‘pyams’ is defined as Elasticsearch index name.
    2. +
+
+
+

NLTK initialization

+

Some NLTK (Natural Language Toolkit) tokenizers and stopwords utilities are used to index fulltext contents elements. +This package requires downloading and configuration of several elements which are done as follow:

+
(end) # ./bin/py
+>>> import nltk
+>>> nltk.download()
+NLTK Downloader
+---------------------------------------------------------------------------
+    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
+---------------------------------------------------------------------------
+Downloader> c
+
+Data Server:
+  - URL: <https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml>
+  - 6 Package Collections Available
+  - 107 Individual Packages Available
+
+Local Machine:
+  - Data directory: /home/tflorac/nltk_data
+
+---------------------------------------------------------------------------
+    s) Show Config   u) Set Server URL   d) Set Data Dir   m) Main Menu
+---------------------------------------------------------------------------
+Config> d
+  New directory> /usr/local/lib/nltk_data (1)
+Config> m
+
+---------------------------------------------------------------------------
+    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
+---------------------------------------------------------------------------
+Downloader> d
+
+Download which package (l=list; x=cancel)?
+  Identifier> punkt
+    Downloading package punkt to /usr/local/lib/nltk_data...
+
+Downloader> d
+
+Download which package (l=list; x=cancel)?
+  Identifier> stopwords
+    Downloading package stopwords to /usr/local/lib/nltk_data...
+
+
+

(1) On Debian GNU/Linux, you can choose any directory between ‘~/nltk_data’ (where ‘~’ is the homedir of user running +Pyramid application), ‘/usr/share/nltk_data’, ‘/usr/local/share/nltk_data’, ‘/usr/lib/nltk_data’ and +‘/usr/local/lib/nltk_data’.

+
+
+

Starting the application

+

When database upgrade process has ended, you can start the web application process with the standard Pyramid’s +pserve command line tool:

+
(env) # ./bin/pserve etc/development.ini
+
+
+

In standard debug mode, all registered components are displayed in the console, until the final line (here using ZEO):

+
2018-01-14 11:37:54,339 INFO  [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=28695) created RW/normal for storage: 'pyams'
+2018-01-14 11:37:54,340 INFO  [ZEO.cache][MainThread] created temporary cache file 3
+2018-01-14 11:37:54,345 INFO  [ZODB.blob][MainThread] (28695) Blob directory `/var/local/env/pyams/var/db/blobs` is used but has no layout marker set. Selected `lawn` layout.
+2018-01-14 11:37:54,345 WARNI [ZODB.blob][MainThread] (28695) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout.
+2018-01-14 11:37:54,346 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
+2018-01-14 11:37:54,347 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7feeb1de7390> None
+2018-01-14 11:37:54,348 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
+2018-01-14 11:37:54,349 INFO  [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
+2018-01-14 11:37:54,355 INFO  [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
+2018-01-14 11:37:54,358 DEBUG [txn.140663320073984][MainThread] new transaction
+2018-01-14 11:37:54,360 DEBUG [txn.140663320073984][MainThread] commit
+2018-01-14 11:37:54,484 DEBUG [config][MainThread] include /home/tflorac/Dropbox/src/PyAMS/pyams_template/src/pyams_template/configure.zcml
+2018-01-14 11:37:54,485 DEBUG [config][MainThread] include /var/local/env/pycharm/lib/python3.5/site-packages/pyramid_zcml/configure.zcml
+...
+2018-01-14 11:37:54,833 DEBUG [PyAMS (utils)][MainThread] Registering utility <class 'pyams_utils.timezone.utility.TimezoneGenerationsChecker'> named 'PyAMS timezone' providing <InterfaceClass pyams_utils.interfaces.site.ISiteGenerations>
+2018-01-14 11:37:54,834 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.timezone.vocabulary.TimezonesVocabulary'> as vocabulary with name "PyAMS timezones"
+2018-01-14 11:37:54,835 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.traversing.PathElementsAdapter'> for (<InterfaceClass zope.location.interfaces.IContained>,) providing <InterfaceClass pyams_utils.interfaces.traversing.IPathElements>
+2018-01-14 11:37:54,839 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.url.AbsoluteUrlTalesExtension'> for (<InterfaceClass zope.interface.Interface>, <InterfaceClass zope.interface.Interface>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_utils.interfaces.tales.ITALESExtension>
+2018-01-14 11:37:54,847 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.widget.decimal.DottedDecimalDataConverter'> for (<InterfaceClass pyams_utils.schema.IDottedDecimalField>, <InterfaceClass z3c.form.interfaces.IWidget>) providing <InterfaceClass z3c.form.interfaces.IDataConverter>
+2018-01-14 11:37:54,942 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.zmi.intids.IntIdsLengthAdapter'> for (<InterfaceClass zope.intid.interfaces.IIntIds>,) providing <InterfaceClass pyams_utils.interfaces.intids.IIndexLength>
+2018-01-14 11:37:54,943 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass zope.intid.interfaces.IIntIds> (<class 'pyams_utils.zmi.intids.IntIdsPropertiesDisplayForm'>)
+2018-01-14 11:37:54,949 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass pyams_utils.interfaces.timezone.IServerTimezone> (<class 'pyams_utils.zmi.timezone.ServerTimezonePropertiesEditForm'>)
+2018-01-14 11:37:54,980 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.zodb.ZEOConnectionVocabulary'> as vocabulary with name "PyAMS ZEO connections"
+2018-01-14 11:37:54,981 DEBUG [PyAMS (utils)][MainThread] Registering class <class 'pyams_utils.zodb.ZODBConnectionVocabulary'> as vocabulary with name "PyAMS ZODB connections"
+2018-01-14 11:37:55,015 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "add-zeo-connection.html" for <InterfaceClass zope.component.interfaces.ISite> (<class 'pyams_utils.zmi.zeo.ZEOConnectionAddForm'>)
+2018-01-14 11:37:55,016 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_utils.zmi.zeo.ZEOConnectionNameAdapter'> for (<InterfaceClass pyams_utils.interfaces.zeo.IZEOConnection>, <InterfaceClass pyams_zmi.layer.IAdminLayer>) providing <InterfaceClass pyams_skin.interfaces.container.ITableElementName>
+2018-01-14 11:37:55,017 DEBUG [PyAMS (pagelet)][MainThread] Registering pagelet view "properties.html" for <InterfaceClass pyams_utils.interfaces.zeo.IZEOConnection> (<class 'pyams_utils.zmi.zeo.ZEOConnectionPropertiesEditForm'>)
+...
+2018-01-14 11:41:13,214 DEBUG [PyAMS (utils)][MainThread] Registering adapter <class 'pyams_default_theme.skin.ResourcesAdapter'> for (<InterfaceClass zope.interface.Interface>, <InterfaceClass pyams_default_theme.layer.IPyAMSDefaultLayer>, <InterfaceClass zope.interface.Interface>) providing <InterfaceClass pyams_skin.interfaces.resources.IResources>
+2018-01-14 11:43:36,665 INFO  [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams'
+2018-01-14 11:43:36,665 INFO  [ZEO.cache][MainThread] created temporary cache file 9
+2018-01-14 11:43:36,673 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
+2018-01-14 11:43:36,674 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642)
+2018-01-14 11:43:36,674 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7ffa54058860> None
+2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328   blob cache size: 0
+2018-01-14 11:43:36,675 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
+2018-01-14 11:43:36,675 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328   -->
+2018-01-14 11:43:36,677 INFO  [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
+2018-01-14 11:43:36,679 INFO  [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
+2018-01-14 11:43:36,682 DEBUG [txn.140713340237568][MainThread] new transaction
+2018-01-14 11:43:36,683 DEBUG [txn.140713340237568][MainThread] commit
+2018-01-14 11:43:36,690 INFO  [PyAMS (scheduler][MainThread] Starting tasks scheduler <SchedulerProcess(SchedulerProcess-1, initial)>...
+2018-01-14 11:43:36,698 INFO  [PyAMS (scheduler][MainThread] Started tasks scheduler with PID 29361.
+2018-01-14 11:43:36,701 INFO  [apscheduler.scheduler][MainThread] Scheduler started
+2018-01-14 11:43:36,702 DEBUG [apscheduler.scheduler][APScheduler] Looking for jobs to run
+2018-01-14 11:43:36,704 DEBUG [apscheduler.scheduler][APScheduler] No jobs; waiting until a job is added
+2018-01-14 11:43:36,719 INFO  [ZEO.ClientStorage][MainThread] [('127.0.0.1', 8100)] ClientStorage (pid=29335) created RW/normal for storage: 'pyams'
+2018-01-14 11:43:36,720 INFO  [ZEO.cache][MainThread] created temporary cache file 15
+2018-01-14 11:43:36,724 DEBUG [asyncio][[('127.0.0.1', 8100)] zeo client networking thread] Using selector: EpollSelector
+2018-01-14 11:43:36,725 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] disconnected <ZEO.asyncio.client.Client object at 0x7ffa557e8b00> None
+2018-01-14 11:43:36,726 DEBUG [ZEO.asyncio.client][[('127.0.0.1', 8100)] zeo client networking thread] try_connecting
+2018-01-14 11:43:36,727 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328 Checking blob cache size. (target: 966367642)
+2018-01-14 11:43:36,728 INFO  [ZEO.asyncio.base][[('127.0.0.1', 8100)] zeo client networking thread] Connected Protocol(('127.0.0.1', 8100), 'pyams', False)
+2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328   blob cache size: 0
+2018-01-14 11:43:36,729 DEBUG [ZEO.ClientStorage.check_blob_cache][[('127.0.0.1', 8100)] zeo client check blob size thread] 140712483907328   -->
+2018-01-14 11:43:36,732 INFO  [ZEO.ClientStorage][[('127.0.0.1', 8100)] zeo client networking thread] [('127.0.0.1', 8100)] Connected to storage: ('localhost', 8100)
+2018-01-14 11:43:36,735 DEBUG [txn.140713340237568][MainThread] new transaction
+2018-01-14 11:43:36,736 DEBUG [txn.140713340237568][MainThread] commit
+2018-01-14 11:43:36,743 INFO  [PyAMS (media)][MainThread] Starting medias converter <MediaConversionProcess(MediaConversionProcess-2, initial)>...
+2018-01-14 11:43:36,751 INFO  [PyAMS (media)][MainThread] Started medias converter with PID 29367.
+Starting server in PID 29335.
+Serving on http://0.0.0.0:6543
+
+
+

From this point, you can launch a browser and open URL http://127.0.0.1:6543/admin to get access to PyAMS +management interface; default login is “admin/admin”, that you may change as soon as possible (see +PyAMS security)!!.

+
+
+ + +