diff -r e3d33ef363fd -r 949d496c4e96 src/source/plugins.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/source/plugins.rst Wed May 09 17:33:41 2018 +0200 @@ -0,0 +1,170 @@ +.. _plugins: + +PyAMS additional features and services +++++++++++++++++++++++++++++++++++++++ + + +Elasticsearch 5.4 +================= + +At first you need to install ElasticSearch (ES), currently PyAMS is compatible with the version 5.4, the Ingest attachment +plug-in is also required to handle attachments correctly. + +Visit https://www.elastic.co/ to learn how to install Elasticsearch Server, and how install `ingest-attachment` plug-in + + +.. tips:: Documentation for installing ElasticSearch 5.4 + + - https://www.elastic.co/guide/en/elasticsearch/reference/5.4/gs-installation.html + - https://www.elastic.co/guide/en/elasticsearch/plugins/5.4/ingest-attachment.html + + +After ElasticSearch installation, following steps describe how to configure ES with PyAMS; + +Initializing Elasticsearch index +-------------------------------- + +If you want to use an Elasticsearch index, you have to initialize index settings and mappings; +Elasticsearch integration is defined through the *PyAMS_content_es* package. + + +1. Enable Service: +'''''''''''''''''' + +In Pyramid INI application file *(etc/development.ini)*: + +.. code-block:: bash + + # ElasticSearch settings + elastic.server = http://127.0.0.1:9200 + elastic.index = pyams + +.. code-block:: bash + + # PyAMS content elasticsearch index settings + pyams_content.es.tcp_handler = 127.0.0.1:5557 + pyams_content.es.start_handler = false + pyams_content.es.allow_auth = admin:admin + pyams_content.es.allow_clients = 127.0.0.1 + + +2. Initialize Elasticsearch Database: +''''''''''''''''''''''''''''''''''''' + +Configuration files for attachment pipeline, index settings and mappings are available `pyams_content_es` package or in PyAMS installation folder: + + +.. code-block:: bash + + (env) $ cd docs/elasticsearch + (env) $ curl --noproxy localhost -XPUT http://localhost:9200/_ingest/pipeline/attachment -d @attachment-pipeline.json + + +With ``elastic.index = pyams`` defined as Elasticsearch index name : *"http://localhost:9200/pyams"* : + +.. code-block:: shell + + (env) $ curl -XDELETE http://localhost:9200/pyams + + (env) $ curl -XPUT http://localhost:9200/pyams -d @index-settings.json + + (env) $ curl -XPUT http://localhost:9200/pyams/WfTopic/_mapping -d @mappings/WfTopic.json + (env) $ curl -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json + (env) $ curl -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json + + +*Troubleshooting*: If you have a 406 error try to add ``-H 'Content-Type: application/json'`` in curl option + + +3. Create or update index: +'''''''''''''''''''''''''' + +You have to index PyAMS objects into ES database. From a shell: + +.. code-block:: bash + + (env) $ ./bin/pyams_es_index ../etc/development.ini + + + +------------------------------- + +Natural Language Toolkit - NLTK +=============================== + + +With the package *PyAMS_nltk* PyAMS can use the NLTK features + +.. seealso:: + + Visit https://www.nltk.org/ to learn more about NLTK + + + + +Initializing NLTK +----------------- + +Some NLTK (Natural Language Toolkit) tokenizers and stopwords utilities are used to index fulltext contents elements. +This package requires downloading and configuration of several elements which are done as follow: + + +*1. Run the Python shell with PyAMS environment:* + +.. code-block:: bash + + (env) $ ./bin/py + + +*2. In the Python shell:* + +.. code-block:: python + + >>> import nltk + >>> nltk.download() + +.. code-block:: python + + NLTK Downloader + --------------------------------------------------------------------------- + d) Download l) List u) Update c) Config h) Help q) Quit + --------------------------------------------------------------------------- + Downloader> c + + Data Server: + - URL: + - 6 Package Collections Available + - 107 Individual Packages Available + + Local Machine: + - Data directory: /home/tflorac/nltk_data + --------------------------------------------------------------------------- + s) Show Config u) Set Server URL d) Set Data Dir m) Main Menu + --------------------------------------------------------------------------- + Config> d + New directory> /usr/local/lib/nltk_data + +.. tip:: + + On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running + Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and + '*/usr/local/lib/nltk_data*' + + +.. code-block:: pycon + + Config> m + --------------------------------------------------------------------------- + d) Download l) List u) Update c) Config h) Help q) Quit + --------------------------------------------------------------------------- + Downloader> d + + Download which package (l=list; x=cancel)? + Identifier> punkt + Downloading package punkt to /usr/local/lib/nltk_data... + + Downloader> d + + Download which package (l=list; x=cancel)? + Identifier> stopwords + Downloading package stopwords to /usr/local/lib/nltk_data...