Mercurial Mercurial > pyams > pyams_user_guide / comparison
summary | shortlog | changelog | graph | tags | bookmarks | branches | files | changeset | file | latest | revisions | annotate | diff | comparison | raw | help
Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
src/source/plugins.rst
branchdoc-dc
changeset 62 dc2bd8068d7a
parent 57 e7d62e94392f
child 104 942151432421
equal deleted inserted replaced
61:cfb7276fdd33 62:dc2bd8068d7a 
     1 .. _plugins:
 
     1 .. _plugins:
 
     2 
     2 
     3 PyAMS additional features and services
 
     3 PyAMS additional features and services
 
     4 ++++++++++++++++++++++++++++++++++++++
 
     4 ======================================
 
     5 
     5 
     6 
     6 
     7 Elasticsearch 5.4
 
     7 Elasticsearch
 
     8 =================
 
     8 +++++++++++++
 
     9 
     9 
    10 At first you need to install ElasticSearch (ES), currently PyAMS is compatible with the version 5.4, the Ingest attachment
 
    10 At first you need to install ElasticSearch (ES); PyAMS is actually compatible with version 5.4. The Ingest attachment
 
    11 plug-in is also required to handle attachments correctly.
 
    11 plug-in is also required to handle attachments correctly.
 
    12 
    12 
    13 Visit  https://www.elastic.co/ to learn how to install Elasticsearch Server, and how install `ingest-attachment` plug-in
 
    13 Visit https://www.elastic.co/ to learn how to install Elasticsearch Server and `ingest-attachment` plug-in
 
    14 
    14 
    15 
    15 
    16 .. tip:: Documentation for installing ElasticSearch 5.4
 
    16 .. tip:: Documentation for installing ElasticSearch 5.4
 
    17 
    17 
    18     - https://www.elastic.co/guide/en/elasticsearch/reference/5.4/gs-installation.html
 
    18     - https://www.elastic.co/guide/en/elasticsearch/reference/5.4/gs-installation.html
 
    19     - https://www.elastic.co/guide/en/elasticsearch/plugins/5.4/ingest-attachment.html
 
    19     - https://www.elastic.co/guide/en/elasticsearch/plugins/5.4/ingest-attachment.html
 
    20 
    20 
    21 
    21 
    22 After ElasticSearch installation, following steps describe how to configure ES with PyAMS;
 
    22 After Elasticsearch installation, following steps describe how to configure ES with PyAMS.
 
       
    23 
    23 
    24 
    24 Initializing Elasticsearch index
 
    25 Initializing Elasticsearch index
 
    25 --------------------------------
 
    26 --------------------------------
 
    26 
    27 
    27 If you want to use an Elasticsearch index, you have to initialize index settings and mappings;
 
    28 If you want to use an Elasticsearch index, you have to initialize index settings and mappings;
 
    28 Elasticsearch integration is defined through the *PyAMS_content_es* package.
 
    29 Elasticsearch integration is defined through the *PyAMS_content_es* package.
 
    29 
    30 
    30 
    31 
    31 1. Enable Service:
 
    32 1. Enable service
 
    32 ''''''''''''''''''
 
    33 '''''''''''''''''
 
    33 
    34 
    34 In Pyramid INI application file *(etc/development.ini)*:
 
    35 In Pyramid INI application files (*etc/development.ini* and *etc/production.ini*):
 
    35 
    36 
    36 .. code-block:: bash
 
    37 .. code-block:: ini
 
    37 
    38 
    38     # ElasticSearch settings
 
    39     # Elasticsearch server settings
 
    39     elastic.server = http://127.0.0.1:9200
 
    40     elastic.server = http://127.0.0.1:9200
 
    40     elastic.index = pyams
 
    41     elastic.index = pyams
 
    41 
    42 
    42 .. code-block:: bash
 
    43 Where:
 
    43 
    44  - **elastic.server**: address of Elasticsearch server; you can include authentication arguments in the form
 
    44     # PyAMS content elasticsearch index settings
 
    45    *http://login:password@w.x.y.z:9200*
 
       
    46  - **elastic.index**: name of Elasticsearch index.
 
       
    47 
       
    48 
       
    49 On startup, main PyAMS application process can start in *indexer* process which will handle indexing requests in
 
       
    50 asynchronous mode; this process settings are defined like this:
 
       
    51 
       
    52 .. code-block:: ini
 
       
    53 
       
    54     # PyAMS content Elasticsearch indexer process settings
 
    45     pyams_content.es.tcp_handler = 127.0.0.1:5557
 
    55     pyams_content.es.tcp_handler = 127.0.0.1:5557
 
    46     pyams_content.es.start_handler = false
 
    56     pyams_content.es.start_handler = false
 
    47     pyams_content.es.allow_auth = admin:admin
 
    57     pyams_content.es.allow_auth = admin:admin
 
    48     pyams_content.es.allow_clients = 127.0.0.1
 
    58     pyams_content.es.allow_clients = 127.0.0.1
 
    49 
    59 
    50 
    60 Where:
 
    51 2. Initialize Elasticsearch Database:
 
    61  - **pyams_content.es.tcp_handler**: IP address and listening port of PyAMS indexer process
 
    52 '''''''''''''''''''''''''''''''''''''
 
    62  - **pyams_content.es.start_handler**: if *true*, the indexer process is started on PyAMS startup; otherwise (typically
 
    53 
    63    in a cluster configuration), the process is supposed to be started from another *master* server
 
    54 Configuration files for attachment pipeline, index settings and mappings are available `pyams_content_es` package or in PyAMS installation folder:
 
    64  - **pyams_content.es.allow_auth**: login and password to be used to connect to indexer process (settings are defined
 
       
    65    in the same way on indexer process and on all it's clients)
 
       
    66  - **pyams_content.es.allow_clients**: list of IP addresses allowed to connect to indexer process.
 
       
    67 
       
    68 
       
    69 2. Initialize Elasticsearch database
 
       
    70 ''''''''''''''''''''''''''''''''''''
 
       
    71 
       
    72 Configuration files for attachment pipeline, index and mappings settings are available into `pyams_content_es` source
 
       
    73 package or in PyAMS installation folder:
 
    55 
    74 
    56 
    75 
    57 .. code-block:: bash
 
    76 .. code-block:: bash
 
    58 
    77 
    59     (env) $ cd docs/elasticsearch
 
    78     (env) $ cd docs/elasticsearch
 
    60     (env) $ curl --noproxy localhost -XPUT http://localhost:9200/_ingest/pipeline/attachment -d @attachment-pipeline.json
 
    79     (env) $ curl --noproxy localhost -XPUT http://localhost:9200/_ingest/pipeline/attachment -d @attachment-pipeline.json
 
    61 
    80 
    62 
    81 
    63 With ``elastic.index = pyams`` defined as Elasticsearch index name : *"http://localhost:9200/pyams"* :
 
    82 And with ``elastic.index = pyams`` defined as Elasticsearch index name: *"http://localhost:9200/pyams"*:
 
    64 
    83 
    65 .. code-block:: shell
 
    84 .. code-block:: shell
 
    66 
    85 
    67     (env) $ curl -XDELETE http://localhost:9200/pyams
 
    86     (env) $ curl -XDELETE http://localhost:9200/pyams
 
    68 
    87 
    71     (env) $ curl -XPUT http://localhost:9200/pyams/WfTopic/_mapping  -d @mappings/WfTopic.json
 
    90     (env) $ curl -XPUT http://localhost:9200/pyams/WfTopic/_mapping  -d @mappings/WfTopic.json
 
    72     (env) $ curl -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json
 
    91     (env) $ curl -XPUT http://localhost:9200/pyams/WfNewsEvent/_mapping -d @mappings/WfNewsEvent.json
 
    73     (env) $ curl -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json
 
    92     (env) $ curl -XPUT http://localhost:9200/pyams/WfBlogPost/_mapping -d @mappings/WfBlogPost.json
 
    74 
    93 
    75 
    94 
    76 *Troubleshooting*: If you have a 406 error try to add ``-H 'Content-Type: application/json'`` in curl option
 
    95 *Troubleshooting*: If you have a 406 error try to add ``-H 'Content-Type: application/json'`` in Curl command lines.
 
    77 
    96 
    78 
    97 
    79 3. Create or update index:
 
    98 3. Update index contents
 
    80 ''''''''''''''''''''''''''
 
    99 ''''''''''''''''''''''''
 
    81 
   100 
    82 You have to Update ElasticSearch indexes with all database contents with ``pymas_es_index``. From a shell:
 
   101 If your ZODB database already store contents, you can update ElasticSearch indexes with all these contents with
 
       
   102 ``pymas_es_index`` command line script. From a shell:
 
    83 
   103 
    84 .. code-block:: bash
 
   104 .. code-block:: bash
 
    85 
   105 
    86     (env) $ ./bin/pyams_es_index ../etc/development.ini
 
   106     (env) $ ./bin/pyams_es_index ../etc/development.ini
 
    87 
   107 
    88 
   108 
    89 
   109 
    90 -------------------------------
 
       
    91 
       
    92 Natural Language Toolkit - NLTK
 
   110 Natural Language Toolkit - NLTK
 
    93 ===============================
 
   111 +++++++++++++++++++++++++++++++
 
    94 
   112 
    95 
   113 PyAMS is using NLTK features through the *PyAMS_calalog*.
 
    96 PyAMS enjoy the NLTK features through the *PyAMS_calalog*
 
       
    97 
   114 
    98 .. seealso::
 
   115 .. seealso::
 
    99 
   116 
   100     Visit https://www.nltk.org/ to learn more about NLTK
 
   117     Visit https://www.nltk.org/ to learn more about NLTK
 
   101 
   118 
   102 
   119 
   103 
   120 Initializing NLTK (Natural Language ToolKit)
 
   104 Initializing NLTK (Natural Language Toolkit)
 
       
   105 --------------------------------------------
 
   121 --------------------------------------------
 
   106 
   122 
   107 Some NLTK Collections like **tokenizers** and **stopwords** utilities are used to index fulltext contents
 
   123 Some NLTK collections like **tokenizers** and **stopwords** utilities are used to index fulltext contents
 
   108 elements. You can enhanced NTKL indexation according to your own needs. This package requires downloading and
 
   124 elements. You can enhance NLTK indexation according to your own needs. This package requires downloading and
 
   109 configuration of several elements which are done as follow:
 
   125 configuration of several elements which are done as follow:
 
   110 
   126 
   111 
   127 
   112 *1. Run the Python shell with PyAMS environment:*
 
   128 *1. Run the Python shell into PyAMS environment:*
 
   113 
   129 
   114 .. code-block:: bash
 
   130 .. code-block:: bash
 
   115 
   131 
   116     (env) $ ./bin/py
 
   132     (env) $ ./bin/py
 
   117 
   133 
   130 
   146 
   131     On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running
 
   147     On Debian GNU/Linux, you can choose any directory between '*~/nltk_data*' (where '~' is the homedir of user running
 
   132     Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and
 
   148     Pyramid application), '*/usr/share/nltk_data*', '*/usr/local/share/nltk_data*', '*/usr/lib/nltk_data*' and
 
   133     '*/usr/local/lib/nltk_data*'
 
   149     '*/usr/local/lib/nltk_data*'
 
   134 
   150 
   135     Please check if you have permission to write to this directory
 
   151     Please check if you have permission to write to this directory!
 
   136 
   152 
   137 
   153 
   138 .. code-block:: shell
 
   154 .. code-block:: shell
 
   139 
   155 
   140     NLTK Downloader
 
   156     NLTK Downloader
 
   188             Downloading package stopwords to /usr/local/lib/nltk_data...
 
   204             Downloading package stopwords to /usr/local/lib/nltk_data...
 
   189 
   205 
   190 
   206 
   191 .. tip::
 
   207 .. tip::
 
   192 
   208 
   193     The full list of NTLK Collection are displayable with the ``l) list`` option
 
   209     The full list of NTLK Collection can be displayed with the ``l) list`` option.
pyams/pyams_user_guide