NLP for informed business decisions

To enable fast searches through the text that would reveal actual valuable information, out-of-the-box and custom approaches were compared:

• Out-of-the-box cloud solution that leverages the AWS Comprehend service.
• Custom solution based on Topic Modeling using LDA and Named Entity Recognition (NER) to tag documents with labels in order to enable a search through raw unstructured texts.

We offered to use Topic Modeling which helps to determine the list of topics discussed in an entire collection of meeting notes and that reveals the presence of one or more topics in each document. Also, our team used Named Entity Recognition for detecting a word or phrase in a text which has either a preset generic entity type (Location, Person, Geographical and Geopolitical entity) or a custom type (Product, Organization, financial entities).

Both approaches allowed for the retrieval of information from unstructured texts, overlaying the context on the content by tagging it with machine-readable metadata. However, the custom approach enabled the labeling of new and unseen documents on the fly, with one or more topics found as a result of a topic modeling algorithm.

This best suits cases where a finer control of the training, optimization, and hosting of a topic model is required (e.g. when we deal with specific texts’ domain, like in our case). Moreover, a custom model is tuned during development, so that it requires no additional effort. Using custom NER solutions, based on a deep learning model, we can detect both default and custom entities in real-time.

Technology used