NLP Search Solution

Introduction

The site search should provided the relevance product results based on the customer search terms.SAP Commerce OOTB search solution have various options for boosting products and Adaptive search,CDS but even which is not really helpful when we come to the production system with various complexity of the product data structures. Its requires frequent configurations and create search profiles.

We have created a small POC  that demonstrates our NLP search solution to provide relevant product results based on the customer search terms.We’ve never seen this concept implemented yet.

Solution

The idea of our Solution to use NLP for search the products. Use NLP Search to look beyond individual words or phrase, and understand the the context they are being search in the site and provide relevant search results.

Design

Key Concept

  1. Needs to generate machine learning models based on the product data
  2. Using the trained model, which help us to tag the search teams with product attribute.
  3. create new Producttag solr property and index the product attribute values.

Challange

Generating the ML Models will be the key challenge and required multiple rounds of testing to ensure the appropriate search results.

Technical Details

  1. Custom NLP Models Creation Job
    1. To create NLP NER (Name entity recognition) custom models for specific Solr index property, it requires two input file *-knownentities.txt and *-sentences.txt.
    2. *-knownentities.txt file will hold attribute specific values, e.g. known-entities for color attribute will be all the colors available in type system and file name will be color-knownentities.txt
    3. *-sentences.txt will hold data from product and category item types.
    4. Custom model creation cron-job will be creating the input text files from the given attribute and then NER model from the created text file.
  1. Create custom Lemmatizer Dictionary to add synonyms and base words related to product or category data.
  2. NLPTermMappingService to load custom NLP NER models, create tokens from search string, create POS tagging for tokens and fetch probability from custom models.
  3. NLPSearchFilterPopulator – override out of the box commerceSearchFiltersPopulator to add filter queries for specific indexed properties having NLPTermQueryFilter enable. It will search for each tokens in the search string against custom model and if the probability is more than NlpTermQueryProbability value then add filter query for that indexed property.
  4. NLPDisMaxFreeTextQueryBuilder – extends out of the box DisMaxFreeTextQueryBuilder to add NLP specific custom changes to Solr query. Depending on the attribute settings in SolrIndexProperty, it will boost relevant properties based on POS tagging or NER token probability from custom models.
Demo Video

Benefits:

  • Natural language processing (NLP) is a branch of Artificial Intelligence (AI) that helps computers understand, interpret and manipulate human language.
  • Automatically understand customer intent & bring relevant search results
  • Derive meanings from unstructured text
  • Detect & distinguish products from their attributes

Features

  • The Long-Tail Keyword Search – Understand not only the words but the concept and linked together to provide relevant results.
  • Automatic Insights – Derive meanings from unstructured text by automatically identifying different product attributes
  • Self-learning search engine – Automatically improvise machine learning models from user interaction and search text
  • Learning to Rank for Keyword Searches –  Rank the products automatically based on customer signals over the whole catalog without any configuration rules
  • Spellcheck & Autocomplete – More advance features for spell correction and autosuggestion can be achieve through trained machine learning models
  • Negation Recognition – Search product by excluding any category or product feature
  • Unit of Measure Conversion – Automatically convert units of measure for accurate search results