Elasticsearch Analyzers for Beginners

Elasticsearch is a powerful search engine that allows you to store, search, and analyze large amounts of data quickly and efficiently. One of the key features of Elasticsearch is its ability to use analyzers to process text before it’s indexed. In this article, we’ll explain what analyzers are and how they work in Elasticsearch.

What are Analyzers?

Analyzers are used to preprocess text before it’s indexed in Elasticsearch. They break down the text into individual tokens (words) based on certain rules or criteria like whitespace, punctuation marks or language-specific stemming algorithms. These tokens are then stored in the inverted index which helps with faster searching later.

Here’s an example: if you have a document containing “The quick brown fox jumped over the lazy dog,” an analyzer might break this sentence down into separate words (tokens), such as “the,” “quick,” “brown,” etc., making each word searchable independently.

How do they Work?

In order for analyzers to work effectively, they need to be configured correctly. There are three main components that make up an analyzer – character filters, tokenizers, and token filters.

Character Filters – Character filters can be used to manipulate the text before it gets analyzed by replacing characters or adding/removing them entirely.

Tokenizer – Tokenizers split the input stream into discrete chunks called tokens based on some defined set of rules such as whitespace or specific characters like commas or hyphens.

Token Filters – Token filters can modify tokens after they’ve been created so that special attention may be given towards removing stop words (commonly occurring words that don’t add much meaning) from being included in indexing.

Types of Analyzers

There are several types of analyzers available in Elasticsearch that cater to different scenarios:

Analyzer Name	Description
Standard Analyzer	The default analyzer used by ElasticSearch when none other is specified; tokenizes text using Unicode Text Segmentation algorithm & applies lowercasing & stopword removal.
Simple Analyzer	Divides each field value into terms whenever it encounters a character which isnt a letter.
Stop Analyzer	A wrapper around a tokenizer which adds support for removing stop words from a stream.
Keyword Analyzer	It doesn’t tokenize field values at all but simply creates one term per input string
Whitespace Analyzer	Splits fields by whitespace . This type is useful when indexing exact phrases.

These pre-built analyzers should suffice most requirements however custom ones can also be built using any combination of character filter, tokenizer and token filter out-of-the-box building blocks provided within elasticsearch APIs.

Conclusion

Analyzing textual data prior-to indexing plays a crucial role while yielding relevant results post-searching with better accuracy level than otherwise achievable without preprocessing them first hand via analysers offered through ElasticSearch API(s). And now you know what Elastisearch offers out-of-the-box 🙂