Removing duplicate documents from Elasticsearch saves disk space and will speed-up searches. This saves you time and makes you more productive. I setup and manage ELK (Elasticsearch, Logstash and Kibana) clusters that process hundreds of millions log lines per day. Once in awhile it is necessary to reprocess a...

Index templates allow you to specify index settings and mappings that will automatically be applied when an index is first created. It uses a simple pattern match to control when a template is applied. An index can have multiple patterns applied by using the order key. The best way...

The _source field contains the original JSON document body of an Elaticsearch document. It is not indexed nor searchable, but it can be returned when executing fetch requests. Keeping the _source field saved with the document comes with substantial storage overhead. Disabling the _source field is easy and does...

As of Nginx 1.11.8 you can use the parameter escape=json in your log_format definition to handle JSON encoding and escaping. Here’s a sample log format you can use. Configure the NGiNX server to use json-log definition. Example log line. Passed through jq. This takes the load off of logstash and puts it on the web server....

Let’s talk about Elasticsearch and some of its advanced features. Elasticsearch is a search and analytics engine that’s built on top of Apache Lucene. It’s a great tool for processing and analyzing large amounts of data in real-time. Sharding and Replication One of the features that makes Elasticsearch so...

Are you new to Elasticsearch and wondering what it is and how it works? Elasticsearch is a powerful search engine that can handle large volumes of data, making it an essential tool for businesses and organizations of all sizes. In this article, we will provide an overview of Elasticsearch,...

This article demonstrates how to migrate a PostgresSQL table to Elasticsearch using a compound index to ensure uniqueness. Examples are in Python, Golang, NodeJS, Java and C#. Note that you will need to replace the PostgreSQL connection parameters and Elasticsearch parameters with your own. Also, make sure to install...

Are you tired of sifting through endless amounts of data to find what you’re looking for? Do you wish there was a better way to organize and search your data? This guide will help you with installing Elasticsearch! This powerful search and analytics engine makes it easy to store,...