I attended the Indy DevOps meetup last week to learn about Elastic Search, LogStash, and Kibana. Also known as the ELK stack. The speaker, Adis Cesar a Solution Architect at Elastic, also talked about something called Beats.
What is Elastic Search?
Elastic Search is a distributed computing platform for rapidly searching text. The inventor of Elastic Search built this to help his wife search for cooking recipes.
What is Kibana?
Kibana is a web application for visualizing data stored in Elastic Search
What is LogStash?
LogStash is an Extract, Transform, Load tool for elastic search. LogStash extracts data from various data sources. Transforms it into a format Elastic search understands and stored it into Elastic Search.
What are Beats?
Beats are lightweight services installed on servers for data collection.
Example
At the meeting the speaker went through an example that I will paraphrase below:
Say I have a machine running an apache web server that I want to collect all the errors from the apache log. In addition, I want to collect CPU usage, disk space and memory usage metrics.
To collect is data I install two Beats on my machine. The File Beat for collecting log data. The metric Beat for collecting metrics such as CPU usage, disk space, and memory usage.
The file beat will periodically check for changes in the apache log. When if finds changes it will send the log to LogStash. LogStash will run a filter on it to find all the errors, then stores the results into Elastic Search.
When the metrics beat runs it will collect the metrics and store it directly into Elastic Search. Since metric data is simple, the metric beat does not have to send the data to LogStash to transform it.
To visualize the data we can use Kibana to create dashboards. There is already a canned dashboard for the metrics data. Here is an example
Cool, how can I use it?
It feels like one of the ways you could use the ELK stack plus beats for event-based monitoring of your infrastructure. Other than that, I am not really sure but am interested in exploring this more.
What should I not use it for?
Elastic search does not follow the ACID principles that RDMS databases follow, therefore I would be hesitant to use this as a primary data source for an application.
