ELK tutorial: part 1

By | 17/07/2015

[keen]

Introduction

I have recently been playing around with Elastisearch, Logstash and Kibana, often referred to as the ELK stack.

Installing ELK

Installing ELK is rather simple. To do so, go to the Elastic website (here) and download each package. As I’m running on Ubuntu (fresh install on Openstack), I’m downloading the equivalent packages for each service:

ubuntu@elk:~$ sudo apt-get update
ubuntu@elk:~$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.6.0.tar.gz
ubuntu@elk:~$ wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
ubuntu@elk:~$ wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz

Next, move ahead on untar these packages:

ubuntu@elk:~$ tar xvzf elasticsearch-1.6.0.tar.gz
ubuntu@elk:~$ tar xvzf logstash-1.5.2.tar.gz
ubuntu@elk:~$ tar xvzf kibana-4.1.1-linux-x64.tar.gz 

When all is well, you’ll see the packages in your directory.

ubuntu@elk:~$ ls -l
total 16
drwxrwxr-x 7 ubuntu ubuntu 4096 Jul 16 15:59 elasticsearch-1.6.0
drwxrwxr-x 7 ubuntu ubuntu 4096 Jun 29 18:07 kibana-4.1.1-linux-x64
drwxrwxr-x 5 ubuntu ubuntu 4096 Jul 17 10:21 logstash-1.5.2

Don’t forget to also install Java in case you haven’t done so:

ubuntu@elk:~$ sudo add-apt-repository -y ppa:webupd8team/java
ubuntu@elk:~$ sudo apt-get update
ubuntu@elk:~$ sudo apt-get -y install oracle-java8-installer

In order to make things a bit easier later on, let’s also create a startup script:

ubuntu@elk:~$ cat startup.sh 
echo "----Starting ElasticSearch----"
cd ~/elasticsearch-1.6.0/
./bin/elasticsearch &

echo "----Starting Kibana----"
cd ~/kibana-4.1.1-linux-x64/
./bin/kibana &

echo "----Starting Logstash----"
cd ~/logstash-1.5.2/
./bin/logstash -f logstash-access.conf 

echo "----Finished booting the ELK stack"

The script obviously is not strictly necessary and can probably be improved dramatically, but for now it does the trick quite well.

If all goes well, you will see messages appearing like below:

ubuntu@elk:~/elasticsearch-1.6.0$ sudo ./bin/elasticsearch
[2015-07-17 11:46:42,569][INFO ][node                     ] [Mindworm] version[1.6.0], pid[3797], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-07-17 11:46:42,577][INFO ][node                     ] [Mindworm] initializing ...
[2015-07-17 11:46:42,581][INFO ][plugins                  ] [Mindworm] loaded [], sites []
[2015-07-17 11:46:42,664][INFO ][env                      ] [Mindworm] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [45.5gb], net total_space [49.1gb], types [ext4]
[2015-07-17 11:46:46,591][INFO ][node                     ] [Mindworm] initialized
[2015-07-17 11:46:46,592][INFO ][node                     ] [Mindworm] starting ...
[2015-07-17 11:46:46,876][INFO ][transport                ] [Mindworm] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/173.39.243.147:9300]}
[2015-07-17 11:46:46,929][INFO ][discovery                ] [Mindworm] elasticsearch/pQUGKeJCT4CbNxjWZK33Jw
[2015-07-17 11:46:50,745][INFO ][cluster.service          ] [Mindworm] new_master [Mindworm][pQUGKeJCT4CbNxjWZK33Jw][elk][inet[/173.39.243.147:9300]], reason: zen-disco-join (elected_as_master)
[2015-07-17 11:46:50,811][INFO ][http                     ] [Mindworm] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/173.39.243.147:9200]}
....

Let’s now also test if ElasticSearch is really running and available. Go to http://:9200. You will see something similar like below picture:

ElasticSearch

In order to reach Kibana, you’ll go to http://:5601. You will find a webpage with the Kibana logo which means you’re good to go! Note that you will have to configure an index first, but we’ll cover that in part 2 of this series.

kibana-default

You have now a full ELK stack up and running. In part 2 of this tutorial, we are going to configure ELK to monitor apache access log files and visualize then using Kibana.