ELK tutorial: part 2

By | 17/07/2015

Introduction

In part 1, we installed the ELK stack. This part will focus on the configuration of it. We’ll keep it rather simple for now. We will process a local Apache access log file and visualize it using Kibana. Let’s get started!

Configuring Logstash

Logstash needs a configuration file.

input { stdin { } }
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

In the input folder, you will see that we define a stdin() function, which will allow Logstash to parse content supplied via the command line. As we want to parse Apache access log entries, we will also use a filter that is able to parse the Apache log format. Luckily this comes out of the box with a Grok filter ‘COMBINEDAPACHELOG’. You can read here what it exactly does:

Next, we need to run Logstash using this configuration file. You do this as follows:

ubuntu@elk:~/logstash-1.5.2$ sudo bin/logstash -f logstash-stdin.conf 
...
Logstash startup completed

If all goes well, you should see the ‘Logstash startup completed’ message appear. If there is an error in your configuration file, Logstash will not hesitate to inform you.

If you now supply the following text to Logstash:

173.38.209.6 - - [17/Jul/2015:06:33:11 +0000] "GET / HTTP/1.1" 200 448 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36"

you will see that Logstash filters it and creates the following:

{
        "message" => "173.38.209.6 - - [17/Jul/2015:06:33:11 +0000] \"GET / HTTP/1.1\" 200 448 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T06:33:11.000Z",
           "host" => "elk",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:06:33:11 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "448",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""

This indicates that everything is working just fine. The only problem is that this is not very useful right? Let’s continue to parse a real Apache access log file. As you could have figured out probably, we can achieve this by changing the configuration file for Logstash:

input {
  file {
    path => "/var/log/apache2/access.log"
    type => "apache-access"
    start_position => "beginning"
  }
}
filter {
  if [type] == "apache-access" {
    grok {
      match => [ "message", "%{COMBINEDAPACHELOG}" ]
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  elasticsearch {
    host => localhost
  }
  stdout { codec => rubydebug }
}
ubuntu@elk:~/logstash-1.5.2$

Here you can see that we monitor the Apache access log file in the folder ‘/var/log/apache2/access.log’. We expect that each time we retrieve an HTML file (or object in general), we will this updated in Kibana. Let’s see what happens…

Don’t forget to run Logstash with this new configuration file:

ubuntu@elk:~/logstash-1.5.2$ sudo bin/logstash -f logstash-access.conf 

The moment Logstash is started, you will see messages appearing in your console:

{
        "message" => "187.102.35.62 - - [17/Jul/2015:11:23:07 +0000] \"GET / HTTP/1.1\" 200 11819 \"-\" \"x00_-gawa.sa.pilipinas.2015\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:23:07.000Z",
           "host" => "elk",
           "path" => "/var/log/apache2/access.log",
           "type" => "apache-access",
       "clientip" => "187.102.35.62",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:23:07 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "11819",
       "referrer" => "\"-\"",
          "agent" => "\"x00_-gawa.sa.pilipinas.2015\""
}
{
        "message" => "173.38.209.6 - - [17/Jul/2015:11:48:22 +0000] \"GET / HTTP/1.1\" 200 3594 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:48:22.000Z",
           "host" => "elk",
           "path" => "/var/log/apache2/access.log",
           "type" => "apache-access",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:48:22 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "3594",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""

Showing something in Kibana

First things first, we left off part 1 with a default Kibana install but it didn’t do a lot yet.

kibana-default

Just take the default settings (as shown in previous picture) for now to create the index. You will get the following:

Kibana-index
And index has now been configured. If you want, you could then go and hit the ‘Discover’ tab. It should not show a lot yet.

When I hit the webpage, I found following entry in my console:

{
        "message" => "173.38.209.6 - - [17/Jul/2015:12:30:20 +0000] \"GET /icons/ubuntu-logo.png HTTP/1.1\" 304 179 \"http://173.39.243.147/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T12:30:20.000Z",
           "host" => "elk",
           "path" => "/var/log/apache2/access.log",
           "type" => "apache-access",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:12:30:20 +0000",
           "verb" => "GET",
        "request" => "/icons/ubuntu-logo.png",
    "httpversion" => "1.1",
       "response" => "304",
          "bytes" => "179",
       "referrer" => "\"http://173.39.243.147/\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""
}

Let’s now see if this also appears in Kibana. If you doubt this is real, check the date and time of the JSON message above and validate it with the picture below 🙂
Kibana_apache_access

This concludes part 2.