ELK tutorial: part 3

By | 17/07/2015

Installing and configuring Logstash-Forwarder

On the central server, create a folder called ‘/etc/pki/tls/certs’ and ‘/etc/pki/tls/private’:

ubuntu@elk: sudo mkdir -p /etc/pki/tls/certs
ubuntu@elk: sudo mkdir /etc/pki/tls/private

Because logstash-forwarder is using an SSL connection to the ELK server, we will need to create a certificate. To do so, add the following to your ‘/etc/ssl/openssl.cfg’ file (most likely the v3_ca tag is already in the file):

[ v3_ca ]
subjectAltName =IP:

Then let’s go ahead and create the certificate as follows:

ubuntu@elk:/etc/pki/tls$ sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

You will see that a certificate is created and copied to ‘etc/pki/tls/certs’. Then obviously we will need to copy this certificate to the webserver that will communicate with our ELK server. Do this as follows:

ubuntu@elk:/etc/pki/tls/certs$ sudo scp -i keypair_ccs.pem logstash-forwarder.crt ubuntu@:/tmp

On each new server, we need to copy the certificate from the ‘/tmp’ folder to the ‘/etc/pki/tls/certs’ folder:

ubuntu@webserver:/tmp$ cp logstash-forwarder.crt /etc/pki/tls/certs

We also need to install logstash-forwarder of course:

ubuntu@webserver:/etc/pki/tls/certs$ echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list
deb http://packages.elasticsearch.org/logstashforwarder/debian stable main
ubuntu@webserver:/etc/pki/tls/certs$ wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
--2015-07-17 10:53:20--  http://packages.elasticsearch.org/GPG-KEY-elasticsearch
Resolving packages.elasticsearch.org (packages.elasticsearch.org)... 50.16.229.96, 23.23.156.102, 174.129.21.167, ...
Connecting to packages.elasticsearch.org (packages.elasticsearch.org)|50.16.229.96|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1768 (1.7K) [binary/octet-stream]
Saving to: 'STDOUT'

100%[==================================================================================================================>] 1,768       --.-K/s   in 0s      

2015-07-17 10:53:20 (391 MB/s) - written to stdout [1768/1768]

OK
ubuntu@webserver:/etc/pki/tls/certs$ sudo apt-get update
ubuntu@webserver:/etc/pki/tls/certs$ sudo apt-get install logstash-forwarder

...
Preparing to unpack .../logstash-forwarder_0.4.0_amd64.deb ...
Unpacking logstash-forwarder (0.4.0) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up logstash-forwarder (0.4.0) ...
 Adding system startup for /etc/init.d/logstash-forwarder ...
   /etc/rc0.d/K20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc1.d/K20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc6.d/K20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc2.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc3.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc4.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
   /etc/rc5.d/S20logstash-forwarder -> ../init.d/logstash-forwarder
Logs for logstash-forwarder will be in /var/log/logstash-forwarder/

We then need to modify the configuration file for logstash-forwarder:

ubuntu@webserver:/etc/pki/tls/certs$ sudo nano /etc/logstash-forwarder.conf

and copy/paste the following:

{
  "network": {
    "servers": [ "" ],
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/var/log/apache2/access.log"
      ]
    ,
      "fields": { "type": "apache-remote2" }
    }
   ]
}

Finally, let’s check if our logstash-forwarder has established a secure connection to our logstash server:

ubuntu@apachewebserver2:/var/www$ cd /var/log/logstash-forwarder/
ubuntu@apachewebserver2:/var/log/logstash-forwarder$ ls
logstash-forwarder.err  logstash-forwarder.log
ubuntu@apachewebserver2:/var/log/logstash-forwarder$ tail logstash-forwarder.err 


2015/07/17 10:59:00.213359 Waiting for 1 prospectors to initialise
2015/07/17 10:59:00.213415 Launching harvester on new file: /var/log/apache2/access.log
2015/07/17 10:59:00.213467 harvest: "/var/log/apache2/access.log" (offset snapshot:0)
2015/07/17 10:59:00.213512 All prospectors initialised with 0 states to persist
2015/07/17 10:59:00.213808 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/07/17 10:59:00.214301 Connecting to []:5000 () 
2015/07/17 10:59:00.309486 Connected to 
2015/07/17 10:59:05.250653 Registrar: processing 3 events
{
        "message" => "173.38.209.6 - - [17/Jul/2015:11:17:52 +0000] \"GET / HTTP/1.1\" 200 447 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:17:52.000Z",
           "type" => "apache-remote1",
           "file" => "/var/log/apache2/access.log",
           "host" => "apache-webserver1",
         "offset" => "627275",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:17:52 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "447",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""
}
{
        "message" => "173.38.209.6 - - [17/Jul/2015:11:17:56 +0000] \"GET / HTTP/1.1\" 200 448 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\"",
       "@version" => "1",
     "@timestamp" => "2015-07-17T11:17:56.000Z",
           "type" => "apache-remote2",
           "file" => "/var/log/apache2/access.log",
           "host" => "apachewebserver2",
         "offset" => "828",
       "clientip" => "173.38.209.6",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "17/Jul/2015:11:17:56 +0000",
           "verb" => "GET",
        "request" => "/",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "448",
       "referrer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36\""
}

When you now look to Kibana, you will …

Kibana