Installing ElasticSearch Logstash & Kibana #EKL #Logstash-forwader #COMBINEDAPACHELOG #AmazonEC2

It’s been a year since I last updated the blog, laziness wins any day! :P. This blog entry will illustrate how to setup an out of the box installation for EKL.This setup was done on Amazon EC2 instances, this will cover the following topics:

     a) Setting up ElasticSearch 

     b) Setting up Logstash Server

     c) Setting up Logstash-Forwader

     d) Setting up Kibana
        [Logstash 1.4.2 and Kibana 3 ElasticSearch 1.3]

Below is the pictorial setup which I have up and running.The Final aim would be to send Apache access logs from Server [EC2_A] to Server [EC2_B] and create a Elastic cluster named (elasticsearch) and show the graphical representation in Kibana

Here the EC2_A Server is our Logstash_forwader/Shipper. On our EC2_B we have the Elastic cluster and the logstash Server which is running, and the UI is shown in Kibana.

There are many online resources, which we can refer to get the above setup. However, they are not at one single place, I had to search at multiple places to get the above setup running.

One of the best resource I came across is the Logstash Cook Book  and the EKL installation guide by Digital Ocean.

Certainly, there were lot of initial breakers I faced, but with this blog entry I suppose one should not face any problems while installing.

For starters, who are not familiar with EKL can read about these components here. ElasticSearch(http://en.wikipedia.org/wiki/Elasticsearch), Logstash(http://logstash.net/), Kibana(http://rashidkpc.github.io/Kibana/).

Let’s Start!

We will try setting up the EC2_B box first.

EC2_B Config: (Micro instance) Ubuntu Server 14.04_32bit: Linux ip-192.168.2.2 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:02:19 UTC 2014 i686 i686 i686 GNU/Linux

Installing Dependencies:

(The only prerequisite required by Logstash is Java runtime)

$ sudo add-apt-repository -y ppa:webupd8team/java

$ sudo apt-get update

$ sudo apt-get -y install oracle-java7-installer

Now try:
$ java -version

java version "1.7.0_45"

Java(TM) SE Runtime Environment (build 1.7.0_45-b18)

Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

Install Elastic Search:

$ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.tar.gz

$ tar zxvf elasticsearch-1.1.1.tar.gz

$ cd elasticsearch-1.1.1/

$ ./bin/elasticsearch &    --This will start Elastic Search

Install Kibana:

$ cd ~; wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gz

$ tar xvf kibana-3.0.1.tar.gz

$ sudo vi ~/kibana-3.0.1/config.js --Now change the text from 9200 to 80

  elasticsearch: "http://"+window.location.hostname+":80”,

$ sudo mkdir -p /var/www/kibana3

$ sudo cp -R ~/kibana-3.0.1/* /var/www/kibana3/

Install nginx to host Kibana:

$ sudo apt-get install nginx

$ cd ~; wget https://github.com/elasticsearch/kibana/raw/master/sample/nginx.conf

$ vi nginx.conf   --Now change the value of root as below

  root to /var/www/kibana3;

$ sudo service nginx restart

Now go to http://[IP]/kibana3 to check if Kibana UI is visible.

Install Logstash:

$ curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz

$ tar zxvf logstash-1.4.2.tar.gz

Now Generate the SSL Certificate:

$ sudo mkdir -p /etc/pki/tls/certs

$ sudo mkdir /etc/pki/tls/private

Now we will edit the openssl.cnf file so that later on we won’t face any issues when we compile our logstash-forwader using go1.3 linux/amd64 on EC2_A (More details here)

$ Vi /etc/ssl/openssl.cnf

In the [v3_ca] section add the following entry

subjectAltName = IP:192.168.2.2

Note: Here the IP address has to be of the EC2_B. machine.

Now lets create a index on our Elastic cluster:

Lets first install a plugin named "head"

$ cd ~/elasticsearch-1.1.1/
$ bin/plugin --install mobz/elasticsearch-head

Now go to http://IP(EC2_B):9200/_plugin/head/

Go to indices tab and create a new index called "apache"

Now Generate the Self signed certs:

$ cd /etc/pki/tls; sudo openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

The same certificate "logstash-forwarder.crt” has to be imported to logstash_forwader server (EC2_A). Please do this using appropriate “scp” commands.

Configure Logstash:
$ nano ~/logstash-1.4.2/logstash.conf

input {

  lumberjack {

    port => 5000

    type => "apache-access"

    ssl_certificate => "/etc/pki/tls/certs/logstash- forwarder.crt"

    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" 

}

}

filter {

  grok {

    type => "apache-access"

    pattern => "%{COMBINEDAPACHELOG}"

  }

}

output { 

  elasticsearch {

 host => localhost

 protocol => http

 index => “apache"

 cluster => "elasticsearch"

 index_type => "apache"

     } }

This creates a configuration file which will make the log stash listen on port 5000 (lumberjack) and accept incoming logs from the logstash forwarder. Also, the grok filter here I have specified as %{COMBINEDAPACHELOG} since we will be sending the apache access logs from the EC2_A server.

Now setting up our EC2_A server (Logstash_Forwader/Shipper):

EC2_A: (Micro instance) Ubuntu Server 14.04_64bit: Linux ip-192.168.2.1 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Make sure your Apace Server in running on this machine and java is also installed, or please refer to the first step.This machine will be used as a shipper to send apache logs to EC2_B.

$ wget https://github.com/elasticsearch/logstash-forwarder/archive/master.zip

$ unzip logstash-forwarder-master.zip

$ cd logstash-forwarder-master

Installing the developer tools:

$ sudo apt-get install build-essential

Installing Go:

$ sudo apt-get install python-software-properties

$ sudo apt-add-repository ppa:duh/golang

$ sudo apt-get update

$ sudo apt-get install golang

$ sudo apt-get install ruby rubygems ruby-dev

$ sudo gem install fpm

Creating the forwarder deb package.

$ umask 022

$ make deb

You'll see a long sequence of compilation and then some final execution as the fpm command runs and creates the DEB package.

Listing 1.34: Forwarder make output

Installing the forwarder:

$ sudo dpkg -i logstash-forwarder_0.2.0_i386.deb

Now create a folder to place the "logstash-forwarder.crt" certificate. Before that we need to import the "logstash-forwarder.crt" cert file that we created on the EC2_B server. Please do it via necessary scp commands.

$ mkdir /etc/certs

Place the "logstash-forwarder.crt" file in the /certs folder.Also, create the logstash-conf file:

$ nano /etc/logstash-forwarder/logstash-forwarder.conf

change Below IP to the IP of your Logstash Server

{

"network":

 {

"servers": [ “IP[EC2_B]:5000" ],

"ssl ca": "/etc/certs/logstash-forwarder.crt",

"timeout": 15

 },

"files": [

    {

"paths": ["/var/log/apache2/access.log"],

"fields": { "type": "apache-access" }

     }

        ]

}

Now Start the forwarder:

$ cd /opt/logstash-forwarder
$ bin/logstash-forwarder -config="/etc/logstash-forwarder/logstash-forwarder.conf" &

Finally Starting the Logstash Server on (EC2_B):
$ cd ~/logstash-1.4.2/
$ bin/logstash -f logstash.conf & --This will start the logstash server 

Any further changes to the access logs will be now visible in your kibana dashboard. To check the above setup, hit the default apache page @ (http://IP[EC2_A]/) and check the changes recorded by your elastic cluster on the kibana dashboard.

The dashboard I use is my personal favourite, which can be found here.
I hope this blog entry will be useful for a successful EKL installation. Do write a comment below if you get stuck anywhere.