It’s been a year since I last updated the blog, laziness wins any day! :P.
This blog entry will illustrate how to setup an out of the box installation for
EKL.This setup was done on Amazon EC2 instances, this will cover the
following topics:
a)
Setting up ElasticSearch
b)
Setting up Logstash Server
c)
Setting up Logstash-Forwader
d)
Setting up Kibana
[Logstash 1.4.2 and Kibana 3 ElasticSearch 1.3]
[Logstash 1.4.2 and Kibana 3 ElasticSearch 1.3]
Below is the pictorial
setup which I have up and running.The Final aim would be to send Apache access logs from Server [EC2_A] to Server [EC2_B] and create a Elastic cluster named (elasticsearch) and show the graphical representation in Kibana
Here the EC2_A Server is our
Logstash_forwader/Shipper. On our EC2_B we have the Elastic cluster and the logstash Server which is running, and the UI is shown in Kibana.
There are many online resources, which we can refer to get the above
setup. However, they are not at one single place, I had to search at multiple
places to get the above setup running.
One of the best resource I came across is the Logstash Cook Book and the EKL installation guide by Digital Ocean.
Certainly, there were lot of initial breakers I faced, but with this blog
entry I suppose one should not face any problems while installing.
For
starters, who are not familiar with EKL can read about these components here.
ElasticSearch(http://en.wikipedia.org/wiki/Elasticsearch),
Logstash(http://logstash.net/),
Kibana(http://rashidkpc.github.io/Kibana/).
Let’s
Start!
We
will try setting up the EC2_B box first.
EC2_B Config: (Micro instance) Ubuntu Server 14.04_32bit: Linux ip-192.168.2.2 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:02:19 UTC 2014 i686 i686 i686 GNU/Linux
EC2_B Config: (Micro instance) Ubuntu Server 14.04_32bit: Linux ip-192.168.2.2 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:02:19 UTC 2014 i686 i686 i686 GNU/Linux
Installing
Dependencies:
(The only prerequisite required by Logstash is Java runtime)
(The only prerequisite required by Logstash is Java runtime)
$ sudo add-apt-repository -y
ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get -y install
oracle-java7-installer
Now try:
$ java -version
$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
Install Elastic Search:
$ tar zxvf elasticsearch-1.1.1.tar.gz
$ cd elasticsearch-1.1.1/
$ ./bin/elasticsearch & --This will start Elastic Search
Install Kibana:
Install Kibana:
$ tar xvf kibana-3.0.1.tar.gz
$ sudo vi ~/kibana-3.0.1/config.js --Now change the text from 9200 to 80
elasticsearch:
"http://"+window.location.hostname+":80”,
$ sudo mkdir -p /var/www/kibana3
$ sudo cp -R ~/kibana-3.0.1/* /var/www/kibana3/
Install nginx to host Kibana:
$ sudo apt-get install nginx
$ vi nginx.conf --Now change the value of root as below
root to /var/www/kibana3;
$ sudo service nginx restart
Now go to http://[IP]/kibana3 to check if Kibana UI is visible.
Install Logstash:
$ tar zxvf logstash-1.4.2.tar.gz
Now Generate the SSL Certificate:
$ sudo mkdir -p /etc/pki/tls/certs
$ sudo mkdir /etc/pki/tls/private
Now we will edit the openssl.cnf
file so that later on we won’t face any issues when we compile our logstash-forwader using go1.3 linux/amd64 on EC2_A (More
details here)
$ Vi /etc/ssl/openssl.cnf
In the [v3_ca] section add the following entry
subjectAltName = IP:192.168.2.2
Note: Here the IP address has to be of the EC2_B. machine.
Now lets create a index on our Elastic cluster:
Lets first install a plugin named "head"
$ cd ~/elasticsearch-1.1.1/
$ bin/plugin --install mobz/elasticsearch-head
Now go to http://IP(EC2_B):9200/_plugin/head/
Now Generate the Self signed certs:
$ cd /etc/pki/tls; sudo openssl req
-x509 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key
-out certs/logstash-forwarder.crt
The same certificate
"logstash-forwarder.crt” has to be imported to logstash_forwader server
(EC2_A). Please do this using appropriate “scp” commands.
Configure Logstash:
$ nano ~/logstash-1.4.2/logstash.conf
input {
lumberjack {
port
=> 5000
type
=> "apache-access"
ssl_certificate => "/etc/pki/tls/certs/logstash- forwarder.crt"
ssl_key
=> "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
grok {
type
=> "apache-access"
pattern
=> "%{COMBINEDAPACHELOG}"
}
}
output {
elasticsearch
{
host => localhost
protocol => http
index =>
“apache"
cluster =>
"elasticsearch"
index_type =>
"apache"
}
}
This
creates a configuration file which will make the log stash listen on port 5000
(lumberjack) and accept incoming logs from the logstash forwarder. Also, the
grok filter here I have specified as %{COMBINEDAPACHELOG} since we
will be sending the apache access logs from the EC2_A server.
Now setting up our EC2_A server
(Logstash_Forwader/Shipper):
EC2_A: (Micro instance) Ubuntu Server 14.04_64bit: Linux ip-192.168.2.1 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
EC2_A: (Micro instance) Ubuntu Server 14.04_64bit: Linux ip-192.168.2.1 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Make
sure your Apace Server in running on this machine and java is also installed, or please refer to the first step.This machine will be used as a shipper to send
apache logs to EC2_B.
$ unzip logstash-forwarder-master.zip
$ cd logstash-forwarder-master
Installing the developer tools:
$ sudo apt-get install build-essential
Installing Go:
$ sudo apt-get install python-software-properties
$ sudo apt-add-repository ppa:duh/golang
$ sudo apt-get update
$ sudo apt-get install golang
$ sudo apt-get install ruby rubygems ruby-dev
$ sudo gem install fpm
Creating the forwarder deb
package.
$ umask 022
$ make deb
You'll see a long sequence of compilation and then
some final execution as the fpm command runs and creates the DEB package.
Installing the forwarder:
$ sudo
dpkg -i logstash-forwarder_0.2.0_i386.deb
Now create a folder to place the "logstash-forwarder.crt" certificate. Before that we need to import the "logstash-forwarder.crt" cert
file that we created on the EC2_B server. Please do it via necessary scp
commands.
$ mkdir /etc/certs
Place
the "logstash-forwarder.crt" file in the /certs folder.Also, create the logstash-conf
file:
$ nano
/etc/logstash-forwarder/logstash-forwarder.conf
change Below IP to the IP of your Logstash Server
change Below IP to the IP of your Logstash Server
{
"network":
{
"servers": [
“IP[EC2_B]:5000" ],
"ssl ca":
"/etc/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths":
["/var/log/apache2/access.log"],
"fields": { "type":
"apache-access" }
}
]
}
Now Start the forwarder:
$ cd /opt/logstash-forwarder
$ bin/logstash-forwarder -config="/etc/logstash-forwarder/logstash-forwarder.conf" &
Finally Starting the Logstash Server on (EC2_B):
$ cd ~/logstash-1.4.2/
$ bin/logstash -f logstash.conf & --This will start the logstash server
$ bin/logstash-forwarder -config="/etc/logstash-forwarder/logstash-forwarder.conf" &
Finally Starting the Logstash Server on (EC2_B):
$ cd ~/logstash-1.4.2/
$ bin/logstash -f logstash.conf & --This will start the logstash server
Any further changes to the access logs will be now visible in your kibana dashboard. To check the above setup, hit the default apache page @ (http://IP[EC2_A]/) and check the changes recorded by your elastic cluster on the kibana dashboard.
The dashboard I use is my personal favourite, which can be found here.
I hope this blog entry will be useful for a successful EKL installation. Do write a comment below if you get stuck anywhere.