View apache archive logs using Logstash + Elastisearch + Kibana

Greetings.

No, so long ago I had to run through the old apache logs. It was necessary to make a selection on several IP addresses, to find some anomalies and attempts of SQL-injection'ov. There were not so many logs, about a million lines, and you could safely do everything with the standard set of grap-awk-uniq-wc, etc.

Since I have been using the Logstash-Elasticsearch-Kibana bunch for analysis-viewing all kinds of logs for some (more than a year) time, I decided to use it in this situation.

A brief description of the main components of the system.

Logstash- A free open-source program in java for collecting and normalizing logs. It can accept logs either from local files or via tcp / udp ports. At the time of writing, there are 26 different input filters. There is even an input module for collecting messages from twitter or irc.

Elasticsearch is a free open-source search engine based on Apache Lucene. Fast, easily customizable and very scalable.

Kibana is a web interface written in ruby ​​to display data from Elasticsearch. Simple setup, but many features - search, graphics, stream.

1. Elasticsearch

1.1 Download Elasticsearch (size 16MB):
It is important to note that for Logstash version 1.1.9 Elasticsearch must be exactly version 0.20.2.
# wget download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz

1.2 Unpacking the file:
# tar -zxf elasticsearch-0.20.2.tar.gz
Those who want to amaze others can add the “v” key :)

1.3 By and large, you can run Elasticsearch with factory settings. But I still change some parameters.
Go to your favorite text editors in the settings file:
# vi elasticsearch-0.20.2/config/elasticsearch.yml

List of my changes for standalone solutions:

cluster.name: logs
index.number_of_replicas: 0
path.data: /elasticsearch/elasticsearch-0.20.2/data
path.work: /elasticsearch/elasticsearch-0.20.2/work
path.logs: /elasticsearch/elasticsearch-0.20.2/logs
bootstrap.mlockall: true
discovery.zen.ping.multicast.enabled: false

Before starting Elasticsearch, make sure that the directories specified in path.data, path.work and path.logs exist.

1.4 Run Elasticsearch in foreground mode to make sure that the server is working correctly:
# ./bin/elasticsearch -f
If we see a line like this, then the server has started

[2013-01-11 1151:35,160][INFO ][node                     ] [Virgo] {0.20.2}[17620]: started

1.5 To start Elasticsearch in the background (daemon) mode, just remove the " -f " switch .
# ./bin/elasticsearch

If two tcp ports 9200 and 9300 appeared on your server in LISTEN mode , this means Elasticsearch is ready to work.

2. Logstash

2.1 Download the latest version of Logstash 1.1.9 (size 60MB)
# wget logstash.objects.dreamhost.com/release/logstash-1.1.9-flatjar.jar

2.2 Create a configuration file (apache.conf) for accepting apache archive logs, normalizing them and entering them into the Elasticsearch database:

input {
  tcp {
    type => "apache-access"
    port => 3338
  }
}
filter {
  grok {
    type => "apache-access"
    pattern => "%{COMBINEDAPACHELOG}"
  }
  date {
    type => "apache-access"
    timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
  }
}
output {
  elasticsearch {
    embedded => false
    cluster => logs
    host => "172.28.2.2"
    index => "apache-%{+YYYY.MM}"
    type => "apache-access"
    max_inflight_requests => 500
  }
}

Short description of some parameters:

port => 3338

In our case, Logstash will listen on tcp port 3338. We will send netcat'm apache logs to it.

cluster => logs

specify the name of the cluster that we registered in cluster.name: in the Elasticsearch settings

host => "172.28.2.2"

ip address on which Elasticsearch runs

index => "apache-%{+YYYY.MM}"

in my case there are not so many 40,000 daily apache logs, so a monthly index is enough. If the logs per day are 500,000 or more, then it is more appropriate to create a daily index “apache -% {+ YYYY.MM.dd}”

2.3 Run Logstash
# java -Xmx64m -jar logstash-1.1.9-flatjar.jar agent -f ./apache.conf

Check that Logstash is running:
# netstat -nat |grep 3338
If port 3338 is present, then Logstash is ready to accept logs.

2.4. We start sending old logs to apache logstash.
# gunzip -c archived.apache.log.gz |nc 127.0.0.1 3338
How quickly all logs are filled in depends on many parameters - CPU, RAM, number of logs.
In my case, 600 thousand lines of logs were completely uploaded to the database in 4 minutes. So your mileage may vary.

2.5 While the process of uploading logs is ongoing, you can check whether the data is in the Elasticsearch database.
To do this, enter in the browserelasticsearch_ip : 9200 / _status? pretty = true if you find lines like this there:

"index" : "apache-2011.09"

then everything works as required.

3. Kibana

3.1 Installing Kibana:
git clone --branch=kibana-ruby github.com/rashidkpc/Kibana.git
cd Kibana
gem install bundler
bundle install

If you are behind a proxy server, then before the git clone ... command specify your proxy server:
git config --global http.proxy proxy.domain.com:3128

3.2 Kibana
# vi KibanaConfig.rb
configuration Settings that may require changes:

Elasticsearch = "localhost:9200"
KibanaPort = 5601
KibanaHost = '172.28.2.2'
Smart_index_pattern = 'apache-%Y.%m'
Smart_index_step = 2592000

3.3 Starting Kibana
# ruby kibana.rb

After a successful launch, the following text appears on the screen:

== Sinatra/1.3.3 has taken the stage on 5601 for development with backup from Thin
>> Thin web server (v1.5.0 codename Knife)
>> Maximum connections set to 1024
>> Listening on 172.28.21.21:5601, CTRL+C to stop

3.4 Starting to view the logs
In the browser, enter the address http://172.28.21.21:5601 and get a convenient, fast interface for viewing old apache logs.

For those who want to see what Kibana + Logsatsh + Elasticsearch is, there is a demo page http://demo.logstash.net/

Thank you for your attention,