Update timezone in logstash

Hello! For a couple of months, our company has been successfully using the logstash-elasticsearch-kibana bundle for production to collect and process a fairly large amount of logs. Having glanced at kibana after transferring the clock, it was found out that all the logs come with a time delay of 1 hour. Under the cat, I want to share a solution to the timezones problem in the logstash-elasticsearch-kibana bundle and the finished logstash assembly with updated timezones.

I will not particularly delve into how the change of timezone influenced the final view of the logs, but in fact in kibana we received events created at the current time as events an hour earlier. The result was quite a funny picture, given that not all rsyslogd restarted automatically and picked up new timezone, a couple of servers continued to work according to the old time zones.

You can solve the problem of course with the crutch setting logstash, but I decided to fully update the timezone in logstash.


In fig. graph of the number of events at the time of transition to logstash with the correct timezones.
Logstash has three sources of time zones at once - one in java , the second in the ruby tzinfo gem , and the third compiled in jruby (artifact of joda-time and joda-timezones).

  1. Let's start with a simple one - update tzdata in java:


    To do this, use the utility from Oracle tzupdater , can be downloaded from the Oracle website .

    Updating occurs literally in one command:

    java -jar tzupdater.jar  -V
tzupdater version 1.4.7-b01
JRE time zone data version: tzdata2014c
Embedded time zone data version: tzdata2014g
java -jar tzupdater.jar  -u
java -jar tzupdater.jar  -V
tzupdater version 1.4.7-b01
JRE time zone data version: tzdata2014g
Embedded time zone data version: tzdata2014g

  2. Build jruby with the new tzdata:


    1. To get started, let's build the joda-time artifact with the new timezones:
      Before building, install the maven and ant packages:
      apt-get install maven ant
      Download the latest available joda-timezones package, replace the tzdata version in it, and specify the more recent joda-time in the dependencies (2.5 from 10.3.2014):
      mkdir joda-time
cd joda-time
wget http://search.maven.org/remotecontent?filepath=org/jruby/joda-timezones/2013d/joda-timezones-2013d.pom -O pom.xml
sed -i 's/2013d/2014g/' pom.xml
sed -i 's/<version>2.2/<version>2.5/' pom.xml
mvn package
mvn instal
      Now, maven has got the latest version of joda-timezones.
    2. Download the source code for the jruby version used in logstash (for logstash 1.4 it is jruby-1.7.11):
      wget https://github.com/jruby/jruby/archive/1.7.11.tar.gz
tar -xvf 1.7.11.tar.gz
cd jruby-1.7.11/
      Replace the artifact version in ./core/pom.xml:
      sed -i 's/<tzdata.version>2013d/<tzdata.version>2014g/' ./core/pom.xml
sed -i 's/<tzdata.jar.version>2013d/<tzdata.jar.version>2014g/' ./core/pom.xml
      In ./pom.xml:
      sed -i 's/<joda.time.version>2.3/<joda.time.version>2.5/' ./pom.xml
      Putting jruby together:
      ~/jruby-1.7.11# mvn -Pcomplete
      After the build, check which version of tz jruby answers, at the same time compare the time with the real one (in the end I run http-server, which would be more convenient to collect logstash):
      ~/jruby-1.7.11# java -jar ./maven/jruby-complete/target/jruby-complete-1.7.11.jar -rrbconfig -e 'p RbConfig::CONFIG["tzdata.version"]'"2014g"
~/jruby-1.7.11# java  -jar ./maven/jruby-complete/target/jruby-complete-1.7.11.jar -e 'p Time.now'
2014-10-29 14:58:07 +0500
~/jruby-1.7.11# cd ./maven/jruby-complete/target/;python -m SimpleHTTPServer

  3. Putting logstash together:


    Download unpack:
    wget https://github.com/elasticsearch/logstash/archive/v1.4.2.tar.gz
tar -xvf v1.4.2.tar.gz
mv logstash-1.4.2 logstash-contrib;cd logstash-contrib
    We make changes:
    diff --git a/Makefile b/Makefile
index 0ec3da1..7fcca1a 100644
--- a/Makefile+++ b/Makefile
@@ -7,7 +7,7 @@ ELASTICSEARCH_VERSION=1.1.1
 WITH_JRUBY=java -jar $(shell pwd)/$(JRUBY) -S
 JRUBY=vendor/jar/jruby-complete-$(JRUBY_VERSION).jar
-JRUBY_URL=http://jruby.org.s3.amazonaws.com/downloads/$(JRUBY_VERSION)/jruby-complete-$(JRUBY_VERSION).jar+JRUBY_URL=http://127.0.0.1:8000/jruby-complete-$(JRUBY_VERSION).jar
 JRUBY_CMD=bin/logstash env java -jar $(JRUBY)
 ELASTICSEARCH_URL=http://download.elasticsearch.org/elasticsearch/elasticsearch
diff --git a/logstash.gemspec b/logstash.gemspec
index 4917d83..6ba8ae4 100644
--- a/logstash.gemspec+++ b/logstash.gemspec
@@ -23,6 +23,7 @@ Gem::Specification.new do |gem|
   gem.add_runtime_dependency "stud"               #(Apache 2.0 license)
   gem.add_runtime_dependency "clamp"              # for command line args/flags (MIT license)
   gem.add_runtime_dependency "i18n", [">=0.6.6"]  #(MIT license)
+  gem.add_runtime_dependency "tzinfo", [">=1.2.2"]#(MIT license)
   # Web dependencies
   gem.add_runtime_dependency "ftw", ["~> 0.0.39"] #(Apache 2.0 license)
diff --git a/tools/Gemfile.jruby-1.9.lock b/tools/Gemfile.jruby-1.9.lock
index dc11fd5..41e4362 100644
--- a/tools/Gemfile.jruby-1.9.lock+++ b/tools/Gemfile.jruby-1.9.lock
@@ -169,7 +165,7 @@ GEM
       http_parser.rb (~> 0.5.0)
       json (~> 1.8)
       simple_oauth (~> 0.2.0)
-    tzinfo (1.1.0)+    tzinfo (1.2.2)
       thread_safe (~> 0.1)
     user_agent_parser (2.1.2)
     uuidtools (2.1.4)

    And run the assembly:
    make tarball

    Upon completion of the build, we get ./build/logstash-1.4.2.tar.gz - the finished logstash with the updated tzdata!


PS All manipulations were performed on ubuntu 14.04 installed Oracle Java (TM) Development Kit ( JDK) 7 (build 1.7.0_72-b14)

The finished assembly logstash 1.4.3 with updated timezone, you can download c yandex-disk or mail.ru .

Tags:

Also popular now: