How We Graylog2 Choose / Pixonic Blog

Each relatively large production sooner or later raises the question of centralized collection and viewing of logs. Currently, there is a huge selection of open source, paid, online and on-premises solutions. I will try to explain the selection process in our particular case.

This review article, the essence of which is to talk about the main features of Graylog2, why we chose it and how we operate it.

A little about our infrastructure (a little more details can be read here): we use 7 data centers around the world, about 500 production servers and a couple of hundred different applications from which we would like to collect logs. All this is running both Linux and Windows, and heterogeneous services are spinning on top. All services have their own log format, while there is also Java, which has a peculiar StackTrace.

Our requirements and what we wanted to get as a result

On the part of programmers and all interested, the requirements for logs were simple:

• the log sending agent should not load the system heavily;
• the ability to add custom fields at an arbitrary point in time, on arbitrary servers;
• search, sorting and so on;
• the ability to send POST logs with requests or something similar (for sending logs, for example, from mobile devices).

There is nothing complicated here, everything is quite usual. But in our case, it was still necessary to satisfy the following requirements for servicing the service:

• authentication in openLDAP (in our case, it is FreeIPA );
• convenient delineation of rights;
• convenient configuration of clients (preferably from one place);
• the ability to automatically install agents on all used system options;
• the ability to conveniently monitor both the serviceability of services and the necessary metrics;
• the presence of community and documentation, or commercial support;
• simple scaling.

This set of requirements was quite critical so that the new service could fully fit into the existing infrastructure, taking into account the peculiarities of our automation, distribution of rights and not be a black sheep, which would have to spend too much time. The service should be convenient for end users and satisfy their requirements.

At this stage, we realized that we only had some commercial solutions and Graylog2 from opensource.

How to count the number of logs and load

In short, no way. Therefore, here I will point out the basic approaches and nuances that helped us in this matter.

In the beginning, we took and looked at the number of logs on the focus group of servers, measured the dynamics of changes over 2 weeks. We multiplied the resulting number by the number of servers. As a result, the average number of logs was about 1TB per day. These logs needed to be stored from 2 weeks to 3 months.

At this stage, when calculating commercial solutions and our own infrastructure, it was decided to use Graylog2. Having decided that the best way to calculate the real load is to get part of the prod traffic to the test server, we deployed one Graylog2 node and sent traffic from a certain focus group there.

About a week we saw a load of 10-20k messages per second and in general were ready to lay on these numbers when deploying a combat cluster. But at some point something broke on the servers, the number of logs grew by almost 10 times, and on one server we saw a surge of up to 100k messages per second. At the same time, part of the StackTrace from Java applications did not fit into the allowed log size. At this moment, we realized that the logs are needed just for convenient work in such critical situations, and all previous calculations were carried out exclusively in normal conditions.

Main conclusions:

• counting logs under normal conditions does not give a picture of what is happening in case of accidents. A collection of logs is needed precisely for the operational solution of these situations;
• different services and languages ​​write messages in their own way and such situations must be taken into account in advance;
• cluster performance should allow processing several times more messages from normal load.

A small description of the functionality of Graylog2

The main reasons why we chose it:

• A well-proven product. With good documentation and coverage of the main problems.
• Ability to configure agents through a web interface through Collectors.
• Simple and functional integration with OpenLDAP, including group synchronization.
• Convenient horizontal scaling.
• Huge selection of different variations of input for obtaining logs.
• The presence of plugins and extensions.
• A fairly simple and convenient query and sampling language.
• The presence of dashboards and the possibility of notifications.

This functionality covered almost all of our needs and greatly simplified life. In just a couple of months, from a test service with a dubious future, it turned into a rather important business-critical unit and for six months it has been doing remarkably well with its tasks.

Of course, the introduction of such solutions is not the most transparent process. We were faced with a fairly large number of nuances and pitfalls both in the initial setup, and during further operation. In the next article I will tell you exactly how we tuned and tuned Graylog2 and its components such as MongoDB and Elasticsearch.