Remote logging in journald or Still “you don't need it”?
Disclaimer
All experiments were conducted on CentOS Linux release 7.2.1511 as the main system, with the latest available from the stock turnip systemd (systemd-219-19.el7_2.13). I hope that some of the data will be irrelevant at the time of publication of the article.
Introduction
Starting to grab linux distributions with the Fedora 15 release, systemd finally won. Bison and aksakals are gradually accustomed to units and systemctl. The last defenders of the Good Old grind their teeth. In these realities, it is not possible to circumvent systemd child products. And today let's talk, for example, about journald.
Journald itself (and journalctl, respectively) is a great tool. Originating as a project slightly alien to systemd, by now journald has become the second tool in the systemd family that system administrators are introduced to. I really like the idea of runtime log storage with the optional ability to reset it for permanent storage, the idea with boot-id journalctl --boot
and machine-id ( journalctl --machine
), the ability to call logs of any registered applications from a single interface journalctl -u
, and the presence of log rotation out of the box (as if in place , and in time) with journalctl --vacuum-size
or journalctl --vacuum-time
. Of the good, one cannot but mention the parameters since
, until
and priority
with which events can be “grabbed” by time and priority, and, of course, the parameterutc
relieving a huge headache with multi-timezone teams and projects. The binary file storage format may be a bit controversial, but this choice was explained by the authors during the first announcements of journald (security and low cost of storage, integration with systemd, forced single format, portability).
Unfortunately, the journald directory mechanism, which allows, for example, organizing the translation of log messages or providing urls to end users to solve problems with specific errors, has not gained much popularity. But on the whole, even the systemd developers do not take full advantage of this opportunity - let alone us mortals.
It is all implemented, working, described in hundreds of manuals, verified. Thanks a lot, but I want more.
Preamble
Not so long ago, a friend asked him to configure a server for collecting logs. Haha! I thought, this is an excellent opportunity to study journald in the light of its capabilities as a central log server!
It’s clear that the tool depends on the task. So this time a brief collection of requirements was carried out:
- It is necessary to organize a single point of logging
- It is necessary to organize the possibility of connecting to a single point of logging in advance an unknown number of clients (and disconnecting them, respectively)
- It is advisable to organize the ability to view logs online, with the automatic receipt of new entries
- When performing the previous paragraph, log storage at a central point can be limited to N hours (not even days)
- The message format is inconsistent up to the multiline
- The key software of interest is written to stdout (yes, you can change the behavior or naprostil, but we buy what they sell), the console does not close
And general technical introduction:
- Data is transmitted over public networks
- Dedicated Host Machine: Centos7.2 (latest)
- Connected machines: Ubuntu 16.04
Systemd and journald seem to be a good choice, right? After all, all the points have already been implemented! Raise the test bench!
Host host
Well, let's get started.
We are interested in three components:
systemd-journal-gatewayd
- http-daemon that opens a port for viewing (or downloading) log entriessystemd-journal-remote
- a daemon downloading or accepting log records on a central serversystemd-journal-upload
- a daemon duplicating log entries to a remote server
All these components are included in the centos package systemd-journal-gateway
, so we do:
yum -y update && yum -y install systemd-journal-gateway
For receiving logs on the host machine, a daemon is used systemd-journal-remote
. We’ll start with him.
systemd-journal-remote
Remote has two modes of operation: active (in which it goes to the remote log and downloads logs - including in tracking mode. For this mode, all remote machines need it systemd-journal-gatewayd
) and passive (in which the daemon hangs and waits for it to be will come). Given the unknown number of cars - select the passive mode. The whole setup, in fact, is to make the directory /var/log/journal/remote
, assign it the rights of the desired user and start the service:
mkdir -p /var/log/journal/remote
chown systemd-journal-remote:systemd-journal-remote /var/log/journal/remote
systemctl start /var/log/journal/remote
Since we are building a demo stand, let's switch the protocol for receiving files from https to http. To do this, you need to edit the start service file /lib/systemd/system/systemd-journal-remote.service
, replacing the option listen-https
with listen-http
. It will also be useful to correct /lib/systemd/system/systemd-journal-remote.socket
by specifying only the address of interest for binding the daemon.
The daemon started, hangs in passive mode, works on http. Hooray!
Devil in detail
Firstly, when creating a directory, /var/log/journal
all your system logs will begin to be written to the directory . To change this behavior and return it as it was, you need to fill in the configuration file (or even better , since the first one can be frayed during updates), namely the line . By default, the value of this parameter , which means "if there is a directory in var, journald writes logs there." In my case, the parameter had to be forced to set to ./var/log/journal/
/etc/systemd/journald.conf
/etc/systemd/journald.conf.d/<ваш конфиг>.conf
Storage=
auto
volatile
systemd-journal-upload
With upload it’s even easier: we create /etc/systemd/journal-upload.d/<ваш конфиг>.conf
by writing there:
[Upload]
URL=http://:<порт удаленного демона>
and run systemctl start systemd-journal-upload
Devil in detail
Secondly, the daemon will not start due to a very old bug in Centos . So do usermod his hands usermod -a -G systemd-journal systemd-journal-upload
. After that, the daemon should start successfully.
Remote clients
On remote clients, I recall, is Ubuntu 16.04. So we put there apt-get install systemd-journal-remote
and make edits /etc/systemd/journal-upload.d/<ваш конфиг>.conf
similar to the previous paragraph (except usermod
).
systemd-journal-gatewayd
And here, in fact, there is nothing to describe. The current version introduced in Centos does not allow specifying non-standard directories for the specified daemon. Moreover, the standard directory for logs from other machines, according to the logic of the developers, is non-standard for the log viewer. In new versions of systemd, this seems to be fixed , but we will not install non-native versions ...
Well, let's at least observe the logs:
journalctl -D /var/log/journal/remote --follow
Well, something seems to work ...
Devil in detail
Thirdly, thanks to the old bug in systemd , your directory will start to swell wildly /var/log/journal/remote
. And then nothing will help you ...
Let's get to the end!
However, since we have come this far, let's get to the end. We start the daemon on the host machine systemd-journal-gatewayd
(again, do not forget to fix /lib/systemd/system/systemd-journal-gatewayd.socket
it to limit the daemon, right?) And carefully study the browser’s web interface:
- There is virtually no swap of logs online
- Fierce
on mouse over
andon mouse scroll
for logs - microhttpd under the hood
- Sorry, but terrifying interface
- It is possible to filter logs by systemd-unit, but (due to the number of non-informative session units, for example, from crown scripts), the inability to use filters
- Все юниты пишутся полностью — если вы используете например пароли к базам данных в крон-строках, ваши пароли будут вас ждать
Нда. Возможно, и хорошо, что не получилось его настроить, м?
Итого
Для proof of concept за пару дней был написан pet-проект, благо journald предоставляет родную библиотеку для питона. Из особенностей:
- Онлайн-просмотр по SSE
- Возможность фильтровать выводимые юниты администратором на уровне конфигурации
- Возможность фильтровать хосты/приоритет/юниты пользователем на уровне веб-приложения
- Ну и немножко бутстрапа
Этот же проект был раскатан приятелю, с описанием имеющихся проблем, отказом от ответственности и принудительной очисткой журналов.
But on large projects, personally, I probably will not use journald as a central repository of logs for a long time to come. Until the above bugs are removed, my choice is definitely in favor of syslog.
Article author: Stepan Karamyshev