Remote logging in journald or Still “you don't need it”?


    Disclaimer


    All experiments were conducted on CentOS Linux release 7.2.1511 as the main system, with the latest available from the stock turnip systemd (systemd-219-19.el7_2.13). I hope that some of the data will be irrelevant at the time of publication of the article.


    Introduction


    Starting to grab linux distributions with the Fedora 15 release, systemd finally won. Bison and aksakals are gradually accustomed to units and systemctl. The last defenders of the Good Old grind their teeth. In these realities, it is not possible to circumvent systemd child products. And today let's talk, for example, about journald.



    Journald itself (and journalctl, respectively) is a great tool. Originating as a project slightly alien to systemd, by now journald has become the second tool in the systemd family that system administrators are introduced to. I really like the idea of ​​runtime log storage with the optional ability to reset it for permanent storage, the idea with boot-id journalctl --bootand machine-id ( journalctl --machine), the ability to call logs of any registered applications from a single interface journalctl -u, and the presence of log rotation out of the box (as if in place , and in time) with journalctl --vacuum-sizeor journalctl --vacuum-time. Of the good, one cannot but mention the parameters since, untiland prioritywith which events can be “grabbed” by time and priority, and, of course, the parameterutcrelieving a huge headache with multi-timezone teams and projects. The binary file storage format may be a bit controversial, but this choice was explained by the authors during the first announcements of journald (security and low cost of storage, integration with systemd, forced single format, portability).


    Unfortunately, the journald directory mechanism, which allows, for example, organizing the translation of log messages or providing urls to end users to solve problems with specific errors, has not gained much popularity. But on the whole, even the systemd developers do not take full advantage of this opportunity - let alone us mortals.


    It is all implemented, working, described in hundreds of manuals, verified. Thanks a lot, but I want more.


    Preamble


    Not so long ago, a friend asked him to configure a server for collecting logs. Haha! I thought, this is an excellent opportunity to study journald in the light of its capabilities as a central log server!


    It’s clear that the tool depends on the task. So this time a brief collection of requirements was carried out:


    1. It is necessary to organize a single point of logging
    2. It is necessary to organize the possibility of connecting to a single point of logging in advance an unknown number of clients (and disconnecting them, respectively)
    3. It is advisable to organize the ability to view logs online, with the automatic receipt of new entries
    4. When performing the previous paragraph, log storage at a central point can be limited to N hours (not even days)
    5. The message format is inconsistent up to the multiline
    6. The key software of interest is written to stdout (yes, you can change the behavior or naprostil, but we buy what they sell), the console does not close

    And general technical introduction:


    1. Data is transmitted over public networks
    2. Dedicated Host Machine: Centos7.2 (latest)
    3. Connected machines: Ubuntu 16.04

    Systemd and journald seem to be a good choice, right? After all, all the points have already been implemented! Raise the test bench!


    Host host


    Well, let's get started.
    We are interested in three components:


    1. systemd-journal-gatewayd - http-daemon that opens a port for viewing (or downloading) log entries
    2. systemd-journal-remote - a daemon downloading or accepting log records on a central server
    3. systemd-journal-upload - a daemon duplicating log entries to a remote server

    All these components are included in the centos package systemd-journal-gateway, so we do:


    yum -y update && yum -y install systemd-journal-gateway

    For receiving logs on the host machine, a daemon is used systemd-journal-remote. We’ll start with him.


    systemd-journal-remote


    Remote has two modes of operation: active (in which it goes to the remote log and downloads logs - including in tracking mode. For this mode, all remote machines need it systemd-journal-gatewayd) and passive (in which the daemon hangs and waits for it to be will come). Given the unknown number of cars - select the passive mode. The whole setup, in fact, is to make the directory /var/log/journal/remote, assign it the rights of the desired user and start the service:


    mkdir -p /var/log/journal/remote
    chown systemd-journal-remote:systemd-journal-remote /var/log/journal/remote
    systemctl start /var/log/journal/remote

    Since we are building a demo stand, let's switch the protocol for receiving files from https to http. To do this, you need to edit the start service file /lib/systemd/system/systemd-journal-remote.service, replacing the option listen-httpswith listen-http. It will also be useful to correct /lib/systemd/system/systemd-journal-remote.socketby specifying only the address of interest for binding the daemon.


    The daemon started, hangs in passive mode, works on http. Hooray!


    Devil in detail


    Firstly, when creating a directory, /var/log/journalall your system logs will begin to be written to the directory . To change this behavior and return it as it was, you need to fill in the configuration file (or even better , since the first one can be frayed during updates), namely the line . By default, the value of this parameter , which means "if there is a directory in var, journald writes logs there." In my case, the parameter had to be forced to set to ./var/log/journal//etc/systemd/journald.conf/etc/systemd/journald.conf.d/<ваш конфиг>.confStorage=autovolatile


    systemd-journal-upload


    With upload it’s even easier: we create /etc/systemd/journal-upload.d/<ваш конфиг>.confby writing there:


    [Upload]
    URL=http://:<порт удаленного демона>

    and run systemctl start systemd-journal-upload


    Devil in detail


    Secondly, the daemon will not start due to a very old bug in Centos . So do usermod his hands usermod -a -G systemd-journal systemd-journal-upload. After that, the daemon should start successfully.


    Remote clients


    On remote clients, I recall, is Ubuntu 16.04. So we put there apt-get install systemd-journal-remoteand make edits /etc/systemd/journal-upload.d/<ваш конфиг>.confsimilar to the previous paragraph (except usermod).


    systemd-journal-gatewayd


    And here, in fact, there is nothing to describe. The current version introduced in Centos does not allow specifying non-standard directories for the specified daemon. Moreover, the standard directory for logs from other machines, according to the logic of the developers, is non-standard for the log viewer. In new versions of systemd, this seems to be fixed , but we will not install non-native versions ...


    Well, let's at least observe the logs:


    journalctl -D /var/log/journal/remote --follow

    Well, something seems to work ...


    Devil in detail


    Thirdly, thanks to the old bug in systemd , your directory will start to swell wildly /var/log/journal/remote. And then nothing will help you ...


    Let's get to the end!


    However, since we have come this far, let's get to the end. We start the daemon on the host machine systemd-journal-gatewayd(again, do not forget to fix /lib/systemd/system/systemd-journal-gatewayd.socketit to limit the daemon, right?) And carefully study the browser’s web interface:


    • There is virtually no swap of logs online
    • Fierce on mouse overand on mouse scrollfor logs
    • microhttpd under the hood
    • Sorry, but terrifying interface
    • It is possible to filter logs by systemd-unit, but (due to the number of non-informative session units, for example, from crown scripts), the inability to use filters
    • Все юниты пишутся полностью — если вы используете например пароли к базам данных в крон-строках, ваши пароли будут вас ждать

    Нда. Возможно, и хорошо, что не получилось его настроить, м?


    Итого


    Для proof of concept за пару дней был написан pet-проект, благо journald предоставляет родную библиотеку для питона. Из особенностей:


    1. Онлайн-просмотр по SSE
    2. Возможность фильтровать выводимые юниты администратором на уровне конфигурации
    3. Возможность фильтровать хосты/приоритет/юниты пользователем на уровне веб-приложения
    4. Ну и немножко бутстрапа

    Этот же проект был раскатан приятелю, с описанием имеющихся проблем, отказом от ответственности и принудительной очисткой журналов.


    But on large projects, personally, I probably will not use journald as a central repository of logs for a long time to come. Until the above bugs are removed, my choice is definitely in favor of syslog.


    Article author: Stepan Karamyshev


    Also popular now: