Logging management in systemd

  • Tutorial
Systemd journal

The systemd initialization daemon has become the de facto standard in modern Linux systems. Many popular distributions switched to it: Debian, RHEL / CentOS, Ubuntu (since version 15.04). Systemd uses a fundamentally different (compared to the traditional syslog tool) logging approach.
It is based on centralization: the specialized journal component collects all system messages (messages from the kernel, various services and applications). At the same time, there is no need to specifically configure sending logs: applications can simply write to stdout and stderr, while journal will save these messages automatically. This mode is also possible with Upstart, but it saves all the logs in a separate file, while systemd saves them in a binary database, which greatly simplifies the systematization and search.

Storing logs in binary files also avoids the difficulties of using parsers for different types of logs. If necessary, the logs can be easily converted to other formats (more on this will be described below).
Journal can work both with syslog and completely replace it.
The journalctl utility is used to view the logs. We will talk about the features and subtleties of working with it in this article.


Time setting



One of the major drawbacks of syslog is that it doesn’t save time zones. The journal eliminated this drawback: for logged events, you can specify both local time and universal coordinated time (UTC). The time is set using the timedatectl utility.
You can view the list of time zones using the command:

$ timedatectl list-timezones


Setting the desired time zone is as follows:

$ timedatectl set-timezone <time zone>

Upon completion of the installation, it will be useful to make sure that everything is done correctly:

$ timedatectl status
Local time: Thu 2015-07-30 11:24:15 MSK
Universal time: Thu 2015-07-30 08:24:15 UTC
RTC time: Thu 2015-07-30 08:24:15
Time zone: Europe / Moscow (MSK, +0300)
NTP enabled: no
NTP synchronized: no
RTC in local TZ: no
DST active: n / a


The very first line (Local time) should show the exact current time and date.

Journalctl: view logs



The journalctl utility is used to view the logs.
If you enter the journalсtl command without any arguments, a huge list will be displayed on the console:

- Logs begin at Wed 2015-07-29 17:12:48 MSK, end at Thu 2015-07-30 11:24:15 MSK. -
Jul 29 17:12:48 host-10-13-37-10 systemd-journal [181]: Runtime journal is using 4.0M (max allowed 20.0M, trying to leave 30.0M free of 195.9M available → current limit 20.0M )
Jul 29 17:12:48 host-10-13-37-10 systemd-journal [181]: Runtime journal is using 4.0M (max allowed 20.0M, trying to leave 30.0M free of 195.9M available → current limit 20.0M )
Jul 29 17:12:48 host-10-13-37-10 kernel: Initializing cgroup subsys cpuset
Jul 29 17:12:48 host-10-13-37-10 kernel: Initializing cgroup subsys cpu
Jul 29 17:12:48 host-10-13-37-10 kernel: Initializing cgroup subsys cpuacct
Jul 29 17:12:48 host-10-13-37-10 kernel: Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4- 1)) # 1 SMP Debian 3.16.7-ckt11-1 + deb8u2 (2015-07-17)
Jul 29 17:12:48 host-10-13-37-10 kernel: Command line: BOOT_IMAGE = / boot / vmlinuz-3.16.0-4-amd64 root = UUID = b67ea972-1877-4c5b-a328-29fc0d6c7bc4 ro console = tty1 console = ttyS0 video = 640x480 consoleblank = 0 panic = 15 c


Here we have given only a small fragment of it; in fact, it includes a gigantic amount of records.

Log Filtering



The journalctl utility has options with which you can filter logs and quickly extract the necessary information from them.

View logs from the current download



Using the -b option, you can view all the logs collected since the last boot of the system:

$ journalctl -b


View logs from previous sessions



Using journalctl, you can view information about previous sessions in the system - in some cases this can be useful.
However, keep in mind that saving information about previous sessions is not supported by default on all Linux distributions. Sometimes it needs to be activated.

To do this, open the journald.conf configuration file, find the [Journal] section in it and change the value of the storage parameter to persistent:

$ sudo nano /etc/systemd/journald.conf
...
[Journal]
Storage = persistent


View a list of previous downloads using the command:

$ journalctl --list-boots
0 9346310348bc4edea250555dc046b30c Thu 2015-07-30 12:39:49 MSK — Thu 2015-07-30 12:39:59 MSK


Her conclusion consists of four columns. The first of them indicates the serial number of the download, the second indicates its ID, and the third indicates the date and time. To view the log for a specific download, you can use identifiers from both the first and second columns:

$ journalctl -b 0


or

$ journalctl -b 9346310348bc4edea250555dc046b30c


Filter by date and time



Journalctl also has the ability to view logs for specific periods of time. The –since and –until options are used for this. Suppose we need to view the logs starting at 17.15 p.m. on July 20, 2015. To do this, you will need to run the command:
$ journalctl --since "2015-07-20 17:15:00"


If no date is specified with the since option, logs starting from the current date will be displayed on the console. If the date is specified, but the time is not specified, the default time value will be applied - “00:00:00”. Seconds are also optional (in this case, the default value is 00).

You can use these human-friendly constructions:

$ journalctl --- since yesterday
$ journalctl --since 09:00 --until now
$ journalctl --since 10:00 --until "1 hour ago"


Filtering by application and service



To view the logs of a specific application or service, use the -u option, for example:

$ journalctl -u nginx.service


The above command will print the nginx web server logs to the console.
Often there is a need to view the logs of a service for a certain period of time. This can be done using a command of the form:

$ journalctl -u nginx.service --since yesterday


With the -u option, filtering by date and time is also used, for example:

$ journalctl -u nginx.service -u php-fpm.service —since today


Thanks to this, you can track the interaction of various services and receive information that could not be obtained by tracking the respective processes separately.

Filtering by process, user and group



You can view the logs for a process by specifying its identification number (PID) in the journalctl command, for example:
$ journalctl _PID = 381


To view the logs of processes launched on behalf of a specific user or group, the _UID and _GID filters are used, respectively. Suppose we have a web server running on behalf of the www-data user. First we define the ID of this user:
$ id -u www-data
33

Now you can view the logs of all processes running on behalf of this user:

$ journalctl _UID = 33


You can display the list of users on the console about which there are entries in the logs, as follows:

$ journalctl -F _UID


To view a similar list of user groups, use the command:

$ journalctl -F _GUID


You can use other filters with the journalctl command. You can view the list of all available filters by running the command
$ man systemd.journal-fields


Filtering by path



You can also view the logs for a process by specifying the path to it, for example:

$ journalctl / usr / bin / docker


Sometimes in this way you can get more detailed information (for example, view records for all child processes).

View kernel messages



To view kernel messages, use the -k or −−dmesg option:

$ journalctl -k


The above command will show all kernel messages for the current boot. To view kernel messages for previous sessions, you need to use the -b option and specify one of the session identifiers (serial number in the list or ID):

$ journalctl -k -b -2


Filter messages by error level



During diagnostics and fixing problems in the system, it is often necessary to look through the logs and find out if they contain messages about critical errors. Specially for this, journalctl provides the ability to filter by error level. You can view messages about all errors that have occurred on the system using the -p option:

$ journalctl -p err -b


The above command will show all error messages that have occurred in the system.

These messages can be filtered by level. Journal uses the same classification of error levels as syslog:

  • 0 - EMERG (system is inoperative);
  • 1 - ALERT (immediate intervention required);
  • 2 - CRIT (critical condition);
  • 3 - ERR (error);
  • 4 - WARNING (warning);
  • 5 - NOTICE (everything is fine, but you should pay attention);
  • 6 - INFO (information message);
  • 7 —DEBUG (delayed printing).


Error level codes are specified after the -p option.

Logging to standard output



By default, journalctl uses the less external utility to display log messages. In this case, it is impossible to apply standard utilities for processing text data to them (for example, grep). This problem is easily solved: just use the −−no-pager option and all messages will be written to standard output:

$ journalctl --no-pager


After that, they can be transferred to other utilities for further processing or saved in a text file.

Select output format



Using the -o option, you can convert log data into various formats, which facilitates parsing and further processing, for example:

$ journalctl -u nginx.service -o json
{ "__CURSOR" : "s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635", "__REALTIME_TIMESTAMP" : "1422990364739502", "__MONOTONIC_TIMESTAMP" : "27200938", "_BOOT_ID" : "81b58db8fd9046ab9f847ddb82a2fa2d", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "3fffffffff", "_MACHINE_ID" : "752737531a9d1a9c1e3cb52a4ab967ee", "_HOSTNAME" : "desktop", "SYSLOG_FACILITY" : "3", "CODE_FILE" : "src/core/unit.c", "CODE_LINE" : "1402", "CODE_FUNCTION" : "unit_status_log_starting_stopping_reloading", "SYSLOG_IDENTIFIER" : "systemd", "MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5", "_TRANSPORT" : "journal", "_PID" : "1", "_COMM" : "systemd", "_EXE" : "/usr/lib/systemd/systemd", "_CMDLINE" : "/usr/lib/systemd/systemd", "_SYSTEMD_CGROUP" : "/", "UNIT" : "nginx.service", "MESSAGE" : "Starting A high performance web server and a reverse proxy server...", "_SOURCE_REALTIME_TIMESTAMP" : "1422990364737973" }


A json object can be represented in a more structured and human-readable form by specifying the json-pretty or json-sse format:

$ journalctl -u nginx.service -o json-pretty
{
    "__CURSOR": "s = 13a21661cf4948289c63075db6c25c00; i = 116f1; b = 81b58db8fd9046ab9f847ddb82a2fa2d; m = 19f0daa; t = 50e33c33587ae; x = e307daadb4858635
    "__REALTIME_TIMESTAMP": "1422990364739502",
    "__MONOTONIC_TIMESTAMP": "27200938",
    "_BOOT_ID": "81b58db8fd9046ab9f847ddb82a2fa2d",
    "PRIORITY": "6",
    "_UID": "0",
    "_GID": "0",
    "_CAP_EFFECTIVE": "3fffffffff",
    "_MACHINE_ID": "752737531a9d1a9c1e3cb52a4ab967ee",
    "_HOSTNAME": "desktop",
    "SYSLOG_FACILITY" : "3",
    "CODE_FILE" : "src/core/unit.c",
    "CODE_LINE" : "1402",
    "CODE_FUNCTION" : "unit_status_log_starting_stopping_reloading",
    "SYSLOG_IDENTIFIER" : "systemd",
    "MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5",
    "_TRANSPORT" : "journal",
    "_PID" : "1",
    "_COMM" : "systemd",
    "_EXE" : "/usr/lib/systemd/systemd",
    "_CMDLINE" : "/usr/lib/systemd/systemd",
    "_SYSTEMD_CGROUP" : "/",
    "UNIT" : "nginx.service",
    "MESSAGE" : "Starting A high performance web server and a reverse proxy server...",
    "_SOURCE_REALTIME_TIMESTAMP": "1422990364737973"
}


In addition to JSON, log data can be converted into the following formats:

  • cat - only messages from logs without service fields;
  • export - binary format, suitable for exporting or backing up logs;
  • short - syslog output format;
  • short-iso - syslog output format with timestamps in ISO 8601 format;
  • short-monotonic - syslog output format with labels of monotonic time (monotonic timestamp);
  • short-precise - syslog output format with timestamps (the time of events is indicated accurate to microseconds);
  • verbose - the most detailed format for presenting data (includes even those fields that are not displayed in other formats).


View recent events



The -n option is used to view information about recent events in the system:

$ journalctl -n


By default, the console displays information about the last 10 events. With the -n option, you can specify the required number of events:

$ journalctl -n 20


Viewing logs in real time



Messages from the logs can be viewed not only in the form of saved files, but also in real time. To do this, use the -f option:

$ journalctl -f 


Logging Management



Determining the current volume of logs



Over time, the volume of logs grows, and they take up more and more space on the hard drive. You can find out the volume of currently available logs using the command:

$ journalctl --disk-usage
Journals take up 16.0M on disk.


Log rotation



Log rotation is configured using the −−vacuum-size and −−vacuum-time options.
The first of them sets the maximum permissible size for logs stored on the disk (in our example - 1 GB):

$ sudo journalctl --vacuum-size = 1G


As soon as the volume of logs exceeds the specified number, excess files will be automatically deleted.
The −−vacuum-time option works similarly. It sets the storage period for logs, after which they will be automatically deleted:

$ sudo journalctl --vacuum-time = 1years


Setting log rotation in the configuration file



Log rotation settings can also be specified in the configuration file /еtc/systemd/journald.conf, which includes, among others, the following parameters:

  • SystemMaxUse = the maximum amount that logs can occupy on disk;
  • SystemKeepFree = the amount of free space that should remain on the disk after saving the logs;
  • SystemMaxFileSize = the size of the log file, upon reaching which it should be deleted from disk;
  • RuntimeMaxUse= максимальный объём, который логи могут занимать в файловой системе /run;
  • RuntimeKeepFree= объём свободного места, которое должно оставаться в файловой системе /run после сохранения логов;
  • RuntimeMaxFileSize= объём файла лога, по достижении которого он должен быть удален из файловой системы /run.


Централизованное хранение логов



One of the most common tasks in the work of the system administrator is to configure the collection of logs from several machines and then placed in a centralized repository.
Systemd has special components for this task: systemd-journal-remote , systemd-journal-upload, and  systemd-journal-gatewayd .

Using the systemd-journal-remote command, you can receive logs from remote hosts and save them (the systemd-journal-gatewayd daemon must be running on each of these hosts), for example:

$ systemd-journal-remote −−url https://some.host:19531/


As a result of the above command, the logs from the host some.host will be saved in the var / log / journal / some.host / remote-some ~ host.journal directory.

Using the systemd-journal-remote command, you can also add the logs on the local machine into a separate directory, for example:

$ journalctl -o export | systemd-journal-remote -o / tmp / dir -


The systemd-journal-upload command is used to upload logs from the local machine to the remote storage:

$ systemd-journal-upload --url https://some.host:19531/


As you can see from the above examples, the "native" systemd utilities to support centralized logging are simple and easy to use. But they, unfortunately, so far are not included in all distributions, but only in Fedora and ArchLinux.

Users of other distributions so far have to transfer logs to syslog or rsyslog, which then forward them over the network. Another solution to the centralized logging problem was proposed by the developers of journal2gelf utilityincluded in the official systemd repository: journalсtl output in JSON format is converted to GELF format and then passed to the application for collecting and analyzing Graylog logs. The solution is not very convenient, but nothing better can be invented in the current situation. It remains only to wait for the "native" components to be added to all distributions.

Conclusion



As can be seen from the discussion, systemd journal is a flexible and convenient tool for collecting system data and application data. A high level of flexibility and convenience was achieved, firstly, thanks to a centralized approach to logging, and secondly, thanks to the automatic recording of detailed metadata in the logs. With the help of journalctl, you can view the logs, getting the necessary information to analyze the work and debug various system components.
If you have questions or additions - welcome to comment. The discussion of systemd and its components will continue in future publications.

Readers who, for one reason or another, cannot leave comments here, are welcome to our blog .

Also popular now: