It seems that systemd is an apple of contention in the Linux community. As if there is no neutral point of view on systemd. Radically opposing opinions suggest that you must either love him or desire destruction. I want to offer some middle ground. First, discuss the nightmare properties of systemd.

Bad and nightmarish

systemd-escape

The fact that there is systemd-escape in itself clearly indicates something terribly wrong. If you have never seen or used this command in business, consider yourself lucky.

In practice, the command is used like this :

/bin/bash -c 'while true; do \
    /usr/bin/etcdctl set my-container \
        "{\"host\": \"1\", \"port\": $(/usr/bin/docker port my-container 5000 | cut -d":" -f2)}" \
    --ttl 60; \
    sleep 45; \
done'

To be honest, this generally sounds like a bad idea in general, but sometimes, if you write cloud-init for CoreOS, then this is your best option. I added the escape characters for the new line for better readability.

If we need to create a command ExecStartwith such contents, then systemd will not be able to recognize quotes, because there it does not work in the shell. Therefore, a command that normally runs in your shell will not work as a systemd unit. The simplest solution for systemd would be to implement something like shlexPython or ShellwordsRuby, but instead a patch was made as if from another world systemd-escape:

$ man systemd-escape | head
SYSTEMD-ESCAPE(1)                                            systemd-escape                                            SYSTEMD-ESCAPE(1)
NAME
       systemd-escape - Escape strings for usage in system unit names
SYNOPSIS
       systemd-escape [OPTIONS...] [STRING...]
DESCRIPTION
       systemd-escape may be used to escape strings for inclusion in systemd unit names.

Convert the script at the top to something acceptable for SystemD:

$ systemd-escape 'while true;do /usr/bin/etcdctl set my-container "{\"host\": \"1\", \"port\": $(/usr/bin/docker port my-container 5000 | cut -d":" -f2)}" --ttl 60;sleep 45;done'
while\x20true\x3bdo\x20-usr-bin-etcdctl\x20set\x20my\x2dcontainer\x20\x22\x7b\x5c\x22host\x5c\x22:\x20\x5c\x221\x5c\x22\x2c\x20\x5c\x22port\x5c\x22:\x20\x24\x28-usr-bin-docker\x20port\x20my\x2dcontainer\x205000\x20\x7c\x20cut\x20\x2dd\x22:\x22\x20\x2df2\x29\x7d\x22\x20\x2d\x2dttl\x2060\x3bsleep\x2045\x3bdone

Agree that if you need to implement a while loop from Bash for the unit to work, then you are already in a bad position, but sometimes it is required for templates.

Binary logs

If you are not up to date, journald stores its logs in binary format . It breaks down the usual tools that we used to use for system monitoring: tail, cat, lessand grepbecome useless. With the binary format, damage to the logs is also possible. If the text logs accidentally falls into the binary content, like most editors vimand lesselegantly handle it. If binary content gets into the wrong place of binary logs, your logs are lost.

The rationale for storing logs in binary format was speed and performance, they are easier to index and faster to search. However, this was clearly a difficult choice with obvious consequences for end users on both sides of the barricades. If you need fast logs / logging, you can get them, but users will have to learn a new command journalctland they will not be able to use familiar tools.

Binary logs do not seem to me something bad, but this is another barrier to the adoption of systemd. I will review the logs later in this article and present arguments for my point of view, which journaldis a good idea.

Good

Now let's pay attention to the advantages that systemd gives us. I think that they became the reason why all Linux distributions implemented systemd.

Sanity

Let's start by comparing the SysV init script for ZooKeeper , which takes 169 lines of fragile code, as noted in the comments throughout its source.

# for some reason these two options are necessary on jdk6 on Ubuntu
#   accord to the docs they are not necessary, but otw jconsole cannot
#   do a local attach
...

We implement it as a systemd unit:

[Unit]
Description=ZooKeeper
[Service]
Type=simple
Restart=always
RestartSec=5
EnvironmentFile=/etc/sysconfig/zookeeper
ExecStart=/usr/bin/java -cp ${ZK_CLASSPATH} ${JVM_FLAGS} org.apache.zookeeper.server.quorum.QuorumPeerMain ${ZOO_CFG_FILE}
[Install]
WantedBy=multi-user.target

I wrote it in less than ten minutes. Of course, he needs an environment file that sets the following variables:

ZK_CLASSPATH=/opt/zookeeper:/opt/zookeeper/lib:/etc/zookeeper
JVM_FLAGS=-Xmx=2g
ZOO_CFG_FILE=/etc/zookeeper/zoo.cfg

But ... that's all . It is done.

If this process is simply logged into standard output and standard errors, then the logs will be saved in the log, they can be tracked, indexed, searched and exported with syslog-ngor rsyslog. Below I will consider how logs are recorded.

Process control

In the old days, we used something like supervisord to track that processes were running. This is because before the appearance of systemd, if you do not write this, then it will not happen . Do not think that init scripts after starting system services will actually monitor these processes, because this was not the case. Services could fall out with a segfault error and stop until the user manually starts them again.

And now systemd:

[Service]
...
Restart=always
RestartSec=1

This tells the system that if there is a failure in the process, then wait one second and always restart it. If you specify stopfor a service, then it stops until you run it again, as expected from this command. In addition, systemd logs when and why the process crashed, so finding problems becomes simple and trivial.

Process planning

Coming back from the dark times of SysV init scripts, what were our options for starting a service after another service? Or further, what were the options for starting service A after service B, but before service C? The best option was this:

while true ; do
  if pgrep serviceB ; then
    start_service
    break
  else
    sleep 1
  fi
done

To start service A before service C, you had to change the init script of service C and add a similar while loop to detect and wait for service A. There is no need to say that this is a disaster.

And now systemd:

[Unit]
Description=Service A
After=serviceB.service
Before=serviceC.service

And that’s all . Nothing more to do. systemd will create a service dependency graph and run them in the correct order, and you are guaranteed serviceAto start after serviceB, but before serviceC. What's even better is drop-in units, which I'll talk about shortly. In short, it’s easy to include additional code in the source unit without having to rewrite the code of the source unit.

Bonus: conditions for launching units

systemd also makes it easy to set conditions for starting units :

[Unit]
Description=Service A
ConditionPathExists=/etc/sysconfig/serviceA

Here, Service A will only start if a file is present /etc/sysconfig/serviceA. There are many different conditions, and all of them can be inverted.

Bonus: Concurrency

Since systemd knows the dependency order for all of its units, loading a Linux machine using systemd is much faster than on older init systems. This is because systemd runs services without dependencies in parallel.

Unit overload

As discussed above, systemd makes it very easy to load additional configurations for a given unit, expanding it. Say we only need to run rsyslogafter launch cloud-final. In turn, this cloud-finalis the last stage of work cloud-init.

The source file for rsyslog.serviceis in /usr/lib/systemd/system/rsyslog.service, but we will not edit this file. We will create a drop-in unit in /etc/systemd/system/rsyslog.service.d/after-cloudinit.conf:

[Unit]
After=cloud-final.service

The final file name is not very important as it ends with .conf. Whatever this file defines, it will be included in the original unit file. This little drop-in verifies that rsyslog does not start until cloud-final.serviceit starts / ends.

Addition: I was told on twitter that systemd runs these files in alphabetical order. To maintain order in the chaos, probably a good idea to add them to the numerical prefixes, then there will be intelligible order, for example %02d-%s.conf.

Rewriting units

What if we need to completely remove certain parts from the unit? Removing is simple:

[Service]
ExecStartPre=

What we did in this unit to overwrite was to remove all blocks ExecStartPrefrom the original unit. If we add another line ExecStartPreunder an empty line, we can specify our own script for preliminary launch, which is completely different from the previous one.

Logging

Logs with systemd are incredibly simple and support all the features and buns you want. If the process is simply logged to standard output and standard errors, then the log is logged by default. Searching for these logs is then trivial:

$ sudo journalctl -u rsyslog.service

This will launch a type browser lessto scan history rsyslog.service. Log tracking for a unit is also simple:

$ sudo journalctl -u rsyslog.service -f

This is literally the equivalent of a command tail -ffor the log.

Logs automatically rotate in the log and this can be configured, so there’s no more nonsense logrotate, the log just takes care of everything.

Connection is rsyslogeither syslog-ngdone simply, and this means that none of your applications need to talk to syslog anymore, their standard output goes to the log immediately and will be imported and sent in accordance with the syslog configuration.

Go ahead and learn

We have reviewed a lot of the main places here. Personally, I have bookmarked the following documentation fragments for systemd that help me write units:

• systemd.exec
• systemd.service
• systemd.unit

We did not even talk about the excellent mount points , timers, and many of the security features that systemd supports. I also did not mention tools from user space systemctland journalctlwhich are described here:

• systemctl
• journalctl

Definitely, I was in the “systemd crap” camp for a long time, until I began to study what systemd really allows. Now I see systemd as an integral part of my infrastructure at the system level, and it would be harder and harder to implement on older distributions without systemd.