June 23, 2016 at 14:35Running service workers with systemd
- Tutorial
Following the release of Ubuntu 16.04 (the new LTS release), systemd became the reality of all the major Linux distributions used on servers. This means that you can lay on the advanced features of systemd, without risking to leave part of the application users “overboard”.
This post is about how to implement a multi-negotiated application using systemd.
Abstract: Using service templates and target'ov to run multiple service instances (implementation of "workers"). PartOf dependency. A little about the [install] section of units.
Many programming languages with poor or no multithreading (Python, Ruby, PHP, quite often C / C ++) use the concept of "worker". Instead of framing the complex relationships between threads within the application, they launch several single-threaded copies of the application, each of which takes on a piece of the load. Thanks to the SO_REUSEPORT option, it is even possible to listen together on the same port, which covers most of the tasks that require workers (in fact, ordinary server applications that implement APIs or serve a website).
But this approach requires a “supervisor”, which is responsible for starting copies, monitors their status, processes errors, finishes with all kinds of stop / reload, etc. With the seeming triviality, this is absolutely not a trivial task, full of nuances (for example, if one of the workers got into TASK_UNINTERRUPTIBLE or received SIGSTOP, then there may be problems when restarting with a not very well written parent).
There is an option to start without a supervisor, but in this case, the reload / restart task is transferred to the administrator. With the “one process per core” model, restarting a service on a 24-core server becomes a candidate for automation, which in turn requires processing all of the same SIGSTOP and other complex nuances.
One solution to the problem is to use systemd service templates along with a dependency on the general target.
systemd supports “templates” for launching services. These templates accept a parameter that can then be inserted anywhere in the command line arguments (man systemd.service). The parameter is passed through the '@' symbol in the service name. The part after '@' (but before the point) is called 'instance name', encoded by% i or% I. A complete list of parameters is www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers . The presence of '@' in the service name (before the dot) indicates that this is a template.
Let's try to write the simplest template:
/etc/systemd/system/foobar-worker@.service
And run a few of these:
We look:
Now we want to somehow start all of them in a common way. There are targets for this.
Target is a systemd unit that does nothing, but can be used as a dependency element (target can depend on several services, or services can depend on target, which also depends on services).
target's have the extension .target.
We will write our simplest target:
vim /etc/systemd/system/foobar.target
(attention to .service, it is necessary!)
About 'Wants' we will talk a little lower.
Now we can run all three foobar-
workers at once: systemctl start foobar.target
(attention to target - in the case of .service it can be omitted, in the case of .target - no).
Three sleeps appeared in the process list. Unfortunately, if we make systemctl stop foobar.target, they will not disappear, i.e. on the "workers" they are little similar. We need to somehow combine target and workers into a single whole. For this we will use the dependencies.
Systemd provides an extensive set of dependencies to describe exactly what we want. We are interested in 'PartOf' from this list. Before that, we used wants.
Compare their behavior:
Wants (which we used) - the mentioned service tries to start if the main unit starts. If the mentioned service crashes or cannot start, this does not affect the main service. If the main service shuts down / restarts, then the services mentioned in the dependency remain unaffected.
PartOf - If the aforementioned shuts down / restarts, then the main service also shuts down / restarts.
Just what we need.
Add the dependency to the description of the worker:
This post is about how to implement a multi-negotiated application using systemd.
Abstract: Using service templates and target'ov to run multiple service instances (implementation of "workers"). PartOf dependency. A little about the [install] section of units.
Introduction
Many programming languages with poor or no multithreading (Python, Ruby, PHP, quite often C / C ++) use the concept of "worker". Instead of framing the complex relationships between threads within the application, they launch several single-threaded copies of the application, each of which takes on a piece of the load. Thanks to the SO_REUSEPORT option, it is even possible to listen together on the same port, which covers most of the tasks that require workers (in fact, ordinary server applications that implement APIs or serve a website).
But this approach requires a “supervisor”, which is responsible for starting copies, monitors their status, processes errors, finishes with all kinds of stop / reload, etc. With the seeming triviality, this is absolutely not a trivial task, full of nuances (for example, if one of the workers got into TASK_UNINTERRUPTIBLE or received SIGSTOP, then there may be problems when restarting with a not very well written parent).
There is an option to start without a supervisor, but in this case, the reload / restart task is transferred to the administrator. With the “one process per core” model, restarting a service on a 24-core server becomes a candidate for automation, which in turn requires processing all of the same SIGSTOP and other complex nuances.
One solution to the problem is to use systemd service templates along with a dependency on the general target.
Theory
Patterns
systemd supports “templates” for launching services. These templates accept a parameter that can then be inserted anywhere in the command line arguments (man systemd.service). The parameter is passed through the '@' symbol in the service name. The part after '@' (but before the point) is called 'instance name', encoded by% i or% I. A complete list of parameters is www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers . The presence of '@' in the service name (before the dot) indicates that this is a template.
Let's try to write the simplest template:
/etc/systemd/system/foobar-worker@.service
[Unit] Description = Foobar number% I [Service] Type = simple ExecStart = / bin / sleep 3600% I
And run a few of these:
systemctl start foobar-worker @ 1 systemctl start foobar-worker @ 2 systemctl start foobar-worker @ 300
We look:
ps aux | grep sleep root 13313 0.0 0.0 8516 748? Ss 17:29 0:00 / bin / sleep 3600 1 root 13317 0.0 0.0 8516 804? Ss 17:29 0:00 / bin / sleep 3600 2 root 13321 0.0 0.0 8516 764? Ss 17:29 0:00 / bin / sleep 3600 300
Now we want to somehow start all of them in a common way. There are targets for this.
Target
Target is a systemd unit that does nothing, but can be used as a dependency element (target can depend on several services, or services can depend on target, which also depends on services).
target's have the extension .target.
We will write our simplest target:
vim /etc/systemd/system/foobar.target
[Unit] Wants=foobar-worker@1.service foobar-worker@2.service Wants=foobar-worker@300.service
(attention to .service, it is necessary!)
About 'Wants' we will talk a little lower.
Now we can run all three foobar-
workers at once: systemctl start foobar.target
(attention to target - in the case of .service it can be omitted, in the case of .target - no).
Three sleeps appeared in the process list. Unfortunately, if we make systemctl stop foobar.target, they will not disappear, i.e. on the "workers" they are little similar. We need to somehow combine target and workers into a single whole. For this we will use the dependencies.
Dependencies
Systemd provides an extensive set of dependencies to describe exactly what we want. We are interested in 'PartOf' from this list. Before that, we used wants.
Compare their behavior:
Wants (which we used) - the mentioned service tries to start if the main unit starts. If the mentioned service crashes or cannot start, this does not affect the main service. If the main service shuts down / restarts, then the services mentioned in the dependency remain unaffected.
PartOf - If the aforementioned shuts down / restarts, then the main service also shuts down / restarts.
Just what we need.
Add the dependency to the description of the worker:
[Unit] Description = Foobar number% I PartOf = foobar.target [Service] Type = simple ExecStart = / bin / sleep 3600% I
All. If we make systemd stop foobar.target, then all our workers will stop.Install Dependencies
Another interesting feature of systemd is install dependencies. Sysv-init had the ability to enable / disable services, but it was very difficult to explain how enable should be done there. What runlevels? What dependencies?
In systemd, everything is simple. When we use the 'enable' command, the service is “added” (via the slice mechanism) depending on what we specified in the [install] section. For our convenience, there is a dependency WantedBy, which in meaning is the opposite of Wanted.
There are tons of standard targets we can cling to. Here are some of them (all - man systemd.special):
* multi-user.target (standard for "need to start", the equivalent of the final runlevel for sysv-init).
* default.target - alias for multi-user
* graphical.
Let's hook onto multi-user.target.
New content foobar.target:[Unit] Wants=foobar-worker@1.service foobar-worker@2.service Wants=foobar-worker@300.service [install] WantedBy = multi-user.target
Now, if we do enable it:# systemctl enable foobar.target Created symlink /etc/systemd/system/multi-user.target.wants/foobar.target → /etc/systemd/system/foobar.target.
Everything, our service, cobbled together from several workers, is ready to start / restart as a whole, plus it will be launched at the start of our computer / server.