Theory and practice of unattended upgrades in Ubuntu

Unattended upgrades is the native mechanism for automatic updates for Debian / Ubuntu (and other GNU / Linux distributions based on them). By default, it is enabled in the system due to the installed package unattended-upgradesand configuration file /etc/apt/apt.conf.d/50unattended-upgrades, and it is configured to update packages only from the security repository , which includes, for example, critical fixes for the libssl package that come out as a result of the next replenishment of the CVE vulnerability database.


Note : hereafter, unattended upgrades are considered in the context of Ubuntu server editions, which is most likely applicable “as is” to other distributions, but there may be some peculiarities that remain outside the scope of the article.

So, what additional features do unattended upgrades provide (besides the security updates included by default) and what problems can they lead to?

The word "unattended" in the name of this mechanism is really important in its literal translation - "unattended." Why is that? It’s enough to recall that packages generate files during installation.dpkg-new, in which the new config for the package appears (if the checksum of the config in the installed package differs from the checksum of the config in the system). This circumstance should be taken into account, because otherwise you can get a new version of software that no longer works with the option you added to the config , and after installing the package, the service / application simply will not start. Therefore, when collecting or borrowing a package into your repository, remember that installing such a package will not update the configs itself, so, for example, if the version configs are not compatible (which is relevant in case of a more significant update than the security fix) , it can turn out to be very unpleasant a situation where everyone has been sleeping for a long time, and your package rolled out on a heap of servers and "Everything has broken, boss!".

Speaking of unattended upgrades package response times: checking for updates and installing them on Ubuntu / Debian systems is defined in /etc/cron.daily/apt. The file is launched from /etc/crontab, in which early morning is set by default (06:25).

Repository / PPA Application

Let's move on to practice. We had the following problem: one package was installed on all servers, but not in its native version, but with certain modifications. What to do? The obvious options are:

• “Let's curse for a few hours one of the trainees and let him roll out!” - Perhaps this will turn out to be useful for the trainee, but only at the stage of training to work with the system and provided that he does not know how to work apt at all. Further it will really turn into a curse. In addition, the result will be more dependent on the human factor than we would like.
  
  
• “Roll out everywhere with Chef / Puppet / Ansible / ...!” - a great idea, but not our case at that time and in that situation. For the sake of one package, you would have to defeat the dragon, i.e. enter many machines into the selected configuration management system.
  
  
• Since the article is devoted to unattended upgrades, it is easy to guess that it is this mechanism that offers a different way to solve the problem without spending a lot of human hours on it ...

Indeed: the configuration of unattended upgrades allows you to activate the automatic receipt of all updates of a package for the selected repository - for example, your own or PPA, which you have good reason to trust. In order for all this auto-operation to work at a minimum level, it is enough to create a config on the target machine /etc/apt/apt.conf.d/51unattended-upgrades-custom, in which there will be only three lines:

Unattended-Upgrade::Allowed-Origins {
  "Origin:Suite";
};

If you have your own repository, the word Originand Suiteshould at least evoke from the category of "Somewhere I have seen it ...." I will tell you that such parameters can be seen in the repository on the machine itself, where the package should be updated, in the file *_InRelease. Illustration for the well-known ppa:nginx/stable:

$ head -10 /var/lib/apt/lists/ppa.launchpad.net_nginx_stable_ubuntu_dists_trusty_InRelease 
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Origin: LP-PPA-nginx-stable
Label: NGINX Stable
Suite: trusty
Version: 14.04
Codename: trusty
Date: Sat, 11 Feb 2017 21:55:33 UTC
Architectures: amd64 arm64 armhf i386 powerpc ppc64el

We see two parameters:

• Origin - “origin” of the repository, which may indicate the name of the maintainer or the repository itself;
• Suite- distribution branch; e.g. stable, testing for Debian or trusty, xenial for Ubuntu.

Note : For more detailed documentation on Release / InRelease files and the parameters used in them, see wiki.debian.org .

Thus, if you have your own package repository, you need to add these parameters. As a result, for this PPA example, anginx/stableconfiguration that allows unattended upgrades for all packages from it will look like this:

Unattended-Upgrade::Allowed-Origins {
  "LP-PPA-nginx-stable:trusty";
};

It is logical to automate these settings (by the method that is closest to you and / or already used), creating all the necessary configuration files when deploying new OS installations. And you can check how the next launch of unattended upgrades behaves, as follows:

$ unattended-upgrade -v --dry-run

The flags are simple: -v- to be more verbose, and --dry-run- not to apply changes. With a trial run, we will immediately see that this solution may have downsides :

• If installing a package requires interactivity, i.e. user intervention (especially relevant if you are doing a large-scale system update because you haven’t done this for a long time) - unattended upgrades simply won’t do anything.
  
  
• The example with automatic updating of nginx from PPA should not be checked in real life production, unless you want to pass the name “dirty Harry” for package updates. Once again, remember the name of the utility and remember that you can only update that which really does not require any supervision . For example, libraries (although even here there are pitfalls, if you recall at least recent Ubuntu dances with libc and a broken DNS resolver ) and software that does not require configuration and runs on demand (atop, htop and the like).

Another “reverse” example, which does not even apply to the addition of specific PPAs in unattended upgrades, but which we have come across more than once in practice, is security updates for PostgreSQL. In the standard Ubuntu Server configuration (i.e., installing only security updates through unattended upgrades), automatic updating of critical vulnerabilities in this DBMS led to its restart at a time when not everyone expected it. If this behavior may be unexpected (undesirable) for your case, see the option Package-Blacklistbelow.

Additional features

You /etc/apt/apt.conf.d/50unattended-upgradescan see what else unattended upgrades can do. There are not many settings, but some of them are useful:

• Package-Blacklist- A list of packages that are not allowed to be updated in this way. Here, in the example, we immediately offer to do this for libc, and another example is described above - with PostgreSQL. But remember, on the other side of the scale: postponing critical corrections, you risk security .
• AutoFixInterruptedDpkg- if the last installation / upgrade process could not be completed for any reason, you probably had to fix the situation manually. This option does the same, i.e. causes dpkg --force-confold --configure -a. Please note that the option --force-confoldis indicated here - it means that old versions of configs will be saved if conflicts arise.
• MinimalSteps- Perform updates as little as possible. Allows you to abort the update by sending SIGUSR1 to the unattended-upgrade process.
• InstallOnShutdown- Install updates before turning off the computer. Personally, it seems to me a bad idea, because I would not want to get a corpse after a scheduled server reboot.
• Mailand MailOnlyOnError- to whom to send letters about updates and / or problems with them. Emails are sent through the standard MTA sendmail (environment variable is used SENDMAIL_BINARY). Unfortunately, only letters, and it is impossible to execute curl to some API here.
• Automatic-Reboot- restart automatically after installation if there is a file /var/run/reboot-required. The file itself appears, for example, after installing the Linux kernel package when the rule is triggered /etc/kernel/postinst.d/update-notifier. In general, another option from the set of "dirty Harry."
• Automatic-Reboot-Time - if you want to do your dark things at night, while no one sees ... sets a specific time for automatic reboot.
• Acquire::http::Dl-Limit- this is from the general set of apt parameters. Limits the speed of downloading updates so as not to clog the channel.

conclusions

Unattended upgrades is an automatic update installation tool built into Debian and Ubuntu-based distributions. It is usually used to install security updates from the corresponding repository, but it is easy to extend the application to any other repositories. The tool will be useful for supporting easy-to-install and update programs and scripts that are packaged in packages - for implementation only the repository itself and a few lines in the config on the updated machine are required.

But remember that the simplicity of the tool does not mean that you will not hit it on the fingers: do not configure automatic updates of complex or critical services if you are not 100% sure of the safety of this action and this will not affect the duration of sleep for you and yours colleagues.