Centralized Let's Encrypt Certificate Update

Hello!

In this article, I will describe how we solved the problem of centrally updating Let's Encrypt certificates and managing infrastructure using ansible.

In our solution we will use:

• ansible
• rsync, rsyncd
• inotify, incron
• certbot
• nginx

I will offer two architecture options in which our solution can be useful. In turn, you can suggest your options in the comments.

Option 1: You have several frontend servers with public ip (for example 3) serving several domains. These domains can be added / removed. In order not to monitor each of the frontend servers, it is more convenient to do this on one letsencrypt server'e:

Option 2: You have only one server with public ip, and you need certificates on servers within the network:

Ansible Role Description

A repository with roles is available here .

There are 4 roles in the repository:

• nginx-simple
  Installs on all hosts nginxand copies basic configs. The role itself does not start from the playbook. It runs on meta dependencies from other roles.
• letsencrypt-server
  Configures rsyncdon the host letsencrypt server. In meta dependencies has a role nginx-simple. Accordingly, it will first be installed nginx, and then the role will be lost letsencrypt server.
• incron
  Installs the necessary packages for incronand copies the base config. The role also does not start directly as well nginx-simple.
• front
  In meta dependencies it has roles incronand nginx-simple. After them, the role frontcopies the configs necessary for example.com nginx, adds the task to cronpick up new certificates with letsencrypt serverand the task to incroncheck for file changes and perform a hooknginx -s reload

Let's move on to practice:

Initial configuration

Initially, we have:

• 1 server for centrally issuing an SSL certificate ( letsencrypt server)
• 1 or more publicly accessible servers with nginx ( front)

Ubuntu 16.04 is installed on all servers.

Install and configure nginx

First, install nginx on all hosts from a common role nginx-simpleand distribute nginx configs common to all hosts (nginx.conf, ssl parameters, certificate paths, etc).

For letsencrypt server in the template, the .../site-available/default.conffolder .well_knownwill be in /var/www/:

{% if letsencrypt_server %}
  location /.well-known {
    root /var/www/;
  }

For the server / servers of the group front, since the folder is .well_knownused not only to obtain certificates, but also for other software, we import into the config example.confletsencrypt-proxy.confand nginx will search for the folder locally on the front server using the directive try_file:

{% if nginx_proxy_well_known %}
  try_files $uri $uri/ @letsencrypt;
{% endif %}

Configs for the domain will be filled depending on the variables from inventory. In the repository, this is the example.com domain

Also, depending on the variable letsencrypt_server, the role will be nginx-simpleinstalled on letsencrypt server certbotand will add a cron task to renew certificates.

Obtaining a Certificate

Since we solved this problem before the appearance of a wildcard certificate from Let's Encrypt, we will consider both options for obtaining a certificate.

On the letsencrypt server server, do the following:

certbot certonly --agree-tos -d example.ru --webroot -w /var/www/

If there are more than one domain, we add the next ones using the key -d.

To get a wildcard certificate, we will need to add DNS TXT records. At the moment, this is the only way to obtain such a certificate:

certbot certonly --agree-tos -d example.ru -d *.example.ru --preferred-challenges dns --manual --server https://acme-v02.api.letsencrypt.org/directory

Certbot will write which TXT records you will need to add.

Certificate renewal

We received the certificates, it remains to configure copying them to the front server / server. To do this, we will configure rsyncdletsencrypt with read permissions on a limited list of ip addresses:

hosts allow = {{ hosts_allow }}
hosts deny = *
list = true
use chroot = no
[cert]
path = /etc/letsencrypt/live/
uid = root
gid = root
read only = true

Every 5 minutes, the cron task from the front servers will check if the certificates have been updated and pick them up. Because certificates are rotated, /etc/letsencrypt/live/{{ domain }}symlinks are in the folder . Add a key -Lto get the original files:

/usr/bin/rsync -zavL --chmod=D0750,F640 --delete rsync://{{ hostvars['letsencrypt-server'].ansible_eth0.ipv4.address }}/cert /etc/letsencrypt/live/

Hook for nginx

We configured nginx, received certificates, took them to the front server. It remains to determine that the files in the folder /etc/letsencrypt/live/{{ domain }}have changed and execute a hooknginx -s reload

The linux kernel subsystem inotifyand the daemon will help us with this incron. You can read more about them here .

The role will incroninstall the necessary packages, and from the role template, the fronttask of monitoring certificates and the necessary hook will be added:

/etc/letsencrypt/live/{{ domain }}/ IN_CREATE,IN_DELETE,IN_MODIFY,IN_MOVED_TO nginx -s reload

Finally

We tried to describe in detail the entire installation and configuration process. everything is described in ansible playbooks - the article turned out to be very compact. As they often like to say, “a little more than 100 lines of code”. We are happy to answer questions, criticism and comments in the comments.