March 17, 2019 at 13:22How to make friends Ovirt and Let's Encrypt
- Tutorial
Walking along the path of improving the infrastructure, I decided to finish off the ancient and painful question - without unnecessary gestures, provide colleagues (developers, testers, admins, etc) with the opportunity to independently manage their virtual machines in ovirt. There are several components in ovirt that I need to configure to solve my question: the web interface itself, the noVNC console, and filling in disk images.
I didn’t find the buttons “Make Flicker”, so I show which pens I turned to solve this problem. Full instructions under the cut:
Before starting, I would like to draw attention to the fact that for some reason unknown to me, infrastructure domains are created in private zones lan, local, and so on.
What prevents me from using the organization’s domain in the public zone is unknown to me. For example, instead of the domain Alex-GLuck-Awesome-Company.local, you can safely use the domain for the site of the company Alex-GLuck-Awesome-Company.com.
If you are afraid that you will not be able to keep track of the domains in your organization, and this will break something, then for a modest 100 rubles a year you can take a separate domain for the aglac.com infrastructure.
Why it is more profitable to use domains in public zones:
1. Services appearing in your public space within your organization: VPN, file sharing (seafile, nextcloud) and others. Configuring traffic encryption on such services usually looks like a blooper, and we will not protect ourselves from MitM, because it is difficult (actually not).
Or inside the office you have one service address, and another from the Internet, and these communications must be maintained, which our limited specialist resources are spent on. Well, employees have to remember different addresses, which is inconvenient.
2. You can use free certificate authorities to encrypt your internal services.
Own PKI is a service that needs to be supported, 100 rubles a year for the opportunity to use PKI from free certificate authorities more than pay for the time of employees who could spend it on other tasks.
3. When using your own certification authority, you will put sticks in the wheels of your remote employees and colleagues who want to work with BYOD (bring their laptops, phone, tablets) and you cannot control their devices. They bring poppies, Linux, androids, iOS, Windows - there is no point in supporting such a zoo.
In all, of course, there are exceptions, and banks with other harsh enterprises that have established security policies will never be able to improve the service for their employees.
There are paid certification authorities for them, which for a certain amount can sign their CA certificate (google “root signing service”).
There are other reasons why it is more profitable to use a public domain (the most important thing is that it belongs to you), but the article is not about that.
ATTENTION! If you add Let's Encrypt's CA certificate to the ovirt’s trusted list, this may affect the security of your systems!
The first thing you need to pay attention to is that putting the interfaces of the Internet on the Internet is a bad practice, because this makes no practical sense, and creates additional security threats.
Therefore, you need to get a certificate on some of our bastion host, and then transfer the certificate and key to our host with ovirt-engine.
We add the external address of our bastion host in the DNS with our ovirtengine.example.com ovirt name , I will leave the installation of certbot and nginx behind the scenes (how to do this on the hub has already been described).
Configuring Nginx version> = 1.15.7
Then we get our certificate and key:
We archive our certificate and key:
Download the archive from the bastion host, upload it to our ovirt -engine:
Next, we unpack our archive and create symlinks to simplify understanding of the file location system:
We configure the built-in pki in the ovirt to use the java certificate store (openjdk) for certificate verification:
We convert the CA from let's encrypt into der format and add an ovirt to the java trust store certificate store (this is such a container where the list of certificates is located, such a system is used in java):
We edit the SSL settings for apache, add the parameter to support symlinks and remove the parameter for CA, which will check certificates (by default, the system will use the trusted CA set for verification):
After that, just in case, we will backup the original files generated through PKI ovirt automatically and replace the symlinks with files from Let's Encrypt:
We restore SElinux contexts on files and restart our services (httpd, ovirt-engine, ovirt-imageio-proxy, ovirt-websocket-proxy):
httpd - web server apache
ovirt-engine - web interface ovirt
ovirt-imageio-proxy - daemon for loading disk images
ovirt-websocket-proxy - service for running noVNC console
All of the above has been checked on version 4.2 of the ovirt.
According to good security practices, there should not be a connection between the bastion host and the ovirt, and the certificate is issued only for 3 months. This is where a controversial point appears about how I have implemented certificate renewal.
I have an ensemble playbook that runs on foreman daily at 5 a.m. on a schedule. This playbook goes into the ovirt, checks the certificate validity period and if less than 5 days are left before the expiration, it goes to the bastion host and starts the certificate renewal.
After updating the certificate, it archives the folder with the files, downloads the forman to the host, and unzips the ovirt host. Then it restores the SElinux contexts on the files and restarts our services.
I didn’t find the buttons “Make Flicker”, so I show which pens I turned to solve this problem. Full instructions under the cut:
DISCLAIMER:
Before starting, I would like to draw attention to the fact that for some reason unknown to me, infrastructure domains are created in private zones lan, local, and so on.
What prevents me from using the organization’s domain in the public zone is unknown to me. For example, instead of the domain Alex-GLuck-Awesome-Company.local, you can safely use the domain for the site of the company Alex-GLuck-Awesome-Company.com.
If you are afraid that you will not be able to keep track of the domains in your organization, and this will break something, then for a modest 100 rubles a year you can take a separate domain for the aglac.com infrastructure.
Why it is more profitable to use domains in public zones:
1. Services appearing in your public space within your organization: VPN, file sharing (seafile, nextcloud) and others. Configuring traffic encryption on such services usually looks like a blooper, and we will not protect ourselves from MitM, because it is difficult (actually not).
Or inside the office you have one service address, and another from the Internet, and these communications must be maintained, which our limited specialist resources are spent on. Well, employees have to remember different addresses, which is inconvenient.
2. You can use free certificate authorities to encrypt your internal services.
Own PKI is a service that needs to be supported, 100 rubles a year for the opportunity to use PKI from free certificate authorities more than pay for the time of employees who could spend it on other tasks.
3. When using your own certification authority, you will put sticks in the wheels of your remote employees and colleagues who want to work with BYOD (bring their laptops, phone, tablets) and you cannot control their devices. They bring poppies, Linux, androids, iOS, Windows - there is no point in supporting such a zoo.
In all, of course, there are exceptions, and banks with other harsh enterprises that have established security policies will never be able to improve the service for their employees.
There are paid certification authorities for them, which for a certain amount can sign their CA certificate (google “root signing service”).
There are other reasons why it is more profitable to use a public domain (the most important thing is that it belongs to you), but the article is not about that.
The bottom line, but the point ...
ATTENTION! If you add Let's Encrypt's CA certificate to the ovirt’s trusted list, this may affect the security of your systems!
The first thing you need to pay attention to is that putting the interfaces of the Internet on the Internet is a bad practice, because this makes no practical sense, and creates additional security threats.
Therefore, you need to get a certificate on some of our bastion host, and then transfer the certificate and key to our host with ovirt-engine.
We add the external address of our bastion host in the DNS with our ovirtengine.example.com ovirt name , I will leave the installation of certbot and nginx behind the scenes (how to do this on the hub has already been described).
Configuring Nginx version> = 1.15.7
/etc/nginx/conf.d/default.conf
server {
server_name _;
listen 80 default_server;
location /robots.txt { alias /usr/share/nginx/html/robots.txt; }
location /.well-known {
root /usr/share/nginx/html;
}
location / {
return 444;
}
}
server {
server_name _;
listen 443 ssl http2 default_server;
location /robots.txt { alias /usr/share/nginx/html/robots.txt; }
location /.well-known {
root /usr/share/nginx/html;
}
ssl_certificate /etc/nginx/ssl/$ssl_server_name/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/$ssl_server_name/privkey.pem;
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
# позволяем серверу прикреплять OCSP-ответы, тем самым уменьшая время загрузки страниц у пользователей
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=15768000;
location / {
return 444;
}
}
Then we get our certificate and key:
certbot certonly --nginx -d ovirtengine.example.com
We archive our certificate and key:
tar Phczf /tmp/ovirtengine.example.com.tgz /etc/letsencrypt/live/ovirtengine.example.com
Download the archive from the bastion host, upload it to our ovirt -engine:
scp bastion-host:/tmp/ovirtengine.example.com.tgz /tmp/
scp /tmp/ovirtengine.example.com.tgz ovirtengine.example.com:/
Go to the goal
Next, we unpack our archive and create symlinks to simplify understanding of the file location system:
tar Pxzf /ovirtengine.example.com.tgz && rm -f ovirtengine.example.com.tgz
mkdir -p /etc/letsencrypt/live
ln -f -s /etc/letsencrypt/live /etc/pki/letsencrypt
We configure the built-in pki in the ovirt to use the java certificate store (openjdk) for certificate verification:
cat << EOF > /etc/ovirt-engine/engine.conf.d/99-setup-pki.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts"
ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
EOF
We convert the CA from let's encrypt into der format and add an ovirt to the java trust store certificate store (this is such a container where the list of certificates is located, such a system is used in java):
openssl x509 -outform der -in /etc/pki/letsencrypt/ovirtengine.example.com/chain.pem -out /tmp/ovirtengine.example.com.chain.der
keytool -import -alias "Let's Encrypt Authority X3" -file /tmp/ovirtengine.example.com.chain.der -keystore /etc/pki/ovirt-engine/.truststore -storepass $(grep '^ENGINE_PKI_TRUST_STORE_PASSWORD' /etc/ovirt-engine/engine.conf.d/10-setup-pki.conf | cut -f 2 -d '"')
rm -f /tmp/ovirtengine.example.com.chain.der
We edit the SSL settings for apache, add the parameter to support symlinks and remove the parameter for CA, which will check certificates (by default, the system will use the trusted CA set for verification):
sed -r -i 's|^(SSLCACertificateFile.*)|#\1|g' /etc/httpd/conf.d/ssl.conf
sed -r -i '0,/(^#?SSLCACertificateFile.*)/ s//\1\nOptions FollowSymlinks/' /etc/httpd/conf.d/ssl.conf
After that, just in case, we will backup the original files generated through PKI ovirt automatically and replace the symlinks with files from Let's Encrypt:
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/fullchain.pem /etc/pki/ovirt-engine/apache-chain.pem
services=( 'apache' 'imageio-proxy' 'websocket-proxy' )
for i in "${services[@]}"; do
cp /etc/pki/ovirt-engine/certs/$i.cer{,."$( date +%F )".bak}
cp /etc/pki/ovirt-engine/keys/$i.key.nopass{,."$( date +%F )".bak}
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/privkey.pem /etc/pki/ovirt-engine/keys/$i.key.nopass
ln -f -s /etc/pki/letsencrypt/ovirtengine.example.com/cert.pem /etc/pki/ovirt-engine/certs/{apache,imageio-proxy,websocket-proxy}.cer
done
We restore SElinux contexts on files and restart our services (httpd, ovirt-engine, ovirt-imageio-proxy, ovirt-websocket-proxy):
restorecon -Rv /etc/pki
systemctl restart httpd ovirt-engine ovirt-imageio-proxy ovirt-websocket-proxy
httpd - web server apache
ovirt-engine - web interface ovirt
ovirt-imageio-proxy - daemon for loading disk images
ovirt-websocket-proxy - service for running noVNC console
All of the above has been checked on version 4.2 of the ovirt.
Auto renew certificates for ovirt
According to good security practices, there should not be a connection between the bastion host and the ovirt, and the certificate is issued only for 3 months. This is where a controversial point appears about how I have implemented certificate renewal.
I have an ensemble playbook that runs on foreman daily at 5 a.m. on a schedule. This playbook goes into the ovirt, checks the certificate validity period and if less than 5 days are left before the expiration, it goes to the bastion host and starts the certificate renewal.
After updating the certificate, it archives the folder with the files, downloads the forman to the host, and unzips the ovirt host. Then it restores the SElinux contexts on the files and restarts our services.