How to manage virtual machines if there are a lot of them

After we released several more projects, and the number of tickets in the tracker on the topic “create a user, deploy a virtual machine, give access” exceeded all conceivable limits, there was a need to change something.
Objective: to organize a linux working environment for several development teams and testers. The total number of virtual machines is three to four dozen.




I’ll say right away that the article is not a detailed step-by-step guide “as it is and as it should”, but only a description of practical experience and a brief overview of the tools we use. It is understood that the reader will be able to solve the problems himself, for example, installing the necessary dependencies, packages, setting up the network, etc. We deliberately do not attach screenshots and detailed configuration files and leave the reader the opportunity to independently explore the listed tools.

It is necessary:

• automate the deployment of virtual machines according to specific patterns, reduce deployment time,
• Simplify user authentication
• provide quick service configuration on each virtual machine,
• provide the administrator with a single control center for the entire economy and save him from the routine editing of many of the same type of configuration files,
• provide real-time monitoring of the state of the system, as a whole, and for each virtual machine.

Requirements for tools: open-source, ease of setup and use, web interface, or a powerful package of utilities for configuration.

Solutions used: ubuntu 10.04 and other versions, proxmox openvz, chef, openldap, GIT, zabbix

1. Authentication.

Deploy openldap. Recently, the openldap package has switched to a dynamic configuration scheme, so all configuration is carried out using special ldif files. I don’t know whether it’s good or bad, it’s unusual and not always trivial at least. For example, you can use this article . Using the same article, you can configure LDAP replication if necessary.
We did not configure the LDAP frontend, then the LAM web control panel (v3.4.0) did this for us . We never found a more convenient dashboard for LDAP. GOSA has a little more functionality, but much more complicated to set up, the rest of the products are frankly not enough in appearance and functionality.

Since we wanted to simplify the authentication procedure for developers and testers as much as possible, it was decided to collect public keys from them in addition to passwords and put them in LDAP.
This scheme will allow us to authenticate without passwords using SSH keys. The article about keys and general SSH configuration is here , only instead of the authorized_keys file we will use information from LDAP.

To do this, you need to collect openssh with the LPK patch (about it below) and connect the openssh-lpk_openldap.schema scheme in LDAP Configuring the LAM web interface is trivial and does not require special attention. After installation, the panel is able to configure the LDAP frontend itself. In the settings panel, you must also connect the SSH Public Key module (ldapPublicKey).
root:/#cat lpk.ldif
dn: cn=openssh-lpk,cn=schema,cn=config
objectClass: olcSchemaConfig
cn: openssh-lpk
olcAttributeTypes: {0}( 1.3.6.1.4.1.24552.500.1.1.1.13 NAME 'sshPublicKey' DES
C 'MANDATORY: OpenSSH Public key' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.
1.1466.115.121.1.40 )
olcObjectClasses: {0}( 1.3.6.1.4.1.24552.500.1.1.2.0 NAME 'ldapPublicKey' DESC
'MANDATORY: OpenSSH LPK objectclass' SUP top AUXILIARY MAY ( sshPublicKey $
uid ) )
root:/#ldapadd -Y EXTERNAL -H ldapi:/// -f lpk.ldif

2. SSH

From the repository, he does not know how to take public keys from LDAP, we will teach him.
You can read about LPK technology and take the patch here. Everything that redzhednilos, and we got two files and two lines there, patch manually. In debian / rules, you need to add to ../configure in two places. In order for this package to successfully get on the working system and not conflict with the working version from the repository, there are several ways, for example:
root:/#apt-get source ssh
root:/#cd openssh-5.3p1
root:/#patch -p1 < contrib-openssh-lpk-5.4p1-0.3.13.patch


--with-libs="-lldap" --with-ldflags="-L/usr/lib" --with-cppflags="-I/usr/include -DWITH_LDAP_PUBKEY"


• upgrade package version
• fix debian / control and debian / rules, name the package by some other name and try to collect it all.

The easiest way is to upgrade. The package will get up without problems, but in the future there will be problems with updating. Since our infrastructure is closed externally, the lifespan of virtual machines is relatively short, security problems in this environment are not particularly important for us, then we chose this path. For production environments, this method, of course, is not suitable.

We make changes to the changelog and collect the package. At the output we get a certain 'openssh.deb', which is able to query the LDAP 'sshPublicKey' if certain conditions are met. User entry in LDAP
root:/#dch -i
root:/#dpkg-buildpackage -b


• with objectclass 'ldapPublicKey'
• c objectclass 'posixAccount'
• with attribute 'sshPublicKey' filled

Group entry in LDAP

• c objectclass 'posixGroup'
• with the attribute 'cn' and the name of the group in it
• with the attribute 'memberUid', which will list the uid members of this group.

Required settings in / etc / ssh / sshd_config Depending on the LDAP settings, you can simply comment out the unnecessary parameters.
UseLPK yes
LpkServers ldap://10.10.10.10/ # адрес сервера LDAP
LpkUserDN ou=People,dc=office # где искать пользователей
LpkGroupDN ou=group,dc=office # где искать группы
LpkBindDN cn=admin,dc=office # под каким пользователем биндиться, если не анонимно
LpkBindPw secret # пароль
LpkServerGroup developers # каким группам разрешено заходить на этот сервер
LpkForceTLS no # Если у вас настроен TLS, то включаем.
LpkSearchTimelimit 3 # Лимиты по поиску и подключению в секундах
LpkBindTimelimit 3 #
LpkPubKeyAttr sshPublicKey # что забирать из LDAP

3. Deploy configurations.

Since the entire fleet of virtual machines is practically the same for us, we used the deployment system for the chef configuration from the opscode repository .
The system is very powerful and flexible, it allows you to automate the installation of almost everything you can think of.

An example of a simple recipe for configuring Postgres + Postgis Configuring LDAP on the client. Here we simply lay out pre-configured configuration files on the client machine so that the client can log in to our LDAP. Using chef, on each virtual machine we also put the already configured zabbix-agentd and a set of ZTC scripts to it
# Cookbook Name:: postgres
# Recipe:: default
package "postgresql" do
action:install
options "--force-yes"
end

package "postgresql-contrib" do
action:install
options "--force-yes"
end

package "postgis" do
action:install
options "--force-yes"
end

package "postgresql-9.0-postgis" do
action:install
options "--force-yes"
end

script "install_postgis" do
interpreter "bash"
user "postgres"
cwd "/tmp"
code <<-EOH
createdb template_postgis
createlang plpgsql template_postgis
psql -d template_postgis -f /usr/share/postgresql/9.0/contrib/_int.sql
psql -d template_postgis -f /usr/share/postgresql/9.0/contrib/postgis-1.5/postgis.sql
psql -d template_postgis -f /usr/share/postgresql/9.0/contrib/postgis-1.5/spatial_ref_sys.sql
createuser -s pgsql
EOH
end


# Cookbook Name:: openldap
# Recipe:: auth

package "nscd" do
action :upgrade
end

package "libnss-ldap" do
action :upgrade
end

package "libpam-ldap" do
action :upgrade
end

service "nscd" do
supports :status => true, :restart => true, :reload => true
action [ :restart ]
end

service "ssh" do
supports :status => true, :restart => true, :reload => true
action [ :restart ]
end

script "prepare dirs" do
interpreter "bash"
user "root"
cwd "/tmp"
code <<-EOH
mkdir -p /etc/ldap
EOH
end

cookbook_file "/etc/libnss-ldap.conf" do
source "libnss-ldap.conf"
mode 0644
owner "root"
group "root"
end

cookbook_file "/etc/pam_ldap.conf" do
source "pam_ldap.conf"
mode 0644
owner "root"
group "root"
end

cookbook_file "/etc/nsswitch.conf" do
source "nsswitch.conf"
mode 0644
owner "root"
group "root"
notifies :restart, resources(:service => "nscd"), :immediately
end

%w{ account auth password session }.each do |pam|
cookbook_file "/etc/pam.d/common-#{pam}" do
source "common-#{pam}"
mode 0644
owner "root"
group "root"
notifies :restart, resources(:service => "ssh"), :delayed
end
end

. After starting the zabbix virtual machine by autodiscovery, it adds it to the pool and immediately attaches the necessary templates. The zabbix control panel is also attached to LDAP. The only thing that had to be done here was the script for automatic synchronization of LDAP and zabbix users, monitoring is not able to create users itself.

Since our public ssh keys are now stored in a centralized repository, this scheme also made it possible to integrate GIT into the GIT process painlessly and transparently for users.

The Proxmox control panel, pre-configured templates for openvz, LVM storage, allowed us to simplify and speed up the deployment of new virtual machines. We control all allocated resources, under load everything behaves predictably and stably. The cost of implementing and maintaining such a virtual machine is extremely low compared to other virtualization systems.

Let me remind you that this is a test and development environment, completely closed to the outside world, without critical data and without the need for backups. Therefore, in this scheme, many very important points were missed in terms of safety and fault tolerance. For production in this scheme, you will need to think about certificates, TLS, SSL, password policy, about various PAM protection mechanisms, about sudo, as well as firewall rules and group policies.

Some more links:
LDAP for the
Chef Internet project or how to manage a thousand servers
Linux virtualization using OpenVZ
Zabbix universal monitoring system - introduction