Speed up development with Vagrant
How often do you have to develop and run the application locally and persistently look for problems, because in production the application does not behave exactly the way you wanted it to? And how often are tickets sent to you to solve the problem in the application, although in fact the problem is precisely the incompatibility of versions of different applications? And how long do you have to wait for the virtual machine when there are not enough resources of the local machine to launch the new version of the application? For us, these issues were rather painful, and we broke thousands of copies in disputes, trying to solve them. Practice shows that one of the options for solving these problems may be Vagrant.
Vagrant is a kind of wrapper over a virtualization system or, if you like, a DSL. Most often they use VirtualBox, but there are drivers for VmWare and even for Amazon EC2. More detailed information, as well as how to install it and get started, can be found on the official website - www.vagrantup.com
We have several proprietary applications that communicate with each other via the RabbitMQ bus and use things like MongoDB and MySQL. Testing of individual applications running in a synthetic environment far from the real environment did not show us the behavior of the application bundle in the real environment when updating any one element. And changes in the environment did not show at all, because QA and staging environments were far from real. Of course, this does not save us from the need to conduct unit testing.
Let's try to solve this problem with Vagrant.
For clarity, I will show such a scheme:
Description of wokflow:
This can help verify changes to the environment before sending to Product: the work of new versions of third-party applications, Puppet manifests, custom scripts, loss of application communications, service crashes, business process emulation, etc. And we definitely won’t get the situation in the style of "aah, we forgot to put the package on the server" or "damn it, the API changed, and now the applications do not work with each other." Well, it is worth paying attention that in this scheme the Puppet Master is used instead of the Puppet Standalone, precisely for repeating the real environment.
In our case, a complication of the task is that Amazon EC2, Openstack or even VmWare ESXi can be used, which again will not help to catch bugs and situations associated with them. The only way out in this situation I see the use of all this bundle, but with a provider other than VirtualBox. And from this there is a small minus - making Vagrant work with other providers is not always a trivial task. Well and also, there is not always a machine for a build agent that will pull N virtual machines with applications that need a lot of resources to work.
And the whole charm lies in the fact that we have a project in the repository, which has a history of changes that are reviewed, and all changes, up to installing the wim, are tested before getting into production. Well, no one bothers us in the same way to check external resources, update databases, the bus and all sorts of frontends on node.js. It all depends on imagination and desire.
Later in the article we will describe how to use Vagrant to implement the above scheme.
The usual usecase for Vagrant is to raise the service and environment and see how my code works in this environment. Especially if you want to see some kind of platform-dependent thing. Well, plus, Vagrant allows you not to breed a zoo on a working machine, and not to simultaneously keep packs of software and services of different versions for different needs and projects.
One of the most pleasant features for me is support for Puppet, both in master and in standalone modes. Of course, Vagrant can do Chef, but at our company we use Puppet, so the choice is obvious to me. I will use Puppet Standalone, or simply puppet apply.
First of all, we go to the project directory and do vagrant init, as a result we get the Vagrantfile file, which contains a description of our virtual machines. I prefer to use the once configured file and copy it from project to project, changing the necessary sections and configs. In the file below Vagrantfile of one VM. For example, configure RabbitMQ and install Oracle Java.
This configuration file will be enough to start the machine with the usual Ubuntu 12.04 LTS without any installed services and programs. But I’m a lazy person, and I don’t want to install and configure software on my virtual machine with my hands each time, but I don’t want to waste time and immediately start chasing the code or conducting any tests.
Add a section to the node description in the Vagrantfile:
And, accordingly, we need to create ./vagrant.d/modules, ./vagrant.d/manifests, ./vagrant.d/files and ./vagrant.d/manifests/site-rmq.pp and ./vagrant.d/fileserver.conf.
The modules directory contains the module files that we will use; the manifests directory contains the site-rmq.pp manifest of the Puppet of our virtual machine; files - the directory with the files managed by fileserver; fileserver.conf - file for using fileserver and attaching any specific files to our virtual machine.
But if we want to use third-party modules for Puppet, we need to install them before running puppet apply. For this, I did not come up with anything better than using a script; if there are craftsmen who will tell me how to put the modules, and then use them in manifests using only Puppet, I would be grateful. =)
Add a section for provisioning the virtual machine with a shell script:
And create a script in the ./vagrant.d directory:
In the script from the repository, git and librarian-puppet are installed and then with the help of librarian-puppet the modules we need are installed. More information about librarian-puppet features can be found on github: github.com/rodjek/librarian-puppet It is important to install git in advance, otherwise we will not be able to use librarian-puppet to install modules from git repositories. And I also had to make a small hack with copying the module for installing Java, it’s just that this repository has a slightly non-standard location of the module itself. But you can also use the Vagrant plugin for librarian-puppet github.com/mhahn/vagrant-librarian-puppet
Now we can describe our node in the manifest:
Now it’s enough to do in the catalog of our project:
and wait a few minutes, depending on the speed of your internet connection. (At the first start, the Ubuntu image is deflated)
Now you can go to our virtual machine :
and check, for example, installed Java:
And you can also open the WebUI RabbitMQ service:
Of course, you can search for the necessary modules and use Puppet to install whatever your heart desires. The benefit of the Puppet community is quite active and big. And you can also write the desired module yourself and use it. The above is just an example for one service and one machine. They can be countless.
For lazy people I uploaded files as a bonus to github: github.com/wl4n/vagrant-skel
I hope this article helps you spend more time on changing the code, rather than setting up the environment for it, and eliminates the need to fix and reproduce the magic bugs Production-environment 'a.
Link to a similar article, but using Chef Solo (from comments): habrahabr.ru/post/140714
Vagrant is a kind of wrapper over a virtualization system or, if you like, a DSL. Most often they use VirtualBox, but there are drivers for VmWare and even for Amazon EC2. More detailed information, as well as how to install it and get started, can be found on the official website - www.vagrantup.com
Vagrant and environment testing
We have several proprietary applications that communicate with each other via the RabbitMQ bus and use things like MongoDB and MySQL. Testing of individual applications running in a synthetic environment far from the real environment did not show us the behavior of the application bundle in the real environment when updating any one element. And changes in the environment did not show at all, because QA and staging environments were far from real. Of course, this does not save us from the need to conduct unit testing.
Let's try to solve this problem with Vagrant.
For clarity, I will show such a scheme:
Description of wokflow:
- The developer makes any changes to the environment in the repository and sends the review code to other developers;
- TeamCity picks up changes, runs unit tests;
- TeamCity runs Vagrant on BuildAgent, which initiates the launch of several virtual machines and the launch of the Puppet agent;
- The changed environment and stable versions of the tested applications are rolled onto virtual machines;
- Tests run, or QA checks the application;
This can help verify changes to the environment before sending to Product: the work of new versions of third-party applications, Puppet manifests, custom scripts, loss of application communications, service crashes, business process emulation, etc. And we definitely won’t get the situation in the style of "aah, we forgot to put the package on the server" or "damn it, the API changed, and now the applications do not work with each other." Well, it is worth paying attention that in this scheme the Puppet Master is used instead of the Puppet Standalone, precisely for repeating the real environment.
In our case, a complication of the task is that Amazon EC2, Openstack or even VmWare ESXi can be used, which again will not help to catch bugs and situations associated with them. The only way out in this situation I see the use of all this bundle, but with a provider other than VirtualBox. And from this there is a small minus - making Vagrant work with other providers is not always a trivial task. Well and also, there is not always a machine for a build agent that will pull N virtual machines with applications that need a lot of resources to work.
And the whole charm lies in the fact that we have a project in the repository, which has a history of changes that are reviewed, and all changes, up to installing the wim, are tested before getting into production. Well, no one bothers us in the same way to check external resources, update databases, the bus and all sorts of frontends on node.js. It all depends on imagination and desire.
Later in the article we will describe how to use Vagrant to implement the above scheme.
Starting a simple service
The usual usecase for Vagrant is to raise the service and environment and see how my code works in this environment. Especially if you want to see some kind of platform-dependent thing. Well, plus, Vagrant allows you not to breed a zoo on a working machine, and not to simultaneously keep packs of software and services of different versions for different needs and projects.
One of the most pleasant features for me is support for Puppet, both in master and in standalone modes. Of course, Vagrant can do Chef, but at our company we use Puppet, so the choice is obvious to me. I will use Puppet Standalone, or simply puppet apply.
First of all, we go to the project directory and do vagrant init, as a result we get the Vagrantfile file, which contains a description of our virtual machines. I prefer to use the once configured file and copy it from project to project, changing the necessary sections and configs. In the file below Vagrantfile of one VM. For example, configure RabbitMQ and install Oracle Java.
Vagrantfile:
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# All Vagrant configuration is done here. The most common configuration
# options are documented and commented below. For a complete reference,
# please see the online documentation at vagrantup.com.
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "precise64"
# The url from where the 'config.vm.box' box will be fetched if it
# doesn't already exist on the user's system.
# config.vm.box_url = "http://domain.com/path/to/above.box"
config.vm.box_url = "http://files.vagrantup.com/precise64.box"
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# config.vm.network :forwarded_port, guest: 3000, host: 3000
# Create a private network, which allows host-only access to the machine
# using a specific IP.
# config.vm.network :private_network, ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network :public_network
# If true, then any SSH connections made will enable agent forwarding.
# Default value: false
config.ssh.forward_agent = true
### Define VM for RabbitMQ
config.vm.define "rmq", primary: true do |rmq|
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
rmq.vm.provider :virtualbox do |vb|
# Don't boot with headless mode
vb.gui = false
# Use VBoxManage to customize the VM. For example to change memory:
vb.customize ["modifyvm", :id, "--memory", "1024"]
end
# Networking options
rmq.vm.network :private_network, ip: "192.168.100.5"
rmq.vm.hostname = "rmq.example.com"
end
end
This configuration file will be enough to start the machine with the usual Ubuntu 12.04 LTS without any installed services and programs. But I’m a lazy person, and I don’t want to install and configure software on my virtual machine with my hands each time, but I don’t want to waste time and immediately start chasing the code or conducting any tests.
Add a section to the node description in the Vagrantfile:
rmq.vm.provision :puppet do |puppet|
puppet.manifests_path = "./vagrant.d/manifests"
puppet.manifest_file = "site-rmq.pp"
puppet.module_path = "./vagrant.d/modules"
puppet.options = "--fileserver=/vagrant/vagrant.d/fileserver.conf --verbose --debug"
end
And, accordingly, we need to create ./vagrant.d/modules, ./vagrant.d/manifests, ./vagrant.d/files and ./vagrant.d/manifests/site-rmq.pp and ./vagrant.d/fileserver.conf.
The modules directory contains the module files that we will use; the manifests directory contains the site-rmq.pp manifest of the Puppet of our virtual machine; files - the directory with the files managed by fileserver; fileserver.conf - file for using fileserver and attaching any specific files to our virtual machine.
But if we want to use third-party modules for Puppet, we need to install them before running puppet apply. For this, I did not come up with anything better than using a script; if there are craftsmen who will tell me how to put the modules, and then use them in manifests using only Puppet, I would be grateful. =)
Add a section for provisioning the virtual machine with a shell script:
# Enable shell provisioning
config.vm.provision "shell", path: "./vagrant.d/pre-puppet.sh"
And create a script in the ./vagrant.d directory:
pre-puppet.sh:
#!/bin/bash
# This script installs modules for puppet standalone
echo "[Info] Running pre-puppet.sh for install modules"
if [ "x$(dpkg -l | grep -E '^ii\s+git\s')" == "x" ]
then
echo "[Info] Installing git"
apt-get -y install git || (echo "[Error] Cant install git" && exit 0)
else
echo "[Info] git already is installed, skipping"
fi
if [ "x$(gem list librarian-puppet|grep -v LOCAL)" == "x" ]
then
echo "[Info] Installing librarian-puppet"
gem install librarian-puppet ||
(echo "[Error] Cant install librarian-puppet" && exit 0)
else
echo "[Info] librarian-puppet already is installed, skipping"
fi
if [ ! -e Puppetfile ]
then
cat > Puppetfile << EOF
#!/usr/bin/env ruby
#^syntax detection
# Warning!
# Do not edit this file, check pre-puppet.sh script!
#
forge "http://forge.puppetlabs.com"
# use dependencies defined in Modulefile
#modulefile
mod 'puppetlabs/rabbitmq'
mod 'saz/timezone'
mod 'saz/locales'
mod 'jpuppet/java-git',
:git => "git://github.com/jpuppet/java.git"
mod 'jfryman/nginx'
EOF
fi
mkdir -p /vagrant/vagrant.d/modules
echo "[Info] Installing puppet modules"
librarian-puppet install --path=/vagrant/vagrant.d/modules/ ||
(echo "[Error] Cant install modules" && exit)
rm Puppetfile*
# Ugly hack for java
cp -r /vagrant/vagrant.d/modules/java-git/modules/java /vagrant/vagrant.d/modules
exit 0
In the script from the repository, git and librarian-puppet are installed and then with the help of librarian-puppet the modules we need are installed. More information about librarian-puppet features can be found on github: github.com/rodjek/librarian-puppet It is important to install git in advance, otherwise we will not be able to use librarian-puppet to install modules from git repositories. And I also had to make a small hack with copying the module for installing Java, it’s just that this repository has a slightly non-standard location of the module itself. But you can also use the Vagrant plugin for librarian-puppet github.com/mhahn/vagrant-librarian-puppet
Final view of Vagrantfile:
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# All Vagrant configuration is done here. The most common configuration
# options are documented and commented below. For a complete reference,
# please see the online documentation at vagrantup.com.
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "precise64"
# The url from where the 'config.vm.box' box will be fetched if it
# doesn't already exist on the user's system.
# config.vm.box_url = "http://domain.com/path/to/above.box"
config.vm.box_url = "http://files.vagrantup.com/precise64.box"
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# config.vm.network :forwarded_port, guest: 3000, host: 3000
# Create a private network, which allows host-only access to the machine
# using a specific IP.
# config.vm.network :private_network, ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network :public_network
# If true, then any SSH connections made will enable agent forwarding.
# Default value: false
config.ssh.forward_agent = true
# Enable shell provisioning
config.vm.provision "shell", path: "./vagrant.d/pre-puppet.sh"
### Define VM for RabbitMQ
config.vm.define "rmq", primary: true do |rmq|
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
rmq.vm.provider :virtualbox do |vb|
# Don't boot with headless mode
vb.gui = false
# Use VBoxManage to customize the VM. For example to change memory:
vb.customize ["modifyvm", :id, "--memory", "1024"]
end
# Networking options
rmq.vm.network :private_network, ip: "192.168.100.5"
rmq.vm.hostname = "rmq.example.com"
# Enable provisioning with Puppet stand alone. Puppet manifests
# are contained in a directory path relative to this Vagrantfile.
# You will need to create the manifests directory and a manifest in
# the file base.pp in the manifests_path directory.
#
rmq.vm.provision :puppet do |puppet|
puppet.manifests_path = "./vagrant.d/manifests"
puppet.manifest_file = "site-rmq.pp"
puppet.module_path = "./vagrant.d/modules"
puppet.options = "--fileserver=/vagrant/vagrant.d/fileserver.conf --verbose --debug"
end
end
end
Now we can describe our node in the manifest:
site-rmq.pp:
#
# This manifest describes development environment
# RabbitMQ-server
#
class { 'timezone':
timezone => 'Europe/Moscow',
}
class { 'locales':
locales => ['ru_RU.UTF-8 UTF-8'],
}
# apt-get update
# ---------------------------------------
class apt_install {
exec {'update':
command => 'apt-get update',
path => '/usr/bin',
timeout => 0,
} ->
package {[
'vim',
]:
ensure => installed,
}
}
# RabbitMQ service
class rabbitmq_install {
class { '::rabbitmq':
service_manage => false,
port => '5672',
delete_guest_user => true,
}
rabbitmq_user { 'developer':
admin => true,
password => 'Password',
}
rabbitmq_vhost { 'habr':
ensure => present,
}
rabbitmq_user_permissions { 'developer@habr':
configure_permission => '.*',
read_permission => '.*',
write_permission => '.*',
}
rabbitmq_plugin {'rabbitmq_management':
ensure => present,
}
}
class java_install {
class { "java":
version => "1.7",
jdk => true,
jre => true,
sources => false,
javadoc => false,
set_as_default => true,
export_path => false,
vendor => "oracle",
}
}
# Include classes
include apt_install
include timezone
include locales
include rabbitmq_install
include java_install
Now it’s enough to do in the catalog of our project:
vagrant up
and wait a few minutes, depending on the speed of your internet connection. (At the first start, the Ubuntu image is deflated)
Now you can go to our virtual machine :
vagrant ssh
and check, for example, installed Java:
> vagrant ssh
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
Welcome to your Vagrant-built virtual machine.
Last login: Sat May 17 12:28:08 2014 from 10.0.2.2
vagrant@rmq:~$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)
And you can also open the WebUI RabbitMQ service:
Of course, you can search for the necessary modules and use Puppet to install whatever your heart desires. The benefit of the Puppet community is quite active and big. And you can also write the desired module yourself and use it. The above is just an example for one service and one machine. They can be countless.
For lazy people I uploaded files as a bonus to github: github.com/wl4n/vagrant-skel
I hope this article helps you spend more time on changing the code, rather than setting up the environment for it, and eliminates the need to fix and reproduce the magic bugs Production-environment 'a.
Link to a similar article, but using Chef Solo (from comments): habrahabr.ru/post/140714