February 25, 2015 at 14:10Stamping Windows: Automated Deployment of Windows Virtual Machines on Hyper-V with Vagrant (Part 1)
Sooner or later, any growing IT company faces the task of optimizing resources, which from the point of view of system administration necessarily involves the maximum automation of all processes. This is necessary for many reasons: reducing the time spent, minimizing the influence of the human factor, increasing due to this scalability and reliability of the systems as a whole.
Our company Wild Apricot is developing software (SaaS) that runs on ASP and .NET, and uses MS SQL Server as a database. Therefore, the lion's share of the infrastructure we have is running Windows, mainly Server 2012 R2. Accordingly, the sysop department periodically has to raise new cars with it, and sometimes even deploy whole farms, both on a live environment and on a test one. Currently, we have about 20 hypervisors and, accordingly, more than a hundred different machines, of which the vast majority are Windows VMs. Now it is planned to install another half-dozen development environments, each of which consists of a dozen components that it is desirable to isolate from each other. To solve such problems, we conceived the automation of the entire process. I made a detailed description of the whole path, so the article has grown and therefore I decided to divide it into three parts. In this part I will try to address general issues, explain the choice of tools and tell how to prepare the environment for further work.
Historically, all new VMs we have are running the Microsoft Hyper-V hypervisor.
My task was to automate the creation of new virtual machines. It is necessary to quickly raise the operating system on the hypervisor with all the necessary software and settings. All this is essentially the first step towards a full-fledged continuous deployment system with a configuration management system, which we plan to implement. This is far beyond the scope of this article, but there are enough materials on this topic, and I am going to continue the story about all the nuances of working in the world of Microsoft products in the next article, if this material turns out to be interesting and useful. As a result, the task was successfully solved using a fairly traditional set of tools, many of which have long been known in the world of open source software, but it was not so easy to apply them fully in the Windows world because of the abundance of various pitfalls.
In this article, we will talk about the first three items of the list.
So what do they do and why were they chosen?
Using this toolkit allows you to get all the magic of Continuous Delivery in one bottle under the familiar system.
Well, let's get down to the fun part, the applied technical part of the deployment.
First you need to prepare the host - the system on which virtual machines will be deployed. If they are needed for local tests and everything will be spinning on the same computer where you are sitting, everything is simple and transparent. However, a situation often arises when virtual computers are needed in the end in a completely different place - in a data center on a live environment, on a neighboring server or on a grandmother's netbook. The fact is that out of the box Vagrant scripts do all operations on the localhost. In principle, no one bothers to deploy it first at home, configure-play, and then migrate to a distant hypervisor, but it turns out that it takes a decent amount of time even when the server is on the same network segment with you. In the case of
Therefore, the optimal solution for myself I chose the Vagrant lift on all hypervisors where it is planned to be used frequently. This speeds up the process (even for one machine - because the box weighs significantly less than the VHD file of the deployed image), but it adds extra software to the server. Fortunately, Chocolatey and Vagrant do not require a GUI, so they can easily be installed even on the free Hyper-V Server.
This task as a whole is rather trivial:
In general, if you don’t like clogging up servers with unnecessary programs or prefer to always use the latest versions (you should understand that, despite the fact that the repository usually contains the latest version - it doesn’t appear there instantly) - you can completely do without Chocolatiers by installing Vagrant manually from the distribution. It comes in the form of an msi package, so there should be no problems installing from the console. But personally, I prefer the first option, not only because I like chocolate - I just used to put all the software in this way, even on a home laptop.
At this moment we are faced with the first pitfall (I already warned that the path will be thorny?). The fact is that our tramp (that is how “vagrant” is translated) puts itself in different directories from version to version, and in the latest release at the moment it again began to be installed directly at the root of the system drive in the C: \ HashiCorp \ Vagrant folder . Everything would be fine, but he periodically forgets to write the path to his folder to the environment variable, so the system may not find it unless you enter the full path to the binary. This is treated with a simple command in Powershell:
For the command line, use the setx command with the / M switch. For example, if you need to change the location of the folder where it will store the boxes (by default, it stores them on its folder on drive C, which may not be very pleasant when there is not enough space on the system drive):
This completes the preparation of the host. About how to make a basic box for deployment and how to work with it further - I will tell in the following parts ( part 2 ). There will also be a separate chapter with a list of useful materials and links.
Our company Wild Apricot is developing software (SaaS) that runs on ASP and .NET, and uses MS SQL Server as a database. Therefore, the lion's share of the infrastructure we have is running Windows, mainly Server 2012 R2. Accordingly, the sysop department periodically has to raise new cars with it, and sometimes even deploy whole farms, both on a live environment and on a test one. Currently, we have about 20 hypervisors and, accordingly, more than a hundred different machines, of which the vast majority are Windows VMs. Now it is planned to install another half-dozen development environments, each of which consists of a dozen components that it is desirable to isolate from each other. To solve such problems, we conceived the automation of the entire process. I made a detailed description of the whole path, so the article has grown and therefore I decided to divide it into three parts. In this part I will try to address general issues, explain the choice of tools and tell how to prepare the environment for further work.
Why Windows, why Hyper-V?
Historically, all new VMs we have are running the Microsoft Hyper-V hypervisor.
There are several reasons for this:
- In recent years, it has developed quite a lot and, on the whole, has sufficient functionality for our tasks. The most important thing is that Vagrant began to support him humanly, which I will discuss in more detail later. Also, there is support for many Linux distributions, which allows, with some reservations, to host a zoo of different machines on Hyper-V, in case you use several OSes (and someone else is wrong?);
- It is convenient to manage it using native tools (MMC toolkit and Powershell), built-in both server and client OSes from Microsoft;
- The role of Hyper-V is very easy to add to the server, and in general the installation is available to a wider circle of administrators (rather than, say, in the case of KVM or ESXi), there are many documentation, guides, virtual laboratories, and so on, much of this is also available in Russian language. This makes it easy to install it not only for administrators, but also for developers or testers (who also often need to quickly raise the environment for their purposes), which expands the range of possible applications;
Migration of physical machines to virtual ones is very easy and fast with the help of a simple utility Disk2VHD - it uses a snapshot mechanism and due to this it allows you to make a VHD image of even a working system running in a live environment. It helped a lot when we virtualized our infrastructure. In the same way, we migrated VMWare hosts. - Microsoft Hyper-V Server does not need to be bought, it is free . For us, this was not very relevant, but this is a great opportunity to “play around” with her at no additional cost.
- From Hyper-V, later, if desired, you can easily migrate to the Azure cloud platform, while continuing to manage it using the same tools.
Goals
My task was to automate the creation of new virtual machines. It is necessary to quickly raise the operating system on the hypervisor with all the necessary software and settings. All this is essentially the first step towards a full-fledged continuous deployment system with a configuration management system, which we plan to implement. This is far beyond the scope of this article, but there are enough materials on this topic, and I am going to continue the story about all the nuances of working in the world of Microsoft products in the next article, if this material turns out to be interesting and useful. As a result, the task was successfully solved using a fairly traditional set of tools, many of which have long been known in the world of open source software, but it was not so easy to apply them fully in the Windows world because of the abundance of various pitfalls.
- Chocolatey
- Vagrant;
- Boxstarter
- Chef.
In this article, we will talk about the first three items of the list.
Why Vagrant and Chef, Chocolatey and Boxstarter?
So what do they do and why were they chosen?
- Chocolatey is a package manager for Windows. Of course, Microsoft announced that Windows Server 10 will have its own, integrated package manager, but before implementing it in our production, it is still far and for now you can use chocolatiers quite successfully. It allows you to automate the installation of the necessary software along with all the dependencies. For this, NuGet packages are used, which you can quite successfully create yourself, which is very convenient when you need to install your own server software.
- Vagrant allows you to create and use a box - a sample of a virtual machine, to subsequently automatically deploy one (or several, or several tens) of virtual machines, configure the necessary parameters, and also implement the so-called provisioning - the process of installing and configuring the necessary software. For example, IIS and Chef-client can be installed there, the latest Windows updates can be installed, folders can be configured to look like file extensions and hidden files, ports can be opened on the firewall, user accounts can be set up and so on - and all this without direct participation administrator, with automatic reboots if necessary, according to a previously described scenario.
Of the analogues, System Center immediately comes to mind, which also allows you to create a template and roll out several virtual machines from it. However, for many reasons, I preferred Vagrant - it is lighter, does not require a heavy central server, and the disk of the new virtual machine does not become dependent on the disk of the original template. In addition, Vagrant runs from a regular command line, which makes it easy to use on hypervisors without a GUI.
The article will contain quite a few references to various pitfalls - I want to say right away that the latest version is currently being considered - 1.7.2, respectively, in subsequent versions (I hope) many problems will be fixed and unnecessary crutches will fall off. But they will surely add new bugs, which will allow me to make a whole series of articles. - Boxstarter is a set of useful scripts that greatly simplify the configuration of both finished machines and those just raised. It works quite closely with Chocolatey and allows you to configure the final system with a single command, downloading the script from a website or network folder. In addition, there are ready-made recipes for Chef from the author for him.
- Chef is used to centrally manage configurations in the broadest sense, including both initial initialization and keeping them up to date. Other systems can be used instead, but I settled on it because of its prevalence, powerful community, and active development: for example, recently a new, significantly improved version of the community cookbooks for IIS has appeared. Well, of course, it was influenced by the fact that he already had a little experience with this before.
Using this toolkit allows you to get all the magic of Continuous Delivery in one bottle under the familiar system.
The main stages of deployment
Well, let's get down to the fun part, the applied technical part of the deployment.
Install Hyper-V and Management Software
First you need to prepare the host - the system on which virtual machines will be deployed. If they are needed for local tests and everything will be spinning on the same computer where you are sitting, everything is simple and transparent. However, a situation often arises when virtual computers are needed in the end in a completely different place - in a data center on a live environment, on a neighboring server or on a grandmother's netbook. The fact is that out of the box Vagrant scripts do all operations on the localhost. In principle, no one bothers to deploy it first at home, configure-play, and then migrate to a distant hypervisor, but it turns out that it takes a decent amount of time even when the server is on the same network segment with you. In the case of
Therefore, the optimal solution for myself I chose the Vagrant lift on all hypervisors where it is planned to be used frequently. This speeds up the process (even for one machine - because the box weighs significantly less than the VHD file of the deployed image), but it adds extra software to the server. Fortunately, Chocolatey and Vagrant do not require a GUI, so they can easily be installed even on the free Hyper-V Server.
This task as a whole is rather trivial:
- Installing the Hyper-V role in Windows 8 or Server 2012 requires a reboot and is done through Server Manager for GUI lovers (see the link at the end of the article) or using PowerShell with administrative privileges:
Install-WindowsFeature –Name Hyper-V -IncludeManagementTools
- Chocolatier is put from Powershell with one command:
iex ((new-object net.webclient) .DownloadString ('https://chocolatey.org/install.ps1'))
- Well, Vagrant rises already from Chocolatey:
choco install vagrant
In general, if you don’t like clogging up servers with unnecessary programs or prefer to always use the latest versions (you should understand that, despite the fact that the repository usually contains the latest version - it doesn’t appear there instantly) - you can completely do without Chocolatiers by installing Vagrant manually from the distribution. It comes in the form of an msi package, so there should be no problems installing from the console. But personally, I prefer the first option, not only because I like chocolate - I just used to put all the software in this way, even on a home laptop.
At this moment we are faced with the first pitfall (I already warned that the path will be thorny?). The fact is that our tramp (that is how “vagrant” is translated) puts itself in different directories from version to version, and in the latest release at the moment it again began to be installed directly at the root of the system drive in the C: \ HashiCorp \ Vagrant folder . Everything would be fine, but he periodically forgets to write the path to his folder to the environment variable, so the system may not find it unless you enter the full path to the binary. This is treated with a simple command in Powershell:
$ env: Path + = ”; C: \ HashiCorp \ Vagrant \ bin”
For the command line, use the setx command with the / M switch. For example, if you need to change the location of the folder where it will store the boxes (by default, it stores them on its folder on drive C, which may not be very pleasant when there is not enough space on the system drive):
setx VAGRANT_HOME "X: / your / path" / M
This completes the preparation of the host. About how to make a basic box for deployment and how to work with it further - I will tell in the following parts ( part 2 ). There will also be a separate chapter with a list of useful materials and links.