December 8, 2016 at 22:39Experience in building Infrastructure-as-Code in VMware. Part 1: Indication of a problem
Greetings, dear reader. I am starting a series of articles on how we were looking for a solution for applying the Infrastructure-as-Code approach in our virtual VMware VSphere environment.
We have a Puppet configuration management system for Linux, there is (at the moment) DSC for Windows Server.
As for Linux - almost everything is automated. We enter the machine configurations in nodes.yaml, we enter the roles in Hiera, build the modules (or take ready-made ones), we have PXE, IP addresses are distributed from DHCP by MAC address.
That is, from the moment the virtual machine starts Linux, until the virtual machine is ready for operation - no action is needed. Try to guess what is done manually in this chain? That's right, creating the virtual machine itself in VSphere.
When I first raised this question, they told me that they were looking for a solution, tried options, but nothing worked. "To hell!" - I thought and argued on a beer box that I would find a solution that would work according to the following scenario: the developer or engineer makes a Pull Request, in which we have the configuration of the virtual machine (kernel, memory, network, template, etc.) - Further, some magic turns to VSphere and creates a machine, according to the settings in the file.
Let me tell you a little about our surroundings, so that you understand what I have to deal with.
We use VMware VSphere as On-Premise virtualization - a couple of data centers, a datastore cluster and several Resource Pools (RP) for each command. Team members have the right to create virtual machines within the RP, the infrastructure guys do not interfere with this and simply maintain the entire platform, periodically reminding developers and engineers to clean up unused machines (non-rubber resources).
We have Windows virtual machines, Linux virtual machines, the scale of the tasks is huge - web servers, reverse proxies, balancers, domain controllers, application and database servers and there is no end to them.
Now I will tell you what tools I tried to use, and why they did not suit me.
As I already wrote in the previous article , I really love Ansible and in matters of automation, the first thing I see is whether it can be used for this.
According to the documentation, there is a good module vsphere_guest which can create and remove virtual machines. Exactly what is needed. This is what my createvm.yaml playbook looks like
I deliberately comment on hostname esxi because I create a virtual machine directly in RP, and not on the host. DRS will decide where to put the virtual machine.
If I run a playbook, he swears that the required hostname parameter is not specified. If I uncomment him, then he will argue about the lack of rights to create a virtual machine on the esx host (which is obvious, because I only have rights to RP). I created a corresponding issue , so I hope the guys from Ansible fix this, because the tool is really good.
Another tool that can create virtual machines in VMware is Terraform, a product from HashiCorp. Initially, it was imprisoned for interaction with Packer and deployed in AWS, but it also solves our problems. Here is the configuration file itself:
variables.tf
terraform plan works great.
What is also great, you can set the IP address, domain name - that is, set the full customization of the machine from the template. Trying to run ...
Hmm, datastore not found. As I said, we have a cluster, so I’ll try to do it in a dirty way by specifying one of the cluster nodes.
Well ... failure again. Later it turned out that Terraform is not able to work with datastore clusters. The corresponding issue was created on GitHub by my colleague, but alas, no success in this field either.
Having failed in the search for working tools from third parties, I decided to turn to a vendor solution.
The vendor offers two solutions - PowerCLI (add-on for Powershell) and vmware-cli (command line interface for * nix).
It was not possible to get vmware-cli to work on CentOS 7 and OS X (one sufferer even wrote a blog about it), so I decided to immediately start using a tool that works.
An attentive reader may wonder why I spent so much time on Ansible and Terraform, while PowerCLI has long been used. The reasons are simple - I do not know Powershell at the proper level to start using it with a snap, plus it forces me to use a windows machine that will deal with a clean provision. However, I have no other options.
A quick study of the documentation gave me enough skills to write a simple script.
This script is working and does everything necessary. Running the script is as follows:
The script will ask me to provide a username and password, replay the variables and create a machine using cmdlet'a new-vm. The reader may wonder why this line is present:
Let experienced powershell guys correct me if I'm wrong. Get-Credential creates an object consisting of login, password and domain (if any). Password is in SecureString state. Unfortunately, PowerCLI does not know how to work with either the Get-Credential object or SecureString, so you have to go to such tricks to pass the username and password to a simple string variable.
Dear reader, if one day you have the task of automating the creation of virtual machines in VMware, then consider the following:
If you have a single node ESX, then I recommend using Ansible, it has a low entry threshold and is quite lightweight and nimble.
If you have the same complex infrastructure as ours, then it’s better not to reinvent the wheel, but develop PowerCLI.
In the next part I will tell how we made our script smarter, and taught him how to do checks for customization, the number of cores and other resources and naming convention.
We have a Puppet configuration management system for Linux, there is (at the moment) DSC for Windows Server.
As for Linux - almost everything is automated. We enter the machine configurations in nodes.yaml, we enter the roles in Hiera, build the modules (or take ready-made ones), we have PXE, IP addresses are distributed from DHCP by MAC address.
That is, from the moment the virtual machine starts Linux, until the virtual machine is ready for operation - no action is needed. Try to guess what is done manually in this chain? That's right, creating the virtual machine itself in VSphere.
When I first raised this question, they told me that they were looking for a solution, tried options, but nothing worked. "To hell!" - I thought and argued on a beer box that I would find a solution that would work according to the following scenario: the developer or engineer makes a Pull Request, in which we have the configuration of the virtual machine (kernel, memory, network, template, etc.) - Further, some magic turns to VSphere and creates a machine, according to the settings in the file.
Let me tell you a little about our surroundings, so that you understand what I have to deal with.
We use VMware VSphere as On-Premise virtualization - a couple of data centers, a datastore cluster and several Resource Pools (RP) for each command. Team members have the right to create virtual machines within the RP, the infrastructure guys do not interfere with this and simply maintain the entire platform, periodically reminding developers and engineers to clean up unused machines (non-rubber resources).
We have Windows virtual machines, Linux virtual machines, the scale of the tasks is huge - web servers, reverse proxies, balancers, domain controllers, application and database servers and there is no end to them.
Now I will tell you what tools I tried to use, and why they did not suit me.
Empirically ...
Ansible vsphere_guest
As I already wrote in the previous article , I really love Ansible and in matters of automation, the first thing I see is whether it can be used for this.
According to the documentation, there is a good module vsphere_guest which can create and remove virtual machines. Exactly what is needed. This is what my createvm.yaml playbook looks like
---
- name: Create a VM in resource pool
hosts: localhost
connection: local
gather_facts: False
vars_prompt:
- name: "user"
prompt: "Enter your username to virtualcenter"
private: no
- name: "password"
prompt: "Enter your password to virtualcenter"
private: yes
- name: "guest"
prompt: "Enter you guest VM name: "
private: no
tasks:
- name: create VM
vsphere_guest:
vcenter_hostname: vcenter.example.com
validate_certs: no
username: '{{ user }}'
password: '{{ password }}'
guest: '{{ guest }}'
state: powered_off
vm_extra_config:
vcpu.hotadd: yes
mem.hotadd: yes
notes: This is a test VM
vm_disk:
disk1:
size_gb: 10
type: thick
datastore: mydatastore
vm_nic:
nic1:
type: vmxnet3
network: mynetwork
network_type: standard
vm_hardware:
memory_mb: 1024
num_cpus: 1
osid: centos64Guest
scsi: paravirtual
resource_pool: "/Resources/MyResourcePool"
esxi:
datacenter: mysite
#hostname: myesxhost01
I deliberately comment on hostname esxi because I create a virtual machine directly in RP, and not on the host. DRS will decide where to put the virtual machine.
If I run a playbook, he swears that the required hostname parameter is not specified. If I uncomment him, then he will argue about the lack of rights to create a virtual machine on the esx host (which is obvious, because I only have rights to RP). I created a corresponding issue , so I hope the guys from Ansible fix this, because the tool is really good.
Terraform
Another tool that can create virtual machines in VMware is Terraform, a product from HashiCorp. Initially, it was imprisoned for interaction with Packer and deployed in AWS, but it also solves our problems. Here is the configuration file itself:
provider "vsphere" {
user = “[email protected]"
password = "${var.vsphere_password}"
vsphere_server = “virtualcenter.example.com"
allow_unverified_ssl = "true"
}
resource "vsphere_virtual_machine" "test" {
name = "${var.machine_name}"
vcpu = 1
memory = 1024
domain = “test.example.com”
datacenter = "mysite"
resource_pool = "Production Cluster #1/Resources/myresourcepool"
network_interface {
label = "test"
ipv4_address = "192.168.1.2"
ipv4_prefix_length = "24"
ipv4_gateway = "192.168.1.1"
}
disk {
datastore = "${var.datastore}"
size = "10"
name = "${var.datastore}/${var.machine_name}/${var.machine_name}.vmdk"
template = "mytemplate"
}
}
variables.tf
variable "vsphere_password" {}
variable "machine_name" {
type = "string"
default = "test"
}
variable "datastore" {
type = "string"
default = "mysite/mydatastore"
}
terraform plan works great.
$ terraform plan
var.vsphere_password
Enter a value: supersecurepassword
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.
The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.
Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.
+ vsphere_virtual_machine.test
datacenter: "mysite"
detach_unknown_disks_on_delete: "false"
disk.#: "1"
disk.1370406802.bootable: ""
disk.1370406802.controller_type: "scsi"
disk.1370406802.datastore: ""
disk.1370406802.iops: ""
disk.1370406802.keep_on_remove: ""
disk.1370406802.key: ""
disk.1370406802.name: ""
disk.1370406802.size: ""
disk.1370406802.template: "mytemplate"
disk.1370406802.type: "eager_zeroed"
disk.1370406802.uuid: ""
disk.1370406802.vmdk: ""
domain: “test.example.com”
enable_disk_uuid: "false"
linked_clone: "false"
memory: "1024"
memory_reservation: "0"
name: "test"
network_interface.#: "1"
network_interface.0.ip_address: ""
network_interface.0.ipv4_address: “192.168.1.2"
network_interface.0.ipv4_gateway: "192.168.1.1"
network_interface.0.ipv4_prefix_length: "24"
network_interface.0.ipv6_address: ""
network_interface.0.ipv6_gateway: ""
network_interface.0.ipv6_prefix_length: ""
network_interface.0.label: "test"
network_interface.0.mac_address: ""
network_interface.0.subnet_mask: ""
resource_pool: "Production Cluster #1/Resources/myresourcepool"
skip_customization: "false"
time_zone: "Etc/UTC"
uuid: ""
vcpu: "1"
Plan: 1 to add, 0 to change, 0 to destroy. What is also great, you can set the IP address, domain name - that is, set the full customization of the machine from the template. Trying to run ...
Error applying plan:
1 error(s) occurred:
* vsphere_virtual_machine.test: Datastore 'mysite/mydatastore not found.
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
Hmm, datastore not found. As I said, we have a cluster, so I’ll try to do it in a dirty way by specifying one of the cluster nodes.
Error applying plan:
1 error(s) occurred:
* vsphere_virtual_machine.test: Datastore 'mysite/mydatastore/mydatastore-vol01' not found.
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
Well ... failure again. Later it turned out that Terraform is not able to work with datastore clusters. The corresponding issue was created on GitHub by my colleague, but alas, no success in this field either.
Powercli
Having failed in the search for working tools from third parties, I decided to turn to a vendor solution.
The vendor offers two solutions - PowerCLI (add-on for Powershell) and vmware-cli (command line interface for * nix).
It was not possible to get vmware-cli to work on CentOS 7 and OS X (one sufferer even wrote a blog about it), so I decided to immediately start using a tool that works.
An attentive reader may wonder why I spent so much time on Ansible and Terraform, while PowerCLI has long been used. The reasons are simple - I do not know Powershell at the proper level to start using it with a snap, plus it forces me to use a windows machine that will deal with a clean provision. However, I have no other options.
A quick study of the documentation gave me enough skills to write a simple script.
Param(
[string]$Name,
[string]$ResourcePool,
[string]$Location,
[int]$NumCPU,
[int]$MemoryGB,
[int]$DiskGB)
$ErrorActionPreference = "Stop"
Try
{
$credential = Get-Credential
Add-PSSnapin VMware.VimAutomation.Core
[string] $username = $credential.GetNetworkCredential().UserName
$username = 'example\' + $username
Connect-VIServer -Server virtualcenter.example.com -User $username -Password $credential.GetNetworkCredential().Password -Force
$params = @{
name = $Name
ResourcePool = $ResourcePool
Location = $Location
NumCPU = $NumCPU
MemoryGB = $MemoryGB
DiskGB = $DiskGB
}
new-vm @params
}
Catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.DuplicateName] {"VM exists"}
This script is working and does everything necessary. Running the script is as follows:
.\createvm.ps1 -Name mytestvm -ResourcePool myresourcepool -Location myteam -NumCPI 1 - MemoryGB 1 -DisckGB 10 The script will ask me to provide a username and password, replay the variables and create a machine using cmdlet'a new-vm. The reader may wonder why this line is present:
[string] $username = $credential.GetNetworkCredential().UserName
$username = 'example\' + $usernameLet experienced powershell guys correct me if I'm wrong. Get-Credential creates an object consisting of login, password and domain (if any). Password is in SecureString state. Unfortunately, PowerCLI does not know how to work with either the Get-Credential object or SecureString, so you have to go to such tricks to pass the username and password to a simple string variable.
conclusions
Dear reader, if one day you have the task of automating the creation of virtual machines in VMware, then consider the following:
- Do you use Resource Pools
- do you use datastore clusters
If you have a single node ESX, then I recommend using Ansible, it has a low entry threshold and is quite lightweight and nimble.
If you have the same complex infrastructure as ours, then it’s better not to reinvent the wheel, but develop PowerCLI.
In the next part I will tell how we made our script smarter, and taught him how to do checks for customization, the number of cores and other resources and naming convention.
