Cloud from scratch using XenServer
Recently we created a small cloud to solve our internal problems and want to share this experience with Habr readers. Here we describe in detail what equipment was chosen for the deployment of the cloud and how to create the infrastructure of the cloud system, relying on XenServer from Citrix. In this product, Citrix decided to abandon the standard approach, when the cloud has some central control node, they split it into several components and offered to place them in the cloud too. Who cares how it all works - welcome under the cut!
In this article, we will focus on such issues as: preparing the hardware, installing XenServer, installing a license, creating a virtual network infrastructure, describe the problems encountered with virtual machines on the Ubuntu OS, and talk about dynamic load balancing, about settings and differentiation of access to the cloud , and, of course, we will show what we did.
The first task is to choose the basis for any cloud, namely, the choice of servers on which virtualization will be performed. We chose IBM servers and chose the IBMx3850 X5.
Each IBM is based on the IBM X-Architecture and has:
• 4 Intel® Xeon® CPUE7- 8860 processors with a clock frequency of 2.27GHz, which ultimately gives 40 cores per server (80 threads);
• 150GB RAM;
• 2 independent power supplies;
• fiber channel expansion card ;
• network card for a 10-gigabit connection;
• 2 hard drives of 500 GB, combined in RAID1 .
Then the question arises: where to store virtual machines? If they are stored on the servers themselves, this reduces the reliability of the system, because when the server fails, we lose all the virtual machines that were on it. Also, this approach greatly complicates the task of load balancing, because the migration of a virtual machine will require moving its disk to another server, and this is a rather long process. Therefore, our stand uses an external DELL md3620f storage equipped with 4 fiber channel outputs. This storage supports up to 24 hard drives, which can be combined into all popular types of raids (RAID0, RAID1, RAID5, RAID6, RAID01). In our case, we use 10 hard drives of 1 TB combined in RAID5.
What is required for quick migration? To ensure fast migration between IBMs, a 10 gigabit summit x670 switch was added to the stand, this theoretically should have accelerated migration (the longest process in migration is transferring data over the network from one server to another) by 10 times, but in practice it only won 5-6 times. To enable servers and virtual machines to have access to the local network and the Internet, the HP ProCurve switch was added to the stand, as well as through it traffic to external clients.
To summarize the hardware, we have a stand that includes:
• 4 IBMx3850 X5 servers
• HP ProCurve switch for connecting up to 1000 Mb / s
• Summit x670 switch for supporting 10 Gbit / s connection
• external storage DELL md3620f with 10TB disk array
• and more (uninterruptible power supply APC Smart-UPS 3000VA, transceivers for fiber, eight ten-meter fiber optic cables, 50 meters twisted pair)
All this looks like in the picture, there are five uninterrupted sources below power supply, which in case of failure of the central power supply, will correctly stop everything running on the server and turn off the servers themselves.
The logical connection diagram is as follows:
If summarized, we have:
• 160 computing cores (320 threads);
• 600GB RAM;
• 14TB of disk space;
• High-speed network infrastructure.
To run virtual machines on our chosen hardware, it was decided to deploy the XenServer Platinum cloud platform.
Brief general information about XenServer, if anyone has not encountered: XenServer is an independent operating system based on the Linux kernel. At the heart of everything is the XEN hypervisor, the latest version at the moment - the used version of XenServer 6.1 - is based on the XEN 4.1 hypervisor. For normal operation of the system, a processor with support for virtualization and 1GB of RAM is required. Citrix guys do not like to write their minimum requirements, preferring to write maximum system requirements, you can find them here . The XenServer family includes several product versions that differ in price and additional features .
All IBM servers must have XenServer installed. Installing XenServer itself is a completely straightforward process and is not very different from installing Ubuntu. Much more interesting is setting up the system for subsequent convenient operation, which will be discussed later, the main focus will be on the extension components for XenServer that Citrix provides. After installing XenServer on all hosts, you need to go to XenCenter (you can download it by going to the web interface of the installed XenServer or from the Citrix website). In XenCenter, you need to create a pool and add a server to it.
After installing XenServer, the first thing you need to do is install the server licenses. This is done not by entering license keys, but by installing a certificate received in your account on the Citrix website. The certificate is a text file with license keys recorded in a special format. It can be obtained at once on several servers. The certificate must be installed in a special licensing center. It is worth noting that if at least one of the certificates is not valid, then all certificates will not be valid.
Citrix suggests installing a licensing center either on a separate host or on a virtual machine in your cloud (it consumes almost no CPU resources and requires only 256 MB of RAM). But if you install it as a virtual machine, you may be in trouble. One day you may encounter a problem that we encountered when we turned off the servers of our stand for their upgrade, and with them turned off, respectively, a virtual machine with a licensing center. After switching on, XenServer generated an error “The trial license has expired” (the licensing center wasn’t launched after all). It seems there shouldn't be any problems: just turn on the virtual machine of the licensing center with a normal license and accept it. But there it was: with an expired license, you can’t turn on ANY virtual machine. True, after the initial panic, the button “activate the free version” was found in the license acceptance menu, after clicking which we saw the phrase “you have 30 days of a trial license”. And only then you can start a virtual machine with a license.
To flexibly manage the network settings of virtual machines and not to be on the same subnet as XenServer servers, you need to establish a virtual network infrastructure. For these purposes, Citrix has a Distributed Virtual Switch Controller (DVSC) component. It is also provided as a virtual machine, but it already consumes more resources: 2 VCPU and 2GB of RAM. DVSC settings are not complicated: you need to specify a free IP address for the virtual machine, and then add the pool to DVSC. All this is done through the web interface. After these actions, it becomes possible to create virtual networks between virtual machines located on different servers (Cross-ServerPrivateNetwork),
If you go to the DVSC web interface, you can see a lot of useful information about network infrastructure: a list of created networks, a list of virtual machines connected to a specific network, network error messages, network activity graphs (both for a specific virtual machine and for the entire network as a whole).
DVSC can also act as a simple firewall, the rules of which can be created in the AccessControl tab. All network security policies are divided into 4 levels: global policies (as multiple pools can be connected to one DVSC), pool policies, specific network policies, and specific virtual machine policies. The rules describe the type of action (allow or deny), protocol (you can specify the recipient and sender ports or specify a known protocol) and for whom to apply this rule.
The firewall enters work immediately after installation, and it immediately contains 4 basic global rules:
• allow all ARP messages on the network
• allow virtual machines to receive an IP address via dhсp
• allow virtual machines to access DNS servers
• allow all network traffic (this applies after checking all the rules of other levels)
generally DVSC all good, and functionality is a bit like vShield Edge from VmWare, but Edge seems more convenient due to different things: the possibility of creating the DHCP-server, user-friendly virtual organization for Nat s machines, etc. All this (the absence of a DHCP server and Nat), of course, is solved by creating a separate virtual machine based on Ubuntu, but it’s convenient when everything is already decided in advance.
In general, creating virtual machines with Ubuntu for the first time causes shock and misunderstanding - how did this get into the release version of the product? Here's the thing:
After creating a virtual machine from a standard template, it cannot start and writes the above error (Error code: INVALID_SOURCE). The error is related to the boot parameters of the virtual machine. You can deal with it as follows (the description is taken here and slightly modified to work with a large number of virtual machines):
1. Go to the XenServer console, which can be done through XenCenter (the Console tab on the server) or via ssh.
2. Find out the uuid of the virtual machine with the command xe vm-listname-label = [VM_NAME]. In our example, it looks like this:
3. Next, you need to set the boot parameters with the following command: xe vm-param-setuuid = [UUID_VM] HVM-boot-policy = BIOS \ orderHVM-boot-params: order = dc.
4. After these simple manipulations, the virtual machine will start successfully.
But this is not the end of Ubuntu-related errors. When creating our stand, it was decided to create virtual machine templates with different OSs, so as not to waste time installing the necessary one, but to immediately take the finished and put the necessary software there. There are no problems with Windows-based machines, while with Ubuntu there is a black screen problem when creating a virtual machine from an image. The solution to this problem turned out to be quite simple, on the one hand, and a little wrong, on the other. The problem is solved by simply installing xen-tools on a virtual machine. The downside of this solution is that it is impossible to provide a clean operating system, which is sometimes required as part of the tasks being solved.
As part of the tasks being solved, dynamic load balancing between servers with XenServer is often needed. For these purposes, Citrix provides the Citrix WLB Virtual Appliance virtual machine, which also needs to be added to the cloud, after which it is easy to configure through its console (when entering the console, the machine will tell you all the necessary actions). After that, you need to go to XenCenter and indicate to the pool that this particular virtual machine will be responsible for balancing the load between the servers (this action is performed in the WLB tab).
This virtual machine monitors the load on the server (the number of cores used, the amount of RAM used, network activity) and distributes it between the servers. This happens both when the virtual machine is turned on (it runs on the most unloaded server) and during its operation (due to migration).
The last task that needs to be solved for normal operation is providing access to the cloud. And here, Citrix, in our opinion, has the biggest problems. Citrix offers two options for accessing the cloud: through XenCenter and through the web-based interface.
If you connect Active Directory (AD) to the pool, you can create users in XenCenter. Citrix decided to abandon the discretionary access model at XenCenter and implemented a role-based model. Hence the main problem: ALL users see and have access to ALL virtual machines, only the type of access is regulated, but it applies to ALL virtual machines immediately (i.e. if you are given a role to start virtual machines, then all at once). It is also worth noting that AD should always be available, as upon reboot, AD is not automatically added to the pool, and you must add it manually each time.
Citrix suggests using web-based access for discretionary access. To configure access via the web-based interface, you must install the Citrix XenServer Web Self Service virtual machine. After simple configuration of the virtual machine through its console (you must specify an IP address or specify that it is obtained via DHCP), you need to perform a number of settings through the web interface. Here Citrix is beyond praise for an accessible and understandable description: if you are logged in as an administrator, you will immediately be shown a list of steps that must be completed, as well as a detailed description of how to do this.
Citrix XenServer Web Self Service can use the same users that are in AD, or create new ones. When the XenServer Web Self Service is first loaded, the administrator needs to specify how he wants to act, and this decision cannot be changed later (of course, you can always rearrange the virtual machine, but this will entail a new setting of access rights to the virtual machines). After configuration, any user can access a specific virtual machine through a browser. And here Citrix is also very happy: any browser can be used to work, and not some limited set, like the cloud from Microsoft (only InternetExplorer) or VmWare (Opera is not supported). In order for the user to access the virtual machine,
The big disadvantages of working through the web-interface include the inability to configure the physical parameters of the virtual machine (number of processors, amount of RAM, setting up connected networks and hard drives). So the web-interface is access to the graphical interface of the virtual machine, and not to the management of its physical parameters. We performed all the necessary actions: prepared the equipment, set up licenses, deployed ... And now the cloud is ready to go!
To assess the real computing power of the servers, we conducted an experiment, the purpose of which was to load all 80 logical cores on the system to the maximum. As the basis for the experiment, we took a program that performs Ray-Tracing of a simple scene, without an operating system and uses all the cores on all processors in the computer. You can read about how this program works and how to get the source codes of this program here .
For the experiment, the program was slightly modified: we added motion animation for one of the spheres in the figure, added a speed calculation, and buffering was added when drawing each frame. To compare the power of the obtained program, we ran it on several computers of different configurations, including our IBM servers. In the experiment, a scene was rendered from 5 spheres in a resolution of 800x600. The experiment was successful and the IBM server showed impressive performance. For all experiments, we recorded a video where the green numbers in the upper left corner of the screen indicate the number of frames per second (FPS), the red numbers indicate the number of seconds per frame. Here are the results we got:
1. Ordinary computer: Intel i3-2100, 3.1 GHz, only 2 cores. For each core, 800x600 / 2 = 240000 points per frame. As you can see from the video, the speed was about 0.5 FPS (more than 2 seconds were spent on one frame).
2. Computer with a modern powerful processor: Intel i7-4770, 3.4 GHz, 8 cores in total. For each core, 800x600 / 8 = 60,000 dots per frame. The result is approximately 2 frames per second, as can be seen in the video.
3. IBM server from the rack: Intel Xeon E7-8860 2.3 GHz, each computer has 4 processors with 10 physical cores (2 logical cores on each core) - a total of 80 cores. For each core, 800x600 / 80 = 6000 dots per frame. The result is 12-14 FPS - which is significantly more than other systems.
Interestingly, if you run rendering on a IBM server in 1280x1024 resolution and let the processor cores work without buffering, you can see how the frame is drawn from 80 stripes!
That's what we got. We hope that after reading our article, you can easily make the cloud yourself, avoiding the problems that we described here, or successfully coping with them!
In this article, we will focus on such issues as: preparing the hardware, installing XenServer, installing a license, creating a virtual network infrastructure, describe the problems encountered with virtual machines on the Ubuntu OS, and talk about dynamic load balancing, about settings and differentiation of access to the cloud , and, of course, we will show what we did.
Hardware preparation
The first task is to choose the basis for any cloud, namely, the choice of servers on which virtualization will be performed. We chose IBM servers and chose the IBMx3850 X5.
Each IBM is based on the IBM X-Architecture and has:
• 4 Intel® Xeon® CPUE7- 8860 processors with a clock frequency of 2.27GHz, which ultimately gives 40 cores per server (80 threads);
• 150GB RAM;
• 2 independent power supplies;
• fiber channel expansion card ;
• network card for a 10-gigabit connection;
• 2 hard drives of 500 GB, combined in RAID1 .
Then the question arises: where to store virtual machines? If they are stored on the servers themselves, this reduces the reliability of the system, because when the server fails, we lose all the virtual machines that were on it. Also, this approach greatly complicates the task of load balancing, because the migration of a virtual machine will require moving its disk to another server, and this is a rather long process. Therefore, our stand uses an external DELL md3620f storage equipped with 4 fiber channel outputs. This storage supports up to 24 hard drives, which can be combined into all popular types of raids (RAID0, RAID1, RAID5, RAID6, RAID01). In our case, we use 10 hard drives of 1 TB combined in RAID5.
What is required for quick migration? To ensure fast migration between IBMs, a 10 gigabit summit x670 switch was added to the stand, this theoretically should have accelerated migration (the longest process in migration is transferring data over the network from one server to another) by 10 times, but in practice it only won 5-6 times. To enable servers and virtual machines to have access to the local network and the Internet, the HP ProCurve switch was added to the stand, as well as through it traffic to external clients.
To summarize the hardware, we have a stand that includes:
• 4 IBMx3850 X5 servers
• HP ProCurve switch for connecting up to 1000 Mb / s
• Summit x670 switch for supporting 10 Gbit / s connection
• external storage DELL md3620f with 10TB disk array
• and more (uninterruptible power supply APC Smart-UPS 3000VA, transceivers for fiber, eight ten-meter fiber optic cables, 50 meters twisted pair)
All this looks like in the picture, there are five uninterrupted sources below power supply, which in case of failure of the central power supply, will correctly stop everything running on the server and turn off the servers themselves.
The logical connection diagram is as follows:
If summarized, we have:
• 160 computing cores (320 threads);
• 600GB RAM;
• 14TB of disk space;
• High-speed network infrastructure.
Install XenServer
To run virtual machines on our chosen hardware, it was decided to deploy the XenServer Platinum cloud platform.
Brief general information about XenServer, if anyone has not encountered: XenServer is an independent operating system based on the Linux kernel. At the heart of everything is the XEN hypervisor, the latest version at the moment - the used version of XenServer 6.1 - is based on the XEN 4.1 hypervisor. For normal operation of the system, a processor with support for virtualization and 1GB of RAM is required. Citrix guys do not like to write their minimum requirements, preferring to write maximum system requirements, you can find them here . The XenServer family includes several product versions that differ in price and additional features .
All IBM servers must have XenServer installed. Installing XenServer itself is a completely straightforward process and is not very different from installing Ubuntu. Much more interesting is setting up the system for subsequent convenient operation, which will be discussed later, the main focus will be on the extension components for XenServer that Citrix provides. After installing XenServer on all hosts, you need to go to XenCenter (you can download it by going to the web interface of the installed XenServer or from the Citrix website). In XenCenter, you need to create a pool and add a server to it.
License Installation
After installing XenServer, the first thing you need to do is install the server licenses. This is done not by entering license keys, but by installing a certificate received in your account on the Citrix website. The certificate is a text file with license keys recorded in a special format. It can be obtained at once on several servers. The certificate must be installed in a special licensing center. It is worth noting that if at least one of the certificates is not valid, then all certificates will not be valid.
Citrix suggests installing a licensing center either on a separate host or on a virtual machine in your cloud (it consumes almost no CPU resources and requires only 256 MB of RAM). But if you install it as a virtual machine, you may be in trouble. One day you may encounter a problem that we encountered when we turned off the servers of our stand for their upgrade, and with them turned off, respectively, a virtual machine with a licensing center. After switching on, XenServer generated an error “The trial license has expired” (the licensing center wasn’t launched after all). It seems there shouldn't be any problems: just turn on the virtual machine of the licensing center with a normal license and accept it. But there it was: with an expired license, you can’t turn on ANY virtual machine. True, after the initial panic, the button “activate the free version” was found in the license acceptance menu, after clicking which we saw the phrase “you have 30 days of a trial license”. And only then you can start a virtual machine with a license.
Creating a virtual network infrastructure
To flexibly manage the network settings of virtual machines and not to be on the same subnet as XenServer servers, you need to establish a virtual network infrastructure. For these purposes, Citrix has a Distributed Virtual Switch Controller (DVSC) component. It is also provided as a virtual machine, but it already consumes more resources: 2 VCPU and 2GB of RAM. DVSC settings are not complicated: you need to specify a free IP address for the virtual machine, and then add the pool to DVSC. All this is done through the web interface. After these actions, it becomes possible to create virtual networks between virtual machines located on different servers (Cross-ServerPrivateNetwork),
If you go to the DVSC web interface, you can see a lot of useful information about network infrastructure: a list of created networks, a list of virtual machines connected to a specific network, network error messages, network activity graphs (both for a specific virtual machine and for the entire network as a whole).
DVSC can also act as a simple firewall, the rules of which can be created in the AccessControl tab. All network security policies are divided into 4 levels: global policies (as multiple pools can be connected to one DVSC), pool policies, specific network policies, and specific virtual machine policies. The rules describe the type of action (allow or deny), protocol (you can specify the recipient and sender ports or specify a known protocol) and for whom to apply this rule.
The firewall enters work immediately after installation, and it immediately contains 4 basic global rules:
• allow all ARP messages on the network
• allow virtual machines to receive an IP address via dhсp
• allow virtual machines to access DNS servers
• allow all network traffic (this applies after checking all the rules of other levels)
generally DVSC all good, and functionality is a bit like vShield Edge from VmWare, but Edge seems more convenient due to different things: the possibility of creating the DHCP-server, user-friendly virtual organization for Nat s machines, etc. All this (the absence of a DHCP server and Nat), of course, is solved by creating a separate virtual machine based on Ubuntu, but it’s convenient when everything is already decided in advance.
Problem creating a virtual machine with Ubuntu
In general, creating virtual machines with Ubuntu for the first time causes shock and misunderstanding - how did this get into the release version of the product? Here's the thing:
After creating a virtual machine from a standard template, it cannot start and writes the above error (Error code: INVALID_SOURCE). The error is related to the boot parameters of the virtual machine. You can deal with it as follows (the description is taken here and slightly modified to work with a large number of virtual machines):
1. Go to the XenServer console, which can be done through XenCenter (the Console tab on the server) or via ssh.
2. Find out the uuid of the virtual machine with the command xe vm-listname-label = [VM_NAME]. In our example, it looks like this:
3. Next, you need to set the boot parameters with the following command: xe vm-param-setuuid = [UUID_VM] HVM-boot-policy = BIOS \ orderHVM-boot-params: order = dc.
4. After these simple manipulations, the virtual machine will start successfully.
But this is not the end of Ubuntu-related errors. When creating our stand, it was decided to create virtual machine templates with different OSs, so as not to waste time installing the necessary one, but to immediately take the finished and put the necessary software there. There are no problems with Windows-based machines, while with Ubuntu there is a black screen problem when creating a virtual machine from an image. The solution to this problem turned out to be quite simple, on the one hand, and a little wrong, on the other. The problem is solved by simply installing xen-tools on a virtual machine. The downside of this solution is that it is impossible to provide a clean operating system, which is sometimes required as part of the tasks being solved.
Dynamic load balancing
As part of the tasks being solved, dynamic load balancing between servers with XenServer is often needed. For these purposes, Citrix provides the Citrix WLB Virtual Appliance virtual machine, which also needs to be added to the cloud, after which it is easy to configure through its console (when entering the console, the machine will tell you all the necessary actions). After that, you need to go to XenCenter and indicate to the pool that this particular virtual machine will be responsible for balancing the load between the servers (this action is performed in the WLB tab).
This virtual machine monitors the load on the server (the number of cores used, the amount of RAM used, network activity) and distributes it between the servers. This happens both when the virtual machine is turned on (it runs on the most unloaded server) and during its operation (due to migration).
Configure and restrict access to the cloud
The last task that needs to be solved for normal operation is providing access to the cloud. And here, Citrix, in our opinion, has the biggest problems. Citrix offers two options for accessing the cloud: through XenCenter and through the web-based interface.
Access through XenCenter
If you connect Active Directory (AD) to the pool, you can create users in XenCenter. Citrix decided to abandon the discretionary access model at XenCenter and implemented a role-based model. Hence the main problem: ALL users see and have access to ALL virtual machines, only the type of access is regulated, but it applies to ALL virtual machines immediately (i.e. if you are given a role to start virtual machines, then all at once). It is also worth noting that AD should always be available, as upon reboot, AD is not automatically added to the pool, and you must add it manually each time.
Web-based access
Citrix suggests using web-based access for discretionary access. To configure access via the web-based interface, you must install the Citrix XenServer Web Self Service virtual machine. After simple configuration of the virtual machine through its console (you must specify an IP address or specify that it is obtained via DHCP), you need to perform a number of settings through the web interface. Here Citrix is beyond praise for an accessible and understandable description: if you are logged in as an administrator, you will immediately be shown a list of steps that must be completed, as well as a detailed description of how to do this.
Citrix XenServer Web Self Service can use the same users that are in AD, or create new ones. When the XenServer Web Self Service is first loaded, the administrator needs to specify how he wants to act, and this decision cannot be changed later (of course, you can always rearrange the virtual machine, but this will entail a new setting of access rights to the virtual machines). After configuration, any user can access a specific virtual machine through a browser. And here Citrix is also very happy: any browser can be used to work, and not some limited set, like the cloud from Microsoft (only InternetExplorer) or VmWare (Opera is not supported). In order for the user to access the virtual machine,
The big disadvantages of working through the web-interface include the inability to configure the physical parameters of the virtual machine (number of processors, amount of RAM, setting up connected networks and hard drives). So the web-interface is access to the graphical interface of the virtual machine, and not to the management of its physical parameters. We performed all the necessary actions: prepared the equipment, set up licenses, deployed ... And now the cloud is ready to go!
Our experiment
To assess the real computing power of the servers, we conducted an experiment, the purpose of which was to load all 80 logical cores on the system to the maximum. As the basis for the experiment, we took a program that performs Ray-Tracing of a simple scene, without an operating system and uses all the cores on all processors in the computer. You can read about how this program works and how to get the source codes of this program here .
For the experiment, the program was slightly modified: we added motion animation for one of the spheres in the figure, added a speed calculation, and buffering was added when drawing each frame. To compare the power of the obtained program, we ran it on several computers of different configurations, including our IBM servers. In the experiment, a scene was rendered from 5 spheres in a resolution of 800x600. The experiment was successful and the IBM server showed impressive performance. For all experiments, we recorded a video where the green numbers in the upper left corner of the screen indicate the number of frames per second (FPS), the red numbers indicate the number of seconds per frame. Here are the results we got:
1. Ordinary computer: Intel i3-2100, 3.1 GHz, only 2 cores. For each core, 800x600 / 2 = 240000 points per frame. As you can see from the video, the speed was about 0.5 FPS (more than 2 seconds were spent on one frame).
2. Computer with a modern powerful processor: Intel i7-4770, 3.4 GHz, 8 cores in total. For each core, 800x600 / 8 = 60,000 dots per frame. The result is approximately 2 frames per second, as can be seen in the video.
3. IBM server from the rack: Intel Xeon E7-8860 2.3 GHz, each computer has 4 processors with 10 physical cores (2 logical cores on each core) - a total of 80 cores. For each core, 800x600 / 80 = 6000 dots per frame. The result is 12-14 FPS - which is significantly more than other systems.
Interestingly, if you run rendering on a IBM server in 1280x1024 resolution and let the processor cores work without buffering, you can see how the frame is drawn from 80 stripes!
That's what we got. We hope that after reading our article, you can easily make the cloud yourself, avoiding the problems that we described here, or successfully coping with them!