Increased disk subsystem performance in the next release of the XenServer hypervisor

Transfer. The original article is available on the Xen Blog .
Author - of Felipe Franciosi .

Recent test builds of XenServer Creedence Alpha are characterized by increased disk subsystem performance compared to XenServer 6.2 (see Marcus Granado's blog for details - Performance Improvements in Creedence ). Basically, the improvements are related to the introduction of a new architecture of the disk subsystem - tapdisk3. We will describe this technology for organizing virtual storage, as well as present and explain the results of experiments in which the performance of approximately 10Gb / s is achieved on a single host with connected cluster storage.

A few months ago I wrote about the Karcygwins project, which included a series of experiments and studies aimed at studying the features of disk I / O. We focused on the case when the load is generated by one VM with one virtual disk. In particular, we were interested in understanding the nature of the overhead for virtualization, especially on devices with low latency, for example, modern SSDs. By comparing the different disk I / O virtualization methods available to the user (that is, blktap and blktap2), we were able to explain where and why additional costs arise, as well as how to significantly reduce them. You can read more about the Karcygwins project here.

From this moment we expanded the scope of research to cases with a more complex load structure. We were especially interested in the joint load from several VMs that completely loaded the storage. We studied the features of the new tapdisk3 subsystem developed for XenServer by Thanos Makatos. Tapdisk3 is written to simplify the architecture and bring it completely into the user's space, and this, in turn, leads to a significant increase in productivity.

There are two significant differences between tapdisk2 and tapdisk3. The first is the way that tapdisk connects to the disk I / O subsystem: the old tapdisk accesses the blkback and blktap2 devices inside the hypervisor, and the new one directly accesses the paravirtual driver inside the VM. The second is the method by which the guest VM and the hypervisor exchange data: the old tapdisk uses access to the displayed pages of memory in the VM's memory and their subsequent copying to the address space of the hypervisor, while the second uses grant copy technology.

All other modifications are necessary in order to make such a change in architecture possible. Most of them affect the level of control of virtual machines (control plane). To do this, we changed the control stack (xapi) so that it maintains constant communication with the tapdisk module. Due to these changes (and also due to some others related to how tapdisk3 processes incoming data), the way of representing a virtual disk in the hypervisor memory has changed. Since tapdisk3 has not yet been officially released, other changes are possible in the future.
To measure the performance achieved with tapdisk3, we selected the fastest server we have and the fastest storage.

• Dell PowerEdge R720 Platform
• 64 GB RAM
• Intel Xeon Processor E5-2643 v2 @ 3.5 GHz
• 2 sockets, 6 cores per socket, hyperthreading = 24 pCPU
• Turbo Frequency - 3.8 GHz
• The hypervisor CPU scheduler is set to Performance mode (set to On Demand by default to save power). Rachel Berry on her blog described the work of the scheduler in more detail.
• The BIOS is set to "Performance per Watt (OS)", the mode "Maximum C-State" is set to 1
• 4 x Micron P320 PCIe SSD (175 GB each)
• 2 x Intel 910 PCIe SSD (400 GB each)
• Each of them is presented as 2 SCSI devices of 200 GB each (total - 4 devices and 800 GB in total)
• 1 x Fusion-io ioDrive2 (785 GB)

After installing XenServer Creedence with build number 86278 (about 5 numbers older than XenServer Creedence Alpha 2 ) and Fusion-io disks, we created storage on every available device. It turned out 9 storages and approximately 2.3 TB of free space. On each storage, we created 10 virtual disks in RAW format of 10 GB each. We connected each virtual disk to its virtual machine, selecting disks “in a circle”, as shown in the diagram below. Ubuntu 14.04 (x86_64 architecture, 2 logical CPUs, not rigidly attached to real ones, 1024 MB RAM) was chosen as the guest OS. We also transferred 24 logical CPUs to the hypervisor and decided not to associate them with real ones ( XenServer 6.2.0 CTX139714 describes in more detail the method of binding logical CPUs to real ones).



First, we measured the aggregate channel performance between the hypervisor and the virtual machine if the disks are connected in the standard way tapdisk2 <-> blktap2 <-> blkback. To do this, we forced one virtual machine to send 10-second write requests to all of its disks at the same time, and then calculated the total amount of data transferred. The request size ranged from 512 bytes to 4 MB. After that, we increased the number of virtual machines to 10, and then replaced the write requests with read requests. The result is shown in the graphs below:





Measurements showed that virtual machines are not able to access the disk with a speed of more than 4 Gb / s. Then we repeated the experiment using tapdisk3. The result has clearly improved:





In the case of writing, the total throughput of the disk subsystem for all VMs reaches 8 Gb / s, while in the case of reading, it is 10 Gb / s. It follows from the graph that in some cases the performance of tapdsik3 is greater than the performance of tapdisk2 by about 2 times for writing and 2.5 times for reading.

To understand why tapdisk3 is so superior to tapdisk2 in performance, you should first consider the architecture of the virtual storage subsystem, which is used by paravirtual VMs and the hypervisor. We will focus on the components that XenServer and a regular Linux VM use. Nevertheless, it should be borne in mind that the information described below is also relevant for VMs running Windows if this OS uses installed para-virtual drivers.

Typically, a guest VM running Linux loads a driver called blkfront at startup. From the point of view of the guest VM, this is a common block device. The difference is that instead of interacting with real equipment, blkfront communicates with the blkback driver in the body of the hypervisor via shared memory and the event channel mechanism, which is used to send interrupts between domains.

Applications inside the guest OS initiate read or write operations (via the libc, libaio libraries, etc.) of files or directly block devices. Operations are ultimately translated into requests for block devices and passed to blkfront through randomly selected memory pages in the guest address space. Blkfront, in turn, provides the hypervisor with access to these pages so that blkback can read and write to them. This type of access is called “grant mapping”.



Despite the fact that the Xen Project development community is conducting a campaign to improve the scalability and performance of the mechanism for accessing displayed pages, there is still a lot of work, as the system is complicated and has a number of limitations, in particular regarding shared access to storage from several VMs. The most notable recent change is a set of patches from Matt Wilson to improve the locking mechanism and better performance.

To reduce the additional cost of allocating and freeing memory for each request in the mechanism for accessing the displayed pages, Roger Pau Monne implemented a new feature in the blkback / blkfront protocol called persistent grant. Such access can be used if both domains (hypervisor and VM) support it. In this case, blkfront gives blkback access to a permanent set of pages, and both drivers use this set for as long as they can.

The flip side is that blkfront cannot indicate which pages will be associated with the request coming from the block device level of the guest VM. In any case, he has to copy the data from the request to this set before transferring the request to blkback. However, even taking this copy operation into account, access to fixed addresses remains a good method for increasing scalability with simultaneous input and output from several VMs.



Both of the above changes are fully implemented in the core space of the hypervisor. However, we do not take into account that the request to the block level of the hypervisor contains a link to pages in the guest address space. This can cause a race condition when using network storage, such as NFS, and possibly iSCSI: if a network packet (which contains a pointer to the exchange page) is queued for retransmission and confirmation of the transmission of the original packet arrives, the hypervisor may send incorrect data again or even crash because the exchange page may either contain incorrect data or be freed altogether.

To avoid problems, XenServer copies pages to the memory of the hypervisor instead of using them directly. This feature first appeared in the blktap2 driver and the tapdisk2 component, along with thin-provisioning technologies and moving VM disks between storages (Storage Motion). As part of this architecture, blktap2 copies pages before transferring them to tapdisk2, to ensure the safe operation of network attached storage. For the same reason, blktap2 provides the hypervisor with a fully functional block device for each virtual disk, regardless of its nature (including “truncated” thin-provisioning disks located on NFS storages).



As we can see from the results of the above measurements, this technology has limitations. It is good for various types of classic storages, however, it demonstrates poor performance when working with modern storage means, for example, SSD disks connected directly to the server via PCIe bus. To take into account the latest changes in storage technologies, XenServer Creedence will contain a new component - tapdisk3, which uses another way to access memory - grant copy.

Starting with the kernel version 3.x as the hypervisor domain (dom0) and the advent of the access device (gntdev), we gained access to the pages of other domains directly from the user space of the hypervisor. This technology is implemented in tapdisk3 and uses the gntdev device as well as the evtchn event channel to communicate directly with blkfront.

Copying upon access is much faster than simply providing access to permanent addresses, and then copying. With this access, most of the operations are performed inside the Xen Project Hypervisor core, and in addition, the presence of data in the address space of the hypervisor domain is ensured for the safe operation of network attached storage. Also, since the logic is fully implemented in the user space, the inclusion of additional functionality (such as thin-provisioning, snapshots or Storage Motion) does not create special difficulties. To ensure access of the hypervisor to the block device corresponding to the disk of the virtual machine (for copying the disk and other operations) we connect tapdisk3 to blktap2



Last (but not least) what we wanted to write about: a sophisticated reader may ask why XenServer does not use qemu-qdisk, which implements access technology at fixed addresses in user space? The fact is that, to ensure the safe operation of network storage (including access to fixed addresses, at which the storage request refers to pages in the guest VM's memory), qemu-qdisk removes the O_DIRECT flag when accessing virtual disks. This leads to the fact that the data is copied to the cache of the hypervisor domain, which ensures secure access to the data. Thus, access to permanent addresses in this case leads to excessive copying and to delays in servicing requests. We believe access-based copying is a better alternative than qemu-qdisk.