VMware, with the release of VMware vSphere 5.1, announced several new ventures in the storage of virtual machine data, including the ability to use a distributed cache on SSD drives on local disks of ESXi servers. This technology had the working name vFlash and was in the Tech Preview stage, later turning into a full-featured vSphere Flash Read Cache (vFRC) of the VMware vSphere 5.5 platform. And this is quite a working tool that can be used in tasks of various levels.

VFRC technology is designed to increase the efficiency of client applications interacting with the disk subsystem by caching read data, while significantly increasing the performance of applications that are actively performing read operations.

Flash Read Cache works at the hypervisor level, while significantly improving the performance of virtual machines that heavily use the I / O system for read operations. For caching, PCIe flash cards and SAS / SATA SSDs locally installed in the host can be used. Devices are combined into a flash pool, from which VMDK disks of virtual machines are allocated space for data caching.

Recall that VMDK (Virtual Machine Disk) is a file format developed by VMware for use as a disk image in its virtual machines.

In terms of performance, the SSD cache is located between RAM and regular disks.

VFRC Architecture Overview

The architectural feature of vFRC is as follows:

When a read request arrives at a VMDK drive with vFRC enabled, it is first determined if there is any required data on vFlash.

• If so, then the virtual machine receives data from the cache. This event is called a “hit” (vFRC hit).
• If the data is not in the cache, then ESXi reads it from the VMDK-disk and gives it to the machine, while writing data to the cache. This event is called a miss (vFRC miss). When a write request arrives, data is written to the VMDK disk and asynchronously to the cache.

What is needed for vFRC?

In order to activate the vFRC functionality, the following conditions must be met:

• Have a configured host with at least one SSD or PCIe SSD.
• Use vSphere 5.5 (vCenter 5.5 and ESXi 5.5).

How does vFRC turn on?

After physically connecting the device to the server with ESXi, you need to add it to the vSphere Flash Infrastructure layer. You can do this on the Virtual Flash Resource Management tab in the host settings.



To enable vFRC in a virtual machine, the Virtual Flash Read Cache item is used in the hard disk settings, where you can specify the amount of allocated space for caching and the block size. The block size for vFRC should be selected depending on which blocks the application writes data to disk. Block statistics for each drive can be collected using the vscsiStats utility on ESXi.

Configuration Features

When configuring vSphere Flash Read Cache, consider the following points:

• VMware ESXi 5.5 in Enterprise Plus edition must be installed on the host server.
• VFRC is configured and managed only through the vSphere Web Client, so VMware vCenter Server is required.
• The maximum cache size for one virtual disk is 400 GB.
• The maximum cache size per host is 2 TB.
• The maximum size of a virtual disk is 16 TB.
• The maximum number of SSDs used for cache is 8.
• It is required to update the Hardware Version of the virtual machine to version 10.
• You must manually configure the cache for each virtual disk, the minimum value is 1 GB.

vFRC in practice in IT CITY

And a little from personal experience. How did we test the SSD cache in the VMware cloud and what pitfalls did we encounter in practice?

Before offering a solution to customers, you need to make sure that it is fully functional and functional, which means you need to test and understand that the result is fully consistent with the declared capabilities and satisfies the requirements of the customer. And if there is a novelty that no one has yet tested and implemented, it should be tested with special close attention. After all, it is no secret to anyone that in everything new there may be hidden flaws, bugs and other unpleasant trifles.

As soon as VMware announced the new possibility of using distributed cache on SSD-drives of local disks of ESXi servers, we decided to test this functionality. Since this technology was in the Tech Preview stage before vSphere 5.5 was released, I wanted to test an already developed solution. Our task was to check the vFRC performance at the constructed stand.

For testing, SSDs were connected to a Dell PERC H710P RAID controller. RAID-0 groups were created by the number of SSD disks, in each group one disk.



Since the Dell PERC H710P RAID controller cannot provide information about the physical type of media connected to it, I had to manually note that the drives connected to ESXI are SSDs. To do this, run the esxcl command:



After running the command in the device parameters, the “Is SSD” flag value changed to “True”:



After that, the devices were added to the vSphere Flash Infrastructure layer. The current procedure was performed in the host settings using the Virtual Flash Resource Management option:



In advance, to test the SSD cache, we prepared a stand with virtual machines based on Windows Server 2008 R2 x64 and two 100 GB each VMDK disks allocated for each virtual machine:

• VMDK1 identified under the OS,
• VMDK2 - for data.

Further, in the parameters of the VMDK2 hard disk of virtual machines, vFRC was turned on, allocating 100 GB for the cache, while determining the block size of 4 KB.



Basic configuration steps completed. Next was the task of verifying the functionality of the included functionality. However, when starting the virtual machines, one simply refused to start, instead of the welcome window, a “blue screen” appeared with the following contents:



There were no obvious problems on the other virtual machines.
Then we decided to use the tools sharpened for monitoring the SSD cache and compare the test results. To begin with, one of the virtual machines launched the FIO utility, which generates the necessary amount of data to the VMDK2 disk. As mentioned earlier, it was he who was allocated for useful data. The FIO utility can work in various modes, we were interested in the "random read" procedure. That is why they launched it in rand-read mode.

Note: More information about the FIO utility can be found at http://freecode.com/projects/fio .

The FIO utility implies the use of a job-file (or, more simply, a configuration file), in which the parameters for testing are written. The utility performs read operations on randomly generated VMDK2 disk data. In the configuration file for reading, the size of the reading block is fixed (in our case, equal to 4 Kb). Then they started the random read operation. The test time was 6 hours 46 minutes.



I was interested in the question: did the read data hit the cache and if so, what is the percentage of hit?
To find the answer, we used the performance graph of the virtual disk of the machine using the vSphere WEB client.



It was interesting to look at the following counters: the average number of output operations per second, read delay, and a counter that provides statistics on cache usage. The latter was a little disappointing, showing a very small percentage of data getting into the cache. With an average number of output operations per second (18,689.328), the value for cached data was 4439.389, which is only 23% of the hit. According to this statistical scenario, cache can simply be considered idle.

Since the standard tool did not show the expected results, we turned to another tool: the esxcli command. It also works with statistics for a specific VMDK drive with the vFRC option enabled. We ran the command with the following parameters:
~ # esxcli storage vflsh cache stats get



In this figure, you can see the cache hit rate value, presented as a percentage. It shows the so-called “hit” vFRC hit, that is, the percentage of data from the cache that is used by the virtual machine. The team in question had to be run several times, since the results at the next start were completely different. For one value, the cache did not work at all, as in the first case, for others it worked, with a percentage of data getting into the cache equal to 96%.

They didn’t dwell on what they got, they used another utility: esxtop (with sending the interactive command “u” (u: disk device)) to display statistics on cache usage. According to the information displayed on the screen, we got the following result: when reading, the data was extracted directly from the cache. Given that the average number of output operations per second was 18 689.328, and the volume of operations for data read from the SSD cache was 18 184.03, the percentage of data getting into the cache was approximately 97%.



The test results did not fully meet our expectations, and we, as a major service provider, VMware partner, turned to vendor-side colleagues for help.

VMware has a wealth of experience interacting with its customers and partners. In case of detection of bugs, bottlenecks and other points regarding the product’s functionality, the developers make every effort to make the necessary corrections.

As a result, in the fall of 2014, VMware ESXi 5.5 Update 2 was released, which fixes the described problem on the blue screen of a virtual machine running Windows Server 2008 R2 x64.

The released update, of course, interested us. We decided to test it by installing it on the previously reviewed test site with vFRC enabled. What is the result? All virtual machines started as one. We put "+" in this test and move towards the counter indicators. As at the very beginning of testing, we launched the FIO utility in rand-read mode with the configuration file used earlier, after which we started the random read operation. The counters for the most part showed working statistics and only periodically indicated incorrect values. That is, VMware ESXi 5.5 Update 2 did not fix the described problem of displaying vFRC statistics. Despite this bug, the vSphere Flash Read Cache technology, as subsequent practice has shown using this functionality,

After the next tests, we moved on to implementing the technology of SSD caching on hosts in an industrial environment. Today, several projects using vSphere Flash Read Cache have been successfully implemented at our sites for our customers especially demanding on performance. The latter, in turn, are satisfied with the results of accelerating the work of their systems and applications.
You can read about other caching mechanisms in the article: “SSD Caching in the VMware Cloud” in the first blog about corporate IaaS .