NetApp FAS and VMware ESXi: Swap

    Continuing the topic of host optimization with VMware ESXi , we’ll look at how to deal with Swap in the infrastructure of NetApp FAS storage . Although this article should be useful not only to owners of NetApp FAS systems . One of the most important features of virtualization is the ability to more efficiently utilize server hardware, which implies Overcommit resources. If we are talking about RAM

    , this means that we can configure each virtual machine more memory than it actually is on the server. And then we rely on ESXi to smooth out the struggle for resources - take away (this process is often called reclamation) the unnecessary memory of one virtual machine and give it to the one that really needs it. At that moment when there is not enough memory, the process of swapping memory begins.

    To begin with, there are two types of swap that can occur on an ESXi host. They are often confused, so let's conditionally call them Type 1 and Type 2.

    VMware ESXi default data location

    Type 1. VMX swapping (vSwap)

    This is the easiest type of swap to describe. Memory is allocated not only for virtual machines, but also for various host components, for example, for virtual machine monitor (VMM) and virtual devices. Such memory can be swaped in order to allocate more physical memory to more needy virtual machines. According to VMware, this feature can free up from 50MB to 10MB of memory for each virtual machine without significant performance degradation.

    This type of swaping is enabled by default, VMX Swap is stored in a folder with a virtual machine as a file with the vswp extension (the location can be changed using the sched.swap.vmxSwapDir parameter ). Vmxswapping can be turned off by setting the virtual machine parameter sched.swap.vmxSwapEnabled = FALSE .

    Type 2. Guest OS Swapping

    The virtual machine “sees” as much supposedly “physical” memory as the RAM is set in its settings. From the point of view of the guest OS , all such allocated memory for it is physical. In fact, the hypervisor hides behind the virtualization layer, for the guest OS , what kind of memory is actually used: physical or vSwap. When the memory "inside" of the guest OS ends, it starts to swap itself. And here you need not to get confused - this process has nothing to do with host-level swapping ( VMX ). For example, for a virtual machine with 4GB of memory configured, the hypervisor can allocate this memory on the host, both in physical memory and in the host swap ( VMX) - for a virtual machine, this is not important, all 4GB will be used as physical memory simply because the guest OS does not have a clue that its memory is virtualized.

    When the guest OS wants to use more than 4GB of memory, it will use its own swap, Windows uses pagefile.sys, in Linux it is a dedicated Swap partition. Thus, the swap is saved inside the virtual disk - inside one of the VMDK files of the virtual machine. Those. The pagefile.sys / swap partition is part of the virtual disk of a virtual machine.

    And this is what happens when the baluning driver is used: when it “swells”, it clogs all the free (from the point of view of the guest OS) memory, while the balancing driver makes sure that it actually receives physical, not swaped host memory, and when such a virtual machine does not have free memory, this forces the guest OS to start swapping data. And this is good, since the guest OS knows better what to swap to disk and what not.

    ESXi Host swapping

    When we understand that the ESXi host will swap a lot, this means that we have spent a lot of resources, all reclamation and techniques for saving memory (ballooning, page sharing and memory compression) do not save - the host simply does not have enough RAM . It is believed that the host is in the “Hard state” state (1% -2% of memory is free) or “Low state” (1% or less). This state of affairs significantly worsens performance and should be avoided. Another problem is when Swapping at the host level does not have control over what gets there - unlike swapping the guest OS (Type 2), the hypervisor does not know what data is more important for the guest OS , so it is important to install VMware Tools so that the driver works baluning.

    vSwap (Type 1) is saved by default to a .vswp file in a folder with a virtual machine. NetApp recommends saving vSwap (Type 1) to a dedicated datastore.

    /vmfs/volumes/4b953b81-76b37f94-efef-0010185f132e/vp # ls -lah |grep swp
    --rw------    1 root    root    2.0G    Jan 11 18:02    vp-c6783a5c.vswp

    Such a file exists only for running virtual machines - at startup, the hypervisor creates this file and deletes it when the virtual machine is turned off. Pay attention to the size of this file - it is equal to the difference between the configured memory for the virtual machine and the physical memory reserve for this machine. This ensures the presence of a given memory space when the virtual machine starts (regardless of which memory is used - real physical or vSwap). The reserve takes care that the virtual machine always gets the necessary space in the physical memory of the host. And the difference between the reserve and the configured memory of the virtual machine is saved in a vswp file.

    Below is an example of starting a virtual machine with 4GB of memory and a reserve of 2GB. Remember that vSwap is deleted after shutdown? And if at that moment the datastore storing this virtual machine is full and 1.8GB of free space remains, what will happen?

    What can be done about this? Of course, you can free up space on the datastore or place vSwap (Type 1) on another datastore. By default, it is stored in a folder with a virtual flank. If it is a “standalone” host: Configuration> Software> Virtual Machine Swapfile Location . The location of vSwap files can also be changed at the settings level of each virtual machine. Open the virtual machine settings and select the Options tab:

    If this host is part of a cluster, look at the cluster settingsStore the swapfile in the datastore specified by the host

    Next, select the datastore where vSwap will be stored (Type 1).

    Why put a swap on a separate datastore?

    Recommended Swap Layout for NetApp FAS

    First, it’s worth noting that the general rule of VMware says to place vSwap (Type 1) in a folder with a virtual machine whenever possible, because putting it on a separate datastore can slow down vMotion. But in the particular case of VMware + NetApp FAS, another recommendation comes in. NetApp recommends storing vSwap (Type 1) on a separate dedicated datastore, one for all virtual machines, since in this case the difference during migration of vMotion is almost leveled. VMware recommends that the space under vSwap (Type 1) be greater than or equal to the difference between the memory configured and reserved for the virtual machine. Otherwise, virtual machines may not start. Example: we have 5 virtual machines with 4GB memory size and a reservation for each 3GB.

    Minimum datastore size for vSwap Type 1:
    5VM * (4 GB - 3GB) = 5GB

    If the host does not have free physical 5VM * 3GB = 15 GB of memory, the machines may not start, displaying the error:
    The host does not have sufficient memory resources to satisfy the reservation

    And at the same time Event log will be created

    Secondlythis is performance - by swapping on a slower datastore, if you are sure there will almost never be a swap. Or if the virtual machines will request more memory than there is on the host, and swap will be constant - on a faster datastore.

    Thirdly, these are snapshots and deduplication at the storage level. Since netap snapshots do not affect performance, they are the norm, they are performed continuously and are part of the NetApp backup paradigm for FAS systems. Swap is absolutely useless information when backing up and restoring. If it is used intensively, and snapshots are often removed, snapshots will be “captured” and stored, all the time the snapshot is alive, this useless information. And since the swap is constantly changing, and the sepshot captures only new changed data, each new snapshot will store these new changed data from the swap, mercilessly eating useful space for useless data in the storage. There is no reason to deduplicate swaps, because the data in the swap is not needed during recovery and will never be read. In this sense, it is convenient to swap to a separate datastore. NetApp recommends placing the swap on a separate datastore with snapshots and deduplication turned off, so we can further reduce overhead in snapshots and replication.

    NetApp does not recommend using local disks (pages 76-78, DEC11) on the host to store vSwap (Type 1) because this configuration has a negative effect on the speed of the vMotion operation.

    Should I Type 2 Swap?

    Placing temporary data, such as Swap and Pagefile.sys on a dedicated datastore (by creating a dedicated virtual disk for this purpose) allows you to not duplicate or backup this data as well as vSwap (Type 1). It is very important not to forget to specify this drive as Independent , so that the VSC agent does not force FAS to remove snapshots from the partition storing the Swap files (Type 2).

    NetApp has no recommendations regarding Swap Type 2, rendering this data is an optional design that has its pros and cons:

    In addition to the vSwap Type 1 rendered, the Swap Type 2 renderer will further reduce overhead on snapshots, and as a result will increase the throughput for replication. In ACB for DRThe presence of a Swap (both types) makes no sense, because it is not used at system startup. If we stop storing Swap (Type 2) in snapshots and backups, local recovery should not cause any problems, since the restored .vmx config file will contain a link to the VMDK file containing the partition with Swap, because it will be be in the same place. But this will add additional steps when recovering at a remote site, in configurations such as SRM , SnapProtect and other DR solutions.

    An optional data layout scheme on NetApp FAS

    So when restoring a virtual machine without a VMDK file storing a swap guest OS, you need to supplement the steps of creating a virtual disk, a file system on it and connecting it to a virtual machine (or connecting an existing VMDK with the created file system). More details TR-3671 .

    Please send messages about errors in the text to the LAN .
    Comments and additions on the contrary please comment

    Also popular now: