VMware ESXi 5.X and NetApp ONTAP 8: Tuning

В продолжение темы об оптимизации хоста для взаимодействия с СХД NetApp FAS, эта статья будет посвещена оптимизации производительности VMWare ESXi, предыдущие статьи были посвящены тюнингу ОСLinux и Windows в среде SAN. Компания NetApp давно тесно сотрудничает с VMware, подтверждением тому может стать тот факт, что нашумевшая технология vVOL была реализована одной из первых ещё в релизе Clustered Data ONTAP 8.2.1 (Август 2014), в то время как vSphere 6.0 ещё даже не выпущен. В связи с чем системы хранения NetApp FAS крайне популярны в этом окружении. Часть про Disk Alignment будет полезна не только владельцам NetApp.

VMWare ESXi settings can be divided into the following parts:

• Hypervisor Optimization
• Guest OS Optimization ( GOS )
• Optimal SAN Settings ( FC / FCoE and iSCSI )
• NAS Settings ( NFS )
• Checking hardware, firmware, and software compatibility

To search for a bottleneck, a sequential exclusion technique is usually performed. I suggest first things first to start with storage . And then move the storage -> Network ( Ethernet / FC) -> Host ( Windows / Linux / VMware ESXi 5.X and ESXi 6.X ) -> Application.

There are a couple of basic documents you should rely on when setting up VMware + NetApp:

TR-4068: VMware vSphere 5 on NetApp Clustered Data ONTAP
TR 3839: Using NFS in VMware (7-Mode)
TR-3749: Guide to Best Practices for Using NetApp Systems with VMware vSphere (7-Mode)
TR-3802: Ethernet for Storage: Best Practices (7-Mode)

Hypervisor

It’s not worth giving all the server resources to the guest OS , firstly, the hypervisor needs to leave at least 4GB of RAM , and secondly, the opposite effect is sometimes observed when adding guest OS resources , this must be selected empirically.

Swap

I will take out this section in a separate post .

Guest OS

Tuning settings is needed for two purposes:

• Guest OS Workload Speed ​​Optimization
• Normal operation in HA pair, in case of failure of one controller (takeover) and resumption of its operation (takeover)

Disk alignment

To optimize performance, you may need to eliminate disk misalignment . Misalignment can be obtained in two cases:

1. due to incorrect moon geometry when creating it in the storage system . This error can only be created in a SAN environment.
2. inside virtual disks of virtual machines. It can be both in SAN and in NAS environment

Let's look at these cases.
To begin, consider fully aligned blocks at the boundaries of the VMFS datastore and storage.


The first case is when there is a misalignment of the VMFS datastore relative to the storage. To fix the first type of problem, you need to create moons with the correct geometry and move the virtual machines there.


The second situation, with shifted file system partitions inside the guest OS relative to the WAFL file structure, can be obtained in older Linux distributions and OSWindows 2003 and older. Typically, this is due to poor placement of the MBR partition table or to machines that have been converted from physical to virtual. You can verify this in Windows guest OS using the dmdiag.exe -v utility (the value of the Rel Sec field must be a multiple of 4KB by WAFL ). Learn more about misalignment diagnostics for Windows machines. Also, the placement of the file system on the disk can be checked using the mbralign utility for the ESXi host , which is part of NetApp Host Utilities version 5.x and VSC . More details on how to eliminate such situations are described inTR-3747 Best Practices for File System Alignment in Virtual Environments .


And of course, you can get misalignment at once on two levels: both at the VMFS datastore level and at the file system level of the guest OS . Learn more about finding misalignment from the NetApp FAS repository .


An example for the VMFS3 file system. In the newly created VMFS5 (not an upgrade with VMFS3), the block has a size of 1MB with sub-blocks of 8KB.

takeover / giveback

To work with takeover / giveback in the HA pair, you need to configure the correct timeouts of the guest OS :

OS	Historical Guest OS Tuning for SAN: ESXi 3.x / 4.x and Data ONTAP 7.3 / 8.0 (SAN)	Updated Guest OS Tuning for SAN: ESXi 5 or later, or Data ONTAP 8.1 or later (SAN)
Windows	disk timeout = 190	disk timeout = 60
Linux	disk timeout = 190	disk timeout = 60
Solaris	disk timeout = 190; busy retry = 300; not ready retry = 300; reset retry = 30; max throttle = 32; min. throttle = 8	disk timeout = 60; busy retry = 300; not ready retry = 300; reset retry = 30; max throttle = 32; min. throttle = 8; corrected VID / PID specification

The default OS values in the case of using NFS are satisfactory, and the settings for the guest OS do not need to be changed.

These values ​​are set manually or using scripts available as part of VSC .
Windows : Set the value of the disk access delay to 60 seconds using the registry (set in seconds, in hexadecimal form).

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Disk]
"TimeOutValue"=dword:0000003c

Linux : Set the disk access delay value to 60 seconds by creating a udev rule (specified in seconds, in hexadecimal form).

DRIVERS=="sd", SYSFS{TYPE}=="0|7|14", RUN+="/bin/sh -c 'echo 60 > /sys$$DEVPATH/timeout'"

(Linux distributions may have a different installation location for udev rules). VMware Tools for a guest Linux OS automatically sets a udev rule with a delay value for a virtual disk of 180 seconds. You can run the grep command for “VMware” vendor ID in the udev rules folder to find a script that sets this value and change it if necessary. Remember to check this value.

Solaris : You can set the delay value to 60 seconds (specified in seconds, in hexadecimal form) for the disk in the / etc / system file :

set sd:sd_io_time=0x3c

Additional settings can be made to the file /kernel/drv/sd.conf :
Solaris 10.0 GA - Solaris 10u6:

sd-config-list="NETAPP  LUN","netapp-sd-config",
"VMware  Virtual","netapp-sd-config";
netapp-sd-config=1,0x9c01,32,0,0,0,0,0,0,0,0,0,300,300,30,0,0,8,0,0;

Solaris 10u7 and later and Solaris 11:

sd-config-list= "NETAPP  LUN","physical-block-size:4096,retries-busy:300,retries-timeout:16,retries-notready:300,retries-reset:30,throttle-max:32,throttle-min:8",
"VMware  Virtual","physical-block-size:4096,retries-busy:300,retries-timeout:16,retries-notready:300,retries-reset:30,throttle-max:32,throttle-min:8";

Please note: there are two spaces between vendor ID NETAPP and ID LUN, as well as between the words "VMware" and "Virtual" in the config above.

FC / FCoE Switch Zoning Settings

Learn more about zoning guidelines for NetApp in pictures .

Alua

For NetApp FAS systems with 7-Mode ALUA is recommended for FC / FCoE . And for NetApp FAS Systems with cDOT, ALUA is recommended for use with all block protocols: iSCSI / FC / FCoE .
ESXi will determine if ALUA is enabled . If ALUA is enabled, Storage Array Type plug-in will be VMW_SATP_ALUA , if ALUA is disabled, it is recommended to use the Fixed Path Balancing Policy+ you must manually specify the optimal / preferred paths. In the case of using ALUA , the algorithm Most Recently Used or Round Robin is allowed to use - any.


Round Robin will be more productive if there are more than one path to the controller. In case of using Microsoft Cluster + RDM disks, the Most Recently Used balancing mechanism is recommended for use .

Below is a table of recommended load balancing settings. Learn more about NetApp FAS, ALUA logic and load balancing for block protocols .

Mode	Alua	Protocol	ESXi Policy	ESXi Path Balancing
7-Mode 7.x / 8.x	Enabled	FC / FCoE	VMW_SATP_ALUA	Most Recently Used or Round Robin
7-Mode 7.x / 8.x	Disabled	FC / FCoE	AA SATP	Fixed PSP (choose optimal paths)
7-Mode 7.x / 8.x	Disabled	iSCSI	AA SATP	Round Robin PSP
cDOT 8.x	Enabled	FC / FCoE / iSCSI	VMW_SATP_ALUA	Most Recently Used or Round Robin

Make sure that the SATP policy applied to your moon has the reset_on_attempted_reserve option enabled:

ESXi host settings

For optimal operation of the ESXi host, it is necessary to install the recommended parameters for it.

Parameter	Protocol (s)	ESXi 4.x with DataONTAP 8.1.x	ESXi 5.x with DataONTAP 7.3 / 8.x
Net.TcpipHeapSize	iSCSI / NFS	thirty	32
Net.TcpipHeapMax	iSCSI / NFS	120	512 (For vSphere 5.0 / 5.1 set 128)
NFS.MaxVolumes	Nfs	64	256
NFS41.MaxVolumes	NFS 4.1	-
NFS.HeartbeatMaxFailures	Nfs	10
NFS.HeartbeatFrequency	Nfs	12
NFS.HeartbeatTimeout	Nfs	5
NFS.MaxQueueDepth	Nfs	-	64
Disk.QFullSampleSize	iSCSI / FC / FCoE	32 (for 5.1 it is configured on each LUNe )
Disk.QFullThreshold	iSCSI / FC / FCoE	8 (for 5.1 it is configured on each LUNe )

There are several ways to do this:

• Using the Command Line Interface (CLI) on ESXi 5.x hosts.
• Using vSphere Client / vCenter Server.
• Using the Remote CLI tool from VMware.
• Using the VMware Management Appliance (VMA).
• Applying the Host Profile, deploying it from the already configured ESXi 5.x to other hosts.

Hba

NetApp generally recommends using “defaults” for HBAs for FAS systems with an ESXi host as set by the adapter manufacturer. If they have been changed, they must be returned to the factory settings. Check out the relevant best practices. For example, if we are talking about DB2 virtualization in VMware on NetApp, it is recommended ( see page 21 ) to increase the queue length to 64 on ESXi (how to do this is written in Vmware KB 1267 ).

Vsc

Плагин NetApp VSC (является бесплатным ПО), устанавливает рекомендуемые настройки на ESXi хосте и HBA адаптере: очередь, задержки и другие. Сам плагин интегрируется в vCenter. Экономит время и устраняет во время теста человечиский фактор при настройке параметров на ESXi хосте для более эффективной работы с NetApp. Позволяет выполнять базовые операции для управления хранилищем из vCenter, необходимые администратору виртуализированных сред. Права доступа к хранилищу при помощи VSC могут быть гибко настроены для нескольких пользователей при помощи RBAC.


Доступна версия как для «толстого» (старого) клиента так и для нового веб-клиента.

Ethernet

Jumbo frames

If you are using iSCSI, it is highly recommended that you use Jumbo Frames on Ethernet at speeds greater than or equal to 1Gb. Read more in the Ethernet article with NetApp FAS .

ESXi & MTU9000

Не забудьте создать правильный сетевой адаптер — VMware рекомендует использовать VMXNEЕ3. Начиная с версии ESXi 5.0 VMXNET3 поддерживает Jumbo Frames. Сетевой адаптер E1000e поддерживает скорость 1GB сетей и MTU 9000 — он устанавливается для всех создаваемых VM по умолчанию (кроме Linux). Стандартный виртуальный сетевой адаптер типа «Flexible» поддерживает MTU 1500. Подробнее.


Также не забудьте что порт-группа установленная для виртуального сетевого адаптера вашей виртуальной машине должна быть подключена к виртуальному свичу с установленной настройкой MTU 9000 для всего свича.

NAS и VAAI

NetApp FAS systems support VMware VAAI primitives by downloading part of the routine data management tasks on the datastore from the host to the storage, where it is more logical to perform this. In SAN environments with ESXi 4.1+ and higher with NetApp FAS Data ONTAP 8.0 and higher systems, VAAI is automatically supported and does not require any manipulation. For NAS environments, NetApp has released a plug-in that allows similar optimization for the NFS protocol . This requires the installation of a NetAppNFSVAAI kernel module for each ESXi host. VSC can perform VAFS NFS installation plugin automatically from vCenter.

Vasa

VASA is a free software that allows vCenter through the API to learn about the capabilities of the storage and make better use of them. VASA integrates into VSC and allows you to create datastore profiles with specific storage capabilities through the GUI interface (for example, the presence / absence of Thing Provitioning, disk type: SAS / SATA / SSD , the presence of a second level cache, etc.) and enable notifications when there is no one level (for example, occupancy or load). Starting with version 6.0, VASA is a required component of VSC and is an important part of the VMware 6 paradigm.vVOL .

Space Reservation - UNMAP

Starting with ESXi 5.1, the return of released blocks from a thin moon (datastore) is supported. This is configured by default on ESXi 5.1, disabled by default on all other ESXi versions 5.X & 6.X (requires manual start), for ESXi 6.X it works automatically with vVOL, and on the ONTAP side this functionality always it is turned off by default; to enable it, you need to execute several simple commands on the storage system .

Compatibility

Widely apply the compatibility matrix in your practice to reduce potential problems in data center infrastructure . For Troubleshoot, contact KB NetApp and VMware .

I am sure that over time I will have something to add to this article on optimizing the ESXi host, so check back here from time to time.

Please send messages about errors in the text to the LAN .
Comments and additions on the contrary please comment