VMware ESXi 5.X and NetApp ONTAP 8: Tuning

    В продолжение темы об оптимизации хоста для взаимодействия с СХД NetApp FAS, эта статья будет посвещена оптимизации производительности VMWare ESXi, предыдущие статьи были посвящены тюнингу ОСLinux и Windows в среде SAN. Компания NetApp давно тесно сотрудничает с VMware, подтверждением тому может стать тот факт, что нашумевшая технология vVOL была реализована одной из первых ещё в релизе Clustered Data ONTAP 8.2.1 (Август 2014), в то время как vSphere 6.0 ещё даже не выпущен. В связи с чем системы хранения NetApp FAS крайне популярны в этом окружении. Часть про Disk Alignment будет полезна не только владельцам NetApp.

    VMWare ESXi settings can be divided into the following parts:

    • Hypervisor Optimization
    • Guest OS Optimization ( GOS )
    • Optimal SAN Settings ( FC / FCoE and iSCSI )
    • NAS Settings ( NFS )
    • Checking hardware, firmware, and software compatibility

    To search for a bottleneck, a sequential exclusion technique is usually performed. I suggest first things first to start with storage . And then move the storage -> Network ( Ethernet / FC) -> Host ( Windows / Linux / VMware ESXi 5.X and ESXi 6.X ) -> Application.

    There are a couple of basic documents you should rely on when setting up VMware + NetApp:

    TR-4068: VMware vSphere 5 on NetApp Clustered Data ONTAP
    TR 3839: Using NFS in VMware (7-Mode)
    TR-3749: Guide to Best Practices for Using NetApp Systems with VMware vSphere (7-Mode)
    TR-3802: Ethernet for Storage: Best Practices (7-Mode)


    It’s not worth giving all the server resources to the guest OS , firstly, the hypervisor needs to leave at least 4GB of RAM , and secondly, the opposite effect is sometimes observed when adding guest OS resources , this must be selected empirically.


    I will take out this section in a separate post .

    Guest OS

    Tuning settings is needed for two purposes:
    • Guest OS Workload Speed ​​Optimization
    • Normal operation in HA pair, in case of failure of one controller (takeover) and resumption of its operation (takeover)

    Disk alignment

    To optimize performance, you may need to eliminate disk misalignment . Misalignment can be obtained in two cases:
    1. due to incorrect moon geometry when creating it in the storage system . This error can only be created in a SAN environment.
    2. inside virtual disks of virtual machines. It can be both in SAN and in NAS environment

    Let's look at these cases.
    To begin, consider fully aligned blocks at the boundaries of the VMFS datastore and storage.

    The first case is when there is a misalignment of the VMFS datastore relative to the storage. To fix the first type of problem, you need to create moons with the correct geometry and move the virtual machines there.

    The second situation, with shifted file system partitions inside the guest OS relative to the WAFL file structure, can be obtained in older Linux distributions and OSWindows 2003 and older. Typically, this is due to poor placement of the MBR partition table or to machines that have been converted from physical to virtual. You can verify this in Windows guest OS using the dmdiag.exe -v utility (the value of the Rel Sec field must be a multiple of 4KB by WAFL ). Learn more about misalignment diagnostics for Windows machines. Also, the placement of the file system on the disk can be checked using the mbralign utility for the ESXi host , which is part of NetApp Host Utilities version 5.x and VSC . More details on how to eliminate such situations are described inTR-3747 Best Practices for File System Alignment in Virtual Environments .

    And of course, you can get misalignment at once on two levels: both at the VMFS datastore level and at the file system level of the guest OS . Learn more about finding misalignment from the NetApp FAS repository .

    An example for the VMFS3 file system. In the newly created VMFS5 (not an upgrade with VMFS3), the block has a size of 1MB with sub-blocks of 8KB.

    takeover / giveback

    To work with takeover / giveback in the HA pair, you need to configure the correct timeouts of the guest OS :
    OSHistorical Guest OS Tuning for SAN: ESXi 3.x / 4.x and Data ONTAP 7.3 / 8.0 (SAN)Updated Guest OS Tuning for SAN: ESXi 5 or later, or Data ONTAP 8.1 or later (SAN)
    Windowsdisk timeout = 190disk timeout = 60
    Linuxdisk timeout = 190disk timeout = 60
    Solarisdisk timeout = 190; busy retry = 300; not ready retry = 300; reset retry = 30; max throttle = 32; min. throttle = 8disk timeout = 60; busy retry = 300; not ready retry = 300; reset retry = 30; max throttle = 32; min. throttle = 8; corrected VID / PID specification

    The default OS values in the case of using NFS are satisfactory, and the settings for the guest OS do not need to be changed.

    These values ​​are set manually or using scripts available as part of VSC .
    Windows : Set the value of the disk access delay to 60 seconds using the registry (set in seconds, in hexadecimal form).

    Linux : Set the disk access delay value to 60 seconds by creating a udev rule (specified in seconds, in hexadecimal form).
    DRIVERS=="sd", SYSFS{TYPE}=="0|7|14", RUN+="/bin/sh -c 'echo 60 > /sys$$DEVPATH/timeout'"
    (Linux distributions may have a different installation location for udev rules). VMware Tools for a guest Linux OS automatically sets a udev rule with a delay value for a virtual disk of 180 seconds. You can run the grep command for “VMware” vendor ID in the udev rules folder to find a script that sets this value and change it if necessary. Remember to check this value.

    Solaris : You can set the delay value to 60 seconds (specified in seconds, in hexadecimal form) for the disk in the / etc / system file :
    set sd:sd_io_time=0x3c

    Additional settings can be made to the file /kernel/drv/sd.conf :
    Solaris 10.0 GA - Solaris 10u6:
    sd-config-list="NETAPP  LUN","netapp-sd-config",
    "VMware  Virtual","netapp-sd-config";

    Solaris 10u7 and later and Solaris 11:
    sd-config-list= "NETAPP  LUN","physical-block-size:4096,retries-busy:300,retries-timeout:16,retries-notready:300,retries-reset:30,throttle-max:32,throttle-min:8",
    "VMware  Virtual","physical-block-size:4096,retries-busy:300,retries-timeout:16,retries-notready:300,retries-reset:30,throttle-max:32,throttle-min:8";

    Please note: there are two spaces between vendor ID NETAPP and ID LUN, as well as between the words "VMware" and "Virtual" in the config above.

    FC / FCoE Switch Zoning Settings

    Learn more about zoning guidelines for NetApp in pictures .


    For NetApp FAS systems with 7-Mode ALUA is recommended for FC / FCoE . And for NetApp FAS Systems with cDOT, ALUA is recommended for use with all block protocols: iSCSI / FC / FCoE .
    ESXi will determine if ALUA is enabled . If ALUA is enabled, Storage Array Type plug-in will be VMW_SATP_ALUA , if ALUA is disabled, it is recommended to use the Fixed Path Balancing Policy+ you must manually specify the optimal / preferred paths. In the case of using ALUA , the algorithm Most Recently Used or Round Robin is allowed to use - any.

    Round Robin will be more productive if there are more than one path to the controller. In case of using Microsoft Cluster + RDM disks, the Most Recently Used balancing mechanism is recommended for use .

    Below is a table of recommended load balancing settings. Learn more about NetApp FAS, ALUA logic and load balancing for block protocols .
    ModeAluaProtocolESXi Policy
    ESXi Path Balancing
    7-Mode 7.x / 8.xEnabledFC / FCoEVMW_SATP_ALUAMost Recently Used or Round Robin
    7-Mode 7.x / 8.xDisabledFC / FCoEAA SATPFixed PSP (choose optimal paths)
    7-Mode 7.x / 8.xDisablediSCSIAA SATPRound Robin PSP
    cDOT 8.xEnabledFC / FCoE / iSCSIVMW_SATP_ALUAMost Recently Used or Round Robin

    Check the applied policy for the checked moon / datastore
    ~ # esxcli storage nmp device list
       Device Display Name: NETAPP Fibre Channel Disk (naa.60a980004434766d452445797451376b)
       Storage Array Type: VMW_SATP_ALUA
       Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
       Path Selection Policy: VMW_PSP_RR
       Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
       Path Selection Policy Device Custom Config: 
       Working Paths: vmhba2:C0:T6:L119, vmhba1:C0:T7:L119
       Is Local SAS Device: false
       Is USB: false
       Is Boot USB Device: false

    Make sure that the SATP policy applied to your moon has the reset_on_attempted_reserve option enabled:
    SATP policy
    ~ # esxcli storage nmp satp rule list
    Name                 Device  Vendor   Model             Driver  Transport  Options                     Rule Group  Claim Options                        Default PSP  PSP Options  Description                                                               
    -------------------  ------  -------  ----------------  ------  ---------  --------------------------  ----------  -----------------------------------  -----------  -----------  --------------------------------------------------------------------------
    VMW_SATP_ALUA                LSI      INF-01-00                            reset_on_attempted_reserve  system      tpgs_on                              VMW_PSP_MRU               NetApp E-Series arrays with ALUA support                                  
    VMW_SATP_ALUA                NETAPP                                        reset_on_attempted_reserve  system      tpgs_on                              VMW_PSP_RR                NetApp arrays with ALUA support       

    ESXi host settings

    For optimal operation of the ESXi host, it is necessary to install the recommended parameters for it.
    ParameterProtocol (s)ESXi 4.x with DataONTAP 8.1.xESXi 5.x with DataONTAP 7.3 / 8.x
    Net.TcpipHeapSizeiSCSI / NFSthirty32
    Net.TcpipHeapMaxiSCSI / NFS120512 (For vSphere 5.0 / 5.1 set 128)
    NFS41.MaxVolumesNFS 4.1-
    Disk.QFullSampleSizeiSCSI / FC / FCoE32 (for 5.1 it is configured on each LUNe )
    Disk.QFullThresholdiSCSI / FC / FCoE8 (for 5.1 it is configured on each LUNe )

    There are several ways to do this:

    • Using the Command Line Interface (CLI) on ESXi 5.x hosts.
    • Using vSphere Client / vCenter Server.
    • Using the Remote CLI tool from VMware.
    • Using the VMware Management Appliance (VMA).
    • Applying the Host Profile, deploying it from the already configured ESXi 5.x to other hosts.

    Example of setting advanced options from ESX 4.x CLI
    The esxcfg-advcfg utility used in these examples is located in the / usr / sbin folder for the ESXi host.
    #Для протоколов iSCSI/NFS
    #esxcfg-advcfg -s 30 /Net/TcpipHeapSize
    #esxcfg-advcfg -s 120 /Net/TcpipHeapMax
    #Для протоколоа NFS 
    #esxcfg-advcfg -s 64 /NFS/MaxVolumes
    #esxcfg-advcfg -s 10 /NFS/HeartbeatMaxFailures
    #esxcfg-advcfg -s 12 /NFS/HeartbeatFrequency
    #esxcfg-advcfg -s 5 /NFS/HeartbeatTimeout
    #Для протоколов iSCSI/FC/FCoE
    #esxcfg-advcfg -s 32 /Disk/QFullSampleSize
    #esxcfg-advcfg -s 8 /Disk/QFullThreshold

    Check advanced settings from ESX 4.x CLI
    #Для протоколов iSCSI/NFS
    #esxcfg-advcfg -g /Net/TcpipHeapSize
    #esxcfg-advcfg -g /Net/TcpipHeapMax
    #Для протоколоа NFS
    #esxcfg-advcfg -g /NFS/MaxVolumes
    #esxcfg-advcfg -g /NFS/HeartbeatMaxFailures
    #esxcfg-advcfg -g /NFS/HeartbeatFrequency
    #esxcfg-advcfg -g /NFS/HeartbeatTimeout
    #Для протоколов iSCSI/FC/FCoE
    #esxcfg-advcfg -g /Disk/QFullSampleSize
    #esxcfg-advcfg -g /Disk/QFullThreshold

    Example of setting advanced options from ESX 5.x CLI
    The esxcfg-advcfg utility used in these examples is located in the / usr / sbin folder for the ESXi host.
    #Для протоколов iSCSI/NFS
    #esxcfg-advcfg -s 32 /Net/TcpipHeapSize
    #For vSphere 5.0/5.1: 
    #esxcfg-advcfg -s 128 /Net/TcpipHeapMax
    #For vSphere 5.5:
    #esxcfg-advcfg -s 512 /Net/TcpipHeapMax
    #Для протоколоа NFS
    #esxcfg-advcfg -s 256 /NFS/MaxVolumes
    #esxcfg-advcfg -s 10 /NFS/HeartbeatMaxFailures
    #esxcfg-advcfg -s 12 /NFS/HeartbeatFrequency
    #esxcfg-advcfg -s 5 /NFS/HeartbeatTimeout 
    #esxcfg-advcfg -s 64 /NFS/MaxQueueDepth
    #Для протоколов iSCSI/FC/FCoE
    #esxcfg-advcfg -s 32 /Disk/QFullSampleSize
    #esxcfg-advcfg -s 8 /Disk/QFullThreshold

    Check advanced settings from ESX 5.x CLI
    #Для протоколов iSCSI/NFS
    #esxcfg-advcfg -g /Net/TcpipHeapSize
    #esxcfg-advcfg -g /Net/TcpipHeapMax
    #Для протокола NFS
    #esxcfg-advcfg -g /NFS/MaxVolumes
    #esxcfg-advcfg -g /NFS/HeartbeatMaxFailures
    #esxcfg-advcfg -g /NFS/HeartbeatFrequency
    #esxcfg-advcfg -g /NFS/HeartbeatTimeout
    #esxcfg-advcfg -g /NFS/MaxQueueDepth
    #Для протоколов iSCSI/FC/FCoE
    #esxcfg-advcfg -g /Disk/QFullSampleSize
    #esxcfg-advcfg -g /Disk/QFullThreshold

    Example of setting advanced options from ESX 5.1 CLI
    #Для протоколов iSCSI/NFS
    #esxcfg-advcfg -s 32 /Net/TcpipHeapSize
    #esxcfg-advcfg -s 128 /Net/TcpipHeapMax
    #Для протоколоа NFS
    #esxcfg-advcfg -s 256 /NFS/MaxVolumes
    #esxcfg-advcfg -s 10 /NFS/HeartbeatMaxFailures
    #esxcfg-advcfg -s 12 /NFS/HeartbeatFrequency
    #esxcfg-advcfg -s 5 /NFS/HeartbeatTimeout 
    #esxcfg-advcfg -s 64 /NFS/MaxQueueDepth
    #Для протоколов iSCSI/FC/FCoE
    # esxcli storage core device set --device naa.60a98000486e5334524a6c4f63624558 --queue-full-sample-size 32 --queue-full-threshold 8

    Checking advanced settings from ESX 5.1 CLI
    #Для протоколов iSCSI/NFS
    #esxcfg-advcfg -g /Net/TcpipHeapSize
    #esxcfg-advcfg -g /Net/TcpipHeapMax
    #Для протокола NFS
    #esxcfg-advcfg -g /NFS/MaxVolumes
    #esxcfg-advcfg -g /NFS/HeartbeatMaxFailures
    #esxcfg-advcfg -g /NFS/HeartbeatFrequency
    #esxcfg-advcfg -g /NFS/HeartbeatTimeout
    #esxcfg-advcfg -g /NFS/MaxQueueDepth
    #Для протоколов iSCSI/FC/FCoE
    # esxcli storage core device list 


    NetApp generally recommends using “defaults” for HBAs for FAS systems with an ESXi host as set by the adapter manufacturer. If they have been changed, they must be returned to the factory settings. Check out the relevant best practices. For example, if we are talking about DB2 virtualization in VMware on NetApp, it is recommended ( see page 21 ) to increase the queue length to 64 on ESXi (how to do this is written in Vmware KB 1267 ).
    Example HBA Qlogic setup on ESXi
    # посмотреть драйвер для Qlogic на ESXi 5.5 и 6.0
    # esxcli system module list | grep qln
    # посмотреть драйвер для Qlogic на ESXi 5.1 и 5.0
    # esxcli system module list | grep qla
    # Установить значение для Qlogic на ESXi 5.5 и 6.0
    # esxcli system module parameters set -p qlfxmaxqdepth=64 -m qlnativefc
    # Установить значение для Qlogic на ESXi 5.1 и 5.0
    # esxcli system module parameters set -p ql2xmaxqdepth=64 -m qla2xxx


    Плагин NetApp VSC (является бесплатным ПО), устанавливает рекомендуемые настройки на ESXi хосте и HBA адаптере: очередь, задержки и другие. Сам плагин интегрируется в vCenter. Экономит время и устраняет во время теста человечиский фактор при настройке параметров на ESXi хосте для более эффективной работы с NetApp. Позволяет выполнять базовые операции для управления хранилищем из vCenter, необходимые администратору виртуализированных сред. Права доступа к хранилищу при помощи VSC могут быть гибко настроены для нескольких пользователей при помощи RBAC.

    Доступна версия как для «толстого» (старого) клиента так и для нового веб-клиента.


    Jumbo frames

    If you are using iSCSI, it is highly recommended that you use Jumbo Frames on Ethernet at speeds greater than or equal to 1Gb. Read more in the Ethernet article with NetApp FAS .

    ESXi & MTU9000

    Не забудьте создать правильный сетевой адаптер — VMware рекомендует использовать VMXNEЕ3. Начиная с версии ESXi 5.0 VMXNET3 поддерживает Jumbo Frames. Сетевой адаптер E1000e поддерживает скорость 1GB сетей и MTU 9000 — он устанавливается для всех создаваемых VM по умолчанию (кроме Linux). Стандартный виртуальный сетевой адаптер типа «Flexible» поддерживает MTU 1500. Подробнее.

    Также не забудьте что порт-группа установленная для виртуального сетевого адаптера вашей виртуальной машине должна быть подключена к виртуальному свичу с установленной настройкой MTU 9000 для всего свича.

    NAS и VAAI

    NetApp FAS systems support VMware VAAI primitives by downloading part of the routine data management tasks on the datastore from the host to the storage, where it is more logical to perform this. In SAN environments with ESXi 4.1+ and higher with NetApp FAS Data ONTAP 8.0 and higher systems, VAAI is automatically supported and does not require any manipulation. For NAS environments, NetApp has released a plug-in that allows similar optimization for the NFS protocol . This requires the installation of a NetAppNFSVAAI kernel module for each ESXi host. VSC can perform VAFS NFS installation plugin automatically from vCenter.


    VASA is a free software that allows vCenter through the API to learn about the capabilities of the storage and make better use of them. VASA integrates into VSC and allows you to create datastore profiles with specific storage capabilities through the GUI interface (for example, the presence / absence of Thing Provitioning, disk type: SAS / SATA / SSD , the presence of a second level cache, etc.) and enable notifications when there is no one level (for example, occupancy or load). Starting with version 6.0, VASA is a required component of VSC and is an important part of the VMware 6 paradigm.vVOL .

    Space Reservation - UNMAP

    Starting with ESXi 5.1, the return of released blocks from a thin moon (datastore) is supported. This is configured by default on ESXi 5.1, disabled by default on all other ESXi versions 5.X & 6.X (requires manual start), for ESXi 6.X it works automatically with vVOL, and on the ONTAP side this functionality always it is turned off by default; to enable it, you need to execute several simple commands on the storage system .


    Widely apply the compatibility matrix in your practice to reduce potential problems in data center infrastructure . For Troubleshoot, contact KB NetApp and VMware .

    I am sure that over time I will have something to add to this article on optimizing the ESXi host, so check back here from time to time.

    Please send messages about errors in the text to the LAN .
    Comments and additions on the contrary please comment

    Also popular now: