November 15, 2016 at 14:53A bit about the disk performance of Windows VM in Proxmox VE. Benchmarks ZFS and MDADM + LVM
If anyone is interested, we recently tested read / write performance inside a windows machine on a node with Proxmox 4.3.
The host system was installed on raid10 implemented in two different ways (zfs and mdadm + lvm)
Tests were carried out on a Windows guest, as it was primarily interested in the performance of this particular OS.
I must admit, this is the second version of the article, the first one made a fatal mistake:
zfs was tested on local storage, and not on zvol, because I until the last thought that proxmox does not support zvol.
Thank you so much winduzoid for noticing this misunderstanding.
Comments on a recent article about Installing PROXMOX 4.3 on Soft-RAID 10 GPT from vasyakrg prompted the thought of writing this article . Not for the sake of holivar, but I decided to publish our recent results.
Water data:
But yes:
CPU: Intel® Core (TM) i7-3820 CPU @ 3.60GHz
RAM: 20GB (1334 MHz)
HDD: 4x500GIB (ST500NM0011, ST500NM0011, ST3500418AS, WDC WD5000AAKX-22ERMA0)
SSD: 250GiB (PLEXTOR PX-256M5Pro)
OS: Proxm Environment 4.3-10
Virtual machine:
CPU: 8 (1 sockets, 8 cores)
RAM: 6.00 GiB
HDD: 60 GiB (virtio)
OS: Windows Server 2008 R2 Server Standard (full installation) SP1 [6.1 Build 7601] (x64)
All results were obtained using CrystalDiskMark 5.2.0 x64 utility .
Each test was conducted in 5 iterations of 32GB .
No additional tweaks and changes not specified in the article were made either in the configuration of the hypervisor or in the configuration of the virtual machine. That is, a freshly installed Proxmox and a Windows virtual machine restored from a backup were used.
Results:
So the results themselves:
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 274.338 MB/s
Sequential Write (Q= 32,T= 1) : 171.358 MB/s
Random Read 4KiB (Q= 32,T= 1) : 3.489 MB/s 851.8 IOPS
Random Write 4KiB (Q= 32,T= 1) : 0.927 MB/s 226.3 IOPS
Sequential Read (T= 1) : 233.437 MB/s
Sequential Write (T= 1) : 183.158 MB/s
Random Read 4KiB (Q= 1,T= 1) : 0.522 MB/s 127.4 IOPS
Random Write 4KiB (Q= 1,T= 1) : 2.499 MB/s 610.1 IOPS
Test : 32768 MiB E: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/08 15:21:41
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 1084.752 MB/s
Sequential Write (Q= 32,T= 1) : 503.291 MB/s
Random Read 4KiB (Q= 32,T= 1) : 31.148 MB/s 7604.5 IOPS
Random Write 4KiB (Q= 32,T= 1) : 203.832 MB/s 49763.7 IOPS
Sequential Read (T= 1) : 1890.617 MB/s
Sequential Write (T= 1) : 268.878 MB/s
Random Read 4KiB (Q= 1,T= 1) : 33.369 MB/s 8146.7 IOPS
Random Write 4KiB (Q= 1,T= 1) : 54.938 MB/s 13412.6 IOPS
Test : 32768 MiB E: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/08 14:55:15
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 1428.912 MB/s
Sequential Write (Q= 32,T= 1) : 281.715 MB/s
Random Read 4KiB (Q= 32,T= 1) : 76.261 MB/s 18618.4 IOPS
Random Write 4KiB (Q= 32,T= 1) : 64.809 MB/s 15822.5 IOPS
Sequential Read (T= 1) : 1337.939 MB/s
Sequential Write (T= 1) : 247.119 MB/s
Random Read 4KiB (Q= 1,T= 1) : 27.926 MB/s 6817.9 IOPS
Random Write 4KiB (Q= 1,T= 1) : 21.005 MB/s 5128.2 IOPS
Test : 32768 MiB E: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/16 14:42:05
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 379.678 MB/s
Sequential Write (Q= 32,T= 1) : 373.262 MB/s
Random Read 4KiB (Q= 32,T= 1) : 12.409 MB/s 3029.5 IOPS
Random Write 4KiB (Q= 32,T= 1) : 150.885 MB/s 36837.2 IOPS
Sequential Read (T= 1) : 931.972 MB/s
Sequential Write (T= 1) : 187.517 MB/s
Random Read 4KiB (Q= 1,T= 1) : 14.106 MB/s 3443.8 IOPS
Random Write 4KiB (Q= 1,T= 1) : 54.419 MB/s 13285.9 IOPS
Test : 32768 MiB E: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/16 14:21:47
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)Later we added a caching SSD to our zfs pool.
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 1518.768 MB/s
Sequential Write (Q= 32,T= 1) : 312.825 MB/s
Random Read 4KiB (Q= 32,T= 1) : 157.763 MB/s 38516.4 IOPS
Random Write 4KiB (Q= 32,T= 1) : 96.962 MB/s 23672.4 IOPS
Sequential Read (T= 1) : 1474.409 MB/s
Sequential Write (T= 1) : 236.638 MB/s
Random Read 4KiB (Q= 1,T= 1) : 28.693 MB/s 7005.1 IOPS
Random Write 4KiB (Q= 1,T= 1) : 24.380 MB/s 5952.1 IOPS
Test : 32768 MiB E: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/16 17:07:45
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 353.932 MB/s
Sequential Write (Q= 32,T= 1) : 401.659 MB/s
Random Read 4KiB (Q= 32,T= 1) : 30.015 MB/s 7327.9 IOPS
Random Write 4KiB (Q= 32,T= 1) : 110.644 MB/s 27012.7 IOPS
Sequential Read (T= 1) : 923.238 MB/s
Sequential Write (T= 1) : 167.356 MB/s
Random Read 4KiB (Q= 1,T= 1) : 31.210 MB/s 7619.6 IOPS
Random Write 4KiB (Q= 1,T= 1) : 56.429 MB/s 13776.6 IOPS
Test : 32768 MiB E: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/16 17:24:12
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)For fun, we also ran tests on one SSD:
-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 526.147 MB/s
Sequential Write (Q= 32,T= 1) : 361.292 MB/s
Random Read 4KiB (Q= 32,T= 1) : 189.502 MB/s 46265.1 IOPS
Random Write 4KiB (Q= 32,T= 1) : 78.780 MB/s 19233.4 IOPS
Sequential Read (T= 1) : 456.598 MB/s
Sequential Write (T= 1) : 368.912 MB/s
Random Read 4KiB (Q= 1,T= 1) : 18.632 MB/s 4548.8 IOPS
Random Write 4KiB (Q= 1,T= 1) : 32.528 MB/s 7941.4 IOPS
Test : 32768 MiB F: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/09 12:56:31
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)-----------------------------------------------------------------------
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s SATA/600 = 600,000,000 bytes/s
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 1587.672 MB/s
Sequential Write (Q= 32,T= 1) : 524.242 MB/s
Random Read 4KiB (Q= 32,T= 1) : 248.953 MB/s 60779.5 IOPS
Random Write 4KiB (Q= 32,T= 1) : 320.532 MB/s 78254.9 IOPS
Sequential Read (T= 1) : 2481.313 MB/s
Sequential Write (T= 1) : 825.351 MB/s
Random Read 4KiB (Q= 1,T= 1) : 58.060 MB/s 14174.8 IOPS
Random Write 4KiB (Q= 1,T= 1) : 59.725 MB/s 14581.3 IOPS
Test : 32768 MiB F: 0.1% (0.1/60.0 GiB) (x5) Interval=5 sec
Date : 2016/11/09 13:28:20
OS : Windows Server 2008 R2 Server Standard (full installation) SP1 6.1 Build 7601 (x64)Charts:
For clarity, I decided to draw some graphs
Sequential read and write speed:
Random read and write speed:
Number of IOPS for random read and write:
Conclusions:
Despite the fact that the results turned out to be rather unusual, and even strange in some places, one can notice that raid10 compiled on zfs, with an option cache=nonefor a disk, showed a better result than raid10 compiled on mdadm + lvm both for reading and writing.
However, with the cache=writebackraid10 option , the compiled on mdadm + lvm noticeably comes forward.
cache=writebackI still have not been able to figure out how much the option increases the risk of data loss, but on the official proxmox forum there is a message that this information is outdated in new versions of the kernel and qemu, it is recommended, in any case, for disks located in zfs.
you refer to an old post, newer kernels and newer qemu version are in place now. for zfs, cache = writeback is the recommended setting.
In addition, IlyaEvseev, in his article, wrote that the writeback cache gives good results for Windows tools.
According to a subjective assessment, writeback is optimal for VMs with Windows, none for VMs with Linux and FreeBSD.
That’s probably all, if you have any suggestions on how else you can increase the performance of disk operations on a guest machine, I will gladly listen to them.