We measure the performance of drives or again about IOPS

Inspired by fasting respected amarao about how to measure disk performance.

Goal:

Test the performance of the available storage media and make sure the chosen method is correct, as well as understand the difference in performance between different types of drives, as well as enterprise-level and consumer-level hard drives.

Equipment:

1. Sandisk Class 10 UHS 1 Extreme Pro 8 GB SD card (up to 95 MB / s read, up to 90 MB / s write)
2. Team Class 10 32 GB SD card (up to 20 MB / s)
3. Transcend 2GB SD card without speed class
4. SSD OCZ-AGILITY3 60 GB
5. Hitachi Deskstar HDS723020BLA642 consumer-level SATA drive 2 TB 7200 rpm, 64 MB
6. SATA-drive enterprise-level Western Digital RE3 WD2502ABYS-23B7A0 250 GB 7200 rpm 16 MB
7. SATA drive consumer-level Seagate Barracuda 7200.11 ST3320613AS 320 GB 7200 rpm 16 MB
8. CD-ROM
9. RAM disk / dev / ram on Linux

Testing Method:

The technique is fully described in the post . There are really a few not entirely clear points:

We select the depth of parallelism of operations so that latency remains within reasonable limits.

The task is to choose such iodepth so that avg.latency is less than 10ms.

Since testing does not use storage systems or SAS disks, but various SATA drives, it makes no sense to measure parallelism.
It is very time consuming to clean the disk before each testing (dd if = / dev / zero of = / dev / sdz bs = 2M oflag = direct), so we will do this before testing once for each drive.
Testing the entire disk completely is very time-consuming, so we will use testing for 30 seconds.
So, we formulate the testing methodology for our case:
Get the IOPS value issued by the drive for arbitrary reading and writing in 4 KB blocks and an avg.latency delay of not more than 10 ms during the test time of 30 seconds . Also, for completeness, measure the speed of linear recording.

Read fio configuration:

[readtest]
blocksize = 4k
filename = / dev / sda
rw = randread
direct = 1
buffered = 0
ioengine = libaio
iodepth = 1024
runtime = 30

Fio record configuration:

[writetest]
blocksize = 4k
filename = / dev / sdz
rw = randwrite
direct = 1
buffered = 0
ioengine = libaio
iodepth = 1024
runtime = 30

Results:

Sandisk Class 10 UHS 1 Extreme Pro 8 GB SD card (up to 95 MB / s read, up to 90 MB / s write)

Sequential read measurement:

root @ host: ~ # dd if = / dev / sdb of = / dev / null iflag = direct bs = 2M
3790 + 0 records in
3790 + 0 records out
7948206080 bytes (7.9 GB) copied, 419.551 s, 18.9 MB / s

Reading Analysis:

When reading, a regularity is clearly observed: the larger the queue size, the more IOPS and more delay. Accordingly, the performance of the drive for reading: iodepth = 8, bw = 4020.6KB / s, iops = 1005, avg = 7.95 msec .










Sequential Write Measurement:

root @ host: ~ # dd if = / dev / zero of = / dev / sdb oflag = direct bs = 1M  
dd: writing `/ dev / sdb ': No space left on device
7581 + 0 records in
7580 + 0 records out
7948206080 bytes (7.9 GB) copied, 555.815 s, 14.3 MB / s

Record Analysis:

With the recording, everything is different. The larger the queue, the lower the speed in MB / s and IOPS, and the higher the delay. Accordingly, the write performance of the drive: iodepth = 1, bw = 618686 B / s, iops = 151, avg = 6.60 msec.
With sequential recording, it was not even possible to get close to the 90 MB / s promised by the manufacturer for recording - the maximum speed is 14.3 MB / s. The difference in speed between arbitrary and linear recording is noticeable - more than 6 times.

Team Class 10 32 GB SD card (up to 20 MB / s)

Sequential Write Measurement:

root @ host: ~ # dd if = / dev / zero of = / dev / sdb oflag = direct bs = 2M
dd: writing `/ dev / sdb ': No space left on device
15600 + 0 records in
15599 + 0 records out
32714522624 bytes (33 GB) copied, 2212.11 s, 14.8 MB / s

Record Analysis:

On record, under no circumstances could I squeeze more than 1 IOPS. Accordingly, the performance of the drive for recording does not meet our requirements for avg.latency <10 ms, since its performance is: iodepth = 1, bw = 5960 B / s, iops = 1, avg = 687.15 msec . Although in sequential recording, the drive really corresponds to Class 10 - 14.8 MB / s.

Transcend 2GB SD Card

Sequential read measurement:

root @ host: ~ # dd if = / dev / sdb of = / dev / null iflag = direct bs = 2M
957 + 1 records in
957 + 1 records out
2007498752 bytes (2.0 GB) copied, 195.504 s, 10.3 MB / s

Reading Analysis:

The linear reading speed corresponds to Class 10, although the flash drive was bought for a long time and was "simple". Read drive performance: iodepth = 4, bw = 2447.4KB / s, iops = 611, avg = 6.53 msec .









Sequential Write Measurement:

root @ host: ~ # dd if = / dev / zero of = / dev / sdb oflag = direct bs = 2M
dd: writing `/ dev / sdb ': No space left on device
958 + 0 records in
957 + 0 records out
2007498752 bytes (2.0 GB) copied, 392.872 s, 5.1 MB / s

Record Analysis:

Failed to get the specified delay level from the drive. An interesting fact is that regardless of the length of the queue, the IOPS value was constant - 8. Sequential write to the media is slower than reading and does not correspond to Class 10. The closest level: iodepth = 1, bw = 35230 B / s, iops = 8, avg = 116.23 msec .

SSD OCZ-AGILITY3 60 GB

Sequential read measurement:

root @ host # dd if = / dev / sdd of = / dev / null iflag = direct bs = 2M 
28620 + 1 records in
28620 + 1 records out
60022480896 bytes (60 GB) copied, 238.535 s, 252 MB / s

Reading Analysis:

The pattern is also visible: the larger the queue, the more IOPS and more delay. Accordingly, the performance of the drive for reading: Read: iodepth = 64, bw = 52705KB / s, iops = 13176, avg = 4849.37 usec . The measured linear read speed of 252 MB / s far falls short of the one declared by the manufacturer (Max Read: up to 525MB / s).













Sequential Write Measurement:

root @ host # dd if = / dev / zero of = / dev / sdd oflag = direct bs = 2M
dd: writing `/ dev / sdd ': No space left on device
28621 + 0 records in
28620 + 0 records out
60022480896 bytes (60 GB) copied, 238.875 s, 251 MB / s

Record Analysis:

Note the non-linear pattern of IOPS from avg.latency delays. Record: iodepth = 128, bw = 70591KB / s, iops = 17647, avg = 7244.71 usec . The measured linear write speed of 251 MB / s far falls short of the one declared by the manufacturer (Max Write: up to 500MB / s).

UPDATE : As amarao suggested , testing SSD records should be done over a longer period of time. Therefore, an additional test for 10 hours was launched. The iodepth number was chosen to be 64, since in a shorter period of time this queue length turned out to be the most effective. The configuration file is as follows:

As a result, the IOPS and bw indices decreased by more than 2 times:



Since fio does not allow you to see the dependence of indicators on time "out of the box" - there was no way to evaluate after what time the performance began to fall.

Hitachi Deskstar HDS723020BLA642 consumer-level SATA drive 2 TB 7200 rpm, 64 MB

Sequential read measurement:

root @ host # dd if = / dev / sdb of = / dev / null iflag = direct bs = 2M
953864 + 1 records in
953864 + 1 records out
2000398934016 bytes (2.0 TB) copied, 15221 s, 131 MB / s

Reading Analysis:

Linear reading speed is very high - 131 MB / s. Failed to achieve the desired avg.latency level. The closest results: iodepth = 1, bw = 258021 B / s, iops = 62, avg = 15.85 msec .

Record Analysis:

Surprisingly, everything turned out better with the recording. I managed to achieve 141 IOPS at 7 ms latency. iodepth = 1, bw = 580929 B / s, iops = 141, avg = 7025.40 usec .

SATA-диск enterprise-level Western Digital RE3 WD2502ABYS-23B7A0 250 GB 7200 об/мин 16 Мбайт

Sequential read measurement:

root @ host: ~ # dd if = / dev / sda of = / dev / null iflag = direct bs = 2M
119237 + 1 records in
119237 + 1 records out
250059350016 bytes (250 GB) copied, 2603.86 s, 96.0 MB / s

Reading Analysis:

Linear read speed is lower than the previous consumer-level drive. Failed to get the result for the given level avg.latency <10 ms. The closest reading result: iodepth = 1, bw = 349284 B / s, iops = 85, avg = 11.69 msec .









Sequential Write Measurement:

root @ host: ~ # dd if = / dev / zero of = / dev / sda oflag = direct bs = 2M   
dd: writing `/ dev / sda ': No space left on device
119238 + 0 records in
119237 + 0 records out
250059350016 bytes (250 GB) copied, 2615.61 s, 95.6 MB / s

Record Analysis:

It should be noted the stability of IOPS on the size of the queue throughout the test. Drive characteristic: iodepth = 1, bw = 768989 B / s, iops = 187, avg = 5294.30 usec .

SATA drive consumer-level Seagate Barracuda 7200.11 ST3320613AS 320 GB 7200 rpm 16 MB

Reading Analysis:

Failed to achieve the desired avg.latency level. The closest results: iodepth = 1, bw = 227153 B / s, iops = 55, avg = 18.00 msec .

Record Analysis:

As it has become a regularity, things are better with burning to disk. I managed to achieve 105 IOPS at 9.5 ms latency. iodepth = 1, bw = 430597 B / s, iops = 105, avg = 9477.07 usec .

CD-ROM

There is no need to test in detail, since the numbers are needed more for orientation. The drive did not allow the condition avg.latency <10 ms. The closest reading result: iodepth = 1, bw = 23655 B / s, iops = 5 .

RAM-диск /dev/ram в Linux

Анализ чтения:

It seems that everything is true, the measurement of reading the RAM disk corresponds to the measured , but there is one caveat:

  cpu: usr = 34.51%, sys = 64.60%, ctx = 1, majf = 0, minf = 0

in other words, we ran into processor performance. And not the fact that the RAM disk itself. The result of reading at the moment: iodepth = 2, bw = 852105KB / s, iops = 213026, avg = 5.51 usec .

Record Analysis:

The same with writing to a RAM disk. We ran into the processor:

  cpu: usr = 20.86%, sys = 76.26%, ctx = 1, majf = 0, minf = 0

Current record result: iodepth = 2, bw = 693857KB / s, iops = 173464, avg = 6.44 usec

Summary table of test results:

The questions that arose:

1. How can one imagine the size of the queue in real life and what does it say about the medium? For example, on SD-cards, the queue size is 1, and on SSD-drives - 64.
2. Is 30 seconds enough for an IOPS test, or do I need to test the entire disk?
3. Performance test RAM-disk rested in the CPU. Is this real disk performance or is it still limited by the processor?

CONCLUSIONS:

1. The technique seems to be true and accurate. There were no differences in testing.
2. UPDATE: Предварительно 30 секунд достаточно для RAM-дисков, чтобы получить более ли менее точную оценку IOPS. Для SSD-дисков время замера должно быть однозначно больше.
3. Измерение производительности на запись и чтение всё же лучше производить отдельно друг от друга.
4. SSD-диски вне конкуренции. Как по IOPS и скорости линейного чтения, так и по latency.
5. Разница в производительности между enterprise и consumer дисками есть в значениях IOPS и Latency. Велика ли разница — решать каждому под конкретные задачи. А вообще недостаточно данных для более точного анализа.
6. Скорость линейного чтения enterprise дисков не выше consumer дисков.
7. Жёсткие диски выдают очень низкую скорость при случайном чтении. Не более полумегабайта в секунду.
8. Read IOPS performance on measured hard drives was lower than on write. This was probably done in order to “equalize” the read and write performance when taking into account the RAID penalty, while the low reading IOPS is offset by the built-in cache in the OS.
9. The linear speed of the tested drives is higher than arbitrary by up to 500 times or more.

PS I would like more results of testing enterprise-level disks, but it was no longer at hand.

Have a nice day!