Comparative testing of software RAID performance. Linux vs FreeBSD (mdadm vs gmirror)

    Introduction I have a long-standing
    opinion regarding various VS - it all depends on the tasks. But no, no, yes there is a desire to dig deeper, to find out who is still stronger - Bruce Lee or Jackie Chan, Stallone or Schwarzenegger, mdadm or gmirror.
    The test does not claim absolute objectivity, rather, it is even subjective in terms of the hardware used. But anyway, numbers are numbers.
    Anyone interested, please come under the cat.


    Test bench
    Motherboard: ECS 865GV-M
    Processor: Intel® Celeron® CPU 2.00GHz
    HDD 1 (system): SATA-II 160Gb Western Digital WD1600AAJS (connected to the motherboard)
    HDD 2,3 (RAID): SATA-II 1 , 5Tb Western Digital [WD15EARS] (connected to Promise)
    SATA controller: Promise SATA300 TX4 PCI, 4-port SATA300
    RAM: 1280Mb
    * God knows what configuration, but as they say, the rich they are.

    Home
    Information about the utilities used in the test:
    Bonnie ++
    As the name implies, Bonnie ++ is based on the Bonnie benchmark. There are many different types of file operations and different programs use them differently. Bonnie ++ tests some of them and displays the result for each of the tests. Distributed under the GPL2 license.
    There are two sections of operations in tests. The first is an IO test that models the behavior of some database applications. The second is a test to create, read and delete a large number of small files.
    Website:www.coker.com.au/bonnie++
    Iozone
    IOzone is a console utility for measuring file system speed. We test the speed of operations such as read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread, mmap, aio_read, aio_write. The results are displayed in a file, the data from which can be used for graphing in Excel.
    Website: www.iozone.org
    DD
    I think that the reader is familiar with this wonderful tool, so I will limit myself to quoting the wiki.
    “Dd (dataset definition) is a Unix program designed for both copying and converting files.” Tested

    OS:
    1.FreeBSD 8.0 RELEASE i386
    2.Debian Lenny 5.0.5 i386

    Preparation of
    FreeBSD: Linux: Let's go (s) Bonnie ++ is the first to go into battle, run with the default settings About the keys: -d: instructed the program in which directory to create temporary files -m: the header displayed in the report - u: from whom to start Next, I redirected the output of the program to the file> out.csv, then using bon_csv2html I translated it into a human-readable html. Results: freebsd linux freebsd linux * I admit, on Linux I was not able to get readable html, so I corrected the html result from FreeBSD. In the screenshots we see the result of the command, let's go in order. Size 2512M
    Загружаем необходимый модуль
    #gmirror load
    Создаем зеркало из двух дисков
    #gmirror label -v gm0 /dev/ad6 /dev/ad10
    Форматируем его в UFS + Soft Updates
    #newfs -U /dev/mirror/gm0
    Создадим каталог
    #mkdir /raid
    Монтируем
    #mount /dev/mirror/gm0 /raid


    Создаем зеркало
    debian:/home/iam# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sd[bc]1
    Записываем информацию в mdadm.conf
    debian:/home/iam# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
    Форматируем в EXT3
    debian:/home/iam#mkfs.ext3 /dev/md0
    Создадим точку монтирования
    debian:/home/iam#mkdir /raid
    Монтируем
    debian:/home/iam#mount /dev/md0 /raid




    # bonnie++ -d /raid/ -m 'FREEBSD_gmirror' -u root > out.csv && cat out.csv | bon_csv2html > gmirror.html







    image

    image

    image

    image



    - RAM * 2
    Sequential Output Sequential Input
    Per Char
    is the result of a read by one byte. Demonstrate the ability of the OS to buffer such operations.
    Block
    read-write blocks of 8192 bytes in size. I think
    Rewrite
    needs no explanation.
    Random Seeks (random read operations)
    Seeks - the number of random read operations
    Sequential Create (sequential file creation)
    Random Create (random / random file creation)
    I think that now the reader does not understand only lines filled with characters +++ - test terminated so quickly that the program cannot display the result.

    Now let's add some instructions to the program
    # bonnie++ -d /raid/ -m 'FREEBSD_gmirror' -n 100:16384:0:20 -u root > out.csv && cat out.csv | bon_csv2html > gmirror_1.html
    -n100: 16384: 0: 20: we say to create 100 files ranging in size from 0 to 16384 b in 20 subdirectories
    Results:
    freebsd
    image
    linux
    image
    freebsd
    image
    linux
    image
    * an attentive reader will notice the difference in the versions of the program. Well, put the latest available in ports / repositories.

    Vavody:
    As you can see, the results of the program are broken for better perception. I will ignore running the program with the default settings, focus on the last four screenshots.
    The Output the Sequential (Serial Output)
    As you can see in the “byte-by-byte” Per Char test, Linux gets to the point so quickly that FreeBSD with its 169 KB / s does not even have time to get to the boot process. 22183 KB / s on Linux versus 169 KB / s on FreeBSD. CPU utilization is minimal.
    In a test where the record size is 8192 B, FreeBSD wins with almost double margin. 48840 KB / s on FreeBSD versus 28895 KB / s on Linux. But FreeBSD loaded the CPU twice as much.
    The Rewrite test showed a slight difference, 23306 KB / s in FreeBSD versus 18529 KB / s in Linux. From which we can conclude that the caching mechanisms of the tested OS are approximately equal in efficiency.
    The Input the Sequential (Seq. Enter)
    Here the victory is clearly for Linux. Per Char 350 KB / s on FreeBSD versus 26435 KB / s on Linux. Blocks in 8192 b, 67290 KB / s FreeBSD versus 86610 KB / s in Linux.
    Random Seeks (random read operations)
    Linux wins this test, 165.1s in FreeBSD versus 148.5s in Linux.

    Let's move on to the last two screenshots of
    Sequential Create (sequential creation)
    . This test sequentially creates / reads / deletes files. Let me remind the reader that we create 100 files in 20 subdirectories of arbitrary size from 0 to 16 KB.
    Speed ​​is measured in files per second. Creating, FreeBSD 103 files / s versus 3099 files / s for Linux, I also ask you to pay attention to the CPU load. Reading, 1556 files / s with FreeBSD versus 34204 files / s with Linux. Removing 12206 files / s from FreeBSD versus 14121 files / s from Linux. Linux, with a huge margin from the pursuer, tears the finish line.
    Random Create (Random / random creation)
    And here Linux is the undisputed leader. Creation, FreeBSD 100 files / s versus 20187 files / s on Linux. Reading, 166 files / s with FreeBSD versus 3299 files / s with Linux. Removing 9139 files / s from FreeBSD versus 11638 files / s from Linux

    Summarizing the results of bonnie ++, you can see that Linux, with some exceptions, won a landslide. Operations to create a large number of small files are many times faster than in FreeBSD. But I advise the reader not to rush to the final conclusions, not everything is so obvious. Move on.

    With bonnie ++ , we 're done with this, and move on to the next tool, Iozone .
    Having played enough with various startup keys, after reading the documentation, the following command was born:
    # iozone -Raz -b /home/iam/results_all.xls -f /raid/1 -g 1g
    * in fact, I’m disingenuous, I looked at the team in the test results on the program’s website))
    About the keys:
    -R: tell the program to create a report file
    -a: fully automatic mode, the recording size is automatically selected from 4k to 16m (4k..8k..16k..32k .... 16384k)
    -f: specify the file on our array
    -g: the maximum file size is 1Gb (for the mode -a)
    -z: forces the program launched in automatic mode to use small recording sizes for large files (i.e. from 4k to 16m (4k..8k..16k..32k .... 16384k))

    For clarity, I I will give in graphs. With the specified keys, the program creates a pretty weighty report file, in the graphs I highlighted the most interesting, in my opinion, results.
    The first will be a recording test.
    image
    image
    image
    The results displayed on the first chart do not leave FreeBSD any chance. When creating a 16Mb file in blocks of 4 kilobytes to 16 megabytes, the average Linux value is ~ 204 MB / s, FreeBSD 44.7 MB / s.
    A file of 128 MB in size, when the block size reaches 128KB, the picture changes dramatically and FreeBSD confidently breaks ahead. The average values ​​are: Linux ~ 47.8 MB / s, FreeBSD 48.3 MB / s.
    1Gb file. FreeBSD holds the same speed as with the previous two files. Linux, on the other hand, slowed down a lot. The average values ​​are: Linux ~ 33 MB / s, FreeBSD 48.5 MB / s.
    As you can see with smaller files, Linux handles much faster, as we already noticed. With a 1Gb file, FreeBSD takes revenge. It is also significant that FreeBSD can withstand the same speed regardless of file size.

    The next one on the list is reading.
    image
    image
    image
    Turn to tear, we give the file size of 16 Mb. Competitors go almost body to body.
    Average values: Linux ~ 534 MB / s, FreeBSD ~ 538 MB / s.
    File 128 Mb. Linux leaves the start with polishing, but already at the 64 KB checkpoint the opponent catches up with him and a fierce struggle begins, Linux is still the first to get to the finish line. Average values: Linux ~ 515 MB / s, FreeBSD ~ 380 MB / s.
    Having reached our heavyweight, the tested OS flutters at the start.
    File size 1Gb. Linux leaving FreeBSD to swallow smoke at the start goes into the gap, as you can see, the speed reaches 1Gb / s. After the checkpoint (starting with a block size of 128KB), FreeBSD keeps its tail with a slight lag behind the leader. Average values: Linux ~ 553 MB / s, FreeBSD ~ 369 MB / s.
    Linux is much faster at reading speed than FreeBSD. Regardless of file size.

    Random (random) record
    image
    image
    image
    File 16 Mb. The results speak for themselves, we see a smooth curve in Linux, FreeBSD leaves something for the pitstop after the 1Mb checkpoint and barely barely reaches the finish line. Average results: Linux ~ 356.8 MB / s, FreeBSD ~ 255.5 MB / s.
    File 128 Mb. FreeBSD apparently didn’t get overboard and shows depressing results, Linux also felt amiss at the finish, the difference between the block sizes of 8 Mb and 16 Mb is striking. Average results: Linux ~ 6 MB / s, FreeBSD ~ 3.8 MB / s.
    File 1 Gb. As with sequentially creating a large file, FreeBSD is steadily stepping forward, Linux is only catching the opponent. Average results: Linux ~ 12 MB / s, FreeBSD ~ 26 MB / s.

    Random (random) reading
    image
    image
    image
    The first graph, Linux only got ahead a bit, but was already overtaken by the 32 KB checkpoint. Further, the rivals moved flush with the very slight advantage of FreeBSD. Average results: Linux ~ 523 MB / s, FreeBSD ~ 458 MB / s.
    The picture with the test results of a 128 Mb file differs very little from the first, with the only difference being that there is no obvious leader, starting with a block size of 64 KB. Average results: Linux ~ 476 MB / s, FreeBSD ~ 362 MB / s.
    The last graph gives us a clear understanding that Linux is far from the forefront when reading large file sizes. Average results: Linux ~ 523 MB / s, FreeBSD ~ 274 MB / s.

    Toward the close, the test results are all known DD utility.
    Create a 1 Gb file from 4 KB blocks
    FreeBSD: 49.02 MB / s
    Linux: 23.9 MB / s
    Read this file in 4 KB blocks
    FreeBSD: 342.7 MB / s
    Linux: 788 MB / s

    Create a file of size 1 Gb of 64 KB blocks
    FreeBSD: 48.64 MB / s
    Linux: 35.0 MB / s
    Read this file in 64 KB blocks
    FreeBSD: 350.14 MB / s
    Linux: 617 MB / s

    Create a 1 Gb file from 1 Mb blocks
    FreeBSD: 47.98 MB / s
    Linux: 23 , 9 MB / s
    Read this file in blocks of 4 KB
    FreeBSD: 348.51 MB / s
    Linux: 460 MB / s

    Create a file of 1000 Mb in size from blocks of 100 Mb
    FreeBSD: 44.24 MB / s
    Linux: 37.7 MB / s
    Read this file in blocks of 4 KB
    FreeBSD: 308.61 MB / s
    Linux: 459 MB / s

    Create a 100 Gb file from 1 Gb blocks
    FreeBSD: 31.7 MB / s
    Linux: 14.7 MB / s
    Read this file is in blocks of 4 KB
    FreeBSD: 57.48 MB / s
    Linux: the result could not be obtained

    FreeBSD is clearly faster than Linux when creating files. Linux is more powerful at reading files.

    Listings dd
    FreeBSD
    * I corrected the dd output on FreeBSD a bit for Linux readability Epilogue The idea to conduct a test, as I wrote, was born spontaneously. Initially, I just wanted to build Software RAID for backups on a budget machine, and then this VS got in , his mother to him. For me, the choice is obvious - FreeBSD, it is faster when creating files, especially when creating large files. Just what I need. After conducting this test, I only once again became convinced of the correctness of the position taken regarding different holly wars - it all depends on the tasks.
    Создаем файл размером 1Gb блоками размером 4k
    # /usr/bin/time -h dd if=/dev/zero of=testfile bs=4k count=262144
    262144+0 records in
    262144+0 records out
    1073741824 bytes transferred in 20.888121 secs (49,02 Mbytes/sec)
    21.05s real 0.45s user 11.12s sys
    Читаем созданный файл блоками размером 4k
    *далее по аналогии
    # /usr/bin/time -h dd if=testfile of=/dev/null bs=4k count=262144
    262144+0 records in
    262144+0 records out
    1073741824 bytes transferred in 2.987966 secs (342,7 Mbytes/sec)
    2.99s real 0.15s user 2.82s sys

    # /usr/bin/time -h dd if=/dev/zero of=testfile bs=64k count=16384
    16384+0 records in
    16384+0 records out
    1073741824 bytes transferred in 21.050771 secs (48,64 Mbytes/sec)
    21.05s real 0.08s user 8.12s sys
    # /usr/bin/time -h dd if=testfile of=/dev/null bs=64k count=16384
    16384+0 records in
    16384+0 records out
    1073741824 bytes transferred in 2.924540 secs (350,14 Mbytes/sec)
    2.92s real 0.05s user 2.86s sys

    # /usr/bin/time -h dd if=/dev/zero of=testfile bs=1m count=1024
    1024+0 records in
    1024+0 records out
    1073741824 bytes transferred in 21.341423 secs (47,98bytes/sec)
    21.57s real 0.00s user 7.87s sys

    # /usr/bin/time -h dd if=testfile of=/dev/null bs=1m count=1024
    1024+0 records in
    1024+0 records out
    1073741824 bytes transferred in 2.938155 secs (348,51 Mbytes/sec)
    2.94s real 0.00s user 2.92s sys

    # /usr/bin/time -h dd if=/dev/zero of=testfile bs=100m count=10
    10+0 records in
    10+0 records out
    1048576000 bytes transferred in 22.601393 secs (44,24 Mbytes/sec)
    22.86s real 0.00s user 7.89s sys

    # /usr/bin/time -h dd if=testfile of=/dev/null bs=100m count=10
    10+0 records in
    10+0 records out
    1048576000 bytes transferred in 3.240300 secs (308,61 Mbytes/sec)
    3.26s real 0.00s user 3.25s sys

    # /usr/bin/time -h dd if=/dev/zero of=testfile bs=1g count=100
    100+0 records in
    100+0 records out
    107374182400 bytes transferred in 3229.788959 secs (31,7 Mbytes/sec)
    53m50.33s real 0.00s user 14m40.89s sys
    # /usr/bin/time -h dd if=testfile of=/dev/null bs=1g count=100
    100+0 records in
    100+0 records out
    107374182400 bytes transferred in 1781.360638 secs (57,48 Mbytes/sec)
    29m41.67s real 0.00s user 6m44.73s sys



    debian:/raid# dd if=/dev/zero of=testfile bs=4k count=262144

    262144+0 записей считано

    262144+0 записей написано

    скопировано 1073741824 байта (1,1 GB), 45,0159 c, 23,9 MB/c

    debian:/raid# dd if=testfile of=/dev/null bs=4k count=262144

    262144+0 записей считано

    262144+0 записей написано

    скопировано 1073741824 байта (1,1 GB), 1,36295 c, 788 MB/c

    debian:/raid# dd if=/dev/zero of=testfile bs=64k count=16384

    16384+0 записей считано

    16384+0 записей написано

    скопировано 1073741824 байта (1,1 GB), 28,2238 c, 38,0 MB/c

    debian:/raid# dd if=testfile of=/dev/null bs=64k count=16384

    16384+0 записей считано

    16384+0 записей написано

    скопировано 1073741824 байта (1,1 GB), 1,73935 c, 617 MB/c

    debian:/raid# dd if=/dev/zero of=testfile bs=1M count=1024

    1024+0 записей считано

    1024+0 записей написано

    скопировано 1073741824 байта (1,1 GB), 30,711 c, 35,0 MB/c

    debian:/raid# dd if=testfile of=/dev/null bs=1M

    1024+0 записей считано

    1024+0 записей написано

    скопировано 1073741824 байта (1,1 GB), 2,33447 c, 460 MB/c

    debian:/raid# dd if=/dev/zero of=testfile bs=100M count=10

    10+0 записей считано

    10+0 записей написано

    скопировано 1048576000 байт (1,0 GB), 27,8344 c, 37,7 MB/c

    debian:/raid# dd if=testfile of=/dev/null bs=100M count=10

    10+0 записей считано

    10+0 записей написано

    скопировано 1048576000 байт (1,0 GB), 2,28437 c, 459 MB/c

    debian:/raid# dd if=/dev/zero of=testfile bs=1024M count=100

    100+0 записей считано

    100+0 записей написано

    скопировано 107374182400 байт (107 GB), 7315,88 c, 14,7 MB/c

    debian:/raid# dd of=testfile if=/dev/null bs=1024M count=100

    0+0 записей считано

    0+0 записей написано

    скопировано 0 байт (0 B), 7,5694e-05 c, 0,0 kB/c




    Although again I do not agree the whole truth. The choice in favor of FreeBSD was made before the test. Since I just know this OS much better than Linux, and I like it a lot more. If you look at the results objectively, then I absolutely deserve to give the victory to Linux. Once again, having run through the test results, there is no doubt about that.
    Someone can rightly notice, why did you bother with these tests so much, bonnie ++ already told you everything, but no. I am used to approaching everything thoughtfully, with feeling, and with sense. Yes, and the free time that appeared allowed me to experiment.
    How competent and informative my report is for you to decide, I am not a * nix professional; my path to this fascinating world began relatively recently.
    Thanks to those who mastered.

    PS
    In preparing the material used:
    1. Linux Administrator Guide. Authors: Nemeth E., Snyder G., Trent H.
    2. FreeBSD. Detailed guide. Michael Lucas
    3. google.com
    4. / dev / hands and / dev / head =)
    This report was prepared with the active help of comrades from unixadmins.su, for which special thanks.

    Also popular now: