Testing flash storage. File system effects

During testing the performance of leading flash systems, at some point, we asked ourselves: What is the effect of the file system on the performance of a real storage system? How significant is it and what does it depend on?

It is known that the file system is an infrastructure software layer implemented at the kernel level of the OS (kernel space) or, less commonly, at the user level (user space). As an intermediate layer between application / system software and disk space, the file system must introduce its own parasitic load, which affects the performance of the system. Therefore, when calculating the real performance of storage systems, one should take into account the dependence of the fixed parameters on the implementation of the file system and software using this file system.

Testing program.

In order to study the overhead created by various file systems (EXT4, VXFS, CFS) for storage performance, a stand was created that was described in detail in the article Testing flash storage. IBM RamSan FlashSystem 820. The
tests were performed by creating a synthetic load with the fio program on a block device, which is a logical volume of the type stripe, 8 column, stripe unit size=1MiBcreated using Veritas Volume Manager from 8 LUNs presented from the tested system. In relation to the file system, tests were performed equivalent to those described in the article Testing flash storage. IBM RamSan FlashSystem 820.
Then, graphs were plotted showing the effect of the file system on the performance of the storage system (difference in performance in% of that obtained during testing of the block device) and conclusions were drawn on the degree of influence of the file system on the performance of the storage system.

Figure 1. Structural diagram of test stand No. 1. (clickable)

Disk array performance tests for various types of load, performed at the ext4 file system level.

Ext4 file system type.
4K file system block.
The file system is mounted with the mount options noatime, nobarrier.
On the created file system, 16 files of the same size for the entire file system volume are generated. The names of all generated files are given as the value of the filename parameter of the fio program (when you run the tests, the generated load will be evenly distributed between all created files).

Disk array performance tests for various types of load, performed at the vxfs file system level.

Type of vxfs file system.
8K file system block.
The file system is mounted with mount options. The cio,nodatainlog,noatime,convosync=unbuffered
additional file system settings implemented through the vxtunefs command are as follows:

• initial_extent_size = 2048;
• read_ahead = 0.

The obtained graphs are superimposed on the results of previous tests and a conclusion is drawn on the degree of influence of the file system on the storage performance.

Disk array performance tests for different types of load generated by two servers on a Symantec CFS cluster file system.

At this stage, another test server is added to the stand.

Figure 2. Structural diagram of test stand No. 2

Software equivalent to the one installed on the first server is installed on the added server, the same optimization settings are made. All 8 LUNs with storage are presented to both servers. Using Symantec Volume Manager, a cluster volume is created on them striped, 8 columns, unit size=1024KB. This volume creates the CFS file system, which is mounted with options cio,nodatainlog,noatime,nomtime,convosync=unbufferedon both servers. 16 files are created on the file system, to which both servers have access. The servers simultaneously run tests similar to the previous ones. First, the entire set of 16 files created. Then, from each server to its own, not intersecting with the other, a subset of 8 files. Based on the results, graphs of the difference in the obtained indicators are built and conclusions are drawn on the degree of influence of the Symantec CFS cluster file system on performance.

Test results

Findings Comparing EXT4 and VXFS

1. The file system has a significant impact on storage performance, up to its 50% drop.
2. As the load on the storage system increases, the effect of the file system on performance, in most cases, decreases (the disk array becomes saturated, and the overhead of the file system becomes not noticeable against the background of a significant increase in latency at the level of the disk array).
3. The vxfs file system demonstrates performance gains on asynchronous write and read operations with a low load on the disk array. Probably due to the use of the mount option, convosync=unbufferedwhich implies direct transfer of data between the buffer in the user's address space and the disk (without copying data to the kernel buffer of the operating system). The indicated effect is not observed on the ext4 file system. Performance The ext4 file system performance is worse than a block device in all dimensions.
4. When storage is in saturation mode, the vxfs file system shows comparable performance with the ext4 file system. As a rule, the configuration of the storage system is chosen so that it is not in saturation mode in the normal mode of operation, therefore, the resulting lower vxfs performance in comparison with ext4 is not a significant indicator of the quality of the file system.
5. Significant fluctuations in the relative performance of file systems during synchronous input-output is probably due to the lack of optimization of file system drivers for low-latency SSD disks and the presence of additional input-output operations to change file system metadata. It is possible that additional file system settings will reduce these fluctuations.

Disk array performance tests for different types of load generated by two servers on a Symantec CFS cluster file system.

CFS vs VXFS comparison findings

1. Performance under load from both servers on the same 16 files at the same time does not differ from the performance obtained when each server loads its 8 files. (small jumps (20% increase in performance when reading in 4-8K blocks when loading on one set of files, most likely due to background processes on the storage itself, because tests were performed in a row). Observation of the Ethernet connection loading between the servers used for interconnect, showed the absence of a significant load, which is the advantage of CFS when working with multiple servers with one set of files
2. Approximately the same write performance, in both cases. Except for small blocks (4-8K) where CFS results are 2-3 times higher than VXFS. On mixed I / O, CFS is 10-20% better than VXFS.
3. The CFS clustered file system does not adversely affect performance. In some cases, even greater performance is obtained. This may be due to better parallelization of the load from two servers than from one.