Integration of FATFS libraries for organizing disk device reading on iOS

Introduction

The article is devoted to the implementation of an open source library on iOS for reading / writing data from an MFI disk device based on FAT12 / FAT16 / FAT32 / Exfat. A method for building the application architecture based on the FATFS library, as well as methods for debugging and testing wired MFI devices, is presented. The article contains almost no code due to NDA compliance .

Formulation of the problem

The main goal was to create some kind of universal SDK (.framework), capable of working with manufacturers of various disk access devices using the MFI protocol, using the same API standard.



The figure shows a generalized structural diagram of such a framework. Depending on the device protocol, this or that library of interaction with the device is selected. Depending on which file system (FS) is used on the disk, the appropriate algorithm for working with the disk is selected: FAT / exFat / Other FS. At the same time, several applications (using this SDK) can be accessed by the device, which are in the device’s memory, however, only one can write / read at a time. This SDK implies its use in a multi-threaded application with competitive read and write tasks.

Search for a solution

Initially, the FAT32 implementation created by Apple was chosen as the base library for the FS FAT32 implementation. However, the complexity of integrating the standard API for the access device due to binding to a specific platform forced us to search for a turnkey solution with a separate abstract API for the physical access level with the MFI device through the manufacturer’s library. EFSL and FATFS open source implementations of FAT32 were also considered . FATFS was chosen for several reasons:

• current library support
• platform independence and ease of porting
• wide range of configuration parameters
• exFAT support
• abstract access level for media drivers

Due to the problems of receiving application logs, given that the connector was physically occupied and there was no possibility of connecting to Xcode , the NSLogger library was connected , which allows transmitting logs over Wi-Fi in almost real time mode. Nevertheless, the problem of debugging such a library has taken place. It was decided to develop a simulator of the MFI device, which accessed a certain area on the device’s iOS / MAC disk and emulated reading / writing using the same API as the real device.

Integration issues

Porting to Obj-C

FATFS was written in pure C, and had to be integrated into a library written in ObjC. Despite the fact that ObjC directly supports the integration of C functions, there were problems with function signatures and using some types of variables.

Unicode Support

The Unicode option only worked adequately with the Obj-C NSUTF16LittleEndianStringEncoding encoding parameter , despite the fact that, according to the stated requirements, UTF-16BE and UTF-8 were additionally supported. All methods that work with file names had to be made thread safe for FAT32 to work correctly.

Thread safe

The thread safety option, activated by the FS_REENTRANT parameter and implemented through the ff_req_grant, ff_rel_grant, ff_del_syncobj functions , required additional implementation via POSIX. Without the thread safety option enabled, file system table corruption was observed and, as a result, data loss. The file descriptor pthread_t was not immediately able to be correctly implemented due to the fact that the ff_ functions for synchronizing threads passed data by value, and not by reference. After replacing with reference values, there were no problems with thread safety of operations.

Caching

To avoid data loss during copying and file table corruption, the data cache of the recorded file was synchronized using the f_sync function. Given that the disk space could be more than 16 GB, some basic types of variables had to be changed to “long log”. FATFS was originally designed for embedded devices with limited memory size and low performance. As a result, the read / write functions were performed discrete per unit of time per unit, which is significantly lower than continuous read / write. A similar problem arose when reading the directory structure when there were more than 100 files in the directory. Reading / writing a single file grew linearly with the increase in the number of files, as shown in the figure. The “x” axis shows the number of files in the directory, and the “y” axis shows the read time in seconds. Types of curves on the graph for: V2 Sim Fat - disk device simulator, V1 Fat file system FAT32,



The block diagram of the cache in this SDK is presented in the figure below. Thus, not only the file structure was cached, but also sectors for low-level media device read / write functions, due to the lack of a hardware cache.

Unique application access to the device

Applications with integrated SDKs must not simultaneously have access to an external disk device. The rules to which the application with the SDK used should obey should look like the diagram below shows the state.



To establish access rules for the device, a mechanism based on the Darwin Notification Center was implementedwhich allows using notifications to allow or block work with the application device when another application with this SDK is working with the device. When you first launch the application, a request is sent to all applications that are subscribed to receive certain notifications. Each application publishes its state for the rest of the applications so that the new application can transition to an adequate state. If the device is free, the application enters the USING state. The PENDING state is intermediate for BUSY and WAIT in case the device is not available or busy. In the event of a communication failure, the INVALID state is used. After using the device, the application enters the INACTIVE state.

Framework Testing

Writing Unit Tests for all library functions was critically necessary due to strict deterministic API behavior requirements for end users. Initially, all tests were performed on a disk device simulator. However, the simulator and the library for low-level disk access were written by different people who did not have the ability to interact, and therefore the behavior of the simulator did not always coincide with the behavior of the device driver. For this, a testing scheme on a real device was developed, presented below. The test framework was integrated into an application written using the Private API, and the installation was done through the Xcode Server Over-the-Air installation. Private API allowed you to remove the application from the Background state of the application by Push Notification and run the necessary tests. Test results were sent via the REST protocol to the statistics server, where they were subsequently presented in the form of graphs and tables. The library also includes Activity Tracing functions to get a more detailed crash log.

Conclusion

Despite the serious flaws of the FATFS opensourse library, which need to be improved, another alternative with the ability to use Fat32 and ExFat, as well as with an abstract level of support for media drivers, could not be found. If the library is correctly modified with additional caching and buffer read / write functions connected, the FAT32 API file copy and read functions are lower than the copy speed of LBA blocks of the media driver of the iron manufacturer by only 10-15%. For beginners in the development and integration of file systems - FATFS may be a worthy choice.

useful links

→ FatFs - Generic FAT File System Module
→ Microsoft EFI FAT32 File System Specification
→ Darwin Notification Concepts
→ About Continuous Integration in Xcode