s3ql - file system based on cloud storage

Description

Using S3QL, you can create a file system based on Selectel Storage cloud storage that can be mounted on any modern version of Linux, FreeBSD, or Mac OS X.

Opportunities

Transparency

S3QL is practically indistinguishable from the local file system. It supports hardlinks, symlinks, standard system permissions, advanced attributes, and files up to 2TB in size.

Dynamic size

The file system size is not limited; it increases when data is saved and decreases after it is deleted. The cached data is stored on the local drive.

Compression

Before saving to the cloud storage, data can be compressed (lzma, bzip2, zlib supported).

Encryption

After compression (but before loading into the storage) the data can be encrypted with AES encryption algorithm with 256 bit key. An optional SHA256 HMAC checksum is also used to protect data from external manipulation.

Data deduplication

If several files have the same content, then duplicate data will be recorded only once, which allows you to effectively store a large number of similar data, with a minimum occupied volume. This functionality works both on whole files and on blocks of data that are repeated in different files.

Copy-on-Write / Snapshotting

S3QL can quickly copy entire directories without the need for additional disk space. Only when making changes to one of the copies, the changed data will take up additional space.

Data Protection Against Change

You can prohibit changing data for directories, for example, for storing backups that will be protected against accidental deletion.

High performance regardless of network latency

Operations associated with changing the structure (creating directories, moving files and changing permissions) are very fast, since they do not require network interaction.

Slow Link Support

S3QL splits the contents of files into small blocks and caches them locally, which reduces the number of network calls for reading and writing data, and also reduces the amount of data transferred, since only changed data blocks are transmitted over the network.

Installation

The s3ql package is available in the Debian (sid) and Ubuntu repositories, the installation of which boils down to the usual launch of apt-get install s3ql.
If you do not have this package in the repository or you are not happy with its version, you can always install it manually using the instructions .

File system creation

mkfs.s3ql swift://auth.selcdn.ru/container
By default, all inodes are divided into 10MB blocks, this parameter can be set when creating the --max-obj-size file system. After creating the file system, this parameter cannot be changed.
Also, by default, when creating a file system, you will need to enter a password to encrypt data, but if you do not need encryption, you can specify the --plain option and then the data will not be encrypted, which can increase the performance of the file system.

Mounting

Mounting cloud storage is very simple:
mount.s3ql swift://auth.selcdn.ru/container /mnt/container

Compression

When mounting a file system, you can specify the necessary data compression algorithm, but you must understand that the stronger the compression, the greater the load on the processor and the lower the speed when saving data. Compression algorithms relate to each other as follows: lzma is slow, but has better compression relative to bzip2, and bzip2, in turn, has better compression relative to zlib, but slower than it. For maximum file system performance, it’s best to choose a compression algorithm based on network bandwidth

Caching

S3QL supports a local file system data cache for faster access. The cache is based on data blocks, so it is possible that only parts of the files are in the cache.

Additional teams

s3qladm- used to change the password for encryption, update the file system, delete all data and download metadata from the store;
fsck.s3ql- sometimes after losing the network connection to the repository or after the fall of s3ql itself, it is necessary to check the file system, without which there is no possibility of further mounting;
s3qlcp- allows you to replicate directories within the same s3ql file system;
s3qlctrl- to manage the current settings of the mounted file system, for example, cache size and logging level, and also allows the forced execution of flushcache and the loading of meta-data into the storage;
s3qllock- makes folders immutable;
s3qlrm- recursively deletes files and directories on the s3ql file system, can also delete directories that are marked as immutable;
s3qlstat - Displays statistics for the mounted file system.

Unmount

umount.s3ql /mnt/container
Only the user who mounted it or root can unmount a file system. When unmounting, data is reloaded into the storage that have not yet been recorded - the command is blocked until the download is completed.

Total

pros

• encryption on the client side, which ensures storage of fully encrypted data in the storage;
• Using data deduplication can reduce the cost of storing data in the cloud relative to the traditional approach;
• data caching can reduce the cost of outbound traffic for storage.

disadvantages

• There is no possibility of multiple mounting (simultaneously on different machines).

PS For a more detailed acquaintance with s3ql, you can use its documentation

Source of article .