s3ql - file system based on cloud storage
Using S3QL, you can create a file system based on Selectel Storage cloud storage that can be mounted on any modern version of Linux, FreeBSD, or Mac OS X.
S3QL is practically indistinguishable from the local file system. It supports hardlinks, symlinks, standard system permissions, advanced attributes, and files up to 2TB in size.
The file system size is not limited; it increases when data is saved and decreases after it is deleted. The cached data is stored on the local drive.
Before saving to the cloud storage, data can be compressed (lzma, bzip2, zlib supported).
After compression (but before loading into the storage) the data can be encrypted with AES encryption algorithm with 256 bit key. An optional SHA256 HMAC checksum is also used to protect data from external manipulation.
If several files have the same content, then duplicate data will be recorded only once, which allows you to effectively store a large number of similar data, with a minimum occupied volume. This functionality works both on whole files and on blocks of data that are repeated in different files.
Copy-on-Write / Snapshotting
S3QL can quickly copy entire directories without the need for additional disk space. Only when making changes to one of the copies, the changed data will take up additional space.
Data Protection Against Change
You can prohibit changing data for directories, for example, for storing backups that will be protected against accidental deletion.
High performance regardless of network latency
Operations associated with changing the structure (creating directories, moving files and changing permissions) are very fast, since they do not require network interaction.
Slow Link Support
S3QL splits the contents of files into small blocks and caches them locally, which reduces the number of network calls for reading and writing data, and also reduces the amount of data transferred, since only changed data blocks are transmitted over the network.
The s3ql package is available in the Debian (sid) and Ubuntu repositories, the installation of which boils down to the usual launch of apt-get install s3ql.
If you do not have this package in the repository or you are not happy with its version, you can always install it manually using the instructions .
File system creation
By default, all inodes are divided into 10MB blocks, this parameter can be set when creating the --max-obj-size file system. After creating the file system, this parameter cannot be changed.
Also, by default, when creating a file system, you will need to enter a password to encrypt data, but if you do not need encryption, you can specify the --plain option and then the data will not be encrypted, which can increase the performance of the file system.
Mounting cloud storage is very simple:
mount.s3ql swift://auth.selcdn.ru/container /mnt/container
When mounting a file system, you can specify the necessary data compression algorithm, but you must understand that the stronger the compression, the greater the load on the processor and the lower the speed when saving data. Compression algorithms relate to each other as follows: lzma is slow, but has better compression relative to bzip2, and bzip2, in turn, has better compression relative to zlib, but slower than it. For maximum file system performance, it’s best to choose a compression algorithm based on network bandwidth
S3QL supports a local file system data cache for faster access. The cache is based on data blocks, so it is possible that only parts of the files are in the cache.
s3qladm- used to change the password for encryption, update the file system, delete all data and download metadata from the store;
fsck.s3ql- sometimes after losing the network connection to the repository or after the fall of s3ql itself, it is necessary to check the file system, without which there is no possibility of further mounting;
s3qlcp- allows you to replicate directories within the same s3ql file system;
s3qlctrl- to manage the current settings of the mounted file system, for example, cache size and logging level, and also allows the forced execution of flushcache and the loading of meta-data into the storage;
s3qllock- makes folders immutable;
s3qlrm- recursively deletes files and directories on the s3ql file system, can also delete directories that are marked as immutable;
s3qlstat- Displays statistics for the mounted file system.
Only the user who mounted it or root can unmount a file system. When unmounting, data is reloaded into the storage that have not yet been recorded - the command is blocked until the download is completed.
- encryption on the client side, which ensures storage of fully encrypted data in the storage;
- Using data deduplication can reduce the cost of storing data in the cloud relative to the traditional approach;
- data caching can reduce the cost of outbound traffic for storage.
- There is no possibility of multiple mounting (simultaneously on different machines).
PS For a more detailed acquaintance with s3ql, you can use its documentation
Source of article .