Cloud.Mail.Ru + EncFS without local file storage

Synopsis

It's no secret that many of us have 1 TB of free space on the MRU Cloud since its beta test. Decent volume by today's standards - but what to do with it? I don’t really want to upload photos and my video there just like that - account hacks often happen, and in any case, the cloud is a cloud, and you can’t discount the simple fact that any cloud storage belongs to a commercial company, in the interests of which it’s used to self-interest. So, you need an additional layer of protection, for example, EncFS. We look through the hub and github, it seems that there is a decision to encrypt data in its cloud. But there is a non-obvious, but very important inconvenience, which is not mentioned in the original article - in order to synchronize, you need to locally store 600GB of encrypted photos. For modest local storages, in which unencrypted 600GB photos can hardly fit, this is too much.

Idea

Therefore, an ambitious plan was invented - to write a FUSE-file system for the Cloud, and already mount EncFS on top of it (for better security, generate a GPG key and use ecryptfs, because modern computer capabilities already allow you to crack EncFS). In a way, it will be like using a MEGA cloud we drag and drop the file into the browser window - E2E encryption is applied and the file is sent to the cloud without occupying anything locally. The name was decided by "MARC-FS" - MAail.Ru Cloud FileSystem. With its help, we get away from the indicated problem of local storage of encrypted data - by copying the data to the EncFS directory, we will also automatically encrypt them and send them to the cloud. In fact, at a lower level we will call a rather long chain of transformations of the form cp -> kernel FUSE module ->

   +---------------+
   |               |
   |   Local data  |
   |               |
   | /media/DATA/  |
   |               |
   +---------------+
           |
           |
           |
           |
   +-------v-------+
   |               |
   |     EncFS     |
   |               |
   |  ~/remote-enc |
   |               |
   +---------------+
           |
           | FUSE
           | encryption
           |
   +-------v-------+
   |               |
   |    MARC-FS    |
   |               |
   |    ~/remote   |
   |               |
   +---------------+
           |
           | FUSE
           | networking
           |
   +-------v-------+
   |               |
   |               |
   | cloud.mail.ru |
   |               |
   |               |
   +---------------+

Implementation

The plan is ambitious, but in fact, I had to face many limitations both from the side of FUSE and the kernel, and from the side of the Cloud. Most probably, the most disturbing thing is that the Cloud uses data deduplication. That is, files that have the same hashes and sizes are stored in a certain common pool, and after uploading, users simply add a “hardlink” to them. Of course, your encrypted files will be unique, because are protected by your key, in connection with which there is concern that Mail.ru would not pay attention to such personal clouds and make a complaint about this. Hopefully, there were not many users of OS X / Linux who dared to also beta test Mail.ru.

Some other interesting details:

• The cloud does not support downloading from Transfer-Encoding: chunked, and in the FUSE interface there is no way to know in advance the file size before it is written. First you have to load the file into memory, and only then, knowing the size, send it over the network.
• EncFS requires random read / write for its work, as it breaks files into pieces, which it later encrypts. For network file systems (NFS, for example), this is fear and horror, because the protocols do not contain instructions such as "write 4096 bytes at this offset from the beginning of the file." Therefore, for the correct operation of EncFS, it is necessary to keep the file in memory again while working with it (Somewhat later, I added the ability to specify the cache directory in which the intermediate data will be stored, for more details see README.md in the repository - author's note).
• The connection with the cloud is established via https, and this, suddenly, adds a rather large memory overhead to initialize the SSL engine. Moreover, it is not clear how to deal with this, and other network file systems also come across this .
• Although the size of the storage and terabytes, Cloud has a limit on the size of the file uploaded by one request - 2 GB, and only paid services make it possible to upload more. This is suitable for a library of photographs or music, but I won’t be able to upload my collection of HD films, so later I added a transparent split of files - if I upload or download files larger than 2GB, the FS will break or glue them myself.
• In MacOS, file descriptor multiplexing is used, this is its difference from Linux, so I had to link the project not with the system present libfuse, but with the osxfuseone preinstalled using Homebrew Cask. It contains the appropriate workarounds. Also, of the possible problems, I can note that mounting one FUSE-file system to another on MacOS is disabled by default due to recursion problems, but this is bypassed using the mount options manually.

As a result (and maybe, nevertheless), a viable option was obtained, with the help of which it was possible to fill the entire collection of music and photos and animeto the cloud as encrypted files. The result of the work on MARC-FS is presented here , there you can also find instructions for setting up the EncFS bundle. Work is underway and the front of improvements is wide, pull requests and suggestions, of course, are welcome.

An example of an encrypted directory:

Locally decrypted:

On the cloud:

A couple of marginal notes

I could not find any description of the Cloud API; I had to pull it out from related articles and projects. But we must pay tribute to Mail.Ru - even when working in 25 threads and hundreds of getattr / read / write requests per second, only occasionally a 500 Internal Server error slips in response. I don’t see him so often, but if someone from the Cloud team reads this article and needs details, I can get a few of these reports.

libfuse when writing the file system, and indeed the FUSE architecture in its concept reminded userspace client-server drivers, for example in GNU / Hurd or in Plan 9 From Bell Labs . It is good to see that interesting ideas that were abandoned with the loss of relevance of these systems received a new life.

Multithreading for FS had to be invented from scratch and on the knee, so this was an interesting challenge. There is no clear certainty that everything turned out in the best possible way (and without bugs), but it works - and it works quite quickly. Unless there are doubts about how the Cloud itself will relate to the simultaneous upload and download of files.

It should be warned that I did not test MARC-FS in detail on Mac OS X. My friend and I achieved a stable build and copied the music in the terminal a couple of times, but no more. I don’t have Apple-iron at hand and I can’t effectively test the operation of the FS, so if someone can support the operation of the FS on this platform, I will be glad.

Disclaimers

I publish the code and this article in the hope that they can come in handy for someone, but I do not offer a guarantee of presentation or suitability for use. Cloud's internal API can change at any second, and the chance to end up with a trough is pretty high. Use at your own risk.

I know that in the license agreement Mail.Ru there are lines about the prohibition of automatic means of interaction, but none of legal_dep could understand my question in letters and objectively clarify the situation. In the end, we all use email clients that interact in the same way with Mail.ru. I was redirected twice to D. A., who honestly replied that legal advice was not his path.

References

• The code
• Readme

PS Before they warn me about the possible consequences of writing file systems - I already know .