Telegram Unlimited network drive. Free

    Hello, Habr.

    Surely many people thought about it, maybe someone has this idea in TODO. She lay with me for about a year, but still managed to implement it in the form of a working prototype.

    TgCloud:


    • Open source virtual file system g **** code.
    • On the local disk - only metadata: names, size, folder structure, etc.
    • Data is stored in Telegram and is downloaded only when working with a file
    • The size and type of files is not limited, can be used with any OS

    Implementation details and a link to the repository under the cat.



    Idea


    The idea to create VFS based on Telegram came up while working on AudioTubeBot and VideoTubeBot - after I found a way to get around the 50mb limit for downloading files for bots - I studied the Telegram source code - I discovered a lot of interesting things. For example, download speed is limited mostly on the client itself, and not on the server.

    As a result, I increased the download speed to the Telegram cloud to ~ 240Mbit / s per session (it has been working stably for more than a year), which is 15 times faster than the official client.

    The logical step was to use unlimited file storage with high download speed as a network drive - many probably do, but the problem is that telegram clients are inconvenient to use to search and organize files. In addition, there is a limit of 1.5 GB per file size.

    Fuse


    To solve these problems, I first decided to make a Linux kernel patch and create my own virtual file system, which would conveniently group files uploaded to Telegram, using only metadata.

    In the process of studying the issue, it turned out that patching the kernel is not necessary at all. Linux has a special FUSE kernel module that allows developers to create new types of file systems that are accessible for users to mount without privileges. There are wrappers for almost all popular languages ​​for the module.



    Telethon


    To access Telegram, I use the Telethon library version 0.19.1 with a multi-threaded file upload modified by me (TelegramClientX).

    Vfs


    In order not to write a file system from scratch - based on python dedupfs , using fuse and sqlite to store metadata, which breaks files into blocks, considers the sha1 hash and reuses the same blocks - you just need to replace the methods of loading data into the storage with your own.

    Tgcloud


    By combining all this, we get a working prototype of tgcloud , so far without space speeds, but still working, I hope to attract the community to the development. The downloaded files are divided into blocks of 10 MB, sha1 is calculated and loaded into SavedMessages, the hash is stored in the local database and when reading the block, the file is located on it. Blocks are glued together in the necessary order and a stream of bytes is issued - as if you are reading them from disk.

    To run the code


    • You need to get api_hash and api_id on my.telegram.org and insert this data along with the phone number of your account in tg_access.py
    • Create a session by running from the project folder

      python3.6 download_service.py

      and entering the verification code
    • Install

      sudo yum install python-fuse
    • Create Mount Folder

      mkdir storage
    • Launch VFS:
      with debugging

      python dedupfs/dedupfs.py -df --block-size 10240000 storage/

      in the background

      python dedupfs/dedupfs.py --block-size 10240000 storage/
    • You can, for example, specify this folder as the source for filebrowser

    Thank you for attention.

    PS About the ethical side (from comments )


    as an option - cut the speed on the server side, limit the number of download threads, etc. - there are enough ways to limit the disk consumption even without “covering the bench”, but even with a cut speed — the presence of an additional level of abstraction over the downloaded data is stupidly convenient when you can organize files not only within chats and channels, but since it’s convenient for the user to this is the opposite - data will be loaded less due to reuse of blocks. If in the current system I need to edit a text document uploaded - I download, edit and upload it again, while 90% of the data is duplicated already downloaded, while using VFS - only changed blocks will be downloaded again.
    That is, for ordinary users, this is in general - good, if someone decides to use it to host porn - well, let them somehow fight with them, it's Telegram’s philosophy - that “even if a small number of people use it for bad deeds, it’s not means that you need to prohibit all this, etc. etc."

    My channel in Telegram
    Chat for questions
    Sources on GitHub



    The latest dual-processor configurations of dedicated servers with 2019 Intel Scalable processors are available on DEDIC.SH :
    • 2x Xeon Silver 4214 - a total of 24 cores
    • 2x Xeon Gold 5218 - a total of 32 cores
    • 2x Xeon Gold 6240 - configuration with 36 cores.

    The cost of a server with two Xeon Silver 4214 - from 15210 rubles / month
    . We are also ready to collect any configuration for you - write to us !

    If large powers of a dedicated server are not required - VDS from 150 rubles / month is what you need!

    Also popular now: