How to attach a BitTorrent tracker to a Python site: integration with XBT Tracker

Suppose you have a website written in Python and you want to attach a BitTorrent tracker to it, like rutracker.org.

Task separation

The task can be divided into two large functionalities:

1. The catalog of torrent distributions on the site (historically usually implemented as a forum),
2. The Tracker itself, directly involved in the distribution process.

The tracker is an http-application, according to the specification of the BitTorrent protocol, informing the client, upon request, about all the participants in the distribution. Since clients send requests constantly periodically, the Tracker should be productive: the response time should be minimal.

In the world of PHP, Catalog and Tracker are often not divided into two dedicated applications. For example, the popular TBDev Tracker exists as an application combining the Catalog Forum and the Tracker. (The author is apparently so tired of the popularity of his application that the site has not been updated for a long time and downloading the application from it is unrealistic. However, you can find many clones on the Web.)

Some Tracker implementations were originally written in Python, but then rewritten in C ++ for performance reasons. So these days, Python trackers do not exist (at least I could not find). Therefore, the only thing left is to install a separate Tracker application and integrate it with the Python Directory.

Control and protection

If you do not need any control, that is:

• You do not care what distributions your tracker will support - an anonymous user can add his distribution to your Tracker,
• and you think, on a routine basis, that an anonymous user can access any existing distribution,

then you just need to install one of the programs , choosing among them the lightest and most productive.
Further, you simply give your users the opportunity to create directory entries and upload and download .torrent files created by them themselves. Having previously informed them the correct announce address of your tracker, which they will write to the .torrent file when creating the distribution.

If you want to establish at least some kind of control, and even more so, as in my case, you want to restrict the access of some user groups to certain distributions, then for this you need the so-called “private tracker” .

In theory, this is done as follows:

• Preventing the addition of anonymous distributions: the tracker should only support distributions that your authorized users have added to the directory. When creating a Directory entry and attaching a .torrent file to it, the code for your site should add the distribution to the list of allowed distributions for the tracker.
• Prevention of receiving .torrent file by an anonymous user is implemented by means of the Catalog
• Restriction of user access to distribution (as well as tracking of users participating in the distribution) is carried out according to the principle: if someone can see the directory entry and download the attached .torrent file, then participation in the distribution is allowed. In this case, the tracker must identify the user. And this is done by flashing a unique user code in announce the tracker URL - on the fly, when the user downloads the .torrent file. That is, the Directory code should do this.
• Prohibition of “broadcast distribution exchange” protocols: Distribute Hash Table (DHT), Peer EXchange (PEX), Local Service Discovery (LSD), which allow working without a Tracker at all, and therefore without access control. It is implemented by setting the flag “private = 1” in the .torrent file.

Implementation

Given all of the above, my choice fell on the XBT Tracker , as the only implementation of a private Tracker, designed to integrate with any site-directory.
XBT Tracker is written in C ++ and installed by building from source. On Ubuntu Server 12.04, 64-bit is built the first time.

Dependencies

Interaction with XBT Tracker is assumed through the Tracker MySQL database, so our Python Directory will need to be able to write and read the MySQL database. For this, I used the package pymysql.
To parse and modify .torrent files, you will also need a packageBitTorrent-bencode

import bencode, pymysql

Functions

Adding an authorized user, if not already. Here the torrent_passtable field is xbt_usersused to link the user ID of your site and the user ID of XBT Tracker. Previously, torrent_pass was used to authorize the user using the key specified in announce URL, but now XBT Tracker uses a different algorithm - see below. The field uidis auto-incrementing.

    c = db.cursor()
    c.execute("SELECT uid, torrent_pass, torrent_pass_version, downloaded, uploaded FROM xbt_users WHERE torrent_pass = %s",
        (request.user.id,))
    rec = c.fetchone()
    if not rec:
        # insert a new user
        c.execute("INSERT INTO xbt_users(torrent_pass) VALUES(%s)", (request.user.id))
        c.execute("SELECT uid, torrent_pass, torrent_pass_version, downloaded, uploaded FROM xbt_users WHERE torrent_pass = %s",
            (request.user.id,)) #rowcount =0
        rec = c.fetchone()
        db.commit()
        uid, torrent_pass, torrent_pass_version, downloaded, uploaded = rec

Reading and parsing a .torrent file:

        with open(fpath, 'rb') as f:
            buf = f.read()
        bt = bencode.bdecode(buf)

Set the private flag by force and calculate info_hash (the private flag is included in the info section and affects its hash):

    if xbt_force_private:
        bt['info']['private'] = 1
    info_hash_obj = hashlib.sha1(bencode.bencode(bt['info']))

Registration of distribution in the table xbt_filesif there is none yet.

    sha = info_hash_obj.hexdigest()
    c = db.cursor()
    c.execute("SELECT flags FROM xbt_files WHERE info_hash=UNHEX(%s)", (sha,))
    flag = c.fetchone()
    if flag is None:
        c.execute("INSERT INTO xbt_files(info_hash, mtime, ctime) VALUES(UNHEX(%s), UNIX_TIMESTAMP(), UNIX_TIMESTAMP())",
            (sha,))
        db.commit()
    elif flag[0] % 2 :
        error_msg(pagename, request, _("Torrent is marked for deletion!"))
        return

Authorization key calculation for announce URL

XBT Tracker uses a very tricky key calculation algorithm. A set of values ​​is taken:

• User ID: xbt_users.uid,
• version of the user's private key: xbt_users.torrent_pass_version.
• server secret key (generated automatically at the first start of XBT Tracker): table xbt_config, value torrent_pass_private_key,
• info_hash value of this distribution: xbt_files.info_hash.

All this is magically mixed according to the principle "I twist and twist, I want to confuse":

    c.execute("select value from xbt_config where name = 'torrent_pass_private_key'")
    torrent_pass_private_key = c.fetchone()[0]
    s = "%s %d %d %s" % (torrent_pass_private_key, torrent_pass_version, uid, info_hash_obj.digest())
    sha = hashlib.sha1(s).hexdigest()[0:24]
    pwd = "%08x%s" % (uid,  sha)
    bt['announce'] = 'http://xbt.host:port/%s/announce' % pwd
    # remove other trackers
    if bt.has_key('announce-list'):
        del bt['announce-list']

From here the purpose becomes clear xbt_users.torrent_pass_version: by changing this value, you can make all previously downloaded .torrent files invalid. That is, it is a kind of password reset analog.
And finally, we encode the .torrent file back to the line, which we will send to the client:

    buf = bencode.bencode(bt)

Deleting a distribution

When deleting the attached file, we must remove the distribution from the list of registered distributions (table xbt_files).
There is a method of polite deletion, in which we mark the distribution as deleted, but in reality it is deleted by the tracker when it is completely downloaded by the last person.

c.execute("update xbt_files set flags = 1 where info_hash = UNHEX(%s)", (info_hash, ))

That's it. Further site twisting: displaying the number and list of distributors, counting the rating, displaying the list of files in the distribution, I refer to the decorations. They are implemented quite obviously: all the necessary information, including statistics, is available in the tracker database, and I, following Pierre Fermat, am sorry to waste a place on them.

The solution outlined is implemented as a plugin to the popular wiki engine: MoinMoin

It should be noted that the proposed solution has a significant drawback: virtual hosting with Python support is generally relatively rare, and I have never seen the possibility of building a C ++ application in nature. The only alternative, it seems to me, is to take some TBDev Tracker clone and bite out the code directly implementing the Tracker functions from there.

I hope my experience is useful to someone.