July 20, 2011 at 12:49Search torrents in DHT network
There are few people among users and readers of Habr who would never have used torrents. And surely, many people understand the vulnerability of the central links of this system - torrent trackers (the history of the Pirate Bay and the former torrents.ru, now rutracker, is a confirmation of this).
However, not everyone knows that for a long time there has been an alternative to traditional trackers - a DHT network that allows you to organize distributions without a tracker. Until now, it has not received much distribution due to the impossibility of searching within the network - the desired file could be downloaded only if there was a magnet-distribution link. However, the first and so far unique search service on the BTDigg.org DHT network has recently appeared .
A service with a very concise design is sharpened for one function - search:
To reduce the dependence on trackers, in 2007, the developers of the bittorrent protocol adopted its modification based on the DHT structure , which allowed distributing without a tracker.
What is a DHT network? In fact, this is a distributed network of torrent clients that stores identifiers (hashes) of all public distributions - the so-called magnet-link, which are a 160-bit random number. Customer DHT nodes are responsible for announcing their torrents and for finding the ones requested. Due to the fact that all modern clients support DHT, we can say that in this network there are almost all public torrents (the network does not “see” private torrents and does not allow downloading). Practically - because DHT in the client can be disabled. The search is based on the meta information that the torrent file contains - that is, in other words, it searches by the name of the torrent and by the names of the files included in it. The user opens a magnet link in his torrent client,
The disadvantage of this solution is that DHT answers only one request - which IP addresses have peers that distribute torrent with this info hash. That is, you can start downloading via DHT only if you have the right magnet link. In this sense, the DHT network resembles the Internet without Google: you can get to the page you need only if you know its exact address. Yes, a DHT network can and is designed to work without torrent trackers, but if they were closed, the process of exchanging files would be much more complicated.
Therefore, it is obvious that this network, since its inception, needed its own search service. However, despite the simplicity of the idea, it turned out to be a very difficult technical task to implement. The creators of the service took about six months to understand that this is generally possible to do.
This is NOT a search service for torrent trackers. The functionality of BTDigg.org is much more powerful: this is the first and so far the only search engine on the DHT network - that is, in fact, on all public torrents on the Internet. As the initial data, info-hash requests from other nodes are used, obtained from our own DHT cluster. Further, after cleaning up the markups, these hashes get torrents from peers participating in file sharing. The contents of torrents are not downloaded at any stage.
The creators of the resource declare respect for the community, therefore they do not use any prohibited methods of data collection (which can increase the speed of information collection, but at the same time lead to an additional load on other network participants). Their 24x7 cluster, on the other hand, supports a DHT network. BTDigg analyzes the activity of distributions, indexing it once a day (the most popular distributions from the top 100 are in real time, every 10 seconds). Search supports the Sphinx search query language (unfortunately, I did not find the language description in Russian).
The search results provide information that contains a torrent file - that is, its name and size. The results can be sorted by relevance and the number of peers (that is, customers who put this file to download).
Of the additional features so far, only the top 100 torrents in 2 weeks and the real-time top 100 in the last 10 minutes, which is updated every 10 seconds without reloading the page.
In addition, the service also provides an API of its functionality - both for calculating the popularity of torrents on the network and for searching (currently used in the qBittorrent client). The API is open, but has limits on the number of requests per unit time.
Due to the technical complexity of the task, the project started recently - only in January 2011. Now its daily audience is 10,000 people, but the potential, of course, is much higher. The project is not yet very well known in RuNet (on Habré it was just a post back in the sandbox a while ago), but his international team clearly expressed Russian participation - at least I talked to her Russian-speaking representative with a rare name John Smith. For obvious reasons, they do not disclose more details about themselves.
By the way, about the copyright holders (who did not understand - they are the “understandable reason”). I will give the floor to John:
Therefore, so far, in order not to give rightholders the opportunity to negotiate in a vein different from an absentee intelligent discussion - well, there, complaints about the host’s abuses, seizure of servers, abuse in the media, etc. - they are forced to remain anonymous, although this complicates the development of the project.
Now we are testing and in the near future a lot of new features will appear on the site, including setting up and filtering issuance, as well as improving network coverage - BTDigg.org will have more (even very rare torrents) and index new distributions in DHT networks faster.
A special hello to the Russian audience - the creators of the service plan to make Russian localization, and the support now understands in Russian. There are no plans to monetize the service yet - perhaps an advertisement will appear if the audience does not mind.
However, not everyone knows that for a long time there has been an alternative to traditional trackers - a DHT network that allows you to organize distributions without a tracker. Until now, it has not received much distribution due to the impossibility of searching within the network - the desired file could be downloaded only if there was a magnet-distribution link. However, the first and so far unique search service on the BTDigg.org DHT network has recently appeared .
A service with a very concise design is sharpened for one function - search:
Torrent Hash Search Engine
To reduce the dependence on trackers, in 2007, the developers of the bittorrent protocol adopted its modification based on the DHT structure , which allowed distributing without a tracker.
What is a DHT network? In fact, this is a distributed network of torrent clients that stores identifiers (hashes) of all public distributions - the so-called magnet-link, which are a 160-bit random number. Customer DHT nodes are responsible for announcing their torrents and for finding the ones requested. Due to the fact that all modern clients support DHT, we can say that in this network there are almost all public torrents (the network does not “see” private torrents and does not allow downloading). Practically - because DHT in the client can be disabled. The search is based on the meta information that the torrent file contains - that is, in other words, it searches by the name of the torrent and by the names of the files included in it. The user opens a magnet link in his torrent client,
The disadvantage of this solution is that DHT answers only one request - which IP addresses have peers that distribute torrent with this info hash. That is, you can start downloading via DHT only if you have the right magnet link. In this sense, the DHT network resembles the Internet without Google: you can get to the page you need only if you know its exact address. Yes, a DHT network can and is designed to work without torrent trackers, but if they were closed, the process of exchanging files would be much more complicated.
Therefore, it is obvious that this network, since its inception, needed its own search service. However, despite the simplicity of the idea, it turned out to be a very difficult technical task to implement. The creators of the service took about six months to understand that this is generally possible to do.
What BTDigg can do
This is NOT a search service for torrent trackers. The functionality of BTDigg.org is much more powerful: this is the first and so far the only search engine on the DHT network - that is, in fact, on all public torrents on the Internet. As the initial data, info-hash requests from other nodes are used, obtained from our own DHT cluster. Further, after cleaning up the markups, these hashes get torrents from peers participating in file sharing. The contents of torrents are not downloaded at any stage.
The creators of the resource declare respect for the community, therefore they do not use any prohibited methods of data collection (which can increase the speed of information collection, but at the same time lead to an additional load on other network participants). Their 24x7 cluster, on the other hand, supports a DHT network. BTDigg analyzes the activity of distributions, indexing it once a day (the most popular distributions from the top 100 are in real time, every 10 seconds). Search supports the Sphinx search query language (unfortunately, I did not find the language description in Russian).
The search results provide information that contains a torrent file - that is, its name and size. The results can be sorted by relevance and the number of peers (that is, customers who put this file to download).
Of the additional features so far, only the top 100 torrents in 2 weeks and the real-time top 100 in the last 10 minutes, which is updated every 10 seconds without reloading the page.
In addition, the service also provides an API of its functionality - both for calculating the popularity of torrents on the network and for searching (currently used in the qBittorrent client). The API is open, but has limits on the number of requests per unit time.
About
Due to the technical complexity of the task, the project started recently - only in January 2011. Now its daily audience is 10,000 people, but the potential, of course, is much higher. The project is not yet very well known in RuNet (on Habré it was just a post back in the sandbox a while ago), but his international team clearly expressed Russian participation - at least I talked to her Russian-speaking representative with a rare name John Smith. For obvious reasons, they do not disclose more details about themselves.
By the way, about the copyright holders (who did not understand - they are the “understandable reason”). I will give the floor to John:
So far, there have been no serious raids on us (unless some porn studios require us to exclude links to their content from the database). From the point of view of the law (in our opinion) we behave like a regular search engine (google, yandex, etc.), even more so - we do not give direct links (a magnet link is not a link in its literal sense). But in the world of lawyers and copyright holders, the law can be turned in any direction and the pirate bay and rutracker examples are proof of this.
Therefore, so far, in order not to give rightholders the opportunity to negotiate in a vein different from an absentee intelligent discussion - well, there, complaints about the host’s abuses, seizure of servers, abuse in the media, etc. - they are forced to remain anonymous, although this complicates the development of the project.
Development plans
Now we are testing and in the near future a lot of new features will appear on the site, including setting up and filtering issuance, as well as improving network coverage - BTDigg.org will have more (even very rare torrents) and index new distributions in DHT networks faster.
A special hello to the Russian audience - the creators of the service plan to make Russian localization, and the support now understands in Russian. There are no plans to monetize the service yet - perhaps an advertisement will appear if the audience does not mind.