January 28, 2011 at 01:17Upgrade
Hi all!
Surely, many of you have noticed that recently attempts to control the Internet are getting stronger and stronger, there are many examples - from the closure of torrents.ru without trial and investigation (and after all they work with copyright holders !!) to demonstration traps of different hackers, so that others would be quieter. Yeah. Fuck you.
I suggest organizing and starting to develop our new (national, hehe) data exchange system, which will be the development of existing torrents, emuls, and other things.
I must say right away that probably all this is somehow implemented in various information exchange tools - the goal is to make it more accessible, more convenient and more lively.
Caution - a lot of text without pictures.
I’ll just try to offer something of my own, but I will be glad of any additions to my thoughts. After all, I can not fundamentally be right, right?)
At the moment (this is just an idea and everything can change dramatically), there is an opinion that the XMPP protocol is very good and use it specifically for the interaction of clients and servers + use some (well, most likely BitTorrent) technology for exchanging data directly.
Accordingly, what we have as a basis:
For the basics, this is enough for us for the eyes and for the ears - you can implement almost everything that your heart desires, and therefore let's try to solve problems with the help of this toolkit.
Here I suggest just sitting and thinking about the structuring of distributions and TPX. At the moment, everything is on the rutracker and a lot of other things are stored, for example, in BB codes and TPX and this is often not very convenient for machines) However, all the same, when creating a distribution on the rutracker, they still need to fill out a form. So why not make it all more perfect and keep the structure instead of plain text?
For this, I see the sufficiency of introducing digital signatures under various content (mainly .torrent files, etc.). From this signature, you can understand what kind of person on the network checked it and, in fact, if anything, questions to him. Perhaps there is an option to organize a multi-level signature - say, first, a small moderator signs it, and then someone more responsible must sign it, etc. or you can organize levels of trust in the content in this way.
To protect against viruses, there is the following idea - to make just bots to which you will send content to be scanned for viruses and they will sign this content if everything passes the test.
When you enter “House MD” into the search bar on the root tracker, you’ll just get a hundred different distributions ... it’s sad and you need to sit and choose what you’d like and look at the headline, it’s kind of quality, but then it turns out that it’s not that season, but it’s not always a good idea to enter “Season 3” in the search bar - again, you can skip something.
You just need to create one distribution for everything and be able to select the .torrent file you need in one place and not rummage through all this. You can put a lot of all kinds of tools nearby - what subtitles are needed, what translation, what season, what picture quality, etc.
In order to be able to combine distributions, you just need to sign them all separately - when there is some distribution, and then more content appears, then he adds his own file to the existing one and he queues for moderation and this new content ( not all distribution) will be marked “not verified” (well, or whatever it will be better) until it is signed by the moderator and just like everything in clause 2. you just need to sign all this separately and separately implement the functionality from clause 2. Looks nice.
Everything is more complicated here. It is necessary to increase the viability of the network - for this you need to implement the following search algorithm for a new tracker.
To do this, we need the following functionality:
The first mechanism is needed in order to understand what to connect when the tracker lies down, and the second if the current tracker is working, but does not want to process this client.
With the second, everything is simple and clear - just with announce, return to him something like “redirect to ****” and that’s all.
But for the first, a special server configuration is needed - you need to create some network of servers (let it be a nexus), it is desirable that they are geographically distant. Each server from this nexus will have a prioritized list of this entire small network, which it will distribute to all its clients. If the tracker suddenly stops responding, then clients will go down the list of these servers and connect somewhere. At this time, there is an exchange of data within the network and determination of who will actually distribute the content now (this will happen very quickly - the network is small and just “agree”, but for implementation, you can estimate some heuristic and come up with some simple and unambiguous rules to determine the successor to the deceased tracker), and then after they have decided, everyone will be redirected to this new tracker.
This should not cause any problems with the configuration - people already set up backups of their torrents, and here you just need to talk to people and the server almost does not need to be configured - they will do everything themselves.
It is still important to remain anonymous and make this network more distributed. It is proposed here to make the hierarchy more complicated, but the complication should be such that it only strengthens the network, not weakens it.
Our trackers are already combined into nexus and can constitute a certain zone with each other, which can function as long as at least one nexus member is alive and all content from the dead nexus members is accordingly stored on it. Yes, there is a problem with the load, but if there are 20 servers in the nexus, then, probably, there will never be such a problem.
If it turns out that the load will be too large, then you can always raise the support server and redirect some clients to it, etc.
In order to identify the server for its participation in the nexus, digital signatures will be used, etc. This will protect the network from external influences. And the subsidiary auxiliary servers may have a different key and, accordingly, when creating a nexus, everyone exchanges a pair of keys - from the main server and from the auxiliary ones. There can be any number of auxiliary ones, but the main only one. This mechanism was created to simplify connecting and expanding the network on the part of one network participant - so that it would not be necessary to communicate and exchange keys with everyone. In fact, auxiliary servers are only proxy caches. But in general, their need is rather illusory.
This is very easy to implement through a response to announce, so let's not get distracted by this.
To do this, you can simply implement for each user a section on the distribution of which he can give links that cannot be opened without, for example, a generated password. In principle, there is also nothing particularly complicated here - trackers are always there and will work. In principle, you don’t even have to store everything on servers, but make a tracker directly in the client - so that users on these private distributions make requests to this client through the XMPP server.
Not all is not there, they are just a mass, and beyond the goal (or beyond desire - to someone like) is to create the perfect tool for the exchange of information. In addition to all of the above, there are lots of introductions to all of this. For example, to create an anonymous semi-automatic way of merging information into the network (for a la Wikileaks), to create opportunities for closed trackers (so that only some members of this super-network could participate in file sharing).
As for the interface - I want to make a single basic interface and I do not want it to be just the same browser. The browser has too few options for real file sharing.
And as a consequence of this - to have a certain hierarchy of nexuses, which would be visible in the palm of your hand "the whole Internet." For example, a breakdown by territorial characteristic, by direction (IT, music, etc.), etc. And in the nexus themselves also create their own hierarchy, etc. I even think that it would be interesting to start from this. And m. so that the nexus did not participate in the hierarchy, but the trackers themselves. It is likely that you will have to do something like DNS with the issuance of names in this hierarchy. But in general, there is already a lot to think about how to organize something, that there would be no cyber squatter, etc. Create the ability to create closed communities without your own tracker. But here the security issue already becomes - the server has all the information and, in principle, it can send everything to other network participants ... here you can use the “tracker in the client” mechanism with synchronization of network participants via an encrypted channel. In general, many, many good things.
In general, I propose to assemble a small team and start creating it all. If not us, then others. People are already scurrying about various things distributed (for example, the diaspora). All come in handy.
Waiting for everyone in the comments.
Surely, many of you have noticed that recently attempts to control the Internet are getting stronger and stronger, there are many examples - from the closure of torrents.ru without trial and investigation (and after all they work with copyright holders !!) to demonstration traps of different hackers, so that others would be quieter. Yeah. Fuck you.
I suggest organizing and starting to develop our new (national, hehe) data exchange system, which will be the development of existing torrents, emuls, and other things.
I must say right away that probably all this is somehow implemented in various information exchange tools - the goal is to make it more accessible, more convenient and more lively.
Caution - a lot of text without pictures.
Key Current Network Issues
- Lots of duplicate content
- Content is not always verified and is often viral
- Several different content formats are scattered across the network (often we find what we need, but not in the format we would like to)
- For a more or less fast jump, a tracker is still required and if it stops working, then the speed starts to fall. (Personally, my DHT generally puts the router pretty quickly, and so it is disabled on all home machines)
- At the moment, it is not so easy to distribute left content on torrents. And often it is completely closed
- There is no way to automatically update torrents (for example, for TV shows this is very important)
- The complexity of initiating the distribution of content without using a tracker (if I just want to transfer a file to a friend via n2p, then I just won’t succeed)
What can we do with this?
I’ll just try to offer something of my own, but I will be glad of any additions to my thoughts. After all, I can not fundamentally be right, right?)
General ideology
At the moment (this is just an idea and everything can change dramatically), there is an opinion that the XMPP protocol is very good and use it specifically for the interaction of clients and servers + use some (well, most likely BitTorrent) technology for exchanging data directly.
Accordingly, what we have as a basis:
- a set of servers that can exchange information with each other
- a set of clients that can exchange information with any of the servers and with any of the clients (but already through the server)
- the information itself can be absolutely any
For the basics, this is enough for us for the eyes and for the ears - you can implement almost everything that your heart desires, and therefore let's try to solve problems with the help of this toolkit.
Problem 1 - Duplicate Content
Here I suggest just sitting and thinking about the structuring of distributions and TPX. At the moment, everything is on the rutracker and a lot of other things are stored, for example, in BB codes and TPX and this is often not very convenient for machines) However, all the same, when creating a distribution on the rutracker, they still need to fill out a form. So why not make it all more perfect and keep the structure instead of plain text?
Problem 2 - Content is not always verified and is often virus-infected
For this, I see the sufficiency of introducing digital signatures under various content (mainly .torrent files, etc.). From this signature, you can understand what kind of person on the network checked it and, in fact, if anything, questions to him. Perhaps there is an option to organize a multi-level signature - say, first, a small moderator signs it, and then someone more responsible must sign it, etc. or you can organize levels of trust in the content in this way.
To protect against viruses, there is the following idea - to make just bots to which you will send content to be scanned for viruses and they will sign this content if everything passes the test.
Problem 3 - Several different content formats are scattered across the network
When you enter “House MD” into the search bar on the root tracker, you’ll just get a hundred different distributions ... it’s sad and you need to sit and choose what you’d like and look at the headline, it’s kind of quality, but then it turns out that it’s not that season, but it’s not always a good idea to enter “Season 3” in the search bar - again, you can skip something.
You just need to create one distribution for everything and be able to select the .torrent file you need in one place and not rummage through all this. You can put a lot of all kinds of tools nearby - what subtitles are needed, what translation, what season, what picture quality, etc.
In order to be able to combine distributions, you just need to sign them all separately - when there is some distribution, and then more content appears, then he adds his own file to the existing one and he queues for moderation and this new content ( not all distribution) will be marked “not verified” (well, or whatever it will be better) until it is signed by the moderator and just like everything in clause 2. you just need to sign all this separately and separately implement the functionality from clause 2. Looks nice.
Problem 4 - Tracker still required for more or less fast downloads
Everything is more complicated here. It is necessary to increase the viability of the network - for this you need to implement the following search algorithm for a new tracker.
To do this, we need the following functionality:
- Obtaining a suitable tracker for distribution from an almost arbitrary other tracker
- Redirect from one tracker to another
The first mechanism is needed in order to understand what to connect when the tracker lies down, and the second if the current tracker is working, but does not want to process this client.
With the second, everything is simple and clear - just with announce, return to him something like “redirect to ****” and that’s all.
But for the first, a special server configuration is needed - you need to create some network of servers (let it be a nexus), it is desirable that they are geographically distant. Each server from this nexus will have a prioritized list of this entire small network, which it will distribute to all its clients. If the tracker suddenly stops responding, then clients will go down the list of these servers and connect somewhere. At this time, there is an exchange of data within the network and determination of who will actually distribute the content now (this will happen very quickly - the network is small and just “agree”, but for implementation, you can estimate some heuristic and come up with some simple and unambiguous rules to determine the successor to the deceased tracker), and then after they have decided, everyone will be redirected to this new tracker.
This should not cause any problems with the configuration - people already set up backups of their torrents, and here you just need to talk to people and the server almost does not need to be configured - they will do everything themselves.
Problem 5 - It is not so easy to distribute on torrents
It is still important to remain anonymous and make this network more distributed. It is proposed here to make the hierarchy more complicated, but the complication should be such that it only strengthens the network, not weakens it.
Our trackers are already combined into nexus and can constitute a certain zone with each other, which can function as long as at least one nexus member is alive and all content from the dead nexus members is accordingly stored on it. Yes, there is a problem with the load, but if there are 20 servers in the nexus, then, probably, there will never be such a problem.
If it turns out that the load will be too large, then you can always raise the support server and redirect some clients to it, etc.
In order to identify the server for its participation in the nexus, digital signatures will be used, etc. This will protect the network from external influences. And the subsidiary auxiliary servers may have a different key and, accordingly, when creating a nexus, everyone exchanges a pair of keys - from the main server and from the auxiliary ones. There can be any number of auxiliary ones, but the main only one. This mechanism was created to simplify connecting and expanding the network on the part of one network participant - so that it would not be necessary to communicate and exchange keys with everyone. In fact, auxiliary servers are only proxy caches. But in general, their need is rather illusory.
Problem 6 - There is no way to automatically update torrents
This is very easy to implement through a response to announce, so let's not get distracted by this.
Problem 7 - The difficulty of initiating content distribution without using distribution on the tracker
To do this, you can simply implement for each user a section on the distribution of which he can give links that cannot be opened without, for example, a generated password. In principle, there is also nothing particularly complicated here - trackers are always there and will work. In principle, you don’t even have to store everything on servers, but make a tracker directly in the client - so that users on these private distributions make requests to this client through the XMPP server.
And are these all ideas?
Not all is not there, they are just a mass, and beyond the goal (or beyond desire - to someone like) is to create the perfect tool for the exchange of information. In addition to all of the above, there are lots of introductions to all of this. For example, to create an anonymous semi-automatic way of merging information into the network (for a la Wikileaks), to create opportunities for closed trackers (so that only some members of this super-network could participate in file sharing).
As for the interface - I want to make a single basic interface and I do not want it to be just the same browser. The browser has too few options for real file sharing.
And as a consequence of this - to have a certain hierarchy of nexuses, which would be visible in the palm of your hand "the whole Internet." For example, a breakdown by territorial characteristic, by direction (IT, music, etc.), etc. And in the nexus themselves also create their own hierarchy, etc. I even think that it would be interesting to start from this. And m. so that the nexus did not participate in the hierarchy, but the trackers themselves. It is likely that you will have to do something like DNS with the issuance of names in this hierarchy. But in general, there is already a lot to think about how to organize something, that there would be no cyber squatter, etc. Create the ability to create closed communities without your own tracker. But here the security issue already becomes - the server has all the information and, in principle, it can send everything to other network participants ... here you can use the “tracker in the client” mechanism with synchronization of network participants via an encrypted channel. In general, many, many good things.
Ok And then what?
In general, I propose to assemble a small team and start creating it all. If not us, then others. People are already scurrying about various things distributed (for example, the diaspora). All come in handy.
- UI programmers / Layout designers - without you, we won’t make a good UI like usability and designers see it!
- Server programmers - without you, we cannot create anything reliable!
- Designers - without you, we can never create a beautiful system that would be pleasant for everyone to use!
- Usability / UI-design - without you, we won’t make a convenient and competent application!
Waiting for everyone in the comments.
