Transcend Library 4: Introduction

Working with a network in c ++ is just one, two, three.

network n = OpenNetwork();
connection c = n.OpenConnection("habrahabr.ru", 80);
html_request h("GET", "/");
c << h;
c.WaitForMessage();
n.Response();
std::cout << c;

Good afternoon, dear Khabravites, today I want to tell you about my work over the past 4 years. About my creation, about my life. It will be about a multifunctional library for working with the network, about its history and the reasons for its creation, about the tasks to be solved, as well as about the internal structure.

I want to make a reservation right away, maybe my work is a bicycle, but neither at the time of creation, nor at the time of the transition to new versions I could find the analogues satisfying me.

History

Start

It all started back in 2008, in the summer. We just went through winsock, tried to make our first server. I didn’t get a teacher, because I really didn’t understand anything. I was a fool and didn’t understand anything.
I was also extremely upset by the inconvenient api, and immediately wanted to encapsulate the whole thing. I did not know C ++ then, and the first version was just a renaming of functions (now lost).

More and more I moved away from the native winsock scheme, making communication more transparent. (For example, when you call the connect function, a socket was created immediately, a dns request was made, and a connection was opened). Gradually, it became so convenient that I decided to try to make a game.

Ancient legend

Be that as it may, prior to TL2, there were no significant improvements, and the "library" itself, God forbid, had 300 lines of code.
With the advent of a real task, a problem arose - the transfer of several data in one connection (chat, texture, music). It is from this that progress itself began. Over the course of six months, I solved this problem, selected the optimal package splitting, the format of service messages, and did tests. As a result, the library successfully coped with the task, with support for the simultaneous transmission of data up to 255 "messages" simultaneously, with a maximum size of 4 gigabytes each.
Unfortunately, we did not write the game in connection with the personnel leak, altruists, after all. Yes, and competitors have appeared. Therefore, I was able to rest in the development of future versions.

TL4

The next iteration was the transition to C ++ and the addition of new standards, the use of new chips, new features.
Now the library fully began to work in two modes at the same time - server and client, allowing you to solve proxy or p2p tasks. The maximum message size has increased indefinitely, and the maximum number of simultaneously transmitted messages is up to 2 ^ 32 (while 255 are actively transmitted). Auto-reconnect features have been added, a priority has appeared for sending messages.
The library became asynchronous, with non-blocking sockets, reducing the load on the main thread at times. (For example, by default, opening a connection took me 443 ms in TL2, this number was reduced to 17 ms TL4).
Added encryption, compression, etc., the list can be continued for a long time.

The main tasks to be solved

The library allows, regardless of the platform, to do the following: To the
client - open a connection, and transfer any data set from any sources (file, array, other connection) and be sure that no matter how many times the connection is interrupted, the data will be delivered.
The server needs to listen to the port and it is always intuitive to have information about the number of connections, traffic on each connection, online time, average speed, and more.

Example "PROXY"

#include 
using namespace ax::tl4;
bool proxy( connection &c, message &m )
{
  std::string s;
  s << m;  // запрос начинается host:pass, не усложняем
  connection nc = c.GetNetwork().OpenConnection(s);
  nc << s;
  nc << c;
  nc >> c;
  return true;
}
void main()
{
  network n = OpenNetwork();
  port p = n.OpenPort(8080);
  p.NonTL4Compatible(true);
  p.OnOpenConnection(proxy);
  while (1)
    n.Response();
}

Of course, the example is simulated, without error handling, and indeed a curve (if you try to convert message to std :: string, the program will be blocked until \ 0 arrives or by timeout), but the essence is clear.

Opportunities

• No protocol restrictions
• Support encryption, compression
• The most accurate synchronization
• The introduction of your handlers (encryption) at the message level, package. Fully modifiable packet construction pipe.
• Combining connections into “virtual networks” (groups processed in one go). All communications between networks are thread-safe.
• Transfer up to 4 billion objects simultaneously through one connection.
• Configurable connection parameters, software speed limit and the like.
• Custom event handlers for all events for each connection or group.
• Priorities for the transfer of objects, full control over "what exactly will be transferred now."
• The ability to transfer incomplete objects - for example, torrent, a stream of Internet radio. No size limit.
• The ability to process a “new object”, “a new piece of an object”, “completion of downloading an object”, “every N bytes”, “every N seconds”.
• Protocol development in one place.

Protocol in one place

This is a very convenient thing that allows you to develop client-server applications without duplicating code.
A small introduction. For each object and each package, a queue of handlers is called (it cannot be changed on the fly for obvious reasons). The handler has two methods - Encode and Decode. Thus, the encryption / decryption, compression / decompression code is always in one place. This allows you to check the correctness of the handler - just call the Test method, it will check on random or provided data that after encryption and decryption the original version is obtained.
The same is in your protocol - for the protocol as a whole, there are the same methods that allow you to convert an object into a message and a message into an object.

How it works

Stream

Everything in the programming world can be represented through a sequence. An array, string, file is all a sequence of bytes. Also about a variable and an inbound-outbound connection.
The stream class system was created in order to abstract from the physical source of the stream and abstract from the transformations actually occurring on the information.
Something similar to stl :: stream but with some improvements.

Pipe

Imagine a simple task, we want to transfer a picture from disk. To do this, you need to download it, convert bmp-> jpg, encrypt it. Now this can be done simply by placing the handlers one by one, and saying that this is an atomic transformation (anology with water supply - pipes go butt to butt).

pipe *ImageTransferPipe( std::string filename )
{
  pipe_constructor c;
  c.add(new BMP2JPG());
  c.add(new BlowFishCrypt());
  pipe p(c);
  return p.GetScheme().Source(new read_file_stream(filename));
}

I’m sure the first time it doesn’t seem very clear and convenient, but be sure, after reading the api it becomes simple.

Out-of-box encryption

As you might have guessed, when creating a connection, there is only one logical functional:

connection::SetHandler( pipe * )

Each byte during transmission will pass through this handler, and each byte during reception will also pass through the corresponding processor (pipe :: GetInverseScheme).

Transferring multiple objects at once

The idea is painfully simple, I hope it is useful to those who need it, and who have not done it yet.
All objects in the queue for sending are shredded by 255 bytes, and these blocks are sent. Blocks are supplied with the object id. Objects are transmitted cyclically - the first second third first second third.
I note that the blocks are always 255 bytes, and are not supplied with size. If suddenly the block is smaller, then through service messages it will be adjusted.
If suddenly the next portion of the object is “not ready” (no data), then it moves to the inactive queue, in exchange for one ready. All this is regulated through system messages.

System messages

They carry several functions. The transmitting side reports the active-inactive rotations, warns of the end of the object approaching, reports the addition of objects, the transmission of "low-level" exceptions, the transmission of the pipe format for the object, and the most interesting thing is synchronization.

Machine Sync

For correct operation it must be turned on at both ends. Because of this task, it was necessary to realize the priority of objects, it is necessary that these service messages are added to the queue at the time of creation.
Hosts exchange their timestamp with milliseconds, thus receiving ping. After setting the average value, they begin to exchange it, too. As a result, both machines have response times.
The most interesting part starts in the server solution. Players with ping more than a couple of hundred will understand me. The enemy will learn something important a few milliseconds earlier, and that’s all.
Now imagine a game where events come to all machines at the same time, and this is done simply. When sending a concurrent message, the connections are sorted in the order of delays, and the actual data is sent so that the estimated arrival time is equal, taking into account the stability of the connection (error less than 10ms for wired solutions).

Why this post?

The fact is that the library is not complete. For example, not all planned connection parameters are supported, I would like encryption algorithms out of the box, check sum, and more. In general, work is still underway.
Here I didn’t appear in order to promote, but to find out if there are analogues (I’d like to steal a couple of ideas to see how people have it), if anyone needs it (it makes sense to go to the market) and find out what you wouldn’t like in such system.

Public apology

I apologize for my manner of presentation, errors and inaccuracies. I was unable to add the tag "I am PR" due to low karma. And indeed in vain, I probably do it.

UPD: Continued