Network Protocol Design
I searched the article on the design of protocols over the hub and to my surprise did not find anything. Perhaps it is worth sharing your thoughts on sabzh then. I must say right away that the division into types is purely mine and may not coincide with what you find in the directories. We also agree in advance that the C / C ++ language is used.
Consider the main types of protocols:
These protocols are based on communication in small pieces of data. The communication protocol is usually strongly typed. An example is the well-known AT-commands for modems.
This type of protocol is the easiest to process (requires an elementary parsing of a line with data isolation). But to communicate with such a protocol on more or less serious tasks is not very easy. Such protocols are well suited for sending small chunks of data of scalar types (strings, numbers).
Nevertheless, designing such protocols is also necessary.
We need to transfer 10 numbers from the client to the server, and get a string or number in response (varies from input data). To do this, we need: the stage of “shaking hands”, the stage of sending data, the stage of receiving a response.
“Handshake” : sending the word “HELLO” back and forth is enough (if we know that it is possible instead of the server to get to another client, we can separate the client and server greetings, for example, “HELLOCL” (from the client) and “HELLOSRV” (from server)). The processing is elementary and comes down to string comparison.
Data transfer : take the command “SEND x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” as the send command from the client and “OK SEND” as the server response. The processing is again elementary and comes down to string comparison and calling sscanf ().
Receiving a response : we agree that the server sends an “ANSWER STR string ” (if the response is a string) or “ANSWER NUM number ” (if the response is a number). The client responds with the OK ANSWER command.
It would seem that everything is simple and clear. BUT. Indeed, in this way we will not be able to understand which set of numbers the sent answer belongs to. To solve this problem is simple. When sending data, we will use the commands: “SEND id NUMS x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” and “OK SEND id ”, where id is a unique identifier for this set of numbers. When responding, the corresponding commands will be as follows: "ANSWER id STR string ", "ANSWER id NUM number"," OK ANSWER id ". Such a protocol will already be exhaustive in most cases.
This type of protocol is the most common. Its basis is the rigid typing of a portion of the data sent. That is, we know in advance that such packets (such an offset and such length) will contain such and such data (the offset and length of some fields can also be specified in the structure, but the initially specified offsets are mainly used). Such protocols are almost all low-level protocols.
The undoubted advantage of such protocols (especially under the condition of rigidly specified offsets and lengths of all fields) is the extreme simplicity of processing. We just need to check the size and execute the memcpy () command in an instance of the structure corresponding to our package.
When designing such protocols, you need to remember about the features of storing structures in C. I mean what is called packing. The fact is that any instance of any structure must be aligned in memory with a certain multiplicity (the multiplicity is set for the whole program one). By default, a multiplicity of 4 is used (I do not advise changing it, since this value strongly affects, for example, the namespace std). This means that the size of any structure will always be a multiple of 4 (if the size of the structure was 14, then 2 meaningless bytes will be added to the end and the size will be 16). Therefore, we need to take care of the same value of this parameter on the server and client.
Also, the main mistake in designing such protocols is inattention to storing multibyte types in memory. It must be remembered that x86 stores them in the form of little-endian (from the youngest to the oldest), and by the standards of network protocols and for example in SPARC computers it is necessary to store them in the form of big-endian (from the oldest to the youngest). Thus, we need to know in what order multibyte types will come to us and, if necessary, turn them over. Moreover, if we need high speed, we have a large flow of exchanged data and we can’t get away from rotations (for example, the criteria are relevant when developing a cross-architecture system of distributed computing), then it is necessary to give the rotation function an extra half an hour, but write it as fast as possible. In such cases, standard htonl (), ntohl () may not be in time.
I refer to tagged protocols now fashionable XML-like protocols. Such protocols, although extremely redundant, are nonetheless easy to process and completely flexible. Their main problems are redundancy and not very high processing speed.
The main mistake in designing such protocols is the desire to shove everything at once. It is necessary to clearly define the requirements for the protocol and isolate that subset of the functionality that is really necessary. Such an approach to designing is perhaps not too extensible, but it will allow us to save on processing time. Moreover, with proper design of the parser module, we can reduce the extensibility problem to a minimum (add a couple of functions and checks to the common code).
To me personally (due to the specialization in speed-demanding and well-structured network applications), this approach seems unnecessary and wasteful.
Perhaps the most interesting type of protocol. Allows you to combine high parsing speed and flexibility.
Packets of this protocol are divided by type. You can also design a tree of type subordination (for example, package A can only be included in package B, but cannot go separately). The basis of such packages is a rigidly defined header structural part (for each type of its own) + non-rigid part of the data (the same approach is used in structural data with a variable length of the last parameter in the structure).
Parsing such protocols, although more complex than parsing structural protocols, is easier and faster than parsing purely tagged protocols.
Typically, the type of package is written as the first field in the header, and we just need to read it and call the necessary function, which will copy the header into the structure, and the data into a piece of memory (usually char * is enough, but in some cases it is more convenient to copy directly to an array of certain structures).
The main mistake in the design is the desire to make "as in the tag protocol" and create a bunch of different types with small headers. This leads to the loss of high parsing speed and negates all the advantages of this type. Thus, it is necessary to balance between slowness and low speed. As practice has shown, in most cases the ideal tag break is the same as the class break if this protocol would be a regular data structure.
PSThis post is more of a review than directed to specific implementations. If anyone has a desire to read about any specific implementation (both well-known, and designing any others), then write in the comments, I will try to write.
Introduction
Consider the main types of protocols:
- question answer
- the structure
- tags
- hybrid of tags and structures
Question answer
These protocols are based on communication in small pieces of data. The communication protocol is usually strongly typed. An example is the well-known AT-commands for modems.
This type of protocol is the easiest to process (requires an elementary parsing of a line with data isolation). But to communicate with such a protocol on more or less serious tasks is not very easy. Such protocols are well suited for sending small chunks of data of scalar types (strings, numbers).
Nevertheless, designing such protocols is also necessary.
Example
We need to transfer 10 numbers from the client to the server, and get a string or number in response (varies from input data). To do this, we need: the stage of “shaking hands”, the stage of sending data, the stage of receiving a response.
“Handshake” : sending the word “HELLO” back and forth is enough (if we know that it is possible instead of the server to get to another client, we can separate the client and server greetings, for example, “HELLOCL” (from the client) and “HELLOSRV” (from server)). The processing is elementary and comes down to string comparison.
Data transfer : take the command “SEND x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” as the send command from the client and “OK SEND” as the server response. The processing is again elementary and comes down to string comparison and calling sscanf ().
Receiving a response : we agree that the server sends an “ANSWER STR string ” (if the response is a string) or “ANSWER NUM number ” (if the response is a number). The client responds with the OK ANSWER command.
It would seem that everything is simple and clear. BUT. Indeed, in this way we will not be able to understand which set of numbers the sent answer belongs to. To solve this problem is simple. When sending data, we will use the commands: “SEND id NUMS x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 ” and “OK SEND id ”, where id is a unique identifier for this set of numbers. When responding, the corresponding commands will be as follows: "ANSWER id STR string ", "ANSWER id NUM number"," OK ANSWER id ". Such a protocol will already be exhaustive in most cases.
Structures
This type of protocol is the most common. Its basis is the rigid typing of a portion of the data sent. That is, we know in advance that such packets (such an offset and such length) will contain such and such data (the offset and length of some fields can also be specified in the structure, but the initially specified offsets are mainly used). Such protocols are almost all low-level protocols.
The undoubted advantage of such protocols (especially under the condition of rigidly specified offsets and lengths of all fields) is the extreme simplicity of processing. We just need to check the size and execute the memcpy () command in an instance of the structure corresponding to our package.
When designing such protocols, you need to remember about the features of storing structures in C. I mean what is called packing. The fact is that any instance of any structure must be aligned in memory with a certain multiplicity (the multiplicity is set for the whole program one). By default, a multiplicity of 4 is used (I do not advise changing it, since this value strongly affects, for example, the namespace std). This means that the size of any structure will always be a multiple of 4 (if the size of the structure was 14, then 2 meaningless bytes will be added to the end and the size will be 16). Therefore, we need to take care of the same value of this parameter on the server and client.
Also, the main mistake in designing such protocols is inattention to storing multibyte types in memory. It must be remembered that x86 stores them in the form of little-endian (from the youngest to the oldest), and by the standards of network protocols and for example in SPARC computers it is necessary to store them in the form of big-endian (from the oldest to the youngest). Thus, we need to know in what order multibyte types will come to us and, if necessary, turn them over. Moreover, if we need high speed, we have a large flow of exchanged data and we can’t get away from rotations (for example, the criteria are relevant when developing a cross-architecture system of distributed computing), then it is necessary to give the rotation function an extra half an hour, but write it as fast as possible. In such cases, standard htonl (), ntohl () may not be in time.
Tags
I refer to tagged protocols now fashionable XML-like protocols. Such protocols, although extremely redundant, are nonetheless easy to process and completely flexible. Their main problems are redundancy and not very high processing speed.
The main mistake in designing such protocols is the desire to shove everything at once. It is necessary to clearly define the requirements for the protocol and isolate that subset of the functionality that is really necessary. Such an approach to designing is perhaps not too extensible, but it will allow us to save on processing time. Moreover, with proper design of the parser module, we can reduce the extensibility problem to a minimum (add a couple of functions and checks to the common code).
To me personally (due to the specialization in speed-demanding and well-structured network applications), this approach seems unnecessary and wasteful.
Tags + Structures
Perhaps the most interesting type of protocol. Allows you to combine high parsing speed and flexibility.
Packets of this protocol are divided by type. You can also design a tree of type subordination (for example, package A can only be included in package B, but cannot go separately). The basis of such packages is a rigidly defined header structural part (for each type of its own) + non-rigid part of the data (the same approach is used in structural data with a variable length of the last parameter in the structure).
Parsing such protocols, although more complex than parsing structural protocols, is easier and faster than parsing purely tagged protocols.
Typically, the type of package is written as the first field in the header, and we just need to read it and call the necessary function, which will copy the header into the structure, and the data into a piece of memory (usually char * is enough, but in some cases it is more convenient to copy directly to an array of certain structures).
The main mistake in the design is the desire to make "as in the tag protocol" and create a bunch of different types with small headers. This leads to the loss of high parsing speed and negates all the advantages of this type. Thus, it is necessary to balance between slowness and low speed. As practice has shown, in most cases the ideal tag break is the same as the class break if this protocol would be a regular data structure.
PSThis post is more of a review than directed to specific implementations. If anyone has a desire to read about any specific implementation (both well-known, and designing any others), then write in the comments, I will try to write.