Redis, hiredis, libev, and multithread. Part 2
In the continuation of the first part I want to tell you how it all really works. A lot of time was put into testing and debugging, and now I want to lay out detailed recommendations on the results of studies that have been conducted.
Attention! Research was not conducted to understand why I need it, but to understand how it works!
So, what did we have in the first part: pure C-code, which was simply included in the C ++ code, joyfully compiled and yielded some results, with not very often entering data into the database. But we still want to approach real time, so we do a stress test, and we get ...
The first thing I stumbled upon is the client’s falling off the server in 10-15 seconds, which is logical, of course, when the data is expected by the server in asynchronous mode, but they do not come. To get rid of this misunderstanding, it is necessary to fasten the sending of the “PING” command to the server with a certain frequency, say, once every 100 microseconds (one hundred microseconds) after the previous request to the server has been completed. This will not affect the operation of the server, and the application will maintain a connection when data is not available.
After screwing up the ping, I began to observe in surprise the fallout to the cortex, and in the place where the hiredis manual recommended to clean the memory, here is the quote: “If the reply for a command with a NULL callback is read, it is immediately free'd. When the callback for a command is non-NULL, it is responsible for cleaning up the reply. ” However, reply was not NULL, but a stable drop in the same freeReplyObject function. Well, what to do, climbed into the source of hiredis, and ... The memory turns out to be cleaned 2 times! To begin with, I cleaned up memory clearing only for ping, but later it turned out that in the Callback function, it is not necessary to clear the memory at all. And you don’t have to clean it yet in functions that relate to receiving requests from other threads (those functions that relate to async-watcher), because the passed pointers are cleaned in the ev_io_stop function, which is called for each of the io-wacher's when the request is removed from the queue. At the same time, of course, one must deal with the so-called “private data” as Taras Bulba: “I gave birth to you - I will kill you.” By the way, for ping, you can specify private data as NULL, nothing falls anywhere, there are no memory leaks.
Surprisingly, on the one hand. When I tried to sip 10k push requests from a single thread into the database for 10 seconds (the figure is nothing at all with the declared Redis performance of 120k requests per second), I again encountered the problem of the connection falling off from the DBMS. We climb into the bowels again, make a trace and ... We begin to realize the phrase "multithreading is not supported by default, because there is no single-valued algorithm to make a thread-safe implementation." What is going on? The following happens: a semaphore that restricts access to write to the buffer of requests from other flows is insufficient, because this same buffer (and in hiredis it is present and can consume an infinite amount of resources) grows, and data in Redis is not sent at the same time, apparently a stream with event loop does not have read access to the buffer while recording is in progress. Initially, I solved the problem by setting a timeout between requests, then moved the semaphore to the bowels of the adapter to libev, setting the wait for the semaphore to add a new request and releasing the semaphore after sending it to the DBMS. However, I have not yet been able to fully understand the problem of the need for a timeout of 1 μs between requests. Perhaps in the next part (if there is one) I will already describe the recipe. Based on all this:
As a result, I managed to achieve ~ 600 push requests per second to Redis in a single stream with preliminary data processing in the service (I did not count how much time it took). In general, for now this is enough for me to start, but I will dig further into an increase in the number of threads and even more correct synchronization of adding requests to the buffer.
Say what you like, hiredis is a young library, in the process of debugging my code, I climbed into its bowels more than once. What didn’t make me happy was the fact that there are often constructions of the following kind:
In general, I patched for myself, wrote it off to the developers, quickly did not respond, but I think they will fix it.
Thanks for attention. I look forward to criticism and comments.
Attention! Research was not conducted to understand why I need it, but to understand how it works!
So, what did we have in the first part: pure C-code, which was simply included in the C ++ code, joyfully compiled and yielded some results, with not very often entering data into the database. But we still want to approach real time, so we do a stress test, and we get ...
Hiredis + Libev: kick me.
The first thing I stumbled upon is the client’s falling off the server in 10-15 seconds, which is logical, of course, when the data is expected by the server in asynchronous mode, but they do not come. To get rid of this misunderstanding, it is necessary to fasten the sending of the “PING” command to the server with a certain frequency, say, once every 100 microseconds (one hundred microseconds) after the previous request to the server has been completed. This will not affect the operation of the server, and the application will maintain a connection when data is not available.
Hiredis + Libev: be clean, but without fanaticism.
After screwing up the ping, I began to observe in surprise the fallout to the cortex, and in the place where the hiredis manual recommended to clean the memory, here is the quote: “If the reply for a command with a NULL callback is read, it is immediately free'd. When the callback for a command is non-NULL, it is responsible for cleaning up the reply. ” However, reply was not NULL, but a stable drop in the same freeReplyObject function. Well, what to do, climbed into the source of hiredis, and ... The memory turns out to be cleaned 2 times! To begin with, I cleaned up memory clearing only for ping, but later it turned out that in the Callback function, it is not necessary to clear the memory at all. And you don’t have to clean it yet in functions that relate to receiving requests from other threads (those functions that relate to async-watcher), because the passed pointers are cleaned in the ev_io_stop function, which is called for each of the io-wacher's when the request is removed from the queue. At the same time, of course, one must deal with the so-called “private data” as Taras Bulba: “I gave birth to you - I will kill you.” By the way, for ping, you can specify private data as NULL, nothing falls anywhere, there are no memory leaks.
Hiredis + Libev + Multithread: Я захлебываюсь!
Surprisingly, on the one hand. When I tried to sip 10k push requests from a single thread into the database for 10 seconds (the figure is nothing at all with the declared Redis performance of 120k requests per second), I again encountered the problem of the connection falling off from the DBMS. We climb into the bowels again, make a trace and ... We begin to realize the phrase "multithreading is not supported by default, because there is no single-valued algorithm to make a thread-safe implementation." What is going on? The following happens: a semaphore that restricts access to write to the buffer of requests from other flows is insufficient, because this same buffer (and in hiredis it is present and can consume an infinite amount of resources) grows, and data in Redis is not sent at the same time, apparently a stream with event loop does not have read access to the buffer while recording is in progress. Initially, I solved the problem by setting a timeout between requests, then moved the semaphore to the bowels of the adapter to libev, setting the wait for the semaphore to add a new request and releasing the semaphore after sending it to the DBMS. However, I have not yet been able to fully understand the problem of the need for a timeout of 1 μs between requests. Perhaps in the next part (if there is one) I will already describe the recipe. Based on all this:
Hiredis + Libev + Multithread: squeeze the maximum out of one thread.
As a result, I managed to achieve ~ 600 push requests per second to Redis in a single stream with preliminary data processing in the service (I did not count how much time it took). In general, for now this is enough for me to start, but I will dig further into an increase in the number of threads and even more correct synchronization of adding requests to the buffer.
Hiredis: file carefully before assembly.
Say what you like, hiredis is a young library, in the process of debugging my code, I climbed into its bowels more than once. What didn’t make me happy was the fact that there are often constructions of the following kind:
<some_struct> *p = (some_struct*)malloc(sizeof(*p));
// Не нуждается в комментариях, думаю.
// Некая переменная типа char[] где-то как-то обретает свое значение, например, строка ошибки (err), а потом следует вызов вот такого вида:
... = ... sizeof(err) ...;
// извините, меня на первом курсе еще по голове били больно за подобное...
In general, I patched for myself, wrote it off to the developers, quickly did not respond, but I think they will fix it.
Thanks for attention. I look forward to criticism and comments.