Rabbit MQ in the processing system of residents

Recently, they successfully put into operation a system for processing citizens' complaints. The bottom line is that when you do not have water, heating or a huge hole in the road near your house, you can complain about a problem in government agencies. There are various platforms where you can file a complaint: websites of state institutions, pages on social networks, call centers.

Our task was to create a single pipeline for processing applications for all departments.
The main goal of the system is to speed up the process of processing calls as much as possible: automate everything automated, control the timing at each stage of the process, inform residents about each step.

One of the specific tasks of the project was the issue of integration with a large number of external systems.

• It was necessary to learn from different sites to pick up the entire flow of complaints, to be able to communicate with them on all changes with applications, to conduct correspondence between public servants and citizens on clarifying the details of complaints.
• In addition to this, we gave away some of the functions to third-party services.

Because There was a lot of data, often had to work in asynchronous mode, then the project team had to solve the problem, so as not to strangle itself and third-party systems. The solution was found in the software message broker Rabbit MQ. It was a new technology for the team at that time.

Below is an interview with the project backend developer, Alexander Shcheglov, WilyLynx , who dealt with the issue and implemented integration.

- Sasha, hello! Please tell in a nutshell what is Rabbit MQ?

The software is intended mainly for the implementation of delayed messaging between different clients, i.e. when you don’t need an answer right now.

- I understand correctly that in general it works like this: the sending service puts data into the created queue as they are generated, the receiving service takes this information as necessary?

That is exactly what works.

- Why did you (development team) choose this solution for the project?

For several reasons. Firstly, in our case, synchronous processing (received and processed at the same time) is not critical, i.e. a message may hang in the queue for a while. In addition, ease of use: to receive messages you only need to declare a queue name and there is no need to write your services. Well, the availability of libraries for common YP. No need to invent anything to work with RabbitMQ.

- I understand correctly that Rabbit MQ allows you to control the exchange of data between systems and web services?

Rather, we still manage the exchange, but the “rabbit” is an excellent tool for organizing this exchange. Here you have the lifetime of messages in the queues, and the maximum length of the queue, and access settings, and clustering, and various exchange protocols, and a plug-in system and more.

- How is it determined that the message has been delivered? - that is, how is it determined that the client has prolonged something after receiving it or has hung in the process?

It is considered delivered if, within a certain period of time, a response from the client arrives. It, in fact, says that he received and was pleased with life. The client can send a response as soon as received, and then try to process the message. Maybe, on the contrary, first try to process and, if successful, send a response. Or you can tell the rabbit in advance so that he does not wait for confirmation of delivery from you and just receive messages. All sent items will be automatically considered delivered.

- Is it possible, for example, to somehow receive not all messages, but for example to subscribe only to messages on a specific application?

There is a slightly different situation. There is an option to send messages in which messages come to all customers. This option is called “publish / subscribe”. A good example: a new message in your public. You sent, all signatories received. And already upon receipt they thought to read or not to read. In general, no one bothers you to create a separate queue for yourself and work only with it. In this case, routing will already be at the application level, and the rabbit as a communication channel.

- Sasha, tell me, is there an option not to create thousands of queues, but to do filtering and routing for one?

One will not work, but Rabbit will allow some routing.

- Please tell us more.

One of them is not, but there are such concepts as “exchange” and “routing key”:

P - producer, the sender of the message in exchange
X - the exchange itself
Red bars - lines
C1 and C2 - recipients



Pabbit can send a message in exchange with a certain key (for example, error / info / warning). And as you can see, different recipients are geared towards receiving messages with different routing keys. Moreover, only C2 will receive a message with the “info” key, and both will receive a message with the “error” key. It is also possible to receive messages according to the template for the key. This is for another type of “Publish / Subscribe” exchange, which I mentioned earlier.



As you can see in any case, for each of the recipients in these types of exchanges there is a turn, but by the end we still have some semblance of filtering / routing.

- What can you recall the problems that arose during the study and implementation of Rabbit?

Besides laziness, no. Seriously, clear documentation, a large number of examples.

- Have you already transferred all exchanges with services and external systems to it?

We now have 38 queues: exchange between circuits, external systems, BI. But unfortunately, some services (read departments) are resisting. because they have implemented rest. In addition, some types of interactions involve the simultaneous receipt of responses to requests.

- What do you think, how successful was this decision?

For interagency collaboration that does not require a synchronous response? For me, it's a great option.

List of materials used