Online checks on the federal network through RabbitMQ, 1C and black magic
Last year, the IT director of one of the largest agro-industrial holdings in Russia addressed us. The approach to the business that our client has implemented has been impressive. He was one of the first to realize the idea of a full-cycle enterprise - from the field to the shelf at the grocery store. Due to the availability and high quality of products, this holding has become a recognized brand that is known and chosen. At that time, the holding included more than 650 outlets and more than 20,000 employees distributed throughout the territory of the Russian Federation.
The customer needed to ensure the fastest possible delivery of checks to the center from all outlets of Russia, including food stalls in remote villages with occasional Internet and minimal computerization.
Given this specificity, the solution of the problem turned into a fascinating adventure with a tambourine, shamans and rabbit's paws in the face of RabbitMQ. How we built a federated cluster of queues and what we encountered under the cut.
About customer issues
A holding with a large number of distributed outlets, operating in the conditions of tough competition, demanded that its managers promptly take management decisions. For this, managers needed information on checks in “real time” mode.
In the current system, the delivery of the check from the cashier to the central system reached 3 days. At the same time, there was periodically recorded not bringing information about the perfect purchase (loss of check).
It was necessary to ensure receipt of information about sales "here and now" to optimize the delivery of products to retail outlets and promptly respond to changes in product and demand balances. This information must simultaneously get into the central IC of the holding on the basis of 1C and on the computer of the merchant in a particular store / outlet.
In addition, it was necessary to ensure a centralized and operational reassessment of goods over the network, and also to connect to the BI system of the holding a stream of online data from all outlets.
As a result, the criteria for project success were formed:
- checks punched at the store’s checkout must be visible in the central 1C system no more than 1 second from the moment of its fiscalization at the checkout if there is (proper operation) communication;
- in case of temporary communication failures, ensure prompt guaranteed delivery of checks to the central system after the restoration of the communication channel.
For the Silver Bullet team, this was a very tempting thing, because at that time we had developed our 1C adapter for RabbitMQ and the ability to launch it into battle at long distances and loads attracted geek minds. The concept of “check position in the center in 1 second” is not so new, we presented ideas for using queues in 1C back in 2013 , and even tried it out on one trading network, but at that time they were very experimental crutches that included Rabbit + 1C is C #, WCF and even some C ++.
Of course, we all saw it in smart books, which were written earlier. Therefore, I will not undertake to judge when exactly the idea of queues in integration projects began to take over the world.
Anyway, the theoretical part and the architectural concept proved to be effective, only a small part remained - to do everything, test, fix bugs, document and cover. : trollface:
Diving headlong
No one knows better than the business owner about his strengths and weaknesses, so all integration projects begin with an audit of client processes, understanding the starting point (AS IS). Usually, a business survey includes:
- analysis of information on how business processes are taking place now;
- gathering system requirements by interviewing key employees to understand how, in their opinion, business processes should be arranged;
- formalization and optimization of business processes.
To exclude discrepancies, we use the formalization of the AS IS and TO BE business processes in the form of BPMN schemes. It is also easier to understand it yourself, and it is easier for the customer not to miss anything when we go through the process diagram with him.
From the technical features, in the course of the audit, we found that:
- retail outlets use Windows machines - with installed heterogeneous operating systems (depending on the year of creation) from Windows XP to Windows 7, 8, 10;
- cash registers are running Frontol firmware with Windows Embedded on board;
- the contour of retail management uses heterogeneous software, including both solutions from vendors 1C, and its own development;
- communication channels are implemented according to the “what is, that is” principle in the range from cable lines to USB whistle modems;
- retail outlets are serviced by local outsourcing companies, and not by a centralized holding service;
- the functioning of the system essentially depends on the serviceability (proper functioning) of the merchandiser’s computer at each outlet.
AS-IS Integration Scheme
The audit also showed that the company's business processes practically do not need to be adjusted, but an infrastructure refinement is required to ensure the smooth flow of information.
In general, we were faced with the task of reliable delivery of documents between the three components of the process: a cash register, a merchandise computer and a central office. At the same time, the computer goods manager is a machine “under the table”. It can be turned on / off at the request of the merchandiser. Or even be turned off 23 hours from 24 hours. However, when leaving the twilight, the merchandiser should see the current set of prices, balances, item positions, etc.
Choosing a solution, collecting rakes and placing crutches
Event integration on the queues has long become a commonly used pattern. When you need to transfer something somewhere, with many links, in an unreliable environment, and at the same time stream data - you need events and queues. Therefore, we chose RabbitMQ, because it is easily integrated into any (so it seemed to us) environment, including the 1C platform, for which we already had an adapter made of AMQP protocol.
RMQ is a kind of data flow manager and allows for integration in the “almost real” time mode, while maintaining weak connectivity of the systems, withstanding the load, and so on and so forth ... A good server, in a word, much has been written about it in Habré.
One of the nice features is out-of-the-box clustering and the ability to build distributed server clusters that work together.
I always liked the pictures with the integration architecture in the queues. They always consist of three cubes, in the center of which is a message broker. Let this picture be in this article, so as not to violate the canon.
When constructing the scheme, the question arose - where should the queue server be located? In which states can we have a system? Found out that there are 5 regular and emergency situations in which work should not stop.
- All blocks are included.
- The center is disconnected, there is access to the merchandiser.
- Merchandise disconnected, there is access to the center.
- Both the merchandiser and the center are disconnected.
- 1C system is disabled.
Checks in all these situations should make their way; trading should not stop. When you restore the channel - checks must arrive to the subscriber. Let me remind you that the outlet - it can be a rural stall. There is no separate server installed on which RMQ could be installed. It turned out that the message broker should stand directly at the checkout. Servers are an unaffordable luxury, and Rabbit is quite lightweight and can work on a small POS terminal. So why not yes?
Of course, we did not make POS the only node of the RMQ cluster, but one of the nodes of the federated cluster was placed directly on the trading terminal, running Windows Embedded. It was a little easier to say than to do, but we did it in a fun and reckless way. What now and tell.
How to put RMQ Frontol terminal
I must say that Erlang and the RMQ server itself on the Windows terminal got up almost without any problems. Problems arose in the client, which must interact with the server from the cash register software.
Frontol cashier software has quite suitable documentation, from which we found out that it is possible to customize behavior using Javascript. “Yohhu!” We said and began to google the JS client for RabbitMQ. It quickly became clear that a bummer was waiting for us. The frontage is not quite javascript. Well, that is formally yes, the syntax is the same there, but the JavaScript machine itself is from the Windows Script Host, the same one that is VBScript, cscript.exe and so on. In short, he is very old, microsofto-specific, and no sane JS client of the rabbit will work on it.
However, in the WSH ecosystem, you can use COM objects, and we headed towards the RMQ client for .NET .
Current versions of this client no longer support .NET 3.5, which we had at the POS terminal, but fortunately, the client’s source code is open, and besides, there are tags on the project github, in which .NET 3.5 is still supported. Glory to Openscore! It remained to deflate the source code of the old version of the .NET client, put the Com-Visible checkbox there and upload it to the terminal.
Interaction with Frontol cashier
Cashier software has an API that can be used to interact.
functioninit() {
frontol.addEventListener("openDocument", "beforeOpenDocument", true);
frontol.addEventListener("closeDocument", "beforeCloseDocument", true);
frontol.addEventListener("closeDocument", "afterCloseDocument", false);
frontol.addEventListener("closeSession", "beforeCloseSession", true);
frontol.addEventListener("closeSession", "afterCloseSession", false);
addPolyfills(); // полифиллы рулят )
initRmqVariables();
createRMQConnection();
}The capabilities of the native JS, which is on board Windows, are extremely poor. For example, there is no Array.indexOf or JSON.stringify. But the world is not without good people. We remembered a popular browser crutch called "polyfills", and happily built them into the cashier. If we throw the jokes away, then all the magic tricks with JS were deliberately commented on meticulously so that future generations of administrators could clearly and quickly understand what was happening, where it came from and how it works.
It quickly became clear that the JS-API does not cover part of our cases, however, since Frontol incorporates Firebird DBMS, and an ODBC provider exists for it, then using the same ADODB we will be able to access Javascript directly from the cash register and get there we need the data.
functionafterCloseSession() {
var connection = getDatabaseConnection();
var qSelect = new ActiveXObject("ADODB.Command");
qSelect.ActiveConnection = connection;
qSelect.CommandText =
"SELECT ChequeNumber " +
"FROM Document " +
"WHERE " +
" State = 1 " +
" AND(ChequeType IN(0, 1, 2)) "Finally, direct work with the queue server from our JS is as follows:
functioncreateRMQConnection() {
factory = new ActiveXObject("RabbitMQ.Client.ConnectionFactory");
factory.UserName = rmqUser;
factory.Password = rmqPass;
factory.VirtualHost = "/";
factory.HostName = "localhost";
try {
rmqConnection = factory.CreateConnection(0);
} catch (e) {
thrownewError("Ошибка подключения к локальному серверу очередей. Обратитесь к администратору!\n" + e.message);
}
rmqChannel = rmqConnection.CreateModel();
rmqMessageProperties = rmqChannel.CreateBasicProperties();
rmqMessageProperties.ContentType = "text/plain";
rmqMessageProperties.ContentEncoding = "string";
rmqMessageProperties.DeliveryMode = 2;
}Overall picture of the solution
The architecture laid down the following principles:
- RabbitMQ server federation - RMQ - Federation servers mode, so that events are delivered to all recipients;
- local cash register survival - if an event occurred at a cash register, then it must be delivered in any case, even if at the moment the cash workplace is not available over the network;
- 2 data providers - in normal mode, data is delivered through the server (merchandise computer), in case of unavailability (as a result of an accident or for other reasons) of the merchandise computer - delivery is carried out by the cash register itself, when the merchandise computer is turned on it will receive its portion of events later than the center, but guaranteed;
- RMQ is a RabbitMQ server with TCP ports for clustering and messaging;
- To ensure the delivery of messages, data stream redundancy has been applied. This minimizes the impact of network connectivity between system nodes;
- in 1C: The center decided to install a set of RMQ servers in ON mode — high availability of the central server, incl. there is dual redundancy and event replication, to guarantee delivery.
In accordance with the architectural solution, the following data flow diagram was created:
- Frontol, 1C - the initial and final points of the exchange - objects that are sources and recipients of messages;
- federated queue - a special type of queue that allows you to create a distributed system for transmitting messages, in which publication to the queue can be performed at a node located at an outlet (upstream) and retrieved from a queue at the center (downstream). For the transfer of messages from upstream to downstream, there is a special plugin (Federation PlugIn) installed on the central queue server;
- Shovel - (literally "shovel") - a mechanism for transferring messages from one object (queue) to another. Objects can belong to one server or different;
- The system architecture provides for remote control of deployment, that is, the ability to install RabbitMQ servers on any computer of the holding network from a single center;
- the configuration and connections to the federation are traversed using the so-called Post-Install scripts, which regularly come with the delivery of RMQ servers, with the necessary configuration of the network parameters adopted in the Customer’s network.
The resulting circuit provides the ordered characteristics for speed and works stably when any of the components fall out. After the loss and restoration of communication, the accumulated data goes to the center within 5-10 seconds. In practice, the technology of loosely coupled systems has proven its effectiveness. All events take place as if in the same office, without taking into account the various types of delays associated with territorial distribution and the various degree of availability of communication channels typical of it.
Brief conclusion
I would like to separately enjoy the amazing time we live. More recently, the use of Open Source in the production was such a kind of devotion. “Does your open source software work? So how do you like this cactus? ”Today, this is an absolute maxim.
I can’t imagine a company that doesn’t have an open source product in production. Thanks to the availability of information, open source, to implement business ideas has become much easier and faster. Need to put RMQ on ancient non-standard Javascript and Windows? There is nothing easier. Is the booze solution a bit wrong? - look how it is done and add the missing. Time-to-Market when using open products is reduced by several times. And everyone knows that a quick release of a solution means a competitive advantage.
Github, Stackoverflow, open documentation and standards allow you to run in a matter of weeks what would have required many years of expertise in various areas of computer knowledge.
And of course, I am particularly pleased that the 1C-nick community every year goes further and further out of its closed world and merges into global IT. For example, “1C: Enterprise” today is one of the official languages supported by Github, and people from the 1C community “taught” it to this language. This story probably deserves a separate article, perhaps I will write it someday. In the meantime - all the best and good luck to you!
Thank you for your time!