February 28, 2019 at 12:04How we use the Zabbix retail monitoring system
Monitoring systems such as Zabbix are not surprising for sophisticated users. However, in retail they are not frequent guests, and if used, then more and more for server control. We went further and use them to monitor cash register software and equipment.
Everything is completely transparent here: retailers and service companies rarely use monitoring systems, because it is difficult to evaluate their economic efficiency. With the introduction of business processes, everything is simple - X money and X effort. But to calculate how much they saved the retailer in the future is harder.
Service contractors usually do not implement monitoring systems also because they do not reduce the importance of their work. This is logical: Zabbix allows you to identify a problem before the client sees it. On the one hand, this improves the quality of the services provided. On the other hand, the client sometimes has the erroneous impression that his business processes are perfectly organized and work without any outside help. But this can be solved by providing reporting on time.
However, even those retailers who agree to implement a monitoring system usually end up with control of servers, office computers, uninterruptible power supplies, and active network equipment. We do this too:
As part of the information received, automatic requests are made in the Service Desk. A number of other data helps us in investigating incidents. Classic example: a user complains that his computer is slow. Without a monitoring system, it’s hard to track - either when the engineer connects everything will be all right, or the employee has a subjective impression (his weak working PC objectively works slower than a fancy gaming computer that is at home). Therefore, we are studying retrospective - graphs for the time when a person was observing a problem.
But all of the above is commonplace, nothing new. It just so happened that we went further and with the help of Zabbix began to monitor the performance of cash register softwareand cash register equipment. We do this for large international retailers, widely represented on the Russian market in both food and non-food segments. Also, our regional monitoring system was acquired by some regional networkers, who now can independently control the performance of their business processes.
Frankly speaking, the monitoring system was implemented in Pilot spontaneously, without any project and in parts. If the decision on this came from above, perhaps we would go along the path of other service contractors and would not bother. But we have initiated the introduction of linear employees - engineers. Faced with a particular breakdown of cash register equipment or software glitches, they were looking for how to prevent it in the future. And they came up with the idea of a monitoring system.
With it, we get three options for solving problems:
Of course, our monitoring system does not affect all cash registers because it does not always make sense. Take a barcode scanner. They either work or not. And in the second case, store employees will report a problem to us much faster than a monitoring system. Therefore, we focused on the control of POS-terminals and cash registers (CCP) .
CCP gives through the driver enough information that allows you to judge its performance. For instance:
Case study: many probably remember the case of a bug in the firmware of one of the manufacturers of KKT in December 2017? As soon as the first information about the problem appeared, we set up a trigger in Zabbix, signaling the firmware version containing the bug, and got a list of CCPs that needed to be addressed urgently.
As part of the cash program control, we monitor various signs:
Most often, Zabbix signals us about network problems: inaccessibility of devices, too long response time. Further there are difficulties with utilization of resources: low-power PCs are usually used for cash registers. The third most common problem is the validity of data from external systems.
Quite often, messages about incorrect local time come. Cash PCs usually do not enter into AD and the ntp service has to be configured there separately, which is sometimes forgotten. And the wrong time at the checkout is fraught with major problems for the store: for example, selling alcohol when it is forbidden, which can lead to a fine or loss of a license.
Another area of activity where Zabbix, quite incidentally, turned out to be quite useful, is the fight against fraud. It happens that contractors in the regions or individual field engineers, who are paid separately, enter into a conspiracy with the users of the customer and solve problems that actually did not exist. We can bring them to clean water by analyzing the indications of the monitoring system. While this is done manually, when a suspicious surge in activity is recorded in a certain location, but we are working to automatically verify applications with Zabbix readings in all cases where this is possible.
Now from our monitoring system receives from 15 to 25% of applications. This is a fairly small amount, but by the end of this year we want to bring it up to 50% for customers who have signed service agreements with us .
Why monitoring systems are rarely found in retail
Everything is completely transparent here: retailers and service companies rarely use monitoring systems, because it is difficult to evaluate their economic efficiency. With the introduction of business processes, everything is simple - X money and X effort. But to calculate how much they saved the retailer in the future is harder.
Service contractors usually do not implement monitoring systems also because they do not reduce the importance of their work. This is logical: Zabbix allows you to identify a problem before the client sees it. On the one hand, this improves the quality of the services provided. On the other hand, the client sometimes has the erroneous impression that his business processes are perfectly organized and work without any outside help. But this can be solved by providing reporting on time.
However, even those retailers who agree to implement a monitoring system usually end up with control of servers, office computers, uninterruptible power supplies, and active network equipment. We do this too:
- from servers we obtain data on the utilization of processors, the performance of fans, hard drives, memory, temperature of processors and motherboards;
- from uninterruptible power sources - statuses, charge level, information about how much time they will work in case of a power outage;
- from network equipment - traffic on ports, utilization of resources.
As part of the information received, automatic requests are made in the Service Desk. A number of other data helps us in investigating incidents. Classic example: a user complains that his computer is slow. Without a monitoring system, it’s hard to track - either when the engineer connects everything will be all right, or the employee has a subjective impression (his weak working PC objectively works slower than a fancy gaming computer that is at home). Therefore, we are studying retrospective - graphs for the time when a person was observing a problem.
But all of the above is commonplace, nothing new. It just so happened that we went further and with the help of Zabbix began to monitor the performance of cash register softwareand cash register equipment. We do this for large international retailers, widely represented on the Russian market in both food and non-food segments. Also, our regional monitoring system was acquired by some regional networkers, who now can independently control the performance of their business processes.
Why did we start doing this?
Frankly speaking, the monitoring system was implemented in Pilot spontaneously, without any project and in parts. If the decision on this came from above, perhaps we would go along the path of other service contractors and would not bother. But we have initiated the introduction of linear employees - engineers. Faced with a particular breakdown of cash register equipment or software glitches, they were looking for how to prevent it in the future. And they came up with the idea of a monitoring system.
With it, we get three options for solving problems:
- preventively - fix the problem before it happens. For example, when monitoring a hard disk, we see that the space on it has been reduced to a critical level. And we are taking measures in this regard;
- after the fact - we solve the problem after it happened. For example, a fan on the processor has failed. The processor is still warming up, but it is working. Sooner or later, of course, it will fail, but so far we have the opportunity to replace the fan. That is, the user has not yet noticed the incident, but he already is. From his point of view, we solve the problem proactively, but from the point of view of equipment - after the fact;
- analytically - we get a large amount of data in retrospect for parsing incidents.
Of course, our monitoring system does not affect all cash registers because it does not always make sense. Take a barcode scanner. They either work or not. And in the second case, store employees will report a problem to us much faster than a monitoring system. Therefore, we focused on the control of POS-terminals and cash registers (CCP) .
CCT Health Monitoring
CCP gives through the driver enough information that allows you to judge its performance. For instance:
- Various inventory data - hardware versions, firmware, drivers, serial numbers. In general, the composition of the equipment on the service is fixed in the annexes to the contracts and stored in CMDB, however, the customer is free to move and replace the equipment as he pleases. Of course, he does not always recall that it would be nice to notify the service company about this. This is where the monitoring system comes to the rescue, which tracks the change in equipment configuration. We wrote an integration module that corrects CMDB according to inventory data from Zabbix. In addition to tracking the actual configuration of equipment at service facilities, it, coupled with the auto-detection functionality of the monitoring system, capitally reduces the time for starting inventory of a new client, if such work is provided for by the contract.
Case study: many probably remember the case of a bug in the firmware of one of the manufacturers of KKT in December 2017? As soon as the first information about the problem appeared, we set up a trigger in Zabbix, signaling the firmware version containing the bug, and got a list of CCPs that needed to be addressed urgently.
- The CCP status code is an excellent parameter that allows you to track almost any malfunction, ranging from incorrectly set time or overheating of the printer head to the presence of unsent fiscal data on the fiscal drive.
Cash Software Control
As part of the cash program control, we monitor various signs:
- the serviceability of the services - whether the software is turned on or not, whether it opens some network ports or is waiting for a connection;
- entries in the logs - usually the software writes to the logs about the problems encountered, generates a set of errors. As an indirect sign, if the logs change, then the software works, if there are no new entries in them, then you need to create a request;
- in fact, the log entries themselves - if an error message occurs, the trigger fires. After processing, the records are transferred to ELK: Logstash logs are scraped out through the Zabbix API;
- the results of the integration software that downloads, converts and sends data (for example, transfers information to EGAIS, OFD, receives a range of goods). So, a recently incorrectly formed data package with the nomenclature disabled the software of self-payment terminals, paralyzing their work in one of our client’s stores. Thanks to the monitoring system, we managed to localize the problem in time;
- software and driver versions - sometimes situations arise when, for example, the versions of two programs are not compatible, but for cash software to work, they need to interact;
- Databases - we monitor the serviceability of services, the availability of network ports, the number of databases, their versions and the number of turned off databases;
- external services (for example, EGAIS, with which we interact through IP networks in automatic mode).
Problems that most often enter the monitoring system
Most often, Zabbix signals us about network problems: inaccessibility of devices, too long response time. Further there are difficulties with utilization of resources: low-power PCs are usually used for cash registers. The third most common problem is the validity of data from external systems.
Quite often, messages about incorrect local time come. Cash PCs usually do not enter into AD and the ntp service has to be configured there separately, which is sometimes forgotten. And the wrong time at the checkout is fraught with major problems for the store: for example, selling alcohol when it is forbidden, which can lead to a fine or loss of a license.
Fraud and downtime
Another area of activity where Zabbix, quite incidentally, turned out to be quite useful, is the fight against fraud. It happens that contractors in the regions or individual field engineers, who are paid separately, enter into a conspiracy with the users of the customer and solve problems that actually did not exist. We can bring them to clean water by analyzing the indications of the monitoring system. While this is done manually, when a suspicious surge in activity is recorded in a certain location, but we are working to automatically verify applications with Zabbix readings in all cases where this is possible.
Now from our monitoring system receives from 15 to 25% of applications. This is a fairly small amount, but by the end of this year we want to bring it up to 50% for customers who have signed service agreements with us .
