How to teach Zabbix to send problem notifications directly to the desktop
Hi, Habr! My name is Ilya Ableev, I work in the monitoring department of Badoo.
You can take a long time on the topic of which monitoring system is steeper. The main tool for reacting to problems in Badoo is Zabbix, and we have repeatedly shared our experience in customizing it.
Out of the box you can learn about new problems either through the web interface or by mail / SMS. It works well and until a certain moment we were satisfied with these methods.
To date, there are several server instances in our system, thousands of hosts, hundreds of thousands of checks, and about the same triggers that need to be addressed as quickly as possible. After a series of incidents, when we only spent up to ten minutes on finding a problem (which is completely unacceptable), we realized that we needed other ways to report problems. Then, perhaps, our reaction would be faster.
The best solution, in our opinion, would be to display important notifications on top of all windows (browsers, chats, consoles). In this article we will talk about how we finished Zabbix, teaching them to send them directly to the desktop.
To suit this solution:
- duty administrators, who not only look at the monitor around the clock, but are also engaged in other tasks;
- all other administrators who do not like to view the PROBLEM sheet and OK in mail or SMS, but who want to timely find out about critical issues.
We'll have to spend time installing, but I think if you are using Zabbix and have already used custom scripts, it will not be difficult for you.
Attention! Receiving notifications in this way is addictive, its long-term use creates the impression that it is an integral part of Zabbix.
I'll tell you in steps what and how we did. The technical implementation is quite simple:
- Zabbix sends an alert for an event.
- The script sender on the server sends data to the client on the desktop via UDP.
- The recipient script receives the notification, and triggers the appearance of a pop-up message through an additional application.
The third item may differ depending on which operating system you are using and which application you like best.
Step One: Configure Zabbix
The official documentation details how to create your own alerts .
The project code is available on GitHub: https://github.com/ableev/ZbxDsktp .
- Beforehand we will put zbxpush.py in AlertScriptsPath on the Zabbix server.
- Next in the web interface: Administration → Media types → Create .
3. We prescribe the IP / FQDN in advance, to which we will send alerts.
This item can be upgraded.
Например, у нас сделано так: существует главный сервер мониторинга, при SSH-логине скрипт запоминает IP зашедшего и начинает отправлять на него алерты. Таким образом, чтобы активировать получение данных у себя, не нужно каждый раз менять экшен.
In the web interface: Administration → Users → % username% → Media .
4. Create an action on the triggers.
In the web interface: Configuration → Actions .
The first is the numeric format of the trigger (4 or 5 affects the displayed icon: High or Disaster), the second the notification header, and the third the message body.
@@@ - so that it is convenient to break into columns, because anything can be written in the names of the triggers (at least, we have :)), but not exactly this combination.
Step two: the server sends data to the client
The logic is simple: we receive an event, send it to the client. Here you can change UDP to TCP. We chose UDP for one simple reason: Zabbix sends all notifications sequentially, which means that if your computer is not available, if you use TCP, notifications will be sent with a long delay.
But UDP is unreliable.
Читатель, до которого доходит шутка про UDP, заметит, что в случае отправки алертов по UDP появляется вероятность потерять их где-нибудь между сервером и десктопом. И будет прав. Но суть не в том, чтобы гарантированно доставить сообщение, а в том, чтобы просто и быстро привлечь внимание к дашборду с проблемами: «Эй, там что-то случилось, пойди и посмотри».
Do not forget to make the script executable!
Step three: receive the event and display notifications
zbxlistenerd.py- a script that will spin in the background and launch notifications
settings.cfg- a file with client settings
icons/4.png- icons for different criticalities of the triggers
Then our paths diverge. The “client” part involves a bit of creativity, as each creates an environment for himself.
What is necessary:
- icons corresponding to the critical points of the triggers;
- sound of pop-up notifications;
- performing an action by clicking on a notification (SSH login, opening a link, initiating a ticket about a problem, etc.).
In our company, Linux and Mac OS are mainly used as the desktop OS, so let's look at a few examples for them. If you implement the same for Windows, welcome to the comments! We are happy to complement the post.
I will give an example that happened to use: notify-send in Xfce (in the screenshot above). Why notify-send, but not the same Python using libnotify? Yes, because it is in all popular desktop distributions (Ubuntu, Fedora, SUSE) and works with all (at least popular) DE (Gnome, KDE, Xfce).
Advice: if you experience periods of mass unavailability of something (for example, the switch fell off - and you are overwhelmed with a mountain of triggers about server unavailability), immediately configure the hotkey to complete the notify-send process.
Since the solution was initially implemented on the basis of a free application, and then we tried a paid one, there will be two examples.
After a brief search, a free terminal-notifier was chosen .
- You can set different icons for different criticalities;
- You can create links to graphics in Zabbix and open them by clicking on the notification;
- You can implement the opening of SSH-links: saw the notification -> clicked -> immediately went to the host.
Disadvantages: none (especially after icon support was added).
To enable it, set growl_enabled = True in settings.cfg.
At the time when the terminal-notifier did not know how to use custom icons, Growl did quite well with this. Therefore, our curiosity defeated greed - and we tried this rather popular application (which, I must say, can not only change icons - this is the whole center of notifications).
- you may already have the application installed;
- You can select different icons for different criticalities;
- as well as in the terminal-notifier, you can customize actions by clicking on notifications.
If you don’t have Growl, I recommend using terminal-notifier.
What we got as a result:
- fast delivery of notifications to admins;
- a decrease in the reaction time to events (from the moment “learned” to “repaired” or “passed on”);
- the ability to perform simple semi-automatic actions:
Further, it all depends on your creativity.
Once upon a time, when I was working at an ISP, when the Internet channel fell from all the speakers and headphones at the workplace, the sound of a siren could be heard. This led the attendant (me) to a stupor, did not adequately report the problem by telephone, but he (I) tried to solve this problem as soon as possible.
Personally, I already know cases of screwing to the script the sound of a pig from a popular antivirus and a talker from a popular translator. :)
With free time, desire and modern technology, you can do so
Quick to you reactions to incidents!