Happy new year with new MQTT / UDP
Hey.
As I wrote recently (the first short article on MQTT / UDP ), MQTT / UDP is an MQTT based protocol, but:
In a sense, this is CAN, but on top of Ethernet.
What for.
My apartment has been living for several years under the control of a system like “the house is not quite oligophrenic”. Mind it would be redundant to call it, but - light and climate are automated. To imagine the scale of the disaster - the system occupies a full jam-packed rack about 19 inches wide and two meters high. Two walls of the rack are occupied almost to the floor.
When I designed all this, the issue of fault tolerance came first. Several years of operation have shown: it is true. Denies everything. Sooner or later. Here are just electromechanical relays have not yet refused, namely, the last echelon of protection against failure.
The next problem after fault tolerance is a zoo. Due to the natural flow of life, my curiosity and urgent needs, the system lives on a normal Unix on a standard PC, ALC PLC, Raspberr + Orange PI, modules on Atmega, modules based on NodeMCU (ESP8266), and all this goes to each other through modbus 485, modbus TCP, http, and on the side hangs restless MQTT broker, as a legacy of an unsuccessful attempt to switch to it with the entire camp.
Why the attempt to switch to MQTT is unsuccessful. Firstly, for a part of the iron it is too heavy or complicated. On that fragment of Pascal, who is hiding in the CodeSys PLC, only a masochist can write an MQTT. But then it is necessary and debug. Similarly, atmega: you can cram, but closely. But this is not the whole trouble.
The MQTT as it is (and it is 3.1.1 everywhere) insists on sending the PUBLISH packet (that is, our message to the broker) to all recipients, including the sender. The effect of this insanity is enchanting - the same OpenHAB cannot send and receive data to the MQTT under the same name. This means that it is impossible to organize a bus on the basis of MQTT (several modules that update the value of the same object and use it). Namely, this is how the PLC communication should be organized, in which the main control of the light lives, OpenHAB, which controls the light from the web interface and the mobile application, and smart switches, which I would like to be able to add here and there. That is, of course, this problem can be circumvented, but in fact it means to build your own protocol OVER MQTT, and this already seems like a brute force.
At the same time, what do I need? Send an update of the parameter value and get it on all interested points. For what, actually, grandfathers and made UDP on the broadcast address.
After the last post on Habré, one of the readers for a long time reproached me with the unreliability of the UDP protocol. I responded to it speculatively that for IoT this UDP is MORE reliable than TCP: with 50% packet loss on the TCP network, it will not just lie down, but will lie ALL OVER, and the temperature sensor that sends measurements via UDP will simply lose half of the samples, which will affect the system works about nothing. At all.
But it was speculative. However, the New Year holidays are given to the person in order to finish the champagne to ask themselves: shall we put home lokalka on the shoulder blades today? And what would not.
I took MQTT / UDP and wrote a primitive test. One side sends consecutive packets without a pause between packets as much as they can. The second one considers speed and packet loss. The experiment was simple: we start this mockery, and in parallel two HD TVs show movies from the Internet, and the desktop computer writes a huge file on the NAS.
Rate the result. I was waiting for everything, but ... the maximum loss on UDP reached a full half percent, and both TVs stopped the show. So I was still a pessimist. In a real home network, UDP delivery reliability is close to absolute. However, the version with confirmations and resend packages I, apparently, will do. Difficulties a bit, but the question removes completely.
The second question is security, but, really, if the home network is hacked to me, the loss of data from the temperature sensors is the last thing I’ll grieve about. However, again, the task of digitally signing packages is present in the issue tracker, and I already understand how to extend MQTT-shny packages for its implementation.
In general, I plan on switching to MQTT / UDP on the first pair of devices in my home network just a few days ago.
And I suggest you try it in your programs and devices: Repository and documentation in English .
What is available:
Quite full implementations in Java, Python and C. Basic implementation on Lua. So far, only sending for Arduino and CodeSys PLC (tested and works on Aries, but should go on anything).
GUI tool for looking at what is happening and intervening:
Programs for sending and receiving data in Python, Lua and Java.
Python connectors for integration with regular MQTT and directly with OpenHAB. They practice loop protection by preventing the message from being broadcast back 5 seconds after passing in the forward direction. In addition, there is a library for limiting the flow of repeated data. It checks if the update has already been updated for the specified time, and if it has, then skips the new update only if the data has changed.
I plan to gradually move to this protocol completely, and here is one example of what I want with sensors:
There are three sensors in one room (well, or on the street, it does not matter). They send samples via MQTT / UDP. These counts are simultaneously obtained by the main system of the smart home (OpenHAB), a historical database and a wall monitor that shows the temperature to the residents.
The beauty of MQTT / UDP is that in such a scheme, the failure of any block does not create any problems for everyone else. And this is with the enchanting simplicity of the protocol.
Moreover, redundant control schemes with redundancy are easily implemented in the same scheme. The database server can be duplicated without any problems. Cunning will duplicate the modules that are involved in management. For example, listening to the temperature give out the control effect on the radiators. But, probably, this is the next story. Today I plan to stay on collecting signals from the sensors and the exchange between modules of the smart home.
As I wrote recently (the first short article on MQTT / UDP ), MQTT / UDP is an MQTT based protocol, but:
- Goes over UDP broadcast (no broker needed, almost no configuration needed)
- Up to indecent simple to implement (10 lines on C + UDP / IP stack - and you send data from the sensor)
- Everyone hears everyone
In a sense, this is CAN, but on top of Ethernet.
What for.
My apartment has been living for several years under the control of a system like “the house is not quite oligophrenic”. Mind it would be redundant to call it, but - light and climate are automated. To imagine the scale of the disaster - the system occupies a full jam-packed rack about 19 inches wide and two meters high. Two walls of the rack are occupied almost to the floor.
When I designed all this, the issue of fault tolerance came first. Several years of operation have shown: it is true. Denies everything. Sooner or later. Here are just electromechanical relays have not yet refused, namely, the last echelon of protection against failure.
The next problem after fault tolerance is a zoo. Due to the natural flow of life, my curiosity and urgent needs, the system lives on a normal Unix on a standard PC, ALC PLC, Raspberr + Orange PI, modules on Atmega, modules based on NodeMCU (ESP8266), and all this goes to each other through modbus 485, modbus TCP, http, and on the side hangs restless MQTT broker, as a legacy of an unsuccessful attempt to switch to it with the entire camp.
Why the attempt to switch to MQTT is unsuccessful. Firstly, for a part of the iron it is too heavy or complicated. On that fragment of Pascal, who is hiding in the CodeSys PLC, only a masochist can write an MQTT. But then it is necessary and debug. Similarly, atmega: you can cram, but closely. But this is not the whole trouble.
The MQTT as it is (and it is 3.1.1 everywhere) insists on sending the PUBLISH packet (that is, our message to the broker) to all recipients, including the sender. The effect of this insanity is enchanting - the same OpenHAB cannot send and receive data to the MQTT under the same name. This means that it is impossible to organize a bus on the basis of MQTT (several modules that update the value of the same object and use it). Namely, this is how the PLC communication should be organized, in which the main control of the light lives, OpenHAB, which controls the light from the web interface and the mobile application, and smart switches, which I would like to be able to add here and there. That is, of course, this problem can be circumvented, but in fact it means to build your own protocol OVER MQTT, and this already seems like a brute force.
At the same time, what do I need? Send an update of the parameter value and get it on all interested points. For what, actually, grandfathers and made UDP on the broadcast address.
After the last post on Habré, one of the readers for a long time reproached me with the unreliability of the UDP protocol. I responded to it speculatively that for IoT this UDP is MORE reliable than TCP: with 50% packet loss on the TCP network, it will not just lie down, but will lie ALL OVER, and the temperature sensor that sends measurements via UDP will simply lose half of the samples, which will affect the system works about nothing. At all.
But it was speculative. However, the New Year holidays are given to the person in order to finish the champagne to ask themselves: shall we put home lokalka on the shoulder blades today? And what would not.
I took MQTT / UDP and wrote a primitive test. One side sends consecutive packets without a pause between packets as much as they can. The second one considers speed and packet loss. The experiment was simple: we start this mockery, and in parallel two HD TVs show movies from the Internet, and the desktop computer writes a huge file on the NAS.
Rate the result. I was waiting for everything, but ... the maximum loss on UDP reached a full half percent, and both TVs stopped the show. So I was still a pessimist. In a real home network, UDP delivery reliability is close to absolute. However, the version with confirmations and resend packages I, apparently, will do. Difficulties a bit, but the question removes completely.
The second question is security, but, really, if the home network is hacked to me, the loss of data from the temperature sensors is the last thing I’ll grieve about. However, again, the task of digitally signing packages is present in the issue tracker, and I already understand how to extend MQTT-shny packages for its implementation.
In general, I plan on switching to MQTT / UDP on the first pair of devices in my home network just a few days ago.
And I suggest you try it in your programs and devices: Repository and documentation in English .
What is available:
Quite full implementations in Java, Python and C. Basic implementation on Lua. So far, only sending for Arduino and CodeSys PLC (tested and works on Aries, but should go on anything).
GUI tool for looking at what is happening and intervening:
Programs for sending and receiving data in Python, Lua and Java.
Python connectors for integration with regular MQTT and directly with OpenHAB. They practice loop protection by preventing the message from being broadcast back 5 seconds after passing in the forward direction. In addition, there is a library for limiting the flow of repeated data. It checks if the update has already been updated for the specified time, and if it has, then skips the new update only if the data has changed.
I plan to gradually move to this protocol completely, and here is one example of what I want with sensors:
There are three sensors in one room (well, or on the street, it does not matter). They send samples via MQTT / UDP. These counts are simultaneously obtained by the main system of the smart home (OpenHAB), a historical database and a wall monitor that shows the temperature to the residents.
The beauty of MQTT / UDP is that in such a scheme, the failure of any block does not create any problems for everyone else. And this is with the enchanting simplicity of the protocol.
Moreover, redundant control schemes with redundancy are easily implemented in the same scheme. The database server can be duplicated without any problems. Cunning will duplicate the modules that are involved in management. For example, listening to the temperature give out the control effect on the radiators. But, probably, this is the next story. Today I plan to stay on collecting signals from the sensors and the exchange between modules of the smart home.