Not a minute offline - Choosing an industrial LTE router

In many systems, connection to the Internet is critical, and even a few minutes of downtime can be expensive, for example, in ATMs, alarms, monitoring and access control systems.
Such systems usually use several Internet connections and automatically switch between different channels if the main channel is unavailable.

I was tired of inventing failover on USB modems myself , and I wanted to delegate this to a separate smart piece of hardware. I decided to compare several industrial routers to find out if they live up to their name and how much better they are than my crutches. The article will compare routers of manufacturers iRZ, Termit, Robustel.

Basic requirements for routers:

• 2 SIM-cards and automatic switching between operators - the router should automatically detect the loss of communication on one operator and switch to the backup. This is especially important for mobile systems when cellular coverage is not perfect.
• Ethernet connection - the main system should see only the default gateway, and should not know about the modem subsystem on the router side and channel switching, therefore no USB.
• Support for modern VPN protocols: IKEv2 and OpenVPN - for ease of administration, and in order not to depend on real IP addresses, routers must be able to VPN. At the same time, support not only obsolete PPTP and L2TP, but also modern protocols.
• (Optional) GPS support, the availability of GPIO, serial RS-232/485 interfaces - these functions are not required for our tasks, but we will also consider them as a bonus, which may be useful to someone.

The problem of homemade crutches

Wi-Fi router with two USB modems (Photo: Dmitry Bevza dgl.ru)

Failover - emergency switching of the Internet channel to the backup, in cases of inaccessibility of the main one.

Usually, to reserve Internet channels, I used a household Wi-Fi router with OpenWRT firmware, and the mwan3 package . It allows you to configure automatic backup and balancing between several Internet channels, including between a wired connection and a USB modem.

The main drawback of this approach is the heterogeneous equipment and flimsy design of external USB-modems. I liked the Huawei E3372h modems with HiLink firmware (Ethernet emulation). These modems are quite stable and cheap, but when external antennas are connected, the whole structure becomes very flimsy. The antenna’s hard wires create a lever, and modems get out of the USB port, and the CRC9 antenna connector itself is not as reliable as the SMA, because it does not have a screw mount.


Assemblies on external USB modems are very fragile: antenna connectors are not fixed, modems fall out of ports

The design of two USB modems with external antennas had to be very hard to fix, but physical damage and dropping out of ports were still the main problem. This solution is suitable for static installation in hard-to-reach places, but it is absolutely not suitable for installation in a car or in an electrical panel on a DIN rail.

Sometimes the USB subsystem hung in the routers, and modems became unavailable. I had to write a watchdog script that would separately monitor the availability of modems, in addition to mwan3, which monitors the availability of the Internet. At some point, I was tired of it, and I began to look for a solution in one box, where all the problems had already been solved for me.

Model Comparison Chart

Industrial routers are more expensive than household routers, even with the separate purchase of modems. Therefore, the main criterion for choosing models for the review was the cost. Additional features like Wi-Fi, GPIO pins and GPS are not needed in our case, but we will still compare them as options that affect the cost.

Testing methodology

The main task of the compared routers is to provide uninterrupted Internet in any conditions, therefore we will compare how well the router switches to the backup channel in case of problems on the main one. To do this, we will simulate a loss of communication, and measure on the server side how quickly the router was able to reconnect.

As a target host, a small TP-Link MR3020 router will be connected to each tested router, which will act as a computer with a payload. It will run OpenVPN connected to the server. Monitoring will be launched on the server, monitoring the availability of the address inside the VPN tunnel.

That is, on the server side, one fixed IP address will be monitored assigned to mr3020 via a VPN tunnel. Regardless of which channel the traffic on the router goes through, the monitoring server will interact with one address. Accordingly, if the address is available, this means that the router has successfully switched the channel, and there is Internet in the network behind NAT.
In order to restore the OpenVPN tunnel as quickly as possible, on the server side, set the keep-alive option to the minimum values, and disable encryption:

cipher none
ping 1
ping-restart 4

Keep in mind that it takes time to reconnect the VPN, and in reality the Internet is recovering faster than can be seen on the graphs. However, such a test allows a more honest assessment of the fact of channel switching, as it shows that the clients behind the router were able to restore TCP connections and return to normal operation.


A TP-Link MR3020 is connected to each router, with a running VPN tunnel. A working Internet is considered to be the availability of the IP address 10.0.0.1 on the server side

PingPlotter - monitoring program

The monitoring program and the OpenVPN server are located on the same physical computer as the VPN server. I will monitor the MultiPing utility using regular ICMP Echo requests. The request interval is 1 second - this allows you to see on the graph even short-term inaccessibility of the host.


The PingPlotter program draws convenient ping graphs showing the delay and packet loss.

I admit, I tried for a long time to choose a monitoring system. I tried zabbix, cacti, bash + rrdtool scripts, but it all seemed too complicated for my tasks. As a result, I found the excellent PingPlotter program and its simplified version without route tracing - MultiPing. The program allows real-time monitoring of a group of hosts with a resolution of up to 1 second, and draw detailed graphs.

Antenna

Broadband antenna Triad MA-2697, suitable for the basic ranges of cellular communications

In order to exclude different reception quality due to different antennas, the same Triad MA-2697 antennas will be connected to all routers . According to the manufacturer, the antenna is broadband, and works in the bands GSM-900 \ 1800 MHz, 3G-2100 MHz, Wi-Fi-2400 MHz and 4G WiMAX-2600 MHz. I will not check the quality of the antenna. Even if it is bad, all routers will work in the same reception conditions.

Mobile operators

MegaFon and MTS SIM-cards will be installed in each router, with the usual tariff for individuals. The megaphone will be selected as the main, MTS standby. There is no logic in such a priority, the choice is random. At the test site, both operators have equally good LTE coverage.

Disadvantages of the test bench

Test stand deployed on the table

Since the test stand is stationary in the room, the quality of coverage of the cellular network does not change. In such conditions, it is impossible to artificially simulate poor coverage when there is a cellular network signal, but packet losses are too large. In a good way, it was necessary to place a stand on a train or car, and carry it to places far from the city. Then it would be possible to compare how well the built-in modems work with a bad signal.
In our case, we can only recreate a complete loss of communication on the main channel, and watch how quickly the router switches to the backup one.

Termit CR41P

The router is provided by Digital Angel (digitalangel.ru)


SoC : Atheros AR9344 rev. 3
RAM: 128MB
Flash memory: 16MB
Wi-Fi: 802.11n (2.4GHz)
Modem : Quectel EC25-E (LTE Cat 4)
Firmware : based on OpenWRT 05.15.1
USB Host
MicroSD
DIN rail mount
GPS / GLONASS: 3V active antenna
Serial interfaces: RS-232/485
I / O pins : 2 digital inputs (0-3V / 0-30V, galvanically isolated), 1 analog input (0-24V), 1 digital output (30V, 250 mA open collector), 1 relay output (40V, 4A SPST)



To install the SIM card, you need to unscrew the four screws and remove the back cover.

Well, this is an occasion to consider the insides in more detail. clickable The modem part is made in the form of a shield that is worn on the main board. In the corner you can see the relay for a powerful load output to the I / O pins. The most feature-rich router in the review. It has a wired WAN, and also allows you to use Wi-Fi in the station (client) mode, as a separate WAN. That is, it can have four backup Internet channels. The web interface immediately issues LuCI from OpenWRT. This can be seen both in the design, which they did not try to disguise, and in the paths in the URL. In my opinion, this is rather a plus, as it makes it easy to finish the system for yourself. Industrial functions added to firmware: Modbus, VRRP










, MQTT. Through Modbus and MQTT, you can control basic functions, get network status, GPS coordinates, temperature, switch profiles, reboot the router, manage I / O pins.

Monitoring of Internet channels is done through the multiwan package , which is called obsolete in the OpenWRT wiki. Instead, mwan3 is recommended.

WAN setup

Priority setting interface for Internet channels. You can select one primary and two backup. In our case, the modem is selected as the primary, and the wired WAN as the backup.


Prioritization of Internet channels. You can choose the primary and backup

You need to keep in mind that the modem (radio module) in the router is one, and only one SIM-card can be simultaneously in the network. The monitoring system checks the Internet availability in each channel using ping requests to a given host. At the same time, you cannot check which of the two SIM cards has Internet access, you can only switch to another channel: wired or Wi-Fi. The conditions for switching SIM cards are configured separately, and do not depend on general monitoring.


Setting up monitoring of the availability of the Internet channel. You can specify the host to be checked, check intervals and the number of attempts

Since accessibility monitoring (multiwan package from openwrt) perceives the modem as one interface, it cannot initiate SIM switching if the Internet is unavailable on the modem. The multiwan package can only switch the main network interface. The conditions for switching the SIM card are set separately, and only the following can be selected among the conditions: weak signal, traffic limit, roaming, no network, no PPP connection. That is, in some cases, the modem may not switch the SIM card, even if it does not have internet. For example, if the operator has a network failure, while the modem has an IP address, or if the operator blocked access to the network for non-payment, and redirects the request for a stub with a payment interface. In this case, not a single condition for switching the SIM card will be fulfilled, since there is a formal connection with the operator.


Setting the conditions for switching between SIM cards does not work in conjunction with monitoring the availability of the Internet

Crash simulation

We were not able to come up with a way to programmatically simulate the conditions for switching the SIM card to the backup one in this router, so we will check the switching to the backup wired channel. To do this, disconnect the antennas from the router and look at the graph. The switching time is an average of 12-15 seconds, taking into account the restoration of the VPN connection. You can reduce this time by reducing the inspection interval and the number of lost packets for switching triggers, but too aggressive monitoring settings will often cause false positives when the connection is gone for only a couple of seconds.


Switching time to the backup channel is about 15 seconds

Conclusion

A router with many additional features. An interesting work was done with the I / O module: there are analog and digital outputs, a power relay. You can control pins remotely and configure triggers. This allows you to build simple automation with external peripherals on a single router. For example, manage a greenhouse or space heating.

pros

• Many extra features
• OpenWRT Platform
• Support LTE Cat 4
• Three types of WAN: modem, Ethernet, WiFi

Minuses

• You cannot specify different monitoring addresses for each SIM card
• The conditions for switching to a backup SIM card are separate from monitoring Internet accessibility

Robustel r2000

The router is provided by Euromobile (euromobile.ru)



SoC : Atheros 533MHz
RAM: 64MB
Flash memory: 16MB
Modem : Huawei ME909s-120 (LTE Cat 4)
Firmware : closed Linux-based

router with two Ethernet ports and a minimum of functions. Only one of the LAN ports can act as a WAN. The firmware is closed, there is no root access via SSH. The console is limited to its interpreter with several commands (like Ubiquiti devices).

WAN setup

There are several backup and balancing modes. The second option can be useful for quickly switching between channels.

• Cold backup: only the main channel is active, backup channels are disabled
• Warm backup: backup channels are active, only the main one is used (it is impossible to activate two SIMs at the same time)
  
• Load balancing: Both channels are used simultaneously.

In the default settings, for some reason, the return to the main channel after switching to the backup channel is disabled. This is set in the Revert Interval, when set to zero, the channel will never switch back after switching to the backup.


Setting WAN Priorities and Modes

Each SIM card is configured as a separate interface: WWAN1 and WWAN2. For each interface, you can separately configure the address to check the availability of the Internet, and your own intervals. This is the only router in the review where you can specify two hosts at once for verification. This is useful for protecting against false positives when the test host crashes, but there is Internet access.


Each SIM card can be configured to check Internet availability.

The firmware of the router does not imply root access to the system, so without opening the case, I could not figure out which chipset the device was made on. Accordingly, no home-made scripts and packages can be installed on the router, all functionality is strictly limited by the capabilities of the native firmware. It is possible to install several additional packages from the robustel.com/products/app-center/ repository , but this does not compare with the set of packages from the OpenWRT repository.

On the other hand, such a closed system can be useful to protect against uncontrolled changes in the firmware, which then can not be seen in the web interface.

Crash simulation

Since we can specify different hosts to check the availability of each SIM card, to simulate an accident, we will indicate our controlled server as the first SIM card and turn it off at some point. For the purity of the experiment, we will set the test intervals to the same as for the first router in the review.


Switching to the second SIM card takes 15-30 seconds.

On average, the router switches to the second SIM card in 15-30 seconds. I believe that the registration of the second operator’s network takes the most time. Since all of our routers use a single modem, only one SIM-card can be in the network at a time. There are industrial routers with two independent modems that are simultaneously connected to two mobile operators. In this case, channel switching is faster.

Conclusion

Compact and simple router with a minimum of functions, easy to configure. A closed platform and lack of root access can interfere with someone. It can be recommended for cases where ease of setup is more important.

pros

• Separate failover settings for each SIM card
• Support LTE Cat 4
• Warm backup mode, in which the backup channel does not turn off
• Easy setup

Minuses

• Closed platform, no root access
• Truncated SSH Console
• Only one Ethernet port can operate as a WAN

iRZ RU22W

The router is provided by Digital Angel (digitalangel.ru)



SoC : MediaTek MT7620A ver: 2 eco: 6
RAM: 64
Flash memory: 16
Wi-Fi: 802.11n (2.4GHz)
Modem : Huawei MU709s-2
MicroSD
GPS card support
Firmware : on base OpenWRT 15.05

The router does not support LTE, only the 3G standards HSPA + / UMTS (900/2100 MHz) and 2G GSM / GPRS (850/900/1800/1900 MHz) are supported, while the most expensive. There is support for GPS, Wi-Fi and serial interfaces. By default, all Ethernet ports are combined into one switch; if desired, each port can be allocated to a separate VLAN and used as a WAN. Wi-Fi can work in the station (client) mode, and also be WAN. In total, with two SIM-cards this allows you to get 7 WAN-interfaces. Not bad.

Настройка WAN

Interface priorities are selected in the Routes menu. You can choose one of two modes: failover and balancing. The first will switch the channel to the backup interface in case the primary is unavailable, and the second will distribute the traffic equally between the interfaces, at the NAT level.

Availability monitoring and interface failover is done using iRZ's homemade bash scripts. For example, the service / usr / bin / pinger sends pings, switches interfaces, and controls LEDs. Pretty harsh Bashism :)

SIM card setup

For each SIM card, you can set your own connection check host. This is convenient for cases when we want to choose different servers, depending on the service provider. For example, in roaming we use one server, and in the home region another.


You can set different hosts to which communication will be checked for each SIM card

GPS

I did not find the ability to somehow use the GPS module, except through gpsctl from the console. The web interface does not mention it at all. I believe that it is activated through native cloud services for centralized management of routers. The inability to use the GPS API on my own is considered a drawback.

Crash simulation

For each SIM card, we indicated different hosts for checking Internet accessibility, so we can easily simulate an accident. To do this, we will block the host from the first SIM card on the firewall and see how the router will respond to this. For the purity of the experiment, we will set the same test intervals and the number of attempts as the first router.



Of several attempts, the average switching time is 30 seconds, sometimes the second SIM card is registered for a long time, and the switching is delayed for a minute. Not sure if this is due to the modem or the features of the mobile operator.

Conclusion

I liked the opportunity to get as many as 4 wired WANs and separate failover settings for each SIM card. The purpose of the GPS module remained incomprehensible; using it without third-party software did not work.

pros

• Open platform based on OpenWRT
• Отдельные настройки failover для каждой SIM-карты
• Все LAN-порты могут работать как WAN

Минусы

• Нет поддержки IPsec IKEv2
• Непонятно, как пользоваться GPS и выводами GPIO без стороннего софта

---

Заключение

In my opinion, specialized industrial solutions are much more convenient than home-made structures based on a SOHO router. Although in most cases the same OpenWRT firmware is used, all the nuances are already taken into account, the adjustment comes down to setting several parameters in the web interface, and you don’t need to write crutches like watchdog scripts yourself, everything is already out of the box.

Additional functions like GPIO and GPS module allow, in many cases, to manage with one device instead of two, for simple tasks of automation and data collection.

All three manufacturers have their own software for centralized management of routers. In the framework of this article, we did not consider it. It is useful for a large number of installations, and allows you to manage all routers from one control panel. It might be worth exploring this software too.