Router Management: The Small Joys of Tech Support

In Europe, the USA and many other countries, providers often use managed subscriber devices to which new firmware can be sent via a special interface, or on which settings can be viewed remotely.
I’ll tell you about our experience and how relieved the employees of the second support line experienced . To begin with, we have ended the dialogue in the spirit: “And do you have the upper green light on?”
Weekdays support
Previously, a support call began with questions about whether the lights on the device were on, then with attempts to understand what kind of operating system (the level of “what color is your Internet”) and no less interesting attempts to explain to the user where and how to see the status of the device and then dictate them.
Then, according to these painfully "squeezed" data, a differential diagnosis was made (usually not the most accurate), the problem began to be solved by voice. The “phone sex” session continued: now it was necessary for the user to take the right actions in the right order. If you are lucky, the diagnosis was correct, and everything will work. If not, for example, in the case of physical damage to the router, we tortured the user in vain for 20 minutes, spent both his time and ours, and there’s nothing sensible in the output. “Perhaps you should contact the service center.”
Now everything is a little different. If the device supports TR-069 (protocol for managing subscriber equipment) and has firmware ready to respond to our authorized server, it is enough for the subscriber to simply say that he has a problem. Further, as a rule, the support employee simply calls the router and looks at its status and configuration.
For example, a very common case is when the user has “brakes” due to the noisy 2.4 GHz band. In Moscow, easily from the same channel can be from 3 to 5 neighbors - in the courtyard in each window there will be a router. Often, just changing the channel is enough - and the subscriber is happy again. Imagine now what it would look like without the ability to control the device: go to the settings, wi-fi section, then select a channel, then wait for a miracle.
The second practical case isremote device diagnostics . If the piece of iron is killed or exotically buggy, then you can make a quick conclusion about it, look at the logs for a more detailed analysis and already say for sure that torturing it further is useless.
Often they call about the fact that no-name devices on Android do not connect. The problem is known, but, again, the user is what and how cares last. He wants to lie on the couch and watch a movie.
Since our server is, in fact, an additional authorization channel under the administrator, traditional everyday problems are also solved. You won’t believe it, but so many users simply cannot press the reset button on the router. It also happens that the subscriber changed the admin password to 1234567, and the neighbor guessed it - the support will install a new one. Forgot your home network password - change. Now, for a number of reasons, we are opening Wi-Fi and talking about how to set a new password. Naturally, some users have already decided everything at this point, and they think little about security. Given the increasing number of calls for forgotten passwords, we will instead of opening open a new cryptographic in the future, sending it by SMS so that only the end user sees it.
Firmware update
Previously, it was necessary to stick into the device physically in order to update its firmware. Either download the update and give it through the home network (by hand or with the help of auxiliary software on the subscriber's computer). A centralized firmware update turned into a story lasting at least a couple of years ... not always ending in success.
Today, ACS - automatic configuration servers - have the ability to "flush" the firmware to the end device and update it at the right time without the participation of the subscriber. PROFIT!
As soon as the manufacturer releases a new version or we are finalizing some settings, the software automatically passes through the entire database of routers and puts them in the queue for updating. During the period of the least statistical activity (for the city as a whole, as a rule - around 4:00 in the morning), the firmware is uploaded to the final device. Even if the subscriber is online at this moment, he only gets a break for a few seconds when the device restarts. All functionality configured by the client is not affected.
Previously, changing the MTU was a fairly non-trivial task.. Now all this is decided by one administrator - and in a few days the entire network uses the new parameters. This approach is very useful, because, for example, soon we are going to transfer part of the network to IPoE - and the opportunity not to go around every house is very pleasing. Most customers will not notice anything at all (well, unless those who play online games will enjoy a slightly reduced ping).
This functionality is available for branded SmartBox routers. All subsequent branded routers will also support remote control.
In Russia, Rostelecom is going to use such functionality, according to my information, MGTS has long been using part of the functionality in view of the historical ADSL heritage.
Here is a list of cities where we already use this solution:
Arkhangelsk
Astrakhan
Bryansk
Vladivostok
Volgograd
Voronezh
Yekaterinburg
Ivanovo
Irkutsk
Kazan
Kaliningrad
Kaluga
Kemerovo
Kostroma
Krasnodar
Krasnoyarsk
Kursk
Lipetsk
Moscow
Nizhny Novgorod
Omsk
Eagle
Perm
Pyatigorsk
Rostov-on-Don
Samara
St. Petersburg
Saratov
Smolensk
Sochi
Stavropol
Tver
Tolyatti
Tomsk
Tula
Tyumen
Ulyanovsk
Ufa
Khabarovsk
Cherepovets
Yaroslavl
Kazan and Yuzhno-Sakhalinsk - in the process of implementation.
Astrakhan
Bryansk
Vladivostok
Volgograd
Voronezh
Yekaterinburg
Ivanovo
Irkutsk
Kazan
Kaliningrad
Kaluga
Kemerovo
Kostroma
Krasnodar
Krasnoyarsk
Kursk
Lipetsk
Moscow
Nizhny Novgorod
Omsk
Eagle
Perm
Pyatigorsk
Rostov-on-Don
Samara
St. Petersburg
Saratov
Smolensk
Sochi
Stavropol
Tver
Tolyatti
Tomsk
Tula
Tyumen
Ulyanovsk
Ufa
Khabarovsk
Cherepovets
Yaroslavl
Kazan and Yuzhno-Sakhalinsk - in the process of implementation.
Solution Architecture
We chose a solution from Friendly Technologies. The core of the platform is an automatic configuration server. It communicates with the subscriber equipment and configures it using the ACS API (in particular, it exposes an interface for external applications - Initialization Portal, Online Support Software, Customer Relationship Management, and so on). The management console enables the administrator to manage the activities of the automatic configuration server, add new types of subscriber equipment, monitor and diagnose subscriber equipment using the TR-069 protocol, send warnings about possible errors and other signals, reports, etc. Nearby are auxiliary services of authorization, management, data transfer, database, authorization services and so on.
All this (devices and central servers) are located in a separate virtual network, the main equipment of which is installed at the MCC (main computer center) in Moscow.
The device manufacturer for the first time rolls firmware at the factory, which allows the device to enter this network and connect to the management server. Then the device each time receives a new firmware, including modifying the connection parameters.
This means that, theoretically, you can upgrade our device with open-source firmware without a connection to our server (and it does not allow you to configure it remotely), or alternatively, upgrade your device with firmware compatible with our subnet. But while there are no open source options either.
Another important advantage of such virtualization and deployment in the MCC is its tight connection with mobile telephony services. In particular, the personal account of the subscriber. Soon it will be possible to log into your personal account from a cell phone, configure the router from there (more precisely, transfer the settings to the configuration server, which he, in turn, will transmit to the router), update the password directly there or block access to the child. For many subscribers, this is many times more convenient and understandable than the standard wired web-based interfaces for administering routers or controlling a device using special software on a computer.