As we wrote URL filtering to Rostelecom

Recently on Habré there was an article about implementation of URL filtering by "Rostelecom" (RTK). It so happened that we suggested they solve this problem a year and a half ago and a year ago we made a decision that passed all the tests and which Rostelecom was ready to include. By that time, our shop fell out of favor, well, yes this is not a matter of technology. Therefore, all of the following is the details of our proposed solution. What exactly will implement, God knows.

DPI and IP blocking


It’s probably worth starting with DPI. There are some of them on the network, but putting them on all the traffic would cost RTK several tens of millions of dollars and would constantly require new investments in connection with the increase in consumption by the bored population. Neither the previous nor the current leaders have decided on this (perhaps only for now), because the introduction of DPI does not promise any new services with tangible volumes of income.

Since the RTK had to block different resources before (the courts are judged without regard to the laws), it did so by IP addresses. Accordingly, the simplest filtering at the borders solved the problem. The problem with the registry of prohibited resources began to be solved in a similar way: a specially trained person unloaded the registry, there are both URL and IP and then added entries to the list, and the script of the rules is access list.

Filtration management software


We automated communication with the registry, and here, as everyone understands, there is no magic (it doesn’t really exist any further). We recorded changes to the registry and the implementation of censorship on the network. We added small tricks associated with the distributed responsibility structure on the RTK network, excluding some subscribers (in particular, out-of-band operators) from the filter. Well, the courts have not been canceled, so it is possible to add any resource from a court decision. In addition, the management software configures traffic redirection to the filtering nodes, prepares reports on user searches, monitors the filtering nodes and logs all activities.



Traffic rotation and filtering


Traffic sent by subscribers to IP addresses corresponding to blocked URLs on PE turned into a special VPN, where defult looks at a couple of routers in the center. Using SCU / DCU (similar functionality is also found on pussies) at the border, traffic from subscribers that are not subject to filtering was excluded by source addresses. The management software generated / 32 routes by registry IP addresses, config rules on the central router, and the changes took effect. I recall that they refused to send BGP updates, because this is not good.

On two central routers we configured the copying of the passing traffic already taking into account the ports. Accordingly, for the entire RTK, several megabits were obtained from the force. You could copy either to an external server or to MS-DPC. Both there and there the principle of further processing was the same. Filtering software catches a package, if it contains get with a URL from the list, then the helmet is reset to the server side, and a redirect to the site is directed to the browser with the story about how important moral principles are for modern Russian society.

The minimum response time from servers that entered the registry a year ago exceeded 100 ms, and the software analyzing the URL returned a response of 5-6 ms at a load of 3 Gb / s. Therefore, they decided that making a proxy, ensuring its reliability and so on, in the same spirit, makes no sense. If the filtering software crashes, everything will work, except filtering is clear. True, at the request of the RTK, they designed it all the same for a pair of filtering servers.

Now it’s clear that it was possible and a little easier to do, use a small DPI in the center instead of your filtering software. But so we have done another operator. It may turn out that the selected vendors will combine their registry software with a regular DPI through which only the selected traffic will go. At that moment, we wanted to understand if there was at least some kind of task that should be implemented on the service modules in the router, and not outside. The fact that Juniper closed access to modules for external developers gives a clear answer to this question, but then we were still tormented by doubts.

As for the cost, out of the million in question, more than half were servers, so there was no big profit planned for the software there.

Also popular now: