Selectel IPv4 prefix route leaking / Selectel Blog

On the night of July 15-16, 2017, one of the most remembered events happened on the Selectel network, which led to the deterioration (up to complete inaccessibility) of Selectel's network connection with the foreign segment of the Internet. And it was so remembered that this event was dedicated to the speech of Selectel CTO at the conference of network operators ENOG-14.

But first things first.

Network description

At that time, the Selectel network looked as follows. Selectel is present in two cities of the Russian Federation - St. Petersburg and Moscow, in each city there are two border routers.



Each of the routers announces “into the world” of the Selectel network from the autonomous system AS 49505. Uplinks and peerings are connected to each of the routers.



The largest uplinks of Selectel are TransTelecom, RETN and Raskom.
The largest peer points to which the Selectel network is connected are MSK-IX, DATA-IX, DE-CIX.

At some peer points (in particular, MSK-IX, DATA-IX and others), more-specific prefixes were used to increase the amount of incoming traffic through these exchange points. On DE-CIX, more-specific prefixes were not used.



The stand-alone 49505 system is also used to provide a distributed network of DNS servers. Servers are hosted in several countries at different uplinks; two / 24 subnets using BGP anycast are announced from the servers.

Internet exchange

Internet Exchange (IX), a traffic exchange point is a peer-to-peer network for different network operators, which is one logical switch to which all operators are connected, and a couple of servers that provide general routing information distribution between the participants of the traffic exchange point (route servers, RS )



Responsibility IX, as a rule, manifests itself in two ways - passing traffic from one participant to another through packet switching, and ensuring the correct functioning of route servers at the traffic exchange point - filtering the received routes, if one of the participants announces the wrong routes, the transmission is correct information from one participant to another and, which is especially important for large traffic exchange points, the organization of a blackhole for routes marked with a special community.

Blackhole

Blackhole, translated as “black hole”, is a mechanism that allows telecom operators to protect themselves from DDoS on their networks. This mechanism does not allow maintaining the availability of the attacked resource, but at least it is aimed at maintaining the operability of the network of the operator who has the given resource, or through which this resource is available.

To determine the resources to which traffic should not be transmitted, the operators use the so-called blackhole community. At the end of 2016, this community was included in the RFC 7999 list of well-known communities ( https://tools.ietf.org/html/rfc7999 ).

Usually, operators use the blackhole community very carefully, and as a rule, this community only applies to / 32 prefixes. But the settings at some traffic exchange points and at some operators allow the reception of blackhole routes with a subnet mask other than / 32.

Alexander Ilyin, Technical Director MSK-IX

MSK-IX introduced the BGP Blackholing service as one of the world's first traffic exchange points. The problem described by colleagues from Selectel did not affect our infrastructure, since according to the rules for providing Blackhole on our Route Servers, only routes from / 25 to / 32 are allowed. In accordance with the policy of regional Internet registries RIR (RIPE, RADB, etc.), the Blackhole community can only be installed on networks already announced by the Member. RS accepts network announcements from the Participant only if the network data is also announced by the Participant without an attribute and such errors will be filtered at the input. We also maintain an up-to-date database of participants' contacts and in such cases we quickly block violators.

Incident: First Blood

So, everything happened on the night of July 15-16, 2017. That is, from Saturday to Sunday. Most network engineers, I dare to assume, still rest on the night of Saturday to Sunday in the summer. At 0:30 (Moscow time hereinafter), the technical director received a call from the Selectel technical support engineers on a mobile phone: “Something strange is happening, we don’t know yet, but the symptoms are that customers complain about the inaccessibility of foreign servers from Selectel servers , or servers in Selectel from foreign servers. Plus, the Slack messenger stopped working on the office network. ”

In response, technical support officers were instructed to collect problematic routes from complaining customers. Analysis of the logs and messages of the monitoring system did not reveal obvious problems. Selectel network from the networks of mobile operators was normally accessible, remote access worked without problems. A deeper analysis showed a drop in traffic at one of the uplinks, TransTelecom, without increasing the amount of traffic at other uplinks.



0:30 - a very unsuccessful time to search for anomalies in the graphs of downloading Internet channels. At this time, as a rule, the CNN (busy hour) on Internet networks ends, and the drop in traffic can be caused not only by emergency reasons, but also by normal behavior during the day, and in general it is not very noticeable.

Given the insufficiency of the collected material and the short time it took to diagnose the problem, the conclusion was drawn: “something must have broken in the TTK”. After that, the BGP session with TTK was deactivated. The restructuring of the routes led to the correction of the problem, Slack worked, the connectivity was restored, the customers began to confirm the full functionality of the service. Technical support sent to contact TransTelecom technical support.

The interrupted night's sleep was continued.

Incident: the main part

Around 02:00, the Slack messenger from the office network again stopped working, the situation with the inaccessibility of a significant part of foreign resources again repeated. This was when the channel was disconnected from TransTelecom, that is, either the first assumption was wrong (but why then after disconnecting the BGP session from the TTK, did it work?), Or the problem was further widespread. Technical support engineers on call reported massive customer calls complaining about unavailability of resources. The mass calls and the analysis of the subnets issued to customers showed that the problems concern most of the IP prefixes used in the Selectel network, that is, the problem is global in nature, and not grouped on any one prefix.

Technical support again began to collect the results of the traceroute command, both from the servers inside Selectel and from the servers outside.
A study of the traces showed that the problems are localized in the DE-CIX region. Moreover, there is a significant asymmetry of problematic traffic flows. If the route from Selectel to the end resources goes through DE-CIX, then there may not be a problem. But if the return route passes through DE-CIX, then the problem of inaccessibility clearly manifests itself. Diagnosis of such problems is very often complicated by traffic asymmetry and the use of ECMP in operators. For example, one packet from the operator A network to the Selectel network can be sent via DE-CIX, the second through DATA-IX, and the third through Cloud-IX.



During attempts to localize and diagnose the problem, the session with TransTelecom was turned on.

Further, the engineers began to carefully analyze the information on the looking glass of various operators. The eye caught on outputting information from TransTelecom routers.



Selectel announces the 188.93.16.0/21 prefix to its uplinks and the more-specific 188.93.16.0/22 ​​prefix to some traffic exchange points. In order to prevent uplinks from taking more-specific prefixes “outside”, the same more-specific prefixes from the no-export community are announced to all uplinks. inside uplink networks, these prefixes must be present in the RIB with the best route to the client session. But when the session with TransTelecom was raised, it was discovered that on the Moscow TTK router (with which Selectel had no direct session at that moment) there was a Selectel more-specific prefix leading through some autonomous system 2854.

The autonomous system 2854 is not listed in Selectel uplinks or peerings, i.e. it should not be a transit autonomous system for Selectel prefixes. Where did the TTF come from? Unclear.

We look at looking glass DE-CIX. Here is the route server, all the information meets expectations.



As expected, on the route server, the prefix points directly to the Selectel router. There is no more-specific prefix, Selectel does not announce more-specific on DE-CIX. But something stopped us from moving to the next looking glass, and routing information was requested from Route 2.

Oh!

Firstly, on route server No. 2, the more-specific prefix 188.93.16.0/22 ​​came from somewhere. Where did he come from? AS path 2854 49505. That is, again AS 2854 suddenly announces Selectel prefixes through itself. Secondly, the community was alerted (65535, 666). This is blackhole community! AS 2854 from somewhere takes the more-specific Selectel prefixes, and then sends them to the DE-CIX route server No. 2 with the installed blackhole community!

Compare the output from PC1 and PC2 to DE-CIX:



Yes. On PC1 there is no more-specific, and the route goes immediately to Selectel, as it should. On PC2 there is a more-specific of 2854 with a blackhole community installed.

It is logical that the routers of operators who take routes from DE-CIX router servers “see” the route in Selectel best by a more-specific prefix, traffic to which is filtered by DE-CIX itself.

When it became approximately clear what the problem of routing through DE-CIX was, the chat group of St. Petersburg signalmen had questions “why we also have no connection with some foreign resources”. Diagnostics “on the beaten track” showed that again there is a problem with AS 2854, which announces through itself a lot of routes, including to St. Petersburg operators, using the blackhole community. In broadband operators, complaints from subscribers began to become widespread only in the morning, this is the specifics of broadband access for individuals and the time of the problems (I recall, the night from Saturday to Sunday, summer - many home users sleep at that time in their dachas, on vacation).

Who's guilty

During the diagnosis, the number of the autonomous system is determined, which announces incorrect routes to the PC2 DE-CIX, AS 2854. Who is it? This information is needed in order to be able to quickly contact them and say what they are doing "not the case." AS 2854 is the Russian subsidiary of the global operator Equant (aka Orange Business Services), previously in the Russian Federation this company was called Rosprint.



Naturally, a letter describing the problem was sent to [email protected], technical support engineers began to call these phones. Among the technical contacts, the mobile phone of one of the engineers was even found. A call to a mobile phone to the engineer did not give the desired results. The engineer of Rosprint was on vacation and generally somewhere near Lake Baikal, and not at the computer. Calls to Rosprint’s technical support initially ended with a proposal to write them a letter describing the problem, and on Monday, “networkers will come and can be sorted out.” The mailbox [email protected] during off hours is apparently ignored. It got to the point that the technical support of Rosprint, having heard “I'm from Selectel,” just started to hang up.

What to do

Besides the fact that we did our best to get in touch with the engineers of Rosprint, attempts were made to somehow rectify the situation or reduce problems.
The first attempt to rectify the situation from Selectel itself. It is necessary to disable more-specific, then they will not get to the PC2 DE-CIX, where 100% become the best routes for the participants of DE-CIX. Disconnected. Yes, these routes stopped reaching PC2, but now there were aggregated routes to Selectel networks on PC2. On PC1 there was route 188.93.16.0/21 through AS 49505 (the correct route from Selectel), but on PC2 there was route 188.93.16.0/21 with AS path 2854 49505 and blackhole community.

Partial connectivity has recovered; now on DE-CIX there are no more-specific routes to Selectel networks. Connectivity did not recover from those operators and from those resources that used routes with PC2 as best, despite the AS path.

OK. We cannot reach the engineers of Rosprint. We will knock on other engineers.

An attempt to write a letter describing the problem, and then call to increase the priority of contacting DE-CIX technical support, was unsuccessful.



DE-CIX technical support engineers began to deny their involvement in managing routing information on the route servers. We asked to disconnect Rosprint from the DE-CIX route servers, since they send incorrect information to the route server, and we cannot reach the engineers of Rosprint.

Then we began to write letters in MSK-IX and DATA-IX with a request to disable Rosprint (since they carry out incorrect actions with prefixes with IX). And there was a calculation that, when two large channels fell onto traffic exchange points, Rosprint's technical support would be obliged to report the incident to network engineers, and they would begin to sort it out.

The night has passed

So, in the diagnosis and attempts to contact Rosprint, the night passed. Suddenly at 11:26 (when MSK-IX and DATA-IX were also convinced of the incorrectness of the actions of Rosprint and were ready to disconnect them from the route servers), a message came from Rosprint:



That's all. No sorry, no contacts, nothing ...

The lessons

We learned a few lessons from what happened. Unfortunately, these lessons were learned from my experience in dealing with our problems, but I hope this text will help other operators in the future as well:

• On IX you should always watch two route servers.
• Connectivity to your network must be monitored externally. Using for monitoring connectivity not just ping / HTTP GET, but various API services, including stat.ripe.net.
• The presence of more-specific prefixes, contrary to the widespread opinion among operators about their harmfulness, helped us diagnose the problem.
• When concluding contracts for an Internet traffic strip with uplinks, it is desirable to provide the possibility of working using only one operator, and not just provide the necessary backup in case of failure of one of the border routers.
• Anycast, if there are such segments, it is advisable to separate the autonomous system from the main network.

Why did the PC on DE-CIX accept the routes on the Selectel network from AS 2854?



Because AS 49505 is a member of AS-SET AS-URAL for anycast prefixes. And already AS 2854 can announce prefixes from AS-URAL.

Alexey Kuznetsov, Deputy Head

Modern IT systems and communication networks are the constant interaction of users and various IT services, which in turn depend on / interact with other IT services both directly from the user (authentication via social networks, etc.) and directly between themselves. Violation of the work of IT services, their connection with users or with each other immediately affects users, and often in an unobvious way. The option “does not work at all” is the simplest method of malfunction, while other cases require “troubleshooting” both from technical support staff interacting directly with users / service owners and engineers. Which is easier, to understand in detail or answer “the error is not in our area of ​​responsibility”, “this is the Internet, no one is responsible for anything”? Yes, Errors / problems of some telecom operators may affect the work of users / services located at other telecom operators who do not have any direct relationship with the “source of the problem”, through whose networks the traffic does not even transit. A separate problem is that telecom operators have personnel of different skill levels, not to mention employees working at night and on weekends, different policies for escalating the problem, because for the telecom operator, the users / services connected to it are primary, and the decision to escalate a “someone else's” problem by their system administrator, and maybe first by the Management, at night already depends on the specific person on duty, on the reaction of his colleagues to previous escalations, etc. Often the only way to quickly escalate the problem is direct contacts of managers and administrators of telecom operators, the ability to go through a chain of contacts and reach the right employee. It’s not accepted to go to a foreign monastery with its own charter, so either national organizations or already recognized international organizations, such as RIPE for the Europe-Russia region, can take the initiative on themselves by setting the “rules of the game” common to all their members.