Projection of corporate conflict on network connectivity

A corporate conflict arose on 06/10/2019 due to an increase in the cost of delivering SMS to users of the Vimpelcom network by Mail.RU Group. As a response, Mail.RU Group ceased to “service” direct Russian IP channels to the Vimpelcom network.

Below is a brief analysis of the situation from the perspective of a network engineer.

Update : 06/14/2019 18:45 - emphasis on Russian routes to the VimpelCom network, adjusted conclusions, added an explanation by Sergey Kubasov (CIO Vkontakte) .
Update : 06/14/2019 19:48 - a description has been added of how to restrict the distribution of routes along the “short” Russian route via Rostelecom, MTS, RETN.NET ..
Update : 06/15/2019 09:39 - Mail.ru blocked its Looking Glass.
Update: 06/18/2019 10:10 - Depyring is over. VimpelCom and Mail.RU Group raised direct BGP sessions.


Introductory:
Vimpelcom has an autonomous system AS3216, all the rest (8402 - home Internet, 16345 - mobile Internet) are located for 3216.

Mail.RU Group - autonomous systems AS47541, AS47542 and AS47764. The main content generator is 47542, the so-called CDN VKONTAKTE (films, music). All autonomous systems are independent of each other (from the position of an external autonomous system).

First, let's look at the situation from the Vimpelcom network. Looking Glass Vimpelcom will help us for this .

We look at the first autonomous system - AS47541.

 2914 47541
    79.104.32.251 (metric 10500) (79.104.32.251)
      Origin IGP, metric 30, localpref 87, valid, internal, best, group-best, import-candidate, import suspect
      Received Path ID 0, Local Path ID 1, version 2865394342
      Community: 2914:410 2914:1214 2914:2213 2914:3200 3216:3000 3216:3103 47541:1 47541:40000 47541:50078

AS-PATH contains two autonomous systems - 2914 (NTT) and 47541 (VKONTAKTE-SPB-AS).
The localpref metric is set to 87, which, as described in the RIPE DB for the AS3216 object, corresponds to international peering.

...
remarks: International peer's routes get local preference in the
remarks: range of 81-89.
...

The 3216: 3000 and 3216: 3103 community (source - RIPE DB for AS3216 ) also confirms the same information :

...
remarks: 3216: 3000 Receieved from an international uplink or peer, specifically:
...
remarks: 3216: 3103 AMS-IX
...

That is, Vimpelcom sees the route to VKontakte through the European junction.

Let's see another VKontakte autonomy - 47542 (VKONTAKTE-MSK-CDN-AS).


What about Mail.ru itself? Let's take a look at AS47764 (mailru-as).

 3356 47764
    194.67.0.215 (metric 10501) (194.67.0.215)
      Origin IGP, metric 0, localpref 77, valid, internal, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 2867605667
      Community: 3216:3000 3216:3007 3356:2 3356:22 3356:100 3356:123 3356:519 3356:2094 47764:1 47764:40000 47764:50077 

VimpelCom sees Mail.ru via as3356 (uplink Level3, Tier1 operator). This information is confirmed by localpref 77:

...
remarks: Uplink's routes get local preference in the range of 71-79.
remarks: Last Update: February 2012 ...

and community (3216: 3000 and 3216: 3007):

...
remarks: 3216: 3000 Receieved from an international uplink or peer, specifically:
...
remarks: 3216: 3007 Level 3 Communications
...

From the information received, it can be seen that traffic from the Vimpelcom network on VKontakte and MailRu is distributed through European connections according to the routes received via the BGP protocol. There are no alternative routes through Russian connections at Looking Glass. No measures have been found to artificially redirect traffic through knowingly worse routes.

And how does the Mail.ru Group see the VimpelCom network?
We will use Looking Glass from Mail . Update : as of the morning of June 15, 2019 Looking Glass does not work, the 500th error is

displayed on requests: From routers from AS47764 (mailru-as):

  Path #6: Received by speaker 0
  Advertised to peers (in unique update groups):
    188.93.60.188   
  1299 1299 1273 3216 3216
    217.20.147.250 (metric 100) from 217.20.147.250 (217.20.147.253)
      Origin IGP, metric 500, localpref 200, valid, internal, best, group-best
      Received Path ID 0, Local Path ID 0, version 1172721494
      Community: 1299:20000 47764:701 47764:41100 47764:41108 47764:50077

AS-PATH contains AS1299 (Telia, Tier1 operator, uplink Mail.RU) and as1273 (Vodafon, Tier1 operator, uplink Vimpelcom).

LocalPreference 200 is the standard for external Mail.ru interfaces (https://net.mail.ru/bgp.html), and the MED 500 corresponds to everything received not from a direct interface, not from IX-s, not from peer-to-peer ones.

But why are there no local routes through Russian telecom operators ???
They are, but their priority is “non-standard”!
Here is the route through Rostelecom (as12389):

  Path #1: Received by speaker 0
  Not advertised to any peer
  12389 3216
    46.61.178.149 from 46.61.178.149 (213.59.207.79)
      Origin IGP, metric 500, localpref 199, valid, external
      Received Path ID 0, Local Path ID 0, version 0
      Community: 3216:2001 3216:2999 3216:4100 12389:5 12389:6 12389:1100 12389:1105 12389:1277 47764:701 47764:41100 47764:41104 47764:50077
      Origin-AS validity: not-found

Here is through the Megaphone (as31133):

  Path #2: Received by speaker 0
  Not advertised to any peer
  31133 3216
    78.25.77.41 from 78.25.77.41 (10.222.253.97)
      Origin IGP, metric 500, localpref 199, valid, external
      Received Path ID 0, Local Path ID 0, version 0
      Community: 3216:2001 3216:2999 3216:4100 31133:300 31133:46170 47764:701 47764:41100 47764:41105 47764:50077
      Origin-AS validity: not-found

Here is through RET.NO:

  Path #3: Received by speaker 0
  Not advertised to any peer
  9002 9002 3216
    87.245.253.24 from 87.245.253.24 (87.245.225.1)
      Origin IGP, metric 500, localpref 199, valid, external
      Received Path ID 0, Local Path ID 0, version 0
      Community: 9002:9002 9002:64667 47764:701 47764:41100 47764:41101 47764:50077
      Origin-AS validity: not-found

And even through MTS!

  Path #5: Received by speaker 0
  Not advertised to any peer
  8359 3216
    212.188.61.105 from 212.188.61.105 (195.34.52.77)
      Origin IGP, metric 500, localpref 199, valid, external
      Received Path ID 0, Local Path ID 0, version 0
      Community: 8359:200 8359:609 8359:5012 47764:701 47764:41100 47764:41103 47764:50077
      Origin-AS validity: not-found

The localpref metric for these Russian routes is underestimated, that is, the routes are degraded compared to foreign ones!

In addition, Mail.Ru Group has prohibitions on distributing its prefixes to VimpelCom through Russian operators!

RET.NET (http://lg.retn.net/):
The announcements from Mail.RU Group include a 3216 community: 65535.

Vimpelcom does not accept routes marked by such a community on its network. Excerpt from RIPE DB by AS3216:

...
remarks: Internal comminuties are assigned only internally.
remarks:
and are always deleted from incoming updates at the border
remarks: routers.... remarks: They are in range 3216: 0000-3216: 4999 and 3216: 6000-3216: 65535
...

Towards Rostelecom (http://lg.ip.rt.ru) Mail.RU Group gives similar routes with the community 12389: 8350.

According to the entries in the RIPE DB for the as12389 object, this community means “do not announce on the as3216 network”:

...
remarks: | 12389: 835y When advertising to GoldenTelecom (AS3216) |
...
remarks: | ... y = 0 - do not advertise |
...

Similarly, towards MTS (http://lg.mtu.ru):
Community 8359: 2120 means:

...
remarks: 8359: 212x when announcing to Sovam (Beeline)
...
remarks: x = 0 - do not announce
...

You can’t look at the announcements of Mail.RU Group towards Megafon - the latter does not have Looking Glass.

Let's see AS47541 (VKONTAKTE-SPB-AS).


AS-PATH points to AS174 - Cogent (uplink Mail.RU, Tier1), then AS6762 - Telecom Italia (uplink Vimpelcom). Local Preference everywhere 150, but so on all external joints, regardless of the written policies .

Let's see the AS47542 (VKONTAKTE-MSK-CDN-AS).

 Router: mx960-m9-0
Command: op lg-sh-bgp prefix 81.211.56.202
0.0.0.0/0                LP:151       MED:        NH:87.240.191.222  AS path: 47541 I
Communities: 
Accepted Best
0.0.0.0/0                LP:151       MED:        NH:95.142.204.251  AS path: 47541 I
Communities: 
Accepted
Inactive-reason: Interior > Exterior > Exterior via Interior

And from the second router:

 Router: mx960-m9-1
Command: op lg-sh-bgp prefix 81.211.56.202
0.0.0.0/0                LP:151       MED:        NH:87.240.191.224  AS path: 47541 I
Communities: 
Accepted Best
0.0.0.0/0                LP:151       MED:        NH:95.142.204.250  AS path: 47541 I
Communities: 
Accepted
Inactive-reason: Interior > Exterior > Exterior via Interior

Only default routes (0.0.0.0/0). This situation was explained by an employee of Mail.RU Group greediness , for which he thanks. In short: the Moscow segment of the VKontakte network is caching (rather than generating), whose task is to optimize the download speed of popular, demanded content. Caring for users, yes.

If there is no route to any network, then this network is not served by caching servers. So the optimization of the download speed does not work, and users suffer. But here it is necessary to emphasize - users are not only their own, but also VimpelCom.

Conclusions:

1. From Vimpelcom, traffic towards Mail.RU Group is distributed naturally. No artificial redirects through Local Preference manipulations detected
2. On the part of Mail.RU Group, there are manipulations with Vimpelcom prefixes . On existing routes towards the Vimpelcom network through Russian operators, priorities are underestimated compared to routes through foreign Tier1 operators.
3. Managers of the BGP community have been added to the routes transferred to Russian operators (MTS, Rostelecom, RETN.NET) from the Mail.RU Group to restrict their distribution to Vimpelcom

Why does Mail.RU Group prioritize routes through Europe? Why Mail.RU Group prohibits a short domestic connection with Vimpelcom?

Is it cheaper for them? Drive traffic through foreign channels and pay currency to tirvans?
Or is there a desire to drive traffic away so that it wasn’t so convenient to take it, huh?
This is not known to the network engineer ...

Depyring results: 06/17/2019
BGP sessions between Vimpelcom and Mail.RU Group raised:
AS47764 (Mail.RU):


AS47541 (Vk):


AS47542 (Vk CDN): It


is not known who initiated the raising of the sessions.