Mikrotik and OSPF. What we had to face and how we overcame it

Good day. Today I would like to talk about what did not allow us to live, it blew our brains for a long time - Mikrotik and OSPF.
Mikrotik itself is a good piece of hardware, with a low cost, a lot of features, but unfortunately, not without flaws.
We have the following network diagram:


ROU1 - Cisco c3845
ROU2 - Cisco c3845
MT1 - Mikrotik 1100Ahx2
MT2 - Mikrotik 1100Ahx2
c3550 - Cisco c3550
GW68 - MikroTik RB751U-2HnD
GW69 - MikroTik RB751U-2HnD MT

to RO1 and MT1 to RO1 MT1 and RO1 MT1 ROU1 (I will skip IPSec), the OSPF process for backbone has been started. Suggested subnet 10.0.0.0/19:


On MT1 and MT2, the tunnel addresses are 172.20.64.2 and 172.20.64.6, respectively. Area backbone is already there by default.


But we must still have a connection between MT1 and MT2 directly.
To do this, we use the new area IRK. We do not only have a Cisco c3550 switch, it has a root device, all vlans are installed on it, providers are forwarded through it, vrf is created on it, etc. (If it is interesting, then I will describe the entire network organization)
Create a c3550 area IRK and two vlan to connect to MT1 and MT2. Area IRK is the main area of ​​RU - a regional node in which all routers connected to RU operate.

Now we will accept these vlan on MT1 and MT2, and we will create a new ospf instance and area, add a link between MT1 and MT2 and attach the address.

It seems that I didn’t forget anything that we got:

1. We have two tunnels to the central routers, one from each Mikrotik.
2. Cost 10 and 40 respectively. That is, with MT1, the main router (provider) is alive, we will get the route from backbone 10.0.0.0/19 through the tunnel to ROU1, while dead through the tunnel to ROU2
3. We have a direct link between MT1 and MT2, so as not to drive traffic always through c3550.

Set up, set. Everything works:

# MT1
ADo dst-address = 10.0.0.0 / 19 gateway = 172.20.64.1 gateway-status = 172.20.64.1 reachable via ipip_yyy_vl03_rou1 distance = 110 scope = 20 target-scope = 10 ospf-metric = 40 ospf-type = external-type-1

# MT2
ADo dst-address = 10.0.0.0 / 19 gateway = 172.20.64.105 gateway-status = 172.20.64.105 reachable via ether2 distance = 110 scope = 20 target-scope = 10 ospf-metric = 50 ospf-type = external-type-1

On MT2, the route is up to 10k via ether2, as in principle it should be.

Problem No. 1:

Once, on MT1, a provider dies. The subnet 10.0.0.0/19 becomes accessible through the tunnel to ROU2, everything is as it should be:

# MT2
ADo dst-address = 10.0.0.0 / 19 gateway = 172.20.64.5 gateway-status = 172.20.64.5 reachable via ipip_yyy_vl03_rou2 distance = 110 scope = 20 target-scope = 10 ospf-metric = 70 ospf-type = external-type-1

BUT! As soon as the provider comes to life on MT1, we see the following:

# MT1
ADo dst-address = 10.0.0.0 / 19 gateway = 172.20.64.1 gateway-status = 172.20.64.1 reachable via ipip_yyy_vl03_rou1 distance = 110 scope = 20 target-scope = 10 ospf-metric = 40 ospf-type = external-type-1

# MT2
ADo dst-address = 10.0.0.0 / 19 gateway = 172.20.64.5 gateway-status = 172.20.64.5 reachable via ipip_yyy_vl03_rou2 distance = 110 scope = 20 target-scope = 10 ospf-metric = 70 ospf-type = external-type -1

We see that each router looks into its own tunnel. Which does not suit us at all. Reboot MT2. He begins to receive a route via ether2 from MT1. The mess.

As a temporary solution, we added all area from regional nodes to ROU1 and ROU2. Everything worked very nicely until one day our tsiska said: too many routing processes

And again we returned to the problem with the backbone. Long battles with support do not lead to anything, but at one point we get the answer that Mikrotik, when redistributing routes between Erias (in this case, the route from the backbone is redistributed via IRK),
starts to incorrectly install the routes. And we get the picture that each router looks into its own tunnel.
Then the solution comes by itself:
Add IP to ether2, add a new subnet to the backbone.

# MT1
/ ip address add address = 172.20.64.121 / 30 comment = "MT-MT BB" interface = ether2
/ routing ospf network add area = backbone network = 172.20.64.120 / 30

# We prohibit redistributing the route via IRK
/ routing filter add action = discard chain = ospf-in prefix = 10.0.0.0 / 19

# MT2
/ ip address add address = 172.20.64.122 / 30 comment = "MT-MT BB" interface = ether2
/ routing ospf network add area = backbone network = 172.20.64.120 / 30

# Forbid re-distributing the route via IRK
/ routing filter add action = discard chain = ospf-in prefix = 10.0.0.0 / 19

And we get that route 10.0.0.0/19 is not redistributed through someone else’s eria, but is accepted within one. Fuf, one problem was won.

Now to consider the second problem, we will give the GW config, I will omit the config for MT1 and MT2, there is nothing complicated there.

In the figure we have GW68 and GW69, I will give the settings for only one and in abbreviation:

On Mikrotik 751:
ether2 - backup
ether3 provider - main
ether5 provider - LAN
The question will be, why is not ether2 the main provider. First we made ether1 - the main provider, but it turned out that the port was half-software and there was absolutely no IPSec performance, we just transferred it to ether3 so as not to edit the config much.


How it works:

GW68 installs two tunnels to MT1 and to MT2, each through its own provider. There is no default on Mikrotik, the addresses of MT1 and MT2 are registered in statics.
On the tunnel to MT1 costs cost 10, on the tunnel to MT2 cost 40. As soon as the tunnel comes to life, we get a default and all routes from MT1 through OSPF. All traffic is wrapped on MT1. As soon as the main provider falls off,
after the OSPF timer expires, routes through the tunnel to MT2 become active.

Problem 2:

With the live main channel, we see in the GW68 log 20:12:26

route, ospf, info Database Description packet has different master status flag
20:12:26 route, ospf, info new master flag = false

At this point, our traffic by itself begins to run through the tunnel to MT2. The tunnel to MT1 is live, and traffic runs through the tunnel to MT2. And since stores usually have megabyte reserves,
we get money exhausted for no reason.
To return everything to the tunnel to MT1, it was necessary to turn off and on again the tunnel to MT1 on MT1. Moreover, this problem arose only in some cities, and not on all routers.
At the microtica forum, in those. nobody helped us with support. Once my colleague, trying to raise the MT1-Juniper SRX-650 tunnel, was armed with a bunch of alcohol, a sniffer and a debugger :)
And while debugging IPSec, I came across OSPF Mikrotik authorization errors. We disabled authorization on OSPF and voila interfaces, the problem disappeared by itself.
How authorization is going on, where is the failure, now neither he nor I remember. But the issue is resolved. Why somewhere it worked perfectly and with authorization, I can’t say.

With Mikrotik one can only guess on the coffee grounds, since each support response is a new question, each new firmware presents a new problem :)