Do you know static routing well?

    A static route is the first thing anyone faces when learning the concept of routing IP packets. It is believed that this is the simplest topic of all, everything is simple and obvious in it. I’ll try to show that even such a primitive technology can contain many nuances.

    Reservation. When writing a topic, I assume that the reader is familiar with the concept of routing, knows how to make static routes, and does not consider the word "ARP" abusive. However, even experienced signalmen will surely find something new here.
    All examples have been tested on the iOS 15.2M line. The behavior of other operating systems may vary.
    And there will be no dynamic routing here.


    We work with the following topology:


    How does a static route appear?


    To begin with, we will execute the command that everyone knows, and see with a debug what will happen:
    R00(config)#ip route 3.1.1.0 255.255.255.0 10.0.0.3

    IP-ST(default): updating same distance on 3.1.1.0/24
    IP-ST(default):  3.1.1.0/24 [1], 10.0.0.3 Path = 8, no change, not active state
    IP-ST(default):  3.1.1.0/24 [1], 10.0.0.3 Path = 2 3 7
    RT: updating static 3.1.1.0/24 (0x0):
        via 10.0.0.3
    RT: add 3.1.1.0/24 via 10.0.0.3, static metric [1/0], add succeed, active state
    IP ARP: creating incomplete entry for IP address: 10.0.0.3 interface GigabitEthernet0/1
    IP ARP: sent req src 10.0.0.1 30e4.db16.7791,
                     dst 10.0.0.3 0000.0000.0000 GigabitEthernet0/1
    IP ARP: rcvd rep src 10.0.0.3 0019.aad6.ae10, dst 10.0.0.1 GigabitEthernet0/1

    IOS created a route, and immediately sent an arp request in search of the next hop, which we have is 10.0.0.3. And immediately the question: how did the router find out that the request should be sent to the Gi0 / 1 interface? Surely someone will say “from the list of local interfaces” and make a terrible mistake. Routing doesn't work like that. In fact, iOS made a recursive query on the routing table to find out how to get to the next hop:
    R00#show ip route 10.0.0.3
    Routing entry for 10.0.0.0/24
      Known via "connected", distance 0, metric 0 (connected, via interface)
      Routing Descriptor Blocks:
      * directly connected, via GigabitEthernet0/1
          Route metric is 0, traffic share count is 1

    And here it is, our Gi0 / 1. IOS learns that with recursive requests to the RIB, you need to end as soon as it finds a route with the “directly connected” flag. But what if, in response to the initial request to 10.0.0.3, the route is not connected at all, but an intermediate one, referring to another next hop? We will return to this a bit later, but for now let's recall what CEF is.

    Approximately all beginner-centric documentation says that each packet moves in accordance with the routing table. In fact, on all more or less modern platforms, this is no longer the case, because the routing table (hereinafter - RIB) is not at all optimized for fast data transfer. The scale of the disaster allows thistable (although process switching has many drawbacks besides non-optimal queries - for example, constantly switching the CPU sheduler between contexts, which is very expensive). CEF is a serious optimization. In the current implementation, he builds two tables - FIB (Forwarding Information Base, packet transfer table, based on it is a connected graph with the terrible name 256-way mtrie) and adjacency table (neighborhood table). The first of them is built on the basis of the routing table and in one pass allows you to get all the necessary information. It is built in advance, even before the first package corresponding to it appears.

    Back to our static route. Here is the entry in the routing table:
    R00#show ip route 3.1.1.0
    Routing entry for 3.1.1.0/24
      Known via "static", distance 1, metric 0
      Routing Descriptor Blocks:
      * 10.0.0.3
          Route metric is 0, traffic share count is 1

    Where to send the package? Where to look for 10.0.0.3? Unclear. We need to query the routing table again, this time about 10.0.0.3, and, if necessary, perform a few more iterations until we find out the connected interface. And about this way we actually reduce the performance of the router several times.

    And here is what CEF says:
    R00#show ip cef 3.1.1.0 detail
    3.1.1.0/24, epoch 0
      recursive via 10.0.0.3
        attached to GigabitEthernet0/1

    Simple and concise. There is an interface, there is next hop, to which you need to send a packet. What did they say about the adjacency table?
    R00#show adjacency 10.0.0.3 detail
    Protocol Interface                 Address
    IP       GigabitEthernet0/1        10.0.0.3(10)
                                       0 packets, 0 bytes
                                       epoch 0
                                       sourced in sev-epoch 2
                                       Encap length 14
                                       0019AAD6AE1030E4DB1677910800
                                       ARP

    Let's pay attention to some long sequence in the penultimate line. Something it reminds ... We watch mac 10.0.0.3:
    R00#show arp | in 10.0.0.3
    Internet  10.0.0.3                1   0019.aad6.ae10  ARPA   GigabitEthernet0/1

    We look at our mac address on gi0 / 1:
    R00#show int gi0/1
    GigabitEthernet0/1 is up, line protocol is up
      Hardware is CN Gigabit Ethernet, address is 30e4.db16.7791 (bia 30e4.db16.7791)

    Yeah. That scary line is just two poppies that need to be substituted into the Ethernet header at the stage of encapsulation, and ethertype 0x0800, i.e. commonplace IPv4. And in the two CEF tables there is absolutely all the information that is needed to successfully send a packet further down the chain.

    If someone has a question, why should the piece of iron keep two tables at once instead of one, then I will give the obvious answer: usually a router has few interfaces (and neighbors at the same time) and many routes. What is the point of duplicating thousands of times the same poppies in the FIB? There is never a lot of memory, especially on hardware platforms, be it newfangled ASRs or even L3 switches of the Catalyst line. All of them use CEF when transmitting packets.

    And by the way, let's return for a moment to the original debug. Disable CEF with the no ip cef command (never do this) and compare the result:
    IP-ST(default): updating same distance on 3.1.1.0/24
    IP-ST(default):  3.1.1.0/24 [1], 10.0.0.100 Path = 8, no change, not active state
    IP-ST(default):  3.1.1.0/24 [1], 10.0.0.100 Path = 2 3 7
    RT: updating static 3.1.1.0/24 (0x0):
        via 10.0.0.100
    RT: add 3.1.1.0/24 via 10.0.0.100, static metric [1/0], add succeed, active state

    Route added. Arp request was not. And rightly so - why did the RIB surrender to the mac address? If you let ping go, for example, 3.1.1.1, then most likely it will be like this:
    R00#ping 3.1.1.1
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 3.1.1.1, timeout is 2 seconds:
    .!!!!
    Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/4 ms

    The first packet is discarded, and the router sends an arp request to find out the mac address 10.0.0.3 if it was not previously known. CEF always knows in advance the mac address of the next hop.

    We figured it out. Now back to the question of what happens if the next hop of the static route is not at all on the directly connected interface. We proceed simply:
    R00(config)#ip route 10.0.0.3 255.255.255.255 100.100.100.101

    where Gi0 / 2 has the address 100.100.100.100/24.
    R00#show ip cef 3.1.1.0 detail
    3.1.1.0/24, epoch 0
      recursive via 10.0.0.3
        recursive via 100.100.100.101
          recursive via 100.100.100.0/24
            attached to GigabitEthernet0/2

    How bad it all is ... But what if we have a route to the whole super network?
    R00(config)#no ip route 10.0.0.3 255.255.255.255 100.100.100.101
    R00(config)#ip route 10.0.0.0 255.0.0.0 100.100.100.101
    R00#show ip cef 3.1.1.0 detail
    3.1.1.0/24, epoch 0
      recursive via 10.0.0.3
        attached to GigabitEthernet0/1

    Now our routing table looks like this:
    R00#show ip route
          10.0.0.0/8 is variably subnetted, 2 subnets, 3 masks
    S        10.0.0.0/8 [1/0] via 100.100.100.101
    C        10.0.0.0/24 is directly connected, GigabitEthernet0/1
    L        10.0.0.1/32 is directly connected, GigabitEthernet0/1
          100.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
    C        100.100.100.0/24 is directly connected, GigabitEthernet0/2
    L        100.100.100.100/32 is directly connected, GigabitEthernet0/2

    Seems alright. The new route at 100.100.100.101 does not apply to 10.0.0.3, since its mask / 8 is much shorter than / 24 on the connected interface. But suddenly Gi0 / 1, which contained the next hop for 3.1.1.0/24, went down for some unknown reason, and its connected route disappeared from RIB.
    %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to administratively down
    %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
    RT: interface GigabitEthernet0/1 removed from routing table
    RT: del 10.0.0.0 via 0.0.0.0, connected metric [0/0]
    RT: delete subnet route to 10.0.0.0/24
    RT: del 10.0.0.1 via 0.0.0.0, connected metric [0/0]
    RT: delete subnet route to 10.0.0.1/32
    IP-ST(default):  updating GigabitEthernet0/1

    Here's what happened:
    R00#show ip cef 3.1.1.0 detail
    3.1.1.0/24, epoch 0
      recursive via 10.0.0.3
        recursive via 10.0.0.0/8
          recursive via 100.100.100.101
            recursive via 100.100.100.0/24
              attached to GigabitEthernet0/2

    Oh. Now the packets on the network 3.1.1.0/24 go somewhere wrong. I cannot imagine a scenario where the expected behavior of a static route is to switch to another interface. If there is a backup path behind that interface, then you still need to create another static route ...

    What should I do? Indicate the interface immediately in the route. Re-create the route:
    R00(config)#no ip route 3.1.1.0 255.255.255.0 10.0.0.3
    R00(config)#ip route 3.1.1.0 255.255.255.0 Gi0/1 10.0.0.3

    Raise Gi0 / 1. We look where the route now leads to 3.1.1.0/24:
    R00#show ip route 3.1.1.0
    Routing entry for 3.1.1.0/24
      Known via "static", distance 1, metric 0
      Routing Descriptor Blocks:
      * 10.0.0.3, via GigabitEthernet0/1
          Route metric is 0, traffic share count is 1

    The interface is already indicated here. Therefore, there will be no recursive queries on the routing table. Checking FIB:
    R00#show ip cef 3.1.1.0
    3.1.1.0/24
      nexthop 10.0.0.3 GigabitEthernet0/1

    Yes, no "recursive". And if you pay off gi0 / 1 again? The route has disappeared.
    R00#show ip route 3.1.1.0
    % Network not in table
    R00#show ip cef 3.1.1.0
    0.0.0.0/0
      no route

    And this despite the fact that the route to 10.0.0.3 was still:
    R00#show ip cef 10.0.0.3
    10.0.0.0/8
      nexthop 100.100.100.101 GigabitEthernet0/2

    But what happens if the path to the next hop gives the default route, and the route to 3.1.1.0/24 does not refer to the interface?
    R00(config)#no ip route 10.0.0.0 255.0.0.0 100.100.100.101
    R00(config)#ip route 0.0.0.0 0.0.0.0 100.100.100.101
    R00(config)#no ip route 3.1.1.0 255.255.255.0 Gi0/1 10.0.0.3
    R00(config)#ip route 3.1.1.0 255.255.255.0 10.0.0.3
    R00#show ip route 3.1.1.0
    % Network not in table
    R00#show ip cef 3.1.1.0 detail
    0.0.0.0/0, epoch 0, flags default route
      recursive via 100.100.100.101
        recursive via 100.100.100.0/24
          attached to GigabitEthernet0/2

    Please note that the first line after “show ip cef” is “0.0.0.0/0” and not “3.1.1.0/24”. Despite the fact that there is a formal next hop, in fact all iterations of polling the routing table (except the first one) ignore the default route, which is logical, otherwise any query to the routing table would almost always be resolved (“resolve” means finding the interface into which need to send the package). Therefore, our static route is absent, but packets still fly to Gi0 / 2. Everything seems to be the same as without an explicit interface? Not really. Let's say the routing protocol was told “redistribute static”. If the static route is gone, then the announcement also responds. And if not, the router will continue to tell everyone to "go there through me", and this will almost certainly turn into an L3 ring for the prefix 3.1.1.0/24, which could be accessible from somewhere else. But stop, we agreed not to touch the dynamic routing ...

    But what if you specify an interface in the static route, but don’t specify the IP address of the next hop? Answer: in the case of Ethernet, if proxy arp is not disabled on the next hop, the connection will not be broken, but the router may VERY blunder. More details. If we say “ip route 3.1.1.0 255.255.255.0 gi0 / 1”, then nothing terrible will happen, even a couple of hundred entries in the arp table will be digested by any router (and there are workaround scripts in which such a crutch is the optimal solution ), but “ip route 0.0.0.0 0.0.0.0 gi0 / 1” on the border router will probably kill him. Therefore, remember the general rule: if a static route is created with the next hop on the Ethernet interface, then its IP address should always be indicated. Exceptions are only when you have a very good idea of ​​what you are doing, why you are doing it and why you cannot do otherwise.

    And finally, we’ll do one very bad thing.
    R00(config)# ip route 3.1.1.0 255.255.255.0 10.0.0.3
    R00(config)#ip route 10.0.0.3 255.255.255.255 3.1.1.1

    The first route is fine, a hundred times tested. But the second is strange - it leads through the first. And the first now refers to the second, and we have infinite recursion. Here's what happened:
    IP-ST(default): updating same distance on 3.1.1.0/24
    IP-ST(default):  3.1.1.0/24 [1], 10.0.0.3 Path = 8, no change, not active state
    IP-ST(default):  3.1.1.0/24 [1], 10.0.0.3 Path = 2 3 7
    RT: updating static 3.1.1.0/24 (0x0):
        via 10.0.0.3
    RT: add 3.1.1.0/24 via 10.0.0.3, static metric [1/0], add succeed, active state
    IP-ST(default): updating same distance on 10.0.0.3/32
    IP-ST(default):  10.0.0.3/32 [1], 3.1.1.1 Path = 8, no change, not active state
    IP-ST(default):  10.0.0.3/32 [1], 3.1.1.1 Path = 2 3 7
    RT: updating static 10.0.0.3/32 (0x0):
        via 3.1.1.1
    RT: add 10.0.0.3/32 via 3.1.1.1, static metric [1/0], add succeed, active state

    Added successfully. But then it was highlighted in debag:
    RT: recursion error routing 3.1.1.1 - probable routing loop
    RT: recursion error routing 10.0.0.3 - probable routing loop

    And there was a log entry with severity 3:
    %IPRT-3-RIB_LOOP: Resolution loop formed by routes in RIB

    R00#show ip cef 10.0.0.3 detail
    10.0.0.3/32, epoch 0
      Adj source: IP adj out of GigabitEthernet0/1, addr 10.0.0.3 359503C0
       Dependent covered prefix type adjfib cover 10.0.0.0/24
      1 RR source [no flags]
      recursive via 3.1.1.1, unresolved
      recursive-looped
    R00#show ip cef 3.1.1.0 detail
    3.1.1.0/24, epoch 0, flags cover dependents
      Covered dependent prefixes: 1
        notify cover updated: 1
      recursive via 10.0.0.3, unresolved
      recursive-looped

    However, RIB does not see any crime:
    Routing entry for 3.1.1.0/24
      Known via "static", distance 1, metric 0
      Routing Descriptor Blocks:
      * 10.0.0.3
          Route metric is 0, traffic share count is 1
    R00#show ip route 10.0.0.3
    Routing entry for 10.0.0.3/32
      Known via "static", distance 1, metric 0
      Routing Descriptor Blocks:
      * 3.1.1.1
          Route metric is 0, traffic share count is 1

    Conclusion - never do that.

    Why can a static route not get into the routing table?


    Any networker should immediately give one of the explanations regarding any source of routes in IOS: there is another route to the same prefix, but with a smaller AD (everyone remembers Administrative Distance?). A route whose source is “connected” always has AD = 0, and no other route source can bring anything lower than “1”, even a static route with an explicit interface. Example connected:
    R00# show ip route
    C        10.0.0.0/24 is directly connected, GigabitEthernet0/1

    Those. as long as the Gi0 / 1 interface is in the up state and has an address from the subnet 10.0.0.0/24, no static route to this prefix will appear in the routing table.

    There is also the option “different route sources add routes to the same prefix with the same AD”. The behavior of IOS in this case is not documented, the general recommendation is "never do this."

    But let's see other, less obvious examples. For example, static routes can be created with the word “permanent”, which translates as “permanent”, and then they will always hang in the routing table. Right? Not.

    Add it and look:
    R00(config)#ip route 3.1.1.0 255.255.255.0 10.0.0.3 permanent
    R00#show ip cef 3.1.1.0
    3.1.1.0/24
      nexthop 10.0.0.3 GigabitEthernet0/1

    Put Gi0 / 1, and see:
    R00#show ip cef 3.1.1.0
    3.1.1.0/24
      unresolved via 10.0.0.3

    It exists in RIB, and other routing protocols can use it:
    R00#show ip route 3.1.1.0
    Routing entry for 3.1.1.0/24
      Known via "static", distance 1, metric 0
      Routing Descriptor Blocks:
      * 10.0.0.3, permanent
          Route metric is 0, traffic share count is 1

    And now, without raising Gi0 / 1:
    R00(config)#no ip route 3.1.1.0 255.255.255.0 10.0.0.3 permanent
    R00(config)#ip route 3.1.1.0 255.255.255.0 10.0.0.3 permanent

    They just recreated it without changing anything. And here is what happened:
    IP-ST(default): updating same distance on 3.1.1.0/24
    IP-ST(default):  3.1.1.0/24 [1], 10.0.0.3 Path = 8, no change, not active state
    IP-ST(default): cannot delete, PERMANENT
    R00#show ip route 3.1.1.0
    % Network not in table
    R00#show ip cef 3.1.1.0
    0.0.0.0/0
      no route

    Permanent, speak? Not. There is one small nuance: in order for a permanent route to fit into the routing table forever, it needs to be resolved for at least a split second. Although what else is “forever”? When it remains hanging in the air without a resolving interface, it is enough to say “clear ip route *” or even more so “reload” so that it disappears from RIB.

    But let's continue. Let's do it like this:
    R00(config)#ip route 3.1.1.0 255.255.255.0 Gi0/1 10.0.0.3
    R00(config)#ip route 3.1.1.10 255.255.255.255 3.1.1.1

    It seems like normal routes. What will happen? With the second - absolutely nothing.
    IP-ST(default):  3.1.1.10/32 [1], 3.1.1.1 Path = 8, no change, not active state
    IP-ST(default):  3.1.1.10/32 [1], 3.1.1.1 Path = 2 3 6 8, no change, not active state

    The point is this. Suppose there is a route to XXXX via YYYY We add a route to X1.X1.X1.X1 (this prefix is ​​completely covered by XXXX) via X2.X2.X2.X2 (and it is also covered by XXXX). IOS makes a logical conclusion: the second route does not carry any new information and is completely useless, so it can not be installed in the RIB.

    And now feint ears.
    R00(config)#no ip route 3.1.1.10 255.255.255.255 3.1.1.1
    R00(config)#ip route 3.1.1.10 255.255.255.255 Gi0/1 3.1.1.1
    IP-ST(default): updating same distance on 3.1.1.10/32
    IP-ST(default):  3.1.1.10/32 [1], GigabitEthernet0/1 Path = 1
    RT: updating static 3.1.1.10/32 (0x0):
        via 3.1.1.1 Gi0/1
    RT: network 3.0.0.0 is now variably masked
    RT: add 3.1.1.10/32 via 3.1.1.1, static metric [1/0], add succeed, active state
    IP ARP: creating incomplete entry for IP address: 3.1.1.1 interface GigabitEthernet0/1
    IP ARP: sent req src 10.0.0.1 30e4.db16.7791,
                     dst 3.1.1.1 0000.0000.0000 GigabitEthernet0/1
    R00#show ip cef 3.1.1.10
    3.1.1.10/32
      nexthop 3.1.1.1 GigabitEthernet0/1
    

    And this brings us to another important point. Specifying an interface in a static route allows you to bypass many checks, since a static route no longer needs to perform recursive queries to RIB in search of a path to the next hop, and when it is added, it will not affect triggers on other routes. But this does not negate the main requirement: next hop must resolve to a specific interface, and that interface must be up. The fact that there will be no more recursive requests to RIB means that the next hop’s IP address is right behind the interface, and it will certainly respond to the arp request (from the point of view of the router). If proxy arp is enabled on the router adjacent to Gi0 / 1, then it will probably return its mac address in response to arp, and everything will be fine. Is that an extra entry in the arp table ...

    But still, you should not do this.

    It is necessary to mention another important point. A static route should, in theory, disappear from the routing table as soon as it stops resolving. But in practice, there are many situations when the next hop disappears, but at the same time the static route remains for some time. For example, when next hop resolves through the route received from the dynamic routing protocol. The thing is that the process that monitors the presence of next hop in RIB can not always receive a notification about the disappearance of a route, and it is forced to periodically (once every 60 seconds by default) check whether everything is fine. This will cause a noticeable delay in network convergence.

    You can change the verification interval, for example, by 10 seconds using the command:
    ip route static adjust-time 10

    Also popular now: