“Perfect storm” and how it is treated

    Hello, Habr!

    Broadcast storm (broadcast storm) is such a nightmare of networkers, when in a few seconds the transmission of useful traffic in the entire data center network is paralyzed. How this happens and what you had to think about before - in our today's post.
    imageimage

    Where did it come from?

    At first there was Ethernet, and there was no routing in it, and now there is no routing either, and therefore the switch has to send data packets to all ports - well, for sure. A reply arrives from a destination to a specific port, the switch remembers the correspondence [port - destination], and the next “package” is sent no longer where, but to memorable places, so to speak.

    If there is no answer, and at the time of sending unknown unicast a couple of switches are closed in a ring, the package is returned to the “sender”, which, of course, again throws it to all ports, because well, what else can he do. The “ownerless” data packet returns again and flies again. Each next round of the “looped” broadcast flood is accompanied by an exponential increase in the number of packets in the network segment. Very soon, this avalanche “clogs” the port bandwidth, in the background, overloaded switch processors “boil” beautifully - and your network turns into a monument to itself.

    The cause of the storm can be either a hacker attack or a misfire of your own engineer when setting up equipment - or even a protocol failure. In other words, no one is safe.

    There was such a case

    In 2009, a Broadcast storm occurred in the network of one of our customers, which instantly spread to us, paralyzing the work of the entire data transmission network of the company. Mail, telephony and the Internet fell off, the monitoring system "went crazy" - and it became impossible even to localize the "source". A complete reboot of the switch did not help. We had no choice but to disconnect ALL ports sequentially, client and our own ... Such a "black Monday".

    As it turned out later, one of the client’s employees mixed up the equipment ports and looped his topology at the broadcast domain level. It happens.

    It is clear that in the risk group here, first of all, commercial data centers: a redundant connection for each client in itself means redundant connections between switches. However, I would also not recommend relaxing to networkers of corporate data centers: a couple of switches can be shorted to each other in the server room, by dispersal.
    The good news is that with proper preparation, you can get off with a slight fright where otherwise you would get a simple service.

    Getting Started

    Start by segmenting your network using VLAN: when the network is divided into small isolated segments (fault domain), the storm that “covered” one of the sections is limited to this section. Well, ideally. In fact, a powerful storm, alas, is capable of “dropping” packets into so-called independent virtual networks too.
    In a good way, you need to abandon the “loopback” VLAN both inside your own infrastructure and when connecting clients to the network (if we are talking about a commercial data center). For example, use the FHRP protocols and the U-shaped topology at the access level.

    image
    U-shaped topology avoids looping

    In a number of cases, however, there is no escape from the “ringing” with all the desire. Let's say we need to deploy a fail-safe (2N) infrastructure for the customer with a connection to our cloud. Here, the client VLANs have to be forwarded inside the cloud infrastructure between all the ESXi hosts of the virtualization cluster — that is, the solution itself implies a complete duplication of all network elements.
    We look at the picture - we see the ring.

    image
    A ring topology arises with a complete reservation of communication channels and customer infrastructure.

    What to do if you are the “lord of the rings”

    You can do different things , and each option, as usual, has its own advantages and costs.

    For example, the RSTP protocol(modification of SpanningTreeProtocol) can quickly - within 6 seconds - find and "break" Broadcast loops. He finds them through the exchange of BPDU (Bridge Protocol Data Unit) messages between the switches, and “breaks” the blocking of backup links. In case of problems with the primary channel, RTSP rebuilds the topology using the backup port.

    Good thing overall, but there is a nuance. One RSTP process is allocated for each VLAN, while the number of processes, unlike VLANs, is very limited, and with a sharp increase in the number of VLANs within the same process network, RSTP may not be enough. That is, it will work for a corporate data center, but not for a commercial one with an ever-growing number of customers (VLANs).

    For this case there is MSTP- Improved and supplemented edition of RSTP. It is able to combine several VLANs in one STP process (instance), which in a good sense of the word affects the scalability of the network: the “ceiling” here is 4096 clients (maximum number of VLANs). MSTP also allows you to manage traffic by distributing MST processes between the main link and the backup one, and makes it possible to unload “corrupted” switches if necessary. However, you need to be able to work with MST, that is, this is plus at least one not cheap wise guy on staff (which is not available to everyone).

    image
    Protocols RSTP and MST "break" the loop, blocking traffic on one of the channels

    From the proven alternatives to MSTP, we can recommend FlexLinksfrom Cisco, which we use when there is one switch or stack under the same control on the client side. FlexLinks can backup switch links without using STP, “assigning” primary and backup ports in each pair of ports. It is used at the access level when connecting equipment of different companies according to the Looped Triangle principle in a commercial data center. A very simple tool in terms of tuning, which is nice in itself, and allows you to count on greater stability of the service (compared, for example, with STP). You will love FlexLinks for instant switching to backup links and load balancing via VLAN - and then, perhaps, fall out of love for the opportunity to use it exclusively in the Looped Triangle topology.

    image
    In the Looped Triangle topology, you can instantly switch traffic between channels in the event of a failure.

    Now let's digress from the technique. Do you like cost-effectiveness as much as I love it? Then you will be interested to know that both MST \ RSTP and FlexLinks, blocking backup links, actually exclude half of the ports from the traffic cycle in nature.

    And here are solutions that don’t do this: Cisco VSS (Virtual Switch System), Nexus vPC (virtual port-channel), Juniper virtual router and other mLAG-like (multichassis link aggregation) technologies. The good thing is that it uses all available links, combining them into one EtherChannel logical channel. It turns out a kind of switching cluster, in which the control module of one of the switches (Control-plane) “steers” all the cluster links (Data-Plane). If the current Control-plane fails, its authority is automatically transferred to the “survivor”. We use Cisco Nexus vPC to balance the load between client links that have one switch or stack each. If on the client side two separate switches connected by a common VLAN are added to the STP scheme.

    image
    Combining links into one logical channel solves the problem with storms and does not affect performance.

    If you have clouds

    Virtualization, disaster-proof cloud services distributed between data centers, and other cluster solutions - all this requires a slightly different approach to organizing a Layer 2 network. We remove STP on the mezzanine - we get TRILL .

    TRILL uses an Ethernet-level routing mechanism and itself builds a loop-free path for Broadcast traffic, thereby preventing storms. Well, isn’t it a miracle? :) Still TRILL allows you to evenly distribute the load between the links (up to 16 links), combine distributed data centers into a single L2 network and flexibly manage traffic. TRILL is a generally accepted standard that quickly developed vendor options: FabricPath from Cisco (which we use) and VCSfrom Brocade. Juniper has developed its own Qfabric technology to create a single Ethernet factory.

    Control Shot

    Which protocol will give you 100% storm protection? That's right, no. Therefore, you may be interested in the following two tools:

    Storm-control
    Allows you to set a per-second “quota” for the number of Broadcast packets passing through one port. Anything beyond the “quota” is discarded, and thus the load is controlled. Some nuance is that Storm-control does not distinguish useful traffic from garbage.
    Control — plane policing (CoPP)
    A sort of storm-control for the switch processor. With a Broadcast storm, among other things, the number of ARP requests increases dramatically. When this amount goes off scale, the processor is loaded at 100% - and the network, of course, tells you “goodbye”. CoPP can “dose” the number of ARP requests and thus manage the load on the processor. It copes well with Broadcast storms from the points of exchange of traffic, and with various DDoS attacks - it is checked.

    How to build a death proof network

    So, which of the possible options we tested on ourselves and use depending on the input:

    1. U, V and U-shaped topologies + RSTP (MST) + storm control + CoPP.

    The basic set, first of all, for a commercial data center, in which you have to connect a large number of external (uncontrolled) networks to your own network - and therefore it is highly advisable to prevent the occurrence of "loops" at all.
    If U, V and U-shaped topologies are not your case, the “shortened” version of RSTP (MST) + storm control + CoPP is also suitable.

    2. If there is a task to maximize the capabilities of equipment and channels, take a look at the option mLAG (VSS, vPC) + storm control + CoPP .

    3. If you already have Cisco or Juniper equipment and there are no topological contraindications, try the Flex Links / RTG + storm control + CoPP combination.

    4. If you have a complex case with distributed sites and other pleasures of virtualization and fault tolerance, your option is TRILL + storm control + CoPP.

    5. If you do not know what your case is, we can talk about it :).

    The main thing is to start doing at least something now, even if you sincerely think that a Broadcast storm is what happens to others. In reality, storms “cover” networks of various scales, and ridiculous mistakes are made even by people who, as they say, twenty years in art. “And may it inspire you to exploit” (c).

    Also popular now: