NetApp Metrocluster

In 2007, the Forrester consulting agency conducted a survey of 250 IT specialists in order to assess the risks of accidents for IT, both inside and outside the data center, for example, the risks of natural accidents and disasters.

After processing and publishing the results, it became clear that the usual means of ensuring fault tolerance in the form of, for example, the traditional “redundant, redundant controller and RAID” can protect against only 31% of all possible failures.

On the graph, you also see IT disasters such as power failures (" - what happened to your electricity? - It blinked . "), Software problems (" ... and it will be closed "), human errors (" as you said the name of the volume, which had to be unmounted and banged?"), network settings errors (" - and which interface to use? - try eth0. "), as well as various natural ( and not so ) disasters, such as fires , floods and so on.

Thus, it becomes clear that traditional means performance data protection to protect it, alas, is not enough, no matter how many "nines" you either have promised in the advertisement. And when the cost of downtime or loss of data becomes very significant, there is a question of finding a solution that provides used for greater reliability than traditional solutions.


Typically, to protect data from local disasters, whether it is power failure, fire, " conducting investigative measures ", and other similar force majeure events, the method of replicating data to a remote storage is used. Almost all manufacturers of storage systems offer similar tools today.

However, for beginners, getting acquainted with fault tolerant solutions is often a surprise that the presence of replication and a remote copy of your operational data does not mean true fault tolerance.

You have saved the data, but in order to use this data, you often need quite extensive reconfigurations. It is necessary to break replication from a source that is no longer working, transfer the copy online, explain to the servers that the data is now not on this system (with such IP and WWPN), but on the one with completely different addresses and properties.
You need to double-check all the settings, rewrite them (do not forget or confuse), make sure that everything starts, and only after that your servers will be ready to start from the saved replica.

All this is often complicated by the fact that not one application stores data on the system, but usually a lot of different ones, each with its own rules and ways of organizing fault tolerance, these replicas are often managed in different ways, created in different programs, and so on.

In addition, the switching process itself is often not uniform.
This is done somewhere completely manually, somewhere semi-automatically, but, as a rule, some general solution, not separately for each application, but not for the entire infrastructure as a whole.
But the storage system at the enterprise often uses many dozens of different applications. And all this “moving” must be done for each of them!
No, not in vain, oh not in vain the Russian people equate two crossings to one fire.

So why not assign all these tasks directly to the storage controller itself? Why not make a storage system that switches to its "replica" simple and "transparent" for applications?

It was from this simple idea that NetApp Metrocluster was born - a software and hardware solution for distributed cluster storage.

The idea behind Metrocluster was, like NetApp often, simple and witty.
For each controller from the pair that makes up the cluster (so far, unfortunately, there can only be two controllers), two sets of disks are connected at the current site. One set is its own, and the second is an exact synchronous copy of the data set of the disk set "neighbor". In addition, since modern FC disks have two equal “access” ports, each disk is connected with one port to the local one and the second to the remote controller, “crosswise”, each controller thus has access to the main set of disks, and to the copy, however, at a particular point in time, until a cluster takeover has occurred, it can only work with "its own set." Each record arriving at the controller is synchronously mirrored to the “second set” of disks in the neighbor.

In the event of an accident or any “abnormal event”, the controller, in addition to accessing “its” data, gains access to the cluster partner’s data and also transfers all its resources, such as the IP addresses of its Ethernet interfaces, WWPN interfaces FC, LUN names and "network ball", DNS names, and so on, so that after switching applications continue to work "as before", another controller simply serves their data.

It is simply amazing why none of the major storage vendors have implemented such a simple but effective model (something very similar in theory, however, is now trying to start selling EMC in its VPLEX product).

NetApp Metrocluster comes in two flavors. This is the so-called Stretched Metrocluster, in which the maximum separation distance of its components is determined by the permissible length of a special cluster interconnect cable, not more than 500 meters; and Switched Metrocluster, in which the length is limited only by the maximum length of the Longhaul LW Fiber Channel, currently 100 km.
(The picture from the VMware website shows the use of Metrocluster for VMware vSphere FT, but there may be any applications in its place)



Sretched Metrocluster works just like the more expensive and sophisticated Switched Metrocluster, but it can protect mainly from “local” accidents, for example, taking “half” of the storage to another floor, to a neighboring data center module, or to a neighboring building within 500 meters of cable . But even such a local option will help maintain operability in the event of a fire in a data center, “excavation”, local power failure, and so on.

Switched Merocluster offers a fully-fledged distributed storage system that can protect against a number of natural disasters (for example, NetApp Metrocluster uses Turkish-made Ford Motors factories, one of the main industrial enterprises of which is located in a seismically dangerous area).



Thus, you can create a storage system in which one half will be in Moscow and the other, for example, in Zelenograd, and both halves will work synchronously, like a logically unified structure. The servers of the Moscow data center will work with half of the storage system on their site, and the servers of Mytishchi or Dolgoprudny will work with their own, but at the same time, in the event of a failure (for example, a power failure in the data center, failure of any part of the storage system, failure of the drives, controller, or data transmission channel to honey data centers) data will remain available.

Any failure or a combination of such failures does not lead to data inaccessibility, and the process of switching and recovery is carried out by one simple command. The very same software applications that use data storage on Metrocluster, outside the very fact of switching the controller, do not require reconfiguration, and the cluster operation for them is completely, as they say, "transparent".

For clarity, let's look at possible failure scenarios.

The easiest option is a host failure.
The application is lifted by means of server clustering on host 2, and continues to work, gaining access to data through the "factory" of the original data center to storage 1. (Blue lines show the data access path)



Also, the usual story is a failure of the storage system controller.
As in the previously shown case, access switching is automatic.



The drama is growing. Failure of half of the entire storage. The operator or monitoring software decides on a cluster takeover, which is carried out in a few seconds by a single cf takeover command and is transparent to applications.



Catastrophe. The entire data center is lost, along with the hosts and half of the storage system. Access to data saved. The second cluster controller serves “its” data as usual, and the partner data from its copy on its website.



The gap of the communication channel. Cluster controllers are isolated. The work continues in the normal way, when communication is restored, resynchronization of the plexes will occur. To prevent a split brain situation, if your software can create such a situation, you may need an arbiter site - “tiebreaker”.



Of course, the solution as a whole turns out to be difficult and not cheap (although cheaper than analogues). Two sets of data disks (a sort of distributed Network RAID-1), one on each site, a dedicated internal FC switching “factory” through which communication within the “storage cluster” and mutual synchronous data replication between the sites, but in those cases when it is necessary to ensure the work is not " in 31% of cases"and" always ", when the cost of downtime or data corruption is high, organizations prefer not to save.

However, with the release of the new FAS3200 / 6200 series of storage systems in which a set of licenses for organizing the metro cluster is already included in the basic delivery, a step has been taken towards more massive application of this solution.