NetApp Data ONTAP 8 - Storage Hypervisor

This year will be the “Cluster-mode year” for the company, despite the fact that the version of Data ONTAP supporting this mode of operation, its own internal OS, which works in all NetApp storage systems, was released in version 8.0 back in 2008, according to its true capabilities to create a "multisite storage cluster" are starting to be implemented only now.

But first, a little history.
They started talking about the prospects of developing storage systems in the direction of “cloud” systems several years ago, although this buzz word was not yet included in the IT vocabulary, and then they talked about GRID systems replacing the “classic” network storage systems.
There is nothing surprising in the fact that NetApp, as a company that traditionally pays a lot of attention to promising areas in the industry, was one of the first to deal with the topic of “cloud” cluster systems. And this is what came of it.

Back in 2003, NetApp bought the startup Spinnaker, which was developing in the field of cluster file systems and Global Namespace tools, and a few years later launched a special OS called Data ONTAP Generation X (GX or v10). It was a special OS for some NetApp systems, allowing you to build multi-node storage clusters. Unfortunately, it had many limitations, in particular, it worked only as an NFS server, and did not have many important and familiar features of the “classical” Data ONTAP 7.x, therefore it did not win much popularity, it was expensive, limited, and required significant special knowledge for launch and use, and, as a result, its use was limited to the market for high-performance file servers for HPC (High-performance Computing) and storage systems for scientific and audio-video information. The total number of customers in the world using Data ONTAP GX did not exceed a couple of hundred.

The next step of NetApp was an attempt to merge, “merge” together these two OSs, the “classic” Data ONTAP 7, which has rich functionality that I described in detail on this blog and runs on tens of thousands of systems, and the new, “cluster”, “ cloudy. " However, the “merger” announced in 2008 turned out to be largely “fictitious,” just from one OS distribution called Data ONTAP 8.x it became possible to install two OSs: either in 7-mode, that is, the “classic” Data ONTAP, or in the mode called “Cluster-mode”, which was the development of Data ONTAP 10. Unfortunately, as I said above, these were just two OSs installed from the same distribution, and that’s all. The familiar Data ONTAP Classic features did not appear in Cluster-mode, and still they were two incompatible OSes,

As a result, the existing couple of hundreds of GX clients were migrated to 8.x Cluster-mode, and the majority of NetApp users continued to use “7-mode”. A significant barrier, in addition to functional limitations, was the price of the solution, as well as the complexity of the infrastructure implementation, requiring expensive components, such as an internal Cluster Network using 10 Gigabit Ethernet.
However, work continued, and gradually, in 8.x Cluster-mode, all those features familiar to 7-mode began to appear, such as replication, block deduplication, and, most importantly, work with block protocols - FC, iSCSI and FCoE. Let me remind you that, prior to version 8.1, Cluster-mode was a purely “file storage” operating under the NFS and CIFS protocols, which, alas, did not suit everyone.

Meanwhile, as NetApp “digested” Spinnaker’s legacy, competitors in the field began to appear on the market, Isilon’s product began to be sold (and eventually sold entirely to EMC), and the growing interest in “cloud” IT systems naturally became “ the locomotive of "development and" clouds "of storage - multi-node clusters, a kind of" cloud storage ", perfectly fit into the" cloud paradigm ".

And so, starting with Data ONTAP 8.1, Cluster-mode became able to work with block SAN protocols, began to be able to use asynchronous replication and snapshots, which are familiar to Data ONTAP Classic , deduplication and compression, and finally, prices were reduced, and the use of Cluster-mode has become somewhat more accessible to users.

What does this Cluster-mode give, from what we have not had so far, and how does it work at all?

In order to answer this question, the easiest way would be to present Data ONTAP in Cluster-mode as a kind of “storage hypervisor”. You are already familiar with how the server virtualization system hypervisor, such as VMware ESX, MS Hyper-V, or Citrix XEN, works, I think. Data ONTAP in Cluster-mode works very similarly, but instead of virtual machines with applications under such a hypervisor there will be user-created “virtual storage systems” with data called Vserver, which, like virtual machines under a server hypervisor, can interrupt operation and access Migrate to them from one physical controller in the cluster to another, depending on their availability or performance requirements, or some other needs that you ask.

Thus, Data ONTAP Cluster-mode is the "hypervisor" for the storage system.



At the risk of inflating the article with many technical details that are of interest only to active NetApp users, I’ll tell you a few more details.

First of all, understanding Cluster-mode, you should understand the difference between the concepts of Global Filesystem and Global Namespace . Data ONTAP 8.1 Cluster-mode is a Global Namespace , but NOT a Global Filesystem .

Global Namespace allows you to see a cluster and its space, as a set of nodes of its components, as a single “entity”, an integral space, regardless of where and how your data is stored. However, from a storage point of view, each controller node still operates on the data stored on its aggregates and volumes. One single file cannot reside on more than one aggregate / controller node. Also, it cannot migrate, distributed partly on one, partly on another controller node.
It seems to me that it is very important to understand this, so I focus on this so much.

Of course, devices that implement the Global Filesystem have fewer restrictions in this place (for example, EMC Isilon with its OneFS is just the Global Filesystem), however, in our world, as you remember, nothing comes free, you have to pay for everything, and implementation The Global Filesystem entails a number of very unpleasant negative side effects, primarily for performance. Isilon is very good for a certain number of tasks, but it is good, first of all, for certain types of workload (mainly sequential access). How important is the ability to store huge file (s) in your particular case, exceeding the total capacity of the disks connected to one controller and distributed across several nodes of the cluster, and whether you are ready to pay for such an opportunity with a deterioration in the characteristics of random access to them is up to you. Today on the market there is both one and the other option.

NetApp showed a convincing performance advantage in recent testing of SPECsfs2008 on the NFS protocol, where the 24-node FAS6240 system running Data ONTAP 8.1 almost 1.5 times exceeded the 140-node EMC Isilon S200 system.
At the same time, it should be noted that, in the case of NetApp, the worst case was specifically tested, the “worst case”, that is, only 1/24 of all operations went to the owner controller, 23 out of every 24 operations went through non-native controllers, transmitted through cluster network, and no NetApp existing optimization tools were used, such as, for example, Load Sharing Mirrors (RO copies) on non-native cluster nodes, which, of course, will increase performance in real life.

Let me remind you that the SPECsfs2008 testThis is a classic and authoritative test that simulates the average type of file access via the NFS (and CIFS) protocols generated by a mixture of operations of the corresponding protocol, and there, of course, a lot of operations with metadata and, mainly, random access.

So - Data ONTAP 8.1 Cluster-mode is a Global Storage Namespace , but NOT a Global Storage Filesystem . Despite the fact that you see the cluster as a single entity, a separate file stored on it can not exceed the aggregate capacity of one controller. However, you can access the data in this file through any of the cluster controllers. This is the difference between Global Filesystem and Global Namespace.

The second point on which I want to dwell in more detail is how the cluster is physically built.

Despite the fact that, formally, the “unit” of cluster size is one node, which is one physical controller, these nodes are always included in HA pairs. For this reason, the number of nodes in a NetApp Data ONTAP 8.x Cluster-mode cluster is always even. This ensures reliability and high availability (High Availability) of the nodes, in the same way as was done for controllers in 7.x.
Therefore, you cannot make a 5- or 15-node cluster, but you can not make a 4-, 6-, or 16-node cluster.

Currently, you can build a cluster that works as a NAS server (NFS or CIFS) from 24 nodes-controllers (nodes), that is, from 12 HA-pairs of controllers.
Support for block protocols (iSCSI, FC, FCoE) has also appeared in Data ONTAP 8.1. However, when using block protocols (only one of them, or in combination with NAS), the maximum cluster size is currently supported in the amount of four nodes, or two HA-pairs . This value, as I think, will grow as soon as everything is debugged and reliability is ensured, yet 8.1 is the first version with such functionality, but for now - keep this in mind. This is due, first of all, to the fact that file protocols such as NFS or CIFS, in fact, are fully managed and monitored on the side of the hundred, and it is easier for him to provide all the necessary work procedures and synchronization of processes between cluster nodes.

The third point, which I would like to cover in more detail, is that at the moment NetApp has rather strict hardware requirements for implementing cluster communications. To build the Cluster network, that is, the internal, intercontroller network of the cluster itself, only two 10-Gigabit switch models are currently supported; these are the Cisco Nexus 5010 (20 ports, a cluster of up to 12/18 nodes) and the Cisco Nexus 5020 (40 ports, a cluster more than 18 nodes), they are sold by NetApp with their affiliate numbers, as part of the overall quota of such a system, and a specially designed NetApp configuration file for these switches is installed on them. Moreover, you can use these switches only for the tasks of the internal cluster network, Combine them with other tasks, such as to connect them to the customer's network - it is impossible . Even if there are still ports left.

However, there is good news. Now, NetApp and Cisco, as a time-limited promotion, when ordering cluster-mode storage from NetApp, give the infrastructure equipment necessary for building a cluster for a symbolic price of $ 1 for Cisco Nexus for the Cluster network and Cisco Catalyst 2960 for the Cluster management network, plus the necessary SFP and cable. At the same time, the price of a Data ONTAP 8.1 Cluster-mode system of two nodes, for promotion, is equal to the price of a similar 7-mode configuration, and the infrastructure part will come at a price of $ 5, for five devices (two Nexus 5010, two Catalyst 2960, cable set ), plus service charges.

Before your eyes light up and your hands shake “buy a Nexus 5010 for one bucks”, I would like to clarify on a separate line that this offer is valid only for buying cluster-mode systems, and, according to the terms of the purchase, cannot be used for what else.
A system purchased through a promotion system with two nodes can be expanded up to 12 nodes (6 HA-pairs) by purchasing only SFP and cable.
The structure of the cluster-mode cluster is as follows (in the figure, for example, a two-node system is shown):



As the switches of the client network, you can use any switches, Ethernet or FC, depending on the needs of the user.

As Cluster network switchesonly the Cisco Nexus 5010 (for clusters with up to 12/18 nodes) or 5020 (for more nodes) can be used. NetApp recommends the Cisco Catalyst 2960

as a Cluster Management Switch , but, at present, it does not oblige you to buy this particular model, if necessary, use a different model from the client, this can be done through the PVR institution, a special request for verification and configuration update from NetApp . NB: SMARTnet for such a switch is a must!

Cluster Network is a dedicated 10Gb Ethernet network dedicated only to this task. The only exception is the FAS2040, which can be used in Cluster-mode using Gigabit Ethernet, but it is not recommended for use with other controllers. Please note that even for the 2040 and its Gigabit Ethernet,switches other than the Nexus 5010/5020 are not supported !

Cluster nodes may be different in model. You can combine any controllers for which cluster-mode compatibility is declared into a single cluster (with the only exception in the form of FAS2040, which is not recommended (although possible) to use with controllers of another type, due to the above reason for the lack of 10G ports)

You can also combine in a cluster and systems with disks of various types, for example, you can build a system with SAS, SATA and SSD disks within one single cluster, and organize data migration between different node controllers and disks of different types.



Thus, using Data ONTAP Cluster-mode, which, I recall, can work on any NetApp controllers sold today, you create “virtual storage systems” in much the same way that you create “virtual servers” under the server virtualization hypervisor that are independent from the hardware of the host, they can arbitrarily and “live” migrate between physical controllers according to your requirement or application, you can increase cluster performance on the go by scaling such a cluster, adding new “ho” to it you "Controller, and do much, already familiar to users of VMware vSphere, MS Hyper-V or Citrix Xen Server. Only for storage system.

Judging by the activity that NetApp is now publishing on its website Best Practices for using Cluster-mode for databases (including Oracle and MS SQL Server), application systems such as SAP, MS Sharepoint, and other “business heavyweights” ", The demand for such solutions is now very high.
I think that this year we will see the introduction of systems using Data ONTAP Cluster-mode in Russia.