On data locality in hyperconverged systems
There are concepts that are commonly used both in professional communication and in marketing, implying that they are strictly defined and everyone understands them the same way. But it often turns out that a number of terms around which there is a lively discussion in the IT community did not have exact definitions and do not, and of course, everyone forgot to agree on a common meaning.
For example, the concept of "enterprise readiness". Since we started talking seriously about enterprise automation, it has been used with confidence both in technical documentation and with the support of sales for everyone who is not lazy. But he does not have a strict definition! There is a common understanding of "enterprise-ready systems", or "enterprise-wide systems" as solutions that are ready for use by large organizations from arbitrary sectors of human activity. Often, enterprise-ready solutions are simply called very expensive solutions - worth more than a million dollars, for example. It would seem - a curiosity. But this curiosity sets the level of discussion.
Reliability, Availability, Serviceability
Speaking of “enterprise-level” IT solutions, “enterprise” means any large organizations and management companies, but not IT companies. This is important because such organizations are guided by certain standards and serial products in the field of IT, it is important for them that they can independently operate the solutions they purchase; for them critical fault tolerance and reliability of the functioning of IT systems. Therefore, we prefer to call systems of "enterprise scale" software and hardware solutions designed to work in large organizations and characterized by:
- reliability: long time between failures;
- high availability (availability): even in the event of a failure of one of the components, the systems should continue to function without a noticeable decrease in operational properties;
- serviceability: i.e. the ability to quickly recover or replace a failed component at the lowest cost from both the operating organization and the supplier.
And this was not invented by us. Back in the 1960s IBM used the abbreviation "RAS" in advertising its mainframes - Reliability, Availability, Serviceability. Other characteristic properties can be distinguished, but, one way or another, they are easily reduced to RAS. It is here that the main expectations of large organizations from IT are concentrated.
However, reliability, availability and serviceability are also understood differently. In particular, there is some particular opinion that the mandatory requirement for high system availability is the data locality property. But even the locality of the data does not have a strict definition! The idea of data locality is that the data must be physically “somewhere close” to the place where it is processed. Of course, each manufacturer implements data locality in its own way. It is funny that it is the implementation of data locality that causes the most questions - although with respect to enterprise readiness as a whole, as we see, this property is not even of the second, but of the third order. But since it is this property of enterprise-ready systems that causes such fierce debate, let's find out
Data locality
First you need to understand the concept of data locality and examples of its implementation in various classes of infrastructure and platform systems.
What level of data locality are we talking about? About data locality in relation to processor sockets? Or organizing interactions between geographically distributed data centers? Let's agree that we are talking about hyperconverged systems, i.e. about x86-based virtualization complexes without external storage systems - today, they are becoming the actual standard for implementing virtualized infrastructure for large and medium-sized organizations.
So, a hyperconverged system is, in fact, a lot of nodes on which virtual machines work. Virtual machines store data on internal drives of virtualization hosts. The data locality property implies that each virtual machine writes data to drives located on the same physical node as the virtual machine - so as not to overload the network. In the future, this data will be copied to other nodes to ensure redundancy of data storage, but the virtual machine reads it in its own physical node.
Fig. 1. Hyperconverged system combines resources from several nodes into a computing pool, and delegates local storage units in a single storage pool. VM Virtual Machine Volume 1, located on the first node, writes the first replica of its data blocks to local drives.
But what if the node fails? A virtual machine, of course, will be moved to another physical node. But should all its data “move” there, to this virtual machine? The main advantage of this approach is that the virtual machine can read its data more locally than over the network. The main minus is that data transfer overloads the network, and, as experience has shown, quite significantly. Therefore, in our implementation option, which we used in the Scala-R hypervergence computing platform, we deliberately did not do the rebuilding of the repository with automatic data transfer of virtual machines. At the same time, we proceeded not from speculative ideas about what the locality of data is, but from actual indicators of the availability of the system to which Scala-R corresponds.
Fig. 2.When a node fails, virtual machines from it, including VM 1 , migrate to another node, the first replica for volume VM 1 is now written to the local drives of the second node - this is its new locality. But is it necessary to automatically rebuild the entire storage pool so that the maximum number of data blocks of VM 1 are on the drives of the second node ?
Why are we doing this? Because we believe that IT infrastructure should not be redundant, its complexity and final cost should be justified, and behavior predictable. Examples of good practices include implementing monitoring and control systems such as HP OpenView, IBM Tivoli, BMC Patrol - in certain situations they could proactively perform proactive and corrective actions, but by default these features were disabled and only the system administrator was notified.
We consider such a policy to be very reasonable, and the analogy with the policies of the behavior of hyperconverged systems is seen directly. Migrating virtual machines from a failed node to others is a natural, predictable action necessary to ensure high availability. The transfer of local data, which inevitably increases the load on the network, from our point of view, should be left to the discretion of the operator.
Cross-site locality in hyperconverged systems
Indeed, is data locality required for enterprise-wide systems operation, and if needed, in which implementation option? In the early 2010s early hyperconverged systems were designed to be independent of network infrastructure performance. Gigabit Ethernet was the most common network solution in organizations' data centers, and for the first systems, such as the pioneer systems of this market, Simplivity and Nutanix, cross-site locality was considered an essential property. These solutions implemented the function of preferred recording to local devices, preferred reading from a local device, and automatic rebuilding of the entire storage network during live migration of the virtual machine to another node.
In software-defined storage (SDS) networks, the best effect was achieved when they were jointly used with virtualization platforms, when the volume blocks of virtual machines were located, if possible, on the same nodes where these machines are running, preferably from a local device storage. One of the historically first SDSs that implemented cross-site locality was Parallels Storage (now Virtuozzo Storage). It formed the basis of the software-defined network of the Skala-R hyperconverged complex (the R-Storage component).
But with the transition to 10-gigabit networks, many manufacturers of hyperconverged systems, such as Maxta, Atlantis, systems based on VMWare vSAN, etc., abandoned the implementation of cross-site locality. Most of the existing SDSs, including Microsoft S2D, Dell-EMC ScaleIO, RedHat CephFS, and RedHat GlusterFS, do not implement internode locality at all, and VMWare implements locality in vSAN as a local hot data cache and denies the need for internode locality . This is motivated by low latencies in a modern 10-gigabit network and the potential damage to the balance of the storage system, subject to the rules of inter-site locality.
Even in Nutanix, which in its early implementations emphasized inter-nodal locality, it has been implemented much thinner since 2015- if the delays from remote reading are lower than from the local one, then reading from the remote replica is performed, and the volume is not completely rebuilt when the virtual machine is migrated (cold blocks remain in place, that is, the block is re-localized at the remote node, carried out on first reading).
Moreover, most hyperconverged systems are currently delivered without a network solution! For our part, we have made a Mellanox network solution for RoCE networks, which has a throughput of 56 Gb / s and provides the functions of unloading central processors (CPU offload), an integrated part of the Skala-R complex. Duplicated switches provide reliability, their properties provide a reliable reserve for bandwidth even in scenarios with mass migration of virtual machines, failure of even an entire switch does not lead to reduced availability.
As for inter-site locality, it was noted that it was inherited by Skala-R from the Parallels Storage implementation: data blocks of a virtual machine are preferably written to local devices, and reading is done locally. But the significance of this property for Skala-R is small - the network solution we use almost eliminates the network factor in performance issues.
Implemented in the "Rock-R" and the function of rebuilding the repository taking into account the locality, but it does not start automatically during live migration of machines. “Automation” would be easy to implement, but the analysis of the operating experience of the system did not confirm the feasibility of such a solution. For example, in a situation of a planned or emergency reboot of one of the nodes (which happens with Skala-R much less than in the case of Nutanix and Simplivity), which takes 1-2 minutes, automatic rebuilding of the storage will not make any sense and at the same time entail noticeable decrease in performance. If, after the migration, the virtual machine remains on the new node, its new data will in any case be written to local devices. One way or another, the system administrator always has complete information to make a decision on the re-migration of machines,
Conclusion
So, how efficient is data locality for hyperconverged systems? In general, cross-site locality is useful in software-defined storage networks, as it allows you to reduce interconnectivity, reduce network load and increase overall system performance. But the function of automatically rebuilding the storage during the migration of virtual machines is not only not necessary - in the context of relatively large virtual machines it is rather harmful.
In general, inter-site locality is not related to readiness for workloads and operation at the enterprise level (in terms of RAS). This is only an additional feature, and the higher the performance of network solutions, the lower its value.
