November 14, 2017 at 11:56Storage Multi-tier software-defined storage. Why, why and how is implemented using the MIRhosting cloud as an example
The topic of clouds has recently become more and more in demand. Now, even in the Russian Federation, companies increasingly understand why they can use this cloud and even begin to use it actively. The more companies show interest in the clouds, the more questions arise for us, the developers of cloud software, for the implementation of new trends and technologies, and for service providers that provide fail-safe operation of the platform as a whole.
In this article, I would like to share experience only about one of the aspects of the cloud, but at the same time, perhaps the most complex and important is the implementation of disk space. This article was prepared by Andrey Nesterenko, a cloud expert at MIRhosting, which is one of the hosting providers using Jelastic PaaS, which last month announced the opening of the third region of the cloud platform - in Russia.
First, some background information on how the cloud platform as a whole works. Yes, we are talking specifically about the cloud platform, and not about selling VPS and naming this action the Cloud. In our understanding, the key differences are fault tolerance (which, in particular, automatically excludes the use of local data storage), access to an "unlimited" pool of resources at any time, and payment upon the fact of using these resources, and not for limits or tariffs. The technical implementation of such a cloud requires the collaboration of all elements: equipment, network, orchestration, technical support.
Below we’ll talk about how we implemented only one component - fault-tolerant disk space.
Technologically, the data warehouse is powered by Virtuozzo Storage . For those unfamiliar with the idea, similar to other SDS implementations, Ceph is the best-known open source solution. The choice of Virtuozzo Storage, rather than Ceph, is primarily associated with commercial support and development, better performance characteristics , and “sharpening” for live migration of containers and, as a result, the fault tolerance of these containers. Another important reason is that Jelastic offers this integration with VZ SDS out of the box. I will write about Virtuozzo Storage, but with a few changes. All this can be applied to Ceph, and to other similar solutions.
So, architecturally in Virtuozzo Storage there are 2 components: Metadata Servers and Chunk Servers. Data is cut into so-called chunks, i.e. essentially binary “pieces”. These "pieces" are spread out on Chunk Servers, which can and should be as many as possible. In practice, Chunk Servers is a separate disk (it doesn’t matter, HDD or SSD or NVME), that is, if we have one server with 12 disks, then we will have 12 Chunk Servers that provide RAID 0 performance. Of course, there should be several such separate servers to guarantee fault tolerance, 5 is the minimum recommended amount. The more servers and chunks on them, the better the performance and faster replication in the event of failure of individual chunk servers.
Server metadata, respectively, manages this whole mess of pieces of data spread over a variety of chunk servers. For correct operation, only one metadata server is required, however, if it fails, the entire cluster will become inaccessible. On the other hand, the more metadata servers, the more performance is sagging, since any i / o cluster action should be recorded on more and more servers. There is also a classic split brain problem: in the event of failure of one or more metadata servers, the rest should remain in the "majority" to be guaranteed to be active and unique. Usually 3 metadata servers are made for small clusters, and 5 for larger ones.
This is what a typical cluster looks like:
(picture from Virtuozzo's blog, I think the guys will not mind)
The cluster administrator can set in global settings how many replicas should be guaranteed to be available. In other words, how many server chunks can fail without data loss? Suppose we have physically 3 servers, each of which has 3 disks (a chunk server) and we set the recommended value: 3 replicas. The first thing that comes to mind: what happens if one server as a whole and, accordingly, 3 chunk servers on it fail?
If 3 replicas of some data were recorded just on these 3 chunk servers within one physical server, then this data will be lost, despite the fact that we still have 2 physical servers online with 6 chunk servers on them. To manage this situation, you can configure different levels of replication: at the level of chunk servers (not recommended), at the host level (default), or at the rack level and at the building / location level.
Returning to our example above, when using the default value - at the host level, the metadata servers automatically determine which chunk servers are located on the same physical servers and provide replication to different physical machines. In order for replication to work at the rack or location level, you need to specify which machine is in which rack and / or location. In this case, fault tolerance will work at the level of different racks or even data centers. This is exactly what we did at MIRhosting: the data is distributed on 2 independent data centers.
So, using such SDS technologies, we get theoretically perfect fault tolerance, but the theory and white paper on the site is one thing, and the practice and demanding clients are another
The first thing I had to face was the network. In a normal situation, the traffic between the chunk servers is about 200 Mbps and rarely exceeds 500 Mbps. The more servers, the more traffic is "smeared" across different servers. But in case of failure of the chunk server (s), urgent replication begins to restore the correct balance of data. And the more servers we have, the more data is missing, the more data begins to fly over the network, easily creating 5 gigabytes or more. So, a 10 gigabit network is the minimum requirement, as is 9000 MTU (Jumbo frames). Of course, if we want to provide really good fault tolerance and overall stability, then it is better to have two separate 10 Gbps switches and connect each data storage via 2 ports.
By the way, this also solves such a complex issue as updating firmware on switches.
The second thing we decided was to ensure the operation of the cluster on the basis of dual data centers. Initially, the data warehouse network was running on pure L2. However, with the growth of the cloud platform as a whole and the volume of the data warehouse as its composite, in particular, we came to the logical decision that we need to switch to pure L3. This step is very much tied to another topic - the work of containers in the conditions of dual data centers and live migration. The network for the data warehouse was made similar to the internal and external network of containers, for unification. The topic is worthy of a separate article, I hope this will be interesting to the community. There were no performance degradations with the transition to L3.
The third and probably most important is performance. Clients are different, they have different requirements, different projects, and in the conditions of a public cloud, this becomes a difficult problem. This in itself is divided into 2 parts: the overall performance of the i / o cluster and the various required performance for different clients. Let's start with the first one.
In many ways, the performance issue was resolved using the “poke method”, since neither the documentation nor the information from Virtuozzo technical support, as such, have the best practices. One Virtuozzo technical support employee helped a lot, who spent 30 minutes of his time on the phone and told a lot of things. Having collected all the available information, adding some brains and passing through the colander of experience, we came to the following scheme:
Compute Node (the node where the containers are launched) - the system runs on SSD, pfcache is located on it. The second SSD, and recently NVMe - under the client’s cache. And one or two SSD / NVMe for local chunks. Using RAID1 is possible, but it seems to us at this stage of the development of SSDs that does not have much significance, given the automatic recovery of containers in case of failure (failover) in any case. Client cache has less effect than chunk on compute node. The system determines the nearest available chunk and tries to use it for “hot” data.
The Storage Node (a node that is used only for chunks, without containers) is obviously clogged with SSD and NVMe. HDDs have historically been used, in fact, to provide “cold” replicas.
HDD caching is not used, and the HDD itself is already trying to completely replace with SSD. Those storage systems, where there is still an HDD, always work through a good raid card, RAID0, with caching by write operation, which improves the performance of these HDDs.
The total available cluster performance directly depends on the number of chunks, i.e. drives.
The second task is the need to regulate i / o in a public cloud and various customer requests. For each container in the cloud, certain i / o limits are set for the number of operations (iops) and throughput (io limit throughput). At the moment, the MIRhosting cloud provides 500 iops by default and a bandwidth of 200 Mb / s, which is almost equal to the double performance of a SATA HDD. For the vast majority of customers, these numbers are more than enough, given that these limits are given for each container, which the client can create in any quantities.
However, some customers require increased i / o performance. Obviously, this is solved by different limits for different customers. But it is economically unprofitable to have a cluster capable of, say, 10,000 iops per container (and have an appropriate supply), and sell its resources with reduced limits.
There is a good solution - it is possible to use different levels in the data warehouse. Let's say the first level will be built on SSD + HDD, and the second level will be built on SSD + NVMe.
We will provide the declared characteristics and do it economically feasible (which is directly related to the cost). Moving containers from one level to another takes place “live” and is implemented in Jelastic PaaS also with an hourly rate, like other cloud resources.
Suppose we have one container that is located on the first Compute node. Its file system is divided into 5 chunks. Also, our cluster is configured to use 3 replicas, that is, each chunk will be placed on 3 different servers. The first slide of the gif is how the chunks will be placed when placed on tier 0. The following shows the process of crawling chunks onto Tier 1 media.
At the moment, cloud storage resources are quite stable and it's hard to imagine how you can live differently. Failure of disks, and even the entire node, does not create visible problems for customers and requires planned actions to replace and add disks.
PS Already in the process of preparing this material, an article was published habrahabr.ru/post/341168which discusses similar issues and topics. Our article reflects a slightly different view of the same tasks and problems, and we hope that it will be useful to the community.
It would be interesting to consider the mentioned minuses in the article separately, but most of these minuses are not related to the data warehouse as such, but to the more global work of the cloud, in particular, migration issues.
If the topic is interesting, we can share our own developments on the operation of containers within the cloud based on dual data centers. In particular, from the side of the network and live migration within the framework of such a scheme.
In general, I have to agree with SyCraft that there are bugs, and Virtuozzo technical support can not always clearly say what is happening and why. Nevertheless, this solution, in fact, is the only one in its class that provides the necessary set of services, is actively developing and is commercial, that is, supported by developers.
You can try this solution of fault-tolerant disk space in conjunction with the automatic distribution and orchestration of containers, taking into account the specifics of a particular application from Jelastic PaaS, as well as the performance of the MIRhosting infrastructure, by registering here . We welcome your feedback.
In this article, I would like to share experience only about one of the aspects of the cloud, but at the same time, perhaps the most complex and important is the implementation of disk space. This article was prepared by Andrey Nesterenko, a cloud expert at MIRhosting, which is one of the hosting providers using Jelastic PaaS, which last month announced the opening of the third region of the cloud platform - in Russia.
First, some background information on how the cloud platform as a whole works. Yes, we are talking specifically about the cloud platform, and not about selling VPS and naming this action the Cloud. In our understanding, the key differences are fault tolerance (which, in particular, automatically excludes the use of local data storage), access to an "unlimited" pool of resources at any time, and payment upon the fact of using these resources, and not for limits or tariffs. The technical implementation of such a cloud requires the collaboration of all elements: equipment, network, orchestration, technical support.
Below we’ll talk about how we implemented only one component - fault-tolerant disk space.
Software-defined storage (SDS)
Technologically, the data warehouse is powered by Virtuozzo Storage . For those unfamiliar with the idea, similar to other SDS implementations, Ceph is the best-known open source solution. The choice of Virtuozzo Storage, rather than Ceph, is primarily associated with commercial support and development, better performance characteristics , and “sharpening” for live migration of containers and, as a result, the fault tolerance of these containers. Another important reason is that Jelastic offers this integration with VZ SDS out of the box. I will write about Virtuozzo Storage, but with a few changes. All this can be applied to Ceph, and to other similar solutions.
So, architecturally in Virtuozzo Storage there are 2 components: Metadata Servers and Chunk Servers. Data is cut into so-called chunks, i.e. essentially binary “pieces”. These "pieces" are spread out on Chunk Servers, which can and should be as many as possible. In practice, Chunk Servers is a separate disk (it doesn’t matter, HDD or SSD or NVME), that is, if we have one server with 12 disks, then we will have 12 Chunk Servers that provide RAID 0 performance. Of course, there should be several such separate servers to guarantee fault tolerance, 5 is the minimum recommended amount. The more servers and chunks on them, the better the performance and faster replication in the event of failure of individual chunk servers.
Server metadata, respectively, manages this whole mess of pieces of data spread over a variety of chunk servers. For correct operation, only one metadata server is required, however, if it fails, the entire cluster will become inaccessible. On the other hand, the more metadata servers, the more performance is sagging, since any i / o cluster action should be recorded on more and more servers. There is also a classic split brain problem: in the event of failure of one or more metadata servers, the rest should remain in the "majority" to be guaranteed to be active and unique. Usually 3 metadata servers are made for small clusters, and 5 for larger ones.
This is what a typical cluster looks like:
(picture from Virtuozzo's blog, I think the guys will not mind)
The cluster administrator can set in global settings how many replicas should be guaranteed to be available. In other words, how many server chunks can fail without data loss? Suppose we have physically 3 servers, each of which has 3 disks (a chunk server) and we set the recommended value: 3 replicas. The first thing that comes to mind: what happens if one server as a whole and, accordingly, 3 chunk servers on it fail?
If 3 replicas of some data were recorded just on these 3 chunk servers within one physical server, then this data will be lost, despite the fact that we still have 2 physical servers online with 6 chunk servers on them. To manage this situation, you can configure different levels of replication: at the level of chunk servers (not recommended), at the host level (default), or at the rack level and at the building / location level.
Returning to our example above, when using the default value - at the host level, the metadata servers automatically determine which chunk servers are located on the same physical servers and provide replication to different physical machines. In order for replication to work at the rack or location level, you need to specify which machine is in which rack and / or location. In this case, fault tolerance will work at the level of different racks or even data centers. This is exactly what we did at MIRhosting: the data is distributed on 2 independent data centers.
Enough theory, tell me something really interesting
So, using such SDS technologies, we get theoretically perfect fault tolerance, but the theory and white paper on the site is one thing, and the practice and demanding clients are another
The first thing I had to face was the network. In a normal situation, the traffic between the chunk servers is about 200 Mbps and rarely exceeds 500 Mbps. The more servers, the more traffic is "smeared" across different servers. But in case of failure of the chunk server (s), urgent replication begins to restore the correct balance of data. And the more servers we have, the more data is missing, the more data begins to fly over the network, easily creating 5 gigabytes or more. So, a 10 gigabit network is the minimum requirement, as is 9000 MTU (Jumbo frames). Of course, if we want to provide really good fault tolerance and overall stability, then it is better to have two separate 10 Gbps switches and connect each data storage via 2 ports.
By the way, this also solves such a complex issue as updating firmware on switches.
The second thing we decided was to ensure the operation of the cluster on the basis of dual data centers. Initially, the data warehouse network was running on pure L2. However, with the growth of the cloud platform as a whole and the volume of the data warehouse as its composite, in particular, we came to the logical decision that we need to switch to pure L3. This step is very much tied to another topic - the work of containers in the conditions of dual data centers and live migration. The network for the data warehouse was made similar to the internal and external network of containers, for unification. The topic is worthy of a separate article, I hope this will be interesting to the community. There were no performance degradations with the transition to L3.
The third and probably most important is performance. Clients are different, they have different requirements, different projects, and in the conditions of a public cloud, this becomes a difficult problem. This in itself is divided into 2 parts: the overall performance of the i / o cluster and the various required performance for different clients. Let's start with the first one.
In many ways, the performance issue was resolved using the “poke method”, since neither the documentation nor the information from Virtuozzo technical support, as such, have the best practices. One Virtuozzo technical support employee helped a lot, who spent 30 minutes of his time on the phone and told a lot of things. Having collected all the available information, adding some brains and passing through the colander of experience, we came to the following scheme:
Compute Node (the node where the containers are launched) - the system runs on SSD, pfcache is located on it. The second SSD, and recently NVMe - under the client’s cache. And one or two SSD / NVMe for local chunks. Using RAID1 is possible, but it seems to us at this stage of the development of SSDs that does not have much significance, given the automatic recovery of containers in case of failure (failover) in any case. Client cache has less effect than chunk on compute node. The system determines the nearest available chunk and tries to use it for “hot” data.
The Storage Node (a node that is used only for chunks, without containers) is obviously clogged with SSD and NVMe. HDDs have historically been used, in fact, to provide “cold” replicas.
HDD caching is not used, and the HDD itself is already trying to completely replace with SSD. Those storage systems, where there is still an HDD, always work through a good raid card, RAID0, with caching by write operation, which improves the performance of these HDDs.
The total available cluster performance directly depends on the number of chunks, i.e. drives.
The second task is the need to regulate i / o in a public cloud and various customer requests. For each container in the cloud, certain i / o limits are set for the number of operations (iops) and throughput (io limit throughput). At the moment, the MIRhosting cloud provides 500 iops by default and a bandwidth of 200 Mb / s, which is almost equal to the double performance of a SATA HDD. For the vast majority of customers, these numbers are more than enough, given that these limits are given for each container, which the client can create in any quantities.
However, some customers require increased i / o performance. Obviously, this is solved by different limits for different customers. But it is economically unprofitable to have a cluster capable of, say, 10,000 iops per container (and have an appropriate supply), and sell its resources with reduced limits.
There is a good solution - it is possible to use different levels in the data warehouse. Let's say the first level will be built on SSD + HDD, and the second level will be built on SSD + NVMe.
We will provide the declared characteristics and do it economically feasible (which is directly related to the cost). Moving containers from one level to another takes place “live” and is implemented in Jelastic PaaS also with an hourly rate, like other cloud resources.
Suppose we have one container that is located on the first Compute node. Its file system is divided into 5 chunks. Also, our cluster is configured to use 3 replicas, that is, each chunk will be placed on 3 different servers. The first slide of the gif is how the chunks will be placed when placed on tier 0. The following shows the process of crawling chunks onto Tier 1 media.
At the moment, cloud storage resources are quite stable and it's hard to imagine how you can live differently. Failure of disks, and even the entire node, does not create visible problems for customers and requires planned actions to replace and add disks.
PS Already in the process of preparing this material, an article was published habrahabr.ru/post/341168which discusses similar issues and topics. Our article reflects a slightly different view of the same tasks and problems, and we hope that it will be useful to the community.
It would be interesting to consider the mentioned minuses in the article separately, but most of these minuses are not related to the data warehouse as such, but to the more global work of the cloud, in particular, migration issues.
If the topic is interesting, we can share our own developments on the operation of containers within the cloud based on dual data centers. In particular, from the side of the network and live migration within the framework of such a scheme.
In general, I have to agree with SyCraft that there are bugs, and Virtuozzo technical support can not always clearly say what is happening and why. Nevertheless, this solution, in fact, is the only one in its class that provides the necessary set of services, is actively developing and is commercial, that is, supported by developers.
You can try this solution of fault-tolerant disk space in conjunction with the automatic distribution and orchestration of containers, taking into account the specifics of a particular application from Jelastic PaaS, as well as the performance of the MIRhosting infrastructure, by registering here . We welcome your feedback.
