SAN Experience at SAS
Clouds and virtualization are already familiar terms that have entered our life tightly. With them came new problems for IT. And one of the main ones is how to ensure adequate performance of shared disk storage in the cloud (read the cluster for virtualization)?
When creating our own virtualization platform, we set tests to find the best answer to this question. I want to introduce you to the test results today.
The correct system integrator will say that Infiniband and a pair of cabinets with 3Par will allow, having implemented them once, never again think about SAN performance, but not everyone has a place in the hardware to install extra cabinets with disks. What then? First of all, you need to identify the bottleneck that prevents virtual machines from enjoying all 2000 IOPs that the disk shelf promises to give.
At our booth, where we once learned to build a cluster, IOMeter from the virtual machine downloaded 100x2Gbe iSCSI 100% and showed excellent results - almost the same promised 2000 IOPs. But when the VMs became several dozen, and they all tried quite actively to use the disk system, an unpleasant effect was found - the network load did not fit even half, the controller on the shelf and IOPs were in stock, and on virtual machines everything slows down. It turned out that in such conditions the bottleneck is delays in the data network (latency). And in the case of implementing SAN on iSCSI 1Gbe, it is simply huge.
So on the basis of which protocol to build a SAN, if the scale does not require the connection of hundreds of storage systems, thousands of servers and everything is planned to be placed in two or three racks of the same data center? iSCSI 10Gbe, FC 8GFC? There is a better solution - SAS. And the best argument in choosing is the price / quality ratio.
Let's compare the costs and the result when building a simple SAN from 2 storage systems with 6 client servers.
SAN Requirements:
The classic solution to this problem looks like this:
Assume that the cost of storage with two iSCSI 10 Gbe, FC or SAS controllers are the same. We list the remaining components in the table (you can find fault with the prices - but the order is this):
For the price, the SAS-based solution looks great! In fact, it is close to the cost of the solution on iSCSI 1Gbe.
This approach was used in the construction of the first stage of the Cloudbox IaaS platform.
What did the tests show?
On a virtual machine running in the release SAN among 200 others, IOMeter was launched.
Storage load before the test:
Storage load during the test:
IOMeter test
settings : IOMeter workflow settings:
IOMeter performance during testing:
Conclusions:
The impact of hundreds of virtual machines running on storage at the same time did not lead to performance degradations typical of iSCSI 1 Gbe. IOMeter showed the expected IOPs for the test virtual machine - the maximum possible minus the workload.
So, the pros of this solution:
There are certainly disadvantages:
With seemingly obvious advantages, the popularity of such a solution is not very high. At least habr is not full of reports. What is the main reason: the youth of technology, a narrow segment of application, poor scalability ...? Or maybe all quietly used for a long time? :)
When creating our own virtualization platform, we set tests to find the best answer to this question. I want to introduce you to the test results today.
The correct system integrator will say that Infiniband and a pair of cabinets with 3Par will allow, having implemented them once, never again think about SAN performance, but not everyone has a place in the hardware to install extra cabinets with disks. What then? First of all, you need to identify the bottleneck that prevents virtual machines from enjoying all 2000 IOPs that the disk shelf promises to give.
At our booth, where we once learned to build a cluster, IOMeter from the virtual machine downloaded 100x2Gbe iSCSI 100% and showed excellent results - almost the same promised 2000 IOPs. But when the VMs became several dozen, and they all tried quite actively to use the disk system, an unpleasant effect was found - the network load did not fit even half, the controller on the shelf and IOPs were in stock, and on virtual machines everything slows down. It turned out that in such conditions the bottleneck is delays in the data network (latency). And in the case of implementing SAN on iSCSI 1Gbe, it is simply huge.
So on the basis of which protocol to build a SAN, if the scale does not require the connection of hundreds of storage systems, thousands of servers and everything is planned to be placed in two or three racks of the same data center? iSCSI 10Gbe, FC 8GFC? There is a better solution - SAS. And the best argument in choosing is the price / quality ratio.
Let's compare the costs and the result when building a simple SAN from 2 storage systems with 6 client servers.
SAN Requirements:
- Highly available connection
- Data transfer rate of at least 8GB / s
The classic solution to this problem looks like this:
Assume that the cost of storage with two iSCSI 10 Gbe, FC or SAS controllers are the same. We list the remaining components in the table (you can find fault with the prices - but the order is this):
For the price, the SAS-based solution looks great! In fact, it is close to the cost of the solution on iSCSI 1Gbe.
This approach was used in the construction of the first stage of the Cloudbox IaaS platform.
What did the tests show?
On a virtual machine running in the release SAN among 200 others, IOMeter was launched.
Storage load before the test:
Storage load during the test:
IOMeter test
settings : IOMeter workflow settings:
IOMeter performance during testing:
Conclusions:
The impact of hundreds of virtual machines running on storage at the same time did not lead to performance degradations typical of iSCSI 1 Gbe. IOMeter showed the expected IOPs for the test virtual machine - the maximum possible minus the workload.
So, the pros of this solution:
- Bandwidth. Each port is a 4-channel SAS2 - i.e. in theory, you can get 24Gbps per port.
- Latency. It is an order of magnitude lower than iSCSI 10GbE and FC 8GFC. In the virtualization task of a large number of servers, this becomes crucial in the battle for virtual IOPs.
- Easy enough to set up.
There are certainly disadvantages:
- No replication via SAS. To back up data, you need to look for a different solution.
- The cable length limit is 20 meters.
With seemingly obvious advantages, the popularity of such a solution is not very high. At least habr is not full of reports. What is the main reason: the youth of technology, a narrow segment of application, poor scalability ...? Or maybe all quietly used for a long time? :)