June 2, 2017 at 11:32Flashcache - cheap and cheerful or an alternative to HW RAID 10 SAS
Until 2014, on FirstVDS servers, we used industrial HDD-drives with
SAS-interface and hardware controllers, assembled in RAID 10. This solution completely satisfied us in terms of reliability and performance. Problems with partial loss of client data were 3 times in 12 years of use. The hardware controllers burned out twice. Once the battery failed and during an emergency power outage the built-in cache memory of the raid was cleared.
However, SAS HDDs are expensive. For one server, we took a set of 4 600 GB disks, a hardware RAID controller with a battery. The whole solution cost 44,806 rubles. for 1 Tb. We did not want to raise prices for VDS. It was necessary to find a cheaper solution, while not losing speed and reliability. And ideally, and increase the space provided for VDS.
Only SSDs are even more expensive. At that time, 240 GB drives cost from 8,000 rubles. It was cheaper to stay on Raid 10 SAS than to use SSDs with a total volume of 1 TB. And to increase storage is even more expensive. Therefore, we reviewed several software solutions and included SSDs in the tests to compare speed. A table with the results below.
zfs is an adaptive replacement cache file system and logical partition manager developed by Sun Microsystems. Zfs cannot be included in the original version of the Linux kernel due to license incompatibility (CDDL vs GPL). The system can be screwed on with DKMS modules, but the efforts are not worth it - judging by public tests, the write / read speed was low. They didn’t start testing themselves.
bcache - development of Google, in 2013 was still raw - was not used in production. It only worked with CentOS 7, and we used CentOS 6. Bcache did not test either.
lvm cache is a technology of the Linux community. It also worked only with CentOS 7, but there were no public tests at that time - we decided to do it ourselves. The numbers did not like.
flashcache- developed by Facebook: the company inspires confidence, and the technology has already been tested in production.
Flashcache works in 3 modes:
Since write back is the fastest mode, we chose it for testing.
MD - software raid. Flashcache works in tandem with MD and Raid 1. We included MD without Flashcache in our testing to test how it works separately.
In order to bring the research conditions as close as possible to the real ones, we started random writing and reading to a 32 GB file (mounted file system).
Flashcache in writeback mode bypassed lvmcache and overtook software raid. He lost a lot to expensive SSDs, but most importantly, flashcache exceeded our solution on SAS HDD.
According to a study in January 2014, we implemented flashcache on SSD + SATA HDD. Since then, one server costs 1 SSD and 2 SATA HDDs of 4TB in the mirror. The technology works in writeback mode: it quickly writes data to the cache and slowly drops to the main carrier.
When implementing and maintaining flashcache, we were faced with some features of the technology.
1) SSD wears out
Due to the exceeded number of records / rewrites, SSD stops recording new data. To prevent this, we monitor SMART attributes:
The monitoring program monitors these values automatically and notifies employees of problematic disks. We can only change them on time.
Previously, we used 240 GB disks, they worked less than a year. Now over-provisioning technology allows us to increase the spare area of the disk and thereby extend the life of the SSD. We cut a disk of 1 TB up to 240 GB, this is the work area, the remaining 760 GB is a reserve for wear. Now SSD on average 1 year.
2) Failures when the SSD burns out and unsynchronized (dirty) data is lost
In writeback mode, the data first goes to the SSD cache and only then to the SATA HDD. Data that did not have time to chip on the SATA HDD is called dirty. In the event of a failure, they burn out irrevocably with the SSD. During an emergency power outage, the SSD can also fail with data loss.
Fortunately, crashes don't happen that often. For 2.5 years, we had two cases with loss of client data that did not have time to register in the repository.
There are two ways to reduce the number of failures:
3) Long clean cache
Change SSD and configure flashcache - 5 minutes. But before that, you need to clear the cache - throw all the dirty data onto disks.
On average, we have 30% of dirty data on SSDs, the maximum is 70%. Clearing the cache takes up to 4 hours.
At this time, the system runs slower because it accesses slower media. We always warn customers about a drop in speed, but we can’t force the process. The write speed to the SATA HDD depends on how intensively the clients use the drive. The more intensively used, the greater the load and slower recording speed.
4) Cache may overflow
Frequently used data is in the cache and is called hot. On our servers there are about 13%, maximum 62%. This amount is enough to quickly read / write all VDS on the server. But overcrowding the cache and slowing down performance can be caused by distrust of just one client.
Suppose a client wants to test a disk subsystem. Runs a random file recording program. If the client’s disk capacity is larger than the cache, then everything is bad. The cache will overflow and everything will slide into poor performance. All VDS on the server will be affected.
If you decide to conduct such a test, do not expect actual results. We programmatically limit the attacker the number of disk accesses, this reduces the speed.
5) Flashcache does not work on Centos 7
After updating the kernel, flashcache became incompatible with Centos 7. Since this version of the distribution kit is installed on 50% of our servers, the problem is acute. Now Centos 7 is used with sw raid1 with SSD. On three clusters, we are testing enhanceio - another caching technology - but are not yet ready to announce the results.
Let us calculate how flashcache on SSD + SATA HDD is more profitable than RAID 10 SAS. To do this, we calculate the cost of each solution.
Since 2013, the dollar has risen in price 2 times. Therefore, a solution with flashcache in rubles costs almost the same as RAID 10 SAS, and in dollars it is 2 times cheaper.
By increasing the storage volume by 4 times, we reduced the price of 1 TB. Now it is 4 times cheaper in rubles and 8 times in dollars.
In 2014, we introduced flashcache - we increased the space provided for VDS by 4 times, and increased the speed of interaction with the disk subsystem. This decision came out cheaper than the previous one, allowing us to reduce costs and not raise prices for VDS.
Reliability remained in question, but there were fewer crashes with the HW RAID 10 SAS. In May 2015, for people for whom reliability and speed are crucial, we introduced tariffs with SSD as the main carrier.
SAS-interface and hardware controllers, assembled in RAID 10. This solution completely satisfied us in terms of reliability and performance. Problems with partial loss of client data were 3 times in 12 years of use. The hardware controllers burned out twice. Once the battery failed and during an emergency power outage the built-in cache memory of the raid was cleared.
However, SAS HDDs are expensive. For one server, we took a set of 4 600 GB disks, a hardware RAID controller with a battery. The whole solution cost 44,806 rubles. for 1 Tb. We did not want to raise prices for VDS. It was necessary to find a cheaper solution, while not losing speed and reliability. And ideally, and increase the space provided for VDS.
Only SSDs are even more expensive. At that time, 240 GB drives cost from 8,000 rubles. It was cheaper to stay on Raid 10 SAS than to use SSDs with a total volume of 1 TB. And to increase storage is even more expensive. Therefore, we reviewed several software solutions and included SSDs in the tests to compare speed. A table with the results below.
Alternative solutions
zfs is an adaptive replacement cache file system and logical partition manager developed by Sun Microsystems. Zfs cannot be included in the original version of the Linux kernel due to license incompatibility (CDDL vs GPL). The system can be screwed on with DKMS modules, but the efforts are not worth it - judging by public tests, the write / read speed was low. They didn’t start testing themselves.
bcache - development of Google, in 2013 was still raw - was not used in production. It only worked with CentOS 7, and we used CentOS 6. Bcache did not test either.
lvm cache is a technology of the Linux community. It also worked only with CentOS 7, but there were no public tests at that time - we decided to do it ourselves. The numbers did not like.
flashcache- developed by Facebook: the company inspires confidence, and the technology has already been tested in production.
Flashcache works in 3 modes:
- Write through - data is first written to disk, and then flushed to the cache. Only a record is cached.
- Write back - data is first written to the cache, then flushed to disk. Writing and reading are cached.
- Write around - data is written to disk, and gets to the cache after the first read. Read-only is cached.
Since write back is the fastest mode, we chose it for testing.
MD - software raid. Flashcache works in tandem with MD and Raid 1. We included MD without Flashcache in our testing to test how it works separately.
Test Results
In order to bring the research conditions as close as possible to the real ones, we started random writing and reading to a 32 GB file (mounted file system).
| Parameter | raid10 sas | SSD | MD | flashcache write back |
|---|---|---|---|---|
| queue depth | 32 | 32 | 32 | 32 |
| IOPSread | 1,401 | 51 460 | 598 | 6 124 |
| IOPSwrite | 999 | 23,082 | 230 | 3 205 |
| read speed | 5 607 Kb / s | 205 842 Kb / s | 2 393 Kb / s | 24 496 Kb / s |
| write speed | 3 998 Kb / s | 92 329 Kb / s | 922 Kb / s | 12 823 Kb / s |
Flashcache in writeback mode bypassed lvmcache and overtook software raid. He lost a lot to expensive SSDs, but most importantly, flashcache exceeded our solution on SAS HDD.
New solution with flashcache
According to a study in January 2014, we implemented flashcache on SSD + SATA HDD. Since then, one server costs 1 SSD and 2 SATA HDDs of 4TB in the mirror. The technology works in writeback mode: it quickly writes data to the cache and slowly drops to the main carrier.
When implementing and maintaining flashcache, we were faced with some features of the technology.
Flashcache features
1) SSD wears out
Due to the exceeded number of records / rewrites, SSD stops recording new data. To prevent this, we monitor SMART attributes:
- Media_Wearout_Indicator is the lifetime or wear of the disk: the value for the new disk is 100, and it decreases over time. The minimum allowable is 10, when this value is reached, the disk becomes read-only.
- Reallocated_Sector_Count - the number of reassigned sectors - should be less than 100.
The monitoring program monitors these values automatically and notifies employees of problematic disks. We can only change them on time.
Previously, we used 240 GB disks, they worked less than a year. Now over-provisioning technology allows us to increase the spare area of the disk and thereby extend the life of the SSD. We cut a disk of 1 TB up to 240 GB, this is the work area, the remaining 760 GB is a reserve for wear. Now SSD on average 1 year.
2) Failures when the SSD burns out and unsynchronized (dirty) data is lost
In writeback mode, the data first goes to the SSD cache and only then to the SATA HDD. Data that did not have time to chip on the SATA HDD is called dirty. In the event of a failure, they burn out irrevocably with the SSD. During an emergency power outage, the SSD can also fail with data loss.
Fortunately, crashes don't happen that often. For 2.5 years, we had two cases with loss of client data that did not have time to register in the repository.
There are two ways to reduce the number of failures:
- Use high-quality server SSDs. What we do is buy drives Intel, Hitachi, Toshiba, etc.
- Configure cache replication (mirror raid). The solution provides for the installation of a second SSD, but due to rare failures, we squeezed money for it.
3) Long clean cache
Change SSD and configure flashcache - 5 minutes. But before that, you need to clear the cache - throw all the dirty data onto disks.
On average, we have 30% of dirty data on SSDs, the maximum is 70%. Clearing the cache takes up to 4 hours.
At this time, the system runs slower because it accesses slower media. We always warn customers about a drop in speed, but we can’t force the process. The write speed to the SATA HDD depends on how intensively the clients use the drive. The more intensively used, the greater the load and slower recording speed.
4) Cache may overflow
Frequently used data is in the cache and is called hot. On our servers there are about 13%, maximum 62%. This amount is enough to quickly read / write all VDS on the server. But overcrowding the cache and slowing down performance can be caused by distrust of just one client.
Suppose a client wants to test a disk subsystem. Runs a random file recording program. If the client’s disk capacity is larger than the cache, then everything is bad. The cache will overflow and everything will slide into poor performance. All VDS on the server will be affected.
If you decide to conduct such a test, do not expect actual results. We programmatically limit the attacker the number of disk accesses, this reduces the speed.
5) Flashcache does not work on Centos 7
After updating the kernel, flashcache became incompatible with Centos 7. Since this version of the distribution kit is installed on 50% of our servers, the problem is acute. Now Centos 7 is used with sw raid1 with SSD. On three clusters, we are testing enhanceio - another caching technology - but are not yet ready to announce the results.
Flashcache implementation results
Let us calculate how flashcache on SSD + SATA HDD is more profitable than RAID 10 SAS. To do this, we calculate the cost of each solution.
| RAID 10 SAS | approximate prices for March 2013 |
|---|---|
| SAS 600 GB, 4 pcs. | 7,714 rub. x 4 |
| hardware controller + battery | 8600 rub. + 4500 rub. |
| cable | 850 rub |
| = 44,806 rubles. or $ 1493 (for a course of $ 1 = 30 rubles) | |
| - the cost of 1 TB of space on the parent server |
| SATA HDD + SSD | prices for May 2017 |
|---|---|
| SATA HDD 4 TB, 2 pcs. | 12 000 rub. x 2 |
| SSD | 17 100 rub. |
| = 41,100 rub. or $ 685 (for a course of $ 1 = 60 rubles) |
Since 2013, the dollar has risen in price 2 times. Therefore, a solution with flashcache in rubles costs almost the same as RAID 10 SAS, and in dollars it is 2 times cheaper.
By increasing the storage volume by 4 times, we reduced the price of 1 TB. Now it is 4 times cheaper in rubles and 8 times in dollars.
Conclusion
In 2014, we introduced flashcache - we increased the space provided for VDS by 4 times, and increased the speed of interaction with the disk subsystem. This decision came out cheaper than the previous one, allowing us to reduce costs and not raise prices for VDS.
Reliability remained in question, but there were fewer crashes with the HW RAID 10 SAS. In May 2015, for people for whom reliability and speed are crucial, we introduced tariffs with SSD as the main carrier.
