May 18, 2017 at 17:55Testing cloud platforms from the Top 3
Hello, Habr! I want to share my results from comparing AWS, Azure, and Google cloud platforms. Under the cut are the performance results that an ordinary user receives, and their comparison according to the “price / quality” criterion.
To begin with, I introduced such an indicator as the number of points received by testing tools, divided by the cost of this virtual machine per hour. The following “instances” were used to compare the performance of virtual machines:
* Current price for promo positions, regular price in brackets.
** Price with a 30% discount, provided that you use the virtual machine 24/7 per month.
update : after the publication of the article, there was news about smaller OS drives in Azure, which can save another $ 2.18 per month. This does not change the results, but it saves about 0.8% of the price of a virtual machine (Standard D3 v2).
All "instances" were "raised" in Europe - Ireland \ Germany \ Belgium. Creation occurred through the respective portals of cloud providers. For the instances, SSDs of various configurations were used:
In order to make the tests as objective as possible, I did not configure virtual machines in any way. All "instances" use Windows Server 2016, and for the last test - Ubuntu 04/16/02 LTS. I also refused any OS settings and had to install GeekBench for testing. I used trial versions from the manufacturer’s sites. Using apt-get for Ubuntu, the Hardinfo benchmark was installed. These are the only settings made with the operating system before running the tests.
On all virtual machines I ran the following tests 5 times in succession:
I found the most detailed description here . In short, the total number of tests is 23, they are divided into several categories - Cryptography, Integer, Floating Point, Memory. But actually, in the results of 25 tests, perhaps because I have a newer version. The tests use compression operations, work with JPEG, parsing HTML, SQLite. I would call it one of the most comprehensive tests for different types of operations.
Here I could not find a description, but as far as I understand, tests for sequential write \ read and random write \ read + the same tests are run in this program, but with a queue depth of 32.
Just leave it here: CPU Blowfish, CPU CryptoHash, CPU Fibonacci, CPU N-Queens, FPU FFT, FPU Raytracing.
The results of multi-core testing are the most interesting, if we look at the information on the systems under test, we can note that AWS and Google Cloud give the following picture: Intel Xeon @ xxx GHz 1 processor, 2 cores, 4 threads , while Azure gives “honest” Kernels: Intel Xeon E5-2673 v3 @ xxx GHz 1 processor, 4 cores . I suspect this is the reason for the almost 1.5-fold performance gap. The larger the number in the test, the better.
Total : Azure , AWS, Google Cloud.
More detailed results can be found in the repository .
CrystalDiskMark's results are the most mixed, but most likely this difference is due to the fact that the Google Cloud \ AWS platforms need to be additionally configured for optimal performance, while Azure in the default configuration shows good performance. In addition, I completely disregard the price, since it is quite difficult to calculate. The more numbers in the tests, the better.
In this configuration, Azure left the competitors far behind in all respects: sequential read / write, 4k random read / write, 4k random read / write from the deep queue equal to 32. I think the result is interesting, because these are default machines, therefore, this is the result that the average user will get.
Total : Azure , AWS, Google
I will leave the interpretation of the Hardinfo test results to readers . All test participants showed approximately the same result.
All clouds have their strengths and weaknesses:
When recalculating the power in monetary terms, the picture changes slightly and Google with a 30% discount starts losing to the Azure platform by an average of ~ 10%, although when comparing the calculators of both platforms head-on, the opposite picture may appear. Amazon, unfortunately, is starting to lose 28% in performance money from GeekBench benchmarks.
Naturally, these tests are synthetic and for each individual solution or application you need to carry out specialized workloads, but this can be a good starting point for choosing.
To begin with, I introduced such an indicator as the number of points received by testing tools, divided by the cost of this virtual machine per hour. The following “instances” were used to compare the performance of virtual machines:
| Cloud | Vm size | Cores | Ram | Price \ Hour on windows |
|---|---|---|---|---|
| Aws | m4.xlarge | 4 | 16GB | $ 0.406 |
| Azure | Standard D3 v2 | 4 | 14GB | $ 0.422 (0.488) * |
| n1-standard-4 | 4 | 15GB | $ 0.306 ** |
* Current price for promo positions, regular price in brackets.
** Price with a 30% discount, provided that you use the virtual machine 24/7 per month.
update : after the publication of the article, there was news about smaller OS drives in Azure, which can save another $ 2.18 per month. This does not change the results, but it saves about 0.8% of the price of a virtual machine (Standard D3 v2).
All "instances" were "raised" in Europe - Ireland \ Germany \ Belgium. Creation occurred through the respective portals of cloud providers. For the instances, SSDs of various configurations were used:
- Google Cloud - 100GB, maximum for trial;
- Amazon - 30GB, default maximum;
- Azure - 127GB, by default.
In order to make the tests as objective as possible, I did not configure virtual machines in any way. All "instances" use Windows Server 2016, and for the last test - Ubuntu 04/16/02 LTS. I also refused any OS settings and had to install GeekBench for testing. I used trial versions from the manufacturer’s sites. Using apt-get for Ubuntu, the Hardinfo benchmark was installed. These are the only settings made with the operating system before running the tests.
What tests were used
On all virtual machines I ran the following tests 5 times in succession:
- GeekBench (Windows only).
- CrystalDiskMark (Windows only).
- Hardinfo (Ubuntu only).
Geekbench
I found the most detailed description here . In short, the total number of tests is 23, they are divided into several categories - Cryptography, Integer, Floating Point, Memory. But actually, in the results of 25 tests, perhaps because I have a newer version. The tests use compression operations, work with JPEG, parsing HTML, SQLite. I would call it one of the most comprehensive tests for different types of operations.
Crystaldiskmark
Here I could not find a description, but as far as I understand, tests for sequential write \ read and random write \ read + the same tests are run in this program, but with a queue depth of 32.
Hardinfo
Just leave it here: CPU Blowfish, CPU CryptoHash, CPU Fibonacci, CPU N-Queens, FPU FFT, FPU Raytracing.
results
Geekbench
The results of multi-core testing are the most interesting, if we look at the information on the systems under test, we can note that AWS and Google Cloud give the following picture: Intel Xeon @ xxx GHz 1 processor, 2 cores, 4 threads , while Azure gives “honest” Kernels: Intel Xeon E5-2673 v3 @ xxx GHz 1 processor, 4 cores . I suspect this is the reason for the almost 1.5-fold performance gap. The larger the number in the test, the better.
| Cloud | GeekBench Score (1) | Price per hour | Perf Score / Price (2) |
|---|---|---|---|
| Aws | 6568.6 | $ 0.406 | 16 177 |
| Azure | 9508.4 | $ 0.422 | 22,530 |
| 6188.2 | $ 0.306 | 20 222 |
Total : Azure , AWS, Google Cloud.
More detailed results can be found in the repository .
Crystaldiskmark
CrystalDiskMark's results are the most mixed, but most likely this difference is due to the fact that the Google Cloud \ AWS platforms need to be additionally configured for optimal performance, while Azure in the default configuration shows good performance. In addition, I completely disregard the price, since it is quite difficult to calculate. The more numbers in the tests, the better.
In this configuration, Azure left the competitors far behind in all respects: sequential read / write, 4k random read / write, 4k random read / write from the deep queue equal to 32. I think the result is interesting, because these are default machines, therefore, this is the result that the average user will get.
Total : Azure , AWS, Google
Hardinfo
I will leave the interpretation of the Hardinfo test results to readers . All test participants showed approximately the same result.
conclusions
All clouds have their strengths and weaknesses:
- Google is an excellent discount system that turns on automatically and, at first glance, really gives a big bonus.
- Amazon is very strong in terms of brand confidence and the most popular platform.
- Azure - has a performance advantage by selling real cores, rather than virtual hyper-threading threads (vCPU).
When recalculating the power in monetary terms, the picture changes slightly and Google with a 30% discount starts losing to the Azure platform by an average of ~ 10%, although when comparing the calculators of both platforms head-on, the opposite picture may appear. Amazon, unfortunately, is starting to lose 28% in performance money from GeekBench benchmarks.
Naturally, these tests are synthetic and for each individual solution or application you need to carry out specialized workloads, but this can be a good starting point for choosing.