Performance testing of popular (and not so) CMS
Greetings habrazhitel! When developing FFCMS for my projects, I highlighted one of my main goals as fast speed and minimal consumption of equipment resources - however, all the values can be estimated only by comparison, which I will try to do in this article.
In my last publication, some users asked about the issue of system performance and I decided to test various popular and young CMS as of October 18, 2014. I would like to note - the system was tested immediately after its installation and the data can significantly change in one direction or another, depending on the period of use of the system - increasing the database, number of files and other conditions.
Foreword
Before conducting such testing, I wanted to look at the results of similar tests - after all, most likely they have already been carried out before me. However, to my disappointment, I could not find a single comprehensive test that would compare at least 3 or more systems according to many criteria (maybe I was looking badly?). This is surprising, because dozens (or even hundreds) of thousands of webmasters, programmers, and large companies use the popular CMS - is it really that nobody is interested in the issue of consumed resources by the system?
Test Objects
For testing, I chose a number of currently popular systems, both commercial and free. 10 systems were selected, their names and versions used are listed below:
- Bitrix standart - 14.5.0
- WordPress 4.0
- Drupal 7.32
- Joomla! 3.3.6
- Umi.cms 2.9.7
- FFCMS 2.0.3
- InstantCMS 2.1.1
- KodiCMS 12.20.37 (master)
- NetCat Standart 5.4
- HostCMS. Freebie 6.1.6
For paid systems, demo versions were selected, which can be downloaded on the website of the creators.
Testing methodology
In order for the test result to be homogeneous for all products, a virtual machine was created on which a web server, database and other necessary testing software were installed. As a stand were used:
- Apache 2.2.27;
- PHP 5.3.28;
- MySQL Community Server 5.5.38-log;
- PHP modules: Zend Engine v2.3.0 (no opcache, xcache, acp, eaccelerator);
- Apache benchmark Version 2.3.
* UPD: the second php 5.5 test was also conducted with OPCache enabled, as critics requested in the comments - its result is proportional to the main test. The results are shown in the last section.
Each CMS was installed in the virtual environment of the stand, after which several lines were added to its code (most often index.php), which made it possible to save the result about the execution time of the script and the amount of memory used:
$_test_loadstart = microtime(true);
// код CMS
$_test_loadend = microtime(true);
$_test_loadtime = number_format($_test_loadend - $_test_loadstart, 3);
$_test_memory = number_format(memory_get_usage()/1048576, 3);
file_put_contents($_SERVER['HTTP_HOST'] . '.txt', $_test_loadtime . ';' . $_test_memory . "\n", FILE_APPEND);
As a result, the domain.local.txt file was received, which contained data on the script loading time and RAM consumption.
To create requests to the site, ab (Apache benchmark) was used without creating parallel requests - the goal was not testing apache, the goal was to get average results for each CMS:
ab -n 1000 http://domain.local/ > ~/bench/ab/domain.local.txt
Methodology for evaluating the results
In total, the testing technology was evaluated based on the above described technology in the following sections:
- Descriptive rating (Basic) - distribution size, number of files, size and characteristics of the database
- Estimate script download speed (phpload_time)
- Estimated RAM size (phpload_memory)
- Evaluating Apache benchmark test results (apache_benchmark)
In each testing section, for each CMS, grades (-2 <= x <= +2) were set depending on the results obtained (personal) and average (avg) of the entire sample:
- -2 - the worst result in this test
- -1 - the result is worse than the average in this test
- +0.5 - the result is close to the average within standart devination
- +1 - the result is better than the average in this test
- +2 - the best result in this test
Based on the test results in all categories according to all criteria, the resulting table was compiled, which shows the total results for all tests.
* I would also like to dwell on 1 point - UMI.CMS during testing showed outrageous results in download speed, therefore its results were excluded from the calculation of AVG in some testing categories.
Testing
Since we familiarized ourselves with the material and testing methodology, we will proceed directly to the testing itself. Below are the final results for each of the categories of testing.
Test results in a convenient format: Google spreadsheet
Results for PHP 5.5 with OPCache (proportional to the main ones): Google spreadsheet .
Descriptive Assessment
Test / CMS | Bitrix | Wordpress | Drupal | Joomla | Umi.CMS | Ffms | Instantcms | KodiCMS | Netcat | Hostcms | Avg | MIN | MAX |
Installed size (MB) | 191 | 18.5 | 11.7 | 20.1 | 105 | 18.9 | 11.9 | 23,4 | 98.4 | 17.5 | 51.64 | 11.7 | 191 |
File count | 29908 | 1175 | 1074 | 5267 | 7128 | 1682 | 1650 | 3009 | 6391 | 2919 | 6020.3 | 1074 | 29908 |
Mysql table count | 241 | eleven | 74 | 68 | 97 | 22 | 84 | 33 | 155 | 114 | 89.9 | eleven | 241 |
Mysql table rows | 21918 | 126 | 1421 | 461 | 14616 | 187 | 12587 | 46 | 31467 | 20302 | 10313.1 | 46 | 31467 |
Mysql database size (MB) | eleven | 0.56 | 4.6 | 3.9 | 6.2 | 0.356 | 6.8 | 0.948 | 2.7 | 1.9 | 3.8964 | 0.356 | eleven |
SUMMARY POINTS | -9 | 6 | five | 5.5 | -five | 6 | one | 6 | -four | one |
Estimation of script loading speed
Test / CMS | Bitrix | Wordpress | Drupal | Joomla | Umi.CMS | Ffms | Instantcms | KodiCMS | Netcat | Hostcms | Avg | MIN | MAX |
AVG loadtime (sec) | 0.50 | 0.497733 | 0.210623 | 0.325381 | 2.99 | 0,090089 | 0.548239 | 0.213725 | 0.3279 | 0.70523 | 0.38 | 0.09 | 0.71 |
MIN loadtime (sec) | 0.45 | 0.467 | 0.19 | 0,305 | 2.9 | 0,083 | 0.518 | 0.194 | 0,304 | 0.144 | 0.30 | 0.08 | 0.52 |
MAX loadtime (sec) | 0.69 | 0.661 | 0.317 | 0.489 | 3,218 | 0.135 | 0.851 | 1,387 | 2,076 | 0.91 | 0.83 | 0.14 | 2.08 |
SUMMARY POINTS | -one | -one | 3 | 2.5 | -6 | 6 | -2.5 | one | -four | -2 |
Estimated RAM Size
Test / CMS | Bitrix | Wordpress | Drupal | Joomla | Umi.CMS | Ffms | Instantcms | KodiCMS | Netcat | Hostcms | Avg | MIN | MAX |
AVG memory (mb) | 28,944115 | 22,992 | 18,144003 | 12,556 | 23,089791 | 3,181 | 5,953808 | 6,485019 | 13,696 | 10,789239 | 14.5830975 | 3,181 | 28,944115 |
MIN memory (mb) | 28,942 | 22,992 | 18,144 | 12,556 | 22,784 | 3,181 | 5,951 | 6,485 | 13,696 | 4,213 | 13.8944 | 3,181 | 28,942 |
MAX memory (mb) | 29,177 | 22,992 | 18,147 | 12,556 | 23,095 | 3,181 | 6,885 | 6,504 | 13,696 | 10,848 | 14.7081 | 3,181 | 29,177 |
SUMMARY POINTS | -6 | -3 | -3 | 3 | -3 | 6 | 3 | 3 | 3 | -one |
Apache benchmarks score
Test / CMS | Bitrix | Wordpress | Drupal | Joomla | Umi.CMS | Ffms | Instantcms | KodiCMS | Netcat | Hostcms | Avg | MIN | MAX |
Time taken for tests (sec) | 527,433 | 520,498 | 230.91 | 340,219 | 13 032.86 | 97,178 | 555,886 | 224,176 | 343,965 | 727,217 | 396.3868889 | 97,178 | 727,217 |
Complete requests | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 |
Failed requests | 965 | 0 | 0 | 0 | 697 | 0 | 0 | 0 | 0 | 955 | 213,3333333 | 0 | 965 |
Requests per second (# / sec) | 1.9 | 1.92 | 4.33 | 2.94 | 0.08 | 10.29 | 1.8 | 4.46 | 2.91 | 1.38 | 3,547777778 | 1.38 | 10.29 |
Time per request (ms) | 527,433 | 520,498 | 230.91 | 340,219 | 13032,857 | 97,178 | 555,886 | 224,176 | 343,965 | 727,217 | 396.3868889 | 97,178 | 727,217 |
Transfer rate | 69.51 | 14.9 | 32,99 | 28.61 | 2.28 | 105.48 | 92.36 | 13.33 | 31.74 | 23.98 | 45,87777778 | 13.33 | 105.48 |
SUMMARY POINTS | -four | -2 | four | 2 | -12 | ten | 0 | four | 2 | -6 |
Final results
Test / CMS | Bitrix | Wordpress | Drupal | Joomla | Umi.CMS | Ffms | Instantcms | KodiCMS | Netcat | Hostcms |
Basic | -9 | 6 | five | 5.5 | -five | 6 | one | 6 | -four | one |
phpload_time | -one | -one | 3 | 2.5 | -6 | 6 | -2.5 | one | -four | -2 |
phpload_memory | -6 | -3 | -3 | 3 | -3 | 6 | 3 | 3 | 3 | -one |
Apache benchmark | -four | -2 | four | 2 | -12 | ten | 0 | four | 2 | -6 |
SUMMARY RESULT | -20 | 0 | +9 | +13 | -26 | +28 | +1.5 | +14 | -3 | -eight |
Test Results
Now it's time to knock out the results of our testing and briefly evaluate the result. All our results are located from 0 to 2 sides: those that got a negative result I would not recommend anyone to use at all if you value computing resources (this is just my personal opinion - I understand perfectly that the functionality of such systems can be an order of magnitude higher than others).
One of the important test results was the fact that not a single open system received a negative rating , and all commercial products were behind this line (+6 / -4 in favor of open source).
I wouldn’t like to draw other conclusions; I’ll just give you the final graphs of important tests, draw your own conclusions (all pictures are clickable):
Afterword
By no means did I want to offend anyone - I only provided the results of subjective testing of various CMS. You always have the right to decide whether resource consumption is of great importance to you, perhaps more than the convenience of developing / managing a site.
Results in a convenient format: Google spreadsheet
Results for PHP 5.5 with OPCache (proportional to the main ones): Google spreadsheet
Testing in plain-text: Pick from Yandex.Disk
Only registered users can participate in the survey. Please come in.