Performance testing of popular (and not so) CMS

    php script load total time seconds


    Greetings habrazhitel! When developing FFCMS for my projects, I highlighted one of my main goals as fast speed and minimal consumption of equipment resources - however, all the values ​​can be estimated only by comparison, which I will try to do in this article.

    In my last publication, some users asked about the issue of system performance and I decided to test various popular and young CMS as of October 18, 2014. I would like to note - the system was tested immediately after its installation and the data can significantly change in one direction or another, depending on the period of use of the system - increasing the database, number of files and other conditions.

    Foreword


    Before conducting such testing, I wanted to look at the results of similar tests - after all, most likely they have already been carried out before me. However, to my disappointment, I could not find a single comprehensive test that would compare at least 3 or more systems according to many criteria (maybe I was looking badly?). This is surprising, because dozens (or even hundreds) of thousands of webmasters, programmers, and large companies use the popular CMS - is it really that nobody is interested in the issue of consumed resources by the system?

    Test Objects


    For testing, I chose a number of currently popular systems, both commercial and free. 10 systems were selected, their names and versions used are listed below:
    • Bitrix standart - 14.5.0
    • WordPress 4.0
    • Drupal 7.32
    • Joomla! 3.3.6
    • Umi.cms 2.9.7
    • FFCMS 2.0.3
    • InstantCMS 2.1.1
    • KodiCMS 12.20.37 (master)
    • NetCat Standart 5.4
    • HostCMS. Freebie 6.1.6

    For paid systems, demo versions were selected, which can be downloaded on the website of the creators.

    Testing methodology


    In order for the test result to be homogeneous for all products, a virtual machine was created on which a web server, database and other necessary testing software were installed. As a stand were used:
    • Apache 2.2.27;
    • PHP 5.3.28;
    • MySQL Community Server 5.5.38-log;
    • PHP modules: Zend Engine v2.3.0 (no opcache, xcache, acp, eaccelerator);
    • Apache benchmark Version 2.3.

    * UPD: the second php 5.5 test was also conducted with OPCache enabled, as critics requested in the comments - its result is proportional to the main test. The results are shown in the last section.

    Each CMS was installed in the virtual environment of the stand, after which several lines were added to its code (most often index.php), which made it possible to save the result about the execution time of the script and the amount of memory used:

    $_test_loadstart = microtime(true);
    // код CMS
    $_test_loadend = microtime(true);
    $_test_loadtime = number_format($_test_loadend - $_test_loadstart, 3);
    $_test_memory = number_format(memory_get_usage()/1048576, 3);
    file_put_contents($_SERVER['HTTP_HOST'] . '.txt', $_test_loadtime . ';' . $_test_memory . "\n", FILE_APPEND);
    

    As a result, the domain.local.txt file was received, which contained data on the script loading time and RAM consumption.
    To create requests to the site, ab (Apache benchmark) was used without creating parallel requests - the goal was not testing apache, the goal was to get average results for each CMS:

    ab -n 1000 http://domain.local/ > ~/bench/ab/domain.local.txt
    


    Methodology for evaluating the results


    In total, the testing technology was evaluated based on the above described technology in the following sections:
    • Descriptive rating (Basic) - distribution size, number of files, size and characteristics of the database
    • Estimate script download speed (phpload_time)
    • Estimated RAM size (phpload_memory)
    • Evaluating Apache benchmark test results (apache_benchmark)

    In each testing section, for each CMS, grades (-2 <= x <= +2) were set depending on the results obtained (personal) and average (avg) of the entire sample:
    • -2 - the worst result in this test
    • -1 - the result is worse than the average in this test
    • +0.5 - the result is close to the average within standart devination
    • +1 - the result is better than the average in this test
    • +2 - the best result in this test

    Based on the test results in all categories according to all criteria, the resulting table was compiled, which shows the total results for all tests.
    * I would also like to dwell on 1 point - UMI.CMS during testing showed outrageous results in download speed, therefore its results were excluded from the calculation of AVG in some testing categories.

    Testing


    Since we familiarized ourselves with the material and testing methodology, we will proceed directly to the testing itself. Below are the final results for each of the categories of testing.

    Test results in a convenient format: Google spreadsheet
    Results for PHP 5.5 with OPCache (proportional to the main ones): Google spreadsheet .

    Descriptive Assessment


    Test / CMSBitrix
    WordpressDrupalJoomlaUmi.CMSFfmsInstantcmsKodiCMS
    Netcat
    HostcmsAvgMINMAX
    Installed size (MB)19118.511.720.110518.911.923,498.417.551.6411.7191
    File count299081175107452677128168216503009639129196020.3107429908
    Mysql table count241eleven74689722843315511489.9eleven241
    Mysql table rows219181261421461146161871258746314672030210313.14631467
    Mysql database size (MB)eleven0.564.63.96.20.3566.80.9482.71.93.89640.356eleven
    SUMMARY POINTS-96five5.5-five6one6-fourone

    Estimation of script loading speed


    Test / CMSBitrix
    WordpressDrupalJoomlaUmi.CMSFfmsInstantcmsKodiCMS
    Netcat
    HostcmsAvgMINMAX
    AVG loadtime (sec)
    0.500.4977330.2106230.3253812.990,0900890.5482390.2137250.32790.705230.380.090.71
    MIN loadtime (sec)
    0.450.4670.190,3052.90,0830.5180.1940,3040.1440.300.080.52
    MAX loadtime (sec)
    0.690.6610.3170.4893,2180.1350.8511,3872,0760.910.830.142.08
    SUMMARY POINTS-one-one32.5-66-2.5one-four-2


    Estimated RAM Size


    Test / CMSBitrix
    WordpressDrupalJoomlaUmi.CMSFfmsInstantcmsKodiCMS
    Netcat
    HostcmsAvgMINMAX
    AVG memory (mb)
    28,94411522,99218,14400312,55623,0897913,1815,9538086,48501913,69610,78923914.58309753,18128,944115
    MIN memory (mb)28,94222,99218,14412,55622,7843,1815,9516,48513,6964,21313.89443,18128,942
    MAX memory (mb)
    29,17722,99218,14712,55623,0953,1816,8856,50413,69610,84814.70813,18129,177
    SUMMARY POINTS-6-3-33-36333-one


    Apache benchmarks score


    Test / CMSBitrix
    WordpressDrupalJoomlaUmi.CMSFfmsInstantcmsKodiCMS
    Netcat
    HostcmsAvgMINMAX
    Time taken for tests (sec)527,433520,498230.91340,21913 032.8697,178555,886224,176343,965727,217396.386888997,178727,217
    Complete requests1000100010001000100010001000100010001000100010001000
    Failed requests9650006970000955213,33333330965
    Requests per second (# / sec)1.91.924.332.940.0810.291.84.462.911.383,5477777781.3810.29
    Time per request (ms)527,433520,498230.91340,21913032,85797,178555,886224,176343,965727,217396.386888997,178727,217
    Transfer rate69.5114.932,9928.612.28105.4892.3613.3331.7423.9845,8777777813.33105.48
    SUMMARY POINTS-four-2four2-12ten0four2-6


    Final results


    Test / CMSBitrix
    WordpressDrupalJoomlaUmi.CMSFfmsInstantcmsKodiCMS
    Netcat
    Hostcms
    Basic-96five5.5-five6one6-fourone
    phpload_time-one-one32.5-66-2.5one-four-2
    phpload_memory-6-3-33-36333-one
    Apache benchmark-four-2four2-12ten0four2-6
    SUMMARY RESULT
    -200+9+13-26+28+1.5+14-3-eight


    Test Results


    Now it's time to knock out the results of our testing and briefly evaluate the result. All our results are located from 0 to 2 sides: those that got a negative result I would not recommend anyone to use at all if you value computing resources (this is just my personal opinion - I understand perfectly that the functionality of such systems can be an order of magnitude higher than others).
    One of the important test results was the fact that not a single open system received a negative rating , and all commercial products were behind this line (+6 / -4 in favor of open source).
    I wouldn’t like to draw other conclusions; I’ll just give you the final graphs of important tests, draw your own conclusions (all pictures are clickable):

    load memory avg graphload time avg graphsummary statistic graph

    Afterword


    By no means did I want to offend anyone - I only provided the results of subjective testing of various CMS. You always have the right to decide whether resource consumption is of great importance to you, perhaps more than the convenience of developing / managing a site.

    Results in a convenient format: Google spreadsheet
    Results for PHP 5.5 with OPCache (proportional to the main ones): Google spreadsheet
    Testing in plain-text: Pick from Yandex.Disk

    Only registered users can participate in the survey. Please come in.

    Is it important for you to consume resources on the Web?


    Also popular now: