How to choose the same PHP framework. Benchmarking

When developing any software product, the team of developers first of all is faced with the task of correctly choosing a software platform that determines the structure of the software system.

To do this, you need to take into account a sufficiently large number of characteristics, from "how quickly everything will work" to "but do we need this feature?" And so every time. It is in moments of brainstorming that the team compares the convenience of the framework, speed, a set of features that are implemented in it or in modules compatible with it.

But which is still better, faster and more productive?

Developers are constantly comparing frameworks to clarify this issue for themselves. For example, in an articleLukasz Kujawa provides a comparison of PHP frameworks. One “but” is an article for 2013. But time goes by ... Therefore, we decided to conduct our own, actual comparison of frameworks.

To evaluate the performance, the PHP Framework Benchmark was used . He offers for comparison a lot of frameworks (not only those mentioned above), but the author is in no hurry to add new versions of projects to the repository, which, of course, is sad, although not fatal. If you want to add a new version is not difficult. 


One of the main goals of this article is also an attempt to find practical improvements in the productivity and efficiency of new versions of PHP. Therefore, testing was conducted on PHP 5.6 / 7.0 / 7.1

What will we compare?

For comparison, the following frameworks were selected:

• slim-3.0
• ci-3.0
• lumen-5.1
• yii-2.0
• silex-1.3
• fuel-1.8
• phpixie-3.2
• zf-2.5
• zf-3.0
• symfony-2.7
• symfony-3.0
• laravel-5.3
• laravel-5.4
• bluz (version 7.0.0 - for PHP5.6 and version 7.4 for PHP7.0 and higher)
• ze-1.0
• phalcon-3.0

Testing is conditionally divided into 4 types:

• throughput
• occupied memory (memory),
• runtime (exec time),
• number of included files.

Testing technique and test bench

The machine on which the testing was carried out has the following characteristics: We enter the command and the frame is already on our machine. Since we used Mint, you need to configure: 
 A little about the structure of the php-framework-benchmark itself: / benchmarks - contains bash scripts responsible for collecting information about the number of requests per second (using the ab utility), the amount of information, how much time was spent and how many files were called from the “start point” file. / lib - the directory in which the files are located that are responsible for processing the information received after displaying the “Hello world” page, displaying tables with results and plotting diagrams. / output

Operation system: Linux Mint 17 Cinnamon 64-bit
Cinnamin Version 2.2.16
Linux Kernel: 3.13.0-24-generic
Processor: Intel Core i3-4160 CPU 3.60 Ghz X 2
Memory: 8 GB

Server version: Apache/2.4.7 (ubuntu)
Server build: Jul 15 2016
php 7.1 / php7.0 / php5.6

git clone https://github.com/kenjis/php-framework-benchmark

# Added
net.netfilter.nf_conntrack_max = 100000
net.nf_conntrack_max = 100000
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

sudo sysctl -p







- directory into which logs are added after testing. There are two files for each tested file: .ab.log - the log after running the ab utility, and .output - contains information that was displayed on the screen (usually it is hello world and data on memory, runtime, used files).

The rest of the folders are frame blanks in which one controller has already been added, which will return the string “hello world” when accessing by URI compiled according to the rules for accessing this framework.

To run the test, you first need to configure the frameworks. Let's consider two approaches.

Commandbash setup.shconfigure those frameworks that are described in the list.sh file. You can edit it: add and delete folders for testing. That is, configure as you need.

With a command, bash setup.sh fatfree-3.5/ slim-3.0/ lumen-5.1/ silex-1.3/you can install some separate frameworks by setting them to the command parameters. In some cases, this is convenient, but we used the first approach.

After setting up the frameworks, we started testing with bash benchmark.sh.

At the end of the work, a table appeared in the terminal with a list of tested frameworks, the number of requests per second, the relative value occupied by the memory, as well as the relative values ​​of these indicators.

To display the graphs, we used the link http: // localhost / php-framework-benchmark / .

As you understand, it was necessary to configure Apache and make it look in the frame folder. All of this is described in readme, so no questions arise.

Framework Testing Results

Each section has a structure consisting of two forms of presentation of the results.

The first form is a visual representation type. Each characteristic contains 4 diagrams. Each diagram displays a comparison of the frameworks among themselves, plus a cumulative diagram. It was built using a specific version of PHP. This way you can trace the evolution of improvements in PHP and frameworks.

The second form is the test result in the form of a table (to be clear, let's talk seriously - give me more numbers!).

Performance (throughput)

In relation to our situation, the throughput characteristic is measured in the number of requests that our framework can process within a second. Therefore, the higher this number, the more productive our application is, since it can correctly process requests from a large number of users.

We got the following results (queries per second):

	php 5.6	php 7.0	php 7.1
phalcon-3.1.2	5058.00	5130.00	7535.00
ci-3.0	2943.55	4116.31	4998.05
slim-3.0	2074.59	3143.94	3681.00
yii-2.0	1256.31	2276.37	2664.61
silex-1.3	1401.92	2263.90	2576.22
lumen-5.1	1316.46	2384.24	2741.81
ze-1.0	1181.14	1989.99	1741.81
phpixie-3.2	898.63	1677.15	1896.23
fuel-1.8	1044.77	1646.67	1770.13
bluz-7.3.1	- *	1774.00	1890.00
zf-2.5	198.66	623.71	739.12
zf-3.0	447.88	1012.57	1197.26
symfony-2.7	360.03	873.40	989.57
symfony-3.0	372.19	853.51	1022.28
laravel-5.3	258.62	346.25	625.99
laravel-5.4	219.82	413.49	600.42

* - bluz-7.3.1 does not support php 5.6.

For clarity, we built graphs for each version of PHP:

PHP5.6:


PHP7.0:


PHP7.1:


Summary cumulative diagram (based on frameworks):

Occupied memory (peak memory)

This characteristic (in megabytes) is responsible for the amount of memory occupied by the framework when performing the task assigned to it. The lower this number, the better for us and for the server:

	php 5.6	php 7.0	php 7.1
phalcon-3.1.2	0.27	0.38	0.37
ci-3.0	0.42	0.38	0.38
slim-3.0	0.61	0.55	0.55
yii-2.0	1.31	0.91	0.91
silex-1.3	0.74	0.65	0.65
lumen-5.1	0.80	0.63	0.63
ze-1.0	0.79	0.56	0.56
phpixie-3.2	1.22	0.82	0.82
fuel-1.8	0.7	0.6	0.6
bluz-7.3.1	- *	0.69	0.69
zf-2.5	3.06	1.34	1.34
zf-3.0	2.12	1.09	1.08
symfony-2.7	3.11	1.41	1.42
symfony-3.0	2.86	1.30	1.32
laravel-5.3	2.91	2.04	2.04
laravel-5.4	3.04	1.45	1.49

* - bluz-7.3.1 does not support php 5.6

PHP 5.6:


PHP 7.0:


PHP 7.1:


Summary cumulative diagram (based on frameworks):

lead time

Execution time - the time taken by the system to complete the task. It is measured from the beginning of the task to the output of the result by the system.

We examined how many requests per second the framework can process, how much memory it takes. Now consider how much we need to expect to receive a response from the server. The lower this value, the better for us , and for the nervous system of the client of our application.

Time is given in milliseconds (ms):

	php 5.6	php 7.0	php 7.1
phalcon-3.1.2	1.300	1.470	1.080
ci-3.0	0.996	0.818	1.007
slim-3.0	1.530	1.228	0.662
yii-2.0	1.478	1.410	1.639
silex-1.3	4.657	1.625	2.681
lumen-5.1	2.121	1.829	1.228
ze-1.0	2.629	2.069	1.528
phpixie-3.2	9.329	4.757	1.911
fuel-1.8	3.283	2.684	1.425
bluz-7.3.1	- *	1.619	1.921
zf-2.5	22.042	5.011	3.998
zf-3.0	12.680	2.506	2.989
symfony-2.7	6.529	3.902	2.384
symfony-3.0	9.335	3.987	2.820
laravel-5.3	19.885	4.840	2.622
laravel-5.4	19.561	4.758	3.940

PHP 5.6:


PHP 7.0:


PHP 7.1:


Summary cumulative diagram (based on frameworks):

Connected files

The characteristic that is responsible for the number of connected files, which are described in the file “entry points” of the framework. It is clear that the system spends some time searching and connecting. Therefore, the fewer the files, the faster the first launch of the application will be carried out, since usually the next time the framework works with the cache, which speeds up the work:

phalcon-3.1.2	5
ci-3.0	26
slim-3.0	53
yii-2.0	46
silex-1.3	63
lumen-5.1	37
ze-1.0	68
phpixie-3.2	163
fuel-1.8	53
bluz-7.3.1	95
zf-2.5	222
zf-3.0	188
symfony-2.7	110
symfony-3.0	192
laravel-5.3	38
laravel-5.4	176

The difference in the number of connected files between Laravel 5.3 and Laravel 5.4 may seem strange and give rise to discussions, disputes, etc. We hasten to clarify the situation. As you know, using the command

php artisan optimize --force

in Laravel 5.3, you can generate the compiled.php file, and thereby reduce the number of included files by collecting them into one. But there is one “but”: there are no more commands for generating this file in Laravel 5.4. The developer decided to remove this feature, as he calculated ( https://github.com/laravel/framework/pull/17003 ) that it is better to use opcache to tune performance.

Is it worth updating?

The summary data on the versions more than clearly shows what will increase the productivity and efficiency of resource use during the transition (or initial choice) to the new version of PHP.

When switching from PHP 5.6 to PHP 7.0, the average performance increase was almost + 90%, while the minimum performance increase was + 33% for Laravel 5.3, and the maximum was> 200% for Zend Framework 2.5.

The transition from version 7.0 to 7.1 is not so shocking, but on average it gives an almost 20% increase in performance.

Having summed up all the obtained data on the performance of various versions of PHP, we get these "mattresses":

Fun fact: Laravel 5.3 showed the smallest performance gain when migrating from PHP 5.6 to PHP 7.0, but the largest increase when migrating from version 7.0 to version 7.1, and as a result, the performance of Laravel 5.3 and 5.4 to PHP 7.1 is almost the same.

Memory consumption has also been optimized, so switching from PHP 5.6 to PHP 7.0 will allow your application to consume 30% less memory.

Upgrading from version 7.0 to version 7.1 practically does not give a gain, and in the last Symfony and Laravel we completely go to the "minus", because they start to eat a little more.

It remains to look at the runtime, and yes, everything is fine here too:

• moving from PHP 5.6 to PHP 7.0 will give you an average acceleration of 44%.
• moving from PHP 7.0 to PHP 7.1 will give you another 14% speedup.

Note. Testing with ab - what we faced

“What about slim and phpixie” - this question prompted an investigation into the behavior of the ab utility when interacting with these frameworks.

Let's run the test separately for Slim-3.0: Something is wrong - the number of requests per second is only 0.4 (!) It was a Keep Alive connection, more details can be found here.

ab -c 10 -t 3 http://localhost/php-framework-benchmark/slim-3.0/index.php/hello/index

Concurrency Level: 10
Time taken for tests: 5.005 seconds
Complete requests: 2
Failed requests: 0
Total transferred: 1800 bytes
HTML transferred: 330 bytes
Requests per second: 0.40 [#/sec] (mean)
Time per request: 25024.485 [ms] (mean)
Time per request: 2502.448 [ms] (mean, across all concurrent requests)
Transfer rate: 0.35 [Kbytes/sec] received



ab -c 10 -t 3 http://localhost/php-framework-benchmark/laravel-5.4/public/index.php/hello/index

Concurrency Level: 10
Time taken for tests: 3.004 seconds
Complete requests: 1961
Failed requests: 0
Total transferred: 1995682 bytes
HTML transferred: 66708 bytes
Requests per second: 652.86 [#/sec] (mean)
Time per request: 15.317 [ms] (mean)
Time per request: 1.532 [ms] (mean, across all concurrent requests)
Transfer rate: 648.83 [Kbytes/sec] received

“When you make requests with“ Connection: keep-alive ”the subsequent request to the server will use the same TCP connection. This is called HTTP persistent connection. This helps in reduction CPU load on server side and improves latency / response time.

If a request is made with "Connection: close" this indicates that once the request has been made the server needs to close the connection. And so for each request a new TCP connection will be established.

By default HTTP 1.1 client / server uses keep-alive where as HTTP 1.0 client / server don't support keep-alive by default. ”

So the test for Slim should look like this:

ab -H 'Connection: close' -c 10 -t 3 http://localhost/php-framework-benchmark/slim-3.0/index.php/hello/index

Concurrency Level: 10
Time taken for tests: 3.000 seconds
Complete requests: 10709
Failed requests: 0
Total transferred: 2131091 bytes
HTML transferred: 353397 bytes
Requests per second: 3569.53 [#/sec] (mean)
Time per request: 2.801 [ms] (mean)
Time per request: 0.280 [ms] (mean, across all concurrent requests)
Transfer rate: 693.69 [Kbytes/sec] received

Conclusion

As you would expect, the undisputed leader in productivity (but not development speed) is Phalcon. The second place - and in fact the first among the PHP frameworks (and not C, on which the source code of Phalcon is written) - is occupied by CodeIgniter 3!

Of course, do not forget that each instrument has its own purpose. If you choose a small and lightweight framework and are going to write something different on it from the simplest applications or the REST API, then most likely you will encounter problems when expanding the functionality. And vice versa - the redundancy of full-featured, large frameworks will entail financial costs for hosting even for elementary applications under heavy load.

This testing was carried out in order to convince / tell / strengthen the position of the PHP language versions 7.0 and 7.1 in your mind and in future projects, to convey information that productivity has really increased.

Refactoring the internal data structures and adding an extra step before compiling the code into an abstract syntax tree - Abstract Syntax Tree (AST) - resulted in superior performance and more efficient memory allocation. The results themselves look promising: tests performed on real applications show that PHP 7 is on average twice as fast as PHP 5.6 and also uses 50% less memory during query processing, which makes PHP 7 a strong contender for the HHVM JIT compiler from Facebook

Tests fully confirm both the twice-accelerated request processing in PHP7 and the reduced amount of memory used.