April 21, 2014 at 14:13Migrating to PHP 5.5 and Unit Tests
4 years have passed since the transition from PHP 4.4 to PHP 5.3 in Badoo, it's time to update PHP, this time immediately to the version of PHP 5.5. In addition to new features, the new version of PHP once again brought us a significant increase in productivity, so we had many reasons for the upgrade. In this article, we will talk about how we switched to PHP 5.5, which rakes we collected, and why we rewrote our system once again to run unit tests based on PHPUnit.
Figure 1. General architecture
Last time we switched from the fourth version of PHP to the fifth, and our version of PHP 5.3 contained patches so that the "old" PHP syntax worked, for example
The main problems that we have encountered:
After switching to PHP 5.5, our unit tests began to take much longer (several times), so we decided once again to refine our “launcher” to solve this problem.
Using the xhprof extension, we quickly found out that our tests were “slowing down” because the runkit extension had a significant drop in performance, so we started looking for a reason. As a result, it turned out that the problem, apparently, was the addition of a “mystical” runtime cache for PHP 5.4, which must be reset every time after calling the “runkit _ * _ redefine” function.
The runkit extension for each call goes through all registered classes, methods and functions and flushes this cache. We naively tried to disable this, but after that PHP started to crash, so we had to look for another solution.
Prior to the transition to PHP 5.5, we already had a unit test launcher in the form of an add-on over phpunit, which divided one large suite of unit tests into several smaller ones: at that time we used to run tests in 11 threads (Ilya Kudinov already talked about this on conferences, including the Badoo LoveQA conference: www.youtube.com/watch?v=gAisPsfbLkg ).
We carried out several simple benchmarks and found out that the tests pass several times faster if we divide our suite not into 11 parts, as it was before, but into 128 or more (with a fixed number of processor cores). In each suite, only about 10-15 files were obtained, so we called this concept "microsuits." We got about 150 such microsets, and each of them suspiciously well suited to be a “task” for some script (the task consists of a list of files for the corresponding suite, and it, in turn, starts phpunit with the appropriate parameters )
It so happened that the author of this article has nothing to do with QA, but was one of the main developers of the “new scripting framework”, which, in fact, is a “cloud” for scripts and supports the concept of tasks (we also repeatedly talk about our cloud told at conferences and we will surely tell about it in more detail on Habré). And since we have tasks in the form of file lists for each phpunit suite, it means that they can also be “thrust into the cloud,” which we decided to do. The idea is very simple: since we have many small tasks, they can be run on several servers independently of each other, which should further accelerate the passage of tests.
We run tests from several different sources:
All these types of test runs are united by the fact that you first need to “pull” (fetch) the branch from some source, and then run the test run on this branch.
This fact determined the architecture of our new “cloud” test launcher (Fig. 1, at the beginning of the article):
First, one task is created for the master process, which:
Fig 2. Available directories in MySQL
After this, the master process analyzes the original phpunit suite for running the tests and divides it into the required number of parts (no more than 10 files per microsuite). The resulting tasks (thread processes) are added as tasks to the cloud and begin execution on the servers available for execution.
The first task, which gets to the new server, prepares the selected directory for running the tests, taking the necessary commit from the server on which the master process is running. To ensure that all other tasks that came to the corresponding server do not do the same at the same time as the first task, file locks are used (Fig. 3).
Several test runs can go at the same time for a more complete utilization of the resources of our cluster: tests pass quickly, and the preparation of the source texts takes a significant part of the time, rather than directly executing the code.
Fig. 3. Locks when preparing a directory.
Some tests can go much longer than others, and we have statistics on the time taken for each test, so we use this information to run “long” tests in the first place. Such a strategy makes it possible to achieve a more uniform loading of servers during the test execution, as well as reduce the total time for their execution (Fig. 4).
Figure 4. Tracking the execution time of the tests
“In good weather”, our entire suite of 28,000 unit tests takes 1 minute, so tests that take longer become a “bottleneck”, and our system posts the authors of the corresponding tests on a “shame board”, which is shown on every test run. In addition, if there are few tests left, a list of those who are left is shown (Fig. 5).
Figure 5. Board of shame: a list of tests that take more than a minute
to run. Unit test launch itself was the first script to be transferred to the cloud. It helped to eliminate many bugs and shortcomings in the cloud itself, at the same time significantly accelerating the passage of unit tests.
After switching to PHP 5.5, we were able to use new language features, significantly reduced the CPU consumption on our servers (by an average of 25%), and also transferred our unit tests to the cloud. The latter allowed us to reduce the total test time from 5-6 minutes (in PHP 5.5 — tens of minutes) to one minute, at the same time moving the load from a shared dev server to the cloud.
Yuri youROCK Nasretdinov, Badoo developer
Figure 1. General architecture
"Rake" when switching from PHP 5.3 to PHP 5.5
Last time we switched from the fourth version of PHP to the fifth, and our version of PHP 5.3 contained patches so that the "old" PHP syntax worked, for example
$a = &new ClassName();, and so that our code base could work on PHP4 and PHP5 at the same time. This time we didn’t have such restrictions, so during the transition we just found and replaced all the obsolete constructions with more relevant ones, and this was the end of the code rewriting. The main problems that we have encountered:
- part of the deprecated features of the language has been removed;
- mysql extension has become deprecated ;
- low performance of the runkit extension, which we use when writing unit tests.
After switching to PHP 5.5, our unit tests began to take much longer (several times), so we decided once again to refine our “launcher” to solve this problem.
Runkit and PHP 5.4+
Using the xhprof extension, we quickly found out that our tests were “slowing down” because the runkit extension had a significant drop in performance, so we started looking for a reason. As a result, it turned out that the problem, apparently, was the addition of a “mystical” runtime cache for PHP 5.4, which must be reset every time after calling the “runkit _ * _ redefine” function.
The runkit extension for each call goes through all registered classes, methods and functions and flushes this cache. We naively tried to disable this, but after that PHP started to crash, so we had to look for another solution.
The concept of "microsites" (microsuite)
Prior to the transition to PHP 5.5, we already had a unit test launcher in the form of an add-on over phpunit, which divided one large suite of unit tests into several smaller ones: at that time we used to run tests in 11 threads (Ilya Kudinov already talked about this on conferences, including the Badoo LoveQA conference: www.youtube.com/watch?v=gAisPsfbLkg ).
We carried out several simple benchmarks and found out that the tests pass several times faster if we divide our suite not into 11 parts, as it was before, but into 128 or more (with a fixed number of processor cores). In each suite, only about 10-15 files were obtained, so we called this concept "microsuits." We got about 150 such microsets, and each of them suspiciously well suited to be a “task” for some script (the task consists of a list of files for the corresponding suite, and it, in turn, starts phpunit with the appropriate parameters )
Tests in the cloud
It so happened that the author of this article has nothing to do with QA, but was one of the main developers of the “new scripting framework”, which, in fact, is a “cloud” for scripts and supports the concept of tasks (we also repeatedly talk about our cloud told at conferences and we will surely tell about it in more detail on Habré). And since we have tasks in the form of file lists for each phpunit suite, it means that they can also be “thrust into the cloud,” which we decided to do. The idea is very simple: since we have many small tasks, they can be run on several servers independently of each other, which should further accelerate the passage of tests.
General architecture
We run tests from several different sources:
- automatic test runs using AIDA :
- on the branch (git branch) of the task;
- according to the "build" - the code that will leave for production;
- on the master branch; - “Manual” test runs initiated by developers or testers from the dev server.
All these types of test runs are united by the fact that you first need to “pull” (fetch) the branch from some source, and then run the test run on this branch.
This fact determined the architecture of our new “cloud” test launcher (Fig. 1, at the beginning of the article):
First, one task is created for the master process, which:
- selects an available directory in the database (Fig. 2);
- downloads the git branch from the right place (shared repository or dev server);
- (optional) does git merge master;
- (optional) creates a new commit with all local changes.
Fig 2. Available directories in MySQL
After this, the master process analyzes the original phpunit suite for running the tests and divides it into the required number of parts (no more than 10 files per microsuite). The resulting tasks (thread processes) are added as tasks to the cloud and begin execution on the servers available for execution.
The first task, which gets to the new server, prepares the selected directory for running the tests, taking the necessary commit from the server on which the master process is running. To ensure that all other tasks that came to the corresponding server do not do the same at the same time as the first task, file locks are used (Fig. 3).
Several test runs can go at the same time for a more complete utilization of the resources of our cluster: tests pass quickly, and the preparation of the source texts takes a significant part of the time, rather than directly executing the code.
Fig. 3. Locks when preparing a directory.
Some tests can go much longer than others, and we have statistics on the time taken for each test, so we use this information to run “long” tests in the first place. Such a strategy makes it possible to achieve a more uniform loading of servers during the test execution, as well as reduce the total time for their execution (Fig. 4).
Figure 4. Tracking the execution time of the tests
“In good weather”, our entire suite of 28,000 unit tests takes 1 minute, so tests that take longer become a “bottleneck”, and our system posts the authors of the corresponding tests on a “shame board”, which is shown on every test run. In addition, if there are few tests left, a list of those who are left is shown (Fig. 5).
Figure 5. Board of shame: a list of tests that take more than a minute
to run. Unit test launch itself was the first script to be transferred to the cloud. It helped to eliminate many bugs and shortcomings in the cloud itself, at the same time significantly accelerating the passage of unit tests.
results
After switching to PHP 5.5, we were able to use new language features, significantly reduced the CPU consumption on our servers (by an average of 25%), and also transferred our unit tests to the cloud. The latter allowed us to reduce the total test time from 5-6 minutes (in PHP 5.5 — tens of minutes) to one minute, at the same time moving the load from a shared dev server to the cloud.
Yuri youROCK Nasretdinov, Badoo developer