Summaries of the report “Monolith for hundreds of client versions” (HL2018, Badoo, Vladimir Yants)

    I continue a series of abstracts with HL2018. The guys from Badoo (Vladimir Janz vyants and Nikolai Krapivny) helped me in checking this summary , for which I thank them very much. I hope this has a positive effect on the quality of the message conveyed.


    Features of the development process:

    Responsibility of the developer does not end with the release of the backend. It responds prior to implementation on the platforms.


    There is manual testing, but the client is not ready at the time of release and comes with a (unpredictable) delay. We most often do not know when customers will begin to implement it. Sometimes (not often) features start to do after a long time. Therefore, testing with your hands is hard and not everything is possible. Therefore, we need autotests.


    Unit tests

    Written on phpunit.

    Test a small unit. They do not go to the base or to the services (they should not interact with anything).

    Legacy is still there and complicates the testing process.

    Developed softMocks library - intercepts all include / require and substitutes for changed.

    You can wind any methods: static, private, final.
    The library is available in open source.

    Problem: softmocks relax and allow you to write untested code (and all should be covered with tests).

    Have accepted the rules:

    • New code should be easy to test phpunit
    • SoftMocks - extreme case (old code / long / expensive / difficult)

    These rules look at the code review.

    Quality tests

    Mutation Testing

    • Take the code
    • We take code coverage
    • Parsing the code and applying mutations (change + => -; true => false and so on)
    • For each mutation, we run the suites (set) of tests.
    • If the tests have fallen, then ok. If not, they are not effective enough. Understand, change / append tests.

    There are ready-made solutions (Humbug, Infection), but they did not fit (not compatible with softmocks, there are difficulties with code coverage). Therefore, they wrote their own.

    Mutation testing is not yet available for a manual test. Available to run manually from the console. Now we introduce in CI pipeline, we build process. The result will be on Habré.

    Integration tests

    We test the work of several components in a bundle; We check the work with the database and / or services.

    Standard approach to database testing (DBUnit):

    1. Raise the test database
    2. Fill it
    3. Run the test
    4. Clearing the database


    • It is necessary to support datatables and dataset (the relevance of the contents of the database)
    • It takes time to prepare the database
    • Parallel launches make tests unstable and spawn deadlocks

    Solution: DBMocks library (your own solution)

    How it works:

    • DB driver methods are intercepted using SoftMocks on the setUp test
    • From query parsim db + table
    • In tmpfs, temporary tables are created with the same schema.
    • All queries go to temporary tables only.
    • On TearDown, they are deleted.

    The library is small, but not yet launched in open source


    • Tests can not damage the data in the original tables.
    • Tests are isolated from each other (can be run in parallel)
    • Compatibility of the query with the MySQL version is being tested.

    API tests

    • Simulate client session
    • They are able to send requests to the backend
    • Backend responds almost like a real customer

    Typically, such tests require an authorized user. It must be created before the test and removed after. This brings additional risks (replication, background tasks).

    Solution: Made a pool of test users. Learned to clean them.


    Test users are in the same real environment, because devel! = Prod. It is necessary to isolate test and live users.

    For isolation, the user added the is_test_user flag. And these users are also excluded from the analytics and the results of a / b tests.

    You can do it cheaper - send test users “to Antarctica”, where no one will see them (except penguins).

    QA API

    A tool for preparing the environment in api tests, essentially a backdoor in the backend for quickly changing user / environment settings.

    • Well documented api methods
    • Manage data quickly and easily
    • Write backend developers
    • Can only be applied to test users.

    Allows you to change the user immutable data (for example, the date of registration).

    Protection required:

    • At the network level (available only from the office network)
    • With each request, a secret is transmitted, the validity of which is checked.
    • Methods work only with test users.

    There is a BugsBounty program on HackerOne. Pay for found vulnerabilities. One cant with QA API found with its help.

    Remote mocks

    Moki for remote backend.

    Work on the base of soft mocks. The test asks the backend to initialize for the mock session. Upon receiving a request, the backend checks the list of mocks for the session and applies them using SoftMocks.

    Test example:


    API tests are too convenient. There is a temptation to write them instead of Unit. But API tests are much slower.

    Adopted a set of rules:

    • The purpose of the API tests is to test the protocol and the correctness of the integration.
    • It is permissible to check complex flow.
    • Cannot test small variability
    • We check the tests for code review too.

    Ui tests

    Do not write backend command.

    The feature is covered with Ui tests when it stabilizes.
    Used selenium for the web. For mobile calabash.

    Test run

    100,000 unit tests. 6,000 integration tests, 14,000 api tests.
    In 1 flow time 40 min / 90 min / 10 hours.

    Have made TestCloud - a cloud for start of tests.


    Distribution of the test between threads:

    • It is possible equally (badly, all tests are different, it turns out unequal in time parts)
    • Run multiple threads and consistently feed phpunit tests one at a time (initialization overhead. Long!)


    • Collect statistics on run time.
    • Build tests so that the chunk is run for no more than 30 seconds

    The problem with api tests is a long time, a lot of resources and they do not allow others to execute.

    To solve this, the cloud was divided into 2 parts:

    1. Runs quick tests only.
    2. Runs both types of tests.

    The result is an acceleration of time to:

    • Unit - 1 min
    • Integration - 5 min
    • API - 15 minutes.

    Code coverage run

    What tests to perform? Show code coverage.

    1. Get branch diff
    2. Create a list of changed files
    3. Get the list of tests for these files.
    4. Run the suite from these tests only.

    Coverage is formed once a day, at night, for a master branch. Results (diff) add to the database.


    • We run fewer tests: less load on iron and feedback from tests faster
    • You can run tests for patches. This allows you to quickly roll out the hotfix. In patches, speed is most important.


    • Release backend 2 times a day. After lunch coverage is less relevant (but when rolling out the beat, we always drive a full suite).
    • Api tests generate extensive coverage. For them, this approach does not give much savings.

    If the developer needs to see the code-coverage immediately, that is, the tool that can be run in the console and immediately get a fresh metric on the coverege of a specific file / component.
    It is considered cunning: the data on the coverege of the master is taken, all the modified tests are added, a small suite is obtained for which the coverage is already considered.


    • Need all levels of tests
    • Quantity! = Quality. Do code review and mutation testing
    • Isolate test users from real users.
    • Backdoors in the backend simplify and speed up writing tests
    • Collect test statistics.


    Also popular now: