Summaries of the report “Monolith for hundreds of client versions” (HL2018, Badoo, Vladimir Yants)

I continue a series of abstracts with HL2018. The guys from Badoo (Vladimir Janz vyants and Nikolai Krapivny) helped me in checking this summary , for which I thank them very much. I hope this has a positive effect on the quality of the message conveyed.

Features of the development process:

Responsibility of the developer does not end with the release of the backend. It responds prior to implementation on the platforms.

There is manual testing, but the client is not ready at the time of release and comes with a (unpredictable) delay. We most often do not know when customers will begin to implement it. Sometimes (not often) features start to do after a long time. Therefore, testing with your hands is hard and not everything is possible. Therefore, we need autotests.

Tests

Unit tests

Written on phpunit.

Test a small unit. They do not go to the base or to the services (they should not interact with anything).

Legacy is still there and complicates the testing process.

Developed softMocks library - intercepts all include / require and substitutes for changed.

You can wind any methods: static, private, final.
The library is available in open source.

Problem: softmocks relax and allow you to write untested code (and all should be covered with tests).

Have accepted the rules:

New code should be easy to test phpunit
SoftMocks - extreme case (old code / long / expensive / difficult)

These rules look at the code review.

Quality tests

Mutation Testing

Take the code
We take code coverage
Parsing the code and applying mutations (change + => -; true => false and so on)
For each mutation, we run the suites (set) of tests.
If the tests have fallen, then ok. If not, they are not effective enough. Understand, change / append tests.

There are ready-made solutions (Humbug, Infection), but they did not fit (not compatible with softmocks, there are difficulties with code coverage). Therefore, they wrote their own.

Mutation testing is not yet available for a manual test. Available to run manually from the console. Now we introduce in CI pipeline, we build process. The result will be on Habré.

Integration tests

We test the work of several components in a bundle; We check the work with the database and / or services.

Standard approach to database testing (DBUnit):

Raise the test database
Fill it
Run the test
Clearing the database

Problems:

It is necessary to support datatables and dataset (the relevance of the contents of the database)
It takes time to prepare the database
Parallel launches make tests unstable and spawn deadlocks

Solution: DBMocks library (your own solution)

How it works:

DB driver methods are intercepted using SoftMocks on the setUp test
From query parsim db + table
In tmpfs, temporary tables are created with the same schema.
All queries go to temporary tables only.
On TearDown, they are deleted.

The library is small, but not yet launched in open source

Results:

Tests can not damage the data in the original tables.
Tests are isolated from each other (can be run in parallel)
Compatibility of the query with the MySQL version is being tested.

API tests

Simulate client session
They are able to send requests to the backend
Backend responds almost like a real customer

Typically, such tests require an authorized user. It must be created before the test and removed after. This brings additional risks (replication, background tasks).

Solution: Made a pool of test users. Learned to clean them.

Test users are in the same real environment, because devel! = Prod. It is necessary to isolate test and live users.

For isolation, the user added the is_test_user flag. And these users are also excluded from the analytics and the results of a / b tests.

You can do it cheaper - send test users “to Antarctica”, where no one will see them (except penguins).

QA API

A tool for preparing the environment in api tests, essentially a backdoor in the backend for quickly changing user / environment settings.

Well documented api methods
Manage data quickly and easily
Write backend developers
Can only be applied to test users.

Allows you to change the user immutable data (for example, the date of registration).

Protection required:

At the network level (available only from the office network)
With each request, a secret is transmitted, the validity of which is checked.
Methods work only with test users.

There is a BugsBounty program on HackerOne. Pay for found vulnerabilities. One cant with QA API found with its help.

Remote mocks

Moki for remote backend.

Work on the base of soft mocks. The test asks the backend to initialize for the mock session. Upon receiving a request, the backend checks the list of mocks for the session and applies them using SoftMocks.

Test example:

API tests are too convenient. There is a temptation to write them instead of Unit. But API tests are much slower.

Adopted a set of rules:

The purpose of the API tests is to test the protocol and the correctness of the integration.
It is permissible to check complex flow.
Cannot test small variability
We check the tests for code review too.

Ui tests

Do not write backend command.

The feature is covered with Ui tests when it stabilizes.
Used selenium for the web. For mobile calabash.

Test run

100,000 unit tests. 6,000 integration tests, 14,000 api tests.
In 1 flow time 40 min / 90 min / 10 hours.

Have made TestCloud - a cloud for start of tests.

Distribution of the test between threads:

It is possible equally (badly, all tests are different, it turns out unequal in time parts)
Run multiple threads and consistently feed phpunit tests one at a time (initialization overhead. Long!)

Decision:

Collect statistics on run time.
Build tests so that the chunk is run for no more than 30 seconds

The problem with api tests is a long time, a lot of resources and they do not allow others to execute.

To solve this, the cloud was divided into 2 parts:

Runs quick tests only.
Runs both types of tests.

The result is an acceleration of time to:

Unit - 1 min
Integration - 5 min
API - 15 minutes.

Code coverage run

What tests to perform? Show code coverage.

Get branch diff
Create a list of changed files
Get the list of tests for these files.
Run the suite from these tests only.

Coverage is formed once a day, at night, for a master branch. Results (diff) add to the database.

Pros:

We run fewer tests: less load on iron and feedback from tests faster
You can run tests for patches. This allows you to quickly roll out the hotfix. In patches, speed is most important.

Minuses:

Release backend 2 times a day. After lunch coverage is less relevant (but when rolling out the beat, we always drive a full suite).
Api tests generate extensive coverage. For them, this approach does not give much savings.

If the developer needs to see the code-coverage immediately, that is, the tool that can be run in the console and immediately get a fresh metric on the coverege of a specific file / component.
It is considered cunning: the data on the coverege of the master is taken, all the modified tests are added, a small suite is obtained for which the coverage is already considered.

Results

Need all levels of tests
Quantity! = Quality. Do code review and mutation testing
Isolate test users from real users.
Backdoors in the backend simplify and speed up writing tests
Collect test statistics.