Mutational analysis, or how to test tests

Tests do not happen much - everyone knows that. Memes about unit and integration testing are no longer very fun. And we still do not know whether it is possible to rely on the results of passing tests, and what percentage of coverage will allow not to let bugs into production. If the fatal changes in the code skip tests without affecting their result, the solution suggests itself - you need to test the tests!



On the approach to the automation of this task was the report of Mark Langovoy on  Frontend Conf . Video and article are short, and ideas are very working - you need to take note.


About the speaker: Mark Langovoy ( marklangovoi ) works in Yandex in the Yandex.Tolok project . This is a crowdsourcing platform for quickly marking a large amount of data. Customers upload data that, for example, needs to be prepared for use in machine learning algorithms, and set a price, and the other side, the performers, can complete tasks and earn money.

In his free time, Mark develops the Krasnodar development community Krasnodar Dev Days - one of the 19 IT communities whose activists we invited to Frontend Conf in Moscow.

Testing

There are different types of automated testing.


In the course of popular unit testing, we write tests for small parts (modules) of an application. They are easy to write, but sometimes during integration with other modules they may not behave exactly as we expected.

To avoid this, we can write integration tests that will check the work of our modules together.


They are a bit more complicated, so today we will focus on unit testing.

Unit testing

Any project that wants at least some minimal stability deals with writing unit tests.

Consider an example.

classSignal{
    on(callback) { ... }
    off(callback) {
        const callbackIndex = this.listeners.indexOf(callback);
        if (callbackIndex === -1) {
            return;
        }
        this.listeners = [
            ...this.listeners.slice(0, callbackIndex - 1),
            ...this.listeners.slice(callbackIndex)
        ];
    }
    trigger() { ... }
}

There is a Signal class - this is the Event Emitter, which has an on method for the subscription and an off method for deleting the subscription - check if the callback is contained in the array of subscribers, then we delete it. And, of course, there is a trigger method that will call signed callbacks.

We have a simple test for this example that calls the on and off methods and then the trigger to verify that the callback was not called after the unsubscribe.

test(’off method should remove listener', () => {
    const signal = new Signal();
    let wasCalled = false;
    const callback = () => {
        wasCalled = true;
    };
    signal.on(callback);
    signal.off(callback);
    signal.trigger();
    expect(wasCalled).toBeFalsy();
});

Quality assessment criteria

What are the criteria for assessing the quality of such a test?

Code coverage  is the most popular and well-known criterion that shows how many percent of the lines of code were executed when the test was launched.


You can have 70%, 80% or all of 90% of Code coverage, but does this mean that when you build a new build for production, everything will be fine, or something may go wrong?

Let's return to our example.

Friday night, you're tired, finish another feature. And then you come across this code, which was written by your colleague. Something in him seemed complicated and scary to you.

            ...this.listeners.slice(0, callbackIndex - 1),
            ...this.listeners.slice(callbackIndex)

You decided that you can probably just clear the array:

classSignal{
    ...
    off(callback) {
        const callbackIndex = this.listeners.indexOf(callback);
        if (callbackIndex === -1) {
            return;
        }
        this.listeners = [];
    }
    ...
}

I made a commit, put together a project and sent it in production. Tests have passed - why not? And he went to rest in a bar.



But suddenly, late at night, the bell rings, they scream into the phone that everything is falling, people cannot use the product, and in general - business is losing money! You burn, you face dismissal.



How to deal with this? What to do with the tests? How to catch such primitive stupid mistakes? Who will test the tests?

Of course, you can hire an army of QA-engineers - let our application sit and just click.



Or hire a QA automator. They can get the job of writing tests - why write by yourself, if there are special people for this?

But in fact it is expensive, so today we will talk about mutational analysis or mutational testing.

Mutation Testing

This is a way to automate the process of testing our tests. Its goal is to identify ineffective and incomplete tests, that is, in essence, this is testing of tests .

The idea is to change pieces of code, run tests on them, and if the tests did not fall, then they are incomplete.

Changes are made using certain operations - mutators . They replace, for example, plus by minus, multiply by divide, and other similar operations. Mutators can change pieces of code, replace conditions in a while, reset arrays instead of adding an element to an array.


As a result of the application of mutations to the source code, it mutates and becomes a mutant .

Mutants are divided into two categories:

1. The dead  - those in which we were able to identify deviations, that is, in which at least one test fell.
2. The survivors  are the ones who ran away from us and got the bug before production.

For quality assessment, there is the MSI (Mutation Score Indicator) metric  - the percentage ratio between killed and surviving mutants. The greater the difference between code coverage tests and MSI, the worse the relevance of our tests reflects the percentage of code coverage.

It was a bit of theory, and now consider how it can be used in JavaScript.

Javascript solution

In JavaScript, there is only one actively developing tool for mutation testing - this is Stryker . This tool was named after the character X-man William Stryker - the creator of "Weapons X" and a fighter with all the mutants.



Stryker is not a test runner, like Karma or Jest; neither is it a framework for tests like Mocha or Jasmine. This is a framework for mutational testing that complements your current infrastructure.

Plugin system

Stryker is very flexible, fully built on the plugin system, most of which are written by the developers of Stryker.


There are plugins for running tests on Jest, Karma and Mocha. There is integration with the Mocha frameworks (stryker-mocha-framework) Jasmine (stryker-jasmine) and ready-made sets of mutators for JavaScript, TypeScript and even for Vue:

• stryker-javascript-mutator;
• stryker-typescript;
• stryker-vue-mutator.

Mutators for React are included in the stryker-javascript-mutator. In addition, you can always write your mutators.

If you need to convert the code before launch, you can use plugins for Webpack, Babel or TypeScript.


This is all relatively simple.

Configuration

Configuration will not be difficult: you only need to specify in the JSON-config which test runner (and / or test framework, and / or transpiler) you use, and also install the appropriate plug-ins from npm.

A simple console utility stryker-cli can do all this for you in question-answer mode. She will ask you what you are using and will configure it yourself.

How it works

The life cycle is simple and consists of the following steps:

• Reading and analyzing the config. Stryker loads the config and analyzes it for various plugins, settings, exclusion of files, etc.
• Loading plugins according to config.
• Running tests on the source code in order to check whether the tests are relevant now (all of a sudden they are already broken).
• If everything is good, a set of mutants is generated for the files that we have allowed to mutate.
• Run tests on mutants.

The above is an example of running Stryker:

• Stryker runs;
• reads the config;
• loads the necessary dependencies;
• finds files that will mutate;
• runs tests on the source code;
• creates 152 mutants;
• runs tests in 8 threads (in this case, based on the number of CPU cores).

This is not a fast process, so it is better to do it on any CI / CD servers.

After passing all the tests, Stryker gives a brief report on the files with the number of created, killed and surviving mutants, as well as the percentage of the ratio of killed mutants to survivors (MSI) and mutators that were applied.

These are potential problems that are not foreseen in our tests.

Summarize

Mutation testing is useful and interesting . It can find problems in the early stages of testing, and without the participation of people. It will reduce the time it takes to test the pull request, for example, because qualified developers will not have to spend time checking the pull request, which already has potential problems. Or save production if you decide to prepare a new release on Friday night.

Stryker is a flexible multithreaded mutation testing tool. It is actively developing, but still damp, still has not reached the major version. For example, during the preparation of this report, its developers finally made it possible for the Babel plugin to specify the configuration file and fix the Jest integration. This is an open source project that can be helped to grow.

You can watch other front-end speeches on our youtube channel , all thematic reports from all our conferences gradually get there. Or subscribe to the newsletter , and we will keep you updated on all new materials and news of future conferences.