Stryker, mutational testing in JavaScript

Recently I got acquainted with a software testing method called “Mutation Testing” and have already become a fan of this approach to writing tests.

Theory first

The purpose of mutation testing is to identify ineffective and incomplete tests, that is, it is essentially testing tests .

The idea is to modify small random fragments of the source code and observe the reaction of the tests. If after making the changes the tests are still passed, then such a set of tests is ineffective or incomplete.

The rule by which the conversion is performed in the source code, for example, substitution trueinstead false, is called a mutator (mutation operator) . As mutators, we also use the replacement of signs of arithmetic operations and Boolean operators, zeroing and rearranging variables in places, removing code branches, and others. Changes made to the source code are called mutations . As a result of the acquisition of mutations, the source code mutates and becomes a mutant . After testing, the mutants are divided into two categories:

• killed (caught) - those in which deviations were detected and at least one test failed
• survivors (escaped) - those who were able to pass the tests successfully

With automatic mutation testing, many mutants of the original source code are created, and test sets are run for each of them.

The metric for the effectiveness of mutation tests is the MSI indicator (Mutation Score Indicator), which reflects the ratio of killed mutants to survivors. The greater the difference between MSI and the percentage of code coverage by tests, the less informative criterion for assessing the quality of tests is their percentage of coverage.

It happens that combinations of mutators cause mutually exclusive mutations, and then they say that the resulting mutant is equivalent (to the original program). This is partly why achieving MSI at 100% can be incredibly difficult even in small projects.

Practice now

I will talk about an automated mutation testing framework called Stryker .

To prepare the project, install the stryker-cli package globally:

npm i -g stryker-cli

Next, install and save the stryker and stryker-api packages in the project devs

npm i --save-dev stryker stryker-api

I will use Mocha as an automatic testing framework , and Chai is familiar to me as a statement library :

npm i --save-dev chai [email protected]

Let's execute stryker init, this initialization utility will ask some questions, I selected everything according to my preferences and configuration, plus I added the html item to the list of reports. This is equivalent to this line:

npm i --save-dev stryker-api stryker-mocha-runner stryker-mocha-framework stryker-html-reporter

At the end of the configuration, a file with the stryker.conf.jsfollowing contents will be created :

module.exports = function(config) {
    config.set({
        files: [{
                pattern: 'src/**/*.js',
                mutated: true,
                included: false
            },
            'test/**/*.js'
        ],
        mutate: [],
        testRunner: 'mocha',
        testFramework: 'mocha',
        mutator: 'es5',
        transpilers: [],
        reporter: ['html', 'clear-text', 'progress'],
        coverageAnalysis: 'perTest'
    });
};

We will figure out the options and configure it for ourselves:

• files- An array of names and name patterns for specifying the files needed for testing. As elements you can use:
  
  • string literals, for example 'src/**/*.js'.
  • InputFileDescriptor objects: { pattern: '', included: true, mutated: false }where
    
    • pattern- A required field with a name or name template, but which does not support file exclusion through! , unlike string literals. That is, if a file or directory starts with !and is needed in the project, then use this method instead of a string literal.
    • included- an optional field that determines whether the file should be loaded into the test runner ( true) or simply copied to the sandbox ( false). At runtime, you can observe how the directory flickered in the project structure .stryker-tmp, and in it sandboxes with mutants, if the project depends on your other module, it must also be specified for copying to the sandbox.
    • mutated - an optional field that determines whether the file should be mutated.
• mutate- an optional array of names and name patterns to indicate the files to mutate. You can do without this array if you use InputFileDescriptor objects when selecting files in the array files.
• testRunner- required field, indicates the test runner for tests. Make sure that the appropriate plugin for Stryker is installed , for example, stryker-karma-runnerfor use karmaas a test runner.
• testFramework- indicates the framework used by the tests. The default value is fromtestRunner
• mutator- an optional field, indicates the plug-in set of mutators used in testing by default es5.
• transpilers - an optional array field, indicates transpilers that must perform code conversions before execution.
• reporter - an optional array field, with which you can select the format for reporting after automatic mutation tests.
• maxConcurrentTestRunners - an optional field that determines the number of simultaneously executed tests.

As a capacious practical example, I created a project with the following structure

├── app.js
├── package.json
├── stryker.conf.js
└── test
    └── app.test.js

the main file contains and exports only one function

// app.js
module.exports = {
    userIsOldEnough: (user) => user.age >= 18
};

To substantiate the concept of mutation testing, I will supply the project with unit tests with 100% coverage, even in 2 passes:

// test/app.test.js
const
    expect = require('chai').expect,
    app = require('../app');
describe('Site', () => {
    it('can be visited by an adult', () => {
        expect(app.userIsOldEnough({ age: 23 })).to.be.true;
    });
    it('can not be visited by a child', () => {
        expect(app.userIsOldEnough({ age: 13 })).to.be.false;
    });
});

Stryker configuration file looks like this

// stryker.conf.js
module.exports = function(config) {
    config.set({
        files: [{
                pattern: 'app.js',
                mutated: true
            },
            'test/**/*.js'
        ],
        testRunner: 'mocha',
        reporter: ['html', 'clear-text', 'progress'],
        testFramework: 'mocha'
    });
};

I also added a couple of scripts in package.jsonfor convenience:

{
  "name": "mutations-demo",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "test": "istanbul cover _mocha",
    "posttest": "stryker run"
  },
  "main": "app.js",
  "devDependencies": {
    "chai": "^4.1.2",
    "mocha": "^3.5.0",
    "istanbul": "^0.4.5",
    "stryker": "^0.13.0",
    "stryker-api": "^0.11.0",
    "stryker-html-reporter": "^0.10.1",
    "stryker-mocha-framework": "^0.6.1",
    "stryker-mocha-runner": "^0.9.1"
  },
  "dependencies": {
    "underscore": "^1.8.3"
  }
}

We execute

npm t

and now the fun part begins: you can make sure that all unit tests are passed and they cover 100% of the code

  Site
    ✓ can be visited by an adult
    ✓ can not be visited by a child
  2 passing (15ms)
=============================== Coverage summary ===============================
Statements   : 100% ( 2/2 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 100% ( 2/2 )
================================================================================

then mutation testing starts automatically, and here we get the bad news in the form of MSI 50%:

Mutant survived!
Mutator: BinaryOperator
-       userIsOldEnough: (user) => user.age >= 18
+       userIsOldEnough: (user) => user.age > 18
Tests ran:
    Site can be visited by an adult
    Site can not be visited by a child
Ran 1.50 tests per mutant on average.
----------|---------|----------|-----------|------------|----------|---------|
File      | % score | # killed | # timeout | # survived | # no cov | # error |
----------|---------|----------|-----------|------------|----------|---------|
All files |   50.00 |        1 |         0 |          1 |        0 |       0 |
 app.js   |   50.00 |        1 |         0 |          1 |        0 |       0 |
----------|---------|----------|-----------|------------|----------|---------|

The report concludes that the tests are incomplete, since their passage was not affected by a change in the logical operation from >=to >and therefore, they do not check the function in case the user of the site is 18 years old. This report looks like a diff between commits, but according to the settings, a more beautiful one is generated in the form of a similar html-document.

The repository with this project is on Github . And so that you could not raise anything and just look at the logs, I added a project to Travis .