PhantomJS + JSCoverage + QUnit or console JS unit tests with coverage calculation

Let's talk about the case when you need to automate the launch of tests and the collection of coverage statistics, for example, for a hypothetical client JS library. The task is not entirely trivial, since a normal browser requires a full-fledged browser — the library is a visual wrapper over the standard components of the form. The library should be written so that all interaction with its objects can be done using the methods that they provide, and not only through direct manipulations with the DOM (i.e. any user action can be triggered not only by an event, say, clicking on something, but also hands through the method). Nevertheless, it is necessary to have this DOM so that the results of the methods, in addition to changing the internal state of objects, are also displayed in the DOM. In general, it resembles Sencha (ExtJS) applications.

To achieve these goals, you need a certain controlled browser, a framework for running tests and a utility that will allow you to calculate code coverage with tests, as well as some code that will connect all components.

Concept

One solution is JSTestDriver , which was already written about here, but in the case under consideration, this option did not fit, since the whole solution should be run console-based, should not produce any unnecessary services, or browser windows, should start and die immediately upon completion of work. Plus at the moment, JSTestDriver is not able to exclude connected files from the Coverage Report, which greatly distorts the picture, because Considering coverage, for example, for connected jQuery or MooTools, is unnecessary. On habr there was an overview of simple start of QUnit tests with the help of PhantomJS , but without calculation of coverage or page assembly (which is necessary just for coverage).

Given the specified requirements, a solution based on a bunchPhantomJS + JSCoverage + QUnit , posted by me on Google Code. Let's dwell on the components:

• PhantomJS - console (headless) browser controlled through the JS API;
• JSCoverage is a console parser of JS files, implements an increment in them for each executable line, respectively. array element to count the number of executions. There are minor restrictions - only lines that can be executed at all are involved, i.e. not comments, for example. For notations of objects and arrays, only the line where the notation began is taken into account;
• QUnit is a small JS framework for conveniently and quickly creating unit tests, including asynchronous ones, launched in the browser as a page with connected scripts.

Launch

For the utility to work correctly, it is necessary to download and unzip two archives with executable files PhantomJS (dynamic) and JSCoverage , the paths to them can be changed in batch files, and also indicate the correct name of the test group to execute there. In order to connect all the components together, the auxiliary functionality in question was written. To configure a set of tests and scripts for inclusion in the launched page, one configuration file of the form is used:

var config = {
    includes: [ // файлы, которые будут включены в запускаемый скрипт
        {
            file: 'lib/jquery-1.7.1.js',
            coverage: false // не включаем публичную либу в состав отчета о покрытии
        },
        {
            file: 'lib/json2.js',
            coverage: false
        },
        {
            file: 'lib/testable.js',
            coverage: true // а вот тестируемую библиотеку — включаем
        }
    ],
    testCases: [ // массив путей до тесткейсов, могут обрабатываться рекурсивно, включены будут только файлы, подходящие по маске
        {
            location: '/tests',
            pattern: /.+Test\.js/g,
            recursive: true
        }
    ],
    target: { // информация о сборочной директории
        location: '/target'
    }
};

After starting the “Most Main Batnik” (in the example from the run-tests.cmd or run-tests2.cmd repository , depending on which set you plan to run), the following series of actions are simplified:

1. To collect in one folder all connected JS files for processing by JSCoverage;
2. Walk through them with JSCoverage;
3. Collect links to other connected files that are not involved in calculating coverage (libraries, QUnit itself, auxiliary components for testing, etc.);
4. Find all files with test cases;
5. Assemble a page from the elements described above for execution in the usual order for QUnit;
6. Run this page with scripts for execution from under PhantomJS;
7. Take a QUnit report and execution statistics from the “browser window” - coverage, number of completed, successful and dropped tests;
8. Generate a human-readable report from all of this;
9. Put reports in the folder, as well as everything you need to restart it.

Reports

After the done manipulations in the folder with the utility there will be a subfolder / target / in which they will be folded:

• test.html - the very page with the necessary list of scripts to run in webkit-like browsers (PhantomJS restriction, it is not allowed to give links with the file: // protocol yet), it allows you to use all your favorite dev tools functionality for debugging test curves in the familiar visual environment (the order is this - we run the console test, look in the browser, fix it, run it again, look in the browser for an updated version);
• test-result.html - test execution report, collected coverage statistics, without running tests directly:
  
• * .js.html - coverage reports for each file included in the report:
  
• coverage.json - raw statistics of the number of lines executed by files, just in case, see what JSCoverage collected.

Total

+ Collection of coverage statistics;
+ Automatic assembly of resources for testing, the user is required to configure once and just run the batch file;
+ Launch from under the console;
+ Standalone (you do not need to start anything other than two programs that do not need to be configured or installed - just unzip the archives);
+ Visual report after execution;
- Testing takes place under the conditions of a webkit browser, which does not allow you to check, for example, the features of browsers with arrays, on the other hand, it is better to run cross-browser tests with something like Selenium to emulate user actions, in the case of the considered API tests are purely;
-The collector currently only works with relative paths, support for absolute paths and links will appear in the near future;
- No integration with Maven or anything else for continuous integration - in the near future.

At the moment, the utility can be considered as proof of concept, which copes well with its task, simplifies life, but which is not without its drawbacks. In any case, I plan to develop the functionality as necessary and whenever possible. The kit also comes with a small utility for running tests from JSTestDriver within the framework described above, so you can run tests in the IDE in PhantomJS, but at the moment it is not able to run asynchronous tests.