Evaluation of the test coverage of a Java project using the example of Apache Ignite

I am participating in the development of the open source Apache Ignite project , while working on the project it became interesting for me to evaluate the test coverage and this is what came of it.



Test coverage is the most popular metric used in evaluating the quality of product testing.


This is one of the few metrics that allows you to identify areas requiring attention due to the risk of missing an error, as well as to prioritize work on the modules or components of the project.


The easiest way to get a complete assessment of the test coverage of a Java project is to use the coverage runner built into IntelliJ IDEA . It allows you to set up a collection of metrics in a couple of clicks and run tests with the subsequent generation of a report.


Testing in the Apache Ignite project


The Apache Ignite project for testing uses its own test framework implemented on the basis of JUnit 3. At the time of this writing, the core module of the project contains ~ 82 thousand tests, most of which are component and require raising a cluster from several nodes, including various JVMs, with concomitant preparation of the environment.


It is worth noting that ensuring the performance of such a huge regression base is not an easy task. The community constantly monitors the state of the product and corrects errors found in the framework of the initiative " Make Teamcity Green Again ".


Designated features of the project do not allow to run all the tests in one JVM at once for the following reasons:


  • possible error OutOfMemoryError;
  • possible failure (crash) JVM;
  • possible deadlocks;
  • the inability to start the test due to not stopped the node in the previous test;
  • the run will take three days on one computer.

All this makes it impossible to use IntelliJ IDEA to get a report on all project tests and requires a special approach to solving the problem.


Preparation and evaluation of test coverage


Based on the work done, the most reliable approach was chosen for the task, containing the following steps:


  1. Definition of a set of test classes;
  2. Execution for each test class:
    2.1. running and running a test suite of a class in a separate JVM with a watchdog timer, which terminates the stream in case of a hang or test problems;
    2.2. operations for obtaining and maintaining test coverage metrics;
    2.3. cleaning the environment at the end of the tests;
  3. Merge all metrics obtained in clause 2;
  4. Generate a full report.

There are many tools designed to assess test coverage, the most popular of them are:



I will not dwell on their differences, a clear table comparing the capabilities of the tools for assessing the test coverage is presented here .


To solve the problem, the JaCoCo library was chosen in order to be able to embed the solution on TeamCity , on which the existing Apache Ignite project testing infrastructure is based . TeamCity can work out of the box with JaCoCo .


A bash script and Maven were used to automate the described algorithm . The configuration of the Jacoco Maven plugin is implemented by a separate Maven profile in pom.xml.


The configuration profile of the JaCoCo plugin is shown below and implies the division into 2 separate launches:


  1. Test run with connected JaCoCo agent ( prepare-agent ) for collecting test coverage metrics. The 'runDirectory' property will be passed by the script at startup, which will allow to save the results of the runs isolated;
  2. Merging the results of the run ( merge ) and generating the report ( report ).

Maven JaCoCo configuration
<profile><id>coverage</id><properties><argLine>
          -ea \
          -server \
          -Xms1g \
          -Xmx6g \
          -XX:+HeapDumpOnOutOfMemoryError \
          -XX:+AggressiveOpts \
          -DIGNITE_UPDATE_NOTIFIER=false \
          -DIGNITE_NO_DISCO_ORDER=true \
          -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true \
          -DIGNITE_QUIET=false \
          -Djava.net.preferIPv4Stack=true \
    </argLine><coverage.dataFile>${runDirectory}/coverage-reports/jacoco-ut.exec</coverage.dataFile><coverage.outputDir>${runDirectory}/jacoco-ut</coverage.outputDir></properties><build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><configuration></configuration><executions><execution><id>default-test</id><phase>test</phase><goals><goal>test</goal></goals></execution></executions></plugin><plugin><groupId>org.jacoco</groupId><artifactId>jacoco-maven-plugin</artifactId><version>0.8.1</version><executions><execution><id>default-prepare-agent</id><goals><goal>prepare-agent</goal></goals><configuration><destFile>${coverage.dataFile}</destFile></configuration></execution><execution><id>post-merge</id><phase>validate</phase><goals><goal>merge</goal></goals><configuration><fileSets><fileSet><directory>${basedir}</directory><includes><include>results/*/coverage-reports/jacoco-ut.exec</include></includes></fileSet></fileSets><destFile>merged.exe</destFile></configuration></execution><execution><id>generate-report</id><phase>validate</phase><goals><goal>report</goal></goals><configuration><dataFile>${basedir}/merged.exe</dataFile><outputDirectory>${basedir}/coverage-report</outputDirectory></configuration></execution></executions></plugin></plugins></build></profile>

Below is a script that implements the steps described earlier.


Managing bash-script
#!/bin/bash# Проект должен быть скомпилирован в соответствии с DEVNOTES.txt## Скрипт необходимо запускать в: ignite/modules/core## Команда запуска скрипта: 'nohup ./coverage.sh >/dev/null 2>&1 &'
SCRIPT_DIR=$(cd $(dirname "$0"); pwd)
echo"***** Старт."echo"***** Поиск тестовых классов..."
tests=()
while IFS=  read -r -d $'\0'; do
  tests+=("$REPLY")
done < <(find $SCRIPT_DIR/src/test/java/org/apache/ignite -type f -name "*Test*" ! -name "*\$*" ! -name "*Abstract*" ! -name "*TestSuite*" -print0)
testsCount=${#tests[@]}echo"***** Количество тестовых классов="$testsCount
idx=0
for path in${tests[@]}do
  idx=$((idx+1))
  echo"***** Запуск "$idx" из "$testsCountecho"***** Расположение класса: "$path
  filename=$(basename -- "$path")
 filename="${filename%.*}"echo"***** Название класса: "$filename
 runDir=$SCRIPT_DIR"/results/"$filename
 mkdir -p $runDirif [ "$(ls -A $runDir)" ]; thencontinuefiecho"***** Запуск тестов..."
 timeout 30m mvn -P surefire-fork-count-1,coverage test -Dmaven.main.skip=true -Dmaven.test.failure.ignore=true -Dtest=$filename -DfailIFNoTests=false -DrunDirectory=$runDirecho"***** Очистка окружения..."
 pkill java
done# Объединение результатов и генерация отчета
mvn -X -P surefire-fork-count-1,coverage validate
echo"***** Финиш."

The run of all tests with coverage coverage took ~ 50 hours on a dedicated server: 4 vCPU, 8RAM, 50 SSD, Ubuntu x64 16.04 .


The described approach can be easily parallelized into several stands, if resources are available, which will significantly shorten the run-time and obtain an estimate of the test coverage. After embedding this solution on TeamCity , the test coverage evaluation time should take about 2 hours.


results


According to the results of the report, the coverage of the project instructions is ~ 61%.


Covering the instructions of the main components:


  • Cache - 66%
  • Discovery - 57%
  • Compute - 60%
  • Stream - 51%
  • Binary - 68%
  • Transactions - 71%

After analyzing the results, it became obvious that the entire hot code was covered, as well as the code for correcting typical problems. With such a toolkit, it will be possible to expand coverage to rare and unusual situations, making the product even more reliable.


PS The full report for the audit .


Also popular now: