Allure-Framework. Work with code / Sberbank company blog

Continuing a series of publications on the capabilities of the Allure-framework , today we will talk about working with code. Under the cut, we analyze what a test step is, how to display information in a report during the steps, and what categories of defects are. In addition, we’ll talk about Allure annotations. Further more interesting!

Test step. Definition and annotation @ Step

To visualize the processes that occur in your tests, you must use the steps. A step should represent some kind of atomic action or test. Actually the sequence of steps represents your test. For clarity, the step should describe the input parameters, as well as the expected and actual result, if the step is a check.

In Allure, an annotation @ Step is required to indicate a test step over a method . Then the necessary method steps are included in the body of the test method: tests with the annotation @ Test.

Let's consider an example.

Create a method with the annotation @ Step, which checks whether the sum of two terms complies with the expected result:

@Step
public static void checkSumStep(int num1, int num2, int expectedSum) {
   Assert.assertTrue("Сумма слагаемых не соответствует ожидаемому значению", num1 + num2 == expectedSum);
}

Now create a test method using this step:

@Test
public void simpleTest2() {
   checkSumStep(3, 2, 5);
   checkSumStep(5, 4, 9);
}

When running this autotest, we get the following report:



Annotation @ Step can accept the parameter “value”, which allows you to specify the name of the step in Russian. In addition, the name of the step can be parameterized - pass in it the values ​​of the arguments passed to the step.

Demonstrate with an example

@Step("Проверка разности числа {num1} и числа {num2}")
public static void checkSubtractionStep(int num1, int num2, int expectedResult) {
   Assert.assertTrue("Результат вычитания не соответствует ожидаемому значению", num1 - num2 == expectedResult);
}

When using this step in the test, we get the report:

It is worth noting that in the case of using an object as an argument to a method, it is possible to display in the step name the value of one of the fields of this object. To do this, specify the field name in the value in the form {object.fieldName}

Information output to the report during the test steps. Attachments

When performing the autotest steps, it can be very useful to display various kinds of information in a report. For these purposes, Allure serves as an attachment.

An attachment is information of various types in the form of a file that can be attached to a test step or test.

Attachments can have 4 characteristics:

• value / name - name of the attachment;
• type - information type in accordance with the MIME standard ;
• content - content of the attachment;
• fileExtension - an optional attachment file extension starting with a period.

Attachments to the report can be attached in 2 ways: using the @ Attachment annotation and using the static addAttachment method of the Allure class.

Attach attachments using the @Attachment annotation

In this method, a method is created that returns an array of bytes, the @ Attachment annotation is placed above it . When using this method, the read information will be added to the report as a file with the corresponding extension. The name of the attachment will be the same as the name of the called method.

Demonstrate with an example

Create a method that will read the attachment:

@Attachment
public static byte[] getBytes(String resourceName) throws IOException {
     return Files.readAllBytes(Paths.get("src/main/resources", resourceName));
}

Create a test step that will use this method:

@Step("Проверка эквивалентности строки {str1} строке {str2}")
public static void checkStringEqualsStep(String str1, String str2) throws IOException {
    Assert.assertTrue("Строки не эквивалентны", str1.equals(str2));
    getBytes("picture.jpg");
    getBytes("text.txt");
}

And finally, we will write a test in which our step will be used:

@Test
public void simpleTest4() throws IOException {
    String darkSouls = "Dark souls 3";
    checkStringEqualsStep(darkSouls, darkSouls);
}

As a result of running this test, we get the report:



In the @ Attachment annotation, you can specify additional refinement parameters:

@Attachment(value = "Вложение", type = "application/json", fileExtension = ".txt")
public static byte[] getBytesAnnotationWithArgs(String resourceName) throws IOException {
    return Files.readAllBytes(Paths.get("src/main/resources", resourceName));
}

When using this method, in its steps, attachment will receive the name “Attachment”, the contents of the attachment will be highlighted in accordance with the “json” template, and the attachment itself will be saved in the “.txt” format:



If you change the type to “plain / text”, then there’s no for json-style, there will be no highlighting of key-values:

You can also experiment with fileExtension, for example, specify ".doc". In this case, the attachment will be saved in a format specific to MS Word '97.

Attach attachments using the static addAttachment method of the Allure class

In this method, you can attach the attachment to the test step / test using the overloaded static addAttachment method from the Allure class. You can pass up to 4 arguments to this method - 2 of them are required (the name of the attachment and the content to be attached), 2 are optional (file extension and MIME type).

@Step("Добавить ссылку на Сбербанк")
public static void addLinkSber() {
    String link = "http://sberbank.ru";
    Allure.addAttachment("Результат", "text/plain", link);
}

Despite the fact that MIME types must be specified quite often, they have to be written “manually”. Allure does not include a class with valid constants.

Other Allure annotations

In addition to the most famous annotations @ Step and @ Attachment, Allure in its composition has a number of other annotations:

Summary @ Description

@ Description - An annotation placed above the test or step. Attaches a description to a test or test step. This annotation can take “value” parameters - description text and “useJavaDoc”. When setting “true” to the last parameter, the description will be taken javadok located above the method.

Here is an example of using this annotation

@Test
@Description(value = "Тест проверяет эквивалентность единицы единице")
public void simpleTest7_1() {
    Assert.assertTrue(1 == 1);
}

Functionality Annotations

@ Epic - annotation placed above the test. Allows you to group tests by epic. This annotation takes the parameter “value” - the name of the epic.
@ Epics is the same as @ Epic, but accepts an array of Epics as the “value” parameter.
@ Feature - annotation placed above the test. Allows you to group tests according to the tested functionality. This annotation takes the “value” parameter - the name of the functional.
@ Features is the same as @ Feature, but accepts an array of Feature as the “value” parameter.
@ Story- An annotation placed above the test. Allows grouping tests by User story. This annotation accepts the “value” parameter - the name of the User story.
@ Stories is the same as @ Story, but accepts an array of Story as the “value” parameter.

Annotations @ Epic / Epics, @ Feature / Features, @ Story / Stories can be attributed to functionality annotations. These annotations group functionality tests in the Behaviors section of the Allure report.

An example of using these annotations

@Epic(value = "Математика")
@Feature(value = "Простые математические операции")
@Story(value = "Сложение")
@Test
public void sumTest() {
    Steps.checkSummationStep(5, 4, 9);
}
@Epic(value = "Математика")
@Feature(value = "Простые математические операции")
@Story(value = "Вычитание")
@Test
public void subTest() {
    checkSubtractionStep(8, 2, 6);
}
@Epics(value = {@Epic(value = "Математика"), @Epic(value = "Геометрия")})
@Features(value = {@Feature(value = "Тригонометрия"), @Feature(value = "Простые математические операции")})
@Stories(value = {@Story(value = "Синус"), @Story(value = "Синусоида")})
@Test
public void checkSinTest() {
    checkSinStep(0, 0);
}

When performing tests and subsequent report generation, in the Behaviors section we will see that the tests are grouped into a multi-level list ( @ Epic → @ Feature → @ Story):

Abstract @ Flaky

@ Flaky - annotation placed above the test. Marks autotest as unstable. If the autotest crashes in at least one restart (for example, the “target” folder between runs of the same test is not cleared) - in the report on this test you will see a bomb sign. In addition, classes with unstable tests can be marked with this annotation.

Here is an example of using this annotation.

Note that you should not do branching in tests: in this case, the if - else block is used only to make the test unstable!

@Test
@Flaky
public void testDemoFlaky() {
    int randomNum = ThreadLocalRandom.current().nextInt(0, 2);
    if (randomNum == 0) {
        Assert.assertTrue(true);
    } else {
        Assert.assertTrue(false);
    }
}

With several test runs in restart mode, the report will look like this:

Test Case and Defect Link Annotations

@ Issue - annotation placed above the test. Allows linking autotests with wired Issue. This annotation accepts the “value” parameter, which indicates the defect ID in the bug tracking system.

@ Issues is the same as @ Issue, but accepts the Issue array as the “value” parameter.

@ TmsLink - annotation placed above the test. Allows linking autotests with test cases, the steps of which are described in test management systems. This annotation accepts the “value” parameter, which indicates the test ID in the test management system.

@ TmsLinks is the same as @TmsLink, but accepts an array of TmsLinks as the value parameter.

Annotations in this group add defect / test case references to the Allure report.

In order to get a full link from the defect / test case ID specified in the “value” parameter, you need to describe the link template to the corresponding defect / test in allure.properties (which should be placed in the classpath, for example, in src / test / resources) case.

allure.link.issue.pattern=https://example.org/issue/{}
allure.link.tms.pattern=https://example.org/tms/{}

When generating the report, Allure will replace the {} characters with the values ​​that were specified in the value parameter of your annotation, and you will get full links.

An example of using these annotations

@Test
@Issue(value = "FGY-4627")
@TmsLinks({@TmsLink(value = "TL-135"), @TmsLink(value = "TL-158")})
public void simpleTest15() {
    Assert.assertTrue(1 == 1);
}
@Test
@TmsLink(value = "TL-678")
public void simpleTest18() {
    Assert.assertTrue(1 == 1);
}

Annotations of other links

@ Link - annotation placed above the test. Allows you to attach links to some external resources to the autotest. This annotation can take the parameters “name” - the name of the link (default, url), “value” - a synonym for “name”, “url” - the address to which you want to jump, and “type” - the parameter used to create the icon for links.
@ Links is the same as @ Link, but accepts an array of Link as the “value” parameter.

An example of using these annotations

@Test
@Link(name = "Ссылка", url = "http://yandex.ru")
public void checkSubtractionWithLinkTest() {
    checkSubtractionStep(15, 5, 10);
}
@Test
@Links(value = {@Link(name = "Ссылка1", url = "http://sberbank.ru"),
        @Link(name = "Ссылка2", url = "http://yandex.ru")})
public void checkSubtractionWithLinksTest() {
    checkSubtractionStep(14, 5, 9);
}

Abstract @ Owner

@ Owner - annotation placed above the test. Allows you to specify the person responsible for the test. This annotation accepts the “value” parameter, which indicates information about the author of the autotest.

Here is an example of using this annotation

@Test
@Owner(value = "Пупкин Валерий Иванович")
public void testDemoOwner() {
    checkSumStep(1, 2, 3);
}

Abstract @ Severity

@Severity - annotation placed above the test. Allows you to specify the criticality level of the functionality checked by the autotest. This annotation takes a “value” parameter, which can be one of the enum SeverityLevel elements: BLOCKER, CRITICAL, NORMAL, MINOR, or TRIVIAL.

Here is an example of using this annotation

@Test
@Severity(value = SeverityLevel.BLOCKER)
public void testDemoSeverity() {
    checkSubtractionStep(6, 1, 5);
}

Parameterized AutoTests in Allure

Allure can work with parameterized autotests. Consider the example of JUnit 4.12.
First, create a test class with parameterized tests of the following form:

@RunWith(Parameterized.class)
public class ParamsTests {
    @Parameter
    public int operand1;
    @Parameter(1)
    public int operand2;
    @Parameter(2)
    public int expectedResult;
    @Parameters(name = "operand1 = {0} | operand2 = {1} | expectedResult = {2}")
    public static Collection dataProvider() {
        return Arrays.asList(new Integer[][]{
                {1, 2, 3},
                {2, 4, 6},
                {5, 6, 11},
                {7, 5, 12},
                {2, 4, 5},
                {4, 1, 5},
                {8, 2, 11}
        });
    }
    @Test
    public void checkSum() {
        Assert.assertTrue("Сумма слагаемых не соответствует ожидаемому значению", operand1 + operand2 == expectedResult);
    }
}

Now let's run our test and collect the report:



Allure presents our parameterized test as a set of tests, if you fail in any test, we will get detailed information about the run of this particular case.

Categories of defects. Defect categorization

By default, on the Categories tab of the Allure report, all run test methods are divided into 2 categories: product defects (failed tests) and test defects (broken tests). In order to make custom classification, it is necessary to put the categories.json file in the “target / allure-results” directory.

If you have the allure-maven plugin connected to pom.xml, then categories.json can be placed in the resources subdirectory of the test directory

We give an example of a custom classification of defects. Create the categories.json file:

[
  {
    "name": "Неизвестная проблема",
    "matchedStatuses": ["broken", "failed"],
    "messageRegex": ".*что-то пошло не так.*"
  },
  {
    "name": "Нет данных",
    "matchedStatuses": ["broken"],
    "traceRegex": ".*NullPointerException.*"
  },
  {
    "name": "Product defects",
    "matchedStatuses": ["failed"]
  },
  {
    "name": "Test defects",
    "matchedStatuses": ["broken"]
  }
]

Description of the attributes :

• name - the name of the category. It will be displayed on the “Categories” tab at the highest level of classification. Required attribute.

• matchedStatuses - a list of statuses with one of which the test must complete in order to fall into this category. The statuses “failed”, “broken”, “passed”, “skipped” and “unknown” are available from the box. Optional attribute.

• messageRegex - a regular expression that an Exception message must match to fall into this category. Optional attribute.

• traceRegex - the regular expression that the Exception-StackTrace must match to fall into this category. Optional attribute.

Now let's run the tests that detect such defects and see how the report will look on the “Categories” tab:

Test environment. ENVIRONMENT block on report main page

Allure-report allows you to display on your home page in a special block information about the test environment on which the tests were run. It looks like this: The



information that is displayed in this block gets there from a special environment.properties file.

Here is an example of this file:

OS.version=Windows 10 Pro
JDK.version=jdk1.8.0_162
MAVEN.version=Apache Maven 3.5.2
allure-junit4.version=2.6.0
allure-report.version=2.6.0

This file must be placed in the directory “target / allure-results” before assembling the html report. You can do this manually, or you can use the maven-resources-plugin .

Here is an example of its use in this situation, provided that environment.properties is placed in the resources subdirectory of the test directory. To do this, modify pom.xml:

      ...
      maven-resources-plugin3.0.2copy-resourcesvalidatecopy-resources${allure.results.directory}src/test/resourcesenvironment.properties
      ...
   

Now, when building the project, environment.properties will fall into “target / allure-results” and participate in building the html report.

When running tests on Jenkins, categories.json will not be copied to “target / allure-results” on its own. It is also very convenient to add it to the includes maven-resources-plugin section.

Conclusion

In the second part of the LongRead we tried to talk in detail about annotations in Allure, gave examples of their use. We touched upon categories of defects, test environment and ways of outputting information to the report. We invite you to a discussion in the comments.

Subscribe to the ESF blog, stay tuned for new publications. Soon we will tell you something new and useful about Allure again.