Allure-Android. Informative reports for mobile automation

The article is published on behalf of Andrey Ivanov and Ekaterina Bateeva, neifmetus

Automation of mobile applications is a rather young sphere: there are many frameworks and many projects are faced with the problem of choosing the most “fast, stable, easy to use”. About two years ago, we also faced the choice of a new automation tool for testing Android applications.
All the popular tools were somehow based on UIAutomator and Espresso, so we decided to test them in their pure form and compare them with the same Appium (the most popular) and seeTest (used before, the best among the paid ones at that time).

Of the advantages of Appium, one can distinguish the usual WebDriver API, the ability to use most popular languages ​​and libraries. In addition, it is widely used in many companies and allows you to write tests directly for iOS and Android platforms. And finally, this is a free boxed solution - what could be better?

So we thought, until we discovered the following shortcomings:

• low stability of Appium Server
• You cannot interact with the public methods of Activity (in 2018, Nikolai Abalov from Badoo talked about creating a backdoor in Appium in his article, you can read here )
• greatly inferior in speed of Espresso tests

For us, these moments were critical, so it was decided to put together our own set of tools around Espresso to build an ecosystem for testing mobile applications.

So, the framework was selected, it remained to find the remaining components:

1. Runner - should allow to run tests in parallel and configure device pools
2. Reporter - must provide a readable report that could be used by any team member

Instruments

Everything was fine with the runner, rummaging a bit on github, shazam / fork was chosen .
It allows you to conveniently configure the device pool, easy to modify, generates a simple html report. Logcat is applied to each report, in the event of a stacktrace test and video crash. Video recording does not work correctly, all videos are 1 minute long, sometimes several tests are recorded on the video.



The fork reports were far from ideal, the end user did not understand what was going on in the test only by its name, without having a test case at hand. I wanted to have steps with file attachments that would allow me to structure the report. The search for a reporter for instrumentation tests yielded 2 variants of spoon and cucumber. Both options were discarded because a bunch of screenshots in the case of spoon and bdd from cucumber did not solve the problem completely ...

Allure seemed the most optimal solution to the problem:

• nesting of steps that allows you to structure the report
• the ability to record custom test data (screenshots, video, task number, test parameters)
• concise view of the report

But there was one caveat, Allure simply does not start on Android.

Allure-android

In connection with the above, it was decided to write a library that combines the simplicity and elegance of Kotlin, the advantages of the Allure framework and can work on Android phones. In order to connect the library, add dependencies to the module in which instrumentation tests are located:

dependencies {
    androidTestImplementation "ru.tinkoff.allure:allure-android:$allureVersion@aar"
    androidTestImplementation "ru.tinkoff.allure:allure-common:$allureVersion"
    androidTestImplementation "ru.tinkoff.allure:allure-model:$allureVersion"
}

After setting up the dependencies, we need to register AllureRunListener to the class responsible for running android tests.

There are three ways to do this:

1. Add to build.gradle

   testInstrumentationRunner "ru.tinkoff.allure.android.AllureAndroidRunner"

2. Add listener to arguments in Runner onCreate (arguments: Bundle)

   arguments.putCharSequence("listener", AllureAndroidListener::class.java.name)

3. Directly inherited from AllureAndroidRunner

Allure reports are based on step - a step, an atomic action that is performed during a test. Allure Step and Parameter framework annotations have been replaced with a direct call to the step () function.

inline fun  step(description: String, vararg params: Parameter, block: () -> T): T

This function not only replaces two annotations at once, but also accepts a lambda into which you should wrap the test logic. For example:


After starting the test, on the phone in the folder / sdcard / allure-results will appear reports in json format prepared for Allure2. Having pulled out the result, the team

adb pull /sdcard/allure-results

we can generate a report

allure generate

Of the additional features, we can distinguish:

• the ability to invest steps in each other
• anywhere, you can call deviceScreenshot (tag: String) to take a screenshot that will be automatically attached to the report in the current step
• FailshotRule () - junit4 Rule, will take a screenshot just before the fall

This is an overview of using Allure on the Android platform. The allure-android solution is available on GitHub , you can see in detail and participate in the development.