Website Test Automation: Pytest, Allure, Selenium + some secret ingredients

    For more than a year, Acronis has a new logo, how it was created can be read here . As part of the rebranding, the website www.acronis.com has also been updated . To build the site, we chose CMS Drupal. The site turned out to be cool on the outside and difficult on the inside: 28 locales (!), Responsive web design, carousels, wizards, pop-ups ... as a result, I had to use many different Drupal modules, some of which were customized and written my own. With such a dynamic, complex system, errors are inevitable. Their prevention and detection is the responsibility of the website quality control and assurance team. In this article we want to share our successful recipe for automating testing of our site.



    Why and why?


    Unfortunately, at the initial stage of our site’s life, quality assurance was mainly limited to manual testing. Given the short release cycles, regression tests had to be run often. A large number of locales exacerbated the situation (checks were multiplied by the number of locales). The costs of manual testing were large, and the benefit of automating site testing was obvious - you had to act.

    What do we want?


    So, we begin to make requirements for an automatic testing system. We noticed that processing simple enough defects requires valuable time, we wanted to minimize the number of defects brought into the bug tracker, but without losing such errors from the field of view. Therefore, it was decided to assign work with such defects to the autotest reporting system. Reporting was required to be very convenient, understandable, informative and enjoyable. So that any member of our friendly web-team could see the report, quickly understand what was happening, and fix the defect. Accordingly, everyone looks: developers fix the site, testers fix the tests. As a result, we have a “green traffic light” and GO in production.

    Historically, another requirement is the Python development language.. Omitting the details of the remaining wishes, we received a list of requirements for the future automatic testing system:

    • Convenient, understandable, informative reporting ;.
    • Parallel launch of tests;
    • Parameterization of tests;
    • Inheritance of test data for locales;
    • Testing in the environment closest to the user;
    • Minimization of false positives;
    • Convenient support and delivery of testing environments;
    • Minimum of own development and customization;
    • Open source
    • Python

    Flour of choice


    The first thing we noticed was the Robot Framework . This is a popular solution that has already been mentioned in Acronis publications.. Installed, began to try. Of course, there are many pluses, but now it’s not about them. The main contraindication for us is the work of RF with parameterized tests and the regular reporting of results for them. It should be noted that the vast majority of the tests we conceived are precisely parameterized tests. In the RF report, if one of the cases in the parameterized test fails, then the entire test is marked as failed. Also, reporting out of the box RF was inconvenient for most of our guys - it took a lot of time to parse the results of trial auto tests. The peculiar “simple” writing of tests using keywords in the case of tests more difficult proved to be a daunting task. At the same time, thanks to including Habr, we drew attention to Allure from Yandex.

    We looked, everyone liked it, everyone had a desire to work with this system, which, it should be noted, is very important. We studied the tool, made sure that the project is actively developing, has many ready-made adapters for various popular test frameworks, including python. Unfortunately (and maybe fortunately) it turned out that for python there is an adapter only for the pytest framework . We looked, read, tried - it turned out to be a cool thing. Pytest can do a lot of things, it’s easy to use, extensible functionality due to ready-made and proprietary plug-ins, a large online community, but the parameterization of tests on it is the way we need it! We started it, wrote a couple of trial tests, everything is fine. Now it's up to the parallel execution of tests, here the choice of solutions is small - only pytest-xdist. Installed, launched and ...

    A spoon of tar


    It turned out that pytest-xdist is in conflict with Allure Pytest Adapter . They started to understand, the problem is known , it has been discussed for a long time and has not been resolved. We also could not find a ready-made solution for the inheritance of test data for locales

    Secret ingredients


    To get around these problems, we decide to write a tool (wrapper) in python. This wrapper will prepare test data taking into account inheritance, pass it to tests, pass test environment data to the tests (browser, for example), and also run tests in a given number of threads. After the tests are completed, combine the reports received in different streams into one and publish the final data on the website.

    They decided to implement parallelization quite simply, as a parallel call of each unit test through the command line. With this approach, I had to implement the transfer of test data to the test myself. Thanks to fixtures in pytest, this is a matter of several lines. Important! It is also necessary to comment out a couple of lines in (allure-python / allure / common.py) that are responsible for deleting the "old" files in the report directory of allure adapter.

    It was decided to store test data for parameterized tests in tsv files, static test data in yaml. We decided to determine the belonging of test data to the test and the locale using the names and hierarchy of the directories in which this data is located. Inheritance is carried out from the main base locale "en-us", by deleting, adding unique data. It is also possible to use the keywords 'skip' and 'comment' in the test data - to cancel the launch of a specific test case with an indication of the reason. Such inheritance, for example, if it is necessary to use the same data for all site localizations, then inheritance occurs automatically without any additional parameters. By the way, for these test configurations (environment, timeout, etc.), inheritance was also implemented, but no longer based on localization,



    Another nuance


    Receiving the first reports, we began to think how it is more convenient to display the test results in the context of locales. We considered that it is most convenient for us to separate the results according to the principle - each locale has its own copy of the allure report. And for aggregation of general information on locales, we quickly wrote a simple but pretty wrapper.





    The last thing that overshadowed our joy was cases of freezes of individual launches in a test environment. I forgot to mention that we decided to use the classic Selenium (as the environment closest to the real one) as a testing environment. With a large number of checks in selenium failures can not be avoided. As well as everyone’s “favorite” false positives, they make it very difficult for continuous integration and the “green traffic light” for production.

    We thought and found a way out. Hangs - overcame the completion of our wrapper. We added the ability to specify the maximum execution time for an individual test, and if it does not run for the specified time, we roughly restart it. And false positives were removed using the rerun-xfails add -on for pytest. This plugin automatically restarts all failed tests, again we set the number of attempts in the configuration yaml-file for each test or general.

    Epilogue


    And finally, here it is - the happiness of a novice automation: a stable, convenient working system. It is easy to maintain, allows testing as quickly as possible and without false positives, and provides very convenient reporting on test results.

    PS


    Friends, on your feedbacks here on Habré we would like to understand how interesting our experience is. There is an idea to publish the resulting turnkey solution as a docker container.

    Also popular now: