Test Case Management Tool: how to make the right choice and not regret it

    The head of the QA-department of Redmadrobot Ilya Gorshkov tells how he chose the toolkit for working with test cases.

    In the process of working on any complex software system, a large number of project documentation is created. Its structural composition in most cases is almost the same - these are requirements for a system of various levels, a description of its architecture, program code, QA-team documentation, project plans, reports, etc.

    Today I will talk about the artifacts of the Redmadrobot QA team that we work with during the projects. These are test strategies, test plans, test runs and test cases, traceability matrix, bug reports, performance and quality metrics, various reports on test results, etc. All of them have a certain hierarchy, are created in a certain sequence and require periodic updating.

    There are several ways to solve the problem of creating a single system for working with all QA artifacts. For example, select Excel and Google Docs, keep everything in a bug tracker (using the Test Case as Issue Type) or use one of the test case management tools integrated with the company's bug tracker. We at Redmadrobot chose the third option, based on the specifics of the projects, the scope of the tasks, the types of QA documentation and the types of testing we use, the number of manual test cases that are developed and performed, and various projects that work simultaneously.

    The next step for us was the selection of a suitable test case management tool. It is very important to approach this choice as responsibly as possible, since the price of an error when choosing such a tool is quite high for the company. The QA team at Redmadrobot went to the final decision in several stages and began by developing the criteria that the tool we needed should meet.

    The criteria we have introduced are as follows:

    • integration with the company’s bug tracker (JIRA)
    • storage and the ability to edit test cases, including the import of test cases that we created earlier
    • easy tracking of test case requirements
    • convenient formation of Test Runs / Test Suites and convenient user interface
    • storing all information on Test Development and Test Execution in one place and creating a single space for the QA-team
    • ability to create Traceability Matrix
    • the ability to distribute tasks and assign them to specific engineers
    • ease of reporting, metrics, statistics
    • ease of installation, implementation, support

    From which to choose:

    After developing the criteria, we examined several of the most popular tools on the market that met our expectations. Some of the tools were discarded upon closer examination; for the remaining ones, we issued evaluation licenses and continued the study. As a result, the list was reduced to three tools on which pilot projects were carried out:

    1. Zephyr
    2. TestRail
    3. Meliora

    Upon completion of the pilots, a clear understanding of the capabilities and usability of each of them developed, the features of each tool, their applicability and ease of use became clear for us. Here are the high-level characteristics of each tool:

    Zephyr(1 user - $ 30) - the distinguishing feature of this tool is that it is an Atlassian product, which means it must be perfectly compatible with JIRA, but the problem is that it is valid for JIRA Server, but for now we use OnDemand, with which there were a lot of problems at the pilot stage. In addition to this drawback, Zephyr is inconvenient to use: adding test cases is long and not so easy, there are a lot of extra actions when creating plans and runs.

    Meliora (1 user - $ 25) - you also need to switch to JIRA Server + by itself. Meliora is a rather cumbersome tool that artificially complicates most of the simple tasks, including its own bug tracker.

    Testral(1 user - $ 20) - a simple and convenient tool. The main plus - customization is possible in almost everything, and any actions are intuitive. It is possible to import / export test cases.

    After analyzing the results of the pilots and feedback for each tool, we decided to concentrate on TestRail, which includes and allows you to:

    • Creating / storing / editing test scripts, managing test plans, launching test cycles, recording test results.
    • A clear description of the test scenarios, their review, correlation with the requirements, division into areas - all this allows us to evaluate how complete the coverage of functional tests is, and is a necessary material for the entire project team.
    • Reporting on completely different criteria: Defect Summary, Comparison for Cases, test results for projects / components / milestones, etc.
    • Opportunities for full customization of the “working dashboard”, as well as convenient obtaining the status of the QA-team for different periods of time (helps when creating a weekly / monthly QA report).
    • Easy integration with JIRA.
    • Reasonable price.
    • Great support.
    • Easy and convenient way to import tests from Excel.

    The ability to easily import previously created test cases was a very important criterion for us. When we were just starting to develop QA practice in Redmadrobot, there was no talk of the Test Case Management Tool yet, but we used Excel to work with the tests. But we immediately approached the question of using Excel with a foundation for the future and developed a clear format for tests, realizing that after a while we would “feed” them to some tool. When we launched TestRail into commercial operation, we were able to export “as is” several thousand test cases, which greatly reduced the effort to implement the tool.

    Below I will examine in detail the capabilities of TestRail for various QA activities:

    Test Development:

    • creating test plans / suits / test cases;
    • convenient storage, updating and organization;
    • import / export editing ability;
    • easily customizable set of any test attributes;
    • Requirements Traceability.

    Test Execution:

    • milestones (according to the quality criterion in the company);
    • compilation test runs;
    • establishment of defects from test runs;
    • distribution of tasks;
    • convenient integration with JIRA.

    Test Management:

    • activity management;
    • distribution of roles;
    • assignment of tasks and control of implementation.


    • testing progress;
    • test results in the form of ready-made reports;
    • project statistics;
    • variety of reporting options;
    • team performance metrics.

    What is the result:

    We made the final choice back in August. In September, most of the QA project activities were transferred to TestRail. We continue to translate all new projects there. For all the time, they have never regretted the choice. We collected several metrics that confirmed our assumptions about the correctness of the choice and the positive effect of the implementation. We were able to quickly teach engineers to use the tool effectively. In the near future, we will finish work on the implementation of Requirements Traceability for all projects and will continue to use TestRail further.

    Also popular now: