Key QA process indicators

    As part of Quality Assurance, various metrics and indicators of product quality and development process can and should be used. Metrics can be divided into groups according to the parameters on the basis of which they are calculated, according to the stages of the development life cycle at which they are applied, according to goals and objectives, according to the stakeholders for which they are intended. This list goes on and on.

    In this article, I decided to bring together and consider the most basic, in my opinion, groups of criteria and meters for the QA process. And in each group I will list only the most important and indicative, again in my opinion, metrics, as well as I will figure out why they are needed, in which situations they are useful and how to use them.



    What should be the metrics?


    The metric itself in the context of software is a numerical expression of any property, quality of the product itself or the process of its development. In other words, this is what we can measure, compare and evaluate software with.

    Now, just a couple of comments about the values ​​and properties of metrics:

    • The main goal of any metric is to improve the development process and the software product itself. The metric allows you to see at what point on the way to the goals we are at the moment, approaching them or moving away, whether success criteria are achieved.
    • Metrics should not exist for the sake of the measurement process itself. It is necessary to use only those metrics that really have practical value and will affect the further development of the product or the optimization of the process. From here follows a simple rule - first you need to identify areas for change / improvement, and then decide how to evaluate them.

    Key metric groups for QA


    It is theoretically possible to come up with your own characteristic, formula or indicator for almost every, even the most insignificant action, stage or status of the QA process. You can take into account each artifact, all transitions of defects by status, calculate the number of tests in the set. However, the most important question that you should immediately ask yourself when there is a desire to measure something is: “Why do we need this information, how can it be used?” When forming a set of metrics, one should proceed from goals, plans for improving processes and a product.

    So, in this article we will not consider the usual quantitative indicators of testing progress, which are used in most reports and statuses. Instead, we will analyze which areas, artifacts, and areas of development in terms of Quality Assurance we should measure and control to assess the quality of work. Analysis and optimization of these points are extremely important for effective interaction with stakeholders ( https://doitsmartly.ru/all-articles/sw-testing/120-stakeholders-for-qa.html ) and software development in general:

    1. Requirements for the developed BY.
    Precisely we must understand that we are developing and testing, and the degree of this understanding must be able to evaluate. Potential risks or missed problems at the specification level can lead to the most serious and expensive errors.

    2. The quality of the developed product.
    Everything is obvious here: it is necessary to be able to evaluate the quality of development and software in order to make forecasts and assess risks. It is important to understand how high-quality and reliable the product is, relying not only on the presence or absence of errors found, but just predicting whether there are many potential problems.

    3. Features of the QA team.
    It is also simple here: in order to manage the testing process, to plan work and predict the timing, it is always necessary to have not only the current status of tasks, but also know the capabilities of the QA team.

    4. The quality of the testing team.
    In addition to the quality of the product itself, you need to measure the effectiveness of the QA process itself and the testing team. In order to constantly optimize and improve the quality of work, you need to know where we are now, which allows us to move forward and what casts us back.

    5. Feedback and product satisfaction.
    The latter area, but, of course, not in importance, but in the opinion of the stakeholders of the process, consumers of our services, users of the product. It is very important to be able to measure the overall degree of satisfaction with the product, highlight trends and draw appropriate conclusions. Properly selected metrics for this group will allow you to identify possible problems in time and quickly apply feedback to improve processes.

    Next, we will consider exactly which metrics are included in each group, and analyze how they can be estimated. For each group, I will give some examples of possible metrics and describe their meaning. These and some other indicators of the QA process are discussed in more detail in my article “The Most Important QA Metrics” ( https://doitsmartly.ru/all-articles/sw-testing/133-the-most-important-metrics-in-qa. html)

    Group 1 - Requirements for the developed software


    This group of metrics will allow you to evaluate how we have worked out the requirements (user story) for software, to identify vulnerabilities and the most complex, potentially problematic features of the software, to understand where special control is required.

    1. Test coverage requirements
    “Total number of tests” / “Total number of requirements”

    Purpose of the metric: to identify weaknesses in the test coverage, highlight risks.

    • This metric will work only if the requirements are well decomposed into equivalent ones. Otherwise, it will turn into an indicator of the presence or absence of tests for each of the requirements.
    • For requirements in which the coefficient is equal to or close to 0, you need to consider adding tests.

    2. The degree of interconnectedness of
    AVERAGE requirements (“Number of requirements associated with requirement No. 1” / “Total number of requirements -1”, ..., “Number of requirements associated with requirement No. n” / “Total number of requirements -1”)

    The metric is calculated as the number of links of each requirement to other requirements. In this case, the average value for all requirements is used.

    Purpose of the metric: to provide a basis for assessing the timing of testing and taking into account possible risks. Knowing the degree of mutual influence of requirements on each other, you can, for example, plan additional time and cases for end-to-end testing, work out regression checks, look towards integration, etc.

    • From my own experience, I can note that an acceptable degree of connectivity does not exceed 0.2-0.3. Otherwise, the refinement of one of the requirements will lead to a processing chain, and therefore possible errors in a significant part of the product.

    3. Requirements stability coefficient
    “Number of changes in existing requirements” / “Total number of requirements implemented during iteration, including new ones”

    Purpose of the metric: show how many already implemented requirements have to be redone from release to release when developing new features. Also, the metric gives an idea of ​​how easily the functionality of the system scales, new features are added.

    • For this metric, the coefficient must be at least less than 0.5. In this case, we introduce 2 times more new features than redo existing ones. Otherwise, the team focuses more on remaking previously released features, rather than creating new values ​​for the business.

    Group 2 - Quality of the developed product


    This group of metrics allows you to evaluate and compare from release to release both the quality of the software and the quality of the development itself.

    1. Defect density
    “Number of defects in a separate module” / “Total number of defects in software”

    Calculated as the proportion of defects in the total number of defects falling on a separate module as part of an iteration or release.
    Purpose of the metric: highlight which part of the software is the most problematic. This information will help in the assessment and planning of work with this module, as well as in risk analysis.

    • This metric will help to draw our attention to the problem module \ system \ functionality, where the proportion of defects is above average.

    2. The regression coefficient
    “Number of defects in the old functional” / “Total number of defects, including new functional”

    Purpose of the metric: show what the team’s efforts are spent on - whether it is more involved in the development and debugging of new features or does it spend most of the time fixing existing parts of the software.

    • For example, if the regression coefficient is more than 0.5, this means that we spend more than half the time restoring previously working software functions. This situation needs to be corrected.

    3. The coefficient of re-discovered defects
    “Number of re-detected defects” / “Total number of errors, including previously corrected and new”

    Purpose of the metric: to assess the quality of development and correction of defects, as well as the complexity of the product or an individual module.

    • The closer the coefficient is to 0, the less old errors are repeated.
    • Moreover, if the value is more than 0.2-0.3, this can indicate either the technical complexity of the module, or problems in the architecture, or about poor-quality error correction.

    Group 3 - QA Team Opportunities and Effectiveness


    The main task of this group of metrics is to express in numbers what the testing team is capable of. These indicators can and should be calculated and compared on a regular basis, analyze trends, observing how certain changes affect the work of the team.

    1. The speed of the QA command
    “Number of story points per N iterations” ”/“ N ”

    It is calculated as the ratio of realized story points \ requirements \ user stories over several iterations \ sprints to the number of selected iterations.
    Purpose of the metric: to quantify the capabilities, speed of the team for further planning the scope of work and analysis of development trends. The metric allows you to monitor the speed of QA, to observe what internal or external influences on the team affect this speed.

    2. The average lifetime of the defect
    “Total time to fix the defects found” / “Number of defects”

    The total time during which the defects found during the iteration or release were open to the sum of the defects.

    Purpose of the metric: to show how much, on average, it takes to work with one defect: its registration, correction and reproduction. This indicator will allow you to evaluate the time required for testing, to identify areas of software with which the greatest difficulties arise.

    • A defect's lifetime is the entire time from its registration to closure minus all possible suspensions. By showing defects with the longest correction time, the metric will help identify particularly complex and problematic modules or a “weak link” in the development team.

    Group 4 - Quality of work of the testing team


    The goal of this set of metrics is to assess how well testers fulfill their tasks, determine the level of competencies and maturity of the QA team. With this set of indicators, you can compare a team with it on different time periods or with other, external testing groups.

    1. The effectiveness of tests and test sets
    “Number of detected errors” / “Number of cases in a test set”

    Purpose of the metric: show how many errors on average allow our cases to detect. This metric reflects the quality of the test design and helps to monitor the trend of its change.

    • The indicator of “slaughter” of tests allows you to monitor the effectiveness of each of the test sets, how it changes over time. This will make it possible to supplement them with "fresh" tests on time.

    2. Error rate missed on the product
    “Number of errors detected after the release of the release” / “Total number of errors detected before and after the release”

    Purpose of the metric: to demonstrate the quality of testing and the efficiency of error detection - what proportion of defects were filtered out and which passed on productive.

    • Of course, the permissible percentage of missed errors will depend on many factors, however, if it is> 0.1, then there is clearly a problem, because in this case every tenth defect hit the productive and led to software crashes among users.

    3. Real time of the QA team
    “Time spent on target QA activities” / “Total working hours of the team”

    The ratio of time spent by the team directly on target QA activities to the total number of hours.

    Purpose of the metric: firstly, to increase the accuracy of planning, and secondly, to monitor and manage the effectiveness of the team.

    • Targeted activities may include: analysis, design, evaluations, testing, working meetings, and much more, non-targeted ones - simple due to blockers, communication problems, inaccessibility of resources, etc.
    • Of course, this metric will never be equal to 1. Practice shows that for effective teams its value can be 0.5-0.6.

    4. Accuracy of time estimation by region / type / type of work
    “Estimated time of work” / “Actual time of work”

    Purpose of the metric: allows you to use the correction factor for subsequent estimates.

    • The degree of accuracy of the assessment can be determined for the entire team or individual testers, for the entire system or individual software modules.

    Group 5 - Feedback and User Satisfaction


    And finally, a group of metrics showing how the product was accepted by end users, how it met their expectations. At the same time, as part of evaluating user interactions, it’s important not only feedback about the software itself. Another significant task of this group of metrics is to show whether users are satisfied with the process of interaction with the IT team in general and QA in particular.

    1. User satisfaction with IT service
    A regular survey of user satisfaction with IT service with scoring is carried out.

    Purpose of the metric: show whether users trust the IT team, understand how and why its work is organized, how much this work lives up to expectations.

    • The metric can serve as an indicator that it is necessary to focus on optimizing the process or to make it more clear and transparent for users.
    • The satisfaction indicator can be calculated based on the results of a survey based on the results of software delivery. To do this, you need to collect all the grades and calculate the average score.

    2. Customer satisfaction with the product.
    Regular user survey of how satisfied they are with the quality of the product.
    Purpose of the metric: to determine how much the product being developed meets the expectations of users, whether we are moving in the right direction, whether we correctly determine the importance of features, and choose solutions.

    • To calculate this metric, we also conduct a survey of users and calculate the average score. By calculating this indicator on a regular basis, you can monitor the trend of user satisfaction.

    3. Involvement of stakeholders
    Number of initiatives and suggestions for improving the process and product received during the iteration (release) by stakeholders.

    Purpose of the metric: to determine the degree of participation of external stakeholders (business, infrastructure, users, support, etc.) in the work on the product. Having such a metric on hand, you can navigate where you want to get feedback so that one day you do not encounter problems and misunderstanding.

    Pavel Novikov,
    Program Manager
    https://doitsmartly.ru/

    Also popular now: