Price quality: 7 principles for optimizing the cost of testing

Thinking how to save on testing your software? You are not alone. There is only one small but: if the software is not tested, the most negative scenarios are possible - from expensive and extremely unfavorable for you to refine the application in the late stages to loss of reputation and customer care / customers to competitors.

Ready to take on the staff of the 50 most experienced testers to ensure the quality of the product? That's cool! What for? You need to understand: if you allocate too much resources for testing in cases where it is unreasonable, you will inflate the budget and the software will be too expensive. Are your users and customers happy? You risk again.

Yes, we are hinting that the truth is somewhere in the middle. In this article we will talk about the basic principles, following which you can find a balance between the cost of testing and the quality of your product.

Principle number 1. Start testing as early as possible.

One of the most common mistakes is to start testing the product at a later stage, when it is almost ready for release. The earlier the testing team (QA) connects to the development process, the lower the probability of missing an error in production. In addition, an error identified at an early stage of development will be cheaper. Factor of. Our experience shows that the fix price at the later stages can be 30 times more expensive than the fix for the same error, for example, at the prototype stage.


A case from practice: we waited a very long time in testing a server application that collected and processed a large array of user data. The developers simply refused to give it, arguing that they have a lot of problems, and testing can be done at the end. As a result, they worked on the new version for four months.

When we started testing the application with real user data, it turned out that the selected database architecture could not cope with such a load. A team of five people had to re-plan everything, rewrite the design and architecture of the application, and we had to test it from scratch. The release took place six weeks later than planned.

Total:the desire to save 150 man-hours for testing and to postpone it to release led to the loss of 1,100 man-hours of its employees and the double cost of testing.

Principle number 2. Save, but not on analytics

Suppose you listened to our arguments and decided to implement the first principle. At what stage of product development is it most advantageous to fork out and connect the testing team? Answer: at the planning stage of the application architecture, or at least at the stage of analysis and preparation of requirements.


Without well-conducted analytics, you risk putting the perfect product on the market that nobody needs. This happens when you do not take into account the peculiarities of your target audience.

We remember the horror story about the application for running, which was sharpened by users from Asia. Central Asia would definitely like it and make a profit if customers could install the application on their smartphones. But they could not, and you know why? At the stage of requirements collection, an error was made, due to which the application was tested on Samsung iPhones and smartphones, which in Asia is at least. After all, Asian users definitely prefer Huawei and OPPO.

In our example, saving the developer again failed. Not only did it have to spend 300 man-hours to finalize and test the product, so also the reputational costs almost put the project on its knees.

Bottom line: you can invite experts from the side or limit yourself, the main thing is to remember that consumer testing at the support stage costs ten times more. And by more expensive we mean not so much monetary losses as reputational costs. Do not save on analytics. Only not on it!

Principle number 3. Identify goals and expectations

Of course, you want everything to work, and work well. But this is not enough. You need to understand how good and why so much.


Do you know how to recognize a good test manager? He immediately begins to "torture" you in order to understand exactly what quality of product you need. We call it the collection of expectations. The quality of your software is made up of certain building blocks. An experienced test manager will want to know what kind of building blocks to put in the foundation of quality.

When a customer comes to us and says: “I want it to work,” we begin to ask him: what is the main purpose of creating a product? What features are most requested by users? how do they use them? After collecting the expectations, we set up SMART goals, decompose them, build tasks and KPI tables. As a result, we clearly define:

• types of testing;
• dates of their holding;
• line-up;
• and even possible risks.

All this is necessary to ensure the desired quality of the customer, without guessing on his nerves.

Sometimes, however, incidents occur. On the project of the online store of insurance services, we conducted an analysis and prepared a proposal. Due to the complexity of the functional and the size of the modules, in addition to the full functional and regression testing of the product, we recommended to automate the regression. Despite the fact that it was precisely the regress that left most of the time, the customer refused to automate it, motivating it with the fact that there is a strong “handbrake” team, and the automation will take a lot of time and become unprofitable. All our further arguments and calculations were ignored. We could not prove our position and for three months we tested the regression manually, which, in fact, was unprofitable for the customer.

Then the project was curtailed, two months analyzed and returned with a proposal to use the automation plan, which we proposed five months ago.

The bottom line: you need to be able not only to understand the goals of the project and the client’s wishes, but also to prove to him that you are on his side and know how to get benefits. Otherwise, it will be like ours: a budget that is merged into manual regression testing, although automation would save both time and money. Speaking of automation ...

Principle number 4. Automate

... But wisely. And pre-calculating ROI.

For automation to save your money, the product must be stable. However, if it changes very dynamically, it does not mean that automation will not help you. The use of bodies and specialized programs is already automation. For example, you can help automate routine operations that eat up the time of your testers. Say, write a utility that collects the necessary configurations.

Case from practice:the customer wanted to be released every day instead of two times a week. At the same time full manual regression testing took two to three days. And since 80% of the product's functionality was well-established and stable, it was decided to automate it. To do this, we developed four automation scenarios and conducted a comparative analysis of their effectiveness. Usually for this we use eight indicators:

1. TC's amount - the number of cases in the script.
2. Automation (man * days) - resource costs for script automation (excluding test data for each test case).
3. 1 TC automation (man * hrs) - the cost of automating a single test case.
4. Manual testing (man * days) - the cost of manual testing of the script as a whole.
5. Results investigation (man * hrs) - the cost of researching the results of the AutoTest run.
6. Execution times - the number of test runs required for the period of work on the project. This number reflects the expected number of runs, taking into account the stability of the functional, the required schedule of test runs (regularity of obtaining information), etc.
7. Automation efficacy (%) - the effectiveness of test automation in% of time saved. Given the errors of computation, automation can be recognized as effective with an indicator of more than 150%.
8. Saved time (man * days) - the number of man-days saved during the entire project.

Here is how it looks on the example of our project: Two automation scenarios passed the final selection. Their use has allowed us to reduce the regression from two days to a couple of hours for running autotests, plus two hours for the remaining manual checks, which were unprofitable to automate. Bottom line: automation can reduce labor costs for testing tenfold, and can drain the budget in vain. Without the ability to analyze and calculate ROI, you always risk forever giving up on automation.

Principle number 5. Learn to use newbies.

Do not want to automate? Are testers with experience hitting the budget? Are you thinking of hiring graduate students for a small but reasonable salary? Not a bad idea, but there are a few nuances. Even if you have a simple product, some newcomers will not be enough for you.

A child may find a bug in the game and break the application using the monkey testing method, but will he find all the bugs? Of course not. We love newbies very much, but they, as a rule, do not know the methods of research testing. But they can "walk" on the tests. Therefore, the company of beginners with burning eyes needs an “old man” who will prescribe checks at a level that your beginners will understand.

For example, we attract newbies under the supervision of mentors even to some complex projects. Initially, the process of integration and adaptation of a newbie on the project took a month, which, of course, increased the time and cost of testing. To solve this problem, we have developed a "newbie packages" with test documentation and all the instructions for installing and configuring the necessary components. This step made it possible to shorten the period of full adaptation, first up to two weeks, and after adding a set of visual cases and training videos to the “package” - up to a week.

A fragment of a set of cases for testing the insurance product is presented below.


A “novice package” can include something quite simple but useful, such as a data generator .

And it can give concrete steps for passing cases. By the way, in our experience, it is best to give test cases to beginners (with detailed steps and preconditions), rather than checklists. Having such a set at hand, a beginner will always know what test to take, what to do and where to look for the missing information. The bottom line: it is profitable to use the work of newbies without loss of quality and speed of testing is quite possible. Junior is able to bring real benefits on the project already from the second week, reducing the time spent on adaptation by 4 times (from 160 to 40 hours).







An experienced mentor and newbies integration business processes will help grow a tough team in the shortest possible time. We regularly do this here, and the number of class testers we have prepared has already exceeded one thousand!

Principle number 6. You do not need a hundred testers, you need those who work head

Testers should not be much, they should be enough. You do not want to spend money on great experienced guys who will just be bored at work or get nobody to use the necessary minors? In the same way, you don’t want the forty developers to overwhelm one unfortunate testing specialist with all their fixes and he didn’t have enough time or energy to do quality work on the tasks. So, the sixth rule of price and quality balance is to look for this balance.

Intuition is important, but still try to use the help of strategists and analysts to help you assess the amount of tasks, the number of people you need and their qualifications.

A common case of practice.Not so long ago, we were approached by a client who had 12 testers in the team: TM, senior, middle and 9 junior. We conducted an audit of the testing processes and abandoned eight thousand unnecessary tasks (such as regression of unaffected functionality, establishment of minors, etc.), plus we figured out how to optimize the tests. It turned out that the project needs three automatizers (offered their own) and another middling that was raised from among junior. From the rest of the newcomers had to be abandoned.


Next, we left the testing of critical functionality in the “Identification and Matching” block on the middle. They checked it by hand, research testing. All other blocks are almost completely gone to the automation.


As a result, it allowed:

- To reduce the cost of pre-release testing from 232,400 to 35,200 rubles.

- Increase ROI testing by automating 5 times.

- Reduce the management burden on management.

- Reduce pre-release testing time by 23 man-hours.

- Improve the quality of testing, leaving only experienced testers on the project.

Principle number 7. Calculate what is more profitable for you: staff or outsourcing

Often, but not always, outsourcing is cheaper and allows you to hire more experienced employees at a discount.

Another case from our practice: How to save more than a half million rubles a year, instead of hiring nine full-time testers with a salary of 70 thousand rubles, nine outsourcers with a salary of 130 thousand.

In order not to be unfounded, let us show a table that explains where this benefit comes from.


In our example, a full-time tester with a salary of 70 thousand rubles per month costs the company 14,877 rubles more than a more expensive outsourcing specialist, who has almost twice as much salary. If we take into account the department of 9 people who have worked for a year, then the benefit will be 1,606,716 rubles. And this is money.

However, sometimes it happens that, even realizing the benefits, companies do not want to give testing to outsourcing. For example, we often encounter the desire of the client to transfer everything to his state. This is due to the fears and unwillingness to share their own experience and documentation with external customers.

This position is fully justified:despite all nondisclosure agreements, there always remains the risk of accidental information leakage, the cost of which is not covered by any lawsuits later. In order not to get into such a situation, we recommend to play it safe and contact the company, for which over the past ten years, no such lawsuit has been filed, having first communicated with the clients stated on its website.

Shortly told

1. According to the National Institute of Standards and Technology, the cost of testing at the end of development may exceed its cost at the initial stages 15 times, and after release 30 times.

Do not want to overpay? Start testing as early as possible!

2. Misunderstanding of our Central Asia and its product requirements buried many excellent projects.

Save on analytics? Get ready to pay for the game of guessing game!

3. It is necessary to analyze not only the target audience, but also the wishes of customers. On their basis, you need to be able to set and decompose SMART targets.

Do not know how to set clear goals? Your goals are not clear to the customer? Mortgage costs for refinement and rework!

4. Automation helps optimize testing costs. But not everywhere and not always.

Do you think that hands testing is more profitable? Calculate ROI from automation in several scenarios and compare the numbers!

5. Newbies in testing are enthusiastic, but they lack experience. A good mentor and “newcomer package” increases the efficiency of the juna on the project by 2-3 times.

Scored yesterday's graduates for a reasonable fee? Get ready to pay extra for their transformation from a "monkey" into a man!

6. Quality is not equal to quantity. A good specialist is more expensive because he gives the best result.

Do not want to save, working on the result? Then you should not audit the testing team and optimize its composition.

7. Outsourcing testing allows you to get an experienced team for a project with a guaranteed result. But outsourcing is not for everyone.

Хотите готовое решение с демократичным ценником? Рассчитайте преимущества от штата и от аутсорса, возможно второй вариант окажется в разы выгоднее.

Вместо эпилога

Quality is a multifaceted concept, and therefore its price may vary depending on the depth of understanding by the company of this term. There are many ways to save at the expense of quality, but there are not many options for improving quality without overpaying. Compliance with the above principles has helped us earn a reliable reputation in the market and the loyalty of our customers, and by the clients themselves - to save time, resources and time.

Try these principles on yourself, they are very simple, and therefore effective. We hope our article will help you save on testing without reducing quality. And we are waiting for your questions, ideas, suggestions and comments in the comments.

Nina Ageeva,
Deputy Director at the Laboratory of Quality.