How to calculate time for testing

... Or, in other words, how to calculate the time for testing so that everyone believes it? After all, in fact, we usually have two goals. The first is to calculate the time so as not to make a mistake and allocate resources correctly - most likely, at first it will not work out well at first. The second goal is more real: to calculate the time for testing in such a way as to prove to someone that you still need people in the team, to explain why you do not have time, etc. Oddly enough, after just 50 do the second, then and the first will come out!

Let's now see how to take the time to test, with specific examples.

Case 1: “Here's the TOR, how long will it take to write test cases?”

Suppose a volume folder comes to us with a well-written technical task (TOR). We are asked how long it will take to write test scripts (TS) for this whole thing.

Before we can answer this question, we should know:

• Who will write the TS?
  
  
  • What is the level of the person who will write the TS: does he have less or more experience than you? If he has less experience, of course, he will need more time. If he has as much experience as you have, then your grades agree. If a person is cooler than you, then it is not at all clear why you were asked to calculate the time.
  • What is the level of technology knowledge ? Will a person be able to conduct low-level testing, it’s good to understand everything, will he understand what is happening with the base, for example?
  • What is the level of knowledge of the subject area ? This is even more important than the level of knowledge of the technology, because if, for example, we need to test some sophisticated financial application, and we don’t understand at all what bonds and options are, it will take a lot of time to understand the subject area. Without this, it is quite difficult to understand how difficult TK is, how complicated the product is, and, as a result, how voluminous and what TS should be in general.
  
  
  
• Why and for whom are vehicles written?
  
  
  • For ourselves - if we test our internal development. We have our own product, our own team, and we write TS specifically for our product.
  • For the customer . Sometimes a client orders only testing, and sometimes only a vehicle. Sometimes he asks to test once, and then leave him an artifact, according to which he can test himself. TS for a client can be less detailed or detailed, but differently than if we wrote a TS for ourselves. So, we ourselves well understand all the terms of testing, but maybe we will need to write down all sorts of complex things from the subject area in the TS. The client may well understand the subject area, but he may not know what form validation is, and he will need to explain it.
  • For acceptance testing - in this case, the TS is even more difficult to write than for the client, because this is testing, the results of which the client may not accept if it does not want to. Therefore, we write TCs for acceptance testing much more formal - there should not be any wording such as “everything works fine,” because on such a TS the client may never accept anything from you.
  • For automation. Here we need to think about data driven testing, how to automate it, etc.
  • For the soul . In this case, we just want to figure it out and there is no request from the testing team.
  
  
  
• Where are they written, in which testing management system?
  
  
  • Familiar environment.
  • Unfamiliar environment. In an unfamiliar environment, of course, we will need more time.
  
  
  

The answer to all these questions depends on how much time it takes to write one positive test case - T1. It is also worth considering the time it takes to write one negative TS (it will be designated as T-1). My experience is that negative TSs usually take longer to write — they also usually need more. Of course, it happens in different ways, but it seems to me that the better we know the subject area, the better we understand the application, the better we understand where and how it can fall, the more we will have negative TS. If, for example, you have one negative vehicle for three positive vehicles, you may misunderstand the application. If for one positive you can come up with 500 negative TS, this is a good job. I have a ratio of positive and negative TS - usually from ½ to 1/5.

That is, we need to calculate the time for writing a positive TS, estimate the ratio of positive and negative, and calculate the time as a whole.

Evaluation Options:

• Rough expert . For example, we know that usually on one page of TK we write 5 TS. Usually we write one vehicle for 20 minutes. In the technical task 300 pages. 5 x 20 x 300 = 30000 - that’s how many minutes offhand we will write the TS.
  
  
  • 
    
    X test cases per page, T1 minutes per vehicle, Y pages (T-1 - negative vehicle)
  
  
  
• Rough deductive . Suppose we have a project plan, we have a deadline, we have a team. And we imagine what the ratio of developers to testers should be in the project. For example, a project for six months. There are five developers (which means we need two testers for these six months). We see that we will write the TS about a quarter of the time from the total time allotted for testing. This is a crude but quick way to count.
  
  
  • Testing takes X from the time of the project.
  • 
    
    Writing test cases Y from time to time for testing
  
  
  
• Inductively experienced . This is a slow assessment method, but it works well. Here we try to write a TS for each part of the TK. It is better to do this with the degree of detail with which we will write them further. However, you can make at least high-level vehicles. It may take a day, two, a week, but it will help you figure out how much time it will take.
  
  
  • We break TK into homogeneous parts (for example, by functionality) and try to write a TS for each part
  • We look at how much time it took to write the TS for a certain number of pages of each part, based on this we calculate how much all the time it takes to write the TS for each part
  • Add up the time calculated for each part
    
    
    
    
  
  

Example

Let's now see how these techniques work, using a concrete example. For example, we write a social network for pets. We have TK for this social network, only 47 pages. Suppose we split it into homogeneous parts. Here is what we ultimately have:

• TK :
  
  
  • Host Account (5 pages).
  • Animal profile (15 pages).
  • Photos, videos (12 pages).
  • Friends (8 pages).
  • Messages (5 pages).
  • Non-functional requirements (2 pages, 12 pieces).
• Programmers : four people.
• Duration : six months.
• Rough expert assessment (accepted by our company): two positive TS per page, 0.5 hours - for a positive TS, 45 min. - negative

Now let's try to calculate how many hours it will take us about writing a TS (but, pay attention, not for testing at all):

• Rough expert assessment : (0.5 + 3 x 0.75) x 2 x 47 = 258 hours.
  
  Comment: negative TS here are 3 to one positive. Two positive TS on one page, only 47 pages. 0.75 is 45 minutes.
• Rough deductive rating : (176 x 4 x 6) / 3/4 = 352 hours.
  
  Comment: 176 hours a month, four programmers for six months, divided into three (the ratio of programmers to testers is approximately 1.5 testers per project (4 / 1.5 ≈ 3)) and divided by four (a quarter of the time from the entire testing will be go to write a vehicle).
• Inductively experienced rating : (1 x 10) + (3 x 15) + (6 x 12) + (2 x 16) + (1 x 10) + (3 x 12) = 205
  
  Comment: we spent an hour on the host account , managed to cover half the page, and we have 10 of these halves (1x10). We spent 3 hours on the profile of the animal, in three hours we covered one page (3x15). We spent 6 hours covering one page of the photo / video (so much because here, let's say, everything is very complicated). 2 hours spent on friends and during this time made half a page (2x16). 1 hour spent on messages, done half an page in an hour (1x10). Here we decided to look at non-functional requirements not on the pages, but on the requirements themselves: we spent 3 hours on one requirement (3x12). All this took us 16 hours, which means that in 2 days we estimated the time at 205 hours.

Our estimates do not agree, but this is normal - at least we were able to calculate approximately. And if we have enough time to evaluate, we can make an assessment in several ways.

Counting time when using Agile

Now let's talk about flexible development methodologies when there is no technical task. It is not very clear how to evaluate, but nevertheless, some approaches can be distinguished. For example, you can:

• just go through the backlog;
• write high-level TS;
• write out all the actions inside the sprint and try to evaluate how much they will take time - this approach will be discussed below.

Digression for QA-managers

Now a small digression for QA-managers, project managers. All actions for creating and launching test cases can be divided into the following components:

CREATING A CU	STARTING THE HARDWARE
Environment Preparation (TCenv)	Fulfillment of Preconditions (TRprec)
Vehicle structure preparation (TCstr)	Run regression tests (TRreg)
Creating TS for ready-made functionality (TCbase)	Running tests for new functionality (TRnew)
Creating TS for new functionality (TCnew)	Build New Configurations (Cnew)
Automated Test Creation (TCa)	Number of configurations (Qconf)
Modifying created TS (TCchng)	Retest after fixing bugs (TRrtst)
Summary: ΣTC = TCenv + TCstr + TCbase + TCnew + (TCbase + TCnew) x TCa + TCchng	Result: ΣTR = (TRprec + TRreg + TRnew) x Qconf + TRrtst x Qconf + Cnew

How much time does testing take then, we see a table in the formulas below - everything is clear in them and without comment.

Case 2: “Here is a piece of functionality, how long will it take to test?”

How long will it take to test this or that part of the functionality? To figure this out, there are different ways:

• 
  You can see how much time it usually takes us to test similar functionality.
• You can look at the same functionality if we have already tested this one. It is very comfortable! If you, of course, wrote down the last time, how much time it took.
  
  
  
• Deductive evaluation - if we know how much this functional takes up places in the general functional (if, for example, we already know how long it takes for a full regression), we just see that testing this functional will take, for example, a fifth of the entire time.
  
  
  
  
• Inductive assessment - we remember how much time we spend on each vehicle. We look at this functionality and understand that we need 10 - 15 TS to test it.
  
  
  
  

What is important to remember in this case? That new functionality may again require time to prepare the environment. It is important to remember that there is a coefficient for getting to know the system: more time is spent on tests if you run them the first time than if you run them the second or third time. But if the second time comes in a month, then you still drive it away, as if for the first time - however, this is different for everyone.

Case 3: “Why don't you have time for anything ?!”

The next very common question asked by testers is: “what do you do at all, why don't you manage to do anything, why do you need to help with automation, why don't you want us to release new versions every day - there aren’t any new bugs ? .. ”

To show what we are doing, for each project we can write out all the tasks that are performed in one sprint and paint them by day: calculate how much time each task takes on a given day. So you can see the average, maximum and minimum load.

An example of such a table:



When I did it, I saw that the load may not always be optimally distributed - sometimes the load is large, and sometimes half a day there is nothing to do. This allows you to draw certain conclusions and somehow optimize the process, distribute the load - for example, to do some things in advance, even if this is not entirely consistent with the logic of the process. For example, if at the beginning of a sprint we have a lot of free time, we can already start writing custom stories for the next sprint. Although we don’t know the structure of the next sprint, we can most likely predict that these two or three stories will definitely enter it — they are usually the most critical, they usually have to be rewritten anyway, so this is a good exercise.

So:

1. Defined a list of tasks.
2. Defined "input parameters".
3. We determined the time for each task.
4. Smeared tasks for the sprint.

Got the download in hours by day. And if it turns out that every day you need 24 hours of testing, and we have only two people ... So, we definitely need a third tester!

Case 4.1: “We release the assembly on Friday, or even earlier - so how much time do you need for testing”?

In this case, the testing process is built as follows:

1. Smoke test + test for full regression x on the number of critical platforms + test for basic functionality on all platforms + tests for full regression on platforms (see matrix).
2. Retesting : smoke test + validation x all platforms.

Tests for full regression on platforms are carried out in this case as follows (Cr, FF, etc. here are different platforms, in this case, browsers):

	Cr	Ff	IE9	IE8	Cr '	Ff '	Σ
Smoke test	1	1	1	1	1	1	6
Core functionality	1	1	1	1	1	1	6
Administration	1						1
Statistics for admins		1					1
User account			1				1
User posts				1			1
Friends					1		1
User photos						1	1
User Files	1						1
User Settings		1					1

Using this matrix, we see that for each platform we are testing some part of the functionality: in the end, it turns out that we checked all the functionality and tested the work on each platform. We can also shift this matrix - in the end it turns out that for a certain number of runs, we will still check each function on each platform. This is still a simple scheme - and there may be 20 configurations without the ability to automate their testing - of course, in this case we will not have time to check everything, and we will test using such a matrix.

As a result, we obtain the following formula:

ΣT = (Tprec + Treg) x (Qmain + 1) + (Tprec + Tbase + Tsmoke) x Qconf + (Tsmoke + Tvalidation) x Qconf x Qreturns

Qmain - the number of main configurations on which we are testing everything.

Qreturns - the number of returns.

According to this formula, it is clear how much time it will take us. On it, you can try to keep track of time, look at which testing takes the most time. It will also be clear where and what is better to automate - we will understand where automation will save us the most time.

Case 4.2: “Too many tests! Hurry up".

If we are required to perform testing faster, we can cut the number of tests according to the following principles:

• We look at the priorities:
  
  • test cases - if we paint vehicle priorities. Then we will work only with the highest priority TS;
  • user stories or bugs if our vehicles are attached to them.
• We look at the release notes, which includes :
  
  • критически важный функционал, без которого приложение не будет работать и никому не нужно;
  • заявленный новый функционал — так как его публично объявляют, пользователи его обязательно проверят;
  • исправления критических ошибок (скорее всего, ошибки будут уже проверены);
  • заявленные исправления ошибок — мы заявили, что мы что-то исправили, и мы это проверяем;
  • известные проблемы (known issues) – как ни парадоксально, но мы обязательно должны перетестировать те ошибки, которые мы заявили как ошибки, с которыми мы выпустили продукт. Зачем это делать? Во-первых, нужно проверить, что их воздействие ровно такое же, какое мы описали. Во-вторых, мы проверяем, работает ли тот обход этой ошибки, который мы предлагаем.
• Instead of reducing the amount of testing, you can increase the command :
  
  • Adding programmers is best from your own project. For example, they can write data generators and more - all for automation or semi-automation. They do it well, and the responsibility for testing is still not on them - therefore, they do not suffer from the fact that their code does not work well. For the same reason, you can ask some programmers to check the code of other programmers.
  • Call testers from other projects, or from this project, but engaged in something else.

In general, the real testing process can be represented as a triangle. The angles of this triangle are:

1. Scope of testing.
2. The number of people in the team.
3. The time period for which testing is required.

The speed and quality of testing depend on these three aspects - you need to think about them in order to successfully plan your work. So, for example, if we are offered to test a product for an unrealistically short time, and we definitely won’t meet it, or refuse such work (if we don’t want to release a low-quality product, of course), or we work with the above aspects. Then we can:

1. Reduce testing.
2. Increase the number of people in the team.
3. Ask to increase the period for testing.

The latter method is perhaps the most reliable: after all, people can be given inept to the team, and a reduction in the testing volume can lead to the release of a crude product.

Also, a method works here - ask why such a deadline. Maybe you need only one function for one client - then we will make him a private release, and later we will test everything else.

And one last thing: finalizing an already released project

If the project has already been released, but there has been a demand to finalize it, then how to calculate the time for testing, development, etc.? In this case, we calculate the time as usual, but at the same time we multiply it by two, because all people have already forgotten everything, already in other projects - it takes time to get acquainted with the system. In addition, the addition may be outside the standard cycle, therefore, in addition to testing this addition, a complete regression may be necessary - then we add time to a complete regression.

Posted by Eugene Efimov, QA Lead