How to create a testing strategy: the version of real engineers
- Tutorial
Without a testing strategy, you can certainly do if there is an infinite number of qualified employees, time and money. In short, the ability to cut one release over the years. In such hypothetical ideal conditions, no strategy is needed, because you can test your product with all existing methods for as long as you wish, using techniques in any order, for several rounds, and sooner or later, you will come to production-ready quality in some way.
In reality, the project always burns the deadline, the team’s ability to work / skills are not rubber, and the product requirements are constantly evolving - and this is impossible without a good plan. Therefore, a testing strategy comes to the rescue.
In this article, we will offer questions that need to be discussed in order to draw up a testing strategy, and we will show with examples how decisions are made about testing tools in a given case.
Why do you need a testing strategy?
Who makes the strategy?
First and foremost, of course, QA and necessarily the project manager.
So, take by the hand the project manager of the tester or the tester - manager, pour the coffee, take the paper, markers, laptops, and ... let's go!
At the very beginning, we propose to recall all existing types of testing and decide on each: whether we will apply it and why. Let's look at some examples of how you can decide on the need for a specific type of testing.
In any case, even if the application is completely new and is being installed (deployed) for the first time, you need to check how it took off: whether it starts, whether you can start working with it. Whether it is a mobile application, desktop or web.
For web applications it is useful to discuss how we verify that the data after the update is not lost. What behavior we expect from an active session.
For the desktop and mobile applications, you need to think about how to deliver the new distribution to the user and the update process that is started by the user.
For all projects, it is useful to discuss such things as warming up the system: installing directories, setting up users and other data that the user needs to get started with the system.
The decision will be influenced by such factors: how many users will work in the system, how much data will be processed.
A complete check of the entire system always takes a lot of time. Therefore, to make a decision, it is necessary to assess the expediency: how much time is required for a full regression, and the risks that can occur if we do not spend it.
To determine the need for integration testing, it is useful to list all external systems with which the product interacts, and indicate which data we are receiving and transmitting.
Here it is important to answer questions about the project:
We can not verify the correctness of the mobile application on all devices in general. In the case of a web application, the question immediately arises: will we support IE9? Before writing this type of testing into the strategy, we analyze (if the solution is already working) or we assume (if it is not working yet) on what they will use it.
Similarly, it is necessary to disassemble all other types of testing:
other.
On all projects, force majeure occurs, so in such a case it is useful to have a cheat sheet with a premeditated plan, rather than grabbing at random buttons.
In normal mode, meaningful priorities are also important. For example, the development of autotests should be planned taking into account priorities: first of all, the verification of the most critical scenarios (smoke testing) is automated.
It is necessary to think over and coordinate with the team what environments are required for work, and who will use them. As a rule, 2-3 environments are enough.
It makes sense to discuss in advance all the activities that we expect from the tester on the project, because they are not necessarily the same on all projects. For someone in a team, it may come as a surprise that the tester is doing something or not doing something.
This item will help managers understand what degree of competency is required from the tester, and what time load is expected. For existing projects it is also important to indicate that we (QA) are able to let the manager understand the means he has.
For example, these may be:
We need to agree on the shore, and perhaps further and regularly review the format for maintaining test documentation:
Conditions and scenarios of operation depend on the specific solution, so it is necessary to discuss:
Different trackers have different capabilities and types of tasks. Based on the capabilities of the tracker, we advise you to agree on who and by what principle determines the priority of tasks, to what type of tasks to issue bugs and tasks.
Speak with the team key points:
(More information about the ways to describe the bugs and userstory we wrote in this article ).
If the tester takes any task at any time, this is not order, but chaos. Because the task may not be ready. Or ready, but not veneloena. Or ready, well-known, but depends on other tasks that are not yet ready. As a result, the tester spends time and nerves on finding and processing errors, but in fact you just have to wait. Therefore, we agree at what point the QA area of responsibility for the task begins and ends.
In the early stages of project development, the readiness of the task / version is determined by eye.
With increasing self-awareness, we understand that we need clear criteria that the task / version is ready. So that this solution is transparent for everyone, and not “the booty tester senses that everything is normal”. For example, in the conditions of a customized CD, we believe that the task is ready for testing as soon as it is applied to the QA environment and has the status Resolved.
The section is useful in order to put the tasks in charge (to the administrator and the manager) at the planning stage of testing and to get all the necessary tools to get started.
It is worth discussing such issues:
In this section, we propose to agree on what are the KPIs for assessing the performance of the application (in addition to the obvious test coverage count). This indicator should be measurable and achievable. In particular, understand and answer the following questions:
There is no uniform and universal testing strategy suitable for everyone. It is made up individually for each project.
In reality, the project always burns the deadline, the team’s ability to work / skills are not rubber, and the product requirements are constantly evolving - and this is impossible without a good plan. Therefore, a testing strategy comes to the rescue.
In this article, we will offer questions that need to be discussed in order to draw up a testing strategy, and we will show with examples how decisions are made about testing tools in a given case.
0. We will understand on the shore
Why do you need a testing strategy?
- For the organization of the process in conditions of limited resources. Therefore, to begin with, it would be nice to realize what resources we have.
- In order for all project participants to understand the role of testing, what it can give, what profits we will get from this. So that everyone has equal expectations and understanding of what is happening in the field of quality control. And also to identify possible problems that will inevitably become apparent in the process of discussion.
Who makes the strategy?
First and foremost, of course, QA and necessarily the project manager.
So, take by the hand the project manager of the tester or the tester - manager, pour the coffee, take the paper, markers, laptops, and ... let's go!
1. What types of testing will we use on the project and why?
At the very beginning, we propose to recall all existing types of testing and decide on each: whether we will apply it and why. Let's look at some examples of how you can decide on the need for a specific type of testing.
Testing the installation version
In any case, even if the application is completely new and is being installed (deployed) for the first time, you need to check how it took off: whether it starts, whether you can start working with it. Whether it is a mobile application, desktop or web.
For web applications it is useful to discuss how we verify that the data after the update is not lost. What behavior we expect from an active session.
For the desktop and mobile applications, you need to think about how to deliver the new distribution to the user and the update process that is started by the user.
For all projects, it is useful to discuss such things as warming up the system: installing directories, setting up users and other data that the user needs to get started with the system.
Performance testing
The decision will be influenced by such factors: how many users will work in the system, how much data will be processed.
| Given | Decision |
|---|---|
| We know that the product will be used by 10 people. And we know that they will roll tables of several thousand records. | It makes sense to schedule volume testing with large amounts of data. |
| We know that in the process of using the product, users will upload a lot of media files to the customer's server. | It is necessary to find out what load this server is designed for and to have this data in mind. |
| The application is used by several people a week, the amount of data is minimal. | Performance testing is required. |
Regression testing
A complete check of the entire system always takes a lot of time. Therefore, to make a decision, it is necessary to assess the expediency: how much time is required for a full regression, and the risks that can occur if we do not spend it.
| Given | Decision |
|---|---|
| We roll out the first version of the product or module of the program. | Regression testing is not required. |
| The project is very large, and it takes time to complete a set of regressions before the release of new features that is comparable to the development period. The required rate of supply does not allow this. | We decide that regression testing is not carried out, but more attention is paid to other types of testing and monitoring. |
| The interaction between microservices is covered by contract tests. | Conducting a full regression is inappropriate. In the framework of manual testing, we perform a regression of functionality as recommended by the developer. |
Integration testing
To determine the need for integration testing, it is useful to list all external systems with which the product interacts, and indicate which data we are receiving and transmitting.
| Given | Decision |
|---|---|
| Sales data from the portal are transmitted to the analytical system. | It is important not only to verify that sales are gone, gone in the correct format, but also that came in the correct format - nothing was lost along the way. |
Localization testing
Here it is important to answer questions about the project:
- what languages should be supported
- do we envisage the connection of additional localizations during the operation of the solution,
- in which countries will our users work,
- whether there will be sales and in what currencies
- Whether work with time zones is required.
| Given | Decision |
|---|---|
| The system provides 2 languages: Russian and English. | It makes sense to discuss how we will understand the interface in which language to show the user. |
| The customer asks to make the application multilanguage. | It is worth answering the questions how we will verify the texts, where we will get the texts in Arabic, how the screen will be adapted for the text that is written from right to left. |
Cross-browser and cross-platform
We can not verify the correctness of the mobile application on all devices in general. In the case of a web application, the question immediately arises: will we support IE9? Before writing this type of testing into the strategy, we analyze (if the solution is already working) or we assume (if it is not working yet) on what they will use it.
| Given | Decision |
|---|---|
| Users of the mobile app are employees across the country. | We look at the existing statistics on the use of the platforms of our application and make a decision on which of them we will test. |
| Our application has corporate users on stationary machines. | Most likely, we can recommend the use of only one browser. And thereby reduce the time for testing and support for cross-browser compatibility. |
Similarly, it is necessary to disassemble all other types of testing:
- Functional (we carry out or not),
- Acceptance (who conducts it and how),
- UX / UI (what are the objective criteria for a good interface)
- Modular and unit (who writes the contract tests, who writes the unit tests and whether we write them at all),
- Security testing (who guarantees data security and how resistant the system is to cracking)
- Failure and recovery (how the system will behave in emergency circumstances)
other.
2. Priorities in testing
On all projects, force majeure occurs, so in such a case it is useful to have a cheat sheet with a premeditated plan, rather than grabbing at random buttons.
In normal mode, meaningful priorities are also important. For example, the development of autotests should be planned taking into account priorities: first of all, the verification of the most critical scenarios (smoke testing) is automated.
| Given | Decision |
|---|---|
| Our program is used at airports for selling services to passengers when boarding an aircraft. And if at this moment some kind of error happens, then we will not have time for a hot fix. Therefore, errors should not be! | We check the use scenario at the airport first. |
3. Environment for work
It is necessary to think over and coordinate with the team what environments are required for work, and who will use them. As a rule, 2-3 environments are enough.
| Given | Decision |
|---|---|
| On the CD / CI project, smoke-based autotests are written for the initial functionality check. | We need an environment for the primary assembly of the code in one branch, the smoke-test run, where the external systems are closed with plugs. You also need an environment for manual and integration testing with customer services (QA). |
| Beta testing sessions are held regularly. | You need an environment that will be visible "outside", more stable than the QA-environment. |
4. Tester's works on the project
It makes sense to discuss in advance all the activities that we expect from the tester on the project, because they are not necessarily the same on all projects. For someone in a team, it may come as a surprise that the tester is doing something or not doing something.
This item will help managers understand what degree of competency is required from the tester, and what time load is expected. For existing projects it is also important to indicate that we (QA) are able to let the manager understand the means he has.
For example, these may be:
- assessment of the tasks of the sprint for completeness and consistency,
- evaluation of labor costs for testing,
- preparing checklists for testing (or test cases),
- review of autotest scripts
- writing functional autotests,
- direct manual testing,
- deploy changes on the environment
- updating testing strategies, etc.
5. Test documentation on the project
We need to agree on the shore, and perhaps further and regularly review the format for maintaining test documentation:
- what test documentation will we keep on the project,
- will we write test cases
- do we check lists
- and how often to update this documentation.
| Given | Decision |
|---|---|
| About 300 test cases have already been written for approximately one third of the system functionality. They should not only be periodically passed, but also updated from time to time. | In conditions of limited resources, we decided not to work with test cases, but to limit ourselves to checklists for each specific task. As a result, we save time on supporting documentation without losing the quality of testing. |
6. How test scripts are generated
Conditions and scenarios of operation depend on the specific solution, so it is necessary to discuss:
- what techniques of test design makes sense to use. Within this framework, it is useful to compile a list of objects with which the application works, and their equivalence classes.
- what heuristics and banal life experience help to come up with test scenarios for a specific project. For mobile applications, it is convenient to use standard heuristics, for example, “I SLICED UP FUN”.
| Given | Decision |
|---|---|
| On the insurance project, the user (agent) must inspect the machine, during which it is necessary to take photos and upload them to the server. Common sense suggests that there is hardly any wifi in the garages. And sometimes, there is no mobile communication. | So, you need to check the application, not only when working with 3G, but also in the absence of communication as such. |
| Any user action in the airline’s mobile application is part of a scenario. For example, you can not issue a ticket without creating a pre-booking. | Therefore, it is logical to use the “use case” technique. |
7. How to work with the tracker
Different trackers have different capabilities and types of tasks. Based on the capabilities of the tracker, we advise you to agree on who and by what principle determines the priority of tasks, to what type of tasks to issue bugs and tasks.
Speak with the team key points:
- How we will distinguish the errors found by us from the errors found by customers (and whether we should distinguish them)
- How to make improvements
- Do we need to distinguish the improvements that we have invented ourselves from those requested by the customer? How?
- What is the status to put a mistake, if it does not happen again,
- What status is considered final.
| Given | Decision |
|---|---|
| Tester creates a bug. The developer can not reproduce it, translates into the status of rejected. | The tester re-checks the task and sets the status to Closed (final) if the error is really gone, or refines the description and returns to New. |
| The customer sets the task for revision. The developer understands that he does not have enough data to implement. | The developer transfers the task to the status of Feedback. |
(More information about the ways to describe the bugs and userstory we wrote in this article ).
8. Criteria for the start and end of testing
If the tester takes any task at any time, this is not order, but chaos. Because the task may not be ready. Or ready, but not veneloena. Or ready, well-known, but depends on other tasks that are not yet ready. As a result, the tester spends time and nerves on finding and processing errors, but in fact you just have to wait. Therefore, we agree at what point the QA area of responsibility for the task begins and ends.
In the early stages of project development, the readiness of the task / version is determined by eye.
With increasing self-awareness, we understand that we need clear criteria that the task / version is ready. So that this solution is transparent for everyone, and not “the booty tester senses that everything is normal”. For example, in the conditions of a customized CD, we believe that the task is ready for testing as soon as it is applied to the QA environment and has the status Resolved.
| Given | Decision |
|---|---|
| The project has a process that for each revision a check-list is prepared for testing. | We consider the task verified if we have passed all the checklist items. |
| The project is working on sprints. | A sprint is considered ready if all the improvements are in the Closed status and there are no errors with a Normal priority or higher. |
| The project has a list of functionality by priorities. | We believe that the version is ready for release if the functionality from the first items in the list is working successfully. |
| Test cases are written on the project and regression testing is conducted before release. | A version is considered to be ready for release if, following the results of regression testing, no errors with a Normal priority or higher were found (or the percentage of unsuccessful scenarios is not higher than 5). |
9. Tools for work
The section is useful in order to put the tasks in charge (to the administrator and the manager) at the planning stage of testing and to get all the necessary tools to get started.
It is worth discussing such issues:
- What tools do we need for test documentation,
- What tools can test data be prepared for?
- What tools are needed for automation.
| Given | Decision |
|---|---|
| To describe the implemented functionality, we decided to use mind-maps. | The guys in the team work on different platforms, so you need to choose a mind-map file format that you can work with in Windows, Linux, and Mac. |
| We agreed that we need automation on the project. | So, we need tools that will allow us to prepare test data (for example, roll scripts on the database), allow launch within the pipeline (for example, newman), and allow us to create stubs (for example, WireMock). |
10. Project quality assessment and monitoring tools
In this section, we propose to agree on what are the KPIs for assessing the performance of the application (in addition to the obvious test coverage count). This indicator should be measurable and achievable. In particular, understand and answer the following questions:
- How we will track errors in the sale,
- How will we collect feedback from users
- How we will collect statistics on the use of our features.
| Given | Decision |
|---|---|
| After the release of a new feature of the system, monitoring of use is configured (for example, the number of sales of the service). | A drop in performance is a trigger for investigation. Perhaps the feature does not work as expected, or we have not implemented part of the script. |
| We set up in the program monitoring the speed of performing basic operations. | The criterion for the quality of work is that with an influx of users, the speed of execution does not fall. |
Conclusion
There is no uniform and universal testing strategy suitable for everyone. It is made up individually for each project.
- It is important to understand that strategy should not be an end in itself - it is only a tool that allows us to achieve our goals.
- It is quite normal that it is not always possible to immediately answer all the questions posed. This is an occasion to talk again with the customer or users of the product.
- The project is growing, and if yesterday it was a bit early to bother with performance, then tomorrow it might be time. Therefore, after drawing up a strategy, it is important to regularly update it and update it and bring changes to the team.
- After writing a strategy, it usually becomes clear where the failures are in testing and what needs to be done. This is an occasion to set goals, watch the dynamics and ... update the strategy over time.