Manual Application Testing Guide: Benefits, Milestones, and Methodologies
We analyze in detail how to conduct manual testing, when it is better automated, what a tester needs and how he can build his career from junior to test lead. The guide was prepared jointly with the head of the testing department of Agima Darina Gordeeva.
Hello! My name is Darina Gordeeva. I have been working at AGIMA as a head of the department for almost 3 years. In the field of testing and quality assurance over 6 years. During this time, I went from junior to head of the department, doing hardware testing, as well as mobile and web applications, automating and setting up QA processes. Today I will tell you about the features, capabilities and hidden problems of manual testing.
Recall that any reader who takes advantage of the Habr promise can receive a discount of 10,000 rubles for the course they like, while the most diligent and attentive can collect a 25,000 rubles discount for themselves, solving puzzles from materials prepared in conjunction with Agima:
Today there are many testing frameworks that support almost all existing languages. It would seem - you can take and automate. But even now, manual tests are important.
One explanation for their need is that with manual testing of the functionality, we can get information on the state of the product that we are analyzing, on the quality of development, much faster. In addition, during automation, previously developed cases often have to be changed and updated, and it takes some time to write autotests.
In the development process, feedback from the customer may come when he sees some function in the finished product that he decides to change before the release - and if you have already prepared software tests for it, they will have to be rewritten. Updating cases, autotests and checking them will take up valuable time that could be used to update this feature.
All this means that the main purpose of the manual tests is to pre-verify that the claimed functionality is functional, has no errors and provides the expected, planned results. Without them, you can not be sure that you can continue to work. This is especially true for the functions for which the subsequent development is tied. In this case, fussing with the creation of autotests for these features becomes a blocking factor for the entire product development, shifting deadlines and breaking deadlines. The moment when the cases will come to automate, anyway, sooner or later will come - but you should not strive to bring it artificially in pursuit of the total exception of manual labor.
In addition to this, in the early stages of application development, automation can be quite expensive. You will need a specialist with specific qualifications (and, possibly, not one). Constant maintenance of tests up to date requires resource expenditures up to the release of features. And the months of inactivity, devoted to licking the autotest will hit the motivation of the team.
This is especially true for mobile development. Most of these projects today are being developed in short sprints, which means that features are being introduced at an accelerated pace. In such conditions, manual testing allows us to give the team as quickly as possible a feedback: to report the presence of bugs - or to please her with the fact that everything is okay and you can move on. You can carry out a series of autotests later, covering with them large arrays of code. Manual testing will help prepare cases for this test.
Self-tests do not allow to check whether it is convenient to use the new features of the application - usability testing is always carried out manually.
In manual tests, you can improvise, creating insane combinations of actions that never occur to the user, but can be committed by him randomly. This allows you to create new cases.
New cases appear also because the tester constantly asks himself the question “what if?”. So he finds original ways to interact with the application - even if they were not in the baseline scenarios.
The biggest disadvantage of manual testing is the human factor. He, of course, influences everything that happens in a team - both the work of the participants and the events taking place on the client side. In the case of QA, the reason that the tester will be poorly immersed in the process and miss a potential error can be anything - a lack of experience, family problems or a headache.
Two testers can test the same scenario in different ways. What to do with it? It is important that each unintended or different from the expected result be recorded in the form of a case. So that any tester could test it by performing the same set of actions. But this may not be enough - if the case is not detailed enough, and the tester does not figure it out. Of course, it is impossible to guarantee 100% exclusion of the human factor - but you can try to minimize the probability of the problems it causes.
This also has a negative impact on the delivery time of features in production and labor costs: after all, now, “cunning” cases, invented by testers in the process, are added to periodically held basic cases and regressions.
The situation is aggravated by the likelihood that some of the bugs encountered will not yet have a strict description because their causes are not yet clear. How to deal with such repeated testing? Either find the source of the error and eliminate it, or - all the same to automate the passage of problem cases - in this case, the transition to software tests will be fully justified both in terms of time and financially. (No, this does not contradict the above - because such situations arise already during active development and by this time you will, in any case, launch the auto-testing process).
In any case, the appearance of the first cases needing regression tests or the release of the second version of the application and the scale of the command corresponding to these events is the moment when the automation becomes relevant (but does not exclude manual testing of new features). At this stage, the automation will start saving time: the developer will be able to start the regression tests of the new feature even before the transfer to the QA department to make sure that it does not break the existing functionality, and the tester does not have to go through the core-cracks. Another advantage of introducing autotests at this stage is that they can be launched according to a specific schedule - every night, on the days when sprints end or when new application assemblies are published.
In this case, we must not forget a few things:
If you look at the development process as a whole and at testing as one of its parts, then with proper planning you will always understand what and when it will be ready. This will allow you to better plan the time of certain actions - since some events logically follow others and you have the opportunity to arrange them in chains based on your expectations.
The tester appears in the process of creating an application at an early stage. A client has a certain idea, business analysts collect from this requirement - and testers start working at this moment, checking these requirements.
How does this happen? They ask questions about the intended functionality. Trying to imagine how the application will look when it is implemented. If we are talking about a new feature in an already existing product, they understand how it will interact with the existing functionality. After the developers have evaluated the workload of the client’s ideas and determined how long it will take to implement them.
After that, the design phase begins. Here there is a need to understand how the transitions between screens will be carried out, certain fields validated, how an application or its individual function will generally interact with the end user. In the case of features, it is important to understand how they will fit into the architecture of an existing application.
At the design stagewhen a transition map between screens is created, the tester clarifies the behavior of the individual elements of which they are composed and what visual effects will be followed by those or other user actions. In addition, the tester validates the layouts for completeness, confirming that they reflect everything that is needed to implement the conceived functionality. Self-checking of layouts for compliance with guidelines will also be useful.
With the start of development, QA specialists immediately start writing test cases. At different stages of development, they may have different levels of detail, but by its completion it is desirable to ensure maximum coverage of the entire product with cases so that, upon receiving the assembly, you can start testing immediately.
The first step of direct testing is the Smoke test: assessment that the application or its new part is generally ready for testing. A smoke test is a test of whether an application or a particular function is running in principle.
Smoke test is a quick way to see if we can even begin a functional test. The term came from the creators of boards and microcircuits, which, for a start, had to make sure that the new circuit would not burn - hence the name: smoked / not smoked.
This is a quick way to make sure that we really have the task and we can take it to work, and not send it back to programmers.
The next step is to conduct regression tests. In manual or automatic mode, the main pre-planned array of tests is conducted.
Regression testing is good because it allows you to find errors even in those places where everything was in order. This is due to the fact that regression is an assessment of the functionality of a standard set of cases when implementing each new module and each application change. After all, when developers add a new functionality, the current version may be damaged or a new feature may conflict with existing ones.
For example, adding new screens, and thus changing the navigation may disrupt the menu or at least display it. On the other hand, a global refactoring of the application code can also bring unpleasant surprises - after it, it is also necessary to carry out regression tests.
Problems can cause and update the library used by the application and change the environment in which it works. The more often you update the application, the more important is the role of regression tests. You should not limit yourself to a one-time check carried out when the feature is already implemented - check all changes. Here automation of regression testing will help you - simply because manual regression testing of a new feature created in just a week can take two, or even more, and the testing department will simply get bogged down in this.
A full check, of course, is carried out when a release candidate with one or several features is ready to roll out in production, but it is still necessary to apply one or another relevant cases at separate stages of development.
Everything ends with testing the final build - release candidate. It includes beta testing by internal testers, business testing — evaluating the resulting product by the customer and receiving feedback from him, as well as offering a specific group of users to get acquainted with the pre-release alpha version of the application or its new features. After that, the application is ready to roll it into production.
But the work of a QA-specialist does not end there.- it will test application updates and their compatibility with earlier versions, create cases for testing innovations and, if necessary, automate the passage of these scenarios.
In parallel with this, testers are involved in further analyzing statistics collected by analysts, monitoring application behavior and how users interact with it. This allows not only to see the live use of the results of their work, but also, at times, to discover new scenarios and unknown bugs that cause the application to crash.
Rebus time: guess it and a 10% discount on any of the Skillbox courses is waiting for you right now or be diligent and collect answers to all the puzzles - these discounts are added together (but with no other discounts on Skillbox courses).
However, it is not necessary to delay too much - the promo works until August 30, 2018. Recall that the subject of the riddle is mobile, and English words can interfere with the Russians here, so be careful! Send a request for the course, and when the manager contacts you - call him a pryword encrypted in the rebus (or a few!).
In order to prepare for functional testing, a QA engineer prepares a test plan. This is the documentation that he will need when testing the product, a list of actions that he will need to perform. The test plan indicates the time of testing, describes the product or a specific task - in order to determine exactly what needs to be checked. A detailed description of all the main test cases. Lists the types of testing that are needed: functional and, for example, load. Describes how to automate certain cases and then at what stages they will be automated.
The test plan ends with a description of the report format: you need to explain in advance how the result will be presented. The most common report format is a list of test cases with the status of their passage. Knowing how much the cases cover the entire volume of requirements and reading the report, it will be possible to draw a conclusion about the current state of the application development. To do this, it will be enough to see how many of them have been successfully completed in one or another assembly.
The final sign that the product is ready is the status “all requirements are covered with cases, all cases are passed successfully”.
In addition, the error report format is formalized in the test plan. This may be a bug list, a list of tasks in the tracker.
First of all, the tester should be able to think and be attentive and diligent. Experience is important - it allows you to accumulate certain achievements and consolidate knowledge of the testing processes, turning them into skills.
A manual test specialist doesn’t have to be able to write code and have a deep understanding of architecture. This does not prevent him from checking the functionality and at the application level to understand the principles of interaction between the application and the server.
Automated testing specialist is a separate profession, with a completely different set of knowledge. High-quality auto-testing is not just scripts, but also an understanding of how an application is built from within, and specific skills, such as knowing how to parallelize tests or how to run them in the cloud or on several servers. Only such a pool of skills allows you to proudly call yourself an “automaton with a capital letter.” So, without knowledge of languages, their frameworks, principles of OOP and attentive surveillance of the development of technologies, this auto tester is indispensable.
The path of each tester begins with mastering the theoretical basis of testing, obtaining primary ideas about how an application interacts with the server and with the environment. If this knowledge is there, and with it a person has a very serious intention to learn, he can already be considered a junior tester. A group of such newcomers with burning eyes is assigned to a senior specialist - either a leading tester, or, if the company can afford it, a coach who purposefully pumps his players. They explain incomprehensible moments to the juniors and give painful tasks, like running features on test cases. In the process, a person learns to read test cases and masters the practical basics of functional testing. At this stage, the juniors still need to check the quality of the tests they conducted, passing them after them.
The next step is to create an individual (or collective, depending on the size of the company) development plan, according to which a novice tester will have to develop, mastering new knowledge and achieving certain results within the time allotted to it. That is the way to become a middle-level specialist.
A middle tester is a person who is already able to independently carry out the tasks assigned to him, to find solutions and generally to do his work consciously, and not blindly following the established algorithms.
With the development of skills in test design, knowledge in the functional and applied areas, as well as the development of new testing methods - which now need to be developed independently - there is a gradual transformation into a leading specialist. Now he can be charged with the management and support of the same juniors as he himself was before.
The next level is a test lead. He can already manage a team that includes representatives of all three previous categories, the processes of which he manages and the time of which he manages, including participating in planning global development processes, estimating labor costs and budgeting for his teams.
Further vertical growth of team lead is the management of the department or the transition to product management. If, however, a person seeks new knowledge, and not new administrative positions, then he can choose to develop, analyze, or such a direction as DevOps, combining the duties of a system administrator, tester and programmer.
Testing is only one of the many areas of mobile development that we consider as part of the “Fullstack Mobile Developer” course. The industry professionals who work with us will share their experience with you and test your homework, and the result of training may be an internship for a large company. Come to us to study! Skillbox recommends:
Hello! My name is Darina Gordeeva. I have been working at AGIMA as a head of the department for almost 3 years. In the field of testing and quality assurance over 6 years. During this time, I went from junior to head of the department, doing hardware testing, as well as mobile and web applications, automating and setting up QA processes. Today I will tell you about the features, capabilities and hidden problems of manual testing.
Recall that any reader who takes advantage of the Habr promise can receive a discount of 10,000 rubles for the course they like, while the most diligent and attentive can collect a 25,000 rubles discount for themselves, solving puzzles from materials prepared in conjunction with Agima:
- How to get to the AppStore from the first time: step by step guide ;
- 8 stages of the development process of a mobile application interface ;
- A / V tests do not work. Check what you are doing wrong .
Efficiency, flexibility, the possibility of improvisation and other advantages
Today there are many testing frameworks that support almost all existing languages. It would seem - you can take and automate. But even now, manual tests are important.
One explanation for their need is that with manual testing of the functionality, we can get information on the state of the product that we are analyzing, on the quality of development, much faster. In addition, during automation, previously developed cases often have to be changed and updated, and it takes some time to write autotests.
In the development process, feedback from the customer may come when he sees some function in the finished product that he decides to change before the release - and if you have already prepared software tests for it, they will have to be rewritten. Updating cases, autotests and checking them will take up valuable time that could be used to update this feature.
All this means that the main purpose of the manual tests is to pre-verify that the claimed functionality is functional, has no errors and provides the expected, planned results. Without them, you can not be sure that you can continue to work. This is especially true for the functions for which the subsequent development is tied. In this case, fussing with the creation of autotests for these features becomes a blocking factor for the entire product development, shifting deadlines and breaking deadlines. The moment when the cases will come to automate, anyway, sooner or later will come - but you should not strive to bring it artificially in pursuit of the total exception of manual labor.
In addition to this, in the early stages of application development, automation can be quite expensive. You will need a specialist with specific qualifications (and, possibly, not one). Constant maintenance of tests up to date requires resource expenditures up to the release of features. And the months of inactivity, devoted to licking the autotest will hit the motivation of the team.
If you want to regularly add new features and keep up with the actions of competitors, then before creating autotests always check the capabilities of the product manually. Just because manual testing speeds up your processes.
This is especially true for mobile development. Most of these projects today are being developed in short sprints, which means that features are being introduced at an accelerated pace. In such conditions, manual testing allows us to give the team as quickly as possible a feedback: to report the presence of bugs - or to please her with the fact that everything is okay and you can move on. You can carry out a series of autotests later, covering with them large arrays of code. Manual testing will help prepare cases for this test.
Self-tests do not allow to check whether it is convenient to use the new features of the application - usability testing is always carried out manually.
Skillbox recommends: Online course Fullstack-mobile developer .
In manual tests, you can improvise, creating insane combinations of actions that never occur to the user, but can be committed by him randomly. This allows you to create new cases.
New cases appear also because the tester constantly asks himself the question “what if?”. So he finds original ways to interact with the application - even if they were not in the baseline scenarios.
Problems of manual testing and their solutions
The biggest disadvantage of manual testing is the human factor. He, of course, influences everything that happens in a team - both the work of the participants and the events taking place on the client side. In the case of QA, the reason that the tester will be poorly immersed in the process and miss a potential error can be anything - a lack of experience, family problems or a headache.
Two testers can test the same scenario in different ways. What to do with it? It is important that each unintended or different from the expected result be recorded in the form of a case. So that any tester could test it by performing the same set of actions. But this may not be enough - if the case is not detailed enough, and the tester does not figure it out. Of course, it is impossible to guarantee 100% exclusion of the human factor - but you can try to minimize the probability of the problems it causes.
This also has a negative impact on the delivery time of features in production and labor costs: after all, now, “cunning” cases, invented by testers in the process, are added to periodically held basic cases and regressions.
The situation is aggravated by the likelihood that some of the bugs encountered will not yet have a strict description because their causes are not yet clear. How to deal with such repeated testing? Either find the source of the error and eliminate it, or - all the same to automate the passage of problem cases - in this case, the transition to software tests will be fully justified both in terms of time and financially. (No, this does not contradict the above - because such situations arise already during active development and by this time you will, in any case, launch the auto-testing process).
In any case, the appearance of the first cases needing regression tests or the release of the second version of the application and the scale of the command corresponding to these events is the moment when the automation becomes relevant (but does not exclude manual testing of new features). At this stage, the automation will start saving time: the developer will be able to start the regression tests of the new feature even before the transfer to the QA department to make sure that it does not break the existing functionality, and the tester does not have to go through the core-cracks. Another advantage of introducing autotests at this stage is that they can be launched according to a specific schedule - every night, on the days when sprints end or when new application assemblies are published.
In this case, we must not forget a few things:
- Creating cases and writing autotests will take time - lay it in sprints;
- make sure that the autotest test case is written well and in detail and describes the possible problem or the existing scenario in its entirety;
- check whether the autotest works correctly and whether it really checks what you need and does it qualitatively.
We summarize: manual testing gives a great advantage in speed and labor costs in the early stages, and as the application grows and a large number of regression tests appear, it goes into the category of “operational testing” of new features in a separate sprint. It will be relevant and, if necessary, urgently check how the application will react to a change in the operating system or an update of the environment.
Test stages: what, when and how
If you look at the development process as a whole and at testing as one of its parts, then with proper planning you will always understand what and when it will be ready. This will allow you to better plan the time of certain actions - since some events logically follow others and you have the opportunity to arrange them in chains based on your expectations.
The tester appears in the process of creating an application at an early stage. A client has a certain idea, business analysts collect from this requirement - and testers start working at this moment, checking these requirements.
How does this happen? They ask questions about the intended functionality. Trying to imagine how the application will look when it is implemented. If we are talking about a new feature in an already existing product, they understand how it will interact with the existing functionality. After the developers have evaluated the workload of the client’s ideas and determined how long it will take to implement them.
After that, the design phase begins. Here there is a need to understand how the transitions between screens will be carried out, certain fields validated, how an application or its individual function will generally interact with the end user. In the case of features, it is important to understand how they will fit into the architecture of an existing application.
At the design stagewhen a transition map between screens is created, the tester clarifies the behavior of the individual elements of which they are composed and what visual effects will be followed by those or other user actions. In addition, the tester validates the layouts for completeness, confirming that they reflect everything that is needed to implement the conceived functionality. Self-checking of layouts for compliance with guidelines will also be useful.
Skillbox recommends: online course Mobile Application Design
With the start of development, QA specialists immediately start writing test cases. At different stages of development, they may have different levels of detail, but by its completion it is desirable to ensure maximum coverage of the entire product with cases so that, upon receiving the assembly, you can start testing immediately.
The first step of direct testing is the Smoke test: assessment that the application or its new part is generally ready for testing. A smoke test is a test of whether an application or a particular function is running in principle.
Smoke test is a quick way to see if we can even begin a functional test. The term came from the creators of boards and microcircuits, which, for a start, had to make sure that the new circuit would not burn - hence the name: smoked / not smoked.
This is a quick way to make sure that we really have the task and we can take it to work, and not send it back to programmers.
By the example of the authorization form, a smook is an assessment of whether it is possible to log in at all, without specifying whether the data entered in the fields are valid, whether additional features work, such as a password reminder and other things. If we were able to log in in principle, we can proceed to further, in-depth testing of this module: take negative cases, boundary values, assess compliance with the established validation rules.
The next step is to conduct regression tests. In manual or automatic mode, the main pre-planned array of tests is conducted.
Regression testing is good because it allows you to find errors even in those places where everything was in order. This is due to the fact that regression is an assessment of the functionality of a standard set of cases when implementing each new module and each application change. After all, when developers add a new functionality, the current version may be damaged or a new feature may conflict with existing ones.
For example, adding new screens, and thus changing the navigation may disrupt the menu or at least display it. On the other hand, a global refactoring of the application code can also bring unpleasant surprises - after it, it is also necessary to carry out regression tests.
Problems can cause and update the library used by the application and change the environment in which it works. The more often you update the application, the more important is the role of regression tests. You should not limit yourself to a one-time check carried out when the feature is already implemented - check all changes. Here automation of regression testing will help you - simply because manual regression testing of a new feature created in just a week can take two, or even more, and the testing department will simply get bogged down in this.
A full check, of course, is carried out when a release candidate with one or several features is ready to roll out in production, but it is still necessary to apply one or another relevant cases at separate stages of development.
Everything ends with testing the final build - release candidate. It includes beta testing by internal testers, business testing — evaluating the resulting product by the customer and receiving feedback from him, as well as offering a specific group of users to get acquainted with the pre-release alpha version of the application or its new features. After that, the application is ready to roll it into production.
But the work of a QA-specialist does not end there.- it will test application updates and their compatibility with earlier versions, create cases for testing innovations and, if necessary, automate the passage of these scenarios.
In parallel with this, testers are involved in further analyzing statistics collected by analysts, monitoring application behavior and how users interact with it. This allows not only to see the live use of the results of their work, but also, at times, to discover new scenarios and unknown bugs that cause the application to crash.
Rebus time: guess it and a 10% discount on any of the Skillbox courses is waiting for you right now or be diligent and collect answers to all the puzzles - these discounts are added together (but with no other discounts on Skillbox courses).
However, it is not necessary to delay too much - the promo works until August 30, 2018. Recall that the subject of the riddle is mobile, and English words can interfere with the Russians here, so be careful! Send a request for the course, and when the manager contacts you - call him a pryword encrypted in the rebus (or a few!).
Testing formalization: test plan, bug reports format, reporting
In order to prepare for functional testing, a QA engineer prepares a test plan. This is the documentation that he will need when testing the product, a list of actions that he will need to perform. The test plan indicates the time of testing, describes the product or a specific task - in order to determine exactly what needs to be checked. A detailed description of all the main test cases. Lists the types of testing that are needed: functional and, for example, load. Describes how to automate certain cases and then at what stages they will be automated.
The test plan ends with a description of the report format: you need to explain in advance how the result will be presented. The most common report format is a list of test cases with the status of their passage. Knowing how much the cases cover the entire volume of requirements and reading the report, it will be possible to draw a conclusion about the current state of the application development. To do this, it will be enough to see how many of them have been successfully completed in one or another assembly.
The final sign that the product is ready is the status “all requirements are covered with cases, all cases are passed successfully”.
In addition, the error report format is formalized in the test plan. This may be a bug list, a list of tasks in the tracker.
The task of the tester is to provide the most complete information about the quality of the product to all team members.
What you need to know and be able to become a tester
First of all, the tester should be able to think and be attentive and diligent. Experience is important - it allows you to accumulate certain achievements and consolidate knowledge of the testing processes, turning them into skills.
A manual test specialist doesn’t have to be able to write code and have a deep understanding of architecture. This does not prevent him from checking the functionality and at the application level to understand the principles of interaction between the application and the server.
Automated testing specialist is a separate profession, with a completely different set of knowledge. High-quality auto-testing is not just scripts, but also an understanding of how an application is built from within, and specific skills, such as knowing how to parallelize tests or how to run them in the cloud or on several servers. Only such a pool of skills allows you to proudly call yourself an “automaton with a capital letter.” So, without knowledge of languages, their frameworks, principles of OOP and attentive surveillance of the development of technologies, this auto tester is indispensable.
The path of each tester begins with mastering the theoretical basis of testing, obtaining primary ideas about how an application interacts with the server and with the environment. If this knowledge is there, and with it a person has a very serious intention to learn, he can already be considered a junior tester. A group of such newcomers with burning eyes is assigned to a senior specialist - either a leading tester, or, if the company can afford it, a coach who purposefully pumps his players. They explain incomprehensible moments to the juniors and give painful tasks, like running features on test cases. In the process, a person learns to read test cases and masters the practical basics of functional testing. At this stage, the juniors still need to check the quality of the tests they conducted, passing them after them.
The next step is to create an individual (or collective, depending on the size of the company) development plan, according to which a novice tester will have to develop, mastering new knowledge and achieving certain results within the time allotted to it. That is the way to become a middle-level specialist.
A middle tester is a person who is already able to independently carry out the tasks assigned to him, to find solutions and generally to do his work consciously, and not blindly following the established algorithms.
With the development of skills in test design, knowledge in the functional and applied areas, as well as the development of new testing methods - which now need to be developed independently - there is a gradual transformation into a leading specialist. Now he can be charged with the management and support of the same juniors as he himself was before.
The next level is a test lead. He can already manage a team that includes representatives of all three previous categories, the processes of which he manages and the time of which he manages, including participating in planning global development processes, estimating labor costs and budgeting for his teams.
Further vertical growth of team lead is the management of the department or the transition to product management. If, however, a person seeks new knowledge, and not new administrative positions, then he can choose to develop, analyze, or such a direction as DevOps, combining the duties of a system administrator, tester and programmer.
Testing is only one of the many areas of mobile development that we consider as part of the “Fullstack Mobile Developer” course. The industry professionals who work with us will share their experience with you and test your homework, and the result of training may be an internship for a large company. Come to us to study! Skillbox recommends: