November 6, 2014 at 09:26SQA Days Report - Test Design: Easier to Read or Easier to Write
Alexander Alexandrov - “grandfather of testing” in the CIS, made a presentation at the anniversary 15th SQA Days in Moscow.
Slides:
www.slideshare.net/VLDCORP/ss-33747358 Performance
video:
Performance plan
1.
Test design 2.
How to check testability of requirements 3. Requirements and testing without test cases and / or without testers
4. Requirements and changes
5. It is easier to read or easier to write
6. How test cases are arranged
7. What arise Difficulties
8. How to overcome them.
Let's start with the fact that there is an increase in the general interest in automation, this is a trend and it is good! But the need for a “foundation” for developing automated test scripts is growing. After all, if there is no such foundation, then we get the complexity of creating, using and maintaining scripts. It is not clear what they check, it is not clear how they check and it is not clear how they relate to the requirements! This situation is familiar to those who develop these scripts and those who use these scripts, as well as those who believe the information that these scripts provide.
Also, the following important thesis is the importance of all of the above outside the context of test automation . Replace everything about automatic testing with manual testing, and you get exactly the same thing!
So what do we want?
We want to simplify and speed up the work, as well as increase the reliability of the results for:
- Tester;
- Developer of automated test scripts;
- Test manager analyzing test results (manual / automated) / customer / future users, etc.
My report will be in two unequal parts. In the first I will remind you what a test design is, and in the second how to get around errors and less often step on the same "rake".
How to test testability of requirements
Mantra requirements:
Every requirement must be checked. There are basic criteria that you need to focus on. A person can make mistakes when checking requirements. Therefore, you need to know how to reliably check them! Please note, I am not saying perfect check! It's impossible! At least reliably check!
Method 1 - Early design of test cases
That is, as soon as you start using UML diagrams, stories, viewed several times, with comments, the design begins, you should ask questions: what is the restriction on this field, and what will happen after this action , and who should do this, and what is the default period. Thanks to this, it is quite easy to detect many gaps at the design stage of test cases.
Method 2 - Visualization of the connection of test cases and requirements(where and how this feature is checked)
Because it may turn out that you wrote 3000 test cases, everything is absolutely wonderful, but we simply didn’t check any important things. The reasons are different: they forgot, did not understand, thought what had been done “obvious” to us by the way, missed something. I point this out, not because I have a universal solution, but because we do this every day!
The next aspect, but why do you need anything other than requirements?
Let's compare! The difference at first glance is very simple, but in fact, it is deep! The idea of why requirements testing is needed.
Requirements vs. Testing
Compliance of the software product with the requirements must answer the following questions
:? Are all requirements implemented?
? Are all requirements implemented correctly?
? Is there no excess?
? Is the diagnosis adequate?
“Why are you writing 1000 cases, and wouldn’t it be easier to take a tester, give him requirements for him to test on them?” As a test manager, I come across this question regularly. And I have to answer him differently all the time. Just recently, I answered the same question: “Here I have a serious complex project for a year, it involves only 2 testers who themselves tested the requirements and wrote cases. Within a year, testers left and new ones thanks to their test cases and the analysis they were able to adopt the essence of the project and continue working on it. ”
Often I hear that they offer to conduct testing, but without test cases and / or without testers.
Requirements and testing without test cases and / or without testers
- Testing by developers - problems are well known (subjectivity, ignorance of the entire project, logic)
- Testing by analysts - do not need test cases? (an interesting idea, but they always test a positive scenario). They check only what works as it should in the requirements, even small deviations from the correct behavior are not considered. They check the linear case, but do not carry out a bunch of requirements, integration testing)
Of course, in such cases the quality and volume of testing vary greatly, testing takes only 20% -30% of the total possible volume + serious errors in the requirements are skipped .
Another important aspect that I would like to draw attention to is that we always talk and mean the situation when we are in a development project and we have a stable team. There is a team that is professional and works together. But it happens that you are given a project for support. A completely different team, in a different country. You absolutely do not know these people, and, of course, there are problems with the requirements. Since there are many development works, few new requirements for systems are being done, and maintenance is becoming more. All this should take into account a number of features.
Requirements and Changes
The aspect of test design is also important here, because if every time in the project that is given to you for support, you will make a riddle: “How can I test this or that change?”, Then spend a lot of time and some attempts can be completely unsuccessful . At the same time, the work on the preparation of the test design performed once before its change can quite help you.
And the penultimate aspect of the general part that we will now finish is data-driven testing :
And:
There are separate steps of the test case and there is separately data that is submitted in the test case. The data may vary, the steps of the test case are stable, and thus we can check for a large number of situations. Not to mention the testing situation, data-driven is very important for cases when you have a data-dependent effect.
And another aspect that I would like to mention is the granularity of the requirements . Details may be more or less detailed. But here are the nuances that arise, if the requirements are detailed, it is difficult to maintain, but easy to use. And if there are few details, it is difficult to use, and it is easier to maintain. You can find a reasonable compromise: checklists and testing according to requirementsbut it will always contain risks that are well known.
We turn to the second part of my report.
The structure of the test case is well known, it is described below and humanity hasn’t come up with anything new.
What they write about the structure of the test case
The test case includes:
- Format;
- content;
The format is the same everywhere;
- Sequence number of the step;
- Impact on the system;
- Expected Result;
And remember! The devil is in the details!
If we make a comparison of the formats, then we will see that everything is the same for everyone. Analysis and comparison of formats:
Here you can see one example.
Pay attention to the table, we have the serial number, action, data that we enter and the expected result. And what do we see?
Firstly, we see the expected result not there, then we see an incorrect description of the action (“if the button is unavailable, then click” - the question is, should the button be available or not?), The numbers in the expected result are indicated incorrectly (if the three-digit number, fractional part, etc.
Formally, all the moments are included, but we can’t get reliable information about the quality of the product from these steps, because the necessary tests are probably missing.
It is well known what testing is data-driven . But we offer it more than d experimentally. The expansion at the expense of data made easy
Separation of steps from data
1. Separate description with links
2. Linking data and expected results
Steps
1. Actions to perform a test case
2. Navigation rules
Data
1. What the user enters and / or selects and / or clicks (field, list , ...)
2. Expected results
The structure of the test case (clarification) . Change the format:
And what do we get as a result of such changes?
Everything here is absolutely wonderful, except that it says “enter data”, and also the frequent use of terms such as “any”, “correct”, “expected”, etc.
Our next suggestion on the structure of the test case . Get rid of:
Notice, is there anything else left?
Yes, this is a link to an external document that a tester will have to contact to obtain confirmation of the correct expected result. Here we recall the name of the report “Test Design: Easier to Read or Easier to Write”
What do I suggest?
I would like to pay special attention to data preparation. It is clear that the test case can be performed in any situation. The system must be in a certain state and this state must be full. There are cases when, in order to prepare the data, you still need to perform some actions, including several hundred steps. Therefore, it is very important to record the data with which you work. And even better, if you have the opportunity to get exactly the data that the customer will work with.
Data :
1. Roles, values
2. Links to the data warehouse (queries)
Data preparation :
1. Preconditions (database status)
2. Execution of SQL queries
3. Actions in the format of test cases
The proposed solutions are a reasonable compromise on the complexity of steps and data sets. Sometimes it’s worth writing two or three similar test cases, reducing the volume of data sets by an order of magnitude (due to repetitions). All this will allow the tester not to go astray and miss something, and it will be easier for the developer to build the mapping of script code to the description of the test case (necessary when analyzing and maintaining scripts).
Yes, we complicate their work with a test design, but we increase its quality. Test design is the most difficult thing in testing. There are no two similar test designs. In most cases, in my practice, the introduction of a test design on an ongoing basis saves the labor of testing a manual and automatic tester. And, of course, this requires a highly qualified test designer. You can take a tester and even a developer and train, you can get a good test designer. It's just that a person cannot teach this from the street.
Slides:
www.slideshare.net/VLDCORP/ss-33747358 Performance
video:
Performance plan
1.
Test design 2.
How to check testability of requirements 3. Requirements and testing without test cases and / or without testers
4. Requirements and changes
5. It is easier to read or easier to write
6. How test cases are arranged
7. What arise Difficulties
8. How to overcome them.
Let's start with the fact that there is an increase in the general interest in automation, this is a trend and it is good! But the need for a “foundation” for developing automated test scripts is growing. After all, if there is no such foundation, then we get the complexity of creating, using and maintaining scripts. It is not clear what they check, it is not clear how they check and it is not clear how they relate to the requirements! This situation is familiar to those who develop these scripts and those who use these scripts, as well as those who believe the information that these scripts provide.
Also, the following important thesis is the importance of all of the above outside the context of test automation . Replace everything about automatic testing with manual testing, and you get exactly the same thing!
So what do we want?
We want to simplify and speed up the work, as well as increase the reliability of the results for:
- Tester;
- Developer of automated test scripts;
- Test manager analyzing test results (manual / automated) / customer / future users, etc.
My report will be in two unequal parts. In the first I will remind you what a test design is, and in the second how to get around errors and less often step on the same "rake".
How to test testability of requirements
Mantra requirements:
- Completeness
- Consistency
- Unambiguity
- Traceability
- Feasibility
- Testability
- ...
Every requirement must be checked. There are basic criteria that you need to focus on. A person can make mistakes when checking requirements. Therefore, you need to know how to reliably check them! Please note, I am not saying perfect check! It's impossible! At least reliably check!
Method 1 - Early design of test cases
That is, as soon as you start using UML diagrams, stories, viewed several times, with comments, the design begins, you should ask questions: what is the restriction on this field, and what will happen after this action , and who should do this, and what is the default period. Thanks to this, it is quite easy to detect many gaps at the design stage of test cases.
Method 2 - Visualization of the connection of test cases and requirements(where and how this feature is checked)
Because it may turn out that you wrote 3000 test cases, everything is absolutely wonderful, but we simply didn’t check any important things. The reasons are different: they forgot, did not understand, thought what had been done “obvious” to us by the way, missed something. I point this out, not because I have a universal solution, but because we do this every day!
The next aspect, but why do you need anything other than requirements?
Let's compare! The difference at first glance is very simple, but in fact, it is deep! The idea of why requirements testing is needed.
Requirements vs. Testing
- Requirements : determine what and how should work
- Testing : determine what does not work or does not work the way it should work
Compliance of the software product with the requirements must answer the following questions
:? Are all requirements implemented?
? Are all requirements implemented correctly?
? Is there no excess?
? Is the diagnosis adequate?
“Why are you writing 1000 cases, and wouldn’t it be easier to take a tester, give him requirements for him to test on them?” As a test manager, I come across this question regularly. And I have to answer him differently all the time. Just recently, I answered the same question: “Here I have a serious complex project for a year, it involves only 2 testers who themselves tested the requirements and wrote cases. Within a year, testers left and new ones thanks to their test cases and the analysis they were able to adopt the essence of the project and continue working on it. ”
Often I hear that they offer to conduct testing, but without test cases and / or without testers.
Requirements and testing without test cases and / or without testers
- Testing by developers - problems are well known (subjectivity, ignorance of the entire project, logic)
- Testing by analysts - do not need test cases? (an interesting idea, but they always test a positive scenario). They check only what works as it should in the requirements, even small deviations from the correct behavior are not considered. They check the linear case, but do not carry out a bunch of requirements, integration testing)
Of course, in such cases the quality and volume of testing vary greatly, testing takes only 20% -30% of the total possible volume + serious errors in the requirements are skipped .
Another important aspect that I would like to draw attention to is that we always talk and mean the situation when we are in a development project and we have a stable team. There is a team that is professional and works together. But it happens that you are given a project for support. A completely different team, in a different country. You absolutely do not know these people, and, of course, there are problems with the requirements. Since there are many development works, few new requirements for systems are being done, and maintenance is becoming more. All this should take into account a number of features.
Requirements and Changes
- Binding Change
- Surprise Change
- Ways to commit changes
- Matrix of the relationship of requirements and test cases (assessment of labor costs for the implementation and testing of changes)
- The effect of changes on the application as a whole
The aspect of test design is also important here, because if every time in the project that is given to you for support, you will make a riddle: “How can I test this or that change?”, Then spend a lot of time and some attempts can be completely unsuccessful . At the same time, the work on the preparation of the test design performed once before its change can quite help you.
And the penultimate aspect of the general part that we will now finish is data-driven testing :
- Expected results are displayed in response to correctly entered data.
- Adequate response to incorrect data , including those not recorded in the requirements (corresponding error messages)
And:
- Equivalence classes
- Boundary values
There are separate steps of the test case and there is separately data that is submitted in the test case. The data may vary, the steps of the test case are stable, and thus we can check for a large number of situations. Not to mention the testing situation, data-driven is very important for cases when you have a data-dependent effect.
And another aspect that I would like to mention is the granularity of the requirements . Details may be more or less detailed. But here are the nuances that arise, if the requirements are detailed, it is difficult to maintain, but easy to use. And if there are few details, it is difficult to use, and it is easier to maintain. You can find a reasonable compromise: checklists and testing according to requirementsbut it will always contain risks that are well known.
We turn to the second part of my report.
The structure of the test case is well known, it is described below and humanity hasn’t come up with anything new.
What they write about the structure of the test case
The test case includes:
- Format;
- content;
The format is the same everywhere;
- Sequence number of the step;
- Impact on the system;
- Expected Result;
And remember! The devil is in the details!
If we make a comparison of the formats, then we will see that everything is the same for everyone. Analysis and comparison of formats:
- Substantially the same:
- You can still search in books, the Internet ...
- Is the content adequate for the format?
Here you can see one example.
Pay attention to the table, we have the serial number, action, data that we enter and the expected result. And what do we see?
Firstly, we see the expected result not there, then we see an incorrect description of the action (“if the button is unavailable, then click” - the question is, should the button be available or not?), The numbers in the expected result are indicated incorrectly (if the three-digit number, fractional part, etc.
Formally, all the moments are included, but we can’t get reliable information about the quality of the product from these steps, because the necessary tests are probably missing.
It is well known what testing is data-driven . But we offer it more than d experimentally. The expansion at the expense of data made easy
Separation of steps from data
1. Separate description with links
2. Linking data and expected results
Steps
1. Actions to perform a test case
2. Navigation rules
Data
1. What the user enters and / or selects and / or clicks (field, list , ...)
2. Expected results
The structure of the test case (clarification) . Change the format:
- Step number
- System Impact
- Link to data (what kind of link is needed)
- Expected Result (possibly a link to data and / or illustration)
And what do we get as a result of such changes?
Everything here is absolutely wonderful, except that it says “enter data”, and also the frequent use of terms such as “any”, “correct”, “expected”, etc.
Our next suggestion on the structure of the test case . Get rid of:
- Loops like “Repeat steps 5-73 for all possible data”
- Designs of the type "any", "appropriate." “Suitable”, “expected” without the necessary clarifications, for example:
Notice, is there anything else left?
Yes, this is a link to an external document that a tester will have to contact to obtain confirmation of the correct expected result. Here we recall the name of the report “Test Design: Easier to Read or Easier to Write”
What do I suggest?
I would like to pay special attention to data preparation. It is clear that the test case can be performed in any situation. The system must be in a certain state and this state must be full. There are cases when, in order to prepare the data, you still need to perform some actions, including several hundred steps. Therefore, it is very important to record the data with which you work. And even better, if you have the opportunity to get exactly the data that the customer will work with.
Data :
1. Roles, values
2. Links to the data warehouse (queries)
Data preparation :
1. Preconditions (database status)
2. Execution of SQL queries
3. Actions in the format of test cases
The proposed solutions are a reasonable compromise on the complexity of steps and data sets. Sometimes it’s worth writing two or three similar test cases, reducing the volume of data sets by an order of magnitude (due to repetitions). All this will allow the tester not to go astray and miss something, and it will be easier for the developer to build the mapping of script code to the description of the test case (necessary when analyzing and maintaining scripts).
Yes, we complicate their work with a test design, but we increase its quality. Test design is the most difficult thing in testing. There are no two similar test designs. In most cases, in my practice, the introduction of a test design on an ongoing basis saves the labor of testing a manual and automatic tester. And, of course, this requires a highly qualified test designer. You can take a tester and even a developer and train, you can get a good test designer. It's just that a person cannot teach this from the street.