Testing domain definitions or more than boundary value analysis
All testers at least have heard about such test design techniques as equivalence classes and boundary value analysis. It would seem that it could be simpler: select classes, take one value in each, check the class boundaries and the values to the left and right of the borders. But is it always that simple? What to do if, after breaking up into classes, it turns out that there is, in general, a problem with borders - they cannot be defined, since the data cannot be sorted? What if the tested parameters are interconnected by some logic and depend on each other? How many tests are enough? Below we will consider the possibilities of the two main test design techniques that exceed those inherent in their direct definition.
Scope - a mathematical term - the totality of all possible values of a variable. Testing the definition areas considers the program as a function of many variables, each of which takes a finite set of values. Each such set can be divided into at least two equivalence classes - valid and invalid values.
Testing domain definitions involves three steps:
- allocation of subdomains for each parameter, all elements of which are supposed to lead to the same program behavior (to reduce the number of tests);
- selection of specific values for testing within each class (including for identifying errors related to the fact that the definition area is set incorrectly);
- a combination of these values (to increase test coverage and identify errors that depend on the interaction of several parameters).
Experienced (and not so) testers will now be skeptical that the tasks above are solved by three test design techniques - partitioning into equivalence classes, analysis of boundary values, and pairwise enumeration. This is true, therefore, we will focus on some features of the application of these techniques that will help to detect more errors and / or reduce testing time, while preserving the level of test coverage as much as possible, which means confidence in the quality of the program.
Equivalence classes
A few simple rules.
- If the domain of the parameter definition is a range, then it makes sense to distinguish three equivalence classes: to the left of the range (invalid values), the range itself (valid values) and to the right of the range (again invalid). When selecting classes, you must use inclusive boundaries for the purpose of uniqueness and accuracy: the same value cannot apply to two classes at the same time.
- If the domain of definition is a set of disordered data, then you can always distinguish at least two classes - valid and invalid values. The resulting partition can be "split" further. For example, many Latin letters can be divided into two subsets: Latin letters in upper and lower case, respectively.
The example above uses the obvious method of splitting into subclasses, but it is not the only one. Such a partition is not always advisable in the sense that with its help errors are not so often found. Here are some other tricks:
- by the frequency of use by end users (for example, for parameters like login and password, it may make sense to select “qwertyQWERTY1234567980” in a separate class of characters);
- random equal-sized subclasses (providing conditional test coverage if there are no other logical ways to subclass).
Distinguish between linear (ordered) and non-linear (disordered) equivalence classes. Obviously, it is impossible to apply the analysis of boundary values to the latter, i.e. There is no logical way to highlight elements that are more likely to lead to errors. An example of such a class can be many special characters that can be entered from the keyboard. For crushing this class, a second trick may come in handy. The input parameter for its application will be the number of subclasses that are planned to be used in testing: one value will be taken from each.
Typical errors of this stage of testing the definition areas: too many or too few classes, classes are allocated incorrectly (in relation to the functionality of the program).
Selection of values
After the division into equivalence classes is completed, you must select the values from each class that will be used in the tests. The analysis of boundary values is only one of the methods, and it is suitable only for linear classes. What to do in other cases?
- Random selection. It is highly desirable that at each subsequent execution of the test, some other value from the class be selected at random. In this case, random selection can be applied with return (the value selected earlier can be re-selected with the same probability) or without return (the value selected earlier can no longer be selected, thereby increasing the likelihood of choosing the values remaining in the class).
- Proportional Partition. There are various algorithms whose main purpose is to reduce the risk of incorrect partitioning into equivalence classes. To do this, you can take several values from each class (the number increases with the utility of a particular function of the program). Another way: to take from each class not a fixed number of values, but a fixed part of the class. Thus, for classes containing more elements, more tests will be obtained.
- Boundary value analysis. It should be remembered that the main idea of this technique is to highlight values that lead to errors with a higher probability than others. This technique is not limited directly to the controls on the program screen. In addition to the numerical boundaries of the ranges, it is worth remembering the time limits (for example, the period of free use of the program), cycle boundaries (the number of incorrect password entries), type boundaries (even if, according to the specification, you can enter an integer that is not limited to above in a certain field, this number is otherwise it will be limited to the maximum value of the integer data type that the programmer has chosen to implement this function). There are other boundaries associated with non-functional types of testing - performance, configurations.
- Empirical knowledge. Some values may be selected more often than others or may require special use in terms of the business logic of the application. Ideas of such tests can be suggested by a person who is well versed in the subject area of the program.
When analyzing boundary values and identifying equivalence classes for numerical parameters, special attention should be paid to the calculation result. Such a breakdown and selection of values for testing will help to find important errors. What restrictions are imposed on the range of values, i.e. the result of the calculations, and what values should the input parameters take? What do you need to set input values to go beyond the boundaries of this area? For example, if the result of the calculation should be positive, it is worth highlighting three equivalence classes and the corresponding boundary values:
- input data in which the result is strictly positive (valid class);
- input for which the result is zero (invalid class);
- input for which the result is negative (invalid class).
It may turn out that the values found in such a partition in invalid classes are allowed for input according to the specification. This might be a drafting error that business analysts should point out.
Value Combinations
Defects that depend on the input data can be divided into those that occur with a specific value of one parameter, and those that require a combination of specific values of more than one parameter. To detect the latter, combinatorial testing techniques are used, one of which is pairwise testing.
Combinatorial techniques can be applied when equivalence classes for each parameter are selected, and values are selected on which tests will be performed for each parameter separately. To make combinations, you can use several strategies:
- “Weak” vs. “Strong” combination - weak allows you to make a minimum of tests to detect all defects that occur at a particular value of one parameter, strong is designed to detect defects that occur at the “junction” of parameter values;
- “Normal” vs. "Reliable" combination - normal uses only valid values selected for testing, reliable - everything.
These are independent complementary characteristics, i.e., for example, weak normal combination can be applied. Such a strategy involves the preparation of tests in which at least once all valid values of each parameter selected for testing are encountered.
Example:
- parameter A can take valid values a1, a2, a3 and invalid values a_4, a_5;
- parameter B can take a valid value b1 and invalid b_2, b_3, b_4;
- parameter C can take valid values c1, c2, c3 and c4 and invalid c_1.
A weak normal combination will produce the following tests:
- A1, b1, c1
- a2, b1, c2
- a3, b1, c3
- A1, b1, c4
Strong normal combination - all possible combinations of valid values of each of the parameters:
- A1, b1, c1
- A1, b1, c2
- A1, b1, c3
- A1, b1, c4
- a2, b1, c1
- a2, b1, c2
- a2, b1, c3
- a2, b1, c4
- a3, b1, c1
- a3, b1, c2
- a3, b1, c3
- a3, b1, c4
Similarly, you can compile test suites using strategies of weak reliable and strong reliable combination (try to do this as an example above).
Obviously, when using a strong and / or reliable combination, the number of tests will increase sharply with an increase in the number of values of any of the parameters and, of course, with an increase in the number of parameters themselves. Pairwise technique is one way to reduce the number of tests, while trying to maintain the quality of testing, i.e. minimize the number of undetected errors. But applying this technique, it is important to understand that errors at the junction of more than two parameter values will remain undetected.
Another way to reduce the number of tests is to find out if there are dependencies between the input parameters, and take this into account in the tests. This is not always possible: often testing a black box does not allow you to "look inside." In this case, you can try to identify the dependencies empirically and take them into account in combinatorial tests. However, there is a risk of making a mistake in highlighting such patterns.
Combinatorial tests can and should be made using appropriate tools to avoid the human factor.
I sincerely hope that the above helps you design effective tests.