What tests do you need? Part 2. Matrix of types of testing

annotation

In the first part of a series of articles, I talked about what the choice of tests depends on. Having in mind an understanding of what you want to achieve by testing, you can take the next step - choose tests. To do this, you need to understand what tests are in general.

Almost all articles on types of testing have a grouping of tests into some categories. Only this division does not always coincide. Reading through such articles, one often finds a difference in terminology among different authors.

My goal is to sort through, visualize the variety of tests to help those who choose which tests they need and those who study the testing itself.

I do not presume to give clear definitions of the types of testing in this article. An interpretation of the terms used to name species will be discussed in the third part. This separation is related to the amount of information.

Classification of types of testing

Types of tests can be viewed from different angles. And to classify them is also difficult, such as culinary dishes. You can group dishes according to national cuisine, you can according to the time of reception (for breakfast, lunch, dinner), you can use the ingredients used, you can calorie content and so on. The products themselves - can also be divided into groups according to various criteria. At the same time, everyone understands that a steak can be a dinner, while it does not cease to be a meat dish and cause debate about belonging to national cuisine. So are tests - a regression test can be integration.

Belonging to one category does not exclude belonging to another.

Another example - fatty foods can be called both a dish and a product - cheburek and vegetable oil. A vulnerability test can be a set of automated scripts and one test case in a set of functional acceptance tests of an input form.

The type of test is a characteristic that can be possessed by either a single test scenario or a collection of tests.

See what a cute mind map with types of wine: It would seem that it fully covers all types of wines. Now look at it again and list the wines by country of production, as they are placed in the supermarket or in the restaurant menu. Does not work? But then how to choose? This card is not suitable for selection by manufacturer. If you take a closer look, there is a note "in style and taste." Here is the same with the types of tests.

By restricting ourselves to one grouping criterion, we lose sight of the whole variety of tests.

A grouping criterion may be a key selection criterion.

Test Types Map

I like the approach that was used in the article about testing software on Wikipedia , since a large number of sections were considered.

I took this list as a basis and supplemented with information gathered from other sources and personal experience and compiled a mind map (knowledge map). This card is useful to me in the process of planning tests - I check to see if I forgot what. I’m sharing a sketch with you:



View full size.
You can download the source in xmind format here .

Test names are given in English to avoid translation inaccuracies.

Flower flowers

• Blue allocated characteristics applicable to a test kit
• Yellow highlights features applicable only to test cases.
• Universal characteristics highlighted in green

This separation is conditional, there may be exceptions.

About the yellow blocks in the knowledge map

Looking at my map, you might think that dividing tests by scenario type is not so important. In fact, concrete test cases are many times more than sets, and

missing a look, you lose a bunch of valuable checks, and, therefore, defects.

The types of tests that are listed in the Scripts node are actually the names of the script preparation techniques. Compliance with these methods gives specific tests. Therefore, in my opinion, it is appropriate to give them as types of tests.

The mentioned methods are considered to be types of Black Box tests. Why these tests on my circuit do not apply to the Black Box? Because even testing the specification, you need to think about what will happen if you enter a value greater than the permissible one, and how the system should respond to errors in general. You should also think about this when writing unit tests that are not related to the Black Box.

About green blocks in the knowledge map

In fact, these names are types of requirements. Tests covering a specific type of requirement can be logically called tests of this requirement. Most of the names of the tests placed in blue blocks also got out of the names of the requirements. For example, “the application must be fault tolerant” generates fault tolerance tests.

Green blocks differ from blue ones in that they are rarely mentioned at all in Russian-language reviews of types of testing. However, if you, for example, look for the phrase “suitability testing” you will find a lot of useful things.

How to use a card

My mind map of types of testing, in fact, is a graph, specifically a tree. My favorite XMind application makes it very easy to change the structure of the knowledge map to a tree and other views. But there are a lot of letters in the text, therefore the tree becomes wide and not convenient for perception.

This tree has 10 large branches - the first level of the graph is the classification criteria. Obviously, they are not in themselves types of tests.

I hope you know how the breadth-first search differs from the depth-based search. In my deep conviction, errors in understanding and choosing tests are due to the desire to find a satisfactory solution in a faster manner, which pushes us to search in depth. 2-level (title - list) lists of types of testing greatly contribute to this.

You need to look in width. Because on each branch of the given graph there are tests that you need. This should also be done when trying to characterize existing test suites. The same test will have several different characteristics.

Spoiler: when you study all types of tests, you will find that in each category there are types of tests you need

Classification criteria

Let’s proceed, we’ll run through the width of the upper level of the tree.

1. Type of requirements

All tests depend on what is required from the software being developed.

If we have something to develop, then there is something to test. The presence of formally described requirements is a very important thing, but it is not mandatory. It doesn’t matter if you have a piece of paper called “TK” / “SRS” or not - there are always requirements on the basis of which you conduct the check.

We can clarify, of course, which operations are needed or understand and do according to our vision, but somehow we already have the minimum requirement.

No requirements - no development. The reverse is also true - there is development -> there are requirements.

We can assume that there are no requirements if the developers did not understand what was expected of them and could not get to work. I know good developers who do not start development until they get the answer to the question: “why should this software do this and that? What is the ultimate goal of use? ”

If developers work, then there are requirements, another question remains the accuracy of their understanding.

For example, “a task, make a calculator that has buttons, and that will perform all mathematical operations” contains a bunch of requirements in itself.

Disassemble your idea of ​​what is expected from the developed software by the types of requirements that you see in my mind map. If you want a complete test coverage, then

You need all kinds of tests from the first group

If you need only the “minimum necessary” and not the “necessary and sufficient” - use functional suitability and accuracy tests.

Functional tests will be discussed in detail in the third part.

Many stop at this step. " We need functional tests ." But we have to go further.

2. Object of testing

The most fertile approach to test classification is categorization by test subject. Great variety of technologies, interfaces, architectures, functional purposes and system characteristics. Everything requires its own approach to testing. Different methods and tools.

I will attach this group separately:


On the right, I wrote three groups, they are conditional , therefore they are not parent nodes. Because these requirements can be both functional and not functional, depending on what we are developing. In the general case, we will carry out infrastructural ones before and for transfer to production. And we will conduct operational ones on a prod-like environment, when it will already be determined on which hardware our system will live.

Full security testing, for example for certification in the FSB, can be part of infrastructure tests if you use hardware protection methods or external system modules that we do not develop ourselves. And some security features will be “embedded” in our software and tested as part of the functional requirements.

For example, as part of testing the function of transmitting a transaction message from an ATM to an issuing bank, we will verify the PIN code encryption.

It is very important to understand that

types of tests for test objects - these are not types of functional tests

As well as performance tests, fault tolerance, etc. - not always types of NFR.

Let's look at some examples.

Example 1. There is no functional requirement "when a component fails, its functions must be performed by another, parallel component." Accordingly, the requirement will not be covered by functional tests - stress tests, reliability and stability tests.

Example 2. Consider the case where fault tolerance is a functional requirement. For example, a failover cluster is developed and tested, not the system in which it is used. The development goal is to create a product that will provide fault tolerance. In this case, we are dealing with functional fault tolerance tests.

Example 3. Application developmentJmeter . This is a popular stress testing tool. Its functionality is load testing. This is the case when the subject of testing became the subject of testing. Recursive, right? JMeter stress testing tests are functional.

Other possible examples: the development of a cryptomodule (IS functional tests), the development of an interface between systems (functional integration tests), the development of a front office web interface as a thin client, while observing the separation of business logic from data presentation (functional UI tests). Etc.

Because of such cases, it is not appropriate to separate functional tests by type of functional (only by type of requirement). And it is not appropriate to attribute the functional tests themselves to objects when categorizing "according to the test object."

3. Knowledge of the system

Depending on the knowledge of the system, tests are tests of black, gray and white box. These terms come from Management Theory, and I hope that they are familiar to you in a broader sense than as types of tests.

It is possible to consider as a monochrome box both the entire system and its individual part.

The most common testing approach is to conduct functional suitability tests of the black box . They are also called acceptance tests at the same time, but we have not yet reached acceptance tests, wait. It is on such testing that most novice testers are dragged.

It is a common misconception that conducting such tests is necessary and sufficient, and if it does not lead to an increase in the quality of the product, then the testers did a poor job.

My whole series of articles on choosing tests is aimed at saving readers from this error.

I will write in detail about the impact of system knowledge on testing.

4. The degree of automation

They write about automation everywhere and a lot. I myself really love this topic.

What I don’t like is that automation is most often considered in this context: “here we have accumulated regression functional tests, there is no one to run them, it’s necessary somehow to have it.” Therefore, many will look at this tree branch and say "well, this is later."

Automation is not only the evolutionary development of tests. Some necessary types of tests simply cannot be manual.

You need to think about automation when answering the question:

“How do we check this?”

Some situations simply cannot be reproduced manually. Sometimes we need simulators of external systems. Sometimes - auxiliary tools for preparing test data. Almost always - tools for conducting stress testing.
Some tools you take ready-made, some write yourself. Resources must be laid on this, therefore

think about automation in advance.

5. The degree of isolation of the components

It is possible to test the “from-and-to” system for safety, for compliance with standards, for ease of use. It's about scale — when you take it all.
You can check individual parts at different levels of aggregation. You can check not the components themselves, but how they interact - whether data is lost or distorted.

The choice of the scale at which this or that test will be carried out depends on the goal - what errors you are looking for, what requirements you want to check. And also from the knowledge and availability of the system.

End-to-end testing can be a black box, for example, on acceptance testing at initial commissioning. And before the transfer - a gray box, when testers know how the system works and where it has bottlenecks. The data is poured onto the very front entrance and they are waiting for the leak from all the slots and the expected result from the rear exit. The preparation of test data depends on the knowledge of the system. Knowing where it can break through, you can slip something that will fit into the desired slot. In this case, it may be that a potentially defective component is not available on its own - therefore, we are not talking about testing the component. The entire machine starts up, and defects can be detected not only where expected.

If you isolate the component and submit data to the input only into it and check the result only at its output, this will be testing the component.

Depending on how finely crumbling the system there are differences in terminology regarding types of testing. This will be discussed in another part.

6. Testing time

When determining the necessary types of tests, you need to answer the question

when will we test?

My favorite army joke: "Dig from the fence to the sunset." Arranged for our work:

"Test from a critical defect to release."

I believe that this approach makes it impossible to achieve effective test coverage.

The question of the timing of this or that test largely depends on the project management methodology. According to SCRUM, you will take only those tests that you can catch in the iteration. You will have a fixed release date, and you can assume when it's time to make a smoke, when it makes sense to start a regression, and the rest of the time you will rake up a bug fix and conduct full-fledged tests of new features. The build can be collected every day, and it can be used to drive smokes and / or regression tests. Someone collects the build just before the release and sits at night to double-check or double-check everything that can be done.

In my understanding, the feature that separates the tests by time is the release. Some of the tests are done before the release, inside the team - these are alpha tests, part - after the transfer to operation - these are beta, gamma, delta ... omega tests.

Everyone knows the law of the dependence of the cost of a defect on the time it was discovered. Therefore, maximum tests should be performed at the alpha stage.

Beta tests usually mean “prerelease”. When the product seems to be ready, and it is already being used, but it is still not finished. The practice of full beta testing is common in the gamedev industry and in open source projects.

7. The degree of preparedness of tests

You need to decide:

• You either lay the time to prepare the tests, or not.
• You either have a common standard test plans and test reports - or not.
• You use or plan to use the test management system or not.

Over time, these points may change for you.

Be that as it may, even if you answered “no” at the beginning on all points, this does not mean that only research testing will be carried out on your project. You can take the TK and check whether what is written in it is completed - it will be a fully prepared test. When you know what to look for.

Lack of design tests does not mean their unpreparedness.

More details are in another part.

8. Depth of testing

I read about Test-to-pass and Test-to-fail in Software Testing By Ron Patton . Here is a quote:

There are two fundamental approaches to testing software: test-to-pass and test-to-fail. When you test-to-pass, you really assure only that the software minimally works. You don't push its capabilities. You don't see what you can do to break it. You treat it with kid gloves, applying the simplest and most straightforward test cases.

Patton writes that tests that must pass successfully (Test-to-pass) should be checked first. If they did not pass, then the rest can not be checked.

It does not surprise me that this breakdown of tests is not mentioned anywhere else. The reason is that this characteristic is almost equivalent to dividing test cases into positive and negative ones. In fact, the way it is, except for one point:

Позитивные и негативные бывают тест кейсы. Метками «test-to-pass» и «test-to-fail» можно сгруппировать тестовые наборы для smoke, acceptance и regression тестов, которые могут содержать в себе как негативные, так и позитивные сценарии.

Tets-to-pass tests are in the normal, most commonly used mode of operation.

Test-to-fail - these are tests in uncharted territory, which may turn out to be a minefield. You will not carry out these tests after each assembly. They are needed only to search for special states of the system in which a previously undetected defect may occur or even the entire system may malfunction. Such tests can be ad hock, and negative tests can be cases that are checked when taking a bug fix, and they must pass successfully.

The purpose of a negative test is to ensure that the system responds correctly to the wrong action. The goal of test-to-fail is to break the system for sure.

Remember that the sooner defects are discovered, the better. And that by the end of the iteration, the system should be checked as much as possible, and the basic scripts must surely pass successfully. Thus, we get the alternation of tests in depth.

The first batch of tests should allow detecting a certain number of show-stopper defects. When the first batch of Test-To-Pass is successful, we move on to the second - Test-To-Fail, which should identify as many defects of all degrees of criticality as possible. And after the last batch of tests, defects should not occur at all - they should be Test-To-Pass again.

9. Scenarios

The relevance of casting script types to the diagram has already been written above.
Thinking through scenarios is not a strategic task. When planning a testing strategy, only one thing is certain

Scenarios need everything possible.

Otherwise, you will miss defects.

10. Dynamic

If during testing there are manipulations with the application - it is dynamic. If the state of the system does not change, this is static testing.
Static testing is often overlooked. How can I test without changing anything?
The answer is in the diagram. Code review and documentation testing help identify a significant share of errors without wasting time putting the system in motion.

This completes the wide view. View in depth leave the next time.

Test matrix compilation

So, each test suite can be described by listing 9 characteristics:

• Type of requirements
• Test object
• System knowledge
• Degree of automation
• Test readiness
• Testing depth
• Testing time
• The degree of isolation of the components
• Dynamism

Speaking about a specific project, it will be possible to break down all the test objects into functional and non-functional groups, that is, by type of requirements. - 1 dimension.

You will take up one or another set of tests, depending on the stage of development, so the tests can be grouped by time. They can be marked as tests that must pass successfully, and as tests with which defects should be detected. - 2 measurements

Static tests can be laid down as standard practice at the organizational level. To conduct a code review after each commit with closing the ticket, and test the specification upon receipt of new requirements or at the test design stage. - 1 dimension.

Carry out ad hock / research testing during downtime or in the early days of the project / new functionality. All other tests should be considered prepared.

There are 4 test characteristics left:

• Type of requirements * Test object = Functional component of the system
• System knowledge
• Degree of automation
• The degree of isolation of the components

Now they can be used to compose a matrix of possible combinations of types of testing.





I then considered that even according to the 4 characteristics of the types of tests it turns out so much that it makes no sense to generate them.

Matrix, matrix ... I consider this mind map with types of tests as a matrix of tests. Looking at it, you just need to keep in mind the fact that each individual test at the same time is a kind of test of all the above categories of the top level.

Conclusion

So, a lot of tests are huge. Having an understanding of the purpose of testing and a systematic idea of ​​the types of tests it is already quite possible to answer the question - what types of tests do you need.

In the end, here is the checklist - questions that need to be answered by determining the types of tests needed:

1. What functional and non-functional requirements are presented to the system?
2. What will the system consist of?
3. How well do testers know the structure of the system?
4. How and how to reproduce test situations?
5. In what areas and on what scale will the system be tested?
6. At what stage of development will the tests be conducted?
7. How will you describe and store tests?
8. Do your testers know well how to write test cases?
9. Will developers test themselves and each other?
10. Is the specification perfect?

(These questions were raised when reviewing the types of tests in width.)

I hope I did not confuse the readers even more, and my matrix will be useful to you.