End-to-end testing: what, why, why
Testing in large companies in an enterprise is often difficult and ungrateful. The gap between business units and IT is huge: when a developer has a vision at the level of code, and verification is at the level of unit tests, and the customer thinks working or non-working not even with services, but entire processes beyond the scope of a single development team, or even the whole divisions of the company. And asks to organize business testing, or end-to-end testing, or testing based on scenarios from beginning to end (end 2 end).
Let's start from the very beginning - from two pillars, whence came this notorious “end-to-end business testing”, namely from the testing pyramid and from the ISO9000 standard.
Testing pyramid is surely familiar to any tester who has become skilled in his profession and has crammed when communicating with adjacent departments. Especially often it is necessary to appeal to it when justifying test automation. What tests are cheaper and more important to develop? And run?
The essence of the pyramid of testing is not tricky: at the heart of testing, you should use the simplest and fastest tests in writing and execution - unit tests. Of course, checking the interfaces of classes and functions is hardly the thing that can be shown to the customer, but without this solid, monolithic, trouble-free foundation, it is unlikely that something will be built higher. As a rule, several dozen functions, methods, classes implement some kind of functionality for the customer, and in fact a dozen of unit tests can be reduced to some high-level tests. The customer already needs a beautiful apartment with decoration, but it is unlikely that he will be satisfied when the skewed windows in his apartment stop opening and the floor and ceiling go cracked from the first breeze that blows. However, the customer himself can go into the apartment and check its quality may not be the best idea. Agree, it is difficult for the user to check the quality of concrete in the foundation, and to reproduce all weather conditions. So in testing, of course, high-level testing is needed, then only when we have completed the unit tests, and the tests are already of a higher level.
The higher-level tests are more difficult to develop and perform longer — integration tests that check the correctness of the work of simultaneously working modules that the whole team worked on, releasing their product (system). That is, the integration of the code is checked, the system is tested without interaction with external systems. Such tests already imply a high-level check, most likely through a system call via the system API or even a GUI (front). Working with this type of test is more difficult - to cover all the branches and nuances of the code, you most likely need to use a large number of highly intersecting checks on various test data, and during automation often develop a whole bunch of conditions and branching in the scripts. That is, on the one hand, we have already approached the user, complicating our lives, but on the other hand, it is still difficult for us to find a common language, it costs us more, and the quality of checks is still not enough. That is, we can launch a customer into a new apartment, he can check everything, but without taking into account interaction with other tenants, weather conditions and public utilities. Agree, the sense of the ideal world, the model, in real life, as a rule, a little.
If we add these conditions - we will see how our system interacts with external systems - suppliers and consumers, with our environment, that is, we will conduct system testing, we can easily see that the complexity of testing will also increase. We will need to strive for the simultaneous operation of all interacting systems, albeit without the involvement of specialists in them. For now, it is enough for us to simply accept some data from our suppliers and transfer our data to our customers. In the correct sequence and format. The further fate of the data does not bother us. The main thing - our system works correctly in the right environment. And everything would be good here - for our customer we can already have a full-scale demonstration, but in real life this is not all the success criteria for our development. OK good,
Therefore, here the first idea for end-to-end testing is to check not only our environment, but also all interconnected systems through which the data received or sent by our system passes. And this, in turn, means that we will have to combine several such “testing pyramids” among themselves. The construction of a fragile bridge, on which we will hold for the handle data that is valuable to the user.
That's just the question of how to do this? Who to do it? How to put together?
A series of standards describing quality management systems, including that any process in an organization should be described, documented, even if it is the process of issuing a rake to the janitor until the fall. And if so, that no process that runs inside the software used and developed in the organization, can not be described. The question is how to do it? Of course, the best description, from the point of view of BDD, is the description of behavior by tests, under which the testing pyramid will lie. But we will immediately return to our dilemma with combining several pyramids with thin ropes from top to top, which our ropewalker customer and his users will walk without insurance.
Therefore, the easiest way to use abstractions is to create at least a process diagram, indicate its inputs and outputs, make the process controllable and measurable, ensure interconnection with parent and child processes, as required by ISO9000
Business diagrams are best suited for this purpose, and standards such as UML, BPMN, ARIS, etc. are most often used. And the processes themselves become flowcharts with “cubes” strung on them. Between the "cubes" there is an interaction, in the standard BPMN is the flow of action and the flow of messages. And this is exactly what we need!
Any company that wants to have a certificate and follows the ISO9000 standard is most likely to have such schemes, and they are an integral part of high-level requirements. If good analysts work in the company, then, most likely, references to specific actions from the schemes will go down to low-level requirements. We need them.
In fact, in the diagrams we can see the whole process, and understand which scenario we need to build, and which system / team to run with what data at what point.
I do not underestimate the work of developers who write literate code that forwards messages between different parts of software and hardware systems, but it’s impossible to keep everything in mind. And when the process is used in many other processes, it is better to have such a “map” with you for conducting competent testing, and even more so for building a test model.
So, we have two introductory - every team / system should have a pyramid of tests prepared - from the smallest unit tests to complex system tests, as well as the fact that we must describe the requirements in the form of business within the organization -processes. This fact will allow us to quickly respond to the customer, which business process works and at what point because of what it breaks, and when receiving defects from commercial operation, it is possible to quickly perform root cause analysis (analysis of root causes of defects). In theory.
But in practice, everything again falls on the tester - how to choose the right ones from a pile of tests, especially strangers, build them into a chain, and send the necessary data to the input of each system and verify it with the correctly defined expected result?
The easiest option is to initially develop tests based on business models, and to divide teams into projects that implement a particular business process. To do this, some test management tools already have the ability to download BPMN schemes (for example, for HPE ALM - downloads are supported in XPDL format). HP ALM itself breaks the scheme into a set of requirements (actions), and if desired, creates a hierarchy of requirements (module Requirements-> Business Models). Further, our business is to cover the requirements with tests, and then build the requirements, and hence the tests in the chains covering our business process. These chains in HPE ALM are called “paths” (path), and allow you to see all combinations of sequences. If desired, the requirements, the chains can be immediately converted into tests.
But even if you do not use testing tools, you still have to make chains from the business process. Especially considering the imperfection of the tools (not everything is so rosy), as well as the fact that, most likely, the test model will need to be assembled post-factum, and executed completely in the form of regression by a “common team” not stuck to new projects.
How many ways can a squirrel walk to a bump?
In this case, we will need to open the tests of each team, find the requirements tied to the requirements in the business model, and build chains of them, retaining them in the “common space”. Creating a common space is some kind of substitute, but in any case it should be, even in the form of a barn book, excel, or a project area in the testing management tool. If we talk again about HPE ALM, then the BPT (Business Process Testing) module is responsible for this functionality, which at the same time allows you to transfer the results of one test to the parameters of another. However, if desired, and hard work on HPE ALM, this is possible and implemented through rebuilding test sets (Test set) into the execution flow (Execution flow). Then, when you run the full set, the testers responsible for passing each of the components of the end-to-end scenario will be called in turn.
And, alas, testing controls alone are indispensable. From my practice, almost all tools have some fatal flaws, and therefore, if you get to the testing automation stage of the business process, you will come to create a script that will pull the tests in the right sequence.
As a result, two conclusions can be made:
1) for end-to-end scenarios, previously developed tests for each of the systems included in the chain (scenario) of a business process are used with high probability.
It is possible to present all complete test suites of a company in the form of a sparse matrix, where the columns are distributed for each system (for simplicity, system tests), and the rows are business processes. That is, for certain business processes, it is necessary to select / create tests covering the business process, to establish relationships. If there is no coverage, this is a reason to fill in the gaps in the test model, or to make sure that the quality is ensured by other levels of testing (integration testing, unit testing, code review and running it through analyzers).
2) A tool is needed to monitor, trace and update the business process for synchronization with the test model.
And if testing tools are more or less tolerably done with the creation of a test model, then everything is really very bad with updating, it is often easier to recreate the model, rather than trying to see the changes in the process and the test model. And the experience of real teams suggests that it is better to create a live visualization of the architecture. The easiest way to do this is in the common area, using a simple whiteboard and stickers. Then, the teams that participate in the business process can visually see how the process is modified (links are removed and added, actions are removed and added). The main thing is that everyone has access to the board. Plus, note that if the process involves messages between systems, then, as a rule, at least there should be two tests from each system - for sending and receiving data. However, instead of stickers, you can use a whole Lego-city (from large blocks), or something else more creative. The main thing here is one language and one information space, which is very lacking in an enterprise.
Organizing visual and proper testing by business processes is a difficult and very expensive thing. Please note that E2E testing is not just an acceptance, user testing that the customer will perform, this is building a bridge, taking into account all the possible situations that the customer will follow and lead the users to follow.
Once again - E2E - this is not a walk on the Lada-Kalina across the bridge, and not even travel by two KamAZ. This is a complex engineering work, hanging bridges with sensors and conducting all possible checks and situations - at least a description of these scenarios.
Whether or not your company needs such an ideal finishing run is solely your goals and needs. Always, as with any testing, the potential risks from missed defects at this stage should be assessed, as well as the cost of preparation and carrying out end-to-end testing. Rate that from this will cost you more, and only then act. But in case of end-to-end testing by business processes, it should be remembered that it does not make sense without a solid foundation in the form of 100% passrate unit tests (~ 90-100% coverage), without integration tests (~ 60-80% coverage, 90 100% passrate), without system tests (20-40% coverage, 80-100% passrate). Establishing success criteria (quality gates) is more of a requirement for the quality of the product being produced, the main thing to remember here is that the amount of E2E tests is only the top of the pyramid (1-2% coverage, ~ 99% passrate), which should not be longer than its base, should not be a plug of holes from the previous stages. This is a supplement that is considered a priori closed in the previous stages.
The organization of such testing is mainly work on the preparation and synchronization of test cases and data (test analytics), as well as a set of organizational measures, the synchronization of teams in one place at one time on a workable test site. Keeping this in mind, one should not try to show the customer “end-to-end testing” ahead of time, in order not to waste time at once a large number of people without all the working components assembled together.
PS The described tools, as well as practices - purely for example, the author did not set himself the goal of advertising products and reciting the only true approach to end-to-end testing.
Let's start from the very beginning - from two pillars, whence came this notorious “end-to-end business testing”, namely from the testing pyramid and from the ISO9000 standard.
Pyramid testing
Testing pyramid is surely familiar to any tester who has become skilled in his profession and has crammed when communicating with adjacent departments. Especially often it is necessary to appeal to it when justifying test automation. What tests are cheaper and more important to develop? And run?
The essence of the pyramid of testing is not tricky: at the heart of testing, you should use the simplest and fastest tests in writing and execution - unit tests. Of course, checking the interfaces of classes and functions is hardly the thing that can be shown to the customer, but without this solid, monolithic, trouble-free foundation, it is unlikely that something will be built higher. As a rule, several dozen functions, methods, classes implement some kind of functionality for the customer, and in fact a dozen of unit tests can be reduced to some high-level tests. The customer already needs a beautiful apartment with decoration, but it is unlikely that he will be satisfied when the skewed windows in his apartment stop opening and the floor and ceiling go cracked from the first breeze that blows. However, the customer himself can go into the apartment and check its quality may not be the best idea. Agree, it is difficult for the user to check the quality of concrete in the foundation, and to reproduce all weather conditions. So in testing, of course, high-level testing is needed, then only when we have completed the unit tests, and the tests are already of a higher level.
The higher-level tests are more difficult to develop and perform longer — integration tests that check the correctness of the work of simultaneously working modules that the whole team worked on, releasing their product (system). That is, the integration of the code is checked, the system is tested without interaction with external systems. Such tests already imply a high-level check, most likely through a system call via the system API or even a GUI (front). Working with this type of test is more difficult - to cover all the branches and nuances of the code, you most likely need to use a large number of highly intersecting checks on various test data, and during automation often develop a whole bunch of conditions and branching in the scripts. That is, on the one hand, we have already approached the user, complicating our lives, but on the other hand, it is still difficult for us to find a common language, it costs us more, and the quality of checks is still not enough. That is, we can launch a customer into a new apartment, he can check everything, but without taking into account interaction with other tenants, weather conditions and public utilities. Agree, the sense of the ideal world, the model, in real life, as a rule, a little.
If we add these conditions - we will see how our system interacts with external systems - suppliers and consumers, with our environment, that is, we will conduct system testing, we can easily see that the complexity of testing will also increase. We will need to strive for the simultaneous operation of all interacting systems, albeit without the involvement of specialists in them. For now, it is enough for us to simply accept some data from our suppliers and transfer our data to our customers. In the correct sequence and format. The further fate of the data does not bother us. The main thing - our system works correctly in the right environment. And everything would be good here - for our customer we can already have a full-scale demonstration, but in real life this is not all the success criteria for our development. OK good,
Therefore, here the first idea for end-to-end testing is to check not only our environment, but also all interconnected systems through which the data received or sent by our system passes. And this, in turn, means that we will have to combine several such “testing pyramids” among themselves. The construction of a fragile bridge, on which we will hold for the handle data that is valuable to the user.
That's just the question of how to do this? Who to do it? How to put together?
ISO9000
A series of standards describing quality management systems, including that any process in an organization should be described, documented, even if it is the process of issuing a rake to the janitor until the fall. And if so, that no process that runs inside the software used and developed in the organization, can not be described. The question is how to do it? Of course, the best description, from the point of view of BDD, is the description of behavior by tests, under which the testing pyramid will lie. But we will immediately return to our dilemma with combining several pyramids with thin ropes from top to top, which our ropewalker customer and his users will walk without insurance.
The process approach is a management strategy. Thus, they need to consider these processes and their supporting processes.
Therefore, the easiest way to use abstractions is to create at least a process diagram, indicate its inputs and outputs, make the process controllable and measurable, ensure interconnection with parent and child processes, as required by ISO9000
All processes have:
• inputs;
• outputs;
• operational control;
• appropriate measurement & monitoring.
Each process will have support processes
Business diagrams are best suited for this purpose, and standards such as UML, BPMN, ARIS, etc. are most often used. And the processes themselves become flowcharts with “cubes” strung on them. Between the "cubes" there is an interaction, in the standard BPMN is the flow of action and the flow of messages. And this is exactly what we need!
Any company that wants to have a certificate and follows the ISO9000 standard is most likely to have such schemes, and they are an integral part of high-level requirements. If good analysts work in the company, then, most likely, references to specific actions from the schemes will go down to low-level requirements. We need them.
In fact, in the diagrams we can see the whole process, and understand which scenario we need to build, and which system / team to run with what data at what point.
I do not underestimate the work of developers who write literate code that forwards messages between different parts of software and hardware systems, but it’s impossible to keep everything in mind. And when the process is used in many other processes, it is better to have such a “map” with you for conducting competent testing, and even more so for building a test model.
What's the practice
So, we have two introductory - every team / system should have a pyramid of tests prepared - from the smallest unit tests to complex system tests, as well as the fact that we must describe the requirements in the form of business within the organization -processes. This fact will allow us to quickly respond to the customer, which business process works and at what point because of what it breaks, and when receiving defects from commercial operation, it is possible to quickly perform root cause analysis (analysis of root causes of defects). In theory.
But in practice, everything again falls on the tester - how to choose the right ones from a pile of tests, especially strangers, build them into a chain, and send the necessary data to the input of each system and verify it with the correctly defined expected result?
The easiest option is to initially develop tests based on business models, and to divide teams into projects that implement a particular business process. To do this, some test management tools already have the ability to download BPMN schemes (for example, for HPE ALM - downloads are supported in XPDL format). HP ALM itself breaks the scheme into a set of requirements (actions), and if desired, creates a hierarchy of requirements (module Requirements-> Business Models). Further, our business is to cover the requirements with tests, and then build the requirements, and hence the tests in the chains covering our business process. These chains in HPE ALM are called “paths” (path), and allow you to see all combinations of sequences. If desired, the requirements, the chains can be immediately converted into tests.
But even if you do not use testing tools, you still have to make chains from the business process. Especially considering the imperfection of the tools (not everything is so rosy), as well as the fact that, most likely, the test model will need to be assembled post-factum, and executed completely in the form of regression by a “common team” not stuck to new projects.
How many ways can a squirrel walk to a bump?
In this case, we will need to open the tests of each team, find the requirements tied to the requirements in the business model, and build chains of them, retaining them in the “common space”. Creating a common space is some kind of substitute, but in any case it should be, even in the form of a barn book, excel, or a project area in the testing management tool. If we talk again about HPE ALM, then the BPT (Business Process Testing) module is responsible for this functionality, which at the same time allows you to transfer the results of one test to the parameters of another. However, if desired, and hard work on HPE ALM, this is possible and implemented through rebuilding test sets (Test set) into the execution flow (Execution flow). Then, when you run the full set, the testers responsible for passing each of the components of the end-to-end scenario will be called in turn.
And, alas, testing controls alone are indispensable. From my practice, almost all tools have some fatal flaws, and therefore, if you get to the testing automation stage of the business process, you will come to create a script that will pull the tests in the right sequence.
As a result, two conclusions can be made:
1) for end-to-end scenarios, previously developed tests for each of the systems included in the chain (scenario) of a business process are used with high probability.
It is possible to present all complete test suites of a company in the form of a sparse matrix, where the columns are distributed for each system (for simplicity, system tests), and the rows are business processes. That is, for certain business processes, it is necessary to select / create tests covering the business process, to establish relationships. If there is no coverage, this is a reason to fill in the gaps in the test model, or to make sure that the quality is ensured by other levels of testing (integration testing, unit testing, code review and running it through analyzers).
2) A tool is needed to monitor, trace and update the business process for synchronization with the test model.
And if testing tools are more or less tolerably done with the creation of a test model, then everything is really very bad with updating, it is often easier to recreate the model, rather than trying to see the changes in the process and the test model. And the experience of real teams suggests that it is better to create a live visualization of the architecture. The easiest way to do this is in the common area, using a simple whiteboard and stickers. Then, the teams that participate in the business process can visually see how the process is modified (links are removed and added, actions are removed and added). The main thing is that everyone has access to the board. Plus, note that if the process involves messages between systems, then, as a rule, at least there should be two tests from each system - for sending and receiving data. However, instead of stickers, you can use a whole Lego-city (from large blocks), or something else more creative. The main thing here is one language and one information space, which is very lacking in an enterprise.
Finally
Organizing visual and proper testing by business processes is a difficult and very expensive thing. Please note that E2E testing is not just an acceptance, user testing that the customer will perform, this is building a bridge, taking into account all the possible situations that the customer will follow and lead the users to follow.
Once again - E2E - this is not a walk on the Lada-Kalina across the bridge, and not even travel by two KamAZ. This is a complex engineering work, hanging bridges with sensors and conducting all possible checks and situations - at least a description of these scenarios.
Whether or not your company needs such an ideal finishing run is solely your goals and needs. Always, as with any testing, the potential risks from missed defects at this stage should be assessed, as well as the cost of preparation and carrying out end-to-end testing. Rate that from this will cost you more, and only then act. But in case of end-to-end testing by business processes, it should be remembered that it does not make sense without a solid foundation in the form of 100% passrate unit tests (~ 90-100% coverage), without integration tests (~ 60-80% coverage, 90 100% passrate), without system tests (20-40% coverage, 80-100% passrate). Establishing success criteria (quality gates) is more of a requirement for the quality of the product being produced, the main thing to remember here is that the amount of E2E tests is only the top of the pyramid (1-2% coverage, ~ 99% passrate), which should not be longer than its base, should not be a plug of holes from the previous stages. This is a supplement that is considered a priori closed in the previous stages.
The organization of such testing is mainly work on the preparation and synchronization of test cases and data (test analytics), as well as a set of organizational measures, the synchronization of teams in one place at one time on a workable test site. Keeping this in mind, one should not try to show the customer “end-to-end testing” ahead of time, in order not to waste time at once a large number of people without all the working components assembled together.
PS The described tools, as well as practices - purely for example, the author did not set himself the goal of advertising products and reciting the only true approach to end-to-end testing.