March 14, 2019 at 15:03Risk Based Testing
To ensure the quality of the information product in medicine, insurance, banking and other industries, along with other testing methods, it is important to use risk-based testing. For verification, the most risky areas of the software being created are selected. This allows us to anticipate negative scenarios and successfully implement the business goals of the customer.
In the article, we will tell how we at SimbirSoft worked with risks (using the Internet acquiring system and other projects as an example) and what methods of risk assessment and management we use in customer projects.
To begin, we give an example from our practice in the development for the banking sector.
Customer's proposal: to focus on the web version of the bank for individuals.
The risk we identified: a possible loss of audience due to the low demand of the web version for individuals, in contrast to the mobile client-bank.
Our arguments: statistics of user preferences based on reviews and distribution of the audience by mobile and web versions.
Conclusions: for individuals, the mobile version is more convenient, since the phone is always at hand. Operations are fast, the authorization system allows you to access all convenient services. In this case, quick access to a limited set of the most popular functions is important.
For legal entities, the completeness of the functions provided, the ability to upload, print, and work with a large amount of information are important. For this, the web version is more convenient.
Our solution: to focus on the mobile client bank for individuals.
At the beginning of work with the project it is important to choose the testing strategy correctly. Let's look at examples of why it is important and how to choose it.
Sooner or later, all companies are faced with the need to organize the testing process and come to understand that building its strategy is an important stage in software development. Sometimes - through their own bitter experience. It is especially dangerous to underestimate the role of testing and strategy selection in the development of large-scale projects. The testing process should be selected according to the business goals and specifics of the project, otherwise it will not lead to positive results either in a month or a year.
For example, consider testing mobile and web applications for a bank. At the start of the project, we selected a strategy based on requirements with a low level of detail. We used checklists to reduce testing time and support the test basis. With the growth of functionality, the addition of acquiring, sms-authorization and notification, more complex systems, checklists no longer cope with their task. Over time, more and more QA specialists joined the team, it was necessary to transmit information and coordinate their actions. With the complication of the product, any change could affect related functions, that is, the risk of regression increased. The need for automation of regression tests was brewing, so we switched to test cases.
Conclusion:depending on the project, its specifics or stage of development, the testing strategy changes.
The strategy must be selected for the project goals in order to ensure the best quality product. She answers the questions “what”, “where” and “when” will be tested. At any moment in time, you know at what point you are and where you will come in the future - according to the strategy.
A business goal may be to ensure the security of customer data, as well as the software itself at the production stage. Security begins with the development process and continues through the testing phase.
For example, at one of the projects, we created a safe environment for development and testing, deployed an infrastructure that met all the requirements and helped protect data. We requested certified tokens and name-based flash drives for each QA specialist to access the test infrastructure. Thus, we ensured the business goal of the project in software security and kept confidential the client and user data.
Due to the testing strategy, emphasis can be placed on really important aspects for a particular project. It is logical that the release of a mobile game or a complex banking CRM-system requires different approaches to testing.
In our practice, we at SimbirSoft used the whole range of development methodologies, but flexible technologies always remain our priority. And even where, for a number of reasons, it is not possible to use them, the team adopts the best practices and applies them in everyday work. Testing becomes an integral part of the whole process and merges into the overall workflow. In this case, it is responsible not only for the quality of the product, but even for the quality of the entire work process.
What technologies do we use:
In full force, the testing strategy reveals itself in projects with complex business logic. For example, software for information support of banks, the construction of an Internet acquiring system, an automated trading platform. In such projects, it is important to apply a suitable testing strategy, since the price of some errors can lead to real losses and greatly affect the company's reputation for the worse.
Also, the main goals of testing - defect search and software verification for compliance - may be added additional. For example, it is important for banks to quickly implement the new requirements of the Central Bank. This means that timely testing with the required quality for critical tasks will be added to the main goal.
Recently, in a banking project, we were faced with a change in federal law - an increase in the VAT rate from 18% to 20%. A lot of preliminary work was done to adapt to the change of legislation, since the change concerned not only the replacement of forms, interfaces, but also the alteration of the logic of several calculation formulas. It was necessary to ensure the correct display on many platforms. Also, in the deferred payment function, it was necessary to take into account the transition period with rates of 18% and 20%.
Now we’ll talk in more detail about how we build our strategy and why we often choose risk-based testing.
There are several testing strategies that are selected for specific purposes:
In the case of working with projects with complex business logic, it is necessary to determine stringent requirements in the design of systems on which testing is based on. A suitable tool is requirements-based testing.
One type of requirements-based strategy is risk-based testing. Moreover, those parts of the system’s functionality that are most exposed to risks are tested first. Risk is a possible negative consequence of a malfunctioning system. The consequence is a risk in the presence of two components, such as opportunity and negativity.
There are two types of risks:
1. Product risk
It can be managed and unmanaged. In the example above, we were faced with a manageable risk: rapid growth and complexity of functionality, which increased the likelihood of regression. Here we solved the problem by having an understandable test base and subsequent automation. The risk that we cannot affect is dependence on external systems and their failure in the integration process. Here we lay the measures that will reduce their impact on our system. For example, using backup, handling exceptional cases, displaying warnings for the user.
2. Project risk
For example, on a project, we were faced with the fact that the customer had not previously worked with a distributed team, and therefore the used workflow did not meet the requirements and created additional communication problems: lack of understanding, duplication of tasks, performance of mutually exclusive tasks, and so on. We agreed on the restructuring and improvement of the process - revised the workflow, introduced all the team members, held rallies, presentations and retrospectives. As a result, the work went in the right direction.
The risk-based approach allows you to compose a certain number of risks, for a short time to test the risks with high priority and further provide the customer with metrics on how well they were tested, showing the number of planned and completed cases and the number of defects.
The implementation of the risk-based approach on the project takes place in four stages:
Risk Identification - at this stage, you need to identify the risks and get their list.
Risk Assessment - here we analyze the list and classify it by priority.
Risk Mitigation - at this stage we determine how deeply we will test the risks.
Risk Management - here we decide how we will continue to work with them and go through them, identify new risks.
Risks are identified and assessed by a group of stakeholders during brainstorming sessions. The team should include business analysts or carriers of knowledge about the system from the customer, developers, manager or project manager, architects, QA specialists. We involve information security specialists, employees who work directly with the current system, and business analysts who are immersed in processes to identify and assess risks.
Consider the risk-based approach by the example of testing the Internet acquiring system.
We
distinguish the following requirements: D1 . Providing 1000 simultaneous connections to the system per second.
D2. Transaction Security.
D3. Access to the transaction should be available only to the person making the transaction.
D4. Providing and supporting the SET (Secure Electronic Transaction) standard.
As a product risk, we can distinguish:
RP1. System crash with simultaneous connections.
RP2. Using SQL injection during the transaction.
RP3. Access to someone else's transaction when changing parameters in the URL.
RP4. Loss of data when the connection with the bank is lost at the time of the transaction.
RP5. The use of invalid certificates when providing the SET (Secure Electronic Transaction) system.
As organizational risks:
RO1. The fall of the developed system due to the inaccessibility of external systems.
RO2. The presence of hard-to-reproduce cases that cannot be detected in a test environment.
Thus, each risk logically follows from the requirements that are in the system, but is not limited to them. Risks complement the requirements and identify additional cases that may arise when working with the system.
Risks can be reduced or compensated depending on the project objectives and system requirements. It is accepted that the risk is covered by the test case at least once:
1. For each product riska set of test cases RP1-RP4 is prepared with the condition that each requirement and each risk should be covered by at least one test case. Depending on the purposes of testing, the set of test cases is expanded to a certain level of detail.
2. For each organizational risk, a list of activities is prepared that can reduce the impact of risk on the product being developed.
There are several methods for assessing and managing risks: lightweight methods (PRAM, PRISMA) and more formal (FMEA, FTA).
With the FMEA model, the project team identifies all processes, components, modules of the system in which a system failure or error can occur that can lead to degradation of product quality. During the brainstorm, the risks of the system are determined by three indicators: severity, priority, probability. Then the Risk Priority Number is calculated for each risk and, depending on the indicators, the testing depth is laid.
With the FTA model, all the causes that can lead to defects in the business processes of the system are determined in stages. The analysis goes from the highest to the lowest level. The difference between FMEA and FTA is that the first approach focuses on the functionality of the system, and the second on business processes.
In addition to these formal and heavy approaches, there are more flexible and informal ones. For example, PRAM (Pragmatic Analysis and Risk Management) and PRISMA (Product Risk Management) methods. They successfully and easily combine risk and requirement based strategies. Basically, these approaches use requirements specifications as input, but stakeholders can also be involved.
Lightweight risk analysis methods are very similar to formal ones. They focus on technical or commercial risks, weighing the probability of occurrence of the risk and the factors affecting this probability.
However, the only two factors used by these methods are the likelihood of risk and its impact. In addition, these approaches use simplified qualitative judgments and scales to make decisions.
Our teams use flexible lightweight methods and adapt PRAM and PRISMA approaches to their goals.
For example, on one of the projects, we identify project and product risks that may work. To do this, analysts, developers and QA participate in the analysis - one representative from the team.
A risk table is formed with the products. They determine the assessment of the probability of a risk occurring and its possible impact on the system on a five-point scale. In table 1 - the strongest influence, 5 - the weakest. Also for probability 1 - high probability, 5 - low probability.
Further, for each of them, the risk coverage of the product is traced with test cases.
We select the following checks:
TC1. Checking the load with more than 1000 connections to the
TC2 system . Load test with 1000 connections to the
TC3 system . Enter SQL injection during transaction
TC4. Enter SQL injection on the successful transaction page, by substituting other
TC5 data . Enter XSS (Cross-Site Scripting) scripts in the available fields for input when the transaction
TC6. Simulation of a disconnected Internet connection during a
TC7 transaction . Exit the transaction session at the confirmation stage
TC8. Validation of expired certificates during transaction
Thus, priority checks are TC2, TC4, TC5.
TC6 and TC8 have a high impact, but less likelihood, so the priority of checking for them is lower, but checks are also required.
TC7 does not apply to any of the requirements, but extends the test for a negative scenario, which is possible with stable operation of the system.
We also determine actions for project risks, by which we assign additional measures and decisions.
On risk “RO2. The presence of hard-to-reproduce cases that cannot be detected in the test environment ”, you can take the following:
It is important to have an action plan in case one of the product or project risks works. Sometimes it may save setting up a backup system of the project. It is not always possible to reduce the risk to a minimum level, but it should be possible to reduce at least its consequences. Our post “Christmas Story” was on this topic : how a risk can work, the probability of which tends to zero.
For example, we must completely eliminate data loss during a transaction, but consider all cases too laborious. Therefore, you need to have ways to handle such cases. One of the options for security is the development of more detailed logging. This will provide a permanent rollback point to the previous action if something went wrong during the transaction.
Risk-based testing allows you to cover the most risky areas with test cases, thereby reducing their impact and likelihood of being triggered. This is the most winning strategy for systems with complex business logic and high cost of error. The solution is suitable for the banking sector, insurance companies, complex internal CRM-systems of a medical profile. With a risk-based approach, we also work with project risks, which improves the overall process of testing and project management.
In the article, we will tell how we at SimbirSoft worked with risks (using the Internet acquiring system and other projects as an example) and what methods of risk assessment and management we use in customer projects.
Risks at the start of the project
To begin, we give an example from our practice in the development for the banking sector.
Customer's proposal: to focus on the web version of the bank for individuals.
The risk we identified: a possible loss of audience due to the low demand of the web version for individuals, in contrast to the mobile client-bank.
Our arguments: statistics of user preferences based on reviews and distribution of the audience by mobile and web versions.
Conclusions: for individuals, the mobile version is more convenient, since the phone is always at hand. Operations are fast, the authorization system allows you to access all convenient services. In this case, quick access to a limited set of the most popular functions is important.
For legal entities, the completeness of the functions provided, the ability to upload, print, and work with a large amount of information are important. For this, the web version is more convenient.
Our solution: to focus on the mobile client bank for individuals.
At the beginning of work with the project it is important to choose the testing strategy correctly. Let's look at examples of why it is important and how to choose it.
Testing strategy must meet business objectives
Sooner or later, all companies are faced with the need to organize the testing process and come to understand that building its strategy is an important stage in software development. Sometimes - through their own bitter experience. It is especially dangerous to underestimate the role of testing and strategy selection in the development of large-scale projects. The testing process should be selected according to the business goals and specifics of the project, otherwise it will not lead to positive results either in a month or a year.
For example, consider testing mobile and web applications for a bank. At the start of the project, we selected a strategy based on requirements with a low level of detail. We used checklists to reduce testing time and support the test basis. With the growth of functionality, the addition of acquiring, sms-authorization and notification, more complex systems, checklists no longer cope with their task. Over time, more and more QA specialists joined the team, it was necessary to transmit information and coordinate their actions. With the complication of the product, any change could affect related functions, that is, the risk of regression increased. The need for automation of regression tests was brewing, so we switched to test cases.
Conclusion:depending on the project, its specifics or stage of development, the testing strategy changes.
The strategy must be selected for the project goals in order to ensure the best quality product. She answers the questions “what”, “where” and “when” will be tested. At any moment in time, you know at what point you are and where you will come in the future - according to the strategy.
A business goal may be to ensure the security of customer data, as well as the software itself at the production stage. Security begins with the development process and continues through the testing phase.
For example, at one of the projects, we created a safe environment for development and testing, deployed an infrastructure that met all the requirements and helped protect data. We requested certified tokens and name-based flash drives for each QA specialist to access the test infrastructure. Thus, we ensured the business goal of the project in software security and kept confidential the client and user data.
Due to the testing strategy, emphasis can be placed on really important aspects for a particular project. It is logical that the release of a mobile game or a complex banking CRM-system requires different approaches to testing.
Banking Testing Strategy
In our practice, we at SimbirSoft used the whole range of development methodologies, but flexible technologies always remain our priority. And even where, for a number of reasons, it is not possible to use them, the team adopts the best practices and applies them in everyday work. Testing becomes an integral part of the whole process and merges into the overall workflow. In this case, it is responsible not only for the quality of the product, but even for the quality of the entire work process.
What technologies do we use:
- flexible planning and preparation of internal releases;
- user story preparation;
- holding of rallies;
- conducting retrospectives.
In full force, the testing strategy reveals itself in projects with complex business logic. For example, software for information support of banks, the construction of an Internet acquiring system, an automated trading platform. In such projects, it is important to apply a suitable testing strategy, since the price of some errors can lead to real losses and greatly affect the company's reputation for the worse.
Also, the main goals of testing - defect search and software verification for compliance - may be added additional. For example, it is important for banks to quickly implement the new requirements of the Central Bank. This means that timely testing with the required quality for critical tasks will be added to the main goal.
Recently, in a banking project, we were faced with a change in federal law - an increase in the VAT rate from 18% to 20%. A lot of preliminary work was done to adapt to the change of legislation, since the change concerned not only the replacement of forms, interfaces, but also the alteration of the logic of several calculation formulas. It was necessary to ensure the correct display on many platforms. Also, in the deferred payment function, it was necessary to take into account the transition period with rates of 18% and 20%.
Now we’ll talk in more detail about how we build our strategy and why we often choose risk-based testing.
Pros of risk-based testing
There are several testing strategies that are selected for specific purposes:
- based on requirements;
- methodical;
- reactive strategies;
- advisory strategies.
In the case of working with projects with complex business logic, it is necessary to determine stringent requirements in the design of systems on which testing is based on. A suitable tool is requirements-based testing.
One type of requirements-based strategy is risk-based testing. Moreover, those parts of the system’s functionality that are most exposed to risks are tested first. Risk is a possible negative consequence of a malfunctioning system. The consequence is a risk in the presence of two components, such as opportunity and negativity.
There are two types of risks:
1. Product risk
It can be managed and unmanaged. In the example above, we were faced with a manageable risk: rapid growth and complexity of functionality, which increased the likelihood of regression. Here we solved the problem by having an understandable test base and subsequent automation. The risk that we cannot affect is dependence on external systems and their failure in the integration process. Here we lay the measures that will reduce their impact on our system. For example, using backup, handling exceptional cases, displaying warnings for the user.
2. Project risk
For example, on a project, we were faced with the fact that the customer had not previously worked with a distributed team, and therefore the used workflow did not meet the requirements and created additional communication problems: lack of understanding, duplication of tasks, performance of mutually exclusive tasks, and so on. We agreed on the restructuring and improvement of the process - revised the workflow, introduced all the team members, held rallies, presentations and retrospectives. As a result, the work went in the right direction.
The risk-based approach allows you to compose a certain number of risks, for a short time to test the risks with high priority and further provide the customer with metrics on how well they were tested, showing the number of planned and completed cases and the number of defects.
The implementation of the risk-based approach on the project takes place in four stages:
Risk Identification - at this stage, you need to identify the risks and get their list.
Risk Assessment - here we analyze the list and classify it by priority.
Risk Mitigation - at this stage we determine how deeply we will test the risks.
Risk Management - here we decide how we will continue to work with them and go through them, identify new risks.
Risks are identified and assessed by a group of stakeholders during brainstorming sessions. The team should include business analysts or carriers of knowledge about the system from the customer, developers, manager or project manager, architects, QA specialists. We involve information security specialists, employees who work directly with the current system, and business analysts who are immersed in processes to identify and assess risks.
Consider the risk-based approach by the example of testing the Internet acquiring system.
Work with risks (on the example of an Internet acquiring system)
We
distinguish the following requirements: D1 . Providing 1000 simultaneous connections to the system per second.
D2. Transaction Security.
D3. Access to the transaction should be available only to the person making the transaction.
D4. Providing and supporting the SET (Secure Electronic Transaction) standard.
As a product risk, we can distinguish:
RP1. System crash with simultaneous connections.
RP2. Using SQL injection during the transaction.
RP3. Access to someone else's transaction when changing parameters in the URL.
RP4. Loss of data when the connection with the bank is lost at the time of the transaction.
RP5. The use of invalid certificates when providing the SET (Secure Electronic Transaction) system.
As organizational risks:
RO1. The fall of the developed system due to the inaccessibility of external systems.
RO2. The presence of hard-to-reproduce cases that cannot be detected in a test environment.
Thus, each risk logically follows from the requirements that are in the system, but is not limited to them. Risks complement the requirements and identify additional cases that may arise when working with the system.
Risks can be reduced or compensated depending on the project objectives and system requirements. It is accepted that the risk is covered by the test case at least once:
1. For each product riska set of test cases RP1-RP4 is prepared with the condition that each requirement and each risk should be covered by at least one test case. Depending on the purposes of testing, the set of test cases is expanded to a certain level of detail.
2. For each organizational risk, a list of activities is prepared that can reduce the impact of risk on the product being developed.
Risk Assessment and Management Techniques
There are several methods for assessing and managing risks: lightweight methods (PRAM, PRISMA) and more formal (FMEA, FTA).
With the FMEA model, the project team identifies all processes, components, modules of the system in which a system failure or error can occur that can lead to degradation of product quality. During the brainstorm, the risks of the system are determined by three indicators: severity, priority, probability. Then the Risk Priority Number is calculated for each risk and, depending on the indicators, the testing depth is laid.
With the FTA model, all the causes that can lead to defects in the business processes of the system are determined in stages. The analysis goes from the highest to the lowest level. The difference between FMEA and FTA is that the first approach focuses on the functionality of the system, and the second on business processes.
In addition to these formal and heavy approaches, there are more flexible and informal ones. For example, PRAM (Pragmatic Analysis and Risk Management) and PRISMA (Product Risk Management) methods. They successfully and easily combine risk and requirement based strategies. Basically, these approaches use requirements specifications as input, but stakeholders can also be involved.
Lightweight risk analysis methods are very similar to formal ones. They focus on technical or commercial risks, weighing the probability of occurrence of the risk and the factors affecting this probability.
However, the only two factors used by these methods are the likelihood of risk and its impact. In addition, these approaches use simplified qualitative judgments and scales to make decisions.
Our teams use flexible lightweight methods and adapt PRAM and PRISMA approaches to their goals.
How we work with risk-based testing
For example, on one of the projects, we identify project and product risks that may work. To do this, analysts, developers and QA participate in the analysis - one representative from the team.
A risk table is formed with the products. They determine the assessment of the probability of a risk occurring and its possible impact on the system on a five-point scale. In table 1 - the strongest influence, 5 - the weakest. Also for probability 1 - high probability, 5 - low probability.
How we continue to work with product risks
Further, for each of them, the risk coverage of the product is traced with test cases.
We select the following checks:
TC1. Checking the load with more than 1000 connections to the
TC2 system . Load test with 1000 connections to the
TC3 system . Enter SQL injection during transaction
TC4. Enter SQL injection on the successful transaction page, by substituting other
TC5 data . Enter XSS (Cross-Site Scripting) scripts in the available fields for input when the transaction
TC6. Simulation of a disconnected Internet connection during a
TC7 transaction . Exit the transaction session at the confirmation stage
TC8. Validation of expired certificates during transaction
Thus, priority checks are TC2, TC4, TC5.
TC6 and TC8 have a high impact, but less likelihood, so the priority of checking for them is lower, but checks are also required.
TC7 does not apply to any of the requirements, but extends the test for a negative scenario, which is possible with stable operation of the system.
How we continue to work with project risks
We also determine actions for project risks, by which we assign additional measures and decisions.
On risk “RO2. The presence of hard-to-reproduce cases that cannot be detected in the test environment ”, you can take the following:
- Prepare a pre-production stand, which is as close as possible to the real application environment, in conjunction with all external systems;
- write end-to-end testing scripts that pass through all adjacent systems and provide transaction verification;
- after carrying out all priority checks, use the error guessing techniques according to the basic requirements of the system and scripts for additional checks in the role of a “system hacker”.
Backup plan
It is important to have an action plan in case one of the product or project risks works. Sometimes it may save setting up a backup system of the project. It is not always possible to reduce the risk to a minimum level, but it should be possible to reduce at least its consequences. Our post “Christmas Story” was on this topic : how a risk can work, the probability of which tends to zero.
For example, we must completely eliminate data loss during a transaction, but consider all cases too laborious. Therefore, you need to have ways to handle such cases. One of the options for security is the development of more detailed logging. This will provide a permanent rollback point to the previous action if something went wrong during the transaction.
Conclusion
Risk-based testing allows you to cover the most risky areas with test cases, thereby reducing their impact and likelihood of being triggered. This is the most winning strategy for systems with complex business logic and high cost of error. The solution is suitable for the banking sector, insurance companies, complex internal CRM-systems of a medical profile. With a risk-based approach, we also work with project risks, which improves the overall process of testing and project management.