Automated regression testing approach

Hello dear readers. Today we would like to coincide with the launch of the course "Python QA Engineer" . Anticipating possible questions, we warn that there is not a word about Python in the article, but nevertheless we find this material useful for testers, so we decided to share it with you.

Testing every smallest detail of the code is not feasible, therefore, regression testing should carry out comprehensive verification and focus on a specific area in its entirety. The main goal is to ensure that not a single regression error will harm a critical business process. It is this effort that makes it possible to capitalize on automation. An automated testing approach aimed at reducing regression errors will help to go a long way towards building good customer relationships and increasing brand value.

My team is responsible for testing an accounting application that uses sophisticated calculations and records book entries in the accounting system. All this is a workflow, and closing books every month in it is the most important.

With each release, there are some changes in the calculations, for example, for the account it may be necessary to increase the debit or credit, or the existing value may be divided between the two accounts. Such changes in a complex application that has many internal and external dependencies often lead to unpredictable behavior, including regression errors, in unexpected and seemingly unrelated places.

Let's deal with a regression error, it is sometimes called a critical incident or production problem. After working for several years in the software industry, you realize that the regression error that leaked to production is much more important than the incorrectly working new feature. The fact is that the new function is still hidden, so when it breaks down, the impact on the business component is minimal, while with a regression error, the user relies on the work function and most likely does not have a backup version of the working system.

Regression errors are caused by changes that were not taken into account by the project manager or product owner during acceptance tests; architect, developer during code review at the design or implementation stage; or by a QA-analyst and tester in test cases. All security networks missed an error and the user came across it. If an error in a recent update was detected at any of the above stages, i.e. teams, interested parties or any other link that is present in your organization, then it was reviewed prior to release, or at least evaluated for criticality.

Regression errors entail several costs: urgent correction, fines for potential violation of the service level agreement and, most importantly, they damage the trust of your users in your product. They may decide that a new release will break something that already works.

It became important for my team to check absolutely everything.
However, it is well known that this is not possible, or at least too expensive and time consuming. Therefore, “testing everything” focused on the following two things: critical business processes and the belief that the behavior in the main critical business processes will be the same as in the latest release, before the change. In general, we wanted to make sure that in the latest release, a regression error did not appear in the critical business process.

We evaluated our typical approaches to automated testing to verify the calculations and to verify that all the logic is described in the code for testing automation. This approach raised the classic question: “Who is testing the tester code?” If the test case was unsuccessful, there remained the same probability that the problem was in the tester code. In addition, with every change in the application, the test code also needs to be updated, and such changes have occurred frequently.

Also, thanks to automated testing, we made sure that we have a fixed set of inputs and a well-known set of outputs. Due to the complexity of the application, it was impossible to submit all possible sets of input data at a time. Various data sets were also needed for the end of the month, quarter, year. The schedule was a serious problem, because it takes a lot of time to generate a huge set of input data and corresponding scripts.

There was another variable: user data. We had a special privilege: we received backup copies of user data every month. The quality of the test directly depends on the data used for testing, and the data from the production is always better than the generated data, so our privilege was a huge advantage that we did not want to miss.

Identification and implementation of regression tests

Our idea was to use automated testing, which requires the minimum necessary maintenance, and which strengthens the confidence of stakeholders that the release will be of proper quality.

So, we needed a testing strategy for critical business cases that would guarantee the absence of regression errors and, of course, that we could quickly put into practice.
Our preparation process looked like this:

• Database backup from production, restored twice;
• Two test systems are installed that work in parallel:
  
  • One with a production code;
  • Another with the current version of the application under test.

This approach provides two identical systems with differences in code in only one version:
Having two systems is quite important, as it helps to understand that any problem arises only due to the latest changes to the code.
Tests are separated, so from the standard process “perform an action and get a reaction”, we proceed to the fact that actions are performed from one point to another while maintaining the workflow, after which the error reports are compared. This is the key to detecting unexpected errors.

When the tester focuses on a new function or some kind of change, the test turns out to be focused specifically on it, i.e. the relevance of a particular change is checked. Regression testing is different in that it must verify that nothing else has changed. This difference is reflected in automation scenarios. It makes specific function testing scripts unsuitable for searching for regression errors, so a different approach is needed here.

For example, if you work with an order management system, you will need a script to place orders with a lot of input data to place orders for two installed testing systems (preferably working in parallel), then you get a report on orders for the past day and compare each value. Then all orders are confirmed or approved (this action), and reports, such as daily confirmed orders, stock orders, warehouse reports, orders from each carrier, type of shipment, type of payment, etc. will be compared in two systems. This continues throughout the entire workflow. You can combine actions, such as placing and confirming orders, and then compare reports at separate stages.

Another example is the hotel management system, in which individual actions are defined as critical business processes, such as check-in, billing at a restaurant, and receiving inventory. All these processes will be assigned their own actions and reports. The difference in this system compared to the previous example is that the test suites can run in parallel, and there is no need to complete the step before starting the next one.

A comparison of the two reports is a moment of truth, and it should be impeccable, in the sense that none of the interested parties should have doubts about its correctness. The difference in the reports is the real difference.

For this operation, we use the web service interface. All reports are sent in parallel on two systems, as a result, the received response in JSON format is compared.

The comparison takes place on three fronts:

• Source (production) minus target (application under test);
• Target minus source;
• Comparison of values ​​to obtain the intersection.

This method will work for XML, XLS, CSV fixed width or any other format. We need to make sure that there are no unnecessary records, there are no missing records, and all the values ​​of the records match.

This is the essence of the approach we are talking about. All this is information readable in the application, which is issued as a report or, in some cases, working as an interface to other applications.

The success of this approach lies in a comparison tool or utility that processes cases related to your application. You can consider yourself lucky if you find something suitable "out of the box", otherwise, it is important to understand that investing in such an instrument is worth it because it will bring good results.

After all the talk about automation, you need to insert a remark. Since some differences in the reports are expected, since they must be there as required, all results must also be analyzed manually. There should be clear successful results of passing the test cases, however, unsuccessful results must also be analyzed and their validity must be confirmed. Since we are talking about regression testing errors, they should be fixed before release. Of course, there are some exceptions that are handled in accordance with the application.

Program setting

All applications are different, and installing and configuring them also happens in different ways. To prepare the application for testing, some additional steps may be required, therefore, they must be taken into account at the right time and in the right place to run the tests. Here are a set of typical steps:

• “Confuse” data from production by deleting e-mail identifiers or other confidential information, or replace it with fictitious data;

  
  

• Get the data in the proper form to run the test;

  
  

• Adapt settings for the QA environment, for example, by changing the integration relationships.

  
  

  The only thing worth remembering is that the above steps must be performed for both installations. Remember that before starting the test suite, the settings must be identical.

  
  

  Often, actions other than requesting a report can return an object, for example, an action, such as creating or modifying an order, can return a new order object. In this case, you need to compare two objects and not wait for the comparison of reports. This can help identify the error in the early stages at the root.

  
  

  It is also recommended that you break the entire set into smaller sets, for example by grouping transactions and related reports together. Sets can be run in parallel to save runtime. However, for an application with a typical workflow, this only works if you can split the cases vertically and horizontally, or vice versa.

  
  

  Variations can begin with technologies - JSON, XML or scalers (int / string / float), and expand until the application under test and production respond with different structures, but still comply with the architecture. For example, the production version may use the old JAR file, which operates in a specific format, and in the new version the JAR file was updated and now the response format has changed, so their comparison will show inconsistencies. In order to compare them properly, you need a temporary plug-in.

  
  

  There will probably be few such situations. In such cases, it is sometimes easier to tweak the design or consider them in the context of a workaround.

  
  

  There are several options for handling such comparisons:

  
  

• Ignore some fields, such as identifiers and dates;

  
  

• Ignore numerical differences less than 0.0001;

  
  

• Игнорировать чувствительность к регистру;

  
  

• Структурировать изменения в два ответа.

  
  

Улучшение регрессионного тестирования

Регрессионное тестирование должно быть целостным и фокусироваться на целой области. Этот баланс поможет извлечь выгоду из автоматизации. Автоматизированный подход к тестированию поможет уменьшить количество регрессионных ошибок и поможет на пути к хорошим отношениям с клиентами и повышению стоимости бренда.

Теперь, в ритме постоянного движения вперед, наша команда хочет попробовать отказаться от двух идентичных установок системы, которыми мы пользуемся сейчас, и реализовать ту же стратегию на одной установке. Мы хотим сохранить ответы прошлых выполнений и использовать их для сравнения. Поход к тестированию всегда можно улучшить. Пожелайте нам в этом удачи!

The translation was prepared especially for students of the "Python QA Engineer" course .