“It is important to prioritize”: about testing at Sberbank Technologies

If you think about for what Russian companies testing can be a particularly hardcore task, Sberbank Technologies immediately comes to mind . Firstly, there is a gigantic scale, secondly, a lot of responsibility (financial operations are not photosharing with geolocation for you), thirdly, a course has been taken for the fastest release cycle: that is, you need to test a lot, very high quality and at the same time very fast.

What is life like with such almost mutually exclusive attitudes? Following the Heisenbug conference , where the company was a sponsor, we asked several questions to Anton Romanov from her Quality Department, deputy head of the corporate credit systems testing department.

- Prior to Sberbank-Technologies, you worked at Aplana, where you were also involved in testing Sberbank systems, but from an external contractor. But how do testing tasks between the company itself and the contractors are generally shared in Sbertech?

- Historically, contractors were initially involved mainly for regression testing. And I worked at Aplan just when mostly regression testing was handed over for a row, and Sberteh controlled. And now, as the volume of work increases, contractors are also involved in testing new projects.

- It is obvious to everyone that Sberbank-Technologies is a big company, but can you give an idea of ​​how large it is in terms of testing using any numbers?

- About 1,500 people work in the Quality Department, these are 38 departments. About five departments are received, which are divided by functional competencies: Load Testing Department, Automated Banking Systems Testing Department, Front-end Systems Testing Department, Administration Department. And each department has about five departments.

- When there are so many departments, it becomes interesting: how much is all this unified? Do you have rigorous uniform standards for testing that everyone obeys, or do departments have a high degree of autonomy?

- I would say we have a middle ground. For example, the testing organization department sets some guiding practices. For them, criteria are defined that are fulfilled or not fulfilled under certain conditions (testing at an early stage, matching their tests with analysts, etc.). That is, the teams work according to the general set of rules, but depending on the features of the system and the development process, the team independently determines which practices are applicable and how they will be implemented.

Naturally, the processes are transparent and open: with active feedback, coordination and understanding of work processes.

- And can the team use some innovative practices that have not yet become mainstream, or in your case always requires time-tested?

- Of course, we need reliability, but this does not mean that you can’t work with something new. For example, mind maps (mental maps) were until recently considered an innovative practice. We use them, but as an additional tool. And the use of only mind maps in exchange for any other test documentation, of course, is not allowed. Thus, using something new, we calculate and mitigate risks, often conduct similar tests in pilot mode. And only if we see efficiency and result, such decisions are practiced and replicated.

- Do we correctly understand from the outside that the high responsibility of financial transactions affects the approach to testing?

- Where it comes to money, there is a huge responsibility - and of course, this affects our approaches to testing. The process is quite strict, with several stages of approval - transfer from development to testing, from system testing to integration testing, from integration testing to acceptance. That is, everything is rechecked several times, there are strict transfer criteria for such a distribution kit, the implementation of regression testing and the quality of testing new functionality are monitored.

- And also, probably, in your case, load testing is especially important?

- Yes it is. For example: there are about 150 different systems interacting with each other in Sberbank. Our department has one large system, which receives about 500 payment documents per second in one operational day. Accordingly, the volume per territorial bank is about four million documents per day. These are payments to customers, and tax and utility payments are the mass of everything.

Thus, load testing has a very important role, because Sberbank is critical for the processing time of these documents - they should not accumulate in lines and slow down banking operations. We pay great attention to how the system behaves after the next update roll-up - it is very important that the level of service does not sag. Therefore, in load testing, we try to create an environment that is most consistent with the industrial one.

- At the same time, you have huge volumes of already written code, which should complicate regression testing, and at the same time, the goal is to speed up releases as much as possible. What helps to cope with such a challenge?

“Good old practices help — risk-based testing.”

The system that we are testing passed to us by inheritance: it was originally developed by a third-party company. The documentation remained somewhere at the level of seven years ago, and new changes were scatteredly documented in different places. That is, the first task is to understand what, where and how it works.

And the next moment - volumes. In this system, there are about 15,000 operations, all payments flow into it - and accounting reports, and postings, and in general everything. Our releases have tight deadlines (of the order of two weeks for everything), so it is important to determine what to check in the first and second turn, and to what extent. Defining priorities, we attract colleagues from development, support and Sberbank - we are conducting a comprehensive dialogue with interested parties.

It is very important to automate processes and thus save time. But still there are checks that can only be done manually, and here we use the Lead-Time metric (we mean the time from the beginning to the end of the regression) - the time it takes to perform regression testing. We use the data on the test execution time, which is stored in our tool, and on the basis of this we estimate how much time we need to perform the regression, compare the same time with the calculation through the date, just the end date - the end date (two calculation methods - through actual completion and start dates and through the volume of tests and the average execution time; so far, “fact” exceeds the calculated value). So far, we have differences and we are thinking about how to reduce the time it takes to run individual tests.

- About a company of the scale of Sbertech, it seems that there should be many tools developed inside that - do you encounter such in your work?

- It depends on what exactly to talk about: we do not have our own JIRA, but many utilities are being developed for individual testing needs. For example, the creation of special messages in a specific format, test data generators, SQL scripts are very common.

- Earlier, Sberteh reported that DevOps is being implemented in the Quality Department - and what does this mean in practice for your department?

- With the advent of devops, practices such as Continuous Integration and Continuous Deployment appear. In the department where I work, they are not there yet, because our system belongs to the legacy class, which will be replaced by a new platform (it is being developed as a web application). But colleagues from other departments are already working in DevOps, which really saves time and speeds up the whole process. Because everything is collected automatically, it rolls and starts at night, and you only have to see the results in the morning.

- About Sberteh, it is known that he is actively hiring developers - and in testing also constantly attract new people?

- Now a new IT platform of Sberbank is being actively developed, which will combine a large number of systems that currently exist separately. This will be a single platform, consisting of a large number of modules that will perform individual functions. But in parallel, we process applications for existing systems. Therefore, of course, we need people to carry out this really large-scale task.

You are a certified tester up to the ISTQB Full Advanced level - how much do the knowledge gained and the fact of obtaining a certificate affect Sbertech?

- In my work, this knowledge is very useful. Often there are some tasks from the series “some kind of business process is being designed, you need to somehow evaluate the number of tests in order to roughly plan the work, roughly understand how much regression will take”. Here we need test design methods that are discussed in ISTQB. They help, having not even very detailed documentation and based on assumptions about existing risks and quality criteria, to determine the possible volume - maybe not in terms of tests, but in terms of checks.

Also, a risk-based strategy is very applicable to our regression model. How to assess the risks, how to prioritize them, whom to bring to this - all this is very useful and has helped me more than once in my work. Well, in general, as it seems to me, the knowledge obtained in preparation for certification determines the further approach to work. The installation is gradually being put into my head that you can’t test everything: you need to proceed from risks, prioritize and ensure maximum quality with available resources. With this approach to solving problems, a completely different result is obtained.