Tape recorder - a tool for recording autotests

Good afternoon, dear readers. My name is Victor Burov. I work as a developer in the company ISPsystem and I want to share the experience of test automation.

It so happened that we had manual testing prevailing, and testers spent a lot of time doing the same actions. Once we thought: why not teach the panel to repeat the actions of the tester, because, in fact, they all turn into specific API calls. This would allow people to write tests even without programming skills.

We decided to write a module for creating automatic tests. So that the tester can simply press the test creation button, fulfill the conditions of the test case, at the end click “finish” - and that’s it, the test was ready! A simple idea, but it turned out to be difficult to implement. Because we wanted this module to be adapted to our products as much as possible and take advantage of the unified interface: to make the recording look like a ready-made test case. This would completely eliminate the manual work of writing tests. The resulting system was called the “tape recorder”.


Test Case Interface Module Interface

Principle of operation

All request parameters (HTTP headers, environment variables, POST data, if any) and the entire response is written to the xml file. Each entry is assigned a sequence number. All requests are divided into modifying and non-modifying. After the test has been recorded, many of the non-modifying requests are cut, as they do not affect the execution of the tests and only delay and confuse the execution process (hence the missing sequence numbers in the screenshot).

During recording, the tape recorder allows you to set up checks in the fields on the forms and in the columns of lists in one click. It also allows you to record negative tests, remembering what error the panel returned during the test recording.

During playback, requests are sent directly to the application API, without using a browser.

In fact, a tape recorder is a module that is embedded in all our products and allows you to install event-sensitive handlers. Written with COREmanager.

An example of recording a single call made by a tape recorder:

Improvements (what we did not think in advance)

Expectation

A person can “just wait”, a computer is not. One of the first problems that the tape recorder had to solve was the writing of the Vaters. What did the people not think of to wait for the completion of the operation. The expectation of background tasks and the ability to add a stop at a specified step for a specified number of seconds was implemented.

Record and edit test steps

Probably everyone remembers the time of typewriters: one mistake - and you have to reprint the whole page.



So that the tester did not have to rewrite the entire test with an erroneous action, a mechanism was added to record the test from any step after saving. The ability to edit comes in handy when you need to adapt the test to changes in the behavior of the tested functions.

Macros for variables

When executing tests, the values ​​in the transmitted and checked parameters changed depending on the server on which they were launched. An example of such data is IP addresses. It is impossible to recognize them at the stage of recording the test, so I added a macro system. This allowed us to create tests that were not so tightly tied to the environment. The lack of solution is that after recording the macros must be specified manually.

Another problem that significantly complicates the work with a tape recorder is the use of non-native keys. We did not notice it immediately, as we tested the tape recorder on ISPmanager, which uses native identifiers. But in some other panels, the entry is identified by a unique ID. Therefore, we had to teach the tape recorder not only to get an identifier after creating a record or an object (since the ID can change from launch to launch), but also to insert it into all subsequent requests.

JUnit format support

The tests created by the tape recorder are launched automatically in the Jenkins continuous integration environment. After executing the tests, an xml file is created containing data in the JUnit format. That the file was correctly formed, restriction on naming of tests was entered. For example, the User.Create.xml test fell into testsuite with the name User and, therefore, it had a Create case. In the event of an error, a failure child node was added to it with a complete description of the error.

Metrics

The number of unique functions called during the test is calculated and the percentage of the total number of functions is determined, excluding those available only for internal use. Thus, the simplest test coverage is measured. In addition, the metrics show the total test time and the number of successful and lowered tests.

Test repository

The test storage first solves the problem of transferring ready-made tests to other servers. It also comes in handy when tests are written by several testers. A small panel for storing tests Storage was developed and deployed also on the basis of our COREmanager. There is a test synchronization module with storage in the tape recorder. When writing a new test or unloading tests from the repository, they are automatically unavailable for loading into the repository, so that there is no confusion. After changing the test, it will increase the revision number so that only tests after the changes are loaded into the repository.

Difficulties (well, where do without them)

Using a tape recorder showed that not all API functions followed our internal recommendations. In particular, not all functions returned a record identifier after its creation. I had to go back, including to the working code, and bring it in line with the requirements.

Deadlock became another problem. The panel implies the execution of some critical actions in exclusive mode. And the tape recorder, being part of the same panel, causing such functions led to the hang of the entire system. It was possible to determine this only with the help of GDB (no one remembered this feature). Unfortunately, it was not without crutches, because it was decided to run these tests in a multithreaded mode when running tests of the tape recorder. Theoretically, it was possible to arrange the tape recorder not as a module, but as a separate panel. But we did not try.

Write and forget, too, unfortunately, did not work. The interface of our products is changing and becoming more complex. Moreover, the number of interface components is actively growing. Therefore, the tape recorder has to be refined from time to time so that when new components appear, it can analyze their structure and process the results of query execution.

Conclusion

The creation of the tape recorder helped to improve the quality of the tested products. Saved time and resources for training testers. Along the way, the tape recorder allowed us to review our API for compliance with internal recommendations, and, therefore, to make the API a bit more “logical”.