Impressions of the best reports at the SQA Days conference

Along with other areas of IT, testing is developing rapidly. And here it is equally important to keep abreast. One of the available ways to get new information is to attend conferences / seminars. And we successfully use this method.

On May 20-21, 2016, the 19th international conference “SQA Days” was held in St. Petersburg. The event was held in three streams, and during the two days of the conference 57 reports on various topics were heard here: the testing process itself, the principles of building a team of testers, professional growth and motivation of a QA specialist.

Reports on test automation went in a separate flow. Many speeches were devoted to such trends as DevOps, Continuous Delivery. We talk about the most interesting reports of SQA Days-19.

Jenkins 2.0: We organize testing as part of Continuous Delivery (Oleg Nenashev)

Oleg is a member of the core team in the open-source project Jenkins, works for CloudBees, where he develops the CloudBees Jenkins Platform. Oleg touched on the topic of Continuous Delivery, as well as the tasks that arise when implementing this methodology.



With Continuous Delivery, product testing takes place in short cycles at all stages, the tests are automated as much as possible. If necessary, Continuous Delivery involves a quick rollback to the previous version of the product. In this methodology, the infrastructure becomes critical to the project. As soon as a failure occurs, we immediately begin to incur losses, especially if you need to roll back the changes, and not release new functionality. Infrastructure changes with the product and should be tested along with it. Within the framework of Continuous Delivery, these are, of course, short cycles, dividing the system into modules. What is needed to organize such testing? Parallelization of processes in the project, versioning of code and assembly procedures, reproducibility of tests and assemblies, testing on a real release.

Implementation of Continuous Delivery requires an automation system. Oleg spoke about the new features of Jenkins 2.0 and focused on the new Pipeline as code technology. The classic Jenkins interaction model is the web user interface. The user is required to manually create a new task, and then supplement it with details. In this regard, additional efforts are required to create and manage projects, in addition, Jenkins task settings are stored separately from the project code. This approach makes it impossible to apply the best Continuous Integration / Continuous Delivery techniques. Jenkins Pipeline allows you to describe the assembly process in the form of code, so that the configuration of Jenkins tasks can be stored in the version control system along with the project code, track changes and test with the project.



In conclusion, Oleg spoke about the successful use of Pipeline in the CloudBees Jenkins Platform, and also concluded that Pipeline as code reduces automation support costs and can be used for Continuous Delivery.

How to stop being afraid and start automating. Or not to start (Alexey Lyazgunov)

This report will be primarily interesting:

• Test managers who want to implement automation in their project
• teams whose automation is “as if working,” but it’s not clear whether it brings results,
• as well as engineers who automate some of their actions and want to bring it to the command level.

The report turned out to be detailed, such moments were touched upon: what is automation, roles in automation and its consumers, what is expected from automation, why automation is expensive and long, goals and expectations from automation, causes of failures. The speaker dwelled in detail on the goals of automation and conditionally divided them into good, bad and doubtful.

• Reduce testing time . At first glance, the goal is good, but it all depends on what needs to be automated. For example, if this is new functionality, then the duration of testing will increase significantly. Therefore, you need to understand that if you need to test quickly and right now, then you probably should not automate.
• Increase the number of checks / coverage .
• Provide more frequent checks . One of the main goals of automation.
• Reduce the influence of the "human factor" . According to Alexei, a dubious goal, but in some cases a good goal. Indeed, sometimes it is necessary that the test is carried out according to predetermined sequential steps. But you must understand that quite often tests do not catch defects that a person can detect.
• Cheaper testing . Automation is expensive, you have to pay for new practice, and constantly.
• Introduce new types of testing . Automation allows us to do what we cannot do with our hands.
• Reduce the release time of a new release . A good goal, but at first it’s a little hard to achieve, since there should be some basic set of tests.
• Replace manual tester with autotests .
• Fast getting current status . Great goal. For example, we want to know whether it is worth giving the build for testing, and for this we run a certain set of tests.
• Automation of manual scripts . From the speaker’s point of view, this is a bad goal. Of course, it is good to automate the set of acceptance scenarios, however, if we cover with autotests in more detail and broadly, then we will encounter a number of difficulties. First of all, manual scripts are mainly UI tests, which is the most expensive and unreliable automation. Further, different details of tests, different breakdowns of checks by script. Sometimes it is better to use different interfaces (for example, creating a user is better through the API than filling out forms on the frontend). Also, not all manual tests are automated, and not all possible tests have manual scripts.
• Automate regression testing . If we want to automate regression testing, then we need to understand that the pace of writing new autotests should be the same as the pace of product development. If possible, you can set yourself such a goal, otherwise it is better to automate regression testing only at the module or component level.
• Automation will simplify testing .

Summarizing the goals, Alexey proposes first to focus on the goals highlighted in green, and only then set “blue” goals.

Then Alex introduces the concept of “Three DOs”: Automation is Good, Expensive, Long. This is a list of questions, but which is worth answering when implementing automation.

Automation is welcome:

• What problems do I want to solve by implementing automation?
• My expectations from automation - not delusions?
• Is there support and understanding of the leadership and team?
• Do I know how to integrate automation into a process?

Automation is expensive:

• Am I willing to pay for automation?
• Am I ready to hire new people?
• Am I willing to pay for infrastructure, tools and training?
• Am I ready to pay constantly?

Automation is Long:

1. I understand that the benefits of automation are not instant?
2. I know that all participants in the development will be involved and need to change the approach to development?
3. I understand that, having started, I can not stop?
4. Is the project long enough to enjoy the fruits of automation?
5. Am I willing to spend time researching in automation?

As a conclusion, Aleksey proposes to draw up a checklist and, if based on the information collected, the question “is it worth implementing automation or not”, you will answer “Yes”, this does not mean that you can implement it. This means that maybe you will succeed. But if your answer is “No,” then most likely you will not succeed. But if you do not try, then it definitely will not work.

Automation Methodology Assessment - MBT (Dmitry Khimion)

Technically, the report was strong and of high quality. It was dedicated to the Model Based Testing (MBT) methodology and its applicability within the Agile framework.

In short, MBT is one way of organizing tests as part of its automated testing system. Any application is a state machine.



In the classical sense, there are no test cases for this methodology, there is only a model of the system under test. There is no such entity as an automated test, test scripts are generated during the autotest run.

Dmitry compares MBT with the traditional approach to automated testing. Any state, like any transition from one state to another, is described once and is used only once in the generation of tests by the combinatorial mechanism. The amount of change is about the same as that of the developers. In the case of using linear scripts, even when we have a good modular system of automated testing, when the general code is passed out, and it is necessary, for example, to add new steps, then we must fix all the tests that use the new functionality. With an increase in the number of tests, the duration of their development and updating will increase. MBT is beneficial for agile software development. Agile methodology always involves a work product, short sprints. All Agile teams ideally seek to complete testing at the time of completion of development. Considering that the generated test suite can be flexibly changed by modifying the model, while the time costs are approximately equal to the development of functionality, if properly organized, MBT automation either keeps up with the development, or if the development increases the timeline, it does not keep up, and in this case there is an increase in the duration of the sprint.

Dmitry further described the concept of the MBT framework, the architectural diagram of which is shown below. Source - the external source from which the model is built. Test Model - a model of the system under test, which is translated from an external source into a program view. Generator reads the model in the form of a graph and forms paths (test sequences). Restriction tool - a mechanism for limiting the generation of test sequences. Test Manager is responsible for system performance and aggregation of results. Logger - module for logging.



Summing up, Dmitry highlighted the positive aspects of using MBT. This methodology works to minimize the amount of code. MBT allows for high test coverage through combinatorics. Medotology is applicable in Agile, since the time for updating the model, in most cases, will correspond to the time of developing new functionality. Among the disadvantages of using MBT is the limited choice of tools. In addition, the modules shown in gray in the previous figure will have to be implemented on their own.