Five ideas for “arming”, or Impressions of the Moscow “Heisenbag”

How to make sure that the conference you are taking the ticket to will not be "messy"? In my opinion, the best way is to become the speaker of one of the conferences of the same organizer and to test the process of selecting and preparing reports on yourself. In the spring I was a speaker at a conference held by the JUG.ru Group, and everything that happened before that was reminiscent of a degree defense. Several times my report was listened to by the program committee, after each of the runs I received a long list of questions and comments. One of the committee members became an “opponent” and in the endless discussions in Slack required additional experiments in order to obtain unconditional evidence of the effectiveness of what I was going to talk about in the report. When the news came that the report was approved, it was not the final: I was invited, along with other speakers, to take two-day oratory courses. After going through this test, I know that the guys from the JUG.ru Group will not allow a drop of "crap" at their conferences. And I trust them: this year I attended most of their conferences.

The Moscow Heisenbag for me is the last conference of the outgoing year, closing the busy season. Judging by the statistics voiced by program director Andrei Dmitriev during the opening remarks, most people at Heisenbug are professional testers. The sensation of communication is mainly people working in large companies with dedicated QA teams.

I am not a typical visitor to the Heisenbug. In our company there are only about 25 people, and many combine roles. I have to be a technical leader, a project manager and just a “problem solver,” and we only have a couple of people involved in test automation. The survival of our company literally depends on the effectiveness of the team. Increasing efficiency is only possible through gaining new knowledge and broadening one's horizons, for which I and two of my colleagues went to the Heisenbag.

Day 1

Conference opening

A good mood has been set from the very opening of the conference. During the opening, program director Andrei Dmitriev gave the audience a task: in thirty seconds to meet someone from the neighbors sitting around. The hall was immediately filled with a loud noise of voices. It was a very useful task! After all, a conference is a tool that you need to skillfully use in order to give you the maximum benefit. Although it seems that the main component of the conference is reports, in reality you can get only about 40% of the total benefit from sitting on the reports, and the remaining 60% you need to take yourself - talking to specialists on the sidelines, discussion zones and at a conference party. If you are an IT introvert, then in order to start a conversation with a new person, you have to go over yourself. But this is difficult only at the very beginning - very soon you will find out

When the noise died down, Andrei gave another task: to remember at least five key ideas for ourselves over the next two days, which, having taken out from the conference, we could put into practice. I took note of this, and reports began in three rooms.

1

For the first time slot, I went to report Dmitry Buzdin "How to build a framework for automated tests?" . Based on the theoretical ISTQB Generic Test Automation Architecture architecture, Dmitry talked about how you can combine open-source solutions so that they completely cover all the tasks of the testing process. For me personally, the report did not contain any ready-made recipes, but on the other hand, I found out about the ISTQB model, and my knowledge of what details any test framework should consist of was seriously ordered.



At the same time, in a parallel room was a report by Sergei Egorov on TestContainers. I didn’t go for it for a very simple reason: for the first time I heard Sergey’s report on TestContainers back in the spring at JEEConf, in early autumn we successfully implemented this technology, and in October at Joker we managed to communicate in detail and successfully with Sergey on all features that we met. The idea of ​​TestContainers is simple and powerful: using the existing Docker infrastructure and downloadable images, in your test, for example, you can get a JDBC connection to a real Oracle database simply by instantiating the class, and it’s no more complicated than some kind of HashMap. Browsers, databases, and more are available out of the box by simply instantiating a class! What used to require complex setup of the test environment and configuration of the tests themselves now boils down to three simple requirements: the machine must have JDK, Maven (Gradle) and Docker.

2

Then I went to Simon Stewart's “Scaling Selenium” report . Simon talked about common sore problems that arise as the UI tests grow and become more complex: fragile locators, expectations, and scaling with Selenium, including in the cloud. Although I did not feel deeply immersed in the topic, I suddenly realized that Simon voiced the first idea, which I definitely want to apply at home: "Check for preconditions". Before starting system tests, check whether all the services on which the system depends are running. Does the database, authentication server, and other components that the application depends on work? After all, if at least one of them “lies”, it makes no sense to test, but it’s worth stopping and clearly reporting the problem. Otherwise, a large number of irrelevant errors fall into the logs, which undermine the credibility of the test results as such.

At the same time, my colleague Nikolai visited the Selenide Puzzlers by Alexey Vinogradov and Andrey Solntsev. Quiz reports are a great idea to sparkle with the speed of your mind and earn a T-shirt. Returning from there, Kolya said that he was pulling his hand with all his might, but he was never given a word. Knowing the delicate Kolya and the hype that usually happens on “puzzles”, I can assume that it was worth pulling your hand even harder :-) Selenide is a wonderful framework that solves most of the problems of the “bare” Selenium. We experimented with it quite successfully, but now we use another one. Although after Heisenbug they thought about the correctness of their choice.

3

Next, we thought whether to go to the report of Vladimir Sitnikov “Testing the performance of web applications on the browser side” or to the round table “What should the tester know in 2018?” We decided that we did not rest on browser-side performance yet, and went to the round table.



There was no table there, but there were five speakers sitting in a row, a presenter and questions asked from the audience and on the telegram channel. The discussion was mainly not about technology, but about the career path of the tester and the organizational structure of the company - things that are not very relevant for such a micro-company with a flat structure like ours. Although there were five speakers, Nikolai Alimenkov spoke most and most actively, and some are good if they managed to insert several remarks in an hour. I consider Nikolay one of the best experts and trainers in the IT field, and just a cool witty dude. But at this conference he already had two reports of his own, and therefore, maybe it was sometimes worth giving more time to other guys to speak.

I understood such a thing: to go to the "round table" makes sense only if you have a specific question that you would like to ask a group of experts. If you go to “listen,” then the thread of the conversation may not go where you expect it to be. Speakers may find it more convenient to ask their specific questions in the discussion area or at a party, which makes the very idea of ​​a round table rather dubious.

But not to say that in this time slot we had a bad time. When I was next to my colleague, I showed him several falling UI tests, because of which Jenkins spammed me in the mail, and I was nervous before the upcoming show next week to the customer. Together, we managed to quickly get to the bottom of the reasons for the fall of these tests, and I left the hall satisfied. Sometimes at a conference it may be necessary to work. And we will definitely see the report of Vladimir Sitnikov in the video!

Dinner

What I can say can be treated with irony, but I consider lunch the most important indicator of the quality of any conference. You spend the whole day, go on a business trip or just come home in the dead of night, constantly focused on obtaining knowledge - this is energy-intensive. The food at the conference should be such that thoughts about food and thirst do not distract, and taking into account the fact that you may have specific requirements for food. The JUG.ru Group in this regard is well done, their food quality is good, although there are punctures: telegrams and kamenty on Habré have been seething for a long time about quinoa at Joker 2017 :-) This time, when I was at dinner, I found that although all the people were already clearly full, the tables were still bursting with snacks and hot dishes. This gave me a pleasant confidence, and I did not deny myself another serving of fish under vegetablesand eggplant with cheese , after which I was ready to continue :-)

4

At the previous St. Petersburg Heisenbug Andrei Satarin talked about special tools for analyzing C ++ code execution. We write in Java, so I thought that his current report “How to check the system without starting it” would not be relevant for us. But still, I went to Andrei himself and asked him. It turned out that the report was not at all limited to C ++, I stayed - and did not regret it. Hence the conclusion: never hesitate to specify in advance what the report will be about, at the speaker or at a member of the program committee. You will be readily answered, and this will increase the efficiency of your pastime at the conference.

The idea that Andrey set forth is at the same time cheap to implement and effective: the configuration files of distributed systems can and should be verified automatically. The simplest thing you can check is that these files generally exist and are valid JSON / YAML / XML, but more complex checks are possible: for example, that the distribution of services by nodes / racks / data centers meets the requirements for system fault tolerance. Or that the hosts and ports are connected to a single network, otherwise it may turn out, for example, that due to incorrect port settings, your cluster has fewer connected instances than you thought.

For me, however, the picture did not add up in one place: Andrei talked about the case when the configuration files are in the version control system in the finished form, but, for example, we have the configuration files created dynamically and laid out on virtual machines using Ansible, and on virtual computers they are no longer available. I went into the discussion area and a little “brainstorming” led to the decision: it’s enough to run the copies of the configs in a separate folder during the ansible-playbook run and immediately after that start checking the resulting configs. And although the risks that Andrei spoke about are not significant for my project, I know that over time they will become significant, and then these checks can and will need to be implemented. So the second idea "to arming" was received .

5

The author of the next report , The Abuse and Misuse of Test Automation, Alan Page participated in Microsoft Windows editions during the years when I wrote hello world in Borland C ++ at the institute. That alone was worth going to him. The report mentioned the well-known concept of the “test pyramid,” which often tends to degenerate into such a ridiculous “ice cream cone”:



“The test pyramid means you should write as few less UI tests as possible,” Alan said. UI tests 1) it is difficult to write, 2) they require huge computational resources and are still slow, 3) they are unreliable (flaky). The most reliable and cheapest tests are Unit tests, and they should be the most. And in case of detection of a bug, the test for it should be written on the lowest possible tier of the test pyramid - ideally, if it is Unit-test.

It would seem - this is the very basics! But it’s strange that I didn’t formulate this for myself in such a crystal clear form before. Thus, the updated idea of ​​the essence of the test pyramid was for me another idea of ​​"arming" from the conference.

6

The first day was closed by a report by Nikolai Alimenkov “Everyday classification of testers from the developer's point of view” , where he witty described archetypal cases of ineffective and destructive behavior of testers in a company: “Nazi tester”, “slug tester”, “dandelion tester” ... Indeed, much more depends on efficiently working people in an organization than on technologies, and such a report will probably open eyes to someone on what’s going on in their team, and managers will be able to diagnose cases of nee ciency behavior of employees.

A party

Parties at conferences are held mainly in order to be able to meet and talk with fellow specialists, without being constrained by the time frame of the break between reports. Therefore, there you need drinks, food and quite quiet comfortable places where you can talk. At huge conferences such as Joker, noisy crowds form, there is a flea market around popular speakers and communication is not easy. In this regard, I prefer less numerous conferences, and Heisenbag turned out to be good in this regard. We had a great conversation with Vladimir Sitnikov on topics related to SQL, and then I moved on to the group formed around Nikolai Alimenkov: as it turned out, there was a full discussion of his closing report, people discussed organizational problems in their companies. Everyone agreed that “fixing” problems with people is a much more difficult task than the most complicated bugs. I also managed to talk with Nikolai and learn something about how best to carry out the transformation of processes in our company, set a goal, build a delivery conveyor. The conversation went far from the topic of testing in the topic of process control, and this is wonderful: you go to the conference in order to communicate with experts on any topics that concern you, not necessarily limited by the name of the conference.

Day 2

1

The second day has come. I came to the first slot sleepy and with my head already swollen from the information received on the first day. I also had to code something and do a pull request. Therefore, although I was in the hall on the report of Artem Eroshenko “Simplicity, trust, control - three pillars of web testing automation,” half of my focus was on my IDE. PageElements is cool and right. Retrofit turns the HTTP API into a Java interface. To the hostess: Yandex htmlelements in his improved incarnation should be taken from the githubnik Artyom eroshenkoam.

Sorry, Artyom, I was not the best listener of your report. Perhaps this report should somehow be revised in the video. It’s okay to miss one of the slots at the conference: it’s impossible to capture everything. Information, like food, requires digestion.

2

In his report “Developer + tester = quality ++” Nikolai Alimenkov continued yesterday's topic of the struggle to improve processes in the organization. This is not the first time Nikolai has been a staunch opponent of the existence of a "testing department", in which, like through a brick wall, developers "drop" a product in order to get it back from the tester after some time. Nikolay talked about the benefits of Agile Feature Teams with the joint work of developers and testers, and the “main ingredients of efficiency” that testers and developers can make during the implementation of the main practices of such a process, and, as a result, about the transition from the ideology of Quality Assurance to the ideology of Quality Assistance .

3

We are long-standing users of Jenkins, so I went without fail to the report by Oleg Nenashev “Building our test framework with Jenkins Pipeline and libraries” . However, as I see from the reports on the mitaps, we are behind in the level of Jenkins usage: Jenkins suggests using Pipeline, but we are still sitting on Freestyle Jobs / Maven Build Jobs. Oleg spoke about the need to transfer all jobs to Pipeline in the conclusions of his report in plain text.

But the examples that are shown in Oleg’s Jenkins reports are, in my opinion, quite complicated, abound with Groovy code that we don’t know, references to APIs that we don’t know, and in general give the impression of hard-to-debug hacks. Although, judging by the depth of questions from the public, there are a lot of people around who use this “to the fullest”. In our simple case, the entry threshold is somewhat scary. But we don’t need so much from Jenkins: for him to collect Maven projects, publish artifacts and at the same time be fully configured through version control code. Where to go - you need to learn all this, you need to deal with Oleg’s examples on the github.

And we also heard how some people, leaving the report, said among themselves: “Everything is clear, we must switch to TeamCity”. There is serious competition between CI systems, and Jenkins has yet to compete for itself ...

4

Pavel Senin in his Selenoid report - hundreds of parallel UI tests easily and quickly demonstrated how to use the Selenoid system to replace an unreliable and poorly scalable Selenium hub with a beautiful solution that launches browsers in containers and issues sessions to tests, builds requests for sessions when there are not enough resources in turn, and, most importantly, it does not require that your existing Selenium tests be rewritten.

Since by the nature of the project work, I well know that the browser is a beast ready to gobble up all resources, and therefore it is better to keep it “in a cage", completely isolating it from other processes, I really liked the idea with Selenoid. This is definitely another takeaway from the conference.. We will try to deploy a Selenoid server as a common resource for UI tests in our company.

In the course of the report, Pavel involuntarily revealed the organizational problems common to many in his team, which did not escape the tweets. The solution to organizational problems is what passed to me on this Heisenbag red line.

5

The report of Andrei Solntsev under the short title “Flaky tests” , in my opinion, turned out to be the best at the conference. Going to him, I thought: “Well, yes, there is such a thing as tests that don’t know why sometimes they don’t work - what can I say?” And really, what can I say? Andrew did not give ready-made simple recipes for ridding this of such situations, although he outlined the main areas of struggle with them.

But he told a number of truly detective stories from his own practice, sometimes lasting months and years. For example, “Curse of the green button”, which was closed for a split second by the pop-up progress bar, because of which the button was not pressed.

Or the best: “Why does Chrome hang?” - a mystery that Andrei could not solve for two whole years, having tried everything that he could: logs, timeouts, debugging ... Once, when he unsuccessfully worked out the most crazy hypotheses, my daughter called play it in cubes. He reluctantly distracted from work, looking regretfully at the monitor. The test was left unattended for twenty minutes. Suddenly, the following appeared on the screen, and the solution came:



Andrei concluded with the same thing Nikolai was talking about: developers should participate in the work of testers so that through “interpenetration” the teams are mutually enriched.

One of the reasons for the “flakiness” of our own tests on one of the projects is that the data in our test base is “polluted” by the results of previous tests. In the discussion area, I wanted to ask Andrey what mechanisms it is better to quickly raise a fresh copy of the database with data before each test. Nearby were guys who were experienced in this field, a discussion ensued, and in just a couple of minutes we worked out a solution that was suitable for us and optimal in our case. And we will implement it in the near future, this is certainly another idea “for arming” . And it’s cool when exactly the best experts in the field help you solve your engineering problem at the conference :-).

6

The conference ended with a report by Alan Page 'Truths about technical testing' . Alan talked about what awaits testers in a transforming world where the separation between the specialties of “developer” and “tester” is erased.

For example, the current stereotype that a “tester” is by all means a “UI test automation tool” must break, especially since the principle of the test pyramid forces us to write as few UI tests as possible. Instead, the tester should integrate as much as possible into the development process. It seemed that Alan directly continued the thoughts of Nikolai and Andrey, who spoke in this room in front of him, and so it was: for example, Nikolai had a slide about “throwing a product through the wall” between the teams of developers and testers, and Alan ridiculously portrayed that throwing a pantomime .

Alan said that when he was a developer, he felt the greatest satisfaction not when he created new functionality, but when he caught a bug, and I completely agree with him - I have the same thing. After his report and the official closing of the conference, Alan went into the empty corridor to the very last discussion area of ​​the conference, and we talked a little on these topics: he turned out to be a very pleasant person to talk to.

Conclusion

In his closing remarks, Andrei Dmitriev asked the public - “how many take-aways do you take away from the conference”? Honestly, by this moment I did not count. But on the other hand, I knew exactly what I was taking away the most-most presumptuous main take-away, I wrote about him on Twitter: