From tanks to nuclear power plants: looking back at Heisenbug 2017 Moscow

While after the Heisenbug conference we were collecting and analyzing viewers feedback, a detailed IvanPonomarev post appeared on Habré with his impressions of the viewer. And in order not to repeat it, but to supplement it, we decided to build our text about the conference differently. Do not describe all two days with the party and do not talk about the reports that Ivan has already described, but look at the audience’s ratings, which of the others are most liked by the audience (all of these ratings were above 4.2), and tell a little about them. The result was a list of “six things that attracted Heisenbug viewers” ​​- it can be used to evaluate specific topics and their scatter.

Tanks

In the summer at St. Petersburg Heisenbug there was a report on load testing from Alexei Lavrenyuk , and Yandex.Tank was mentioned there, to which Alexey has a direct relationship. This time at the conference, there were also tanks, but completely different ones: Alexander Shukov from Wargaming talked about how to deal with autotests in the case of World of Tanks . While microservices are raging around somewhere, games remain "monolithic" - so a game of such proportions is also a kind of "tank", which we still have to be able to handle, and for this we created our own framework in Wargaming.

For whom was the report - for those who work directly in gamedev (“learn from experience”), or, conversely, for everyone else (“while you indulge in plushies, this is what’s going on with us”)? Perhaps for those, and for others. It was not required to know the game specifics in advance - on the contrary, the report just helped to understand the main points. For example, it’s easy not to think from the outside that the difference between “boxed” and MMO affects the testing of games: it’s one thing to test a large project that “has been preparing, releasing and exhaling loudly for two years,” and another thing that is constantly developing. And not only people from game dev could benefit for themselves. In the end, the monoliths not only remained in the games.

Was it possible on the sidelines of the conference to try out from Alexander the way to cheat in World of Tanks? On the contrary, it was possible to get an explanation from him why this game is better protected from cheating than many others: almost everything is processed on the server, and not on the client, so no matter how you try to fake with the installed application, you can do little.

Smartphones: a pair of paired reports

iOS and Android divided the mobile world into two parts. It's funny that the same number popped up on Heisenbug in connection with the mobile theme: exactly two reports with the #mobile tag got into the audience’s top 5 reports, and each of them was presented precisely by two speakers.



The first is “Testing geolocation in Badoo: bumps, stones, crutches and a selfie stick” by Alexander Khozi and Nikolai Kozlov . For dating with the “users nearby” function, geolocation is obviously important, and the speaker’s thorough approach to the topic was illustrated by the fact that instead of “GPS” they said “GNSS” (“global navigation satellite system”): GPS is the most popular, but not the only satellite navigation system.

Tricks with testing geolocation begin with the fact that it involves moving in space. If other functions of the application can be directly called at your workplace, then how to check the movement without leaving the office? As it turned out, the separation between Android and iOS here just affects. Both systems can be “tightened up” in official emulators, but if you can just set specific coordinates for Android (the default is the Icelandic village of Dalvik, which gave the name to the Android virtual machine), then iOS also has modes that simulate moving. But if standard tools are not enough, third-party helpers come to the rescue, which were also discussed. And later, energy consumption was also affected - whoever finds himself in an unfamiliar place with a discharged phone, because he often looked at the map, he understands



Just about energy consumption was the second favorite "mobile" report from Alexei Lavrenyuk and Timur Torubarov from Yandex. Moreover, they even started with the words that they listened to the performance of Alexander and Nikolai. But then they went almost into the format of "Crazy hands." How to measure the energy consumption of an application without wasting time on discharging a smartphone to zero? Theoretically, there are software ways to see this, but they didn’t work for a number of reasons - for example, due to the insufficient frequency of measurements (in addition, there the difference between Android and iOS also made itself felt). Among third-party hardware projects like Batt0r, right now, too, was not found suitable.

As a result, Yandex decided to act “hard” and “hardcore” themselves, opened the circuit, made their own Arduino-based circuit for measuring current, and wrote their library to work with it — and we talked about this in detail at the conference. In general, although Alexei went far from his spring report on stress testing, there was also something to boot up with.

A / B tests

What could be simpler than conducting an A / B test? I launched two different options for a different audience, compared, left in the production proved to be better.

But it was not there. Roman Poborchiy loudly stated in the title of the report “Your A / B tests are broken”, and this provocative statement was supported by arguments. Here, for example, is one of the first. Typically, when comparing, the results are considered to be different with a probability of 95%, and this number looks solid. But if, instead of a single test, we conduct a whole series (“now we will choose the best of dozens of options”), then the probability that the result was at least one of them by random fluctuation increases sharply - which means that it’s very likely that you will take for “the most users liked "option is one that actually shot by accident.

It is curious that in the audience’s reviews Roman was praised “for statistics with mathematics” and “for the fact that there were few statistics with mathematics”. This paradox has an explanation: the report spoke about the importance of the statistical approach in A / B tests, but at the same time it was carefully structured so that an unprepared listener would not be buried under a pile of formulas and would not cease to understand anything.

White box

Nikita Makarov (Odnoklassniki) has been participating in the Heisenbug program committee since the very first conference. It is not surprising that he understands what kind of reports the audience is waiting for, and his own was highly appreciated.

And Nikita is sure that in our time testers need to understand the code. It is not surprising that his report was devoted to “white boxes”, and he immediately explained what “blacks” limit.

Add a much less predictable: a curious historical fact from his report. The Chaos Monkey tool is now well known., which randomly disables various elements of a large system, letting you know how the system will respond to such a failure. And Netflix, the creators of Chaos Monkey, are now considered the main apologists for this approach. But it is much less well known that such an idea was first implemented back in the 70s, and not at the software level, but at the hardware level. With the development of circuitry, when the circuits became complex and the consequences of the failure of any element were not obvious, to test them they began to be physically excluded from the circuit, thereby giving rise to fault injection .

Error messages

At Antonina Hisametdinovoy of the company "Pavlov's Dog" was a report on the subject, adjacent to the test. Finding errors and getting rid of them is good, but all the same, situations inevitably arise when the end user encounters them. And this means that the matter is not limited to search: you need to understand what exactly the user in this case should see. And with this now everything is sadder than we would like: many act on the principle of "what is there to sort out with error messages, we need to cut features."

For example, what should I do if the entire service has a malfunction? It’s clear that “repair as soon as possible” - but is it possible to tell users something better than “service not available”? Yes: estimates of the time to restore its operability can also help, and instructions on alternative ways to get what you want, and messages to users even before they bury themselves in the problem with their nose.

This is particularly interesting: among other things, Antonina talked about the trust of users in infrastructure technologies such as cash registers and ATMs, and that a mistake could undermine this trust. And after the conference there was a massive failure of cash registers, becoming a clear illustration of such an error.

Nuclear power plants

Finally, there was a completely unexpected report. Vyacheslav Alenkov represented the ASE group of companies belonging to Rosatom, and nuclear energy is not the first thing that comes to mind when the words “testing conference” are presented. And from the outside it would be possible to assume that the construction of a nuclear power plant is a conservative area, where all the main principles were laid down many years ago, innovations come leisurely, and the waterfall model reigns unshakable. Is total digitization of everything and agile necessary when it’s not about a fashionable mobile photo service, but about a huge, complex and very responsible physical object?

It turned out that we needed. Without flexibility, today is simply nowhere - with a “waterfall” you will simply find yourself uncompetitive in the global market (“ASE” works in a number of countries). And the transition to digital projects occurred just for reasons related to testing. A mistake is much more expensive when it is already embodied physically in the form of an incorrectly laid pipe - and the more they can be caught before implementation (for example, in a 3D model), the better. In general, everything is closer to the life of typical Heisenbug participants than you might think.

Bugs

And finally, after the reports, we will share a couple of funny “bugs” of the past Heisenbug.



Firstly, the badges were not designed for very long inscriptions, and, for example, speaker Roman Poborchiy turned out to be an "independent consultant on." In general, of course, when organizing a testing conference, you should be prepared for the fact that participants can enter unnaturally long values ​​in any field!

Secondly, a completely unexpected problem arose. In one of the halls, the image on the projector sometimes disappeared, and it was not immediately possible to understand that when the speakers walked on the carpets of this hall, statics accumulated, making itself felt when you touch the laptop. That's really a tricky bug - try to reproduce one until you understand what’s the matter.

But we carefully fix all the problems that arise, so that each next conference is better. So by the next Heisenbug (St. Petersburg), these bugs should be fixed. See you there!