Quality Guaranteed: What the Game Tester's Work Actually Looks Like (Part 2)

The first part is here .

Westwood College's old video ad has become a bit of a joke in the video game world. Two guys, comfortably sitting on the couch, kill controllers in the trash, enthusiastically playing on the Sony PlayStation. A girl comes in and says: “Hey guys, have you finished testing this game? I have one more here. ” “We just passed the third level, the schedule needs to be pulled up a bit,” one of the guys replies. Then, turning to his friend, he smiles as if he had just won the lottery: "I can’t believe that we play games, and we are still paid for it." “I know,” the second answers him. “And my mother said that my passion for video games will not lead to anything good.”

That is how for a long time people imagined the life of those who are testing computer games - not as work 5-9 hours a day, but as the dream of all teenagers. Who wouldn’t want to sit on a comfortable sofa and play games all day with short breaks to “pull up” the graphics in the third level?

Lunch disaster

Rob Hodgson, a video game tester, once tested the early build of the Fallen Earth multiplayer game at lunchtime when game servers suddenly crashed. Soon after, they failed again. The puzzled employee tried to reproduce the bug, but his attempts were unsuccessful - neither he himself nor any other member of the team could understand what causes this critical error.

“It all seemed very strange,” Rob told me. - We asked one programmer to check the logs during server freezes, but after some time he only shrugged, being unable to determine the problem. It all looked as if someone had reached the borders of the game world, and our servers simply choked and died in an attempt to track the player. However, none of the QA team tested the limits of the world, and the designers left for lunch! ”

After some thought, Hodgson and other team members began to question other employees, and eventually identified the source of the problem, laying the servers on both blades. “It turns out that the lunch break of the designers was most directly related to this problem. - says Hodgson. - The fact is that one of the guys working in the design department put his horse on the "autopilot" and left to have a bite. Sometimes she just found a tree on her way and stopped. But there were days when she went into the vast expanses of the outside of the game world, as a result of which the servers were simply hard-boiled. ”

***

Starting from the game Assassin's Creed Unity and ending with the version of Batman: Arkham Knight for PC - it seems that modern games come with a lot more problems than before. Of course, it’s easy to put the blame on the shoulders of the testing department. And sometimes this is confirmed - in particular, several testers with extensive experience in the QA sector immediately told me that they had the opportunity to work with indifferent employees who are careless about their duties and perform them completely differently. Two “seasoned” testers reported that they had seen their colleagues smoking weed at the lunch break almost every day, one of my interlocutors noted that some even wore dark glasses in unsuccessful attempts to hide their hobbies (apparently, so that they were not visible dilated pupils - approx.

At the same time, many testers say that they manage to find the vast majority (if not all) of the bugs that modern games are crammed with. The problem, in their opinion, is that no one is correcting errors.

Most testers work using a priority setting system, where each bug is prioritized for criticality and the importance of removing it. Bugs with the highest priorities include errors that cause a critical crash in the game (crash or freeze). All other glitches are assigned to different categories in accordance with the level of importance - it all depends on the position of the tester. As a rule, producers and programmers take the time to eliminate critical errors, because with them it will be difficult for the game to even pass standard certification. But minor errors and bugs of "moderate severity" are tied to the game, like a bath sheet, being victims of tight deadlines for the project and programmers who can do exactly as much as they allotted time for it - and no more.

“We usually come to the conclusion that the risk associated with fixing a bug, as well as the time taken to solve the problem, is not worth it - especially when we find a mistake by doing something very unusual in the game world or intentionally“ breaking ”the game "Says the tester who worked on the popular role-playing action movie The Elder Scrolls V: Skyrim. The latter is widely known to gamers, being a project with unprecedented scope and at the same time an extremely buggy game.

"Part of the bugs is marked as" will not be fixed "and" fix after release. " The latter means: “It would be good to eliminate it, but right now you can do without it. Now, if people write about this bug on forums and other online resources, then maybe we will change its status. ” I think that now in the gaming industry we are working with a much larger margin in time than, say, 20 years ago. Because now the developer / publisher simply can say: “Well, we took into account this annoying bug, but it can be eliminated with the help of day-one-patch (the so-called“ first day patch ”, which is released on the day of the official release of the game - comment of the translator). ”



Games are becoming larger and larger, developing both in terms of graphics, and in terms of gameplay and game mechanics. As a result of this evolution, more and more bugs are generated - including those that are very difficult to catch or fix. Game development studios cannot change the game’s release schedule simply because the product is full of glitches. The release date of the product, dedicated to some holiday, will not change, no matter what the raw game is - unless the publisher himself agrees to postpone the release date, and it will cost him dearly. Therefore, as several testers told me, during their work, they focus on “catching” precisely those bugs that will not allow them to undergo mandatory certification for release on the console. Usually this behavior implies ignoring some of the others,

According to one specialist who tested the product for compliance with certain requirements for one major publisher, developers and publishers have independent QA teams. “We found a bug and sent maximum information to developers, which allowed us to reproduce it. The development team determined whether this was a bug, and then either canceled it or tried to fix it. We received an updated build with the indicated number of fixed bugs, then carefully checked whether the problem really was resolved, and then went on to the next bug. From the player’s point of view, such an algorithm was a complete disappointment. The fact is that bugs that spoiled the game to the user were usually marked as “wrong”, because they did not affect the release of the game.

Other testers say that because of hard deadlines, they had to perform only the necessary minimum actions and look for only those gaps with which the game could be certified, after which it would be possible to start printing discs and promise the release of the “first day patch”.

“If we were running out of time, and in multiplayer there were still a lot of glitches, then we focused on testing the single-player game mode and other offline components of the product,” one of the testers wrote to me by e-mail. - “Console manufacturers can easily reject a game if it has serious problems with multiplayer, but at the same time they allow us to send the game to print, provided that the company provides an output of the“ first day patch ”to fix problems A, B and / or C. This policy is used because neither the developer nor the console manufacturer is able to guarantee the expected “patch” that eliminates problems in the offline components of the game, because such modes do not require an Internet connection (a gamer can play one of night mode, without connecting to the network and the mother of the developers, Unaware of the release of the patch - approx. translator). Therefore, we worked on eliminating those bugs that were related to offline content, polishing only this component of the game for successfully passing the certification procedure and realizing that there are errors in the multiplayer. And by the time the game was released, we tried to prepare a patch that should fix the known bugs. Unfortunately, this is a common practice, which was often applied in conditions of unrealistic deadlines for the completion of the project and the resulting too large workload. Usually, precisely because of this behavior (and not the low qualification of the development team and testers), a lot of bugs remained in the game. ” polishing only this component of the game for successfully passing the certification procedure and realizing that there are errors in the multiplayer. And by the time the game was released, we tried to prepare a patch that should fix the known bugs. Unfortunately, this is a common practice, which was often applied in conditions of unrealistic deadlines for the completion of the project and the resulting too large workload. Usually, precisely because of this behavior (and not the low qualification of the development team and testers), a lot of bugs remained in the game. ” polishing only this component of the game for successfully passing the certification procedure and realizing that there are errors in the multiplayer. And by the time the game was released, we tried to prepare a patch that should fix the known bugs. Unfortunately, this is a common practice, which was often applied in conditions of unrealistic deadlines for the completion of the project and the resulting too large workload. Usually, precisely because of this behavior (and not the low qualification of the development team and testers), a lot of bugs remained in the game. ” which was often used in conditions of unrealistic deadlines for the completion of the project and resulting in too much workload. Usually, precisely because of this behavior (and not the low qualification of the development team and testers), a lot of bugs remained in the game. ” which was often used in conditions of unrealistic deadlines for the completion of the project and resulting in too much workload. Usually, precisely because of this behavior (and not the low qualification of the development team and testers), a lot of bugs remained in the game. ”

Sometimes bugs are intentionally left in the game for one reason or another, incomprehensible to people from the outside. According to one tester who worked for the American division of the legendary game company Nintendo, even before the international release of the game, they managed to catch a well-known bug related to Sky Drop in the game Pokémon Black and White, however, by this time the title had already been released in Japan, so the company Instead of fixing the glitch, I decided to leave it in the game to maintain parity. At the same time, the tester notes that, in addition to a strange policy, Nintendo has created an internal standard for careful recording of each bug, according to which testers performing a submission are asked to attach a corresponding video that clearly shows the “nature” of the error found: “It seems to me

The explanation of another specialist who checks the product for compliance with certain requirements regarding the increasing complexity of testing will certainly anger gamers even more than publisher policies. “It's all about the DLC,” the tester says.

“Imagine that we have a game with 40 DLC (elements of downloadable content). Even if we imagine that some or even all of these 40 packages simply open hidden content originally placed on the disk, we still need to test every possible combination of these 40 parts:

• in all game modes;
• using various physical media to store information (say, 20 DLCs are located on the hard drive, 15 on the USB drive, 5 on the memory cards, etc.);
• comparing the packages installed by the owner of the game with other DLCs activated by his rivals in multiplayer;
• checking the proper behavior of the game in the event that some jerk has all the DLCs stored on an external drive, and he suddenly decides to disconnect this device;
• with each update of the game (1.01, 1.02, etc.);
• with various save files;
• etc. etc.

As a result, this can lead (and leads) to the moment when the task of testing all possible combinations and scenarios becomes physically impossible for a person. The complexity here increases exponentially with the addition of new variables. ”

As a result, the current situation with testing can be described in a nutshell: everything is complicated. It is especially difficult for testers to work with multiplayer games: even hundreds of testers sometimes will not be able to reproduce such bugs that hundreds of thousands of people can tirelessly scorch aliens on game servers. And even though some testers say that now developers are working more closely and fruitfully with the QA-team than in the past, no one can change unrealistic deadlines.

“The deadlines for completing projects now set are absurd,” says one former tester. - Apparently, most people simply do not understand that even for a game that has been in development for, say, 3 years, only half of this period can be called full-time production. Only 9 months have the game been tested, of which only 3 months are tested in conditions of maximum load of the QA-team. At this time, they may completely block access to the game, and by the time we can provide information about the product, the game is already in beta. Here I exaggerate a little, if we talk about large well-known titles, but for most of the projects that I worked with, this is a very real situation. When the QA department is finally allowed to speak, it turns out

Farewell to no offense

One former tester says:

“On one of the projects, I happened to work in a room with many other testers - a kind of big“ paddock ”. Most of them did their work on the basis of a temporary agreement. I sat next to people who had been testing one game for years and were damn exhausted by this work, but continued to work because they had no other choice, they were still hoping. As far as I remember, only two or three employees from hundreds of testers got a permanent job. We regularly received condescending letters of this kind: “Congratulations! We just earned 8 hours of vacation! ” It felt like we were little children, and teachers draw a star on our palm for diligent behavior. Personally, I did not need it at all. Better put your praise on the payroll.

Meanwhile, in the corporate parking lot, every couple of months, new, vibrant cars appeared, as if coming down from the covers of expensive magazines. The cost of each of these cars was probably higher than the total amount that four or five testers from our team earned over the year. The company kept throwing parties for its regular employees. When you work hard and in spite of this you cannot afford new clothes, and such things happen nearby, it strongly pressures and depresses. As soon as the game was released, a wave of layoffs swept through the testing department. I think about 100 people lost their jobs, but I do not know the exact numbers. I only know that the QA department was almost erased from the structure of the company. A few hours after the security took us out of the company’s territory,

At that time, a lot of bad things happened in my life. But I also met many good people who knew how hard it was for us temporary workers - for this reason, good people left the company themselves. The company policy disappointed everyone with whom I had a chance to work there. I was transferred between departments, as a result, before leaving, I got into a small team with a large load. But I learned a lot and met really good people. When I told my colleagues about leaving, no one was offended by me. On the contrary, the team members almost openly approved my move, as if I told them that I was stopping drinking or something like that. ”

Is it possible to build a successful long-term career in QA? Or will this area forever remain a means to achieve something more, a kind of transit point for moving to more interesting sectors of game development?

During the preparation of this article, I spoke with approximately 60 testers, and only four of them said they were considering a long-term career in QA. Many used this area to “jump” to the position of producer or designer in gaming companies, while others worked as testers for several months or years, after which they gave up and transferred to other, more profitable work areas. Those who feel they have the strength to stay and succeed in testing can try themselves as a manager in the QA department - that specialization, where the ability to manage other people is much more important than helping to create video games. For some, however, this does not bring the expected satisfaction, especially given the fact that the stereotype of young, immature,

“Immediately after you start working as a manager, managing a team, you get such experience that no one can be prepared in advance,” the experienced tester says. - Testers come to work in pajamas, arrange sexual harassment to each other, steal things ... Even such an elementary thing as ordering food can be a real test for the manager. QA is a work area that is clearly biased towards young “specialists”. The team, which consists entirely of 18-19 year olds, is a very common occurrence, and often this is their first serious work. Managing such a team is a real test for a manager who has to deal with young fragile personalities and raging hormones. ”

Several experienced game developers told me about employees who are valued higher than the head of the QA department. These are those who know how to correctly write a report on a caught bug, how to concentrate on working with critical errors instead of finding small bugs, how to work with other people to achieve maximum productivity and avoid craps. Such people are rarely found in the field of QA - the combination of low salaries and poor attitude towards employees discourages many testers from working in this area long before they get the proper experience and learn useful things.

Another former tester by the name of James, who asked not to write his name, said the following: “Reflecting on my departure, I asked my manager, assistant producer of the game project, to give me a couple of minutes of my time to ask for advice. I wanted to find out how you can move from a temporary tester position to a permanent position in the field of QA - for example, in development or another sector. He honestly advised me to distance myself from testing in general: "The faster you can leave this area, the better for you." After eight months, I still quit QA, moreover, I completely left the gaming industry. Now I am a commercial software developer. I admit, now my life has become less fun, but stability has increased, and the salary has increased significantly. ”

Based on real events.

The film “ The Boy for Three ” (Grandma's Boys), which was released in 2006 and the image of actor Alan Covert (Allan Covert), who played the tester, became a bit of a joke in the QA world. People constantly ask real testers: which of what is shown in the film is true? Usually the answer is “little.”

One of the episodes of the film, in which the narcissistic game designer JP looks through the early version of the game and sees how the main character's head falls off, is often attributed to moments indicating a mismatch of reality (scroll to 1:31):



Stupid, right? Bugs do not really appear like this. Except in special, isolated cases ...

“Once I had a chance to work on one game, sharpened by the Kinect touch controller, which included various obstacle competitions taught as a TV show,” says a tester who worked with a major publisher. - Before the start of each round, short introductory videos with the announcement of participants and introductory comments were played. One, and only once, the commentator’s head suddenly moved from the usual position on the neck to an empty place near the character’s body, and then simply fell down, disappearing off the screen. No one subsequently could reproduce this bug, and we tried, believe me.

It looked surreal, especially after I already got an aversion to that famous film because of all the inaccuracies and simulated simplification (if you want, vulgarization) of my work ("So this guy just fell off his head? DOESN'T HAPPEN such crap in real games! ”). And a similar situation happened in a real video game that I worked on! I will never forget this moment. "

***

The fantasies of gamers from Westwood College ads are far from real game testing. In fact, we are dealing with difficult work and low salaries. As a result, many testers find themselves in an unenviable position, outsourcing or contracting for a game developer who does not even know who does the testing for them. One QA specialist wrote to me that one day he received an email from the HR manager of the company where he worked: “Employees were invited to write their thoughts and wishes about the corporate festive evening. When I replied to the letter, suggesting a couple of my ideas, it turned out that the employees working under the temporary agreement were not even invited to the party. ”

Even those who find testing a worthwhile job that brings satisfaction note that the situation could have been better — much better. Large-scale changes in the industry could improve the working atmosphere and increase the productivity of the work of testers who would fruitfully work in conjunction with game developers to eliminate the maximum number of bugs.

“Testing video games is the work of unsung and unskilled heroes,” an analytical article reported.from the very famous company ST Labs. - For some companies, a typical testing strategy is to give the game to a large number of testers on the eve of the release and hope that specialists will find serious problems. The desire to understand the game specifics and modern technologies, building a systematic, rigorous testing methodology - all this is still perceived by most development companies as a completely new approach. ”

This article was written by Dury Price (Duri Price) and Ilya Pearlman (Ilya Pearlman) back in 1997. After 18 years and 4 generations of game consoles, you can confirm that nothing has changed. Good game testers are still unsung and unskilled heroes who do everything in their power to eliminate all-all the bugs in the game. Despite the absurd deadlines and their colleagues completely devoid of motivation, they do their work, which they always underestimate - at best.

However, there is another view of this situation.

Here is how one tester with great experience voiced it: “You will deal with shitty games, with shitty people and a shitty producer (or project manager) who thinks that“ all these fucking testers ”are not good for him to work . You will work for many hours, you will be underestimated, and you will notice every now and again that the very unusual bug with which you spent almost the whole working day has been canceled with the note “It is known. Not critical for release. " But at the same time, you will see your name in the final credits of the game, you will understand that you have contributed to a really cool project ... and you will get to work on another cool game. ”

Do you work as a tester? Do you agree with the opinion of the author of the original article? Please tell us about your work, its pros and cons - the way you see it.

PS I apologize to the Megamind community for such a protracted translation of the second part. I sincerely hope you enjoyed the fiction.