Testing games in Innova: a story about the work of the department

As a preface, I’ll say that I came to Innova a little over a year ago, my task was “to do testing in the company”. My testing department consists of two groups: a web application testing group and a gaming application testing group. This division has developed because these areas have different tasks and different requirements for employees.

Then the story will be about the direction of game testing. This is a story about my guys, about our processes, about our organization of work. I invite you to a verbal tour.



External environment

I'll start with a description of the external system. We have 11 game projects. They are of different sizes, they are developed by different companies (most often Korean), they have different processes, different life cycles, different update rates, different quality and different initial quality requirements.

At first glance, it seems that Innova only deals with localization, and it would seem that what is there to test? They give us a finished product, test only localization. But these different initial requirements for product quality play a role. Various development companies consider it normal to release a product on their (Korean) market with a certain number of known bugs. And this amount is different for everyone, as you know.

We try to minimize this amount in all our games. Because as soon as we release the game in Russia, it becomes "ours." We think so.

Testing Process

However, we do not test the entire game. That would be wrong. In the ideal case, we really should only test localization and assembly. To this we add more rigorous testing of new functionality and beta testing by users.

That is, the update testing plan looks something like this:

- localization testing:
- - lists for verification, deadlines

- testing new functionality:
- - check lists for verification, priorities, deadlines

- assembly testing:
- - smoke tests, deadlines

- beta testing:
- - task for players, deadlines.

I talked about the full cycle of testing a localized game using the Atlantic as an example.

The process of interaction with developers depends on the project and on the development company. Bug reports can be executed in BTS, on our or their side, can be collected in Excel or Google-docs. Most often, testers interact with them to fix the developers, but in some places all communication passes through the project manager. We adapt to the project, but make changes to the processes to optimize them.

Test environment

In addition to battle servers (which different games have from one to thirteen), there is a Public Test Server (TCP) system and an internal test server (QA-stand). Any player can enter the TCP if desired. All beta tests are conducted there. In order to get there, you don’t need any special difficulties, you don’t even need a separate account, the one that you play on battle servers is suitable.

At the QA stand, internal testing is carried out before giving it to the players on the TCP.

Not every game update goes all the way QA-stand -> TCP -> battle servers. Some updates cannot be put on the TCP without hitting the combat system. These immediately go into battle after internal testing. Some, on the contrary, cannot be properly tested at the internal stand, and they immediately go outside to our beta testers.

The quality of the combat product

And yet, it is necessary to produce products with known bugs. And - it happens that with critical. For example, the last major update of the legendary Lineage II High Five Part 3 project, installed on the server on December 28, brought a serious bug: the game client crashed randomly with users with a critical error when teleporting.

This may depend on the fact that, for example, this bug occurs when the load on the servers. And then it pops up only in combat operation. We do not recreate the load in our testing cycle, because the development company does it. But, not always effective. Since updates are often launched at the same time as Korea and often earlier than Europe, we run the risk of launching with unknown bugs.

This may depend, for example, on the planned update launch date, which all players are waiting for.

This may depend on the uncertain timing of the solution to the problem from the developers.

I’m sure that if a poll was conducted among the players who suffered from a teleport bug in High Five, if they agreed to play without an update to this day, they would still choose New Year’s update.

Of course, known errors are reported to the players. And the status of their decisions is constantly updated. And, of course, they are unhappy with the answer "Sent to a developer company, they are working on it!"

Team organization

Almost every project has a test manager. This is one of my guys who is the best versed in this project, who plans to test the project, participates in the planning of installing updates. He knows the weaknesses in his project, and knows what is most important for his audience.

Its tasks include:

- planning of testing the project (or its updates)
- organization of testing the project (or its updates)
- - internal testing
- - external testing (using beta testers)
- informing all interested parties about the testing status and product status

Test -manager interacts on work:

- with the project manager (together they plan updates, approve the launch)
- project engineer (information about installing patches, fixing build errors)
- developers (reports on found bugs, test results)
- fellow game testers (sets tasks, collects results)
- project editor (message about localization errors, fixing them)
- community project manager (known errors, work with beta testers)
- user support team (sets tasks, collects results, collects information about errors found by users,)

“There are plants, some managers in the country”, You say :) Of course, the test manager is not a position, but a role. My employee enters this role when there is testing activity on his project. The rest of the time, he is a tester. He is the "hands of the other head." One person will never cope on time with testing a big update to a big game. Therefore, when he needs, the whole team is at his service. At the time of the activity of his project - he is the main one, he sets tasks and collects the result.





In addition, if a massive attack is required (for example, checking the client build on all project servers), then a user support team comes to the rescue: they know the content of the game and are ready to help. Here the tester again acts as the task director.

Team work planning for the day takes place at morning stand-ups. The main issues that the team discusses are:

- “I will do this today”
- “I need 4 people today to test this”
- “I am not loaded today, who needs help?”

Owner approach

We have the concept of “driver tasks". It means “to be its owner”, to be most interested in its decision. It is clear that the tester can not do much on his own:

- he cannot directly affect the speedy fixing of the bug by the developers
- he cannot directly affect the acceleration of the installation
- he cannot control the engineer
, etc.

But he is the driver of his tasks. First of all, he needs to receive a patch with a fix as soon as possible. First of all, he needs release notes. He needs to install the update on a test server on Friday so that beta testers can help us over the weekend. And he seeks to resolve these issues from the project manager, from the engineer, from me, in the end.

A person who, in case of difficulty, simply writes a letter, folds his hands and waits, does not suit us.

My testers are their project managers. Projects for testing projects. Process owners.

From the information that I gathered from the guys who come for interviews as a game tester, this approach to work, to put it mildly, is very rare in companies that localize games. Most often, testing departments consist of executives who are set clear objectives and require clear results. All decisions are made by the head, all plans are written and approved by the head.

I think it’s wrong to do it for everyone myself, so I teach my guys to do it. This develops them, and it gives me freedom as a leader to engage in more strategic tasks.

Side-effect

True, this approach also has a side-effect. The guys see all sides of the project, communicate with all project participants, with almost all departments in the company. They are seen, they are being watched. And they are growing.

Some grow in other departments. Two of my game testers went to junior project managers. At the same time, one of them was on the project on which he worked, and the head of the neighboring project grabbed the second. Right now, he (my former tester) in Seoul is establishing communications with the development company. Another goes to the department of analytics of game products, just now I am looking for a replacement for him. I rejoice for them. So I work well.

And no less I rejoice for those for whom testing is the industry where they want to develop and bring benefits. They grow as testers, as test managers, and are not going to hobble skiing anywhere.

Company level

Not all projects are tested by the testing department. In Innova, our project manager is fully and entirely responsible for his project. Including for its quality, and for its testing. As the head of the service department, I am the owner of my business, and in order to remain competitive, I must provide the project with a testing service with such quality that the managers order it from me. That is, if suddenly my testers start messing up, then the project manager has the right to refuse the services of my department and arrange testing for himself. By any methods.

Or he may not order it for other reasons. For example, in several small projects, testing is carried out by the project team itself. Because the update testing cycle is very small (a few hours), there is not much content, and it’s easier to allocate time inside the team “Right now everyone has sat down and ran” than to put a task in my department.

In another project, the head does not give testing to us, because this activity is interesting to his employees, and they cope with it, along the way, studying new content, the knowledge of which they need to work.

In such projects, we act as carriers of expertise: we can suggest, help write tests, suggest how to properly arrange artifacts.

It’s difficult, but interesting. We are scolded, and users say thanks to us. We fight the system and are friends with the developers.

Thank you guys!