How game statistics differ from playtest data

Now, while working in a research company, I sometimes meet developers who confidently say that they do not need any audience research, since they have statistics. They have data from hundreds of thousands or even millions of players behind their backs and this gives them confidence. But is everything so simple? Quite ambiguous, because the statistics show what the players are doing, but does not answer the questions “why”.

In 2013, at my largest World of Battles (MMO RTS) project, we received statistics about each significant player’s action (there were hundreds of thousands), application and server behavior. Some statistics were available “live”, and some appeared ex post the next day in the process of automatic data processing on the server.

The project was large and I, like other responsible employees, collected and analyzed this data. These statistics showed us many of the strengths and weaknesses of the game. In the process of receiving answers based on statistics, we had a lot of new questions. In this article I will write about what I lacked to analyze the behavior of users of the game while working on World of Battles.

When does the statistics appear?

Game statistics will appear immediately after you have a working client and, if necessary, a server. At first it will be just logs for debugging, but later it will be necessary to think over and set up a statistics system that will allow you to keep your finger on the pulse of your application. After all, the program itself will not tell you what is wrong with it, as well as its users.


This figure shows what is known about the game before it began to study and analyze data from an array of users

We begin to collect game statistics, what happens?

Performance statistics

• Server statistics - data on server operation, for example, status or system load
• Client statistics - data on client performance, for example, performance, application crashes

Community statistics

The picture of community activity - summarized data on the use of chats, a forum, a cut of posts and activity on social networks, information about broadcasts and fan resources

User statistics

User metrics - a strategic view of user activity. These are DAU, MAU, ARP (P) U, CCU, Retention Rate, Average Session time and many other indicators

Gameplay Statistics

• Players behavior - data on the number of games played (or levels), visits to various screens, reached levels, and so on
• Economics - data on the in-game economy, both on the economy of Soft and Hard- currency
• Balance - the balance of the game has many faces and statistics can cover only some of them, for example, the balance of success in games, the balance of the battle, the financial balance, the balance of the speed of progress

What do the processed data allow?

• Learn basic product metrics
• Learn about its weakest points
• Construct optimization hypotheses based on data

This figure shows what you will know about the game after reviewing and analyzing the data.

For example, while working on the World of Battles, thanks to statistics, we learned that certain things are not bought by players, but we did not know the exact reasons - either this price, either the power of things, or their appearance. And without conducting playtests, we would have wondered over the reasons for the low purchase of certain things.

Now let's try to look at the product through the eyes of users, how to do it?

We find representatives of the target audience, and let you play your game. We record the entire process on video, as well as collect feedback from players during and at the end of testing.

What are the main tools for playtests?

Video recording - this tool allows you to visually see the activity of a player in your game. The video of the first game session is very important, it will show how newcomers behave - most of your users.



At the World of Battles project, a video game showed us that our first interface could only be understood by developers and hardcore players from the 90s and that players didn’t open some important screens at all.

Questioning and interviews - these tools allow you to collect quantitative results and answers to the most exciting questions for developers, such as estimates of complexity, graphics, music, balance, etc.
They also provide answers to the questions “Why?”. For example, why don't players pay? Why stop playing? Will they advise the game to their friends and what will they say about it?



After a survey of World of Battles respondents, we learned that the speed of battles in the game is slow, and the speed of movement of units is generally worthless. Players were bored of waiting for one squad to kill another in battle, and especially while they were moving around the map. We are used to the game, we didn’t notice it at all, and no statistics told us about this problem.

Focus group - this tool allows you to deeply and thoroughly work out any questions of interest to the developer, ranging from the attitude to existing mechanics and ending with plans for future updates. With a good organization of the process, even subliminal information can be obtained from respondents.



During the discussion of the World of Battles fight at the focus group, we learned why players choose particular elite units, how they see giants, and which cards and why they seem more interesting than others.

There is another very important advantage of playtests over statistics, and it consists in the fact that gameplay statistics are available only from your own projects, or from publisher’s projects. A playtest can be conducted on any existing product, and as a result, you can know which of the mechanics and elements of a successful game are more popular and which are less, whether the balance is good and so on.

Total

The bottom line is that statistics allow you to see the problems, and play tests to understand their causes and subsequently correctly corrected. Those. only using a combination of statistics and playtests can you see a complete and clear picture of what is happening.