[Translation] Heatmaps: We customize gameplay with a simple system for collecting and analyzing game indicators
In this article , taken from the September 2010 magazine Game Developer's Magazine, Chris Pruett, working in Google consultant for the development of games, he tells how he quickly and easily implement a convenient system of assessment of the gameplay of your game on the Android platform, under the name Replica Island .
Nothing beats the feeling you experience while watching someone play the game you created. In the process of developing a game, you play every day and develop, perhaps unconsciously, a certain style of play. But giving your game into the hands of another person, you get the opportunity to see what happens to your creation when it is used without your daily gaming experience.
Every problem that arises, a failure in the animation, incomprehensible text in the manual and periodic errors make a very strong impression when a novice plays the game. It doesn’t matter how long you have been honing the game process and how many mistakes you have corrected, your elaborated style of play and your special involvement in the gameplay that you create the game may prevent you from clearly seeing the problems that other players may immediately encounter.
That's why gameplay testing is a vital part of making a good game. To get the most out of testing, we will need to save information about the process of passing the game. This article describes my experience in solving this problem.
I took my first steps in the gaming industry, creating games for Game Boy Advance. At that time, the testing approach was quite simple: we gave the neighboring guys a special Game Boy Advance device connected to a VCR, recorded their game, and then watched the recordings. This allowed us to catch obvious and major errors.
The sections of the game, the passage of which caused strong negative emotions in the guys, were accepted for additional study. When a player fails over and over at a specific place in the game, this is a clear signal that this part of the gameplay needs to be reworked. Several game runs with the help of the neighboring guys and we could significantly improve our games.
I am currently working as a consultant in creating games for Google Android. My first game for this platform, the Replica Island arcade, is not much different from the games for Game Boy that I wrote 10 years ago. But something has changed - I did not work anymore for the company that created the games, I wrote it for myself, with the help of one artist, mainly working in my free time.
Now I do not have access to the audience of young beta testers that was at that time. And even if this access, the current target audience for the game is somewhat older.
After all, there is no easy way to collect game progress information on the phone. The only way is to stand "above the soul" during the game, but this is inconvenient, and can affect the way the player plays, which reduces the purity of the experiment.
What should an independent developer do? Having completed the development of Replica Island, I realized that there is no guarantee that it will be interesting to play this game. The game was developed “in a vacuum” and I needed a third-party look at the gameplay before I could be sure that it was time to release the game.
The first thing I tried was polls. I placed the game on the internal server of the company where I worked, and sent letters to my colleagues asking them to play and send me their impressions. I even launched a small forum with questions about the game. This approach did not work.
Despite the fact that many downloaded the game, very few, less than one percent of those who downloaded, bothered to answer my questions. In addition, those who answered the questions did not provide enough information. It is difficult to understand what exactly was the reason for the assessment "the game is difficult to play." Whether the reason was inconvenient control in the game, poor design of levels, the location of obstacles in the game, guides to passing levels, or something else.
After that incident, I thought of a game-gathering and processing system developed by Naughty Dog for the Crash Bandicoot game .
The system recorded statistics about the game process on a memory card. This data was then collected (offline) in order to detect areas that were difficult to pass through, or areas where the game character most often died.
These problem areas were reworked, and this information was also used for a system for dynamically adjusting the complexity of the game.
One of the interesting principles that guided me in creating my system was the idea of Naughty Dog that we should avoid ending the game with all our might. Their ultimate goal was to eliminate the moments when the player was stuck at some stage and could not continue the game.
I liked this idea, but then I did not understand how feasible it is for playing on the phone. I asked around a little to understand the current situation with recording statistics for games with large budgets. And I learned that many companies use mechanisms to collect information about player activity.
Several people said that despite the fact that they collect a large amount of information, they experience difficulties in processing and presenting this data in a form convenient for perceiving and understanding what needs to be changed in the games being studied.
On the other hand, some studios have tools that can recreate the player’s movement in terms of level and generate statistics about which weapons the players prefer, which opponents are more difficult to defeat, and which parts of the map are clearly visible in one case or another.
It seems that the formation of a set of indicators for the player is applicable for a wide class of games, but this is justified only for large companies that invest significant resources to build tools for processing the collected data.
An example of how this type of system can be most effectively used can be seen in a presentation by Georg Zoeller about the wonderful telemetry system they use in BioWare.
It turned out that collecting data about the gameplay is an easy part of the task, but interpreting this data in such a way as to give a clear and useful picture to the creators of the game is much more complicated.
It sounded discouraging as I tried to keep my toolbox as simple as possible. But I decided nevertheless to experiment with some key indicators.
My Android phone did not have a memory card, but there was a permanent Internet connection. Perhaps, I thought, I can capture some important events in the game, send them to the server and get results from it. My goal was to understand as much as possible about how they play my game, while leaving the system as simple as possible.
The system I wrote consisted of three parts: a thread of execution, which during the game process collects data on the player’s actions and sends it to the server, the server itself, and a tool for processing the collected data.
Server - this is said too loudly, of course. My server was represented by 30 lines of a PHP script that checked a GET request via HTTP and wrote the results to a MySQL database. The structure of the GET request, in turn, was also very simple. This is the name of the event, coordinates, code version, session identifier and timestamp.
These data, directly in the received form, were entered into the database. The current data processing was also performed in PHP when a special page was opened on the server. Not a very good choice in the long run, this will be discussed below.
I started by registering two events: the death of a game character and the completion of a level. Each time a game character died or a player completed a level, the game sent the corresponding event to the server. From this data, I subsequently was able to form a rather detailed idea of the gameplay.
I was able to see which levels require the most time to pass, at which levels the player most often dies, and which of them passes too quickly. Having grouped these data by players, I saw what percentage of all players die at one level or another, and the average failure rate for each player.
Analyzing the coordinates of this or that event, I could say in which cases the game character died from the actions of the enemy, and in which from falling into the pit. For starters, my simple scorecard was pretty detailed.
As soon as my base system began to work, I released an update for beta testers and began to monitor the incoming data. Quite quickly, recurring situations were identified.
At some levels, almost one hundred percent of the players failed at least once, while at other levels the players got stuck for several hours, which meant an unsuccessfully designed level, since it was supposed to pass within a few minutes. By analyzing this data, I got a clear idea of what levels need improvement.
But defining problem levels was not enough. Sometimes I could not say why a problem arose at a particular level.
And I took a step forward. Using the same data, I wrote a tool to indicate the location of the death of a game character on a level map. And I was able to see exactly where the game character died, and where not.
The first run of the system showed only a small point at the level where the game character died. But with the increase in the number of players, I began to get something like heat maps (heat maps), indicating the places of death of players on level maps, which were much easier to perceive. The chapter on creating a heat map will be presented below.
Heat map generated based on game character death data in Replica Island
The combination of high-level game statistics and the designation at the level of places of death of game characters shed light on the shortcomings in the game design. I realized, for example, that a large number of players died when they met with the very first monster. This was not because the monster was very strong. After a detailed study of the situation, I came to the conclusion that the monster appeared in a place where the main method of attack - jumping from above - was difficult to implement because of the low located ceiling.
I also realized that my simple system for dynamically adjusting the complexity of the game itself required correction. This system, without publicizing this fact, increased the life of the game character and the energy for his flight after a certain number of successive deaths. Studying the statistics, I realized that she should have done this much earlier.
Also, the structure of my levels has undergone major changes. I had a sufficient number of levels with high time, which was spent on their passage, but with a small number of deaths. And I realized that the players simply got lost at the level. I redid these levels to make the order of their passage more understandable. In a couple of cases, I even completely redid the levels from scratch.
But the biggest problem I encountered was pits. Replica Island is a platformer, and, as you can guess, in it the player needs to make a large number of jumps over the pits. Unlike the constantly rotating marsupials and plumbers living in the pipes , my game character had flight as the main way of moving.
I needed a control system for a game character that would not require a gamepad. Since the protagonist of the game, the green Android robot used rocket engines on its feet to fly. The basic model of movement was to get an impulse still on the surface before the jump and, using this impulse, with the help of engines fly in the right direction. Engines consumed energy fairly quickly, but energy was renewed upon landing. And the idea was that the player would make a jump, and then, carefully spending energy, reach the right places, or make precise jumps on opponents, attacking them.
All this was good, but when I looked at the statistics of the death of game characters that came from beta testers, I found that, for the most part, they died in bottomless pits. Crowds of players failed even in the smallest pits. And more worried by the fact that over the course of the game the number of deaths in the pits did not decrease. Players did not improve their skills in overcoming obstacles during the game.
After which I carefully studied the design of the game and the design of the levels and formed several theories. The main problem, in my understanding, was that players do not see holes when they are in a jump. In the end, during the jump, the presence of a simple hole or a trap hole at the landing site was not noticeable. And since my levels are often very high, it is difficult to determine which pits led to the underground levels, and which of them led to a terrible death.
The second and more important problem was that the behavior of the camera showing the main character of the game was not very well worked out. When the main character made a high jump, the surface of the earth disappeared from sight. As a result, it was difficult to determine the right place to land.
Well-known platformers like Super Mario Bros almost never perform vertical scrolling. Mario has a whole set of complex rules that determine situations in which the camera can move up and down. However, in my game, the presence of flight mechanics meant that I allow vertical scrolling in all cases. After a large number of corrections made, I got a smarter camera behavior that does not start moving up, unless the player himself is close to the border of the displayed area.
After all these changes, I released another update for my beta testers and compared the results with the results obtained from the previous version of the game. The changes were very promising. The total number of deaths has decreased, the time for passing the levels has basically returned to normal. And mortality from falling into the pits decreased markedly.
I repeated the testing process several times before I was ready to release the final version of the game. With my scorecard, it was easy to determine how these or other changes affect the progress of the game by beta testers.
So, after several test runs, my graphs began to show the standard normal distribution , which I wanted. It's time to release the game and I decided to leave the scorecard built into the game. I was wondering if the information that I will receive from new users will be different from the information that I received from the beta tester group. There was only one way to find out.
Of course, every time the game sends data to the server, the best solution would be to notify the user.
The first time after the release of the game in the welcome message, which is displayed with information about the latest changes, was information that the game sends anonymous and not personalized information about the game process to the server in order to improve the game subsequently. And players who did not want this could turn off this system through the menu.
This approach seemed like the best solution. Despite the fact that the game code is open, and anyone could study the structure of the sent data packets (I made sure in advance that none of the sent data can be compared with a specific person or device), I gave the players the opportunity to say “no, thanks”.
Comparing the number of installations from the Android Market with the number of unique players in my scorecard, I came to the conclusion that less than 20% of game users refused to transfer data about the game process to the server.
As a natural result, I received a huge amount of data for analysis - more than 14 million information points, about a gigabyte of information about events that were generated by users of my game. At the time of writing, their number was 1,200,000 players.
In fact, this amount of information broke my system pretty quickly. I have statistics collected from the first 13,000 players that I posted on the Replica Island website. But after the release of the game, most of my analysis tools stopped working.
The good news was that the first 13,000 players gave statistical information that was very similar to the smaller beta tester group, which probably suggests that the results obtained in the test group can be applied to large groups players.
I was very satisfied with my event accounting system at Replica Island. With a little effort that cost me almost nothing (the server side, which records events, cost me less than my Xbox Live account), and using only two types of events, I was able to quickly and efficiently determine the places in the game where the players ran into problems.
In addition, as soon as I began to collect this information, I got the opportunity to compare the summary data from my system received from different versions of my game, which gave me the opportunity to make sure that the changes I make to the game have a positive effect.
Using PHP and MySQL to implement the server side was a good solution. Capturing events was so trivial that I'm sure it could just be implemented in any language. In PHP, the implementation of the entire server side took me less than half an hour.
Using a separate thread of execution to send events to the server was also a good move. I did not want the user interface to be blocked while sending HTTP requests, and put the process of communication with the server in a separate thread of execution.
At first, I was afraid of a slowdown in the game on the device because of this, but as it turned out, there was no reason for concern. The additional load was so insignificant that I could not even notice it during the profiling of the game.
After all, keeping the whole system as simple as possible was a good solution. I looked at many different event options that I could record in my game. But tracking the moment of the death of the protagonist of the game and completing the level provided more than enough information for analysis.
Larger amounts of information would complicate the processing of data, and possibly make it more difficult to get a clear picture as a result. Now that I have some experience in creating an automatic system for collecting and processing indicators, I may decide to increase in the future the amount of data that I send to the server. But starting with a simple system was definitely a good move, in my opinion.
Not everything in the event registration system worked well. I made several decisions that did not give the expected results or simply were a waste of time.
The decision to use PHP for the server side was a good move. However, it was a mistake to use PHP to process the received data. My idea was to do everything through a web interface. I even wrote my level editor in PHP and JavaScript.
But PHP began to slow down when the amount of data to process increased significantly. PHP worked in a very limited environment in terms of memory and computing resources. And I almost immediately ran into these limitations. As soon as I began to receive information from 20,000 players, most of my PHP tools simply stopped working.
In particular, image processing in PHP turned out to be problematic. I implemented the whole heatmap generation mechanism in PHP, but I should have written something that would run locally instead of running on the server.
I came across a lot of errors in the PHP GD interface (forming images with an alpha channel simply did not work), and decided to simply reduce the size of the level images for further processing.
For this article, I rewrote this tool using Python and ImageMagick .
And the results were impressive. The code for this implementation can be downloaded from the official website of Game Developer magazine .
As a result, even though these data told me about when the game character died and how much time was required to complete the levels, they did not give me information in order to determine the moments not related to the death of the main character of the game, when the players quit playing my game (game shelf moments).
I came to the conclusion that I released a game with some key miscalculations in level design that my scorecard would never have discovered. In the most extreme situations, the players faced a task that they did not understand how to solve, and simply quit playing, not completing the passage of the level.
This has never been displayed in my system, since the condition for fixing the event of completion of the level did not occur. I only found out about this when the players began to send me complaints that they were stuck in the same place when passing the game.
Although my automatic system was very convenient, it did not show me the full picture of the gameplay. In my case, the indicators were good for identifying problem areas at the levels, but would be ineffective in determining the miscalculations associated with the order of interaction of the elements of the game with each other.
For my next game, I will again use a similar automatic system. In addition to fixing the coordinates of the death of a game character, I may add events based on the form of the death that has overtaken him. It may be useful to know exactly how the game character died, and not just know where exactly it happened.
And, depending on the game, it can be useful to send the history of the movement of the game character to the server before its death at one or another level.
However, the key point of such a system is its simplicity. Data collection does not make sense until reliable tools have been created for their processing.
For my next game, I would rather leave the basic system for sending data to the server and saving them to the database intact. And I will focus on creating the best tools for processing the received data.
I am also interested in how the summary data obtained by players can be used to configure dynamic game complexity control systems.
If the game is able to receive summary data from the server, this will change its gameplay, based not only on the results of the game of one player, but also on the average data of millions of other players. In my opinion, this opens up new interesting opportunities.
Collecting and analyzing player performance is not an ideal substitute for custom testing. But they give a very useful averaged picture. And since the scorecard allows you to test the game on larger groups of players than is possible with individual beta testers, the system tells you more about the game in the long run.
The benefits received from the scorecard more than covered the cost of it in Replica Island. Keeping the simplicity of the client and server parts, I received a lot of useful information about the design of game levels and the habits of players, and as a result, the game only got better.
The only thing I regret is that I did not implement such a system in my previous games. It seems that it is applicable in almost all games of any genres for any platform.
Generating heatmaps is not that complicated, but finding the exact instructions took me some effort. I used a method similar to that described here .
When performing all these steps, make sure that you maintain the color space in the 8-bit range per channel (in particular, when calculating the opacity in step 5). Or consider using a format that supports floating point data for imaging.
You can easily identify precision bugs, which will be noticeable with a large number of events, so that the contribution of one event to the overall picture will be less than one percent.
Tools like ImageMagick can help you solve this problem.
Nothing beats the feeling you experience while watching someone play the game you created. In the process of developing a game, you play every day and develop, perhaps unconsciously, a certain style of play. But giving your game into the hands of another person, you get the opportunity to see what happens to your creation when it is used without your daily gaming experience.
Every problem that arises, a failure in the animation, incomprehensible text in the manual and periodic errors make a very strong impression when a novice plays the game. It doesn’t matter how long you have been honing the game process and how many mistakes you have corrected, your elaborated style of play and your special involvement in the gameplay that you create the game may prevent you from clearly seeing the problems that other players may immediately encounter.
That's why gameplay testing is a vital part of making a good game. To get the most out of testing, we will need to save information about the process of passing the game. This article describes my experience in solving this problem.
Starting small
I took my first steps in the gaming industry, creating games for Game Boy Advance. At that time, the testing approach was quite simple: we gave the neighboring guys a special Game Boy Advance device connected to a VCR, recorded their game, and then watched the recordings. This allowed us to catch obvious and major errors.
The sections of the game, the passage of which caused strong negative emotions in the guys, were accepted for additional study. When a player fails over and over at a specific place in the game, this is a clear signal that this part of the gameplay needs to be reworked. Several game runs with the help of the neighboring guys and we could significantly improve our games.
I am currently working as a consultant in creating games for Google Android. My first game for this platform, the Replica Island arcade, is not much different from the games for Game Boy that I wrote 10 years ago. But something has changed - I did not work anymore for the company that created the games, I wrote it for myself, with the help of one artist, mainly working in my free time.
Now I do not have access to the audience of young beta testers that was at that time. And even if this access, the current target audience for the game is somewhat older.
After all, there is no easy way to collect game progress information on the phone. The only way is to stand "above the soul" during the game, but this is inconvenient, and can affect the way the player plays, which reduces the purity of the experiment.
What should an independent developer do? Having completed the development of Replica Island, I realized that there is no guarantee that it will be interesting to play this game. The game was developed “in a vacuum” and I needed a third-party look at the gameplay before I could be sure that it was time to release the game.
The first thing I tried was polls. I placed the game on the internal server of the company where I worked, and sent letters to my colleagues asking them to play and send me their impressions. I even launched a small forum with questions about the game. This approach did not work.
Despite the fact that many downloaded the game, very few, less than one percent of those who downloaded, bothered to answer my questions. In addition, those who answered the questions did not provide enough information. It is difficult to understand what exactly was the reason for the assessment "the game is difficult to play." Whether the reason was inconvenient control in the game, poor design of levels, the location of obstacles in the game, guides to passing levels, or something else.
Reflecting on a scorecard
After that incident, I thought of a game-gathering and processing system developed by Naughty Dog for the Crash Bandicoot game .
The system recorded statistics about the game process on a memory card. This data was then collected (offline) in order to detect areas that were difficult to pass through, or areas where the game character most often died.
These problem areas were reworked, and this information was also used for a system for dynamically adjusting the complexity of the game.
One of the interesting principles that guided me in creating my system was the idea of Naughty Dog that we should avoid ending the game with all our might. Their ultimate goal was to eliminate the moments when the player was stuck at some stage and could not continue the game.
I liked this idea, but then I did not understand how feasible it is for playing on the phone. I asked around a little to understand the current situation with recording statistics for games with large budgets. And I learned that many companies use mechanisms to collect information about player activity.
Several people said that despite the fact that they collect a large amount of information, they experience difficulties in processing and presenting this data in a form convenient for perceiving and understanding what needs to be changed in the games being studied.
On the other hand, some studios have tools that can recreate the player’s movement in terms of level and generate statistics about which weapons the players prefer, which opponents are more difficult to defeat, and which parts of the map are clearly visible in one case or another.
It seems that the formation of a set of indicators for the player is applicable for a wide class of games, but this is justified only for large companies that invest significant resources to build tools for processing the collected data.
An example of how this type of system can be most effectively used can be seen in a presentation by Georg Zoeller about the wonderful telemetry system they use in BioWare.
It turned out that collecting data about the gameplay is an easy part of the task, but interpreting this data in such a way as to give a clear and useful picture to the creators of the game is much more complicated.
It sounded discouraging as I tried to keep my toolbox as simple as possible. But I decided nevertheless to experiment with some key indicators.
My Android phone did not have a memory card, but there was a permanent Internet connection. Perhaps, I thought, I can capture some important events in the game, send them to the server and get results from it. My goal was to understand as much as possible about how they play my game, while leaving the system as simple as possible.
Base system
The system I wrote consisted of three parts: a thread of execution, which during the game process collects data on the player’s actions and sends it to the server, the server itself, and a tool for processing the collected data.
Server - this is said too loudly, of course. My server was represented by 30 lines of a PHP script that checked a GET request via HTTP and wrote the results to a MySQL database. The structure of the GET request, in turn, was also very simple. This is the name of the event, coordinates, code version, session identifier and timestamp.
These data, directly in the received form, were entered into the database. The current data processing was also performed in PHP when a special page was opened on the server. Not a very good choice in the long run, this will be discussed below.
I started by registering two events: the death of a game character and the completion of a level. Each time a game character died or a player completed a level, the game sent the corresponding event to the server. From this data, I subsequently was able to form a rather detailed idea of the gameplay.
I was able to see which levels require the most time to pass, at which levels the player most often dies, and which of them passes too quickly. Having grouped these data by players, I saw what percentage of all players die at one level or another, and the average failure rate for each player.
Analyzing the coordinates of this or that event, I could say in which cases the game character died from the actions of the enemy, and in which from falling into the pit. For starters, my simple scorecard was pretty detailed.
Mark failures on the map in bright red
As soon as my base system began to work, I released an update for beta testers and began to monitor the incoming data. Quite quickly, recurring situations were identified.
At some levels, almost one hundred percent of the players failed at least once, while at other levels the players got stuck for several hours, which meant an unsuccessfully designed level, since it was supposed to pass within a few minutes. By analyzing this data, I got a clear idea of what levels need improvement.
But defining problem levels was not enough. Sometimes I could not say why a problem arose at a particular level.
And I took a step forward. Using the same data, I wrote a tool to indicate the location of the death of a game character on a level map. And I was able to see exactly where the game character died, and where not.
The first run of the system showed only a small point at the level where the game character died. But with the increase in the number of players, I began to get something like heat maps (heat maps), indicating the places of death of players on level maps, which were much easier to perceive. The chapter on creating a heat map will be presented below.
Heat map generated based on game character death data in Replica Island
Visual design miscalculations in game design
The combination of high-level game statistics and the designation at the level of places of death of game characters shed light on the shortcomings in the game design. I realized, for example, that a large number of players died when they met with the very first monster. This was not because the monster was very strong. After a detailed study of the situation, I came to the conclusion that the monster appeared in a place where the main method of attack - jumping from above - was difficult to implement because of the low located ceiling.
I also realized that my simple system for dynamically adjusting the complexity of the game itself required correction. This system, without publicizing this fact, increased the life of the game character and the energy for his flight after a certain number of successive deaths. Studying the statistics, I realized that she should have done this much earlier.
Also, the structure of my levels has undergone major changes. I had a sufficient number of levels with high time, which was spent on their passage, but with a small number of deaths. And I realized that the players simply got lost at the level. I redid these levels to make the order of their passage more understandable. In a couple of cases, I even completely redid the levels from scratch.
But the biggest problem I encountered was pits. Replica Island is a platformer, and, as you can guess, in it the player needs to make a large number of jumps over the pits. Unlike the constantly rotating marsupials and plumbers living in the pipes , my game character had flight as the main way of moving.
I needed a control system for a game character that would not require a gamepad. Since the protagonist of the game, the green Android robot used rocket engines on its feet to fly. The basic model of movement was to get an impulse still on the surface before the jump and, using this impulse, with the help of engines fly in the right direction. Engines consumed energy fairly quickly, but energy was renewed upon landing. And the idea was that the player would make a jump, and then, carefully spending energy, reach the right places, or make precise jumps on opponents, attacking them.
All this was good, but when I looked at the statistics of the death of game characters that came from beta testers, I found that, for the most part, they died in bottomless pits. Crowds of players failed even in the smallest pits. And more worried by the fact that over the course of the game the number of deaths in the pits did not decrease. Players did not improve their skills in overcoming obstacles during the game.
After which I carefully studied the design of the game and the design of the levels and formed several theories. The main problem, in my understanding, was that players do not see holes when they are in a jump. In the end, during the jump, the presence of a simple hole or a trap hole at the landing site was not noticeable. And since my levels are often very high, it is difficult to determine which pits led to the underground levels, and which of them led to a terrible death.
The second and more important problem was that the behavior of the camera showing the main character of the game was not very well worked out. When the main character made a high jump, the surface of the earth disappeared from sight. As a result, it was difficult to determine the right place to land.
Well-known platformers like Super Mario Bros almost never perform vertical scrolling. Mario has a whole set of complex rules that determine situations in which the camera can move up and down. However, in my game, the presence of flight mechanics meant that I allow vertical scrolling in all cases. After a large number of corrections made, I got a smarter camera behavior that does not start moving up, unless the player himself is close to the border of the displayed area.
After all these changes, I released another update for my beta testers and compared the results with the results obtained from the previous version of the game. The changes were very promising. The total number of deaths has decreased, the time for passing the levels has basically returned to normal. And mortality from falling into the pits decreased markedly.
I repeated the testing process several times before I was ready to release the final version of the game. With my scorecard, it was easy to determine how these or other changes affect the progress of the game by beta testers.
Game release
So, after several test runs, my graphs began to show the standard normal distribution , which I wanted. It's time to release the game and I decided to leave the scorecard built into the game. I was wondering if the information that I will receive from new users will be different from the information that I received from the beta tester group. There was only one way to find out.
Of course, every time the game sends data to the server, the best solution would be to notify the user.
The first time after the release of the game in the welcome message, which is displayed with information about the latest changes, was information that the game sends anonymous and not personalized information about the game process to the server in order to improve the game subsequently. And players who did not want this could turn off this system through the menu.
This approach seemed like the best solution. Despite the fact that the game code is open, and anyone could study the structure of the sent data packets (I made sure in advance that none of the sent data can be compared with a specific person or device), I gave the players the opportunity to say “no, thanks”.
Comparing the number of installations from the Android Market with the number of unique players in my scorecard, I came to the conclusion that less than 20% of game users refused to transfer data about the game process to the server.
As a natural result, I received a huge amount of data for analysis - more than 14 million information points, about a gigabyte of information about events that were generated by users of my game. At the time of writing, their number was 1,200,000 players.
In fact, this amount of information broke my system pretty quickly. I have statistics collected from the first 13,000 players that I posted on the Replica Island website. But after the release of the game, most of my analysis tools stopped working.
The good news was that the first 13,000 players gave statistical information that was very similar to the smaller beta tester group, which probably suggests that the results obtained in the test group can be applied to large groups players.
One way or another, this plan worked
I was very satisfied with my event accounting system at Replica Island. With a little effort that cost me almost nothing (the server side, which records events, cost me less than my Xbox Live account), and using only two types of events, I was able to quickly and efficiently determine the places in the game where the players ran into problems.
In addition, as soon as I began to collect this information, I got the opportunity to compare the summary data from my system received from different versions of my game, which gave me the opportunity to make sure that the changes I make to the game have a positive effect.
Using PHP and MySQL to implement the server side was a good solution. Capturing events was so trivial that I'm sure it could just be implemented in any language. In PHP, the implementation of the entire server side took me less than half an hour.
Using a separate thread of execution to send events to the server was also a good move. I did not want the user interface to be blocked while sending HTTP requests, and put the process of communication with the server in a separate thread of execution.
At first, I was afraid of a slowdown in the game on the device because of this, but as it turned out, there was no reason for concern. The additional load was so insignificant that I could not even notice it during the profiling of the game.
After all, keeping the whole system as simple as possible was a good solution. I looked at many different event options that I could record in my game. But tracking the moment of the death of the protagonist of the game and completing the level provided more than enough information for analysis.
Larger amounts of information would complicate the processing of data, and possibly make it more difficult to get a clear picture as a result. Now that I have some experience in creating an automatic system for collecting and processing indicators, I may decide to increase in the future the amount of data that I send to the server. But starting with a simple system was definitely a good move, in my opinion.
The bumps I filled
Not everything in the event registration system worked well. I made several decisions that did not give the expected results or simply were a waste of time.
The decision to use PHP for the server side was a good move. However, it was a mistake to use PHP to process the received data. My idea was to do everything through a web interface. I even wrote my level editor in PHP and JavaScript.
But PHP began to slow down when the amount of data to process increased significantly. PHP worked in a very limited environment in terms of memory and computing resources. And I almost immediately ran into these limitations. As soon as I began to receive information from 20,000 players, most of my PHP tools simply stopped working.
In particular, image processing in PHP turned out to be problematic. I implemented the whole heatmap generation mechanism in PHP, but I should have written something that would run locally instead of running on the server.
I came across a lot of errors in the PHP GD interface (forming images with an alpha channel simply did not work), and decided to simply reduce the size of the level images for further processing.
For this article, I rewrote this tool using Python and ImageMagick .
And the results were impressive. The code for this implementation can be downloaded from the official website of Game Developer magazine .
As a result, even though these data told me about when the game character died and how much time was required to complete the levels, they did not give me information in order to determine the moments not related to the death of the main character of the game, when the players quit playing my game (game shelf moments).
I came to the conclusion that I released a game with some key miscalculations in level design that my scorecard would never have discovered. In the most extreme situations, the players faced a task that they did not understand how to solve, and simply quit playing, not completing the passage of the level.
This has never been displayed in my system, since the condition for fixing the event of completion of the level did not occur. I only found out about this when the players began to send me complaints that they were stuck in the same place when passing the game.
Although my automatic system was very convenient, it did not show me the full picture of the gameplay. In my case, the indicators were good for identifying problem areas at the levels, but would be ineffective in determining the miscalculations associated with the order of interaction of the elements of the game with each other.
Future
For my next game, I will again use a similar automatic system. In addition to fixing the coordinates of the death of a game character, I may add events based on the form of the death that has overtaken him. It may be useful to know exactly how the game character died, and not just know where exactly it happened.
And, depending on the game, it can be useful to send the history of the movement of the game character to the server before its death at one or another level.
However, the key point of such a system is its simplicity. Data collection does not make sense until reliable tools have been created for their processing.
For my next game, I would rather leave the basic system for sending data to the server and saving them to the database intact. And I will focus on creating the best tools for processing the received data.
I am also interested in how the summary data obtained by players can be used to configure dynamic game complexity control systems.
If the game is able to receive summary data from the server, this will change its gameplay, based not only on the results of the game of one player, but also on the average data of millions of other players. In my opinion, this opens up new interesting opportunities.
Collecting and analyzing player performance is not an ideal substitute for custom testing. But they give a very useful averaged picture. And since the scorecard allows you to test the game on larger groups of players than is possible with individual beta testers, the system tells you more about the game in the long run.
The benefits received from the scorecard more than covered the cost of it in Replica Island. Keeping the simplicity of the client and server parts, I received a lot of useful information about the design of game levels and the habits of players, and as a result, the game only got better.
The only thing I regret is that I did not implement such a system in my previous games. It seems that it is applicable in almost all games of any genres for any platform.
How to generate heatmaps
Generating heatmaps is not that complicated, but finding the exact instructions took me some effort. I used a method similar to that described here .
The main actions are as follows:
- We generate an image of a circle in grayscale, the color of which changes from black in the center to transparent at the edges along a radial gradient. This is an image to display the point at which the event occurred.
- We generate a rectangular image with a color gradient. Let the bottom of the image be white or red, or any color that you choose as the designation of the “hottest” spot on the heat map. The top of the image should be black with several colors in between. This image will be used as a “dictionary” to generate a graphical report on statistical data later.
- We generate a list of coordinates of past events.
- We calculate the maximum number of points of events that are in close proximity to each other. This will be the maximum “heat” value for the heat map.
- For each unique location in the list of events, draw a circle image based on the coordinates of the event. We draw an image with an opacity coefficient calculated by the formula:
(количество событий в этом месте)/(максимальное значение "тепла") * 100%
Use the multiply transfer model (src * dst) to mix the points of the drawn circles with each other.
Overlay the resulting image on a transparent canvas.
When the process is completed we will get an image with a lot of dark spots with a varying intensity of black. This is an intermediate image that will be processed in the next step. - We take the image obtained in the previous step and add color to it. We take the transparency level (alpha) for each point and based on it we calculate the Y-coordinate in the “color dictionary” built in step 2 in order to calculate the color for the processed point.
- Take the resulting image and overlay it on top of the game-level image. Event locations will be shown as colored areas, where increasing color intensity indicates areas in which more events have occurred.
When performing all these steps, make sure that you maintain the color space in the 8-bit range per channel (in particular, when calculating the opacity in step 5). Or consider using a format that supports floating point data for imaging.
You can easily identify precision bugs, which will be noticeable with a large number of events, so that the contribution of one event to the overall picture will be less than one percent.
Tools like ImageMagick can help you solve this problem.