"How to measure the pleasure of the game" and other frequently asked questions about application analytics

Every month we arrange an AMA (ask me anything) session on reddit - we answer any questions about application analytics. We decided to select the most interesting questions, and provide answers to them here. What types of games do you need analytics for? How to conduct experiments? How to increase retention? The answers to these and many other questions are under the cut.


[-] jakkarth
Do I need analytics for all types of games? I admit that it’s useful to use analytics in such large and complex games as World of Warcraft, but is it needed in casual mobile games that people play for 5 minutes? Where is the critical mass upon reaching which analytics is required? Are there any genres or types of monetization that are harder to analyze than others?
devtodev:
My opinion is that analytics is important regardless of the size of the game. Of course, the larger the project, the more sophisticated analytics it requires. When you prepare the game for launch, you still do not know its future scale, but this does not mean that you do not need analytics.
By the way, do not underestimate the volume of casual mobile games where people play from time to time. Check out the post by Paul Murphy, founder of Dots, for example. I can imagine how big and complex analytics are built in their company.
And answering your last question, I’ll say that the depth and complexity of analytics depends more on the scale of the project than on the genre.

[-] seardluin
What metrics would you recommend to use in a RPG game for PC, in which there are no built-in purchases, but there is the possibility of playing online? Would you analyze an offline game, or just an online component?
devtodev:
I think that in any case, you are interested in user loyalty, for example, to accumulate a user base for subsequent games. Therefore, I would recommend tracking both online and offline metrics.
What metrics to use?
First of all, audience metrics: DAU, MAU, loyal DAU (minus one-day users).
Then I would recommend using retention and player activity metrics: retention from 7 to 90 days, sticky factor, number of sessions per user.

[-] Novamonk1
How can I understand if users enjoy the game?
[-] koorb How do you measure custom fan?
devtodev:
Metrics such as pleasure or fan simply do not exist, and therefore we cannot measure them. We can only approximate them in some way, approximate them with existing metrics:
Retention (1 day, 7 days, 30 days and above).
Sticky Factor. It is calculated as the ratio of DAU to MAU. Good metric talking about regular user logins. It has a strong correlation with the income of the game.
The number of sessions per day. It is believed that for a good healthy gaming mobile project, the optimal value is 3 sessions per user per day. In games with a longer session (say, RPG) - two sessions. In games with a short session (casual games, runners) - 4-5 sessions.
Average session length and average time spent by the user in the game per day (session length * number of sessions).
K-factor as an indicator of virality. If the user likes the game, he is more likely to tell friends about it. Typically, the K-factor is calculated as the average number of invitations sent, multiplied by the average conversion from the invitation to registration. However, for mobile projects it is difficult to track the user base covered by the invitation, and therefore the K-factor is calculated as the ratio of the number of new organic users to (DAU - New Users).
I would not recommend using these metrics to compare the pleasure users receive from two different games. But they are perfect for measuring changes in the attitude of users of one particular game to changes in it.

[-] gruegoo
What should I compare the values ​​of my metrics with? I want to compare DAU / MAU indicators and the percentage of paying with other indicators in the industry, but I do not know where to get the data. It is not only about specifically these metrics, but generally about the search for samples for comparison.
devtodev:
I would also like to have an inexhaustible source of benchmarks throughout the industry, it would be really useful. But in practice, you just have to search for information on the Internet yourself.
A good resource is Quora.com, a Q & A service. There you can ask a question about the meaning of metrics in a particular genre, in a particular country. The audience of this service is happy to share information. Here is an example .
I also recommend subscribing to the blogs of devtodev and other analytical services. There you can often find articles with data that can be used for benchmarking. Here are examples: a GameAnalytics article and a devtodev webinar entry.

[-] Jerseywhat
What do you think, how to organize experiments correctly?
What role should analysts play in this process? Decide which experiments to launch, set priorities?
If the team wants to run several experiments at the same time, will you try to somehow analyze their results, or will you suggest to divide the experiments in time? Run many experiments in parallel, or run one experiment with many different options (for example, group A: high complexity, low costs, group B: medium complexity, low costs, and so on)?
How do you work with statistical significance? Which is better, wait until a large enough sample is collected to make the result significant, or as soon as possible choose one of the options without waiting for the end of the experiment?
devtodev:
I believe the analyst’s task is not to offer options for testing, but to provide enough information to the game designer so that he himself offers various hypotheses. And subsequently, it is the analyst who must evaluate the result.
I am in favor of running unit tests. Yes, it will take more time, but the results will be more reliable. If you run many experiments at the same time, there is a high risk that the sample for each option will be too small, and the results may turn out to be erroneous.
This is a matter of project scope. Typically, small projects want to make decisions and make changes instantly, while large projects take longer. As an analyst, I would not recommend making important decisions in a hurry. But perhaps this is just a professional deformation from the fact that as a rule you have to deal with large projects.

[-] bimbompaul
I know there are two ways to calculate retention - classic and rolling. What is the difference? Which one is better to consider in practice?
devtodev:
Good afternoon!
In the case of classic retention, the user is considered withheld on the first day if he entered the application a day after the first visit. In rolling retention, the user is considered withheld on the first day if he entered the application a day after the first visit or later. Therefore, rolling retention values ​​are always higher.
Both retention serve the same purpose - measuring user retention, and they are strongly connected to each other. It is hard to imagine that many users will not log in the first days, and then will begin to use the application regularly. The higher the classic retention, the higher the rolling, and vice versa.
Hereexample of classic and rolling retention values ​​for different projects. If your project is based on long-term retention and communication with the user, I would recommend using sliding retention. If you need benchmarks and benchmarks, then it’s better to use the classic hold, simply because most of the benchmarks on the Internet are for the classic one.

[-] cairmen
An excellent topic for the heading 'question and answer'!
In your opinion, with what metrics can you find the simplest ways to increase revenue and retention?
devtodev:
Metrics alone will not increase your revenue and retention, but they will allow you to generate ideas on how to achieve this.
Let's discuss. The key to successful monetization is long-term retention (the longer the user is with you, the higher the chances of paying him and the average check). The easiest way to increase long-term retention is to optimize short-term retention.
Well, the easiest way is to increase short-term retention by optimizing the first user session (FTUE, first time user experience).
Another reason for optimizing FTUE is the fact that it is during the first session that the basics of user activation are laid, which, in turn, leads to success in monetization.
So, try to optimize the first user session:
look at your product through the eyes of a novice user: what does he feel, what does he think about after one, two, three minutes in your service?
integrate user events into your tutorial (our practice shows that many users leave the tutorial simply for technical reasons, and your task is to understand and optimize)
show the advantages of your product over everyone else
intrigue the user, let him know that it will only be more interesting next.
Use the following metrics to measure success:
percentage of users who successfully complete the tutorial;
1-day retention (both classic and rolling);
conversion rate on the first purchase.

[-] aschearer
What three metrics would you highlight among those that best describe the state of the game, and why?
devtodev:
Good afternoon! Let's look at the game as a model - new users are entering, money is leaving. Therefore, if you choose three metrics, these will be:
new users;
percentage of paying users;
Paying User Average Revenue (ARPPU)
If I know these metrics, I can calculate key indicators and compare businesses with each other. And if you allow me to add a fourth metric, I would add retention.

[-] BikerMouseFromMars
From which books would you recommend starting to study game analytics?
devtodev:
Most of all I like this book .
I also recommend the book “The Goal” by Eliyahu Goldratt. It is not about games, but in general about the search for bottlenecks in business, and this is true for everyone.

[-] svero6
I have a few big questions right away.
What are the standard metrics for MMO games? And how much differ from each other the values ​​of the metrics of hit games, average games (for example, platformers) and unsuccessful games?
If you have an in-game store selling virtual goods for virtual currency, how will you optimize the selling price? How do you understand that this or that price is too high? Or too low? Usually AB tests are used for such purposes, but in online games users communicate a lot with each other, and if you experiment with prices, this will not go unnoticed and break the experiment for you. How else can you optimize the price, except how to change it and see what happens?
Finally ... How to understand how much product to give users for free? Let's say you earn gold in Hearthstone by winning matches, and you can spend that gold on buying sets of cards or other goods inside the game (instead of buying them for real money). In other games, you accumulate virtual currency, completing missions, and so on. How to understand how much to give gold to users? How to optimize it?
devtodev:
Thanks for the interesting questions!
> What are the standard metrics for MMO games? And how much differ from each other the values ​​of the metrics of hit games, average games (for example, platformers) and unsuccessful games?
First of all, for MMO games, I would recommend using a standard set of metrics .
I also recommend paying attention to in-game currency, the dynamics of its spending and savings. Do not let the average currency balance on the players' accounts grow, otherwise the day will come when you will have to raise prices.
Regarding the difference between hit, average and unsuccessful games, I recommend the GameAnalytics article , in which you can see how the values ​​of the metrics differ.
> If you have an in-game store selling virtual goods for virtual currency, how will you optimize the selling price?
Good question. You are right, the first idea is to conduct an AB test, but this is not possible in projects where users communicate a lot with each other.
I would recommend using ABC / XYZ analysis for your virtual goods. This will allow you to highlight categories of products where you can easily change the price (sub-segments BY, CY, BZ, CZ).
If we talk about sub-segments with the letter A, then it's hard to change something when the game is already running. Therefore, it is better to form and fully calculate the consumer basket before launch. Say during beta.
> How to understand how much product to give users for free?
This is more of a game design than an analytical question. From my position, I can only recommend a few links to materials that can help you find the answer:
www.dubitlimited.com/blog/2010/12/31/design-a-social-game-economy-how-much-should-a- virtual-t-shi-html
www.quora.com/How-do-you-set-prices-for-virtual-goods-in-social-games
Freemium Economics (Eric Seufert)
Virtual Economies: Design and Analysis (Vili Lehdonvirta, Edward Castronova)
Free-to- Play: Making Money From Games You Give Away (Will Luton)
Real economics in virtual worlds: A massively multiplayer online game case study, runescape (Tanla E. Bilir.)

[-] thealchemistbr
I am working on a game in which players need to build pyramids of numbers in a special way. The game has three difficulty levels. What metrics are worth using? How do I increase retention? And how do I choose a monetization model based on this information? Thanks!
devtodev:
Good afternoon!
> What metrics are worth using?
If your game is not an MMO with a complex economy, a standard set of metrics is all you need . I would recommend paying special attention to session metrics (number of sessions in general and per player, average session length). I think this is important for casual gameplay. If I'm wrong - let me know.
> How do I increase retention?
There are many ways to increase retention. One of the most effective is the optimization of the tutorial. Teach your users how to play, show them the benefits of your game. A good tutorial is a good one-day retention, which means a good long-term retention, which means money.
I can also recommend daily quests, regular tasks. Well, nobody canceled deep-linking with push notifications.
Hereour webinar is about retention, there are more tips on how to increase this indicator. is our webinar about retention, there you can find some more advices how to increase it.
> And how do I choose a monetization model based on this information?
However, usually the monetization model is chosen before you begin to collect data. For casual games, the most popular and effective model is, of course, free-to-play. If you run a f2p game with long-term retention, you have every chance to succeed. See , for example, how Clash of Clans succeeded.

Conclusion
This is a large part of our favorite questions, but, of course, not all that we can answer with the help of analytical tools. If you are interested in anything else, we will be happy to answer in the comments or private messages.