Vivisection success

    For the past 13 years I have been working at My.Com. This is a very big company consisting of more than 10 various studios making games, first of all mobile. Such a scale implies a greater density of launches of new games, about 3-4 games a year. As you understand, working with such a stream requires a great ability to quickly and accurately evaluate projects at each stage of their creation.

    And here the main difficulty is associated with projects that demonstrate controversial results. If everything is clear with weak / strong, then there are always problems with the assessment of the latter, since the task leaves the assessment plane of direct indicators to the plane of assessment of the potential potential of the project.

    For example, if the casual project Retention of the 1st day has 20%, then this is definitely a weak project that needs to be closed. And if this is a midcore game with Ret1 30%? It's not so obvious.

    Of course, this midkorn game can be closed, but then there is a chance that we will lose the hit, because the potential of the game may not yet be revealed and developers need another year to turn the ugly duckling into a swan.

    If you give the developers this very year, and the beautiful swan doesn’t work out as a result, the company will lose the funds and lose the benefit, since the developers could have made another game this year, which would have become a swan.

    To correctly get out of this fork, you need to be able to assess not the current state of the project, but its potential. But how to see it, judging by industry standards? Unfortunately, no way. Industry standard is too superficial and versatile to "look into the soul" of a single project. To see the potential, a little simple magnifying glass, you need a powerful microscope.

    My practice shows that today all decisions on controversial projects are made on the basis of intuition, credibility, or through infinitely long Soft Launch, which makes the decision-making process expensive, unpredictable, and sometimes even painful. Therefore, I would like to talk today about this “microscope” and tell about my experience of its use.

    Opportunities of a single person are overvalued.

    We begin our conversation by questioning your current ideas about the ability to see the potential of a project and to predict the success of certain games. We all, of course, have our own opinion on this matter, but, with a high probability, it is erroneous, because it relies on the subjective perception of the project, which makes your opinion hostage to all sorts of cognitive distortions.

    We at have come a long way before we realized that there are no experts capable of analyzing the potential of games on a flow basis. Undoubtedly, success happens from time to time, and people immediately appear around him who begin to tell with a clever look how they believed in this success or predicted it. But if you look at the history of all the statements of these people, you will see a lot of conclusions that did not lead to success ... So how can you solve this problem?

    I personally found a solution using our own track record. My studio is part of the company, and I, like all other developers, have access to the statistics of not only my projects, but also the projects of allied studios. Of these, I identified those that meet my criteria for success, and began to dissect their data in search of common formulas for success.

    Success criteria

    First you need to decide on the criteria for the game, which can be considered successful. In my coordinate system, these are games that earn $ 1,000,000 net per month and have a margin of> 30% a year after the global release.

    At, such projects appear fairly regularly, so there was enough data to study and the first conclusions were formed very quickly. I drew them into the following two axioms:

    1. Successful play without successful marketing does not exist.
    2. Successful marketing without a successful game does not exist.

    Yes, a study of our in-house products showed that successful products always combine the history of successful marketing and successful play. Although these areas do not directly depend on each other, it is at their intersection that the level of success we need is formed.

    Therefore, I want to draw your attention to compare the success of various projects on game metrics in isolation from marketing metrics is absolutely useless. It's like talking about the color of the object, considering its shadow.

    Therefore, further preparation of success, I produced already in terms of marketing and in terms of game metrics.

    Evaluation of product success from a marketing perspective

    An analysis of the marketing of our successful games quickly enough showed that their formula for success lies in the project's ability to attract many cheap registrations. Due to this volume, revenue targets were formed, and subsequent optimization provided the necessary marginality.

    To describe the effect of allowing a large amount of traffic to be attracted to a project, we need a parameter such as Install Rate (IR) ads. In my opinion. It is this indicator most clearly reflects the marketing success of the game. I consider it as follows:

    IR = CTR * CR

    Where CTR is the conversion from impression to click; CR - click to install conversion.

    Thus, the higher the IR, the easier the project attracts users and the cheaper they cost it.

    To get a high Install Rate, you need to ensure high clickability of advertising and high conversion from the page page. This can be achieved only if the advertisement and the store page carry a message, which at the same time catches the audience and forms the correct expectations of the product. The latter is needed to keep the user listed. You can create a catchy message, but if it does not meet the real expectations of users when they next enter the game, they will leave, and it will not work to recoup the costs of attracting them.


    This graph illustrates well the strong relationship between CTR and CR. Here you can see the same conflict cling message and real expectations from the game, when the launch of a marketing campaign with a very high CTR causes the collapse of the CR.

    As my practice shows, the first sign of a game’s success is the presence of a high CTR in video ads that demonstrate its gameplay (> 1%). After this, the task of the Store page is reduced to supporting and stronger opening the message formed by these video ads. If everything is done correctly, you get a CR in the region of 30-40%, and the Install Rate itself will be in the region of 0.4-0.5. This value can be considered a sign that the project from a marketing point of view will be a success, and its development must be addressed, even if there are a number of other problems.

    How to collect Install Rate values?

    For this, I advise you to test video ads on video grids, since their targeting is quite common, and therefore the results will be as impartial as possible. Also, this approach will allow you to compare different projects with each other. But make no mistake here and do not forget to take into account the stage of the life cycle of the compared projects! If you compare projects of different ages, you will get incorrect conclusions. Not taking into account the life cycle is a very gross mistake that even professionals often make. For an example, see how the ROI of marketing campaigns of our two hits changed just a year after launch:


    Such a strong difference in payback suggests that at the start of the project attracts the target audience, which is very cheap and pays well. But over time, it ends, and payback begins to fall even despite the product refinements of the project over the past year. Therefore, comparing projects with each other, be sure to make an amendment to the lifetime of the project.

    Can I influence the Install Rate?

    Since IR is a reflection of the reaction of the audience, it is an external input that cannot be influenced; it can only be calculated. This is a consequence of the work that was carried out by the producer at the Preproduction stage, when the game concept, the campaign setting and the date of product launch were approved. This is a curse or blessing that the producer imposed on the project when he gave the development a "green light".

    Of course, I say this so categorically only on the condition that:

    • The work done on positioning the project and the production of promotional materials is done at the proper level and does not contain gross errors.
    • We are talking about the long term. Because at short distances, the Install Rate can be periodically swung through the creation of particularly successful promotional materials, but sooner or later they burn out and the project falls to its base level. And in the long run, we can only slow down the decline in the Install Rate.

    This graph illustrates well the inability to strongly influence IR. We see that at the start the project has a high IR value, but after some time there is a strong downward correction (the cream of the target audience is collected) and the IR goes to its base level. After this, a long-term tendency to its gradual decline is formed, even despite periodic successful advertising activities.

    Evaluation of product success from the perspective of gaming metrics

    Unlike marketing metrics, game metrics are very difficult to compare. For all my time working in the company, we had a lot of holivars in an attempt to develop a unified concept for assessing the quality of the game. But none of these attempts were unsuccessful, since they all broke up about the genre difference.

    Although all of our products are called games, in fact they are completely different products that are united except the presence of an element of entertainment. All games entertain players, but in each genre in their own way. It follows from this absolutely natural rule that it is correct to evaluate the quality of the game within the framework of its genre. But! If you analyze the history of the genres of mobile games, you will see that they are constantly evolving, and particularly explosive combinations appear when genres mix and form new genres and subgenres.

    This is due to competition in the mobile market, which rewards, first of all, those games that are the first to give a unique gaming experience that can monetize in the long run. All this immerses mobile games in an environment of extreme elasticity, thereby blurring the definitions of the genre and presenting it in the form of some kind of game metrics.

    Therefore, when we talk about mobile games and the formation of criteria for evaluating their metrics, we find ourselves in an extremely chaotic environment that does not allow us to make mathematically reasoned comparisons. All that we can do in these circumstances is to roughly compare game metrics by threshold values, which will allow us to divide the games into bad and good, but not to evaluate the controversial games in terms of their potential.

    How to be? Is there an answer, argued by mathematics, and not by credibility of the developers? I think there is.

    In our studio, we did not immediately come to this understanding, and did not come systematically, but rather by chance, for which it is worth saying thanks to Google Play - his market research became the starting point for our own, which led us to answer.

    This happened in 2017 in Brazil. There, Google Play organized a conference for its developers and shared a variety of information. One of the slides of this presentation told about the existence of 4 groups of projects whose success is characterized by a retention curve in the first 10 minutes of the game.

    At that time, we were obsessed with the problem of improving retention in the game Juggernaut Wars, so Google Play research inspired us a lot because it offered to look at retention from an unusual side: how high is its value in the first minute of the game and what is the growth dynamics with each subsequent minute.

    According to Google Play, the stronger the growth from minute to minute and the higher the initial value, the more hits the game. Armed with this concept, we analyzed our portfolio and it worked! We received the following graph, in which the position of the curves of our top projects and one mediocre very accurately reflected their subsequent box office success.

    “Wow!” We thought at that moment. It turns out that you can estimate the hit of the game by the first 10 minutes!

    To feel the importance of this conclusion, point out that even one minute spent in the game has a stable mathematical probability of returning the user to the next day. And if we are talking about a hit, then even this minute is enough to form a significant difference in the retention relative to a non-hit game. And what can a player see at that moment? Virtually nothing, it will be the first impression of the setting and the basic mechanics. He will see that it is impossible to change without completely redoing the project. This can be compared with the presence of their DNA in the game.

    Thus was born the hypothesis about the existence of DNA in the project, which can be interpreted as the very metric that can mathematically express the potential of the game.

    A project's DNA is a fundamental set of game qualities that determines the likelihood of a user returning after the first minute, and also determines the degree of growth in the likelihood of returning with each subsequent minute spent in the game. The higher the initial value and the stronger the growth from minute to minute, the better the DNA of the game and the stronger its potential.

    Such a micro level of analysis allows erasing genre boundaries and evaluating the game project almost at the physiological level. Evaluating the player to return to the project after one minute, as well as measuring the growth of this thrust with each subsequent minute, we work with the player's unconscious reaction. On such a period of time, the game data is still too little for the player to make an informed choice, but unconsciously he still makes it. If the game concept and its implementation is strong enough, then it will even drag the player to an unconscious level, and the longer he is in the game, the stronger his influence will be.

    At what point can DNA be measured?

    This concept begins to work when the project has at least 10 minutes of release-quality gameplay.

    It turns out, if you ensure proper performance, then measure the hits of the game, theoretically, you can already at the stage of Vertical Slice. In practice, however, it will not be possible to measure Closed Alpha, since for the construction of the graph it is necessary to provide a large sample size to smooth out possible fluctuations. This study is based on an analysis of the classic hold of the first 10 minutes and requires representative cohorts of users for each of the 10 minutes. And to ensure the formation of all these cohorts and remove the risk of fluctuations, you will need a minimum of 40,000 registrations. As soon as you can provide such a volume of registrations for a 10-minute gameplay of release quality, you can learn the DNA of the project.

    How to interpret DNA values?

    At the output you will have a curve, the absolute values ​​of which have value only when compared with other curves, calculated under the same conditions. When work is being done with minute retention, the task becomes very project-dependent, when the nuances of login settings and user logins on the server begin to play a huge role. Therefore, you will get maximum data when comparing projects made within similar server solutions.

    But even if there is nothing to compare with, you can evaluate the project in terms of the constancy of growth. For a successful game, it is important that with each minute spent, retention grows: the more people spend time in the game, the more they should like it. If there is no growth of retention, or there is a drop in it, then this is a sign that there are fundamental problems in the game, which I call wormholes. Here is an example of one of our projects where this wormhole is pronounced.

    Here you can see that in 10 minutes of the game, the growth of retention is practically not observed. If our hit games it grows by 250%, then this one is only 20%. At the same time, we see that holding the second minute is significantly lower than holding the first minute, and this suggests that the game not only does not entice the audience, but also roughly loses already interested users. As long as the project has such a wormhole, it cannot absorb the audience effectively.


    Working in a big company that combines more than 10 different studios, each of which works simultaneously on several projects, I got the opportunity to study all our positive experience and look at the success stories from the point of view of analytics and my own experience.

    My task was to learn to objectively evaluate projects not only from the standpoint of overtly weak / strong projects, but also to form an approach for evaluating controversial projects by analyzing their potential potential, which, for a number of reasons, may not yet be disclosed.

    I hope the work I have done will help developers make informed decisions on such momentous issues as the start or closure of projects. After all, any work on the game is not only work, but also time. And if the hard work of game developers do not take, then the time - the most limited resource, use it wisely. Good luck!

    Also popular now: