Six experiments from Skyscanner that brought growth to the mobile application

    The main growth manager at Skyscanner, Caterina Toniolo, told Epic Growth Conference about how the team tested hypotheses and conducted experiments to increase the growth of the mobile application.


    Watch the video and read the notes under the cut.

    The Skyscanner website is visited by 60 million unique users per month, and the application was downloaded by more than 70 million people. The product allows you to search for tickets on various devices, not only desktop and mobile, but also through Cortana, Alexa, Apple Watch.

    Main challenges on the road to Mobile First


    Before you run the experiments, you need to decide on the main metric in the product. For Skyscanner, the “guiding metric” is the Total Transaction Value or the total transaction price for a period of 12 months. For the team, this means that it is necessary to increase the total cost of transactions over the next 12 months.

    In its practice, Skyscanner uses the AARRR marketing model, which includes the so-called “pirated” metrics: attraction, activation, retention, referrals and revenue. Retention, as with the apparent majority of companies, is the most important metric, which shows the percentage of users who return to the product.

    The Skyscanner growth team is focused on the Mobile First strategy, because more and more travel-vertical users are using mobile apps, rather than laptops, to plan their travels.

    However, there are many obstacles to the Mobile First strategy. In the case of Skyscanner, the obstacles are in the product itself, which was originally developed as a website, and not as an application. Other unresolved issues are cross-platform, halo effect, attribution, diplinks, and working with data.

    Cross platform and halo effect


    First of all, before launching the tests, the products need to calculate the LTV (life-time value). To do this, it is necessary to understand the behavior of users on different devices. But, if authorization is not required, as in Skyscanner, it becomes a challenge.

    Another important point is the understanding of the halo effect and how it affects the company's money, namely, how much is spent on attracting users to the application and on increasing organic attraction. For this, it is extremely important to understand the general user behavior and the LTV indicator.

    Attribution


    The most popular attribution model is last-click, attribution on the last click, but it works far from all companies.

    For example, Skyscanner has reached a scale at which attribution on the last click stopped working. Skyscanner is a low-margin product, and the company is trying to get the most out of its resources. Therefore, the team has identified a multi-touch attribution model, which allows you to multiply the growth scale.

    Diplink


    It is extremely important to provide the user with maximum comfort when interacting with the site or application, and when switching between these platforms. But working with diplinks is not as easy as with regular web links.

    Data


    On the one hand, there is an application market that provides various sources and useful data types. On the other hand, the team must make great efforts to process everything and make sure that it is not separate data.



    Hypothesis Testing and Growth Experiments at Skyscanner


    In any company, growth must always be put into practice. The more you test, the more you discover opportunities to grow.

    Experiment with halo effect


    Hypothesis: if the number of paid plants in the Indian market is increased daily, the number of organic plants will also increase.

    Metrics: organic downloads.

    Result: as a result of five iterations of the test - an optimal halo effect was achieved: one organic plant per 1.6 paid plants. The team tested different spending options per day to find the best one. We started with 500 paid installations per day, it took five paid installations to make one organic. But after the optimal level of 2400 paid installations was found, the growth rate was fixed at 1.6.



    App Store Optimization Experiment


    Experiment Skyscanner about growth opportunities in the Russian market.

    Hypothesis 1: if you add relevant keywords in iTunes Connect, you can increase the ranking of these keywords in the App Store.

    Hypothesis 2: keywords in the description of the application in the App Store will help to increase the rating of the application and the number of downloads from Russia.

    We analyzed the description of the application in Russian and found out that of the 100 characters available, we used only 95. One of the words was as many as 12 characters (14% of the entire description length!). We decided to use two words instead of one.

    The use of adjectives is another potential for growth. Thanks to them, you can provide a much greater variety of keywords. There are, of course, such high-frequency keywords as “car.” However, they would strongly compete with other words and would not be as relevant as possible. In addition, we began to consider keywords that were automatically suggested on iOS.

    As a result, the team created a new list of keywords. The main indicator of success was an increase in the rating, the secondary metric was organic downloads. According to the hypothesis, the success will be an increase in the rating by replacing at least 50% of keywords.

    Method:Skyscanner applied a two-sample t-test and used two different data sets with a 95% confidence level to compare them with each other and find out which one would win. The first group studied the average rating for three months before the experiment. The second group studied the data obtained during the experiment, one month after the test.

    Result: the team improved the rating of more than 50% of keywords.

    As a result, we got one more conclusion: the possibilities for optimization are everywhere, even there, from where you do not expect.

    Tip: do two-sample t-tests.



    Experiment with app optimization on Google Play


    Hypothesis: if you place the flag of the Russian Federation on the application icon on Google Play, it will increase the number of organic plants from Russia.

    The Skyscanner team created a null hypothesis - this is a hypothesis that the researcher is trying to disprove, to prove the opposite. To disprove the hypothesis, Skyscanner set a minimal detectable effect — 14 days after launching the test, see an increase in organic plants of 4.25%.

    Result: with the flag on the icon in Google Play, the number of organic installations of the local version of the application decreased by 4.8%. We failed, but we learned an important lesson.

    Experiment with paid optimization to attract users


    Hypothesis: using paid optimization, you can increase the conversion.

    Using anonymous IDs in the AppsFlyer mobile tracker, we tracked the conversion in the application and optimized the costs of those IDs that showed the best performance.

    Metric: ROI growth.

    Result: in the first iteration, ROI increased by 3%. As a result of further optimizing the cost of an anonymous ID with the most effective indicators, ROI increased by another 66%. This experiment was recognized as one of the most effective at Skyscanner and is now practiced every month.

    Experiment with rating optimization


    Hypothesis: if you invite users to put a rating right in the app, the number of ratings on the App Store will increase.

    In the Skyscanner mobile app, you could leave feedback in two steps. At the first stage, users were asked if they liked the application. Those who answered positively were offered to put stars on the App Store. Here the team decided to remove the first step and send the users immediately to the App Store. The purpose of the experiment was to increase the number of reviews and organic downloads. Before the experiment, the average rating was 4.36 points.

    Metric: the total number of reviews, the average rating of stars in the application and the number of organic downloads.

    Result:the average rating did not change, but the number of reviews increased by 48%. After that, it was decided to repeat the test across the entire mobile product Skyscanner.

    We also noticed an increase in organic downloads, but did not find a direct connection with the experiment.



    Experiment with engagement and engagement


    Hypothesis: if we place the Star Wars direction in the search box, the user engagement ratio will increase.

    The experiment was carried out for one day, on Easter. In the search bar, along with recommendations and great deals, planets from the Star Wars universe were added with fictional descriptions, illustrations and prices.

    Result: the level of interaction with the search string increased by 27.5%. In addition, growth opportunities were identified with this element.



    For more product growth reports, see the @epicgrowth Telegram channel .

    Transcript of speeches published on VC .

    Also popular now: