June 6, 2019 at 12:51Cases of successful (and not so) experiments of Yandex.Navigator
The Yandex.Navigator product team shared what experiments are being conducted, how the work is organized internally, how the product is monetized, and what approaches it uses in predictive analytics.
Epic talks is an Epic Growth project where we communicate with teams about well-established processes, testing product hypotheses, analytics and much more. The project was shot with the support of the analytical platform AppMetrica .
Lyosha Gozhev, senior product analyst and Misha Vysokovsky, head of Yandex.Navigator
How is the product team of Yandex.Navigator arranged?
Inside Navigator, we have several product teams, each of which has its own focus. Focus sets the direction of product development. For example, one team is guiding the route, the other focuses on the search script. Unlike a formed product team, focus may change over time. Typically, each team has a product manager, designer, and developer.
We now have two analysts, so they work for several product teams. Mostly the tasks of the analyst are divided into two types.
The first type is adhoc tasks that need to be quickly counted, or tasks that require verification of coding. The second type is research tasks that the whole team needs to dive into. In such tasks there is no separation between the contractor and the customer. The whole process of research tasks is very similar to traversing a growing tree: we generate hypotheses, and then test them. During their refutation or confirmation, new hypotheses arise.
Give an example of successful experiments?
We try to guess where a person will go and offer him a “destination suggest”. For example, the user opens the navigator, and the application offers to go along the most frequent routes. Now we already have about 13% of the routes so constructed. This indicator is constantly growing. We are engaged in improving the predictive model, see how it affects and how much more convenient it becomes for the user.
Another example of a successful experiment is the redesign of the interface using ordinary electrical tape. Since we have a specific product, testing a navigator sitting at a table is a waste of time. At some point, we looked at the interface of Y. Navigator and realized that it had a lot of duplicate information.
We thought: “Listen, the third screen above is a huge black panel. Maybe we should throw it away and without it everything will be fine? ” The easiest option is to take a regular electrical tape and glue the top plate.
The first feedback that appeared was "I understand where to go, but it is not clear when I will come." We took the scissors and cut it a bit so that the arrival time was visible. We were surprised by the positive feedback. People calmly got to a place without information that was sealed with black electrical tape, while driving along an unfamiliar route.
How many hypotheses are tested per week?
We used to use the hypothesis testing approach every Monday from two to four hours. The goal was to make it a habit for product managers to test some new hypothesis, test a new solution, prepare a prototype, and conduct an in-depth interview. After we realized that it was common for everyone, we changed the system.
We did this in the form of an event on Friday, where everyone shared their impressions about testing the hypotheses. We brought in products not only from the Navigator, but also from Geoservices. We compiled a rating of employees where everyone received stars for their research.
This was after the first small team took a course in design thinking. It was an online course, but with an offline part that required the entire team to complete tasks. When we realized that this approach was popular within the company, we began to conduct a course for product managers from Geoservices on our own. We have successfully implemented this initiative, and in the future it helped the internal growth of employees.
We conduct all analytical hypotheses in the task management system. Therefore, this helps us analyze their number and apply in the future the calculations that were carried out for research. On average, in a quarter, we have dozens of different tasks for generating hypotheses, then confirming or refuting them. You can probably evaluate them in the hundreds.
What about failed experiments?
Case with parking. Now navigators are set up just to bring you to your final destination and do not help, for example, find a free parking space for a car. We decided to gradually come closer to this. First, we added a layer of traffic jams on which we showed where you can stand, then we added prices for parking lots, then payment for parking lots.
We wanted to make the product as profitable for the user as possible. But the task of showing free parking spaces was not easy for us. We tested various hypotheses for tracking parking spaces. But all the hypotheses were not confirmed. Now we decided to use this service in a micro route. Micro route - building a small circular route around the destination point so that the user does not go astray and finds a parking lot.
An unsuccessful case for us is the experiment in which, when analyzing metrics, we notice that people more often began to climb into the settings in order to disable a new feature. This happened when we replaced the familiar interface with another, more designed, in our opinion. As a result, people began to use the navigator less.
After testing, we made sure that the most important information should be shown in one place, and not duplicated throughout the product.
What approaches do you use in predictive analytics?
We have an offer that is based on the user's past trips and offers automatically saved routes. This is machine learning in the product.
We also use the classic forecasting model. We distinguish components from user behavior - this is seasonality, time, including a look at the weekly cut, we take into account the criteria of the region and conduct the trend. Summing up all this, we get a forecast for the future.
Since we have two regions - Russia and Turkey, the frequency of use of the product varies. For example, in Russia the audience grows on weekdays, and in Turkey on the weekend. We also have forecasts of both important KPI metrics and those that could potentially become KPI metrics.
We use forecast data not only in analytics, but also in development. When we predict traffic jams by the time of the route, the trend for the next hour and a half is taken into account. This development was introduced through experiments, working with the audience.
For us, one of the main metrics is quality. Accordingly, we consider the relative error of this indicator and look at how the indicator has changed after the introduction of the traffic forecast. We also take it into account under various conditions. For example, the history of product use during rush hour on the road or use on growing traffic jams. This is important because the situation on the roads changes very quickly, and the navigator should be able to take this into account.
When we look at a data slice, we take into account statistics not only for the whole day, but also take into account each individual slice, which can be very different from the positive dynamics of a whole day.
What is the Yandex.Navigator monetization model?
Now geoservices at Yandex are separate business units. Therefore, we are trying to find good forms of monetization for Yandex.Navigator. We conducted several experiments, launched pins along the route a couple of years ago - this is the most light format, it talks well about the location in which your business is located.
We are also looking for different formats, how you can natively help the advertiser reach the motorist inside the navigator. In addition, we use advertising formats such as billboards that hang in the product interface when following a route.
We have special projects that create some emotion in a person. Together with the advertiser, we are creating a message about what a person is cool to do in the near future. For example, we had special projects using additional voices in the navigator. Or, for example, by the centennial of Porshe, we replaced the cursor arrow that leads you along the route with the Porsche model.
We are trying to usefully suggest where a person should go. For example, now we are testing this format when the user has built a route, and we know that there is a MacAuto right on the road. If the route time is not very different, then we will offer him, with the help of one button, to add Makavto to the script.
This is a cool format. Firstly, because he has high conversions, and secondly, because he is super contextual.
What resources for self-education do you read?
Things that expand my professional horizons and are not directly related to the area of work help me. For example, the company IDEO, which are the main creators of research on design thinking in the world. They have a guide where they list what research they use with the Design Kit cases.
This is very similar to the programming approach when new algorithms are invented. They are invented by borrowing from things already existing in life.
More reports on product marketing can be found on the @epicgrowth Telegram channel .
Epic talks is an Epic Growth project where we communicate with teams about well-established processes, testing product hypotheses, analytics and much more. The project was shot with the support of the analytical platform AppMetrica .
Command
Lyosha Gozhev, senior product analyst and Misha Vysokovsky, head of Yandex.Navigator
How is the product team of Yandex.Navigator arranged?
Inside Navigator, we have several product teams, each of which has its own focus. Focus sets the direction of product development. For example, one team is guiding the route, the other focuses on the search script. Unlike a formed product team, focus may change over time. Typically, each team has a product manager, designer, and developer.
We now have two analysts, so they work for several product teams. Mostly the tasks of the analyst are divided into two types.
The first type is adhoc tasks that need to be quickly counted, or tasks that require verification of coding. The second type is research tasks that the whole team needs to dive into. In such tasks there is no separation between the contractor and the customer. The whole process of research tasks is very similar to traversing a growing tree: we generate hypotheses, and then test them. During their refutation or confirmation, new hypotheses arise.
Product Experiments
Give an example of successful experiments?
We try to guess where a person will go and offer him a “destination suggest”. For example, the user opens the navigator, and the application offers to go along the most frequent routes. Now we already have about 13% of the routes so constructed. This indicator is constantly growing. We are engaged in improving the predictive model, see how it affects and how much more convenient it becomes for the user.
Another example of a successful experiment is the redesign of the interface using ordinary electrical tape. Since we have a specific product, testing a navigator sitting at a table is a waste of time. At some point, we looked at the interface of Y. Navigator and realized that it had a lot of duplicate information.
We thought: “Listen, the third screen above is a huge black panel. Maybe we should throw it away and without it everything will be fine? ” The easiest option is to take a regular electrical tape and glue the top plate.
The first feedback that appeared was "I understand where to go, but it is not clear when I will come." We took the scissors and cut it a bit so that the arrival time was visible. We were surprised by the positive feedback. People calmly got to a place without information that was sealed with black electrical tape, while driving along an unfamiliar route.
How many hypotheses are tested per week?
We used to use the hypothesis testing approach every Monday from two to four hours. The goal was to make it a habit for product managers to test some new hypothesis, test a new solution, prepare a prototype, and conduct an in-depth interview. After we realized that it was common for everyone, we changed the system.
We did this in the form of an event on Friday, where everyone shared their impressions about testing the hypotheses. We brought in products not only from the Navigator, but also from Geoservices. We compiled a rating of employees where everyone received stars for their research.
This was after the first small team took a course in design thinking. It was an online course, but with an offline part that required the entire team to complete tasks. When we realized that this approach was popular within the company, we began to conduct a course for product managers from Geoservices on our own. We have successfully implemented this initiative, and in the future it helped the internal growth of employees.
We conduct all analytical hypotheses in the task management system. Therefore, this helps us analyze their number and apply in the future the calculations that were carried out for research. On average, in a quarter, we have dozens of different tasks for generating hypotheses, then confirming or refuting them. You can probably evaluate them in the hundreds.
What about failed experiments?
Case with parking. Now navigators are set up just to bring you to your final destination and do not help, for example, find a free parking space for a car. We decided to gradually come closer to this. First, we added a layer of traffic jams on which we showed where you can stand, then we added prices for parking lots, then payment for parking lots.
We wanted to make the product as profitable for the user as possible. But the task of showing free parking spaces was not easy for us. We tested various hypotheses for tracking parking spaces. But all the hypotheses were not confirmed. Now we decided to use this service in a micro route. Micro route - building a small circular route around the destination point so that the user does not go astray and finds a parking lot.
An unsuccessful case for us is the experiment in which, when analyzing metrics, we notice that people more often began to climb into the settings in order to disable a new feature. This happened when we replaced the familiar interface with another, more designed, in our opinion. As a result, people began to use the navigator less.
After testing, we made sure that the most important information should be shown in one place, and not duplicated throughout the product.
Analytics
What approaches do you use in predictive analytics?
We have an offer that is based on the user's past trips and offers automatically saved routes. This is machine learning in the product.
We also use the classic forecasting model. We distinguish components from user behavior - this is seasonality, time, including a look at the weekly cut, we take into account the criteria of the region and conduct the trend. Summing up all this, we get a forecast for the future.
Since we have two regions - Russia and Turkey, the frequency of use of the product varies. For example, in Russia the audience grows on weekdays, and in Turkey on the weekend. We also have forecasts of both important KPI metrics and those that could potentially become KPI metrics.
We use forecast data not only in analytics, but also in development. When we predict traffic jams by the time of the route, the trend for the next hour and a half is taken into account. This development was introduced through experiments, working with the audience.
For us, one of the main metrics is quality. Accordingly, we consider the relative error of this indicator and look at how the indicator has changed after the introduction of the traffic forecast. We also take it into account under various conditions. For example, the history of product use during rush hour on the road or use on growing traffic jams. This is important because the situation on the roads changes very quickly, and the navigator should be able to take this into account.
When we look at a data slice, we take into account statistics not only for the whole day, but also take into account each individual slice, which can be very different from the positive dynamics of a whole day.
Monetization
What is the Yandex.Navigator monetization model?
Now geoservices at Yandex are separate business units. Therefore, we are trying to find good forms of monetization for Yandex.Navigator. We conducted several experiments, launched pins along the route a couple of years ago - this is the most light format, it talks well about the location in which your business is located.
We are also looking for different formats, how you can natively help the advertiser reach the motorist inside the navigator. In addition, we use advertising formats such as billboards that hang in the product interface when following a route.
We have special projects that create some emotion in a person. Together with the advertiser, we are creating a message about what a person is cool to do in the near future. For example, we had special projects using additional voices in the navigator. Or, for example, by the centennial of Porshe, we replaced the cursor arrow that leads you along the route with the Porsche model.
We are trying to usefully suggest where a person should go. For example, now we are testing this format when the user has built a route, and we know that there is a MacAuto right on the road. If the route time is not very different, then we will offer him, with the help of one button, to add Makavto to the script.
This is a cool format. Firstly, because he has high conversions, and secondly, because he is super contextual.
About education
What resources for self-education do you read?
Things that expand my professional horizons and are not directly related to the area of work help me. For example, the company IDEO, which are the main creators of research on design thinking in the world. They have a guide where they list what research they use with the Design Kit cases.
This is very similar to the programming approach when new algorithms are invented. They are invented by borrowing from things already existing in life.
More reports on product marketing can be found on the @epicgrowth Telegram channel .