June 24, 2019 at 10:42Blitz with Ilya Krasinsky: how to shoot bad hypotheses, why dismiss a product and how to grow for a minimum of action?
Rick.ai CEO Ilya Krasinsky answered questions posed by product managers in Q&A format at the Epic Growth Conference.
See the decryption under the cut.
Any feature always lowers metrics. We automate a particular scenario of human behavior. If the percentage of people who understood the benefits of the feature and took advantage of it is practically zero, then the value of the feature will be minimal.
There are disadvantages after launching a new feature in the product: the database code grows, new bugs and defects appear, users are given a more difficult first session and activation is more difficult.
There is an assessment technique. You take a stream of users who will use this feature. So you can evaluate how much the conversion and revenue from a paid user will change. Next, you can calculate what kind of cash flow a new feature can bring to you.
Very often a company’s development is hindered by a lack of funding. In this case, the business becomes for you not an asset, but a suitcase without a handle, which is a pity to throw, but it’s hard to drag.
Accordingly, in this situation, squeeze the maximum until you shoot this “dead horse”, because next time you will need two to five years to get closer to your current point of development. Determine what skills and experience you can still bring to yourself and do not be afraid to get rid of ballast.
The main problems of current analytics:
A large number of companies already have a Big Data division. The vast majority of these units are engaged in data storage, in the smaller - the compilation of algorithms. There is usually a technological gap between top managers, product managers, and data science analysts. Business guys often do not understand what question to ask analytics.
Basically, these data are not consistent, that is, they already contain errors at the collection stage.
The analytics is very fragile, it is very easy to break. Accordingly, the key question is whether you have a built-in monitoring system for robotic analytics.
The main barrier to the transition to robotic analytics is that the data does not accumulate in the systems you use and, therefore, give incorrect numbers. Therefore, any conclusions and managerial decisions will also be a lie.
Until this problem is solved at the level of integration and data flow, everyone will probably be sawing their own Death Star, believing that it works. I have already saw five such systems in my life, and each time the developers found bugs and defects in them.
Two types of machine learning need to be distinguished:
The latter type is much more used. But, unfortunately, in practice it is very poorly implemented. These presentations do not translate well into a working product. Accordingly, the main problem of machine learning is that people see a black box at the exit.
I believe that for a long time people cannot process such flows of information. I see what we're all going this way: either there will be black boxes, like the attribution models that Google does, or some kind of system that will analyze the data and explain to the person how she analyzed what is in this section (company , domain, conversion).
You already have them: "Google spreadsheets" or Excel.
Most hypotheses cannot change a single metric, cannot do anything good to the user, they need to be shot. And out of 50 hypotheses, if you leave seven, you have a chance of success.
In 2019, it is clear that a person still considers worse than a calculator. But it seems that a person knows how to come up with non-standard ideas.
The easiest way: talk to ten industry professionals in a networking format at the conference. You will receive a list of fifty questions. Leave the questions that you like and you will get some kind of framework.
How is it in our team:
- A person must have a high level of energy. If a person is low in energy, then the whole team will be toxic.
- A person must be systemic and with experience of reflection. Developing a systematic skill is very expensive and time consuming. It is checked quite simply: ask a person about his previous experience, including negative, and what conclusion he made from this experience.
About 50% of people say, “Thank you, great question! I'll go think about it. ” This means that over the last year, when this situation occurred, they did not do this work. They have no such habit.
- A man should not be afraid. In the course of work, a large number of decisions must be made; most likely, the product will be wrong. It is important that he is not afraid to do this.
Trigger analysis. You take the user segment, see all user sessions and the chain of events. Divide people into two groups: those who are in retargeting, and those who are not.
In practice, you need to understand that we never have the task of measuring something accurately. Often this is simply pointless. If your investment in retargeting is less than the amount of work that I just described, then the work of analyzing retargeting will be more expensive than just doing it.
You need an accurate attribution model. Let's look at the concepts: it is not necessary to accurately attribute this or that income to any advertising campaign. We have only four management decisions:
- If his model of the world does not correspond much to reality.
- If his hypotheses are weak and poorly correlated with our users.
- If you do not like to communicate with users.
- If you do not like to make corridors, custom houses.
- If you do not test your hypotheses.
- If using an irrelevant toolkit.
- This means that he will be very mistaken in the conclusions, does not want to learn to do correctly and just does not follow the latest frameworks that occur in the industry, which means he has lagged behind.
Aggravation occurs at the end of the year. At the end of the year, people remember the goal they set for themselves.
The meaning is - you need to be able to lose. I need to repeat myself: there were a lot of experiments, so I just did not take into account and did not understand something. We changed the unit economy, changed the approach, raised the conversion, but this does not mean that the project will be all right.
Encourage support and care within the team. One of the skills that I’m developing in myself now: how to explain to the product manager, designer, marketer, analyst that they did everything wrong, but at the same time so that they don’t give up and they go to work the next day with the words: “Ok, the seventh time we will redo everything, and we will succeed.”
I believe that there are a lot of them. For example, Ultimate Guitar, Skyeng, RealtimeBoard. Behind the success of such companies are not only the first persons who are in the public eye, but also the performers who are doing tremendous work every day.
It’s cool to be friends with them. It's just a free train where you get new ideas, books and frameworks. Therefore, to surround myself with such a list of people, it seems to me, is one of the important tasks.
More narrow-profile product skills practices at Epic Workshop Day .
See the decryption under the cut.
What is an effective technique for assessing the prospects of features?
Any feature always lowers metrics. We automate a particular scenario of human behavior. If the percentage of people who understood the benefits of the feature and took advantage of it is practically zero, then the value of the feature will be minimal.
There are disadvantages after launching a new feature in the product: the database code grows, new bugs and defects appear, users are given a more difficult first session and activation is more difficult.
There is an assessment technique. You take a stream of users who will use this feature. So you can evaluate how much the conversion and revenue from a paid user will change. Next, you can calculate what kind of cash flow a new feature can bring to you.
At what point should I say “enough is enough”? Or do you have to jump and try until the lack of finance stops my startup?
Very often a company’s development is hindered by a lack of funding. In this case, the business becomes for you not an asset, but a suitcase without a handle, which is a pity to throw, but it’s hard to drag.
Accordingly, in this situation, squeeze the maximum until you shoot this “dead horse”, because next time you will need two to five years to get closer to your current point of development. Determine what skills and experience you can still bring to yourself and do not be afraid to get rid of ballast.
When, in your opinion, more than 50% of companies (at least IT) will switch to robotic analytics by analogy with Rick.ai? What are the main barriers at the moment?
The main problems of current analytics:
A large number of companies already have a Big Data division. The vast majority of these units are engaged in data storage, in the smaller - the compilation of algorithms. There is usually a technological gap between top managers, product managers, and data science analysts. Business guys often do not understand what question to ask analytics.
Basically, these data are not consistent, that is, they already contain errors at the collection stage.
The analytics is very fragile, it is very easy to break. Accordingly, the key question is whether you have a built-in monitoring system for robotic analytics.
The main barrier to the transition to robotic analytics is that the data does not accumulate in the systems you use and, therefore, give incorrect numbers. Therefore, any conclusions and managerial decisions will also be a lie.
Until this problem is solved at the level of integration and data flow, everyone will probably be sawing their own Death Star, believing that it works. I have already saw five such systems in my life, and each time the developers found bugs and defects in them.
My advice: duplicate the data so that you have different analytics systems and that you can verify the numbers with each other. One system is a very unreliable thing, errors happen very easily.
What are the prospects for ML in predictive analytics?
Two types of machine learning need to be distinguished:
- Compiled in the Python programming language.
- Composed using a PowerPoint presentation.
The latter type is much more used. But, unfortunately, in practice it is very poorly implemented. These presentations do not translate well into a working product. Accordingly, the main problem of machine learning is that people see a black box at the exit.
I believe that for a long time people cannot process such flows of information. I see what we're all going this way: either there will be black boxes, like the attribution models that Google does, or some kind of system that will analyze the data and explain to the person how she analyzed what is in this section (company , domain, conversion).
How likely are tools to test hypotheses before implementing changes to the product?
You already have them: "Google spreadsheets" or Excel.
Most hypotheses cannot change a single metric, cannot do anything good to the user, they need to be shot. And out of 50 hypotheses, if you leave seven, you have a chance of success.
In 2019, it is clear that a person still considers worse than a calculator. But it seems that a person knows how to come up with non-standard ideas.
What questions to ask the product at the interview?
The easiest way: talk to ten industry professionals in a networking format at the conference. You will receive a list of fifty questions. Leave the questions that you like and you will get some kind of framework.
How is it in our team:
- A person must have a high level of energy. If a person is low in energy, then the whole team will be toxic.
- A person must be systemic and with experience of reflection. Developing a systematic skill is very expensive and time consuming. It is checked quite simply: ask a person about his previous experience, including negative, and what conclusion he made from this experience.
About 50% of people say, “Thank you, great question! I'll go think about it. ” This means that over the last year, when this situation occurred, they did not do this work. They have no such habit.
- A man should not be afraid. In the course of work, a large number of decisions must be made; most likely, the product will be wrong. It is important that he is not afraid to do this.
How to measure the incremental effect of retargeting?
Trigger analysis. You take the user segment, see all user sessions and the chain of events. Divide people into two groups: those who are in retargeting, and those who are not.
In practice, you need to understand that we never have the task of measuring something accurately. Often this is simply pointless. If your investment in retargeting is less than the amount of work that I just described, then the work of analyzing retargeting will be more expensive than just doing it.
You need an accurate attribution model. Let's look at the concepts: it is not necessary to accurately attribute this or that income to any advertising campaign. We have only four management decisions:
- disable the process, do not converge at all; we spend too much, get almost nothing;
- can be slightly modified;
- do not touch;
- to strengthen.
Why would you fire the product?
- If his model of the world does not correspond much to reality.
- If his hypotheses are weak and poorly correlated with our users.
- If you do not like to communicate with users.
- If you do not like to make corridors, custom houses.
- If you do not test your hypotheses.
- If using an irrelevant toolkit.
- This means that he will be very mistaken in the conclusions, does not want to learn to do correctly and just does not follow the latest frameworks that occur in the industry, which means he has lagged behind.
You conduct experiments, 95% of failures, few successes, constantly perishable and pain in the head. How to be
Aggravation occurs at the end of the year. At the end of the year, people remember the goal they set for themselves.
The meaning is - you need to be able to lose. I need to repeat myself: there were a lot of experiments, so I just did not take into account and did not understand something. We changed the unit economy, changed the approach, raised the conversion, but this does not mean that the project will be all right.
Encourage support and care within the team. One of the skills that I’m developing in myself now: how to explain to the product manager, designer, marketer, analyst that they did everything wrong, but at the same time so that they don’t give up and they go to work the next day with the words: “Ok, the seventh time we will redo everything, and we will succeed.”
The coolest food teams in Russia?
I believe that there are a lot of them. For example, Ultimate Guitar, Skyeng, RealtimeBoard. Behind the success of such companies are not only the first persons who are in the public eye, but also the performers who are doing tremendous work every day.
It’s cool to be friends with them. It's just a free train where you get new ideas, books and frameworks. Therefore, to surround myself with such a list of people, it seems to me, is one of the important tasks.
More narrow-profile product skills practices at Epic Workshop Day .