August 20, 2014 at 12:15Measure seven times, cut once: how not to get confused in the metrics of the product, process and team happiness
Today my goal is to briefly talk about the approaches of data-informed product management, which I profess and try to interest you in using its basic tools in your products.
Short disclaimer - I came to product development from project management to outsource. It came as a surprise to me that while close attention is paid to product metrics, process and team often often undeservedly fade into the background.
For myself, I formulated that measuring the success of a product consists of three blocks:
- user happiness;
- success (qualitative and quantitative) of iterations and releases;
- the happiness of the team.
Often we try to measure it in such detail and on so many sides that the analysis of the information received turns into real torture.
Therefore, in the projects, we tried to use the lightweight HEART framework and the Goal-Signals-Metrics approach, which the guys from Google came up with and which helps to focus on really important things for the UX product.
Happiness
Metrics of happiness include, for example:
- user satisfaction;
- the feeling that the product is easy to use;
- net promoter score.
Engagement
For example:
- the number of user visits per week;
- the number of photos uploaded by the user per day;
- the number of likes and shares.
Adoption (Acceptance) Acceptance
can include:
- Updates to the new version;
- user-created subscriptions;
- purchases made by new users in the application.
Retention The
specific metrics here may be:
- the number of users who remain active over time
- churn
- repeated purchases
Task Success The key
tasks may, for example, be:
- successful searches;
- photo upload time;
- a profile completely filled by the user.
You do not have to try to come up with metrics that are important to your product in all of these categories. HEART is about choosing 1 or 2 really important ones and focusing on them for some period.
For example, in an enterprise product that users must use one way or another every day, engagement may not be that important. But the happiness and success of key tasks is what you should concentrate on at least at the starting stage.
HEART makes it easy to “convert” your important categories into meaningful metrics that you can directly track.
All you need to do is not start with brainstorming interesting pieces that you would like to track in the project or which allows you to track the new tricky product analytics system. You need to start with higher-level things: decide what goals your project is pursuing.
No matter how captainly it sounds, sometimes it is not so simple to formulate product goals, with which the whole team unanimously agrees, and HEART helps with this well. It is also important here not to make a typical mistake - not to formulate goals in terms of existing metrics.
For example, one of our goals in my current product is to interest users with recommendations and make them spend more time studying them.
Often, several signals can potentially be suitable for one target, so at the stage of their determination it is useful to analyze and understand which ones are easier or harder to track, which are more sensitive to changes in your product and determine the optimal one for your purpose.
The signal in this case is the amount of time that users interact with recommendations. Metric is the number of minutes per user spent studying recommendations.
The described process helps to naturally prioritize metrics and get rid of excess noise in your measurements. This is a kind of Occam's razor, which from the side looks very trivial, but often leads to unexpected conclusions and actions.
But product measurements are not everything. Although HEART gives a lot to understand how to make a product better for our users, for long-term successful work we lack an understanding of whether everything is fine with the process and the team.
Often in products I come across the fact that these issues are hushed up, because everything seems to be pretty good, people have eyes burned, we are doing a cool thing here, and indeed it’s all some kind of old-fashionedness.
But entropy tends to accumulate in any systems. And if today you deliver a cool product to the market, but don’t think about why each of your releases takes a little longer than the previous one, and people started to leave work a little earlier, a time bomb may ripen in your project.
Another terrible triviality that everyone knows about, but which is very easy to ignore: time-to-market, the speed of reaction to the market and the actions of competitors are critically important for the success of products.
Lack of attention to how your technical duty is developing, how successful your iterations are, what strategic problems are accumulating in your process, lead to the fact that your projects turn into an attempt to jump into a leaving tram.
My little bootstrap of process metrics has three parts. They are easy to measure, and if you are not already doing this, I urge you to try to think about your own set of metrics that reflect the transparency and health of your process. Specific examples of metrics may not correspond to what is important to measure in your project, so be careful with the literal use of this block.
Quantitative Release Metrics
It's simple. To understand how you are doing, just measure the speed vs the quality of your work.
The speed metric can be directly the team speed or a composite metric: user stories made during this iteration / not completed, but planned / transferred to the next iteration.
The metrics that reflect whether you compromise quality to the detriment of speed can be:
- the number of defects found during the iteration / vs that made it to the production as a result
- the distribution of defects by type / component and source
. Moreover, such a thing helps to catch technological risks early on. project.
I happened to work in a project where the external component used by the team became the main source of the defect. At that moment when it was discovered, he was just an annoying nuisance, and they decided to get rid of him, so as not to spoil the statistics.
About six months later, it was rather sad to watch the neighboring teams, who began in a hurry to abandon this component, which by that time had turned for many into a real blocker for the further development of products.
Qualitative release metrics
- the ratio of the number of user stories made to the total number of stories in the project;
- deviation from the planned release date;
- technical debt.
Surely many of you have come across projects that are said to be “yes, it's easier to write from scratch than to support.” This is often due to technical debt, which has expanded to incredible proportions. If you do not know about him and do not manage him today, it is possible that tomorrow he will hurt you. This seems amazingly obvious in engineering-centered companies, but I have seen so many cases where the business and the product dominate healthy engineering practices so much that it was impossible not to mention this point.
And finally, quality metrics.
First of all, this is a simple list of what was good and worth improving. It is important to say that this is not just a metric that needs to be tracked. This is a process of continuous improvement that you need to try to implement on your project. If we do not constantly try to become better, then gradually we become a little worse than before. At least because the world around us does not stand still.
The list of things worth improving can include absolutely any thing - from adding new guides to your code to buying comfortable chairs for the team. And this is not easy - they got together, talked and parted - but a plan of specific actions that need to be done to improve.
And about the last, most important, in my opinion, part of an excellent product.
A simple test - do you know what makes every person in your team come to work every morning? Do you know what makes you grit your teeth and think “how did they get me, once again I’ll go write a statement”?
If you, like me, think that without a motivated and satisfied team a product cannot become completely successful in the long run, then it's time to deal with these issues.
Every time I say such things, people start asking, “Well, how do you know such intimate things? What if my colleague wants to leave our large company in those startups or save up money and move to Thailand? How will he tell me about this? ”
But you really do not want to know about it at the moment when he puts the statement on the table? Therefore, I suggest starting with the questionnaire developed by Max Landsberg and which allows you to get a bunch of important information about what drives and upsets your team, and further understand what to do about it. And be sure to read Stas Davydov's Motivate Me Right , if for some reason you haven't done it yet.
In any product at different stages of development, we strive to get the maximum feedback from users. It serves as a kind of guiding light and will often help to understand whether we are on the right path.
Personal professional feedback within the team is the same guiding star. If you do not regularly receive feedback about your work from your team and do not give it to those people with whom you create a product together, problems may accumulate in your product that you do not even realize.
At one of the projects, I happened to observe the departure of a talented developer.
It's funny, but none of the management learned about it for 2 years that this guy really needed professional recognition, was very kind to speaking at conferences, mentoring newcomers, and participating in open-source projects. Participation in conferences was postponed all the time, sometimes deadlines in projects, sometimes not enough money to go to Java Day somewhere in the states, and after 2 years he could not stand it and went to work in a large tech company as an evangelist.
I am sure this developer had something to say to his managers as a feedback. And such examples are the sea. The feedback helps to receive and give information important for the development and maintenance of the team, and it should not be neglected. You can go even further and try to collect feedback on the level of several teams or a large project. Readas Valve does , it’s not at all as difficult as it might seem.
1. When working with a product, remember that data is just a tool. Focus on goals, not numbers in statistics.
2. Do not forget that your product also depends on the health of your processes. They deserve your love and attention.
3. The team deserves them even more. And even if in your company hr or a functional manager deals with the things I described, you should be the person who really cares that you are doing well on this front.
Short disclaimer - I came to product development from project management to outsource. It came as a surprise to me that while close attention is paid to product metrics, process and team often often undeservedly fade into the background.
For myself, I formulated that measuring the success of a product consists of three blocks:
- user happiness;
- success (qualitative and quantitative) of iterations and releases;
- the happiness of the team.
User Happiness
Often we try to measure it in such detail and on so many sides that the analysis of the information received turns into real torture.
Therefore, in the projects, we tried to use the lightweight HEART framework and the Goal-Signals-Metrics approach, which the guys from Google came up with and which helps to focus on really important things for the UX product.
HEART divides all metrics into 5 categories
Happiness
Metrics of happiness include, for example:
- user satisfaction;
- the feeling that the product is easy to use;
- net promoter score.
Engagement
For example:
- the number of user visits per week;
- the number of photos uploaded by the user per day;
- the number of likes and shares.
Adoption (Acceptance) Acceptance
can include:
- Updates to the new version;
- user-created subscriptions;
- purchases made by new users in the application.
Retention The
specific metrics here may be:
- the number of users who remain active over time
- churn
- repeated purchases
Task Success The key
tasks may, for example, be:
- successful searches;
- photo upload time;
- a profile completely filled by the user.
You do not have to try to come up with metrics that are important to your product in all of these categories. HEART is about choosing 1 or 2 really important ones and focusing on them for some period.
For example, in an enterprise product that users must use one way or another every day, engagement may not be that important. But the happiness and success of key tasks is what you should concentrate on at least at the starting stage.
HEART makes it easy to “convert” your important categories into meaningful metrics that you can directly track.
All you need to do is not start with brainstorming interesting pieces that you would like to track in the project or which allows you to track the new tricky product analytics system. You need to start with higher-level things: decide what goals your project is pursuing.
Goals-signals-metrics
No matter how captainly it sounds, sometimes it is not so simple to formulate product goals, with which the whole team unanimously agrees, and HEART helps with this well. It is also important here not to make a typical mistake - not to formulate goals in terms of existing metrics.
For example, one of our goals in my current product is to interest users with recommendations and make them spend more time studying them.
Often, several signals can potentially be suitable for one target, so at the stage of their determination it is useful to analyze and understand which ones are easier or harder to track, which are more sensitive to changes in your product and determine the optimal one for your purpose.
The signal in this case is the amount of time that users interact with recommendations. Metric is the number of minutes per user spent studying recommendations.
The described process helps to naturally prioritize metrics and get rid of excess noise in your measurements. This is a kind of Occam's razor, which from the side looks very trivial, but often leads to unexpected conclusions and actions.
But what about the process?
But product measurements are not everything. Although HEART gives a lot to understand how to make a product better for our users, for long-term successful work we lack an understanding of whether everything is fine with the process and the team.
Often in products I come across the fact that these issues are hushed up, because everything seems to be pretty good, people have eyes burned, we are doing a cool thing here, and indeed it’s all some kind of old-fashionedness.
But entropy tends to accumulate in any systems. And if today you deliver a cool product to the market, but don’t think about why each of your releases takes a little longer than the previous one, and people started to leave work a little earlier, a time bomb may ripen in your project.
Another terrible triviality that everyone knows about, but which is very easy to ignore: time-to-market, the speed of reaction to the market and the actions of competitors are critically important for the success of products.
Lack of attention to how your technical duty is developing, how successful your iterations are, what strategic problems are accumulating in your process, lead to the fact that your projects turn into an attempt to jump into a leaving tram.
My little bootstrap of process metrics has three parts. They are easy to measure, and if you are not already doing this, I urge you to try to think about your own set of metrics that reflect the transparency and health of your process. Specific examples of metrics may not correspond to what is important to measure in your project, so be careful with the literal use of this block.
Quantitative Release Metrics
It's simple. To understand how you are doing, just measure the speed vs the quality of your work.
The speed metric can be directly the team speed or a composite metric: user stories made during this iteration / not completed, but planned / transferred to the next iteration.
The metrics that reflect whether you compromise quality to the detriment of speed can be:
- the number of defects found during the iteration / vs that made it to the production as a result
- the distribution of defects by type / component and source
. Moreover, such a thing helps to catch technological risks early on. project.
I happened to work in a project where the external component used by the team became the main source of the defect. At that moment when it was discovered, he was just an annoying nuisance, and they decided to get rid of him, so as not to spoil the statistics.
About six months later, it was rather sad to watch the neighboring teams, who began in a hurry to abandon this component, which by that time had turned for many into a real blocker for the further development of products.
Qualitative release metrics
- the ratio of the number of user stories made to the total number of stories in the project;
- deviation from the planned release date;
- technical debt.
Surely many of you have come across projects that are said to be “yes, it's easier to write from scratch than to support.” This is often due to technical debt, which has expanded to incredible proportions. If you do not know about him and do not manage him today, it is possible that tomorrow he will hurt you. This seems amazingly obvious in engineering-centered companies, but I have seen so many cases where the business and the product dominate healthy engineering practices so much that it was impossible not to mention this point.
And finally, quality metrics.
First of all, this is a simple list of what was good and worth improving. It is important to say that this is not just a metric that needs to be tracked. This is a process of continuous improvement that you need to try to implement on your project. If we do not constantly try to become better, then gradually we become a little worse than before. At least because the world around us does not stand still.
The list of things worth improving can include absolutely any thing - from adding new guides to your code to buying comfortable chairs for the team. And this is not easy - they got together, talked and parted - but a plan of specific actions that need to be done to improve.
Team is the king
And about the last, most important, in my opinion, part of an excellent product.
A simple test - do you know what makes every person in your team come to work every morning? Do you know what makes you grit your teeth and think “how did they get me, once again I’ll go write a statement”?
If you, like me, think that without a motivated and satisfied team a product cannot become completely successful in the long run, then it's time to deal with these issues.
Every time I say such things, people start asking, “Well, how do you know such intimate things? What if my colleague wants to leave our large company in those startups or save up money and move to Thailand? How will he tell me about this? ”
But you really do not want to know about it at the moment when he puts the statement on the table? Therefore, I suggest starting with the questionnaire developed by Max Landsberg and which allows you to get a bunch of important information about what drives and upsets your team, and further understand what to do about it. And be sure to read Stas Davydov's Motivate Me Right , if for some reason you haven't done it yet.
Feedback
In any product at different stages of development, we strive to get the maximum feedback from users. It serves as a kind of guiding light and will often help to understand whether we are on the right path.
Personal professional feedback within the team is the same guiding star. If you do not regularly receive feedback about your work from your team and do not give it to those people with whom you create a product together, problems may accumulate in your product that you do not even realize.
At one of the projects, I happened to observe the departure of a talented developer.
It's funny, but none of the management learned about it for 2 years that this guy really needed professional recognition, was very kind to speaking at conferences, mentoring newcomers, and participating in open-source projects. Participation in conferences was postponed all the time, sometimes deadlines in projects, sometimes not enough money to go to Java Day somewhere in the states, and after 2 years he could not stand it and went to work in a large tech company as an evangelist.
I am sure this developer had something to say to his managers as a feedback. And such examples are the sea. The feedback helps to receive and give information important for the development and maintenance of the team, and it should not be neglected. You can go even further and try to collect feedback on the level of several teams or a large project. Readas Valve does , it’s not at all as difficult as it might seem.