December 25, 2009 at 13:53Web analytics: analyze it! Part 6. Action
Part 1. Introduction.
Part 2. Data collection.
Part 3. Basic metrics.
Part 4. From statistics to analytics
Part 5. Divide and think
The most active web analytics evangelist Avinash Koshik in his blog pays considerable attention to topics related to the organization of work: how to convince superiors of the importance of analytics, and most importantly - how to make the company use the data. The fact that they contain information important for business seems to be beyond doubt, but even if management agrees with this, it usually ignores the conclusions of the analyst. As a result, measurements on the site are not accompanied by an analysis of changes in indicators, the site itself is done “by the eye of the boss”, and statistics are used only to draw beautiful graphs and general estimates such as monthly visits. Koshik calls it HiPPO, or “MSaVS” - the opinion of the highest paid employees. Given the fact that every employee has an opinion on the site’s account, the activist has a hard time.
In this situation it is very easy to make a mistake: for example, insisting on experimental measurements, getting the changes for the worse, and stop there, ashamed of the disappointed views of the authorities. Excessive directness will also hurt (you should not start by swearing at the director who insisted on the changes that led to the problem) and, of course, lying (if the experiment weakened the site’s performance, the worst thing you can do is falsify the results!). Stories about analytics-oriented companies (like Amazon) will not help the situation either: the usual answer in this case is “you understand, everything is different with us ...”. You need to start with the most striking and trouble-free steps. Of course, everything is much simpler if the boss is your ally, or you are your own director. In any case, whoever is warned is armed,
Before you rush into battle, you should make a plan. At its head, set the goals pursued by the site. At the same time, one should focus not on what is declared by the creators of the site or the management of the company, but what users can actually see on the site. If the goal of “informing users about new products” is declared, and information on them appears on the site with a six-month delay, the goal can be safely deleted. However, do not overwork: while you can safely list the goals.
The next step is to select numerical indicators for assessing the achievement of goals and create a chain from business requirements to measured website indicators.
For example, for an online store, a chain might look like this:
Creating such chains is very important. They are needed for two purposes: firstly, you can identify important metrics to be measured, and secondly, immediately establish their relationship with the goals of the Big bosses. Now you can speak his language!
When you drew this staircase (by the way, modify it as you work: who said that the first version should be good right away?), It was time to move on to the measurements. Run Google Analytics, look for familiar metrics, segment, think. At first, if you have little experience with Analytics, you will have to get used to it, configure it, use it until you become familiar with it, and most importantly, you can quickly and correctly take the necessary readings.
Then measure your metrics. Just measure, follow them, try to correlate changes in values to external circumstances. After you become this super-guru, the real work will begin: you will have to offer, monitor and test on the site in order to improve these indicators. Without this, all your work does not make sense: the goal of analytics is not to make beautiful reports with numbers and graphs, but to improve the site, increase profits, and make users happy. To do this, you need not only to be able to work with Analytics, but also to navigate in search promotion, interface development, web marketing - of course, here you can work in a team with other specialists.
All changes. which are made on the site, you need to check: they should improve the performance of the site. The beauty of web analytics is that you can check any change quickly and cheaply: just offer it to users and see what changes! In most cases, this is highly recommended: the scope for improvements usually lurks in the most unexpected places. Similarly, you can test other people's ideas (you can even try to introduce “strange” ideas from your bosses - but don’t be surprised if they work very well against your expectations!).
For testing, you can supplement the site engine or use third-party solutions. I recommend to master and use Google Website Optimizer. It allows you to conduct comparative testing of the old and new versions of pages, calculate metric changes and verify the accuracy of measurements.
Reliability, by the way, must be checked for any measurements related to comparing values. The fact is that we always deal with a limited sample, which means that the measured value may differ from the true one. The smaller the sample, the greater the possible deviation. Therefore, if the sample is too small, and the measured value has changed little, there is no certainty that it actually improved. How to be?
There is a fairly simple rule to determine the sufficiency of the sample. The actual value with a probability of 95% is in the range from X – 2√ (X) to X + 2√ (X) .
For example, we’re trying to figure out which of the two banner ads is more clickable. One showed 1,500 times, of which 70 clicked. Another appeared 700 times, 30 users clicked on it. It would seem that the CTR of the first is 4.7%, and the second is 4.2%. The first is better? Is not a fact.
We calculate the possible real number of clicks. 70–2√ (70) = 53, 70 + 2√ (70) = 87. Possible CTR - from 3.5% to 5.8%! It’s too early to be sure. By calculating confidence intervals in this way when increasing impressions and clicks, you will see that they will decrease until they stop overlapping. Only then will it be possible to say with confidence that one of the options is better! Probability theory has always been among the analyst’s best friends.
This ends the first part of my cycle. In it, I tried to collect the most important milestones for the analyst, without pretending to be a complete disclosure of the topic. The next part will most likely be more specific. However, I highly recommend that you continue to research this topic right now, and here's what I can recommend:
Part 2. Data collection.
Part 3. Basic metrics.
Part 4. From statistics to analytics
Part 5. Divide and think
Knockin 'on Heaven
The most active web analytics evangelist Avinash Koshik in his blog pays considerable attention to topics related to the organization of work: how to convince superiors of the importance of analytics, and most importantly - how to make the company use the data. The fact that they contain information important for business seems to be beyond doubt, but even if management agrees with this, it usually ignores the conclusions of the analyst. As a result, measurements on the site are not accompanied by an analysis of changes in indicators, the site itself is done “by the eye of the boss”, and statistics are used only to draw beautiful graphs and general estimates such as monthly visits. Koshik calls it HiPPO, or “MSaVS” - the opinion of the highest paid employees. Given the fact that every employee has an opinion on the site’s account, the activist has a hard time.In this situation it is very easy to make a mistake: for example, insisting on experimental measurements, getting the changes for the worse, and stop there, ashamed of the disappointed views of the authorities. Excessive directness will also hurt (you should not start by swearing at the director who insisted on the changes that led to the problem) and, of course, lying (if the experiment weakened the site’s performance, the worst thing you can do is falsify the results!). Stories about analytics-oriented companies (like Amazon) will not help the situation either: the usual answer in this case is “you understand, everything is different with us ...”. You need to start with the most striking and trouble-free steps. Of course, everything is much simpler if the boss is your ally, or you are your own director. In any case, whoever is warned is armed,
Paris capture plan
Before you rush into battle, you should make a plan. At its head, set the goals pursued by the site. At the same time, one should focus not on what is declared by the creators of the site or the management of the company, but what users can actually see on the site. If the goal of “informing users about new products” is declared, and information on them appears on the site with a six-month delay, the goal can be safely deleted. However, do not overwork: while you can safely list the goals.
The next step is to select numerical indicators for assessing the achievement of goals and create a chain from business requirements to measured website indicators.
For example, for an online store, a chain might look like this:
For a social network site, the goal can be “more registered users”, and the corresponding metrics can be traffic and the percentage of registered users.
Creating such chains is very important. They are needed for two purposes: firstly, you can identify important metrics to be measured, and secondly, immediately establish their relationship with the goals of the Big bosses. Now you can speak his language!
To battle!
When you drew this staircase (by the way, modify it as you work: who said that the first version should be good right away?), It was time to move on to the measurements. Run Google Analytics, look for familiar metrics, segment, think. At first, if you have little experience with Analytics, you will have to get used to it, configure it, use it until you become familiar with it, and most importantly, you can quickly and correctly take the necessary readings.
Then measure your metrics. Just measure, follow them, try to correlate changes in values to external circumstances. After you become this super-guru, the real work will begin: you will have to offer, monitor and test on the site in order to improve these indicators. Without this, all your work does not make sense: the goal of analytics is not to make beautiful reports with numbers and graphs, but to improve the site, increase profits, and make users happy. To do this, you need not only to be able to work with Analytics, but also to navigate in search promotion, interface development, web marketing - of course, here you can work in a team with other specialists.
Check don't trust
All changes. which are made on the site, you need to check: they should improve the performance of the site. The beauty of web analytics is that you can check any change quickly and cheaply: just offer it to users and see what changes! In most cases, this is highly recommended: the scope for improvements usually lurks in the most unexpected places. Similarly, you can test other people's ideas (you can even try to introduce “strange” ideas from your bosses - but don’t be surprised if they work very well against your expectations!).
For testing, you can supplement the site engine or use third-party solutions. I recommend to master and use Google Website Optimizer. It allows you to conduct comparative testing of the old and new versions of pages, calculate metric changes and verify the accuracy of measurements.
Reliability, by the way, must be checked for any measurements related to comparing values. The fact is that we always deal with a limited sample, which means that the measured value may differ from the true one. The smaller the sample, the greater the possible deviation. Therefore, if the sample is too small, and the measured value has changed little, there is no certainty that it actually improved. How to be?
There is a fairly simple rule to determine the sufficiency of the sample. The actual value with a probability of 95% is in the range from X – 2√ (X) to X + 2√ (X) .
For example, we’re trying to figure out which of the two banner ads is more clickable. One showed 1,500 times, of which 70 clicked. Another appeared 700 times, 30 users clicked on it. It would seem that the CTR of the first is 4.7%, and the second is 4.2%. The first is better? Is not a fact.
We calculate the possible real number of clicks. 70–2√ (70) = 53, 70 + 2√ (70) = 87. Possible CTR - from 3.5% to 5.8%! It’s too early to be sure. By calculating confidence intervals in this way when increasing impressions and clicks, you will see that they will decrease until they stop overlapping. Only then will it be possible to say with confidence that one of the options is better! Probability theory has always been among the analyst’s best friends.
The end?
This ends the first part of my cycle. In it, I tried to collect the most important milestones for the analyst, without pretending to be a complete disclosure of the topic. The next part will most likely be more specific. However, I highly recommend that you continue to research this topic right now, and here's what I can recommend:
Books
- Ashmanov’s book “Optimization and promotion of sites in search engines” . Direct search engine optimization should also be borne in mind when conducting analytical work (for example, you need to understand that adding variable markers to the page link actually changes its address). However, the book describes not only optimization for search engines. It is very systemic and helps to build a chain of conclusions in the head.
- Of course, here we should also recommend Avinash Koshik’s book on web analytics , but he has already published a second book in which he pays more attention to the organization of work and details. We are waiting for the translation into Russian.
- For those who can read English and communicate with Amazon (or with me ;) ), I recommend the book Landing Page Optimization: The Definitive Guide to Testing and Tuning for Conversions . It reveals many aspects of testing and gives many tips for improvements that you can and should try to make on the site.
Blogs and Resources
- Blog of Alexey Skobelev . Interesting and detailed articles on aspects of analytics. For example, here's an article explaining how to interpret Alexa.com data.
- Google Conversion University . Google Analytics training course with online end certification (paid). Very useful, even if you are not going to take a certificate. It helped me a lot. Unfortunately, only in English.
- Blog of Roman Zykov . The novel has extensive experience in analytical work (including at Ozon.ru) and writes, and sometimes talks at conferences, a lot of interesting things.