August 2, 2011 at 12:05Web analytics: Not all numbers are equally useful
We are constantly being asked: what is the error in collecting data in Google Analytics? Which counter is better to trust? Is it possible to get rid of all the discrepancies and get the exact numbers of attendance?
When we run an advertising campaign, we pay for each visitor. Web analytics data should tell us which of the ads are wasting a budget, which keywords are more important to us. We need to know the result as soon as possible! When will he be ready?
Common sense in solving this problem is not an assistant. The fact is that in the analysis of statistical indicators we are faced with random processes. Our brain is trying to imagine a uniform process: if the conversion on the site is 10%, it seems to us that every tenth will be converted:
Even if we understand that reality looks different, it is difficult to get rid of the illusion: “wait for a few conversions and get accurate data”. In fact, this process is much more chaotic, for example:
This means that with a limited number of visits, even having accurate data on the number of visitors, views, conversions, we cannot accurately predict future sales. There is no such amount of data that provides firm confidence in the calculated indicators: we can only roughly estimate the “real” percentage of conversion.
There is good news: we can estimate the accuracy with which we calculated this coefficient and realize whether this accuracy is enough for us, or if we need to wait for more data. I will not go into mathematical calculations, telling the theoretical part. I will give only two simple formulas by which it is easy to calculate the numbers you need.
The main concept of mathematical statistics that every analyst should learn is the confidence interval. This is the range in which the true value of the value we need can lie: for example, the conversion rate. The conversion value calculated in the usual way (for example, by Google Analytics) lies somewhere inside this interval; the true value of the conversion (which we learn only on endless traffic) is also most likely located inside it.
With an increase in the data sample, the bell narrows, and the probability that the true value is closer to the measured one increases. We choose what reliability is considered sufficient. Different confidence intervals can be constructed from the same curve. The chart shows two options (red and green).
Let's get down to business. Let's calculate the confidence interval for the conversion of one of the advertisements - “Buy an elephant at a discount!”. For example, in Google Analytics we see 143 visits and a 2.1% conversion for this ad. We want to compare this ad with another one (“Elephant with delivery in Moscow”) - it has 184 visits and 2.7% conversion - and choose the best one. Many marketers have already concluded that the second ad is better, but we need to check it mathematically.
Before starting the analysis, we make an important decision: what reliability will be sufficient. It is most convenient to use standard values that simplify calculations: these are 68%, 95% and 99.7% (they are called “one sigma”, “two sigma” and “three sigma”, respectively). I would like to count with maximum confidence (three sigma), but this extends the interval and often makes it completely uninformative.
The formula that helps us , is as follows:
,
wherein R - calculated by simply dividing the conversion ratio, N - the number of visits, and α - Sigma amount, i.e. measurement accuracy.
With an accuracy of 68%, the conversion of the first ad is in the range from 0.8% to 3.3% ; second ad conversion from 1.7% to 3.8% .
Discouraged, right? It is impossible to compare efficiency: intervals intersect. Both ads may be leaders. If we want to build intervals for a probability of 95%, they will be twice as wide. We need to wait for the data, and the trouble is that we don’t even know when these intervals “parted” and a leader emerges: it depends on the difference in the true conversion rates. Usually, for certainty, you need about a hundred conversions for each ad (word, campaign or other segment whose conversion we consider). Please note: our formula calculates well only when the conversion does not exceed 30%.
But what if our ad didn’t produce a single conversion? The toad begins to strangle us: let's turn off the ads and save the budget!
Zero conversions cannot be calculated according to the previous formula: it turns out that 0 is the exact result. This is not true. If we have 0 conversions and N visits, then the confidence interval is considered simple :
If we have a third ad (“Elephants as a match”) that resulted in 93 visitors without a single conversion, we can only say with certainty of one sigma that its conversion is below 1.2%. This announcement is worse than “Elephant with delivery”, but it is too early to talk about “Elephant at a discount”. Most people in this place are surprised, and some do not even believe: is everything really so unsteady? However, the experience of conducting large campaigns only confirms the random nature of conversions and the fact that at first the “zero” ad on the second hundred clicks can start to show a good result.
Why do not popular analytics tools consider these values for us? I can only speculate. This information will significantly complicate the reports, and most importantly, expose the unpleasant truth: the noticeable number of smart and accurate numbers in these reports are actually useless and even dangerous (if you rely on common sense). Having forgotten about the counting errors, you can stop an effective advertising campaign, considering it unsuccessful, choose a bad version of the advertisement, incorrectly distribute the advertising budget or make a mistake when developing an SEO strategy.
Confidence intervals are excellent for Google Website Optimizer, a tool for conducting and analyzing test results on websites. Optimizer draws conclusions only when it is convinced that the intervals have “diverged” and one of the page options is statistically significantly superior in conversion to other samples.
Many tools take this data into account “under the hood”: for example, Yandex.Direct draws conclusions about the announcement only when it collects enough data and is sure of the result. Of course, we would be interested to see them in the traffic reports, so as not to be mistaken in the estimates by eye. You can write a script for Excel that automatically calculates confidence intervals, but this requires constant data uploading. You can add this functionality to Metric or Analytics using the API of these systems and external scripts.
Are we forced to wait for hundreds of conversions for each ad to evaluate its performance? This would be an ideal solution, but usually impossible: there is not enough advertising budget. Try not to make premature fatal decisions.(remove or modify ads that have not proven to be insolvent) until statistically significant results are obtained; while nothing prevents, for example, temporarily reduce the cost of clicking on them.
It’s useful to add a record of intermediate goals : for example, placing the product in the basket is a sign of an interested buyer, but there will be more such conversions than confirmed purchases, so you will see a statistically significant result earlier. And at the same time, evaluate how many people leave the site due to the inconvenient checkout process!
Do not be disappointed in the numbers, but be on the lookout: then you can see the guide to action where others find only dirty lies.
We always answer: the error is usually about 10%, there is no obvious leader in accuracy, it is impossible to remove all errors - this is how the technology works.Almost no one understands that inaccurate data collection is not the only error affecting the analysis result. Even perfectly collected data will not allow us to accurately calculate the necessary indicators on the site (first of all, the percentage of conversion). The data collected may not be enough! Everyone understands this: if only 15 visitors came to the site and none of them filled out a loan application form, it's too early to talk about conversion. So common sense tells us; but at what point can we say that there is enough data? Should I expect another 100 visits? 200? 500?
When we run an advertising campaign, we pay for each visitor. Web analytics data should tell us which of the ads are wasting a budget, which keywords are more important to us. We need to know the result as soon as possible! When will he be ready?
Common sense in solving this problem is not an assistant. The fact is that in the analysis of statistical indicators we are faced with random processes. Our brain is trying to imagine a uniform process: if the conversion on the site is 10%, it seems to us that every tenth will be converted:
Even if we understand that reality looks different, it is difficult to get rid of the illusion: “wait for a few conversions and get accurate data”. In fact, this process is much more chaotic, for example:
This means that with a limited number of visits, even having accurate data on the number of visitors, views, conversions, we cannot accurately predict future sales. There is no such amount of data that provides firm confidence in the calculated indicators: we can only roughly estimate the “real” percentage of conversion.
There is good news: we can estimate the accuracy with which we calculated this coefficient and realize whether this accuracy is enough for us, or if we need to wait for more data. I will not go into mathematical calculations, telling the theoretical part. I will give only two simple formulas by which it is easy to calculate the numbers you need.
The main concept of mathematical statistics that every analyst should learn is the confidence interval. This is the range in which the true value of the value we need can lie: for example, the conversion rate. The conversion value calculated in the usual way (for example, by Google Analytics) lies somewhere inside this interval; the true value of the conversion (which we learn only on endless traffic) is also most likely located inside it.
Most likely?!Yes, we can only say with absolute accuracy that the conversion is non-negative (and less than one hundred percent). Mathematical formulas will help us build a confidence interval in which a real conversion will be very likely. The probability distribution for our estimate looks like this:
With an increase in the data sample, the bell narrows, and the probability that the true value is closer to the measured one increases. We choose what reliability is considered sufficient. Different confidence intervals can be constructed from the same curve. The chart shows two options (red and green).
Let's get down to business. Let's calculate the confidence interval for the conversion of one of the advertisements - “Buy an elephant at a discount!”. For example, in Google Analytics we see 143 visits and a 2.1% conversion for this ad. We want to compare this ad with another one (“Elephant with delivery in Moscow”) - it has 184 visits and 2.7% conversion - and choose the best one. Many marketers have already concluded that the second ad is better, but we need to check it mathematically.
Before starting the analysis, we make an important decision: what reliability will be sufficient. It is most convenient to use standard values that simplify calculations: these are 68%, 95% and 99.7% (they are called “one sigma”, “two sigma” and “three sigma”, respectively). I would like to count with maximum confidence (three sigma), but this extends the interval and often makes it completely uninformative.
The formula that helps us , is as follows:
, wherein R - calculated by simply dividing the conversion ratio, N - the number of visits, and α - Sigma amount, i.e. measurement accuracy.
With an accuracy of 68%, the conversion of the first ad is in the range from 0.8% to 3.3% ; second ad conversion from 1.7% to 3.8% .
Discouraged, right? It is impossible to compare efficiency: intervals intersect. Both ads may be leaders. If we want to build intervals for a probability of 95%, they will be twice as wide. We need to wait for the data, and the trouble is that we don’t even know when these intervals “parted” and a leader emerges: it depends on the difference in the true conversion rates. Usually, for certainty, you need about a hundred conversions for each ad (word, campaign or other segment whose conversion we consider). Please note: our formula calculates well only when the conversion does not exceed 30%.
But what if our ad didn’t produce a single conversion? The toad begins to strangle us: let's turn off the ads and save the budget!
Zero conversions cannot be calculated according to the previous formula: it turns out that 0 is the exact result. This is not true. If we have 0 conversions and N visits, then the confidence interval is considered simple :
- For one sigma (confidence 68%): interval from 0 to 1.15 / N
- For two sigma (95% confidence): interval from 0 to 3 / N
- For three sigma (99.7% confidence): interval from 0 to 6 / N.
If we have a third ad (“Elephants as a match”) that resulted in 93 visitors without a single conversion, we can only say with certainty of one sigma that its conversion is below 1.2%. This announcement is worse than “Elephant with delivery”, but it is too early to talk about “Elephant at a discount”. Most people in this place are surprised, and some do not even believe: is everything really so unsteady? However, the experience of conducting large campaigns only confirms the random nature of conversions and the fact that at first the “zero” ad on the second hundred clicks can start to show a good result.
Why do not popular analytics tools consider these values for us? I can only speculate. This information will significantly complicate the reports, and most importantly, expose the unpleasant truth: the noticeable number of smart and accurate numbers in these reports are actually useless and even dangerous (if you rely on common sense). Having forgotten about the counting errors, you can stop an effective advertising campaign, considering it unsuccessful, choose a bad version of the advertisement, incorrectly distribute the advertising budget or make a mistake when developing an SEO strategy.
Confidence intervals are excellent for Google Website Optimizer, a tool for conducting and analyzing test results on websites. Optimizer draws conclusions only when it is convinced that the intervals have “diverged” and one of the page options is statistically significantly superior in conversion to other samples.
Many tools take this data into account “under the hood”: for example, Yandex.Direct draws conclusions about the announcement only when it collects enough data and is sure of the result. Of course, we would be interested to see them in the traffic reports, so as not to be mistaken in the estimates by eye. You can write a script for Excel that automatically calculates confidence intervals, but this requires constant data uploading. You can add this functionality to Metric or Analytics using the API of these systems and external scripts.
Are we forced to wait for hundreds of conversions for each ad to evaluate its performance? This would be an ideal solution, but usually impossible: there is not enough advertising budget. Try not to make premature fatal decisions.(remove or modify ads that have not proven to be insolvent) until statistically significant results are obtained; while nothing prevents, for example, temporarily reduce the cost of clicking on them.
It’s useful to add a record of intermediate goals : for example, placing the product in the basket is a sign of an interested buyer, but there will be more such conversions than confirmed purchases, so you will see a statistically significant result earlier. And at the same time, evaluate how many people leave the site due to the inconvenient checkout process!
Do not be disappointed in the numbers, but be on the lookout: then you can see the guide to action where others find only dirty lies.