March 11, 2017 at 22:55Find for a spy: AAB testing as the best option for a split test
Which gives the addition of another letter “A” in the good old AB test and why this is also not enough to assess the changes.
An AAB test or an ABA test is when traffic is divided into three parts: the first and second are sent to the site unchanged, and the third with changes.
1. Insufficient amount of traffic / operations.
Conclusion is done early, high error.
2. Incorrect rotation.
A system or script that engages in rotation does not give an arbitrary and uniform distribution between test blocks.
3. Incomplete cross-browser compatibility
In one of the browsers, the changes do not work correctly. The user group of this browser may have features. Their exclusion from the test distorts its result.
4. Influence of devices
On different devices, changes may look different. For example, on large screens, small details are not so noticeable.
5. The effect of novelty.
Users noticed a new element on the site or its change, pay more attention to it and at first it works better.
6. The effect of addiction.
Users are accustomed to some element and do not notice its change; they work with it as with the old one.
7. Failure to enter the customer’s cycle
During the conversion evaluation, the test period may not coincide with the cycle in which the purchase takes place. Other factors may work earlier, and for the duration of the test, users will already be prepared, the test will show an incorrect result. Also, the consumer cycle may have a long or delayed effect and it will not fall into the test results.
8. Other changes to the
Clean Test is very difficult to achieve, during its implementation other changes on the site may occur. At first glance, it might seem that they are not connected, but this will indirectly affect the test.
9. Seasonal factors, sales and changes in inventory and others.
In my experience, this is:
Each project has its own numbers and the AAB test will help determine them.
A lot has been written about this, I see no reason to repeat myself. Perhaps it makes sense to give links to services with which you can do tests, including multivariate ones:
Experiments in Google Analitycs are a great option for a beginner.
Changeagain.me , Convert.com , Vwo.com, Optimizely.com - powerful, paid services with integration with your account in Google Analytics.
A / Bingo - server frameworks for Ruby.
In the Google Adsense advertising system , there is also an experiment feature.
I also note the online calculator of the criteria for the significance of the split test
and the graphing service based on its results.
To assess the correct time and the required amount of data, sometimes AA tests are used, when the test is put the same thing, without changes. This approach has the right to life, but it takes a separate time. In addition, from the time an AA is conducted to an AB test, something can change.
Summing up, we can say that AAB tests require more resources (traffic, time, money), but give a more accurate and reliable result. To improve it, the AABB system is also used. In any case, for a new project or type of change, it is worth doing an AAB test once or twice, then you can switch to classic ABs.
Have a good test!
See also: How to get the maximum income from advertising systems on your site .
An AAB test or an ABA test is when traffic is divided into three parts: the first and second are sent to the site unchanged, and the third with changes.
What does it give?
- Higher test accuracy. We believe that B gives a relatively reliable result only when A and A are equal.
- Sharp fluctuations in the values of A and A are visible, it is easier to understand which of the factors distorting the test at that moment affected.
- We get the time and amount of data necessary for future tests.
Factors that distort test results
1. Insufficient amount of traffic / operations.
Conclusion is done early, high error.
2. Incorrect rotation.
A system or script that engages in rotation does not give an arbitrary and uniform distribution between test blocks.
3. Incomplete cross-browser compatibility
In one of the browsers, the changes do not work correctly. The user group of this browser may have features. Their exclusion from the test distorts its result.
4. Influence of devices
On different devices, changes may look different. For example, on large screens, small details are not so noticeable.
5. The effect of novelty.
Users noticed a new element on the site or its change, pay more attention to it and at first it works better.
6. The effect of addiction.
Users are accustomed to some element and do not notice its change; they work with it as with the old one.
7. Failure to enter the customer’s cycle
During the conversion evaluation, the test period may not coincide with the cycle in which the purchase takes place. Other factors may work earlier, and for the duration of the test, users will already be prepared, the test will show an incorrect result. Also, the consumer cycle may have a long or delayed effect and it will not fall into the test results.
8. Other changes to the
Clean Test is very difficult to achieve, during its implementation other changes on the site may occur. At first glance, it might seem that they are not connected, but this will indirectly affect the test.
9. Seasonal factors, sales and changes in inventory and others.
What can be done to reduce these effects?
- conduct a split test only on new users or on new and old separately.
- Use multiple systems for control.
- select a separate traffic source for the test.
- do AAAB tests.
Minimum Values for Tests
In my experience, this is:
for advertising campaigns 500,000, 2 weeks.
for site conversion: 500 transactions, 2 months.
to evaluate small changes on the site: 3 thousand sessions, a week.
Each project has its own numbers and the AAB test will help determine them.
But what if there isn’t such a quantity on the site?
- To accept that split tests do not work for you yet.
- To leave the tests working, let them gather statistics.
- Use other methods of testing the hypothesis (surveys, study of competitors, assessment of user behavior).
- Work to minimize distortion in future tests. Sometime their time will come, and you are ready.
Classic AB Testing
A lot has been written about this, I see no reason to repeat myself. Perhaps it makes sense to give links to services with which you can do tests, including multivariate ones:
Experiments in Google Analitycs are a great option for a beginner.
Changeagain.me , Convert.com , Vwo.com, Optimizely.com - powerful, paid services with integration with your account in Google Analytics.
A / Bingo - server frameworks for Ruby.
In the Google Adsense advertising system , there is also an experiment feature.
I also note the online calculator of the criteria for the significance of the split test
and the graphing service based on its results.
AA testing
To assess the correct time and the required amount of data, sometimes AA tests are used, when the test is put the same thing, without changes. This approach has the right to life, but it takes a separate time. In addition, from the time an AA is conducted to an AB test, something can change.
Pros and Cons of AAB
Summing up, we can say that AAB tests require more resources (traffic, time, money), but give a more accurate and reliable result. To improve it, the AABB system is also used. In any case, for a new project or type of change, it is worth doing an AAB test once or twice, then you can switch to classic ABs.
Have a good test!
See also: How to get the maximum income from advertising systems on your site .
