August 13, 2014 at 00:11Theory of Your First A / B Analysis
Testing site content is one of the most important elements of its optimization in terms of the conversion process. A wide selection of technical capabilities for implementing this method of analysis and its “simplicity” made A / B analysis one of the most popular ways to optimize a site. Moreover, the lack of elementary basic knowledge and the desire to achieve results now, coupled with confidence in the correctness of their hypothesis, lead to many errors, which sometimes have critical consequences.
As in any other study, it is necessary to adhere to the basic principles of the methodology.
In order to understand exactly where you should pay attention when developing hypotheses for A / B testing, you need to define pages that belong to the following groups:
- Pages from which the user most often leaves your site;
- Pages that are visited most often;
- Pages, the visit of which is necessary for the order (purchase, subscription).
(Information about this can be obtained based on the site architecture and data from Google Analytics or similar tools)
For pages from each group, you need to look for solutions in the appropriate direction. For example, pages with a large number of failures mean that either the user did not find there what he expected; either he did not manage the site interface or found it too complicated; or something caused him doubts about the reliability of your resource.
There are four main questions that will help you understand the needs and motives of visitors:
1. Why do they come to your site?
2. What prevented them from completing the purchase?
3. Did they find what they were looking for?
4. If they made a purchase, what was the most annoying in the process?
The first question will help you understand the visitor’s intentions. If you find out why they come to you, you can give them what you need, and if you are not able to do this, you should pay serious attention to the ways and methods of your promotion and advertising. In order to get an answer to this question, use the statistics of search queries for which visitors get to your site, as well as the sites from which they come to you.
You can get answers to other questions from the users themselves. But be careful, creating polls on the site is a process that requires feeling and tact, otherwise it is the “polls” that will be the answer to the fourth question.
Now that you have identified the attack area and received the necessary data, it's time to begin work on preparing the test. At this stage, you need to decide on the specific elements with which you will work.
Do not rush with a complete redesign, start small. Changes that take a few minutes can produce an unexpected effect. Typically, these elements are headings and images (or similar content).
Despite the apparent simplicity of such tests, each of the hypotheses must be supported by theory and statistics in order to have a chance of success. You should not expect radical results due to changes in the background or a few sentences, although no one says that changes cannot occur.
Most likely, the best solution will be a set of several simple options that are easy to manage.
It is much more likely to achieve a significant effect from a complete change in the design of the site. But it is worth remembering that such a test requires much closer attention. You must constantly analyze the data received and make appropriate adjustments, while conclusions about the overall effectiveness of the new design can only be made when the results meet all the requirements (see step 5).
Sean Ellis claims that the best option for the first 10 tests is the ratio:
- 4 Test Messages;
- 4 “Aha” tests;
- 2 Large-scale design tests /
Testing Messages - these are tests in which the main task is to work with the submission of your proposal, elements of its presentation and description. Often problems arise at the stage of understanding by the user what exactly you offer them, on what conditions and at what price. Answers to all these questions should be contained in your proposal.
“Aha” tests : Tests of the key elements that are involved in the conversion process. You should track such elements based on statistics of user actions and transitions, and session heatmaps can help you.
This can be an introductory video, a description, a discount coupon, or something completely different. Keep your visitors focused on these elements as quickly as possible.
Large-scale design trials are not necessarily a complete change in design concept. These are tests that affect most of the existing interface, site content and other elements.
After your tests are launched, it is very important to carefully evaluate their results and decide on the fate of each option. Remember that the goal of A / B analysis is not to prove your hypothesis, but to determine the most effective option.
When analyzing the results, it is worth remembering “industry standards”; the hypothesis can be considered confirmed provided that the level of statistical significance of the hypothesis is not lower than 95%, and the statistical power exceeds 80%.
A good calculator for calculating the amount of traffic needed to ensure that the data obtained can be considered statistically significant.
In addition, do not forget that:
• Changes must be tracked on the entire site, and not on a single page. The fact that you are testing a change to a specific page does not mean that it will not affect the entire resource;
• Big changes give great results, but this is not always related to their effectiveness. There is always an effect due to the reaction of regular users to a sharp change in the general concept;
• Reducing the requirements for the statistical weight of the results will reduce the time that will be spent on decision-making, but increases the risk of error.
As in any other study, it is necessary to adhere to the basic principles of the methodology.
Step 1: Selecting the Best Opportunities
In order to understand exactly where you should pay attention when developing hypotheses for A / B testing, you need to define pages that belong to the following groups:
- Pages from which the user most often leaves your site;
- Pages that are visited most often;
- Pages, the visit of which is necessary for the order (purchase, subscription).
(Information about this can be obtained based on the site architecture and data from Google Analytics or similar tools)
For pages from each group, you need to look for solutions in the appropriate direction. For example, pages with a large number of failures mean that either the user did not find there what he expected; either he did not manage the site interface or found it too complicated; or something caused him doubts about the reliability of your resource.
Step 2: Understand User Profile Requirements
There are four main questions that will help you understand the needs and motives of visitors:
1. Why do they come to your site?
2. What prevented them from completing the purchase?
3. Did they find what they were looking for?
4. If they made a purchase, what was the most annoying in the process?
The first question will help you understand the visitor’s intentions. If you find out why they come to you, you can give them what you need, and if you are not able to do this, you should pay serious attention to the ways and methods of your promotion and advertising. In order to get an answer to this question, use the statistics of search queries for which visitors get to your site, as well as the sites from which they come to you.
You can get answers to other questions from the users themselves. But be careful, creating polls on the site is a process that requires feeling and tact, otherwise it is the “polls” that will be the answer to the fourth question.
Step 3: Data Usage and Test Planning
Now that you have identified the attack area and received the necessary data, it's time to begin work on preparing the test. At this stage, you need to decide on the specific elements with which you will work.
Do not rush with a complete redesign, start small. Changes that take a few minutes can produce an unexpected effect. Typically, these elements are headings and images (or similar content).
Despite the apparent simplicity of such tests, each of the hypotheses must be supported by theory and statistics in order to have a chance of success. You should not expect radical results due to changes in the background or a few sentences, although no one says that changes cannot occur.
Most likely, the best solution will be a set of several simple options that are easy to manage.
It is much more likely to achieve a significant effect from a complete change in the design of the site. But it is worth remembering that such a test requires much closer attention. You must constantly analyze the data received and make appropriate adjustments, while conclusions about the overall effectiveness of the new design can only be made when the results meet all the requirements (see step 5).
Step 4: First Tests
Sean Ellis claims that the best option for the first 10 tests is the ratio:
- 4 Test Messages;
- 4 “Aha” tests;
- 2 Large-scale design tests /
Testing Messages - these are tests in which the main task is to work with the submission of your proposal, elements of its presentation and description. Often problems arise at the stage of understanding by the user what exactly you offer them, on what conditions and at what price. Answers to all these questions should be contained in your proposal.
“Aha” tests : Tests of the key elements that are involved in the conversion process. You should track such elements based on statistics of user actions and transitions, and session heatmaps can help you.
This can be an introductory video, a description, a discount coupon, or something completely different. Keep your visitors focused on these elements as quickly as possible.
Large-scale design trials are not necessarily a complete change in design concept. These are tests that affect most of the existing interface, site content and other elements.
Step 5: Measure Results
After your tests are launched, it is very important to carefully evaluate their results and decide on the fate of each option. Remember that the goal of A / B analysis is not to prove your hypothesis, but to determine the most effective option.
When analyzing the results, it is worth remembering “industry standards”; the hypothesis can be considered confirmed provided that the level of statistical significance of the hypothesis is not lower than 95%, and the statistical power exceeds 80%.
A good calculator for calculating the amount of traffic needed to ensure that the data obtained can be considered statistically significant.
In addition, do not forget that:
• Changes must be tracked on the entire site, and not on a single page. The fact that you are testing a change to a specific page does not mean that it will not affect the entire resource;
• Big changes give great results, but this is not always related to their effectiveness. There is always an effect due to the reaction of regular users to a sharp change in the general concept;
• Reducing the requirements for the statistical weight of the results will reduce the time that will be spent on decision-making, but increases the risk of error.