Guidelines for optimizing sites for beginners. Part 2

Original author: Charles Shimooka
  • Transfer
Part 1

Optimization process

To establish a well-defined and formal optimization process in the organization is a very useful practice, because it:

  1. organizes the workflow and sets real completion dates
  2. sets quality control standards and reduces errors
  3. adds weight to the whole operation - the logic of the process can be explained to the owners of the company

At the general planning level, I would recommend organizing optimization planning meetings 1-2 times a week, at which it is necessary:

  1. View current tests to see if they need to be stopped or considered “completed” (see below). There are two options for finished tests:
    1. there is a clear winner. In this case, it is necessary to develop its output in production
    2. There is no clear winner in the current control group. In this case, you need to determine whether additional study of the issue is required, or if you just need to stop the experiment.

  2. Consider data sources and think of new test ideas
  3. Discuss and prioritize any new ideas.

How to understand when the test is completed?

Completion criteria - a difficult thing and even are trade secrets. I will determine the minimum necessary conditions for declaring the test “completed”. There are no generally accepted standards, and the criteria depend mainly on the perceptions of your team. We have developed the following criteria for ourselves:

  1. Time. Tests should take at least two weeks to smooth out fluctuations associated with the days of the week.
  2. Statistical Confidence. We used a confidence interval of 90-95%
  3. Time stability. Options should be at least a week in their places.
  4. The total number of conversions. 200 pcs minimum

Creating a new optimization test can go along the same lines as product development. I recommend the following basic structure:

  1. data analysis
  2. search for ideas for improvement
  3. development of test cases
  4. writing test plan
  5. development
  6. quality control
  7. running tests
  8. analysis of results and reporting

Step 1: Data Analysis

First you need to decide what to concentrate on. We used the following list:

  1. Recent releases of products, or pages that are not yet optimized
  2. Especially valuable pages:
    1. Highly profitable (basket, description of expensive products, etc.)
    2. Highly visited (home page)
    3. Special strategic locations important for some other reason.
  3. Pages with bad statistics:
    1. Low conversion
    2. High percentage of care

Step 2: Search for Improvement Ideas

The question of improving the page is as big as the question of the user interface, and is beyond the scope of this article. You can improve texts, design of forms, display of media data, page rendering, appearance, accessibility ...

I advise you only to collect ideas together - use the strength of the whole team to look for new ideas. Include not only designers, but also developers, copywriters, business analysts, marketers, testers in the process ... A good idea can come up anywhere.

Step 3: writing a test plan

A plan is the basis of any test. At the top level, it is used to plan, communicate and document the experiment, and moreover, it teaches the team to correctly and clearly formulate goals and analyze the results.

In a good plan should be the following points:

  1. Test name
  2. Description
  3. Goals
  4. Opportunities (what will we get if successful)
  5. Methodology
    1. Expected Test Dates
    2. Resources (who will work on it)
    3. Tracking metrics
    4. Completion criteria
    5. Options (screenshots of different designs that visitors will see)

Here's a sample test plan .

Step 4: Test Design and Development

Usually they go along the path of product development - but since the test is simpler than a full-fledged product, I use a lightweight version of the path.

But omit for speed should be secondary things - you can not do the documentation, but do not save on design quality. Remember to do basic usability testing of the options.

Step 5: Quality Control

Check the quality of the tests as carefully as any other code. I recommend at least functional, visual and analytical tests.

The plus of optimization tests is that you can arrange any targeting. You can target different options to specific browsers, platforms, audiences, etc. Let's say that your team has checked the operation of only one A / B test - for desktop browsers, but not for mobile. Then you can test its results exclusively for desktop users. If you still have some problems with displaying in mobile browsers, they will not affect the test results.

Step 6: Launch

At the end of quality checks and decision-making on targeting, you need to run tests. There are several things to keep in mind.

Options must be shown at the same time.

The first principle is so obvious that they don’t talk about it. But I very often heard statements like “after the launch of the new design, our sales / conversions increased - which means that the new design is better.”

The problem is that you do not know what other factors could affect the work of the project before and after the launch of the new design. Perhaps the conversion would have already increased, thanks to the promotion of the brand, seasonal fluctuations, or just by chance. Therefore, all options must be checked in parallel. Only in this way can we eliminate extraneous influence.

Track multiple conversion metrics

One of the A / B tests that we performed was used on the movie description page on the Latin American site DIRECTV. We increased the size and visibility of the “Ver adelanto” button (viewing the trailer), deciding that if people watch the trailer, it will encourage them to buy movies from the site.


And so it happened - in a few weeks we saw an increase in the number of purchases by 4.8%. In a year, such an increase would lead to an increase in profits of $ 18,000. Fortunately, since we also tracked other site parameters, we saw that this option reduced the purchase of channel packages (HBO, Showtime) by as much as 25%. This would reduce profits much more. Therefore, we did not begin to introduce this option in production.

It is important to remember that changes can affect your site unpredictably. Always track different metrics.

Tests should reach an acceptable level of statistical significance.

In one of the presentations, the consultant said that preliminary email segmentation tests showed promising results.


On the graph for the last segment of users (logged in more than 4 times per year), the conversion was 0.00139% (0.139 upgrades per 1000 emails). And although this conversion is very small, according to the consultant, it shows a 142% increase, which is a good result.

Without even mentioning the dubious benefits of these statistics (is it proposed to send emails based on the report only to those users who have logged in more than four times?), There is another problem in the test. If you look at the Upgrades column, you will see that the results were deduced from only five cases of ordering an upgrade. Five out of forty eight thousand letters sent. It turns out that plus / minus one order would radically change all the statistics.

Although this is not an example of an optimization test, but simply a study of email segmentation, it contains an important lesson: do not announce the winner without collecting an acceptable amount of statistics.

What is “acceptable”? The concepts of “significant” (95% confidence) and “highly significant” (99% confidence) in the results are accepted in science. And then, in them, accordingly, there is a 5% and 1% chance that your conclusions are incorrect. In addition, the more statistics you need to collect, the more time it will take. I would recommend dwelling on results in the region of 90-95% confidence, depending on the importance of the situation.

The duration of the tests should take into account natural variations (working days and weekends, etc.) and be stable over time

In an article on, Yan Petrovich describes the problem of premature termination of tests. The test was conducted on the popular site for only one day, and in the end it was announced that the winning option increased conversion by 87% with a confidence of 100%.


Jan writes: “If we stopped the test right then and patted each other on the shoulder, we would have made a mistake. We didn’t test our test on Friday or Monday traffic. But, since we did not stop the test, the real result was completely different. "


Four weeks later, it became clear that the new design, although it worked better than the control, showed an improvement of only 10.49%.

Do not forget about short-term fluctuations in the behavior of website visitors, take into account the difference between working days and weekends and seasonal traffic.

Step 7: Analysis and Reporting

At the end of the test, when you clicked the stop button, you need to collect the results in a report. The report may be a continuation of the plan from step 2, but with the following additional sections:

  1. results
  2. Discussion
  3. Further steps

It’s very good to include graphs and various details in the “results” section so that those who are familiar with the report can themselves follow the trends and analyze the data. This will add credibility to your research and attract people to the optimization program.

The discussion is useful for explaining the details and describing the reasons that led to the results. It should make the team think about user behavior and develop further product improvements.

Also popular now: