Search Experiments: Big and Small

  • From RSS
Ben Gomez: Lead Google Engineer

In my previous post, I described the principles and basic components of a convenient, fast and efficient web search. For simple functions such as spellchecking and two-line descriptions of the site in the search results, there are complex algorithms. How much the features we develop are useful and convenient, we usually evaluate using experiments. These are small tests performed at our request by small groups of users.

Experiments are a very effective tool, and we use it extensively to test potential changes to search algorithms. At any given time on Google sites around the world, between 50 and 200 experiments are conducted. I will start with a description of very small experimental changes that you might not even notice at first glance. And then I’ll talk about some more illustrative examples. There are many people who closely monitor all changes to Google search (sometimes even attributing to us absolutely fantastic things!). Such people are usually aware of serious experiments. At the same time, experiments concerning small changes remain almost unnoticed.

For example, can you find the differences between the two pages?

Option 1:



Option 2:



I am absolutely sure that I myself could not distinguish them if I saw them separately. But, of course, you can! You need to look at each picture as a whole, and then you will see that there is a measurable difference between them. For those who haven’t guessed yet, I’ll tell you that the empty space around the first search result has become a little more, and therefore the first result in Option 2 has become a little more noticeable. This change reflects the fact that our ranking algorithms believe that the first result is significantly more relevant. The advantage of this arrangement is that it helps you concentrate on the very first search result. But, on the other hand, this may interfere with viewing other results. The experiments help to understand which change is more significant, and how the speed of information search changes.

Another change that seems to be minimal:




Here the difference is so obvious that it seems not at all difficult to say which option will work better. In this case, the difference is in the thickness of the plus sign next to the link to the stock price. But when it comes to what is better in terms of search, everything here is not so obvious. Does it follow from the fact that people will click on the big plus sign more often that it is necessarily better? Is it possible that other useful results will seem inconspicuous against the background of results with a big plus? Follow the development of Google, and you will see which version wins! And if everything goes as it should, then although you hardly notice it, the search for you will work better and better. The world will turn pink. And the birds will sing merrily.

Maybe not - but then, at least, you will have the best badge badge that we can come up with. :)

Of course, not all of our experiments are crazy eyesight tests. And I talked about these two experiments in order to emphasize that we are testing almost any changes, even the smallest ones, which, it would seem, need not be worried. In fact, small changes matter, and we take them seriously.

Another type of experiment concerns changes that are not purely visual, but that relate to internal algorithms for presenting information. For example, the algorithm responsible for the formation of headings and descriptions in the search results, now selects not only the query words themselves, but, in addition, their variations and synonyms. For example, at the request of [poems by Pushkin], the results with the highlighted word “poems” will be displayed.



Highlighting morphological variations of words is, in general, a good idea. This helps you better understand which results are relevant to your request. True, of course, not always. Such experiments help us confirm (and sometimes refute) our ideas about changes in these algorithms.

There is another important type of experiment - it is testing functions that significantly affect the search on the network. Even for significant functions, the purpose of the experiments remains the same: are we adding something really useful to people or just distracting their attention? Google is an intuitive product, and the features should speak for themselves without the need for a detailed description or user manual. There is, of course, a well-written help, but most users do not need it. One of the goals of the experiment is to understand how a particular function will be used in practice. And this can differ significantly from our initial ideas.

So, you got acquainted with examples of experiments that we conduct when testing any functions - from barely noticeable to the most vivid and significant. So if the next time you go to Google and see that it has become a little different, maybe we did it ... just for your sake.

Also popular now: