
Search Results Page Experiment
The search results page is one of the most popular Yandex pages. It is downloaded about 130 million times a day. This with an average page size of 25KB gives us 3TB of traffic per day.
Despite the apparent simplicity, behind what this page consists of is the huge work of a large number of people and a lot of sophisticated technologies.
Developing interfaces, we usually go along the evolutionary path, change the page in stages. We check our solutions, introducing them to a small percentage of users - we conduct experiments. We are no longer satisfied with small changes: we want to develop the product by building a new technological platform on which we will implement our projects in the future.
Today we are starting an experiment with a new page of search results. And for this, we chose our site for testing search on the world Internet - yandex.com . The first thing you notice when you see a new page is the difference between its design and the current version of the search results page on yandex.ru. But this experiment is not primarily about design, but about the implementation of significant technological changes. Of course, we are also working on design and plan to test new solutions, but building a new one begins with a quality foundation. We want to tell you exactly how we built it: what decisions were made and which technologies were used.
Template Development
Even before the first mockups appeared, we had to decide on the development approach. There were two options: to lay out a static page and then cut it into templates, or develop immediately on real data and write real templates.
In the general case, especially if one team is involved in layout and template development, a second approach is preferable. It ensures that the project is ready for release (or at least for testing) at any given time. That is, at the very beginning of development, a “raw”, but already functioning prototype of the project arises, which can be “touched”. As a result, this is the option we have chosen.
Our project assembly mechanism, based on the BEM ( Block-Element-Modifier) methodology, helped greatly reduce the time for development and testing of the first prototype) It allows you to connect external block libraries, so we assembled a substantial part of the interface from ready-made blocks that are already on other Yandex services and are located in different internal git and svn repositories.
JavaScript is everywhere
Under the hood of the new version of the search results page are JavaScript templates. They work faster than the template engine for the Perl TT2 language, which we used earlier, and it is more convenient and simple to write them. Now two levels of standardization are written in JS, client scripts, utilities and build technologies, a daemon collector.
Currently, YUICompressor is used to compress JS and CSS code, which requires Java on the server, so in the future we want to switch to UglifyJS and CSSO JS obfuscators - they work faster and are less resource intensive.
Using one language at all technological levels, from the browser to the assembly utilities, greatly simplifies the life of many participants in the process. Developers can configure (and optionally add) tools for assembly, equally easily write code for the client and server. No longer requires in-depth knowledge of a large number of different technologies, it is good enough to know the basic ones: HTML, CSS, JS. Thanks to this, it has become easier to connect new people to the team: all front-end developers in Yandex know JavaScript, and server specifics can be mastered very quickly.
AJAX, History API
Quite a long time ago, it became "fashionable" to use the AJAX technique to update data on a page without a complete reboot. The advantages of this technique are quite obvious: you do not need to load the entire page (markup, styles, scripts, pictures) every time you update the data, initialize the scripts. This greatly speeds up the process of delivering new data to the user. The difference in download speed on slow channels is radical.
However, perfectly understanding all the advantages of AJAX, we were in no hurry to apply this approach in Yandex.Search. We wanted to offer our users not just technology, but a complete product with a redesigned interface. Therefore, a new version of the search results page will require the widespread use of AJAX and the History API.
(We talked in detail about our implementationJune 2, 2012 in Minsk at Y. Subbotnik)
CSS
Initially, we did not have a goal to use the full CSS3 capabilities, but almost all of them were necessary for solving our problems. Menu arrows on transforms, block shadows, first-child / last-child pseudo-selectors instead of modifier classes - all this is in the new interface. At the same time, degradation is provided for older browsers: instead of shadows, simple single-pixel outlines, instead of arrows in the menu, ordinary rectangles, svg gradients in IE9 and a solid background fill for IE8 and older.
Another important decision was to use absolute units of measure to indicate font sizes. Almost all of our services now use the indication of sizes in relative percentages or tagsbut in the new project we decided to abandon them in favor of pixels. There are several arguments for this decision. Firstly, the proportion of browsers that cannot scale fonts in pixels is vanishingly small. Secondly, the sizes in absolute units exclude the influence of the context, and layout with completely independent blocks is one of the basic principles of the BEM methodology. And thirdly, the search results page should work fine on touch devices. The screen resolution on them is often known in advance and strictly fixed, and only its orientation can change.
Deploy
Almost all of our projects store their static files on a static cluster - yandex.st. These are several dozens of cars located in different cities and countries that are able to give data very quickly.
There are several reasons for using a static cluster:
- no need to generate files before sending - everything is ready and lies on the disk;
- To transfer ready-made files, a lightweight web server is enough;
- high machine productivity is not required;
- statics are cached forever, and invalidation is carried out either by changing the package version, or by changing the hash amount;
- service cookies are not sent to the yandex.st domain, this reduces the amount of traffic;
- “through” files (used on several services, for example jquery.min.js or the Yandex logo) do not need to be downloaded every time you switch from service to service.
To put service files on a static cluster, you need to build a deb package. The package is built either manually or with a special script. After assembling a new package, the application for the calculation is sent to the Conductor - our internal tool that allows you to lay out the package on hundreds of machines without the participation of people. This is very important, given the number of Yandex services and the frequency of their updates.
A static cluster is a very complex system that solves many tasks of optimizing the speed of the browser at the client and delivering files to the user. At the same time, using it is very simple: to start a new project in the Conductor, just fill out a small application form and receive confirmation from the responsible administrator, after which the service developer himself completely controls the calculation process. The time to prepare the statics of a new service for deployment is reduced literally to a couple of hours of one specialist, and the new version of the package is laid out and deployed in a few seconds.
Updating the search results page will continue: both on yandex.com and on other versions of Yandex - from Turkish to Russian. We have plans for a lot of experiments, including with interfaces.
Mikhail Troshev,
Search Interface Development Team Leader
Despite the apparent simplicity, behind what this page consists of is the huge work of a large number of people and a lot of sophisticated technologies.
Developing interfaces, we usually go along the evolutionary path, change the page in stages. We check our solutions, introducing them to a small percentage of users - we conduct experiments. We are no longer satisfied with small changes: we want to develop the product by building a new technological platform on which we will implement our projects in the future.
Today we are starting an experiment with a new page of search results. And for this, we chose our site for testing search on the world Internet - yandex.com . The first thing you notice when you see a new page is the difference between its design and the current version of the search results page on yandex.ru. But this experiment is not primarily about design, but about the implementation of significant technological changes. Of course, we are also working on design and plan to test new solutions, but building a new one begins with a quality foundation. We want to tell you exactly how we built it: what decisions were made and which technologies were used.
Template Development
Even before the first mockups appeared, we had to decide on the development approach. There were two options: to lay out a static page and then cut it into templates, or develop immediately on real data and write real templates.
In the general case, especially if one team is involved in layout and template development, a second approach is preferable. It ensures that the project is ready for release (or at least for testing) at any given time. That is, at the very beginning of development, a “raw”, but already functioning prototype of the project arises, which can be “touched”. As a result, this is the option we have chosen.
Our project assembly mechanism, based on the BEM ( Block-Element-Modifier) methodology, helped greatly reduce the time for development and testing of the first prototype) It allows you to connect external block libraries, so we assembled a substantial part of the interface from ready-made blocks that are already on other Yandex services and are located in different internal git and svn repositories.
JavaScript is everywhere
Under the hood of the new version of the search results page are JavaScript templates. They work faster than the template engine for the Perl TT2 language, which we used earlier, and it is more convenient and simple to write them. Now two levels of standardization are written in JS, client scripts, utilities and build technologies, a daemon collector.
Currently, YUICompressor is used to compress JS and CSS code, which requires Java on the server, so in the future we want to switch to UglifyJS and CSSO JS obfuscators - they work faster and are less resource intensive.
Using one language at all technological levels, from the browser to the assembly utilities, greatly simplifies the life of many participants in the process. Developers can configure (and optionally add) tools for assembly, equally easily write code for the client and server. No longer requires in-depth knowledge of a large number of different technologies, it is good enough to know the basic ones: HTML, CSS, JS. Thanks to this, it has become easier to connect new people to the team: all front-end developers in Yandex know JavaScript, and server specifics can be mastered very quickly.
AJAX, History API
Quite a long time ago, it became "fashionable" to use the AJAX technique to update data on a page without a complete reboot. The advantages of this technique are quite obvious: you do not need to load the entire page (markup, styles, scripts, pictures) every time you update the data, initialize the scripts. This greatly speeds up the process of delivering new data to the user. The difference in download speed on slow channels is radical.
However, perfectly understanding all the advantages of AJAX, we were in no hurry to apply this approach in Yandex.Search. We wanted to offer our users not just technology, but a complete product with a redesigned interface. Therefore, a new version of the search results page will require the widespread use of AJAX and the History API.
(We talked in detail about our implementationJune 2, 2012 in Minsk at Y. Subbotnik)
CSS
Initially, we did not have a goal to use the full CSS3 capabilities, but almost all of them were necessary for solving our problems. Menu arrows on transforms, block shadows, first-child / last-child pseudo-selectors instead of modifier classes - all this is in the new interface. At the same time, degradation is provided for older browsers: instead of shadows, simple single-pixel outlines, instead of arrows in the menu, ordinary rectangles, svg gradients in IE9 and a solid background fill for IE8 and older.
Another important decision was to use absolute units of measure to indicate font sizes. Almost all of our services now use the indication of sizes in relative percentages or tagsbut in the new project we decided to abandon them in favor of pixels. There are several arguments for this decision. Firstly, the proportion of browsers that cannot scale fonts in pixels is vanishingly small. Secondly, the sizes in absolute units exclude the influence of the context, and layout with completely independent blocks is one of the basic principles of the BEM methodology. And thirdly, the search results page should work fine on touch devices. The screen resolution on them is often known in advance and strictly fixed, and only its orientation can change.
Deploy
Almost all of our projects store their static files on a static cluster - yandex.st. These are several dozens of cars located in different cities and countries that are able to give data very quickly.
There are several reasons for using a static cluster:
- no need to generate files before sending - everything is ready and lies on the disk;
- To transfer ready-made files, a lightweight web server is enough;
- high machine productivity is not required;
- statics are cached forever, and invalidation is carried out either by changing the package version, or by changing the hash amount;
- service cookies are not sent to the yandex.st domain, this reduces the amount of traffic;
- “through” files (used on several services, for example jquery.min.js or the Yandex logo) do not need to be downloaded every time you switch from service to service.
To put service files on a static cluster, you need to build a deb package. The package is built either manually or with a special script. After assembling a new package, the application for the calculation is sent to the Conductor - our internal tool that allows you to lay out the package on hundreds of machines without the participation of people. This is very important, given the number of Yandex services and the frequency of their updates.
A static cluster is a very complex system that solves many tasks of optimizing the speed of the browser at the client and delivering files to the user. At the same time, using it is very simple: to start a new project in the Conductor, just fill out a small application form and receive confirmation from the responsible administrator, after which the service developer himself completely controls the calculation process. The time to prepare the statics of a new service for deployment is reduced literally to a couple of hours of one specialist, and the new version of the package is laid out and deployed in a few seconds.
Updating the search results page will continue: both on yandex.com and on other versions of Yandex - from Turkish to Russian. We have plans for a lot of experiments, including with interfaces.
Mikhail Troshev,
Search Interface Development Team Leader