January 29, 2015 at 18:09Crawl and index pages in multiple languages
- Transfer
Many large sites attract visitors every day, not only in the country from which they come. Localized versions are made for foreign visitors: some are better translated, some are worse, some have as much information as the native language of the site, others have very few. However, it can be difficult for Google to index content on multilingual sites.
Often, the choice in which language to give information by default is made either manually with the information about the selected language stored in cookies, or automatically: according to the language settings of the browser, operating system, using geolocation and determining the country / city by IP. The Googlebot crawler is not a user, it will not select the language itself, and the automatic detection system will not recognize it and will not display all languages at once, so it took some effort to properly index the content.
Previously, Googlebot had a US IP address, and it sent requests without the Accept-Language header. In this situation, most sites silently gave him their English version, and multilingual resources could have problems with indexing content.
For example, when entering a Russian site where users from abroad were always provided with English content, the robot was not able to get the main content of the site in Russian. As a result, in the search results in Russia the pages of such a site could be displayed lower than they deserved, or not be displayed at all if the option “display results only in Russian” was set.
The robot’s algorithm has changed: if materials can be selected depending on the user's location and language settings on his device, Googlebot will process such pages using new scanning configurations that take into account the location and language. Here are the solutions that are used for this:
New crawl configurations are automatically activated, so your pages may be processed differently by Googlebot and appear in the search results even if you did not change the server or content management system settings.
Do you want to provide the most effective indexing and ranking of your content and make the site more convenient for users to search? We still recommend that you use different URLs for the language versions and select the appropriate options using the rel = "alternate" hreflang = "x" attributes .
If you have any questions or suggestions, we are waiting for you on our webmaster help forum or in the webmaster community on Google+: g.co/ruwebmasters .
Multiregional and multilingual sites (Help Center section)
Six tips for creating sites for a multilingual audience from Google Web Studio
Often, the choice in which language to give information by default is made either manually with the information about the selected language stored in cookies, or automatically: according to the language settings of the browser, operating system, using geolocation and determining the country / city by IP. The Googlebot crawler is not a user, it will not select the language itself, and the automatic detection system will not recognize it and will not display all languages at once, so it took some effort to properly index the content.
As it was before
Previously, Googlebot had a US IP address, and it sent requests without the Accept-Language header. In this situation, most sites silently gave him their English version, and multilingual resources could have problems with indexing content.
For example, when entering a Russian site where users from abroad were always provided with English content, the robot was not able to get the main content of the site in Russian. As a result, in the search results in Russia the pages of such a site could be displayed lower than they deserved, or not be displayed at all if the option “display results only in Russian” was set.
How googlebot works now
The robot’s algorithm has changed: if materials can be selected depending on the user's location and language settings on his device, Googlebot will process such pages using new scanning configurations that take into account the location and language. Here are the solutions that are used for this:
- Sending a request from other IP addresses that are not related to the USA in addition to the addresses used so far from the USA.
- Using Accept-Language headers in the request, which allows Googlebot to switch to different language versions of the page and crawl them.
New crawl configurations are automatically activated, so your pages may be processed differently by Googlebot and appear in the search results even if you did not change the server or content management system settings.
What should be done to make sure a multilingual site is displayed correctly?
Do you want to provide the most effective indexing and ranking of your content and make the site more convenient for users to search? We still recommend that you use different URLs for the language versions and select the appropriate options using the rel = "alternate" hreflang = "x" attributes .
If you have any questions or suggestions, we are waiting for you on our webmaster help forum or in the webmaster community on Google+: g.co/ruwebmasters .
Reference materials
Multiregional and multilingual sites (Help Center section)
Six tips for creating sites for a multilingual audience from Google Web Studio