December 14, 2016 at 10:17How to teach a web application to speak 100 languages: localization features
A key feature of online services is that they are accessible to users from almost all over the world, speaking different languages. If you are developing such a service and want people from different countries to use it, then you need to translate and adapt it, in other words, localize it.
The idea of writing this article arose after the MoscowJS meeting , at which I talked about how the localization process in Badoo is going on . But in this article I would like to tell a little more about the features of localization on the example of web applications, about what localization solutions exist and why in Badoo went their own way. To all who care - welcome under cat.
Why and why?
If you focus only on the Russian market, then most likely you will never need localization, or at least not need it until your product is interesting to an international audience. Then you will have to adapt your application to the new realities very quickly, and this is not so simple. Therefore, you should immediately decide and answer the question: do you really need localization ? If you are inclined to a positive answer, then your service will need to be prepared for various language features. In general, the techniques and tools that allow you to take into account such features are called internationalization, which is the process of creating an application that can work in different languages and with different regional features without any additional changes.
The next question to answer is: why is localization important ?? And it is important primarily for users and customers, because each of them should feel comfortable using your application. For example, residents of a country, even if they know English well, prefer to shop in their native language. Most of them also prefer to use the support services in their native language. If we take Europe, in which there are about 50 countries, then practically each of them will have its own regional features for displaying dates, numbers or currencies. And if we expand our audience to the whole world, then countries like China, Iran, Afghanistan or Saudi Arabia appear in which the text is written from right to left or from top to bottom, and the numbers themselves are written using Indo-Arabic or Persian numbers.
Language Features
What features of the language should you pay attention to first of all , if you decide to introduce localization? First of all, it is worth paying attention to the display of date and time in the format familiar to them. The table below shows several format dependencies by country. As you can see, in most countries the format is different.
| Format | Date example | The country |
|---|---|---|
| yyyy.MM.dd | 09.09.22 | Hungary |
| yyyy-mm-dd | 2016-09-22 | Poland, Sweden, Lithuania, Canada |
| yyyy / mm / dd | 2016/09/22 | Iran, Japan |
| dd.MM.yyyy | 09/22/2016 | Russia, Slovenia, Turkey, Ukraine |
| M / d / yyyy | 09/22/2016 | USA |
The next feature is the number format and currency display. As you can see from the table below, the thousandth and decimal separator may look like a comma, period or whitespace character. And the position of the currency sign may differ not only in different languages, but also in different countries. For example, countries such as Germany and Austria speak the same language, but the money format is different.
| Example | Locale | The country |
|---|---|---|
| 123 456,79 € | ru-RU | Russia |
| € 123,456.79 | en-US | USA |
| 123,456,79 € | de-de | Germany |
| € 123,456.79 | de-AT | Austria |
In addition, the English system of measures, which is used in the United States, Myanmar, and Liberia, may become a problem. Why it is important: it is worth recalling the Mars climate orbiter satellite , which flew up to Mars and fell on it, because the teams on space equipment used force in Newtons, and the software on Earth used pound forces. For the entire duration of the flight no one suspected a mistake. As a result, it cost $ 125 million. So do not forget to display the results in customary measures for users.
When you have figured out the format of dates and numbers, you can move on to the translation features. And the most obvious problem is the declension of nouns after the numerals. As we know, in the Russian language there are three plural forms, while in English there are only two. But there are languages in which there can be six such forms. For example, at this link you will find a table of forms for each language.
| Russian language | English | |
|---|---|---|
| You have 1 gift | Singular | You have 1 gift |
| You have 5 gifts | Plural | You have 5 gifts |
| You have 2 gifts | Few |
1. Translate whole phrases and sentences. They cannot be divided into components, since the sequence of words in different languages can be different.
For example, take this phrase: 8,283 out of 15,311 people liked you!
For English, it looks like this:
{{num_voters_yes_maybe}} out of {{num_voters_total}} {{people}} liked you!But in Japanese, the same sentence already looks different:
{{num_voters_total}}{{people}}中{{num_voters_yes_maybe}}人があなたを気に入っています！As you can see from this example, in Japanese, the reverse sequence of words. Therefore, as many often do, one cannot just write
'Страница ' + pageNum + ' из ' + total2. Some translations differ depending on the gender of the person.
As you can see from the example below, if for the English language you can specify one phrase for both male and female, then in the Slovak language for each gender there will be a phrase.
English
You got an award on {{award_date}}Slovak
М: Toto ocenenie si získal {{award_date}}
Ж: Toto ocenenie si získala {{award_date}}3. Line feeds should be context specific. The translator must know the meaning of the whole sentence, phrase or paragraph, otherwise he may misunderstand and translate incorrectly. For example, a sentence like " You can save this {{item}} " may have a different translation: " You can save / save this {{item}}) ." Ideally, the translator should not only see a set of lines for the translation, but also an image of the area where this line is located.
4. Reusing translation resources may be unsafe. For example, “Save” (file) and “Save” (settings) may have different names in some languages. Or a word like thread can be translated as “thread”, or it can be translated as “thread”.
Here, perhaps, we have identified the most popular features that are encountered when localizing web applications. But this is not all that localization can affect, because design features can also be attributed to it (for example, Japanese and Chinese require a larger font, in some languages the text will be 2 times longer than, for example, in English); color palette (red and green in different cultures mean opposite things, for example, a red checkmark in Japanese means that you did something wrong); the images used (yes, it would be nice to show Asians in Asia, and Europeans in Europe) and many other aspects that are characteristic of a particular country and culture. All this is beyond the scope of the article, but it is worth remembering.
Now look at the Internet and see, andWhat are the tools for client localization ?
Localization methods on the client
The development of interfaces and the implementation of complex business logic already required developers to solve many problems of localization on the client. The internationalization capabilities that ECMAScript provided up to a certain point were rather meager, so libraries such as Closure, Globalize, YUI, Moment.js or some of the developers' own solutions began to appear. All of them expanded the possibilities of ECMAScript and filled in the gaps in internationalization, but the solutions had a different programming interface and certain limitations, associated, for example, with string comparison. So, in December 2012, the ECMA-402 standard appeared, which was supposed to simplify the life of front-end developers in the internationalization of applications. But is this really so? Let's see what this standard offers us now.
ECMAScript Internationalization API
This is the standard that describes the ECMAScript programming interface for adapting to the linguistic and cultural characteristics of languages or countries. Work is done through an Intl object that provides functions for formatting numbers (Intl.NumberFormat), dates (Intl.DateTimeFormat), and string comparison (Intl.Collator). Currently supported by all modern browsers. The last browser that recently added support was Safari, for older browsers you can use polyfill .
The big plus of this standard is that it was developed with the support of Google, Microsoft, Mozilla, Amazon, and, as we are promisedhe will develop. Features will be added to format strings taking into account the plural and gender forms, parsing numbers and much more. It is a pity that this is all happening quite slowly. For example, the standard itself was approved back in 2013, and support by the most popular browsers was realized only in 2016. In the meantime, the functionality of the Intl object is quite limited and does not provide the ability to translate. Therefore, you have to use third-party solutions or use a polyfill for an format not yet approved.
Pros:
- native implementation in the browser;
- high performance;
- does not require downloading additional resources;
- formatting strings with different locales without loading JavaScript resources;
- development of the ECMAScript 2017 Internationalization API standard.
Minuses:
- Legacy browsers require polyfill downloads;
- system dependency. Some locales may not be supported by the client;
- There may be different results in different browsers.
ECMAScript Internationalization API Examples
var mFormat = new Intl.NumberFormat("ru", {
style: "currency",
currency: "GBP"
}).format(1234567.93);
console.log(mFormat); // 1 234 567,93 £
var nFormat = new Intl.NumberFormat('ru-RU').format(1000.15);
console.log(nFormat); // "1 000,15"
var utc = new Intl.DateTimeFormat("en-US", {
timeZone: "utc",
hour: "numeric",
minute: "numeric"
});
console.log(utc.format(new Date())); // 2:38 PMA little more examples can be found here .
As you can see, a lot of things are not yet implemented in the standard, not all the features that web application developers face are taken into account. Therefore, as I wrote above, many have to either look for ready-made solutions, or develop their own. At the moment, there are a lot of solutions, and each has its own advantages and disadvantages. If you turn to Google, then when you search, the first results will be i18next , FormatJS , Globalize , jQuery.i18nother. Some of these libraries offer their own solutions; others try to follow the ECMA-402 standard. Take for example two libraries that give us the first results in a Google search and see what they can do.
i18next
According to the developer, this is a very popular library for internationalization both on the client and on the server (node.js). There are many plug-ins, utilities for it, it integrates with different frameworks. It provides an interface for translators where you can upload translation files, but unfortunately it is already paid. It really has a lot of things implemented, and the library continues to grow, which, of course, is good. But it does not follow the ECMA-402 specification and has its own structural format for messages, not ICU Message syntax . In addition, formatting numbers and dates requires loading moment.js or numeral.js . Accordingly, you will have to load these libraries into the project and add locales for the necessary languages to them.
Pros:
- support for many features of the language;
- the ability to download resources from the backend;
- additional plugins and various utilities;
- extensions for popular frameworks, template engines.
Minuses:
- requires loading resources (i18next 35kb + moment 20kb + necessary locales);
- does not follow the ECMA-402 standard;
- paid interface for translators.
More detailed information on working with the library and more examples can be found on the official website .
Format js
Format JS is a modular collection of JavaScript libraries for internationalization. It is based on the ECMA-402, ICU, CLDR standards and has integration with many frameworks and template engines, for example, such as Dust, Ember, Handlebars. This library, if necessary, either downloads a polyfill to work with internationalization, or uses the capabilities of the browser. In addition, it supports work both on the client and on the server.
Pros:
- modularity;
- uses the capabilities of ECMA-402 or polyfill;
- extensions for popular frameworks, template engines.
Minuses:
- requires loading resources if necessary;
- not all opportunities for translations.
For example, the text for translation in ICU format will look like this:
{ gender, select,
female {{
count, plural,
=0 {У нее нет яблок}
one {У нее # яблоко}
few {У нее # яблока}
other {У нее # яблок}
}}
other {{
count, plural,
=0 {У него нет яблок}
one {У него # яблоко}
few {У него # яблока}
other {У него # яблок}
}}
}You can check the work at this link . Use the example above and enter the locale “ru”. The format, at first glance, is quite complex, but allows you to take into account many features of the language. The only thing I have not yet met convenient systems for translators that would work with a similar format.
As you can see, there are many solutions, you just need to choose. But the localization process does not end simply by choosing a system for localization and trying to cope with various features of languages. Any translation system should be closely integrated into your development process, it should represent a single infrastructure for both the developer and the translator, and answer several important questions, for example:
- What will the translation process look like?
- how will the translation files get to the translators and back to the system?
- How do I know the translator where the specific text is?
And only when you have answers to these questions, you can say that you have a well-integrated localization system, convenient for work.
We had to pay attention to all these questions and language features when Badoo entered the international market. In those already distant times, even if similar systems for localization existed, they did not satisfy all our requirements, and, of course, we had to develop our own system for localization (which we already wrote about on Habré , and also toldabout the layout features of multilingual applications). This system should integrate well into our overall process, be transparent and not delay the development process (since, for example, we are “released” twice a day, and it is very important for us that new product ideas quickly find themselves in a production environment). In addition, she needed to be able to work not only with the web, but also with all our other platforms, such as iOS, Android, Windows Phone, and also be used for email newsletters.
With the advent of a common communication format between our clients and the server (hereinafter referred to as the protocol), or, as we call it, “authentication”, many texts began to come from the server. This approach turned out to be convenient for us, since we do not need to store large amounts of translations on the client, and thanks to this approach, for example, we can conduct AB testing of tokens or create such tokens that depend on the user's actions. Each client can also store the necessary translations at home. The decision where to store translations - on the client or server, is made by the team responsible for the development of the protocol. If some translations are updated, then each client can request new ones (since translations are updated frequently, and new releases appear in application stores with a certain frequency). We call this mechanism Hot Lexem update.
As can be seen from the figure above, in the localization process, we are busy not only with client developers and translators, but also with many other teams. For example, the MAPI team, as I wrote above, designs the protocol and decides where the translations will be stored. The BackOffice team provides a convenient interface for translators, translators, of course, make translations, and the SRV (server developers) or Frontend (client developers) teams generate and display the desired translation. In addition, when we created such a system, on its basis we were able to create a collaborative translation system ( https://translate.badoo.com/ ), in which our users can participate. And they really help us to make translations taking into account the local characteristics of each country.
Conclusion
It is clear that the process of localization of any application is quite serious and painstaking work, because it affects different teams of the project, and not just developers and translators. And at the end of this article, I want to once again draw your attention to the main, in my opinion, moments when localizing applications:
- Localization is a rather complicated procedure for implementation “on top”. If you need it, it should be laid down in the project from the very beginning.
- Localization resources must be application independent.
- Localization extends not only to lines, and this should also be taken into account when designing.
- Make your system convenient not only for developers, but also for translators, automate the translation process.
- If you are not sure about the quality of the translation, it is better not to translate the text at all.
- Try to take into account the cultural characteristics of each country and language.
- Designs, layouts, colors, used images should be subject to localization.
Perhaps I have everything on this topic. I hope that you learned for yourself some new subtleties of the localization process. If you have your own interesting experience, then share it and comments in the comments. Make web great again!
Vyacheslav Volkov, frontend developer, Badoo
