How we implemented HTTPS on the main page of Mail.Ru portal

Paradoxically: HTTPS has been around for many years, but over the years it has not become the default standard for all Internet resources that work with security-critical data. Last year, we were the first among major Russian portals to include HTTPS on the main page.

Now encryption works from the first steps of the user on the Mail.Ru portal, and is also always enabled by default. In our opinion, this is just a must-have, because it is on the main page that the login form and email password are located. Although the transfer of user data during authorization has been going on over HTTPS for many years, but if the main page is loaded via HTTP, then at the data entry stage it is possible to intercept it during an SSLstrip attack, which is quite popular among cybercriminals. Let me remind you of its essence, taking for example the non-existent site example.com:

• Suppose a user accesses the Internet through public Wi-Fi and writes example.com in the browser;
• The request goes to the example.com servers on which the response is generated. It contains, among other things, a link to the URL where authorization takes place (https://auth.example.com);
• An attacker intercepts the response and changes this URL to any other (https://some-fraudlent-server.com). Without HTTPS, a hacker can not only “listen” to traffic, but also send fake data to the client on behalf of the server;
• An attacker sends a fake page to the browser of the user who displays it. It looks exactly like a regular page;
• An unsuspecting user enters their username and password there, then clicks "Enter";
• The browser transfers the user to the action of the authorization form. If it weren’t for the actions of the attacker, he would translate it to https: //auth.example.com, but due to the spoofing of the URL, the browser will transfer it to his page http: //some-fraudlent-server.com, sending there username and password of the user.

Now, even if the user turns out to be very inventive (for example, he has saved in the bookmarks the http-address of the page - http: //mail.ru/ and will try to go to the main page using this link), and the attacker will be very persistent - and so, even in this case, user data will be protected thanks to Strict Transport Security technology. This is a mechanism that activates a forced secure connection via HTTPS. It is supported by most modern browsers. That is why it was so important for us to implement HTTPS on the main page.

Thus, the transition to HTTPS protects, firstly, from listening to traffic and, secondly, from the fact that the page code can modify MiTM. It can be either an attacker or, for example, a public Wi-Fi provider who wants to show the user their own banner ads. Such banners can overlap part of useful content, for example, navigation elements - of course, this is not always liked by users. By the way, theoretically, a provider could just as well introduce malicious content by intercepting user data.

And now about how we implemented HTTPS on the main page of one of the largest and most visited runet portals and what difficulties we had to face.

Translate content to HTTPS

The first step was to transfer to https all the content displayed on the main page. We get most of the information from other Mail.Ru services (News, Weather, and so on). Some of them have not switched to HTTPS yet. But the content, namely the pictures taken from other services, are physically transferred to our servers, so there were no problems with receiving them via HTTPS, or with a possible increase in the load on these services.

It was more difficult with the general banner system, which works on the Mail.Ru portal, since it also contains external content from partners. These are pixels (transparent 1x1 pixel images) that twitch to calculate statistics when a partner’s banner is displayed. Even when we introduced HTTPS on the Mail, we did two things: firstly, we agreed with partners that they also switched to HTTPS and, secondly, so that the display of banners with insecure content was technically prohibited on the side of our banner system . Translating the main page to HTTPS, we used these solutions.

Another example of affiliate content is audience counters. If you switch to HTTPS, we recommend that you check all such content for compatibility with a secure protocol.

Before enabling HTTPS for live users, we forcibly transferred Mail.Ru Group employees to it. This could not give any noticeable load, but it made it possible to once again make sure that all page content is loaded using a secure protocol.

Enable HTTPS on live

The main problem we were preparing for: switching to HTTPS significantly increases the load on the server. In addition, it increases the overall page load time by increasing the connection time to the server and the time to load scripts and statics. Thus, the main tasks that we solved were, firstly, to try to start on the current amount of hardware and, secondly, to get as low as possible in page loading speed.

We decided to do a battle check and turned on HTTPS for some users, measuring how much the load on iron increased. Over the course of several days, we gradually increased the share of HTTPS and followed the charts - CPU on the server, page load time by blocks and audience indicators (hits and transitions to projects with the main one). Here it should be noted that for the main page we consider it necessary to keep a multiple supply of load. However, a test launch showed that this margin would not be present when starting HTTPS for 100% of users. And then it's time for optimization.

Firstly, we turned on keep-alive, thanks to which time is not required to establish a reconnection with the server. Since the main page Mail.Ru contacts the server for information on the current user once a minute, we use the established connection to update it. Thanks to this, keep-alive has reduced the number of established connections. As a result, the load decreased and the average waiting time for a response from the server decreased by about 30%.

Secondly, we reduced the number of requests for updating user data by a third. The main page makes about 900 thousand such requests per minute. However, if the user was and remains unauthorized, then we won’t know anything new how many servers you try (this is like looking in the refrigerator every couple of minutes, checking to see if there is something new). As a result, we began to apply for user data updates only when we assume that this data could be updated - when the user is authorized or the authorization has changed. These changes allowed us to reduce the load to an acceptable level and enable HTTPS for all users, without adding additional servers.

When HTTPS was turned on “at all,” we monitored changes in audience indicators, since it could indicate some problems with page loading in non-standard browsers or OS if users encountered them.

There was an interesting point: we noticed an increase in the number of impressions of the mobile version of the main page immediately after enabling HTTPS. This growth, of course, caused concern. It turned out that with HTTPS, the page completely reloads if the user returns to it by clicking back in the browser. And users return to the main one in this way often - links from the mobile version of the main one open in the same window, unlike the large version. In HTTP, the page in mobile browsers is often shown from the cache (the behavior depends on the OS and browser). Therefore, the number of credited impressions of the main and increased. To test this hypothesis, we built a graph of page reloads using the back button (for some browsers), and he confirmed it.

So, we have successfully translated the main page of our portal. In addition, we implemented HTTPS on our content projects, where not so much user data is stored in comparison with, for example, Mail, but there is an authorization form. Now the secure protocol is enabled on Mail.Ru News, Auto, Cars, Hi-Tech, Weather, Health and Children, and in the future this list will continue to expand (in one of the following posts we will tell how their transition to HTTPS affects traffic from search engines )

We hope that our example will inspire colleagues, and for HTTPS all the main pages of more or less large resources will gradually appear. Protect user data from SSLstrip attacks!