In the process of improving Mail.Ru Mail, we are constantly rolling out new functionality. Today I want to share with you the intricacies of how a product is updated with a multi-million audience, as an example of a complex and interesting task - the redesign of Mail. Looking ahead, I will list the stages of testing that a large and complex feature goes through before implementation:

1. Small group (usually the developers themselves)
2. UX lab
3. Company colleagues
4. Beta users
5. Users who themselves enabled the update
6. Split
7. All Mail Users

Mail in a new design - inside we called it "Septima" - had to successfully go through all these stages.

When choosing a development method, we considered two options: program the product side by side or update the old code. The temptation to write everything from scratch was great, but it was important for us to send Septim to the first and second stages as soon as possible: after all, ideas for updating the design and changing the user experience should be tested in practice, and the only practice in our case is the daily use of the mailbox. If we chose development from scratch, we would start testing much later than in the version with the modification of the old code. Given this, we settled on the second method. Developing a new Mail on top of the old code, we were able to quickly implement the basic functions in the new design, which allowed us to quickly start using it.

Septima on developers (prototype)

It is important to understand that the cost of correcting design errors multiplies with each week. If you write a product for half a year that nobody uses, your risks are already man-months (and if there are several developers, then man-years). Moreover: after six months to understand that the key thing had to be done differently, is equivalent to half a year behind competitors.

In Septima, we changed the look and behavior of the toolbar and focused on getting the Mail with the new toolbar as quickly as possible and testing it in action.

Bottom line: having received the prototype, we were able to feel what previously existed only on mock-ups. This is crucial for further decisions.

UX lab

While testing the prototype, the developer looks at him with a blinded look: after all, he knows about new changes long before he tries them. Users have a fresh look, so it was important for us to conduct testing in the laboratory and see their reaction to updates.

For testing, we selected the main functions of the Mail (write a letter, open, reply, etc.) and asked users to complete these actions. It was important to make sure that for users these actions remained as simple and intuitive as before. The changes were not to be confusing; moreover, they had to make daily activities easier.

Bottom line: the UX laboratory helped us to make sure that the direction was chosen correctly, and we can move on.

Corporate users

In terms of testing, the difference between colleagues from other departments and other users is not very large: for them, changes are also unexpected. The only fundamental difference is that colleagues can quickly report a problem or leave feedback. An important point. With each significant feature, we roll out a form with a vote and the ability to leave feedback about this feature. An indicator of 75% or more who voted more positively is considered good.

Bottom line: rolling out to colleagues helped us solve two problems - get the first mass reviews and check the product for defects. The reviews turned out to be more than positive, we eliminated the repaired defects. You could move on.

Beta users

Updating Mail.Ru Mail as a whole is a huge burden for testing: autotests are not yet written, and manual testing cannot increase several times in size for a short time. In such situations, beta users are required. It is very important to choose them correctly. We decided that we need fairly advanced users who are not afraid of our offer to participate in beta testing, and also can clearly explain what problem they encountered (at least know the word “browser”, “version”, “OS”, etc. .P.). In addition, they should be fairly loyal users who are used to our interface. We came up with the following criteria for "advancement":

• use Chrome or Firefox
• created more than three folders in Mail
• use our mobile application

Those who met all of the above criteria, we showed a die with a proposal to sign up for beta users. We invited them to test the new design with the ability to disable it. The result was an even greater number of reviews and comprehensive testing of the Mail with quality reports of problems. Interestingly, some reports were right away with screenshots of the console. And in one there was a piece of CSS code that was proposed to be added to the project.

Bottom line: in the reviews, users reminded us of several features from the old Mail that we lost sight of in the new version. And the vote once again confirmed that people like the new product.

Self inclusion

For the next stage, we have compiled a smart letter. As soon as the user entered the Mail.Ru Mail web-interface, he received a letter with the description of Septima and the “Enable” button. If the user opened the letter in Septima, then the button in the letter was changed to "Leave feedback." On average, 30% of those who read the letter switched over, which, together with the rest of the restrictions, accounted for 11% of the entire audience.

An interesting fact: for the sake of experiment, we made two buttons for switching to the new interface: “Update” and “Update to version 7.0”. The first clickthrough rate was 5% higher.

At this point, it was important to understand the user response. The number of users (90,000 voted) and reviews (52,000, all reviews have been read and processed) have already made it possible to collect accurate statistics. To our joy, only 2% of those who voted were against the new design.

This stage was difficult for the team, because with such a number of users rare configurations of systems begin to appear (I’ll digress: the configuration includes the operating system, a browser of a certain version, browser plug-ins, antivirus, network environments, etc.; each of these items , as well as their combination, can cause problems with your product). Therefore, rolling out to a significant percentage of the audience requires compliance with several conditions: this should happen at the beginning of the week, you need to make sure that all key developers are not sick and not on vacation, and the support service is on alert. There are several busy days ahead: users begin to report problems, and you need to use the entire arsenal of tools to identify them - logs on the server, reports from browsers, individual logging and even Team Viewer,

Bottom line: in our case, the main problems were brought by third-party browser plug-ins that were counting on the old site layout. For example, the NCH Toolbar broke a scroll in Mail, and WebBars broke a letter. Representatives of a whole family of plug-ins (ExSmile, SmileEx, MegaSmiles), in addition to breaking the display of Mail, also turned out to be trojans that illegally penetrated users' computers and browsers.

After all the problems were fixed, we were ready for the next step.

Measure retention

Reviews are reviews, but before switching the bulk of people, we needed to make sure that the new design works, at least not worse than the old one, and we do not lose users who switched to it. A good indicator for this case is user retention. To evaluate it, we divided the users into three groups identical in all respects. For users from the first group, we switched the interface and showed a tutorial about Septima, which talked about the main changes in the Mail and mentally prepared for the fact that the familiar interface will change. The second group simply switched the interface, and for the third, everything remained the same. The highest percentage of return was found in the first group of users.

Total: we made sure that the new interface not only does not scare away old users, but also increases their return to Mail.

All users

If you launched changes to the current functionality, then you know that the biggest enemy is “Return everything as it was, why did you change everything ?!” My favorite of the reviews I read is “Get your hands off our site!” It doesn’t matter that as a result of the update, we reduced the number of clicks from five to two: for the user, these are native five clicks, he’s used to them.

Everyone wanted to roll out the new interface in the same year in which they started to make it. And we were ready on all counts. All monitoring indicators - download speed, the number of emails sent and read, return to Mail - were either higher or the same as in the old Mail. It remained to measure one more indicator - how much the users liked the new interface. At that time, Septim had a positive rating of 98%, but it had to be taken into account that this was the opinion of users who themselves decided to switch. We “pulled the switch” - switched everyone — and then again measured the ratio of those who were “for” and “against”.

Bottom line: at the moment, the proportion of those who like Septima is 93% - this is a new record for us.

If you had experience of large-scale updates, I will be glad to hear about it in the comments: do you try to minimize the stress for users associated with the transition to the new version, how do you test updates, what are the stages of rolling out updates?