From monoliths to microservices: the experience of M.Video-Eldorado and MegaFon

On April 25, we at Mail.ru Group held a conference about clouds and around - mailto: CLOUD . A few highlights:

• The main Russian providers gathered on one stage - Mail.ru Cloud Solutions, #CloudMTS, SberCloud, Selectel, Rostelecom Data Center and Yandex.Cloud spoke about the specifics of our cloud market and their services;
• Colleagues from Bitrix24 told how they came to multiclaud ;
• Leroy Merlin, Otkrytie, Burger King and Schneider Electric provided an interesting view on the part of cloud consumers - what tasks their business sets for IT and which technologies, including cloud ones, they see as the most promising.

All videos from the mailto: CLOUD conference can be viewed here , but here you can read how the discussion about microservices went. Alexander Deulin, head of the MegaFon business systems research and development center, and Sergey Sergeyev, information technology director of the M. Video-Eldorado group, shared their - successful - cases of getting rid of monoliths. They also discussed related issues of IT strategy, processes, and even HR.

Panellists

• Sergey Sergeev , Director of Information Technology , M.Video-Eldorado Group ;
• Alexander Deulin , head of the MegaFon business systems research and development center ;
• Moderator - Dmitry Lazarenko , head of PaaS-direction Mail.ru Cloud Solutions .

After a speech by Alexander Deulin “How MegaFon Expands Its Business Through a Microservice Platform”, Sergey Sergeev from M.Video-Eldorado joins it for discussion and moderator of the discussion Dmitry Lazarenko, Mail.ru Cloud Solutions.

Below we have prepared for you a transcript of the discussion, but you can also watch the video:

Switching to microservices - a response to market needs

Dmitry:

Have you had any successful experience switching to microservices? And in general: where do you see the greatest benefit in business from the use of microservices or the transition from monoliths to microservices?

Sergey:

We have already come some way to the transition to microservices and have been using this approach for more than three years. The first need that justified the need for microservices was the endless integration of different front-line products with the back office. And each time we had to do additional integration, development, implementing our rules for the operation of a service.

At some point, we realized that we needed to speed up our systems and output functionality. At that moment, such concepts as microservices, microservice approach already existed on the market, and we decided to give it a try. It started in 2016. Then the platform was laid and the first 10 services were implemented, by a separate team.

One of the first services, the most heavily loaded, was a price calculation service. Each time you come to any channel, to the M.Video-Eldorado group of companies, whether it’s a website or a retail store, pick up goods there, see the price on the website or in the “Basket”, one service automatically calculates the price. Why this is needed: before that, each system had its own principles of working with promo - discounts and so on. The back office is engaged in pricing, the discount functionality is implemented in another system. It was necessary to centralize and create such a unique, detachable service in the form of a business process that would allow us to implement this. That’s how we started.

The value of the first results was very great. Firstly, we were able to make detachable entities that allow us to work separately, aggregated. Secondly, we have reduced the cost of ownership in terms of integration with a large number of systems.

Over the past three years, we have added three front-line systems. It was difficult to maintain them in the same amount of resources that the company could afford. Therefore, the task arose of looking for new exits, responding to the market in terms of speed, in terms of internal costs and efficiency.

How to evaluate the success of the transition to microservices

Dmitry:

How is success in switching to microservices determined? What was the “before” in each company? What metric did you use to determine the success of the transition, who actually determined it?

Sergey:

First of all, it was born inside IT as enabler - “unlocking” new features. We had a need to do everything faster for relatively the same money, responding to market challenges. Now, success is expressed in the number of services reused by different systems, the unification of processes among themselves. Now it’s so, and at that moment it was an opportunity to create a platform and confirm the hypothesis that we can do this, this will give the effect and calculation of the business case.

Alexander:

Success is rather an inner sensation. Business always wants more, and the depth of our backlog is the confirmation of success. It seems to me.

Sergey:

Yes, I agree. For three years we have already more than two hundred services and backlog. The demand for resources within the team is only growing - annually by 30%. This is because people felt: it is faster, differently, other technologies, all this is developing.

Microservices will come, but core will remain

Dmitry:

This is like an endless process where you invest in development. Has the transition to microservices for business itself already ended or not?

Sergey:

It is very easy to answer. What do you think: replacing phones is an endless process? We ourselves buy phones every year. And here it is: while there is a need for speed, in adapting to the market, some changes will be required. This does not mean that we refuse standard things.

But we cannot immediately embrace and redo everything. We have legacy, standard integration services that were before this: enterprise buses and so on. But there is a backlog, and there is a need too. The number of mobile applications and their functionality are growing. At the same time, no one says that they will give you 30% more money. That is, there are always needs on the one hand, and on the other, a search for efficiency.

Dmitry:

Life is in good shape. (Laughs)

Alexander:

In general, yes. We do not have revolutionary approaches to removing core parts from the landscape. Systematic work is underway to decompose systems so that they are more consistent with microservice architecture, to reduce the influence of systems on each other.

But we plan to keep the core part, as in the operator’s landscape there will always be some platforms that we buy. Again, you need a healthy balance: we should not rush into the "cutting" core. We put the systems side by side, and now, it turns out that already over many core parts. Further, developing the functionality, we make the necessary representations for all channels that work with our communication services.

How to sell microservices to a business

Dmitry:

I’m still interested - for those who haven’t switched, but are going to: how easy was it to sell this idea to business and was it an adventure, an investment project? Or it was a deliberate strategy: now we go to microservices and that’s all, nothing will stop us. How was it with you?

Sergei:

We did not sell the approach, but the business gain. There was a problem in business, and we tried to remove it. At that moment, it was expressed in the fact that in different channels different pricing principles were used - separately for promotions, for stocks and so on. It was difficult to maintain, errors arose, we listened to customer complaints. That is, we sold the elimination of the problem, but came with the fact that we need money to create a platform. And they showed a business case on the example of the first stage of investment: how will we continue to pay for it and what will it allow us to do.

Dmitry:

Did you somehow fix the time of the first stage?

Sergei:

Oh sure. We took 6 months to create core as a platform and pilot testing. During this time, we tried to create a platform on which the pilot was skated. They further confirmed the hypothesis, and since it works, then we can continue. They began to replicate and strengthened the team - they brought it to a separate unit, which is engaged only in this.

Next is the systematic work proceeding from the needs of the business, opportunities, availability of resources and all that is now in work.

Dmitry:

Okay. Alexander, what do you say?

Alexander:

Our microservices were born from the “sea foam” - due to saving resources, due to some balances in the form of server capacities and redistribution of forces within the team. Initially, we did not sell this project to business. It was a project where we both researched and developed accordingly. We started at the beginning of 2018 and just enthusiastically developed this direction. Sales have just begun, and we are in the process.

Dmitry:

But it happens that a business allows you to do such things on Google’s principle - on one free day a week? Do you have such a direction?

Alexander:

At the same time as research, we were engaged in business tasks, because all of our microservices are solutions to business problems. Only at the beginning we built microservices that cover a small part of the subscriber base, and now we are in almost all flagship products.

And the material impact is already clear - we can already be counted, we can estimate the speed of product output and lost revenue if we were to go the old way. This is where we are building the case.

Microservices: hype or need?

Dmitry:

There are numbers. And revenue or money saved is very important. And if you look at the other side? It seems that microservices are a trend, hype and many companies abuse it? How clearly do you distinguish between what you translate and don’t translate into microservices? If legacy is now, will you still have legacy in 5 years? What will be the age of information systems that work in M. Video-Eldorado and MegaFon in 5 years? Will there be ten-year, fifteen-year information systems or will it be a new generation? How do you see it?

Sergei:

It seems to me that very far away is difficult to make. If you look back, who suggested that this would develop the market for technology and the same machine learning, user identification by face? But if you look in the coming years, it seems to me that core-systems, enterprise-systems of the ERP class in companies - they have been working for a long time.

Our companies have been together for 25 years, in them the classic ERP is located very deep in the system landscape. It is clear that we take out some pieces from there and try to aggregate them into microservices, but core will remain. It’s hard for me now to imagine that we will replace all core systems there and quickly cross over to the other, brighter side of new systems.

I am a supporter that everything that is closer to the client and the consumer, where there is the greatest business benefit and value, where you need adaptability and focus on speed, on changes, on “try, cancel, reuse, do something else "- there the landscape will change. And box products are not built in very well there. At least we do not see this. It requires the most lightweight, simple solutions.

We see such a development:

• core information systems (mostly back office);
• middle layers in the form of microservices connect core, aggregate, create a cache and so on;
• front-line systems are aimed at the consumer;
• integration layer, which is generally integrated into marketplaces, other systems and ecosystems. This layer is as light as possible, simple, with a minimum of business logic in it.

But at the same time I am a supporter of continuing to use the old principles, if they are used to the place.

Let's say you have a classic enterprise system. It is located in the landscape of one vendor, consists of two modules that work with each other. There is also a standard integration interface. Why redo it and bring a microservice there?

But when there are 5 modules in the back office, from which pieces of information are collected into the business process, which are then used by 8-10 front-end systems, the benefits are immediately noticeable. You take out of five back-office systems, create a service, moreover, isolated, which is focused on the business process. You make a service technological - so that it caches information and is fault-tolerant, and also works with documents or business entities. And integrate it on a single principle with all front-line products. They canceled the front-line product - they simply turned off integration. Tomorrow you need to write a mobile application or make a small site and only one part to land in the functional - it's simple: you put it together as a constructor. I see development in this direction more - at least in our country.

Alexander:

Sergey fully described our approach, thanks. I’ll just say where we definitely won’t go - to the core part, to the topic of online charging. That is, the rating and charging will remain, in fact, a “threshing” thresher that will reliably write off money. And this system will continue to be certified by our regulatory authorities. Everything else that looks toward customers, of course, is microservices.

Dmitry:

Here certification is just one story. Probably more support. If you spend a little on support or the system does not require support and improvement, it is better not to touch it. Reasonable compromise.

How to develop reliable microservices

Dmitry:

Good. But I'm still interested. Now you are telling a success story: everything was fine, we switched to microservices, defended the idea in front of the business, and everything worked out. But I heard other stories.

A couple of years ago, a Swiss company, which invested two years in the development of a new microservice platform for banks, eventually closed this project. Completely folded. Many millions of Swiss francs were spent, but in the end they dispersed the team - it didn’t.

Have you had similar stories? Were or are there any difficulties? For example, microservice maintenance, the same monitoring is also a headache in the company's operations. After all, the number of components increases tens of times. How do you see this, were there any unsuccessful examples of investments here? And what can people be advised so that they do not encounter similar problems?

Alexander:

Unsuccessful examples were that business changed priorities and canceled projects. When at a good stage of readiness (MVP is actually ready), the business said: "We have new priorities, we are going to another project, and we are closing this one."

We did not have global files with microservices. We sleep peacefully, we have a 24/7 duty shift that serves the entire BSS [business support system].

And yet - we rent microservices according to the rules by which boxed products are rented. The key to success is that, firstly, you need to assemble a team that will fully prepare the microservice for production. Development itself is, conditionally, 40%. The rest is analytics, DevSecOps methodology, the right integrations and the right architecture. We pay special attention to the principles of building secure applications. In each project, representatives of information security participate both in the planning stage of architecture and in the implementation process. They also manage code analysis systems for vulnerabilities.

Suppose we deploy our stateless services - we have them in Kubernetes. This allows everyone to sleep peacefully due to auto-scaling and auto-raising services, and the shift on duty picks up incidents.

For the entire time of the existence of our microservices, there was only one or two incidents that reached our line. Now there are no operational problems. Of course, we have not 200, but about 50 microservices, but they are used in flagship products. If they failed, we would be the first to know about it.

Microservices and HR

Sergey:

I agree with my colleague about the transfer in support - that you need to organize the work correctly. But I will say about the problems, which, of course, are.

Firstly, the technology is new. This is a hype in a good way, and finding a specialist who will understand and be able to create this is a big challenge. The competition for resources is crazy, respectively, experts are worth its weight in gold.

Secondly, when creating certain landscapes and a growing number of services, the problem of reuse should be constantly solved. As developers like to do: “Let’s write here a lot of interesting things here ...” Because of this, the system grows and loses its effectiveness in terms of money, cost of ownership, and so on. That is, you need to lay reuse in the architecture of systems, include in the road map the introduction of services and the transfer of legacy to the new architecture.

Another problem - although this is good in its own way - is internal competition. "Oh, new fashion guys have appeared here, they speak a new language." People, of course, are different. There are those who are used to writing in Java, and those who write and use Docker and Kubernetes. These are completely different people, they talk differently, use different terms and sometimes do not understand each other. The ability or inability to share practice, knowledge sharing, in this sense is also a problem.

Well, scaling resources. “Great, let's go! And now we want faster, more. Can’t you? Is it impossible to deliver twice as much in a year? And why? ”Such growth problems are standard, probably for many things, many approaches, and they are felt.

Regarding monitoring. It seems to me that services or industrial monitoring tools are already learning or are able to work with Docker and Kubernetes in a different, non-standard mode. So that you, for example, do not have 500 Java machines under which all this is running, namely, aggregates. But these products still lack maturity, they have to go through this. The topic is really new, it will still be developed.

Dmitry:

Yes, very interesting. And this applies to HR. Perhaps your HR process and HR brand have changed a bit over these 3 years. You began to recruit other people with a different competency. And there are probably pros and cons. Previously, blockchain and data science were high-tech, and specialists in them were worth just millions. Now the price is falling, the market is saturated, and there is a similar trend in microservices.

Sergei:

Yes, absolutely.

Alexander:

HR asks the question: “Where is your pink unicorn between the backend and frontend?” HR does not understand what a microservice is. We revealed a secret to them and said that this backend did everything, and there is no unicorn. But HR is changing, quickly learning and recruiting people who possess basic IT knowledge.

The evolution of microservices

Dmitry:

If you look at the target architecture, then microservices look like such a monster. Your journey took several years. Others have a year, others have three years. You foresaw all the problems, the target architecture, did something change? For example, in the case of microservices, gateways, service meshes now appear again. Did you use them at the beginning or did you change the architecture itself? Do you have any such challenges?

Sergei:

We have already rewritten several interaction protocols. Initially, one protocol, now switched to another. We increase safety and reliability. We started with enterprise-technology - Oracle, Web Logic. Now we are moving away from technological enterprise products in microservices and moving to open source or to completely open technologies. We abandon the databases, move on to what works more efficiently for us in this model. We no longer need Oracle technologies.

We started just as a service, without thinking about how much we need the cache, what we will do when there is no connection with the microservice, and the data is needed, etc. Now we are developing a platform so that the architecture can be described not in the language of services, and in business language, take business logic to the next level when we start talking in words. Now we have learned to speak letters, and the next level is when the services will be assembled into some kind of aggregate, when this is already a word - for example, an entire product card. It is built from microservices, but it’s such an add-on for this API.

Security is very important. As soon as you begin to be accessible and you have a service by which you can get a lot of interesting things, and very quickly, in a split second, you want to get this in a not the safest way. To get away from this, I had to change approaches to testing and monitoring. I had to change the team, supply management structure, CI / CD.

This evolution is like with phones, only much faster: first there were push-button phones, then smartphones appeared. We rewrote, remade the product because the market had a different need. So we are evolving: first grade, tenth grade, work.

Iteratively in the year something is laid in terms of technology, something else - in terms of backlog and needs. We connect one with the other. The team spends 20% on the technical debt and technical support of the team, 80% on the business entity. And we are moving, with an understanding of why we are doing it, why we are making these technological improvements, what they will lead to. Like that.

Dmitry:

Cool. And what about MegaFon?

Alexander:

The main challenge during our arrival at microservices is not to fall into chaos. MegaFon’s architectural office immediately joined, even became the initiator and driver, now we have a very strong architecture. His task was to understand what target model we are going to and what technologies need to be piloted. With architecture, we carried out these pilots ourselves.

The next question was: "How then is it to exploit all this?" And one more: "How to ensure transparent interaction between microservices?". The service mesh helped us answer the last question. We piloted the Istio, and we liked the results. Now we are at the stage of rolling into productive zones. We take all the challenges positively - the fact that you need to constantly change the stack, learn something new. We are interested in developing, not exploiting, old solutions.

Dmitry:

Golden words! Such challenges keep the team and business in good shape and create the future. The GDPR has created chief data protection officers, and current challenges create major ones for microservices and architecture. And it pleases.

We have discussed a lot. The main thing - a good study of microservices and the architecture itself avoids many errors. Of course, the process is iterative and evolutionary, but the future lies with it.

Thanks to all participants, thanks to Sergey and Alexander!

Questions from the audience

Question from the audience (1):

Sergey, how has IT management changed in your company? I understand when there is a large stack of several systems, then how it is managed is a fairly understandable and logical process. How did you rebuild management in the IT component after a very large number of microservices were integrated in such a short time?

Sergey:

I will join my colleague that architecture as a driver of changes is very important. We started with the fact that we have an architectural unit. Architects at the same time are the owners of the distribution of functionality and requirements for how it will look in the landscape. So they also act as coordinators of these changes. As a result, specific changes occurred in the specific delivery process when we created the CI / CD platform.

But no one canceled the standard, basic principles of development, business analysis, testing and development. We just added speed. Previously, the cycle took so much, installation on test environments - so much more. Now the business sees a profit and says: “Why can't you do the same in other places?”

It’s like, in a good way, an injection in the form of a vaccine, which showed: it can be like this, but it can be different. Of course, there is a problem in personnel, in competencies, in knowledge, in resistance.

Question from the audience (2):

Critics of microservice architecture say that there are difficulties with testing and development. This is logical where something gets complicated. What difficulties did your team face and how did you overcome them? Question to all.

Alexander:

There are difficulties in the transition from microservices to the platform, but they are solvable.

For example, we make a product that consists of 5-7 microservices. We need to provide integration tests throughout the microservice scopes to give a green light to the transition to the master branch. This task was not new for us: we did this for a long time in BSS, when the vendor supplied us with the solutions already shipped.

And our problem is only in a small team. One conditional product needs one QA engineer. And now, we ship a product from 5-7 microservices, of which 2-3 can be developed by outsiders. For example, we have a product in the development of which our vendor of the billing system, Mail.ru Group and R&D MegaFon participate. We need to cover this with tests before shipping to the prod. For a month and a half, a QA engineer has been involved in this product, and the rest of the team is left without its support.

This complexity is caused only by scaling. We understand that microservices cannot exist in a vacuum, absolute isolation does not exist. Changing one service, we are sure to try to keep the API contract. If something changes under the hood, the front of the service remains. If the changes are fatal, there is some kind of architectural transformation and we generally switch to another data metamodel, which is completely incompatible - only then we are talking about the API specification of the v2 service. We support both the first and second versions, and after all consumers have switched to the second version, we simply close the first one.

Sergei:

I want to complement. I absolutely agree about the complications - they occur. The landscape is getting complicated, overheads are growing, especially for testing. How to deal with this: switch to auto-testing. Yes, you will have to invest in writing autotests and unit tests. So that developers could not commit without passing the test, they could not change the code. So that even the push button does not work without an autotest, unit test.

It is important to maintain the previous functionality, and this is an additional overhead. If you rewrite the technology to another protocol, then you rewrite until you close the whole thing.

We sometimes do not specifically do end-to-end testing, as we do not want to stop development, although we also cling to one another. The landscape is very large, complex, there are many systems. Sometimes just stubs - yes, you lower the security margin, more risks appear. But at the same time release the delivery.

Alexander:

Yes, autotests and unit tests allow you to make a quality service. We are for a pipeline that cannot be completed without unit- and integration tests. It is often necessary to drag emulators, systems with sales into test zones and development environments, because not all systems can be placed in test zones. Moreover, they do not just get wet - we generate a complete response from the system. This is a serious part of working with microservices, and we invest in it too. Without this, chaos will come.

Question from the audience (3):

As far as I understand, initially microservices grew from a separate team and now exist in this model. What are its pros and cons?

We just have a similar story: a certain microservice factory has emerged. Now we have conceptually come to the fact that we are extending this approach to production by streams and systems. In other words, we are moving away from the centralized development of microservices, microservice models, and are getting closer to systems.

Accordingly, our operation also goes to systems, that is, we decentralize this topic. What is your approach and what is the target story for you?

Alexander:

The name "microservice factory" you removed from the language - we also want to scale. Firstly, we really have one team now. We want to provide all development teams that are in MegaFon with the opportunity to work in a common ecosystem. We do not want to completely take on all the development functionality that is now. The local task is to scale out, the global task is to develop all the teams in the microservice layer.

Sergey:

I’ll tell you which way we went. We really started working as one team, but now it is not alone. I am a supporter of the following: there must be an owner of the process. Someone needs to understand, manage, control and build the development process for microservices. He must own resources and engage in resource management.

These resources, which know the technology, specifics and understand how to build microservices, can be found in product teams. We have a mix where people from the microservice platform are in the product team that makes the mobile application. They are there, but they are working on the microservice platform management department process with their development manager. Inside this unit there is a separate team that deals with technology. That is, we mix together a common pool of resources and share them, giving them inside teams.

At the same time, the process remains general, controlled, it proceeds according to general technological principles, with unit testing and so on - everything that is built on top. There may be columns in the form of resources collected from different units of the product approach.

Alexander:

Sergey, you actually own the process, right? Is the backlog of tasks common? Who is involved in its distribution?

Sergey:

Look: here is the mix again. There is a backlog, which is formed on the basis of technological improvements - this is one story. There is a backlog that is formulated from projects, and there is a backlog from products. But the sequence of putting inside each of the products of the service or the creation of this service is developed by the product manager. He is not in the IT-directorate, he is specially removed from it. But my people definitely work on the same process.

The owner of the backlog in different directions - the backlog of changes - will be different people. The connectedness of technological services, their organization principle - all this will be in IT. I own the platform, and resources too. Above is what concerns the backlog and functional changes, and architecture in that sense.

Suppose a business says: “We want this function, we want to create a new product - redo the loan.” We answer: "Yes, we will redo it." Architects say: “Let's think: where do we enter microservices in the loan and how do we do it?” Then we lay it out on projects, products or on the technological stack, put it in teams and implement it. Created a product inside and decided to use microservices in this product? We say: "Now the legacy-systems that we had, or front-line, should go to these microservices." Architects say: “So: into the technological backlog inside front-line products - the transition to microservices. Go". And product managers or business owners understand how much capacity is allotted when it will be done and why.

End of discussion, but not all

The mailto: CLOUD conference was hosted by Mail.ru Cloud Solutions .

We also do other events - for example, @Kubernetes Meetup , where we always look for cool speakers:

• Follow the news of @Kubernetes and other @Meetup in our Telegram channel t.me/k8s_mail
• Want to speak at one of @Meetup? Leave a request at mcs.mail.ru/speak