Microservices: size matters, even if you have Kubernetes
On September 19, the first thematic HUG (Highload ++ User Group) meeting took place in Moscow , which was dedicated to microservices. It included the report “Operation of microservices: size does matter, even if you have Kubernetes”, in which we shared the extensive experience of the company “Flant” in the area of operating projects with microservice architecture. First of all, it will be useful to all developers who are thinking about applying this approach in their present or future project.
We present the video with the report (50 minutes, much more informative than the article), as well as the main squeeze out of it in text form.
NB: Videos and presentations are also available at the end of this publication.
Usually a good story has a plot, a main plot and a denouement. This report is more like a plot, and a tragic one. It is also important to note that it presents a look at microservices from the exploitation side .
I'll start with this schedule, the author of which (in 2015) became Martin Fowler:
For it is evident, as in the case of a monolithic application, which achieved a certain value begins to decrease productivity. Microservices are different in that the initial productivity with them is lower, however, as the complexity increases, the degradation of efficiency is not so noticeable for them.
I will add to this schedule for the case of using Kubernetes:
Why is the application with microservices better? Because such an architecture places serious demands on the architecture, which, in turn, are perfectly covered by the possibilities of Kubernetes. On the other hand, a part of this functionality will be useful for the monolith, especially since the typical monolith today is not quite a monolith (details will be given later in the report).
As you can see, the final schedule (when both monolithic and microservice applications in the infrastructure with Kubernetes) is not very different from the original one. Further, we will talk about applications that are operated using Kubernetes.
And then the main thought:
What is a normal microservice architecture? It should bring you real benefits by increasing your work efficiency. If you go back to the schedule, here it is:
If you call it useful , then on the other side of the schedule there will be harmful microservice (interferes with work):
Returning to the “main idea”: is it worth trusting my experience at all? Since the beginning of this year I have looked at 85 projects.. Not all of them were microservice (approximately one third to one half of them had such architecture), but this is still a large number. We (the company "Flant") as outsourcers manage to see a wide variety of applications developed both in small companies (with 5 developers) and large ones (~ 500 developers). An additional advantage is that we see how these applications live and develop over the years.
When asked about the benefits of microservices, there is a very specific answer to the already mentioned Martin Fowler:
I talked a lot with architects and software developers and asked why they need microservices. And he made his list of their expectations. Here's what happened:
If we describe “in sensations” some of the points, then:
To explain why, in reality, things are not as we expect, I will present a collective image of microservice architecture based on the experience of many different projects.
An example would be an abstract online store that is going to compete with Amazon or at least OZON. Its microservice architecture looks like this:
For a combination of reasons, these microservices are written on different platforms:
Since each microservice should have autonomy, many of them need their own database and cache. The final architecture is obtained as follows:
Fowler has an article on this subject - about “payback” for using microservices:
And we will see if our expectations were met.
But how many microservices do we really need to fix in order to roll out the change? Can we even figure out how everything works, without a distributed tracer (after all, any request is processed by half of microservices)?
There is a “ big clump of dirt ” pattern , and here we have a distributed clump of dirt. In confirmation of this, here’s a rough illustration of how requests go:
Technically, it has been achieved: we can roll each microservice separately. But in practice it is necessary to take into account that many microservices always roll out , and we need to take into account the order of their rollout . In an amicable way, we generally need to test in a separate circuit, whether we roll out the release in the correct order.
She is. Only it is worth remembering that often freedom borders on lawlessness. It is very important here not to choose technologies only in order to “play” with them.
How to make a test loop for the entire application (from so many components)? But still need to keep it up to date. All this leads to the fact that the actual number of test circuits , which in principle we can contain, turns out to be minimal .
And to deploy all this locally? .. It turns out that often the developer does his work independently, but “at random”, because he has to wait for the circuit to be released for testing.
Yes, but it is limited in the area of the used DBMS. In the given example of architecture, Cassandra will have no problems, but MySQL and PostgreSQL will have problems.
Moreover, in reality, disabling one microservice often breaks the correct functioning of the entire system, so there is also a new problem: every microservice is very difficult to make fault-tolerant . Because microservices use different technologies (memcache, Redis, etc.), everyone needs to think over and implement everything, which, of course, is possible, but requires huge resources.
This is really all good.
Not only did we have huge network overheads (multiply queries for DNS, etc.), but also because of the many subqueries we began to replicate data (store caches), which led to a significant amount of storage.
And here is the result of meeting our expectations:
Because:
Start with a monolithic application . Experience Fowler'a says that almost all successful microservice applications began with a monolith, which became too large, after which it was broken. At the same time, almost all systems built as microservices from the very beginning, sooner or later experienced serious problems.
Another valuable thought is that in order to succeed in a microservice architecture project, you should know the subject area very well , and how to make microservices . And the best way to know the subject area is to make a monolith.
The first step to solving any problem is to agree with it and understand that this is a problem, that we no longer want to suffer.
If in the case of an overgrown monolith (when we’re out of the opportunity to re-buy resources for it), we cut it, then in this case we get the opposite story: when excessive microservice doesn’t help anymore, but hinders, cut off the excess and enlarge !
For example, for the collective image discussed above ...
Get rid of the most dubious microservices:
Combine all microservices responsible for the generation of the frontend:
... into one microservice written in one (modern and normal, as you yourself think) language / framework:
It will have one ORM (one DBMS) and first a couple of applications:
... but in general there can be transferred much more, having obtained the following result:
Moreover, we start all this in Kubernetes in separate instances, which means that we can still measure the load and scale them separately.
Look at the picture wider. Very often, all these problems with microservices arise from the fact that someone took his task, but wanted to "play microservices."
In the word “microservices” the part “micro” is superfluous . They are “micro” only for the reason that they are smaller than a huge monolith. But do not think of them as something small.
And for the final thought, let us return to the original schedule: A
note written to it (top right) means that the skills of the team that makes your project are always primary - they will play a key role in your choice between microservices and monolith. If a team does not have enough skills, but it starts doing microservices, the story will definitely be fatal.
Video from the speech (~ 50 minutes; unfortunately, it does not convey the numerous emotions of visitors, which largely determined the mood of the report, but as it is):
Presentation of the report:
Other reports on our blog:
You may also be interested in the following publications:
We present the video with the report (50 minutes, much more informative than the article), as well as the main squeeze out of it in text form.
NB: Videos and presentations are also available at the end of this publication.
Introduction
Usually a good story has a plot, a main plot and a denouement. This report is more like a plot, and a tragic one. It is also important to note that it presents a look at microservices from the exploitation side .
I'll start with this schedule, the author of which (in 2015) became Martin Fowler:
For it is evident, as in the case of a monolithic application, which achieved a certain value begins to decrease productivity. Microservices are different in that the initial productivity with them is lower, however, as the complexity increases, the degradation of efficiency is not so noticeable for them.
I will add to this schedule for the case of using Kubernetes:
Why is the application with microservices better? Because such an architecture places serious demands on the architecture, which, in turn, are perfectly covered by the possibilities of Kubernetes. On the other hand, a part of this functionality will be useful for the monolith, especially since the typical monolith today is not quite a monolith (details will be given later in the report).
As you can see, the final schedule (when both monolithic and microservice applications in the infrastructure with Kubernetes) is not very different from the original one. Further, we will talk about applications that are operated using Kubernetes.
Useful and harmful microservice
And then the main thought:
What is a normal microservice architecture? It should bring you real benefits by increasing your work efficiency. If you go back to the schedule, here it is:
If you call it useful , then on the other side of the schedule there will be harmful microservice (interferes with work):
Returning to the “main idea”: is it worth trusting my experience at all? Since the beginning of this year I have looked at 85 projects.. Not all of them were microservice (approximately one third to one half of them had such architecture), but this is still a large number. We (the company "Flant") as outsourcers manage to see a wide variety of applications developed both in small companies (with 5 developers) and large ones (~ 500 developers). An additional advantage is that we see how these applications live and develop over the years.
Why microservices?
When asked about the benefits of microservices, there is a very specific answer to the already mentioned Martin Fowler:
- clear limits of modularity;
- independent deployment;
- freedom of choice of technology.
I talked a lot with architects and software developers and asked why they need microservices. And he made his list of their expectations. Here's what happened:
If we describe “in sensations” some of the points, then:
- clear borders of the modules: here we have a terrible monolith, and now everything will be neatly laid out in Git-repositories, in which everything is “on the shelves”, not warm and soft;
- Deployment independence: we will be able to roll out services independently so that development goes faster (in parallel, publish new features);
- development independence: we can give this microservice to that team / developer, and that one is different, thanks to which we can develop it faster;
- b of lshaya reliability: if there will be a partial degradation (fall mikroservis one out of 20), then stop working, only one button, and the system as a whole will continue to function.
Typical (harmful) microservice architecture
To explain why, in reality, things are not as we expect, I will present a collective image of microservice architecture based on the experience of many different projects.
An example would be an abstract online store that is going to compete with Amazon or at least OZON. Its microservice architecture looks like this:
For a combination of reasons, these microservices are written on different platforms:
Since each microservice should have autonomy, many of them need their own database and cache. The final architecture is obtained as follows:
What are its consequences?
Fowler has an article on this subject - about “payback” for using microservices:
And we will see if our expectations were met.
Clear module boundaries ...
But how many microservices do we really need to fix in order to roll out the change? Can we even figure out how everything works, without a distributed tracer (after all, any request is processed by half of microservices)?
There is a “ big clump of dirt ” pattern , and here we have a distributed clump of dirt. In confirmation of this, here’s a rough illustration of how requests go:
Deployment independence ...
Technically, it has been achieved: we can roll each microservice separately. But in practice it is necessary to take into account that many microservices always roll out , and we need to take into account the order of their rollout . In an amicable way, we generally need to test in a separate circuit, whether we roll out the release in the correct order.
Freedom to choose technology ...
She is. Only it is worth remembering that often freedom borders on lawlessness. It is very important here not to choose technologies only in order to “play” with them.
Development independence ...
How to make a test loop for the entire application (from so many components)? But still need to keep it up to date. All this leads to the fact that the actual number of test circuits , which in principle we can contain, turns out to be minimal .
And to deploy all this locally? .. It turns out that often the developer does his work independently, but “at random”, because he has to wait for the circuit to be released for testing.
Separate scaling ...
Yes, but it is limited in the area of the used DBMS. In the given example of architecture, Cassandra will have no problems, but MySQL and PostgreSQL will have problems.
B of lshaya reliability ...
Moreover, in reality, disabling one microservice often breaks the correct functioning of the entire system, so there is also a new problem: every microservice is very difficult to make fault-tolerant . Because microservices use different technologies (memcache, Redis, etc.), everyone needs to think over and implement everything, which, of course, is possible, but requires huge resources.
Load measurability ...
This is really all good.
“Lightness” of microservices ...
Not only did we have huge network overheads (multiply queries for DNS, etc.), but also because of the many subqueries we began to replicate data (store caches), which led to a significant amount of storage.
And here is the result of meeting our expectations:
But that's not all!
Because:
- Most likely we will need a message bus.
- How to make a consistent backup at the right time? The only real option is to turn off traffic for this. But how to do this in production?
- If we are talking about supporting several regions, then organizing sustainability in each of them is a very time consuming task.
- There is a problem of making centralized changes. For example, if we need to update the version of PHP, then we need to commit to each repository (and there are dozens of them).
- The growth of operational complexity offhand turns out to be exponential.
What to do with all this?
Start with a monolithic application . Experience Fowler'a says that almost all successful microservice applications began with a monolith, which became too large, after which it was broken. At the same time, almost all systems built as microservices from the very beginning, sooner or later experienced serious problems.
Another valuable thought is that in order to succeed in a microservice architecture project, you should know the subject area very well , and how to make microservices . And the best way to know the subject area is to make a monolith.
But what if we are already in this situation?
The first step to solving any problem is to agree with it and understand that this is a problem, that we no longer want to suffer.
If in the case of an overgrown monolith (when we’re out of the opportunity to re-buy resources for it), we cut it, then in this case we get the opposite story: when excessive microservice doesn’t help anymore, but hinders, cut off the excess and enlarge !
For example, for the collective image discussed above ...
Get rid of the most dubious microservices:
Combine all microservices responsible for the generation of the frontend:
... into one microservice written in one (modern and normal, as you yourself think) language / framework:
It will have one ORM (one DBMS) and first a couple of applications:
... but in general there can be transferred much more, having obtained the following result:
Moreover, we start all this in Kubernetes in separate instances, which means that we can still measure the load and scale them separately.
Summarizing
Look at the picture wider. Very often, all these problems with microservices arise from the fact that someone took his task, but wanted to "play microservices."
In the word “microservices” the part “micro” is superfluous . They are “micro” only for the reason that they are smaller than a huge monolith. But do not think of them as something small.
And for the final thought, let us return to the original schedule: A
note written to it (top right) means that the skills of the team that makes your project are always primary - they will play a key role in your choice between microservices and monolith. If a team does not have enough skills, but it starts doing microservices, the story will definitely be fatal.
Video and slides
Video from the speech (~ 50 minutes; unfortunately, it does not convey the numerous emotions of visitors, which largely determined the mood of the report, but as it is):
Presentation of the report:
PS
Other reports on our blog:
- “ Monitoring and Kubernetes ” (Dmitry Stolyarov; May 28, 2018 at RootConf) ;
- “ Best CI / CD practices with Kubernetes and GitLab ” (Dmitry Stolyarov; November 7, 2017 on HighLoad ++) ;
- “ Our experience with Kubernetes in small projects ” (Dmitry Stolyarov; June 6, 2017 at RootConf) ;
- “ We assemble Docker images for CI / CD quickly and conveniently with dapp ” (Dmitry Stolyarov; November 8, 2016 on HighLoad ++) ;
- “ Practices of Continuous Delivery with Docker ” (Dmitry Stolyarov; May 31, 2016 at RootConf) .
You may also be interested in the following publications: