January 29, 2018 at 08:522018 Microservice Madness Death
- Transfer
Note perev. : This material, written by an experienced developer, is not intended to bury the idea of microservices, as you might think, looking at the title. This article is a reasonable warning for those who have decided that microservices are a “silver bullet”, which in itself solves all architectural and operational problems. To demonstrate this, the author collected and systematized the popular problems often encountered in today's projects that already use microservices or migrate to them.
In recent years, microservices have become a very popular topic. Microservice Madness looks something like this:
There are a great many cases when significant efforts have been made to introduce microservice patterns without necessarily understanding what minuses and pluses this will lead to in the context of the particular problem being solved.
I will tell you in detail about what microservices are, why their pattern is so attractive, and what are some of the main difficulties in the way of their use. I will end with simple questions that I should ask myself when considering the relevance of the microservice pattern for your needs. These questions are presented at the end of the article.
Let's start with the basics. Here is an example of a possible implementation of a hypothetical video sharing platform: first in a monolithic representation (one large block), and then in a microservice.
The difference between the two systems is that the first is a single large block, i.e. monolith. The second is a set of small specific services. Each service has its own specific role.
When a diagram is presented at such a level of detail, it is easy to see its attractiveness. There is a whole set of potential advantages:
Independent development . Small independent components can be created by small independent teams. A group can work on changes to the Upload service without affecting the Transcode service.and not even knowing about him. The amount of time required to study the component is significantly reduced, and it becomes easier to develop new functions.
Independent Deployment . Each individual component can be deployed independently. This allows you to release new features quickly and with less risk. Corrections or features for the Streaming component can be deployed without the need to deploy other components.
Independent scalability . Each component can be scaled independently of the other. During high user demand, when new transfers are coming out, the Download component can be scaled to increase the load without the need to scale each component, which makes scaling flexible and reduces costs.
Reusable . Components implement their small specific function. This means that they are easier to adapt for use in other systems, services or products. The Transcode component can be used by other business divisions or even turned into a new business offering transcoding services to a different audience.
At this level of detail, the advantages of the microservice model over the monolithic model seem obvious. If so, then why is the pattern now in vogue? Where has he been all my life?
There are two answers to this question. The first - they actually did , to the extent of technical capabilities. Second, recent technological improvements have allowed us to take this approach to a new level.
When I started preparing the answer to this question, I got a long description, so it’s better to single it out in a separate article and publish it a little later. Here, the path from one program to many will be omitted, ESB (enterprise service bus, ie, “enterprise service bus” - approx. Transl.) And SOA (service-oriented architecture, ie “service-oriented architecture” - approx. Transl.) , Component design and bounded contexts (“limited context” - a pattern from Domain-Driven Design - approx. Transl.) , Etc.
Instead of all this, I’ll write that in many ways we’ve done it all for some time, but only with the recent explosive growth of container technologies (especially Docker) and orchestration technologies (such as Kubernetes, Mesos, Consul, etc.) the pattern has become much more appropriate to implement from a technical point of view.
If you take for granted the possibility of implementing a microservice approach, you need to carefully think about the need . We saw high-level theoretical advantages, but what about the difficulties?
If microservices are so beautiful, what's the matter? Here are some of the most significant issues that I have seen.
Жизнь разработчиков может стать значительно тяжелее. Если разработчик захочет поработать над фичей, затрагивающей множество сервисов [journey], ему придётся запускать все их на своей машине и подключаться к ним. Зачастую это сложнее, чем просто запустить одну программу.
Эту сложность можно частично облегчить с помощью соответствующих инструментов [tooling], но чем больше количество сервисов, составляющих систему, тем больше сложностей у разработчиков будет возникать при запуске системы в целом.
Teams that do not develop services, but support them, will experience an explosion of potential complexity. Instead of possibly managing several running services, they will have to work with dozens, hundreds or thousands. More services, more ways to interact and more opportunities for potential problems.
Looking at the two previous points, it may seem - especially because of the popularity of DevOps as a practice (of which I am a big supporter) - that operation and development are considered separately. So is it not getting better thanks to DevOps?
The difficulty is that many organizations still have separate development and operation teams, and in such cases they will most likely have a hard time adapting microservices.
For organizations already deploying DevOps, things are not that simple either. Being both a developer and a sysadmin is already difficult (but critical for creating good software), and with the need to also understand the nuances of container orchestration systems and especially systems that are developing rapidly, is even more difficult. So I come to the next point.
The results can be great if the work was done by experts. But imagine an organization in which not everything is perfect with the work of a single monolithic system. Why, with an increase in the number of systems, which complicates the operation, will the situation become better?
Yes, with efficient automation, monitoring, orchestration, etc. - all this is possible. However, technology is rarely a difficulty - usually it’s the search for people who can use it effectively. The demand for such skills is huge now, finding them is not easy.
In all the examples used to describe the benefits of microservices, they were talking about independent components. However, in many cases, the components are not simply independent. If certain areas of paper may look connected, then in reality, when you dig into all the details, it is easy to find that everything is much more complicated than in the proposed model.
This is where everything becomes very complex. If the boundaries are not really well defined, it will happen that - even in the case of the theoretical possibility of an isolated deploy of services - mutual dependencies between services will pop up, because of which you will have to deploy sets of services as a group.
In turn, this means that it is necessary to maintain consistent versions of services that have been verified and tested in working with each other. It turns out that there is really no system, parts of which can be deployed independently, because for deploying a new feature you will have to carefully manage the deployment of many services.
In the previous example, the sometimes arising need for simultaneous roll-out of many versions of many services is mentioned when the feature is deployed. I want to say that thoughtful deployment techniques make life easier: for example, a blue-green deployment [blue / green deployments] (require minimal effort on most platforms for orchestrating services) or parallel launch of many versions of a service with the option of choosing the right one for their user.
These techniques remove many obstacles if the services are stateless. But working with stateless services is actually quite simple. Actually, if you have stateless services, I recommend thinking about skipping all these microservices and using the serverless model.
In reality, many services need to keep state[state] . An example for a video sharing platform is the subscription service. Let the new version of the subscription service store data in a new kind of database. If both services are running in parallel, then you have a system that simultaneously uses two schemes. If you make a blue-green deployment, and other services depend on the data in a new form, they must be updated at the same time. If the deployment of the subscription service failed and rolls back, then most likely it is necessary to roll back these services too, and then "cascade".
Again, it is tempting to think that with databases from the NoSQL world, similar schema-related problems will go away, but this is not so. Databases that do not require strict schemas do not mean that the system does not have schemas, because in essence it simply means the need to manage the schema at the application level, and not at the DBMS level. The fundamental problem of understanding the nature of your data and how it is changing cannot be eradicated.
Since you are creating a large network of services that are dependent on each other, a large field for interservice interaction is likely to appear. There are a number of difficulties. Firstly, there are more places for potential failures. It is necessary to anticipate the possibility that network calls will not work, that is, every time one service accesses another, it should at least try to repeat its attempts. When a service needs to access many services, the situation will become even more complicated.
Imagine a user uploading a video to a sharing service. We need to start the download service, transfer data to the transcoding service, update subscriptions, update recommendations and so on. All these challenges require a certain orchestration, and if something breaks, the actions need to be repeated.
The logic of repeated calls can become difficult to manage. Attempts to do everything synchronously often fail, because they have too many points of failure. In this case, the use of asynchronous interaction patterns will become a more reliable solution. But the difficulty is that due to asynchronous patterns the system becomes stateful. And as mentioned in the last paragraph, it is very difficult to manage stateful systems and systems with distributed states.
When message queues are used in a microservice system for interservice communication, you basically have a large database (message queue or broker) that sticks together all services. Again, although all this at first does not seem to be a problem, the circuit will catch up with you and remind yourself. A version X service can write messages in a specific format, so services that depend on such a message also need to be updated when the sender service changes something in its message.
You can have services that process messages in many different formats, but they are difficult to manage. When deploying new versions of services, it will happen that two versions of the service try to process messages from the same queue and, possibly, even from sender services of different versions as well. This can lead to confusing, inappropriate situations. It may turn out that to avoid them it is easier to allow only certain versions of messages to exist: this means that you have to deploy consistent sets of versions for sets of services, ensuring that messages of older versions are processed accordingly (first).
This again confirms the idea that, if you go into details, independent deployments do not always work as intended.
To overcome the mentioned obstacles, you must very carefully manage versions. Again, there is a tendency to believe that following a standard like semver will solve the problem. This is not true. Semver is an agreement that is worth using, but you still have to keep track of the versions of services and APIs that may interact.
You can quickly come to significant difficulties and find yourself in a situation where you do not know which versions of the services actually work correctly with each other.
It is well known how difficult it is to manage dependencies in software systems, be it modules for Node or Java, libraries for C, etc. It is very difficult to deal with conflicts of independent components used by one entity.
It is difficult to cope with these problems when dependencies are static and can be patched, updated, edited and so on. If the working services act as dependencies, you can’t just update them: you will need to launch several versions (with the problems described above) or suspend the system until everything is fixed.
In situations where transactional integrity is required during operation, microservices can be a big pain. It is not easy to work with distributed state, and many small components that can break make transaction orchestration really difficult.
It may seem attractive to try to avoid the problem by making operations idempotent, suggesting retry mechanisms, etc. - and in many cases it will work. However, there may be scenarios in which a successful or unsuccessful transaction is simply needed, without an intermediate state. The complexity of a workaround for this task or its implementation in a microservice model can be very high.
Yes, certain services and components can be deployed in isolation, but in most cases some kind of platform for orchestration like Kubernetes will be launched. If you use a managed service like GKE from Google or EKS from Amazon, most of the complexities of managing a cluster are solved for you.
However, if you manage the cluster yourself, then you are working with a large, complex, and critical system. Although individual services may have all the advantages described above, you need to carefully manage the cluster. Deploying such a system can be difficult, upgrades can be complicated, failover can be complex, and so on.
In many cases, the overall benefits remain, but it’s important not to simplify or underestimate the additional complexity caused by managing another large and complex system. Managed services can help, but they are often too young (for example, Amazon EKS was announced only at the end of 2017).
Avoid insanity by making prudent and deliberate decisions. And to help you do this, I have prepared several questions that you can ask yourself and explanations of the answers to them. Note perev. : The original image is translated and presented in text form below.
Is it possible to seat your whole team at one big table?
Is your system predominantly stateless?
Are you building a solution for one application or service?
Do you have monolithic dependencies?
Do you have experts in containers, orchestration, DevOps?
Download original PDF with all the questions and answers (in English) can be here .
I deliberately avoided the word “a” in this article. But my friend Zoltan , checking her, made a very good point.
Microservice architecture does not exist. Microservices are just another pattern or implementation of components, no more, no less. Whether they are represented in the system or not, does not mean that the system architecture is ready.
Microservices in many respects are more related to technical processes related to packaging and operation than to the system architecture itself. Appropriate boundaries for components remain one of the main challenges in engineering systems.
Regardless of the size of your services, whether they are in Docker containers or not - you always need to think carefully about how to put the system together. There are no right answers, but there are many options.
Read also in our blog:
In recent years, microservices have become a very popular topic. Microservice Madness looks something like this:
“Netflix is good at DevOps. Netflix make microservices. So, if I'm doing microservices, I'm good at DevOps. "
There are a great many cases when significant efforts have been made to introduce microservice patterns without necessarily understanding what minuses and pluses this will lead to in the context of the particular problem being solved.
I will tell you in detail about what microservices are, why their pattern is so attractive, and what are some of the main difficulties in the way of their use. I will end with simple questions that I should ask myself when considering the relevance of the microservice pattern for your needs. These questions are presented at the end of the article.
What are microservices and why are they so popular?
Let's start with the basics. Here is an example of a possible implementation of a hypothetical video sharing platform: first in a monolithic representation (one large block), and then in a microservice.
The difference between the two systems is that the first is a single large block, i.e. monolith. The second is a set of small specific services. Each service has its own specific role.
When a diagram is presented at such a level of detail, it is easy to see its attractiveness. There is a whole set of potential advantages:
Independent development . Small independent components can be created by small independent teams. A group can work on changes to the Upload service without affecting the Transcode service.and not even knowing about him. The amount of time required to study the component is significantly reduced, and it becomes easier to develop new functions.
Independent Deployment . Each individual component can be deployed independently. This allows you to release new features quickly and with less risk. Corrections or features for the Streaming component can be deployed without the need to deploy other components.
Independent scalability . Each component can be scaled independently of the other. During high user demand, when new transfers are coming out, the Download component can be scaled to increase the load without the need to scale each component, which makes scaling flexible and reduces costs.
Reusable . Components implement their small specific function. This means that they are easier to adapt for use in other systems, services or products. The Transcode component can be used by other business divisions or even turned into a new business offering transcoding services to a different audience.
At this level of detail, the advantages of the microservice model over the monolithic model seem obvious. If so, then why is the pattern now in vogue? Where has he been all my life?
If everything is so cool, why hasn’t anyone done this before?
There are two answers to this question. The first - they actually did , to the extent of technical capabilities. Second, recent technological improvements have allowed us to take this approach to a new level.
When I started preparing the answer to this question, I got a long description, so it’s better to single it out in a separate article and publish it a little later. Here, the path from one program to many will be omitted, ESB (enterprise service bus, ie, “enterprise service bus” - approx. Transl.) And SOA (service-oriented architecture, ie “service-oriented architecture” - approx. Transl.) , Component design and bounded contexts (“limited context” - a pattern from Domain-Driven Design - approx. Transl.) , Etc.
Instead of all this, I’ll write that in many ways we’ve done it all for some time, but only with the recent explosive growth of container technologies (especially Docker) and orchestration technologies (such as Kubernetes, Mesos, Consul, etc.) the pattern has become much more appropriate to implement from a technical point of view.
If you take for granted the possibility of implementing a microservice approach, you need to carefully think about the need . We saw high-level theoretical advantages, but what about the difficulties?
What is the problem of microservices?
If microservices are so beautiful, what's the matter? Here are some of the most significant issues that I have seen.
Increased complexity for developers
Жизнь разработчиков может стать значительно тяжелее. Если разработчик захочет поработать над фичей, затрагивающей множество сервисов [journey], ему придётся запускать все их на своей машине и подключаться к ним. Зачастую это сложнее, чем просто запустить одну программу.
Эту сложность можно частично облегчить с помощью соответствующих инструментов [tooling], но чем больше количество сервисов, составляющих систему, тем больше сложностей у разработчиков будет возникать при запуске системы в целом.
Возросшая сложность для эксплуатации
Teams that do not develop services, but support them, will experience an explosion of potential complexity. Instead of possibly managing several running services, they will have to work with dozens, hundreds or thousands. More services, more ways to interact and more opportunities for potential problems.
Increased complexity for DevOps
Looking at the two previous points, it may seem - especially because of the popularity of DevOps as a practice (of which I am a big supporter) - that operation and development are considered separately. So is it not getting better thanks to DevOps?
The difficulty is that many organizations still have separate development and operation teams, and in such cases they will most likely have a hard time adapting microservices.
For organizations already deploying DevOps, things are not that simple either. Being both a developer and a sysadmin is already difficult (but critical for creating good software), and with the need to also understand the nuances of container orchestration systems and especially systems that are developing rapidly, is even more difficult. So I come to the next point.
Serious Competency Needed
The results can be great if the work was done by experts. But imagine an organization in which not everything is perfect with the work of a single monolithic system. Why, with an increase in the number of systems, which complicates the operation, will the situation become better?
Yes, with efficient automation, monitoring, orchestration, etc. - all this is possible. However, technology is rarely a difficulty - usually it’s the search for people who can use it effectively. The demand for such skills is huge now, finding them is not easy.
Real systems usually do not have clearly defined boundaries
In all the examples used to describe the benefits of microservices, they were talking about independent components. However, in many cases, the components are not simply independent. If certain areas of paper may look connected, then in reality, when you dig into all the details, it is easy to find that everything is much more complicated than in the proposed model.
This is where everything becomes very complex. If the boundaries are not really well defined, it will happen that - even in the case of the theoretical possibility of an isolated deploy of services - mutual dependencies between services will pop up, because of which you will have to deploy sets of services as a group.
In turn, this means that it is necessary to maintain consistent versions of services that have been verified and tested in working with each other. It turns out that there is really no system, parts of which can be deployed independently, because for deploying a new feature you will have to carefully manage the deployment of many services.
Stateful complexities are often ignored
In the previous example, the sometimes arising need for simultaneous roll-out of many versions of many services is mentioned when the feature is deployed. I want to say that thoughtful deployment techniques make life easier: for example, a blue-green deployment [blue / green deployments] (require minimal effort on most platforms for orchestrating services) or parallel launch of many versions of a service with the option of choosing the right one for their user.
These techniques remove many obstacles if the services are stateless. But working with stateless services is actually quite simple. Actually, if you have stateless services, I recommend thinking about skipping all these microservices and using the serverless model.
In reality, many services need to keep state[state] . An example for a video sharing platform is the subscription service. Let the new version of the subscription service store data in a new kind of database. If both services are running in parallel, then you have a system that simultaneously uses two schemes. If you make a blue-green deployment, and other services depend on the data in a new form, they must be updated at the same time. If the deployment of the subscription service failed and rolls back, then most likely it is necessary to roll back these services too, and then "cascade".
Again, it is tempting to think that with databases from the NoSQL world, similar schema-related problems will go away, but this is not so. Databases that do not require strict schemas do not mean that the system does not have schemas, because in essence it simply means the need to manage the schema at the application level, and not at the DBMS level. The fundamental problem of understanding the nature of your data and how it is changing cannot be eradicated.
Complexities of interaction are often ignored.
Since you are creating a large network of services that are dependent on each other, a large field for interservice interaction is likely to appear. There are a number of difficulties. Firstly, there are more places for potential failures. It is necessary to anticipate the possibility that network calls will not work, that is, every time one service accesses another, it should at least try to repeat its attempts. When a service needs to access many services, the situation will become even more complicated.
Imagine a user uploading a video to a sharing service. We need to start the download service, transfer data to the transcoding service, update subscriptions, update recommendations and so on. All these challenges require a certain orchestration, and if something breaks, the actions need to be repeated.
The logic of repeated calls can become difficult to manage. Attempts to do everything synchronously often fail, because they have too many points of failure. In this case, the use of asynchronous interaction patterns will become a more reliable solution. But the difficulty is that due to asynchronous patterns the system becomes stateful. And as mentioned in the last paragraph, it is very difficult to manage stateful systems and systems with distributed states.
When message queues are used in a microservice system for interservice communication, you basically have a large database (message queue or broker) that sticks together all services. Again, although all this at first does not seem to be a problem, the circuit will catch up with you and remind yourself. A version X service can write messages in a specific format, so services that depend on such a message also need to be updated when the sender service changes something in its message.
You can have services that process messages in many different formats, but they are difficult to manage. When deploying new versions of services, it will happen that two versions of the service try to process messages from the same queue and, possibly, even from sender services of different versions as well. This can lead to confusing, inappropriate situations. It may turn out that to avoid them it is easier to allow only certain versions of messages to exist: this means that you have to deploy consistent sets of versions for sets of services, ensuring that messages of older versions are processed accordingly (first).
This again confirms the idea that, if you go into details, independent deployments do not always work as intended.
Versioning can be complicated
To overcome the mentioned obstacles, you must very carefully manage versions. Again, there is a tendency to believe that following a standard like semver will solve the problem. This is not true. Semver is an agreement that is worth using, but you still have to keep track of the versions of services and APIs that may interact.
You can quickly come to significant difficulties and find yourself in a situation where you do not know which versions of the services actually work correctly with each other.
It is well known how difficult it is to manage dependencies in software systems, be it modules for Node or Java, libraries for C, etc. It is very difficult to deal with conflicts of independent components used by one entity.
It is difficult to cope with these problems when dependencies are static and can be patched, updated, edited and so on. If the working services act as dependencies, you can’t just update them: you will need to launch several versions (with the problems described above) or suspend the system until everything is fixed.
Distributed transactions
In situations where transactional integrity is required during operation, microservices can be a big pain. It is not easy to work with distributed state, and many small components that can break make transaction orchestration really difficult.
It may seem attractive to try to avoid the problem by making operations idempotent, suggesting retry mechanisms, etc. - and in many cases it will work. However, there may be scenarios in which a successful or unsuccessful transaction is simply needed, without an intermediate state. The complexity of a workaround for this task or its implementation in a microservice model can be very high.
Microservices can be masked monoliths
Yes, certain services and components can be deployed in isolation, but in most cases some kind of platform for orchestration like Kubernetes will be launched. If you use a managed service like GKE from Google or EKS from Amazon, most of the complexities of managing a cluster are solved for you.
However, if you manage the cluster yourself, then you are working with a large, complex, and critical system. Although individual services may have all the advantages described above, you need to carefully manage the cluster. Deploying such a system can be difficult, upgrades can be complicated, failover can be complex, and so on.
In many cases, the overall benefits remain, but it’s important not to simplify or underestimate the additional complexity caused by managing another large and complex system. Managed services can help, but they are often too young (for example, Amazon EKS was announced only at the end of 2017).
The death of microservice madness!
Avoid insanity by making prudent and deliberate decisions. And to help you do this, I have prepared several questions that you can ask yourself and explanations of the answers to them. Note perev. : The original image is translated and presented in text form below.
1. Team size
Is it possible to seat your whole team at one big table?
- Yes! Microservices may not be needed yet. The difficulties associated with deployment, development, operation, etc., are probably easily solved with good communications and a good architecture , and microservices may turn out to be a solution to a problem that you do not have.
- Not! Microservices can help. If you have a large team or several teams, it can be difficult to strictly define the boundaries of components using architecture alone. Separating components into isolated services can help to realize these boundaries.
2. Stateless / stateful
Is your system predominantly stateless?
- Yes! Consider serverless. If your system is basically stateless, you can probably skip the microservice phase and go straight to serverless - at least in part.
- Not! Microservices will bring complexity. This does not mean that you should not use microservices, but remember that it is not easy to implement and manage them - especially because the system changes over time .
3. "Users" of the system
Are you building a solution for one application or service?
- Yes! Be careful - there may be “blurred” subject areas. If everything you collect is intended for the same user application, it may turn out that building features will require updating multiple services at the same time. Microservices may be appropriate, but be very careful with the design of subject areas.
- Not! Microservices can be very useful. If you are designing a system that will be used by different applications, microservices can be a very appropriate pattern to quickly deliver new features to new “user” applications.
4. Dependencies
Do you have monolithic dependencies?
- Yes! Performance can cause problems. In this case, independently scalable services are unlikely to help because dependency performance remains. It turns out that one of the main advantages will not be relevant. In addition, the boundaries of your services may be worse defined.
- Not! Microservices can be very useful. If monoliths do not pull you to the bottom, you can achieve the high level of independence required to effectively scale microservices.
5. Competency
Do you have experts in containers, orchestration, DevOps?
- Yes! Microservices can be very useful. If you have the appropriate personnel, you should study microservices. Available skills will allow you to deal with potential difficulties and take advantage.
- Not! Test the soil first! In the absence of proper competence or in case of existing difficulties with DevOps, you can jump above your head. Consider adapting one simple service as a proof of concept. Get the first experience you need on projects that are not business critical.
Download original PDF with all the questions and answers (in English) can be here .
Last Thoughts: Do Not Confuse Microservices with Architecture
I deliberately avoided the word “a” in this article. But my friend Zoltan , checking her, made a very good point.
Microservice architecture does not exist. Microservices are just another pattern or implementation of components, no more, no less. Whether they are represented in the system or not, does not mean that the system architecture is ready.
Microservices in many respects are more related to technical processes related to packaging and operation than to the system architecture itself. Appropriate boundaries for components remain one of the main challenges in engineering systems.
Regardless of the size of your services, whether they are in Docker containers or not - you always need to think carefully about how to put the system together. There are no right answers, but there are many options.
PS from the translator
Read also in our blog:
- “ The New Stack Statistics on Kubernetes Implementation Difficulties ”;
- “ Our experience with Kubernetes in small projects ” (video report, which includes an introduction to the technical device of Kubernetes);
- “ Infrastructure with Kubernetes as an affordable service ”;
- “ Why is Kubernetes needed and why is it bigger than PaaS? ";
- “ What is a service mesh and why do I need it [for a cloud application with microservices]? ";
- " Operators for Kubernetes: How to Run Stateful Applications ."
