November 10, 2017 at 10:14Designing a Microservice Architecture with Failure
- Transfer
Translation of Designing a Microservices Architecture for Failure article .
Microservice architecture, thanks to precisely defined service boundaries, isolates failures . However, as with any distributed system, there is a higher likelihood of problems at the network, hardware, or application level. As a result of service dependency, any component may be temporarily unavailable to users. To minimize the impact of partial failures, we need to build services resistant to them that can correctly respond to certain types of problems.
This article presents the most common techniques and architectural patterns for building and operating.highly accessible microservice system .
If you are not familiar with the templates mentioned here, then it is not at all necessary that you are doing something wrong. Building a reliable system always requires additional investment.
With this architecture, application logic is transferred to services, and the network layer is used to interact between them. Interaction over the network, and not through calls within the memory, increases the latency and complexity of the system, which requires the cooperation of numerous physical and logical components. And the growing complexity of the distributed system leads to the fact that the chances of certain network failures are growing .
One of the main advantages of microservice architecture compared to monolithic is that teams can independently design, develop and deploy services. They fully manage the entire life cycle of their services. It also means that teams do not have control over the dependencies of services, as usually other people are in charge. When using the microservice architecture, you need to remember that the services of the provider may be temporarily unavailable due to mowing releases, configurations and various changes, since this does not depend on the developers, and the components change independently of each other.
One of the most attractive aspects of microservice architecture is the ability to isolate failures, and due to the fact that the components fail separately from each other, graceful service degradation can be achieved. For example, if the application that allows sharing photos fails, users will probably not be able to upload new images, but they will be able to view, edit, and share existing photos.
Separate failures of microservices (in theory).
However, in most cases it is difficult to implement this kind of gradual degradation of service, because applications in distributed systems are dependent on each other, and you need to apply several different types of fault-handling logic (some of which we will discuss below) to prepare for temporary difficulties and failures.
Without fault handling logic, services are interdependent and all fail together.
The Google Reliability Team found that about 70% of failures were caused by changes in live systems. Changing something in your system - deploying a new version of the code or changing some configuration - you risk causing a crash or introducing new bugs.
In a microservice architecture, services are interdependent. Therefore, it is necessary to minimize failures and limit their negative impact. To cope with problems caused by changes, you can implement change management strategies and automatic rollbacks .
For example, as you make changes, gradually apply them to a subset of your instances, track and automatically roll back the deployment if you notice a deterioration in key metrics.
Change Management - Rolling Deployment.
Another solution might be to use two production environments. Always deploy in only one of them, and apply a load balancer to it only after making sure that the new version works as expected. This is called a "blue-green" or "black-red" deployment.
Code rollback is not a problem . You cannot leave broken code in production and then puzzle over what went wrong. Always roll back changes as necessary. The sooner the better.
Instances constantly start, restart, and stop due to failures, deployments, or auto-scaling. And therefore they become temporarily or permanently inaccessible. To avoid such problems, your balancer should exclude failed instances from rotation if they cannot serve clients or other subsystems.
The health of application instances can be determined through external monitoring. You can do this with regular endpoint calls
"Reanimate" the application using self-healing. You can talk about this mechanism if the application performs the necessary actions to exit the failed state. In most cases, self-healing is implemented by an external system that monitors the health of instances and restarts them if they are in a state of failure for a certain period. Self-healing is often very useful, but in some situations it can cause problems due to the constant restart of the application. This is possible if the application cannot report a positive state due to overload or timeouts when connecting to the database.
It may not be easy to implement an advanced self-healing mechanism that will be ready for delicate situations like losing a connection to the database. In this case, you need additional logic that will handle extreme cases and let the external system know that it is not necessary to restart the instance immediately.
Usually services will crash due to network problems and system changes. However, most failures are temporary due to self-healing mechanisms and advanced balancing. And we need to find a solution that allows services to work during such incidents. Failover caching can help here , which will provide applications with the data they need.
Failover caches typically use two different expiration dates . A shorter one indicates how long you can use the cache in a normal situation, and a longer one means how long you can use the cached data during the failure.
Failover caching.
It is important to mention that you can only use fail-safe caching when legacy data is better than nothing .
You can use the standard response headers in HTTP to configure the regular and fail-safe cache.
For example, using the header,
Modern CDNs and balancers provide various caching and fault tolerance schemes, but you can also create a common library for your company that contains standard reliability solutions.
There are situations when we cannot cache data, or when we need to make changes to them, but our operations will fail. Then you can try to repeat our actions if there is a chance that the resources will be restored some time later, or if our balancer sends our requests for a working instance.
Be careful when adding retry logic to your applications and clients, because a large number of retries can make the situation worse or even prevent applications from recovering.
In a distributed system, repeats in a microservice structure can generate multiple responses or other repeats, which will create a cascading effect. To minimize the effect of repeats, limit their number and use an exponential delay algorithm to increase the delay between repeats each time until you reach the limit.
Since the repetition is initiated by a client (browser, other microservice, etc.), which does not know whether the operation was unsuccessful before or after processing the request, the application must be able to handle idempotency . For example, when you repeat the purchase operation, you should not duplicate the collection of funds from the buyer. Using a unique idempotency key for each transaction will help you .
Speed limit is a technique for determining the number of requests that can be received or processed by a specific consumer or application within a certain time. With the help of speed limits, we can, for example, filter out our customers and microservices that cause traffic spikes . Or we can make sure that the application will not be overloaded until autoscaling comes to the rescue.
You can also limit low-priority traffic to allocate more resources to mission-critical transactions.
A speed limiter can prevent traffic spikes.
Another type of speed limiter is called the concurrent request limiter. It can be useful when you have “expensive” endpoints that are not recommended to be called more than a certain number of times if you want to serve traffic.
To ensure that you always have enough resources to service critical transactions , use fleet usage load shedder. It holds part of the resources for high-priority requests and does not allow low-priority transactions to use them. Load shedder makes its decisions based on the general state of the system, not the size of a single user request. Also LSs will help your system recover.as they provide key functionality during the incident.
Read more about speed limiters and load shedders in this article: https://stripe.com/blog/rate-limiters .
In microservice architecture, you need to prepare your services to fail quickly and separately . To isolate problems at the service level, we can use the bulkhead template .
A fast component failure is needed because we do not want to wait until the timeouts of the broken instances are over. Nothing is more annoying than a hung request and an interface that does not respond to your actions. This is not only lost resources, but also a spoiled user experience. Services call each other along the chain, so you need to pay special attention to preventing hanging operations, avoiding the accumulation of delays.
You probably immediately came up with the idea of using small grade timeouts for each service call. But the problem is that you cannot know which timeout value will be suitable, because there are situations when network failures and other emerging problems affect only one or two operations. In this case, you probably won’t want to reject these requests because some of them time out.
We can say that the application of the “fast failure” paradigm in microservices through timeouts is an antipattern that should be avoided. Instead of timeouts, you can use the circuit-breaker template , which depends on the statistics of successful / failed operations.
Bulkheads are used in shipbuilding to divide a ship into sections so that each section can be pulled up in the event of a breakdown of the hull.
The principle of bulkheads can be applied in the development of software for sharing resources to protect them from exhaustion . For example, if we have two types of operations that interact with one database instance that has a limit on the number of connections, then you can use two connection pools instead of one common one. As a result of such a client-resource separation, an operation that initiates a timeout or will abuse the pool will not affect the operation of other operations.
One of the main reasons for the death of the Titanic was the unsuccessful bulkhead design, in which water could overflow onto decks into other compartments, filling the entire hull.
Bulkheads on the Titanic (they did not work).
Timeouts can be used to limit the duration of operations. They can prevent operations from freezing and keep the system responsive to your actions. The use of static, finely tuned timeouts in the microservice architecture is antipattern , since we are talking about a highly dynamic environment in which it is almost impossible to choose suitable time limits that work well in any situation.
Instead of using small static transaction-specific timeouts, you can use circuit breakers to handle errors. The action of these software mechanisms is similar to the electrical devices of the same name. With circuit breakers, you can protect resources andhelp them recover . They can be very useful in distributed systems, where a repeated failure can cause an avalanche effect that will put the whole system.
The circuit breaker opens when a certain type of error occurs several times within a short time . An open circuit breaker prevents the transmission of requests - like a real circuit breaker interrupts an electrical circuit and prevents current from flowing through the wires. Circuit breakers usually close after a certain time, giving the service a respite for recovery.
Remember that not all errors must trigger a circuit breaker. For example, you probably want to skip errors on the client side like requests with 4xx codes, but at the same time respond to server failures with 5xx codes. Some circuit breakers may be half open. This means that the service sends the first request to check the availability of the system, and the remaining requests are cut off. If the first request is successful, the machine goes into a closed state and does not prevent the flow of traffic. Otherwise, the machine remains open.
Circuit breaker.
You should constantly check the behavior of your system in the face of common problems in order to make sure that your services can survive various failures . Test more often so your team is prepared for incidents.
You can use an external service that identifies groups of instances and randomly interrupts the work of one of the group members. This will prepare you for a single instance failure. And you can overlap entire regions to emulate a failure with a cloud provider.
One of the most popular solutions is the ChaosMonkey fault tolerance test tool .
Implementing and maintaining a reliable service is not an easy task. It requires a lot of effort and costs a lot of money.
Reliability has different levels and aspects, so it is important to find the solution that is best for your team. Make reliability one of the factors in the process of making business decisions and allocate enough money and time for it.
Microservice architecture, thanks to precisely defined service boundaries, isolates failures . However, as with any distributed system, there is a higher likelihood of problems at the network, hardware, or application level. As a result of service dependency, any component may be temporarily unavailable to users. To minimize the impact of partial failures, we need to build services resistant to them that can correctly respond to certain types of problems.
This article presents the most common techniques and architectural patterns for building and operating.highly accessible microservice system .
If you are not familiar with the templates mentioned here, then it is not at all necessary that you are doing something wrong. Building a reliable system always requires additional investment.
Microservice Architecture Risk
With this architecture, application logic is transferred to services, and the network layer is used to interact between them. Interaction over the network, and not through calls within the memory, increases the latency and complexity of the system, which requires the cooperation of numerous physical and logical components. And the growing complexity of the distributed system leads to the fact that the chances of certain network failures are growing .
One of the main advantages of microservice architecture compared to monolithic is that teams can independently design, develop and deploy services. They fully manage the entire life cycle of their services. It also means that teams do not have control over the dependencies of services, as usually other people are in charge. When using the microservice architecture, you need to remember that the services of the provider may be temporarily unavailable due to mowing releases, configurations and various changes, since this does not depend on the developers, and the components change independently of each other.
Gradual degradation of service
One of the most attractive aspects of microservice architecture is the ability to isolate failures, and due to the fact that the components fail separately from each other, graceful service degradation can be achieved. For example, if the application that allows sharing photos fails, users will probably not be able to upload new images, but they will be able to view, edit, and share existing photos.
Separate failures of microservices (in theory).
However, in most cases it is difficult to implement this kind of gradual degradation of service, because applications in distributed systems are dependent on each other, and you need to apply several different types of fault-handling logic (some of which we will discuss below) to prepare for temporary difficulties and failures.
Without fault handling logic, services are interdependent and all fail together.
Change management
The Google Reliability Team found that about 70% of failures were caused by changes in live systems. Changing something in your system - deploying a new version of the code or changing some configuration - you risk causing a crash or introducing new bugs.
In a microservice architecture, services are interdependent. Therefore, it is necessary to minimize failures and limit their negative impact. To cope with problems caused by changes, you can implement change management strategies and automatic rollbacks .
For example, as you make changes, gradually apply them to a subset of your instances, track and automatically roll back the deployment if you notice a deterioration in key metrics.
Change Management - Rolling Deployment.
Another solution might be to use two production environments. Always deploy in only one of them, and apply a load balancer to it only after making sure that the new version works as expected. This is called a "blue-green" or "black-red" deployment.
Code rollback is not a problem . You cannot leave broken code in production and then puzzle over what went wrong. Always roll back changes as necessary. The sooner the better.
Health-check and load balancing
Instances constantly start, restart, and stop due to failures, deployments, or auto-scaling. And therefore they become temporarily or permanently inaccessible. To avoid such problems, your balancer should exclude failed instances from rotation if they cannot serve clients or other subsystems.
The health of application instances can be determined through external monitoring. You can do this with regular endpoint calls
GET /healthor through automatic reporting. Modern service discovery solutions constantly collect health information from instances and configure balancers to allow traffic only to fully working components.Self-healing
"Reanimate" the application using self-healing. You can talk about this mechanism if the application performs the necessary actions to exit the failed state. In most cases, self-healing is implemented by an external system that monitors the health of instances and restarts them if they are in a state of failure for a certain period. Self-healing is often very useful, but in some situations it can cause problems due to the constant restart of the application. This is possible if the application cannot report a positive state due to overload or timeouts when connecting to the database.
It may not be easy to implement an advanced self-healing mechanism that will be ready for delicate situations like losing a connection to the database. In this case, you need additional logic that will handle extreme cases and let the external system know that it is not necessary to restart the instance immediately.
Failover caching
Usually services will crash due to network problems and system changes. However, most failures are temporary due to self-healing mechanisms and advanced balancing. And we need to find a solution that allows services to work during such incidents. Failover caching can help here , which will provide applications with the data they need.
Failover caches typically use two different expiration dates . A shorter one indicates how long you can use the cache in a normal situation, and a longer one means how long you can use the cached data during the failure.
Failover caching.
It is important to mention that you can only use fail-safe caching when legacy data is better than nothing .
You can use the standard response headers in HTTP to configure the regular and fail-safe cache.
For example, using the header,
max-ageyou can set the time during which the resource will be considered fresh. And with the help of the header stale-if-error- how long the resource will be provided from the cache in case of failure. Modern CDNs and balancers provide various caching and fault tolerance schemes, but you can also create a common library for your company that contains standard reliability solutions.
Retry Logic
There are situations when we cannot cache data, or when we need to make changes to them, but our operations will fail. Then you can try to repeat our actions if there is a chance that the resources will be restored some time later, or if our balancer sends our requests for a working instance.
Be careful when adding retry logic to your applications and clients, because a large number of retries can make the situation worse or even prevent applications from recovering.
In a distributed system, repeats in a microservice structure can generate multiple responses or other repeats, which will create a cascading effect. To minimize the effect of repeats, limit their number and use an exponential delay algorithm to increase the delay between repeats each time until you reach the limit.
Since the repetition is initiated by a client (browser, other microservice, etc.), which does not know whether the operation was unsuccessful before or after processing the request, the application must be able to handle idempotency . For example, when you repeat the purchase operation, you should not duplicate the collection of funds from the buyer. Using a unique idempotency key for each transaction will help you .
Speed Limit and Load Shedders
Speed limit is a technique for determining the number of requests that can be received or processed by a specific consumer or application within a certain time. With the help of speed limits, we can, for example, filter out our customers and microservices that cause traffic spikes . Or we can make sure that the application will not be overloaded until autoscaling comes to the rescue.
You can also limit low-priority traffic to allocate more resources to mission-critical transactions.
A speed limiter can prevent traffic spikes.
Another type of speed limiter is called the concurrent request limiter. It can be useful when you have “expensive” endpoints that are not recommended to be called more than a certain number of times if you want to serve traffic.
To ensure that you always have enough resources to service critical transactions , use fleet usage load shedder. It holds part of the resources for high-priority requests and does not allow low-priority transactions to use them. Load shedder makes its decisions based on the general state of the system, not the size of a single user request. Also LSs will help your system recover.as they provide key functionality during the incident.
Read more about speed limiters and load shedders in this article: https://stripe.com/blog/rate-limiters .
Fail quickly and separately
In microservice architecture, you need to prepare your services to fail quickly and separately . To isolate problems at the service level, we can use the bulkhead template .
A fast component failure is needed because we do not want to wait until the timeouts of the broken instances are over. Nothing is more annoying than a hung request and an interface that does not respond to your actions. This is not only lost resources, but also a spoiled user experience. Services call each other along the chain, so you need to pay special attention to preventing hanging operations, avoiding the accumulation of delays.
You probably immediately came up with the idea of using small grade timeouts for each service call. But the problem is that you cannot know which timeout value will be suitable, because there are situations when network failures and other emerging problems affect only one or two operations. In this case, you probably won’t want to reject these requests because some of them time out.
We can say that the application of the “fast failure” paradigm in microservices through timeouts is an antipattern that should be avoided. Instead of timeouts, you can use the circuit-breaker template , which depends on the statistics of successful / failed operations.
Bulkheads
Bulkheads are used in shipbuilding to divide a ship into sections so that each section can be pulled up in the event of a breakdown of the hull.
The principle of bulkheads can be applied in the development of software for sharing resources to protect them from exhaustion . For example, if we have two types of operations that interact with one database instance that has a limit on the number of connections, then you can use two connection pools instead of one common one. As a result of such a client-resource separation, an operation that initiates a timeout or will abuse the pool will not affect the operation of other operations.
One of the main reasons for the death of the Titanic was the unsuccessful bulkhead design, in which water could overflow onto decks into other compartments, filling the entire hull.
Bulkheads on the Titanic (they did not work).
Circuit Breakers
Timeouts can be used to limit the duration of operations. They can prevent operations from freezing and keep the system responsive to your actions. The use of static, finely tuned timeouts in the microservice architecture is antipattern , since we are talking about a highly dynamic environment in which it is almost impossible to choose suitable time limits that work well in any situation.
Instead of using small static transaction-specific timeouts, you can use circuit breakers to handle errors. The action of these software mechanisms is similar to the electrical devices of the same name. With circuit breakers, you can protect resources andhelp them recover . They can be very useful in distributed systems, where a repeated failure can cause an avalanche effect that will put the whole system.
The circuit breaker opens when a certain type of error occurs several times within a short time . An open circuit breaker prevents the transmission of requests - like a real circuit breaker interrupts an electrical circuit and prevents current from flowing through the wires. Circuit breakers usually close after a certain time, giving the service a respite for recovery.
Remember that not all errors must trigger a circuit breaker. For example, you probably want to skip errors on the client side like requests with 4xx codes, but at the same time respond to server failures with 5xx codes. Some circuit breakers may be half open. This means that the service sends the first request to check the availability of the system, and the remaining requests are cut off. If the first request is successful, the machine goes into a closed state and does not prevent the flow of traffic. Otherwise, the machine remains open.
Circuit breaker.
Check for failures
You should constantly check the behavior of your system in the face of common problems in order to make sure that your services can survive various failures . Test more often so your team is prepared for incidents.
You can use an external service that identifies groups of instances and randomly interrupts the work of one of the group members. This will prepare you for a single instance failure. And you can overlap entire regions to emulate a failure with a cloud provider.
One of the most popular solutions is the ChaosMonkey fault tolerance test tool .
Conclusion
Implementing and maintaining a reliable service is not an easy task. It requires a lot of effort and costs a lot of money.
Reliability has different levels and aspects, so it is important to find the solution that is best for your team. Make reliability one of the factors in the process of making business decisions and allocate enough money and time for it.
Key Findings
- Dynamic environments and distributed systems — such as microservices — have an increased risk of failure.
- Services should fail separately to ensure smooth degradation of the service and not to ruin the user experience.
- 70% of crashes are caused by changes, so don’t be shy about rolling back the code.
- Failures should go quickly and separately. Teams have no control over the dependencies of their services.
- Architectural patterns and techniques such as caching, bulkheads, circuit breakers, and speed limiters help create robust microservices.
