Learning the sound management of Kubernetes / OTUS Blog. Online education

Good day to all!

We are returning to our beloved tradition - the distribution of the utilities that we collect and study as part of our courses. Today we have the DevOps course and one of its tools, Kubernetes, on our agenda .

We recently created a distributed cron jobs planning system based on Kubernetes, a new, exciting platform for container orchestration. Kubernetes is becoming more popular and gives many promises: for example, engineers will not have to worry about which device their application is running on.

Distributed systems are complex in their own right, and managing services on distributed systems is one of the most difficult problems management teams face. We take the introduction of new software into production and training in its reliable management very seriously. As an example of the importance of managing Kubernetes (and why it is so complicated!), Check out the excellent post-mortem hourly outage caused by a bug in Kubernetes.

In this post, we will explain why Kubernetes was chosen. We will study the process of its integration into the existing infrastructure, the method of building trust (and improving) the reliability of our Kubernetes cluster, and the abstraction created based on Kubernetes.

What is Kubernetes?

Kubernetes is a distributed system for planning programs to run in a cluster. You can tell Kubernetes to run five copies of the program, and it will dynamically schedule them in your work nodes. Containers should increase utilization, thereby saving money, powerful deployment primitives allow you to gradually roll out new code, and Security Contexts and Network Policies allow you to safely run multi-tenant workloads.

Kubernetes has many different scheduling options built in. You can schedule long HTTP services, daemonsetsrunning on all the machines in your cluster, tasks that run every hour, and more. Kubernetes is even more. And if you want to find out, check out Kelsey Hightower's great performances: you can start with Kubernetes for system administrators and healthz: Stop doing reverse engineering applications and start watching them from the inside. Do not forget about the good community in Slack.

Why Kubernetes?

Each infrastructure project (I hope!) Begins with the needs of the business, and our task is to improve the reliability and security of the existing cron job planning system, which already exists. Our requirements are as follows:

• Implementation and project management are possible by a relatively small team (only 2 people working full time on this project);
• Ability to plan about 500 different tasks on 20 machines;

And here are a few reasons why we chose Kubernetes:

• The desire to create on the basis of an existing open-source project;
• Kubernetes includes a distributed task scheduler, so we don’t have to write our own;
• Kubernetes is an actively developing project in which you can take part;
• Kubernetes is written in Go, which is easy to learn. Almost all Kubernetes bug fixes were produced by programmers who did not have previous experience with Go;
• If we manage to manage Kubernetes, then in the future we will be able to create already on the basis of Kubernetes (for example, now we are working on a system based on Kubernetes to train machine learning models).

Previously, we used Cronos as a task scheduler , but it has ceased to meet our reliability requirements and is now almost not supported (1 commit in the last 9 months, and the last time a pull request was added in March 2016). Therefore, we decided to no longer invest in improving our existing cluster.

If you are considering Kubernetes, keep in mind: do not use Kubernetes simply because other companies do it. Creating a reliable cluster requires a huge amount of time, and the way it is used in business is not always obvious. Invest your time wisely.

What does “reliable” mean?

When it comes to service management, the word “reliable” does not make sense in itself. To talk about reliability, you first need to install SLO (service level objective, that is, the target level of service).

We had three main tasks:

1. 99.99% of the tasks need to be planned and run within 20 minutes of their planned time. 20 minutes is a rather large time period, but we interviewed our internal clients and no one demanded more accuracy;
2. Assignments must be completed in 99.99% of cases (without interruption);
3. Our move to Kubernetes should not cause client-side incidents.

This meant a few things:

• Short downtimes in the Kubernetes API are acceptable (if it fell by 10 minutes - not critical, if it is possible to recover within 5 minutes);
• Bug planning (when an already running task crashes or cannot start at all) is unacceptable. We took bug planning reports very seriously;
• You need to be careful with removing hearths and destroying instances so that tasks are not interrupted too often;
• We need a good moving plan.

Creating a Kubernetes Cluster

The basic approach to creating the first Kubernetes cluster was to create a cluster from scratch, without using tools like kubeadm or kops . We created the configuration using Puppet, our standard configuration management tool. Creating from scratch attracted for two reasons: we could deeply integrate Kubernetes into our architecture, and we gained wide knowledge of its internal structure.

Building from scratch allowed us to integrate Kubernetes into the existing infrastructure.We wanted seamless integration with our existing systems of logging, certificate management, secrets, network security, monitoring, AWS instance management, deployment, proxy databases, internal DNS servers, configuration management, and many others. The integration of all these systems sometimes required a creative approach, but, in general, it turned out to be simpler than potential attempts to force kubeadm / kops to do what we need from them.

We already trust and know how to manage existing systems, so we wanted to continue using them in the new Kubernetes cluster. For example, reliable certificate management is a very difficult problem, but we already have a way to solve it. We managed to avoid creating a new CA for Kubernetes due to competent integration.

We were forced to understand how exactly the settings that affect our Kubernetes installation. For example, more than a dozen parameters were used to configure certificate / CA authentication. Understanding all of these options made it easier to debug our installation when we found authentication problems.

Build Confidence in Kubernetes

At the very beginning of working with Kubernetes, no one in the team had any experience in using it (only for home projects). How to come from “None of us have ever used Kubernetes” to “We are confident in using Kubernetes on production”?

Strategy 0: Talk with other companies

We asked several people from other companies about their experience with Kubernetes. They all used it in different ways and in different environments (for starting HTTP services, on bare metal, on the Google Kubernetes Engine , etc.).

When it comes to large and complex systems like Kubernetes, it is especially important to think about your application cases, experiment, find confidence in your environment and decide for yourself. For example, you should not decide after reading this post “Well, Kubernetes are successfully used in Stripe, so we can!”.

Here's what we learned from conversations with various Kubernetes cluster management companies:

• Prioritize the reliability of your etcd cluster (the states of your Kubernetes cluster will be stored in etcd);
• Some Kubernetes features are more stable than others, so be careful with alpha features. Some companies use stable alpha functions after their stability has been confirmed for more than one release in a row (that is, if a function becomes stable at 1.8, they wait 1.9 or 1.10 before using it);
• Take a closer look at Kubernetes hosted systems like GKE / AKS / EKS. Installing a fail-safe Kubernetes system from scratch on your own is a difficult task. AWS did not have a Kubernetes managed service during this project, so we did not consider it;
• Be careful of the extra network latency caused by overlay networks / software networks.

Conversations with other companies, of course, did not give a clear answer as to whether Kubernetes is suitable for us, but they provided us with questions and concerns.

Strategy 1: Read the code

We planned to depend heavily on one component of Kubernetes - the cron job controller. At that time he was in alpha, which was a little cause for excitement. We tested it on a test cluster, but how can I say for sure whether it will suit us on production? Fortunately, all the controller functionality code consists of 400 Go lines. Reading the source code quickly showed that:

1. The controller is a stateless service (like any other Kubernetes component, with the exception of etcd);
2. Every 10 seconds, this controller calls syncAll function: go wait.Until (jm.syncAll, 10 * time.Second, stopCh);
3. The syncAll function returns all tasks from the Kybernetes API, goes through this list, determines which tasks should be started next, and then starts them.

The basic logic was easy to understand. More importantly, if a bug were detected in the controller, most likely we would be able to fix it ourselves.

Strategy 2: Perform stress testing

Before starting to build our cluster, we conducted a load test. We were not worried about the number of nodes that the Kubernetes cluster could handle (we planned to deploy about 20 nodes), it was important to make sure that Kubernetes was able to withstand as many tasks as we needed (about 20 per minute).

We conducted a test on a three-day cluster, where 1000 cron jobs were created, each of which was launched once a minute. All tasks just ran bash -c 'echo hello world'. We chose simple tasks because we wanted to test the capabilities of cluster planning and orchestration, rather than its overall computing power.

The test cluster was unable to withstand 1000 tasks per minute. We found that each node launches a maximum of one under per second, and the cluster could run 200 tasks per minute without any problems. Given that we need about 50 tasks per minute, it was decided that such a restriction did not block us (and that if necessary, we could solve them later). Forward!

Strategy 3: Prioritize the creation and testing of a failover etcd cluster

When configuring Kubernetes, it is very important to run etcd correctly. Etcd is the heart of your Kubernetes cluster, that's where all the data about it is stored. Everything except etcd has no states. If etcd is not running, you cannot make changes to your Kubernetes cluster (although existing services will continue to work!).



When starting up, it’s worth keeping in mind two important points:

• Set up replication so that your cluster does not die when a node is lost. Now we have three copies of etcd;
• Make sure you have sufficient I / O bandwidth. In our version of etcd, there was a problem that one node with a high fsync delay could cause a long leader selection, resulting in cluster inaccessibility. We fixed this by making sure that the I / O bandwidth of our nodes is higher than the number of records made by etcd.

Configuring replication - the task is not from the category of made-and-forgot. We carefully tested that when the etcd node is lost, the cluster is still gracefully restored.

Here are some of the tasks we performed to configure the etcd cluster:

• Configure replication
• Checking the availability of the cluster etcd (if etcd fell, we should be aware of this right away);
• Writing simple tools to easily create new etcd nodes and add them to the cluster;
• Adding etcd Consul integration so that you can run more than one etsd cluster in our production environment;
• Testing recovery from backup etcd;
• Testing the ability to rebuild the cluster completely without downtime.

We are very pleased to have tested early. One Friday morning, one of the etcd nodes stopped pinging in our production cluster. We received a warning about this, destroyed the node, raised a new one, added it to the cluster, and all this time Kubernetes continued to work without interruptions. Amazing

Strategy 4: Gradually transfer tasks to Kubernetes

One of our main goals was to transfer tasks to Kubernetes without causing disruptions. The secret to a successful production transfer lies not in an error-free process (this is impossible), but in designing the process in such a way as to reduce the harm from errors.

We were happy owners of a wide variety of tasks requiring migration to a new cluster, so among them were those that had a low level of influence - a couple of errors in transferring them was acceptable.

Before starting the migration, we created an easy-to-use tool that allows you to move tasks between the old and new systems in less than five minutes. This tool reduced the damage from errors - if we transferred a task with a dependency that we did not plan, not a problem! You can simply bring it back, fix the problem and try again.

We adhered to this migration strategy:

1. Roughly distribute tasks according to their importance;
2. Repeatedly transfer multiple tasks to Kubernetes. If you find new problem areas, quickly roll back, fix the problem and try again.

Strategy 5: Investigate Kubernetes Bugs (and fix them)

At the very beginning of the project, we set a rule: if Kubernetes does something unexpected, you need to understand why and propose a fix.

Parsing each bug takes a lot of time, but is very important. If we simply dismissed the strange behavior of Kubernetes, deciding that the complexity of the distributed system was to blame, then each new bug would make us think that it was only our fault.

After taking this approach, we found (and fixed!) Several bugs in Kubernetes.
Here are some of the problems we found during these tests:

• Cronjobs with names longer than 52 characters silently fail to schedule tasks (fixed here );
• Pods can permanently freeze in the Pending state (fixed here and here );
• The scheduler crashes every three hours (fixed here );
• The Flannel hostgw backend has not replaced obsolete root values ​​(fixed here ).

Fixing these bugs helped us better relate to the use of Kubernetes in the project - it not only worked well enough, but also accepted patches and had a good PR review process.

Kubernetes is full of bugs, like any software. For example, we use the scheduler heavily (because our tasks constantly create new pods), and the scheduler’s use of caching sometimes turns into bugs, regression and crashes. Caching is hard! But codebase is accessible and we were able to cope with all the bugs we encountered.

Another noteworthy issue is the hearth transfer logic in Kubernetes. Kubernetes has a component called a node controller that is responsible for eliminating hearths and moving them to other nodes in case of non-response. There may be a case where there is no response from all nodes (for example, with a network and configuration problem), then Kubernetes can destroy all the pods in the cluster. We faced this at the beginning of testing.

If you are running a large Kubernetes cluster, read the documentation carefullynode controller, consider tuning and test hard. Every time we tested changes to the configuration of this setting (e.g. --pod-eviction-timeout) by creating a network restriction, amazing things happened. It is better to learn about such surprises from testing, and not at 3 in the morning on production.

Strategy 6: intentionally create problems for the Kubernets cluster

We have previously discussed the launch of gaming day exercises in Stripe , and continue to do so now. The idea is to come up with situations that may occur in production (for example, the loss of the Kubernetes API server), and then specifically create them (during the working day, with a warning) to make sure that they can be fixed.

Conducting several exercises on our cluster revealed some problems, like monitoring gaps and configuration errors. We were very pleased to find these problems in advance and in a controlled environment, and not by chance six months later.

Some game exercises that we conducted:

• Shutting down a single Kubernetes API server;
• Shutting down all Kubernetes API servers and raising them (surprisingly, everything worked fine);
• Shutdown etcd node;
• Break of working nodes in our Kubernetes cluster from API servers (so that they lose the ability to communicate). As a result, all the pods from these nodes were transferred to other nodes.

We were very pleased to see that Kubernetes adequately withstood our breakdowns. Kubernetes is designed to be error-resistant - it has one etcd cluster that stores all states, an API server, which is a simple REST interface for the database and a set of stateless controllers that coordinates cluster management.

If any of the root components of Kubernetes (API server, controller management, scheduler) is interrupted and restarted, then immediately after recovery, it reads the state from etcd and continues to work. This is one of those things that attracted in theory, and in practice proved to be no worse.

Here are some of the problems we found during the tests:

• “Strange, I was not informed about this, although I should have. It is necessary to correct the observation here ”;
• “When we destroyed the server API instances and raised them back, they required human intervention. Better fix it ”;
• “Sometimes, when we check the fault tolerance of etcd, the API server will timeout requests until we restart it.”

After starting these tests, we made corrections for the problems found: improved monitoring, fixed found configuration settings, documented Kubernetes bugs.

Making cron jobs easy to use.

We’ll briefly look at how we made our Kubernetes-based system easy to use.
Our initial goal was to design a system to run tasks that our team could confidently manage. When we became confident in Kubernetes, it became necessary to easily configure and add new cron jobs by our engineers. We have developed a configuration YAML format so that our users do not need to understand the internal structure of Kubernetes to use our system. Here is the format:

name: job-name-here
kubernetes:
  schedule: '15 */2 * * *'
command:
- ruby
- "/path/to/script.rb"
resources:
  requests:
    cpu: 0.1
    memory: 128M
  limits:
    memory: 1024M

Nothing complicated - just wrote a program that takes this format and translates it into the cron job Kubernetes configuration, which we use with kubectl.

We also wrote a test suite for checking the length (Kubernetes task names cannot exceed 52 characters) and the uniqueness of task names. Now we do not use cgroups to limit the amount of memory in most of our tasks, but this is in the plans for the future.

Our format was easy to use, and given that we automatically generated both Chronos cron jobs and Kubernetes cron jobs, moving tasks between the two systems was easy. This was a key element in the good work of our gradual migration. Once the task transfer to Kubernetes caused an error, we could transfer it back with a simple three-line configuration change in less than 10 minutes.

Kubernetes Monitoring

Monitoring the internal states of our Kubernetes cluster has proven enjoyable. We use the kube-state-metrics package for monitoring and a small Go program called veneur-prometheus to collect the Prometheus metrics that kube-state-metrics issues and publish them as statsd metrics in our monitoring system.

Here, for example, is a graph of the number of Pending hearths in our cluster over the past hour. Pending means that they are awaiting assignment to the working node to start. You can see the peak value at 11 am, because many tasks start at zero minute of the hour.



In addition, we make sure that there are no pods stuck in the Pending state - we check that each pod starts in the working node for 5 minutes, otherwise we get a warning.

Future plans for Kubernetes

Configuring Kubernetes, reaching the point of being ready to launch production code and migrating our cron jobs to a new cluster, took three months for five months for three full-time programmers. One of the significant reasons we have invested in learning Kubernetes is the expectation of a wider use of Kubernetes in Stripe.

Here are some guidelines for managing Kubernetes (and any other complex distributed system):

• Define a clear business reason for your Kubernetes project (and all infrastructure projects!). Understanding the case and the needs of users greatly simplified our project;
• Агрессивно сокращайте объемы. Мы решили не использовать многие базовые функции Kubernetes для упрощения кластера. Это позволило нам запуститься быстрее — например, с учетом того, что pod-to-pod сеть не была обязательной для нашего проекта, мы смогли ограничить все сетевые соединения между нодами и отложить размышления о сетевой безопасности в Kubernetes до будущего проекта.
• Отведите много времени изучению правильного управления Kubernetes кластера. Внимательно тестируйте острые кейсы. Распределенные системы очень сложны, поэтому многие вещи могут пойти не так. Возьмем пример, описанный ранее: контроллер нодов может убить всех подов в вашем кластере, если они потеряют контакт с API серверами, в зависимости от вашей конфигурации. Изучение поведения Kubernetes после каждого изменения настройки, требует времени и внимательности.

Keeping the focus on these principles, we can confidently use Kubernetes in production. Our use of Kubernetes will continue to grow and evolve - for example, we are watching AWS EKS releases with interest. We are completing work on another machine learning model training system, and are also considering the option of moving some HTTP services to Kubernetes. And in the process of using Kubernetes in production, we plan to contribute to the open-source project.

THE END

As always, we are waiting for your comments, questions here or on our Open Day .