Kapitan at the helm of Kubernetes / Southbridge Blog

Meet Kapitan . It will help you bring beauty and order to your Kubernetes configuration .

Kapitan earns a reputation from satisfied user reviews and therefore dispenses with extensive documentation and expensive marketing. We have enough stars and a couple of mentions from Kubernetes bloggers and preachers. Kapitan even became the protagonist of an entire chapter in the book . Most importantly, he attracted the attention of several promising companies, because Kapitan, like no one else, is able to unravel the configuration tied to the sea knot .

At kubernetes.slack.com #kapitan managed to gather a small but dedicated community (join in!), So we are proud of our work :)

Many still believe that Kapitan is a mixture of jsonnet and jinja, but they miss the point.
In this post I will tell you how Kapitan manages Kubernetes deployments, but in general it is capable of more than that. This is important: Kapitan is universal and not fixated on one Kubernetes. Kubernetes is simply one of many uses.

This is not a guide (although I promise a guide too). I just want to tell you why we did it and what problems with deployments of Kubernetes configurations it should solve.

What i didn't like

I started experimenting with Kubernetes in 2015 and immediately fell in love.
True, there are several drawbacks that I do not want to put up with:

• Context . For me, this is one of the most difficult to understand concepts of Kubernetes - and one of the most dangerous. The context refers to the client, it is difficult to understand, it is confusedly called and creates confusion when running kubectl commands . I can not stand contexts in kubectl!
• Verbose nested configuration (yaml!) . I had to sweat to figure out each level of yaml configuration in the manifest. What is the point of repeating labels two to three times in several places?
• A mess with imperative and declarative teams . Newcomers to Kubernetes are encouraged to learn by imperative teams, although it is clear that normal people do not use them. In my opinion, it’s just harder to fit Kubernetes into the right deployment strategy for your company. Spoiler: There is no single “right strategy”.
• Runtime configuration . Jesse Suen is right when he advises against passing configuration parameters to the helm command line (or kubectl and others like that). With parameters, the same command can be executed differently each time.
• Configuration application . We learned how to manage yaml manifests in Kubernetes, we are great. That's just under and Deploy - this is an empty bowl. An application with all its configuration should still float on it.
• Developers want a holiday , and the workflow at Kubernetes is still somehow slurred. Kubernetes fans are forcing everyone to do it right there. Is it necessary? Obey Kelsey Hightower ( Kelsey Hightower )!
• The operators . I have mixed feelings for them, so for now leave this topic :) I can only say that they are often abused.
• Idempotency . Rather, her absence. If we add up all the shortcomings above, we get insufficiently idempotent workflows, which is sad for Kubernetes.

What to do?

I tried to solve these problems and put together a small templating system that used j2cli and a couple of bash scripts to manage Kubernetes configurations.

The system put everything into the environmentA.yaml file and used it in the Jinja2 template. Deploying microservice-style applications from several components was possible with a simple command:

bin/apply.sh environments/environmentA.yaml

Cool! Yaml was all about deploy. It’s very convenient, because I could use the same file as a source of information for something else. Say for ... bash scripts !

I figured out how to import values ​​from yaml into scripts to execute similar commands:

bin/create_kafka_topics.sh environments/environmentA.yaml

And then everything got out of control at once :

• I could not do anything with the structure in the yaml file. It was a mishmash of the same fields, values, confused configuration.
• You will never know how to behave deploy in the environment until you try. Often this was due to changes in jinja2 templates due to a new inventory value (for example, feature_X) that did not work in environments where this function is not defined.
• The same trouble with scripts: if you don’t try, you don’t know.
• Sometimes Kubernetes changed so quickly that it bothered me to constantly change manifests for different versions, especially messing with annotations to the values.
• External factor : the development team switched from configuration files to command line parameters. Such a small change confused us all the cards, and we had to think about a new solution.
• Most importantly : templating yaml with Jinja (or Go templates) is NOT FUN! We then had a riddle: “ What looks like text, reads like text, smells like text, but not text? ". Or, as Lee Briggs aptly put it: “ Why the hell are we template yaml? "

Kapitan: Becoming

We have gathered all their bitter experience and together with Ricardo Amaro ( of Ricardo Amaro ) began to fantasize about the ideal configuration of the control system. Then we still did not have a clear picture, but we knew that we loved and that we did not love.

Love :

• Git.
• Patterning in general: data / values ​​separate from patterns.
• Separate values ​​for different aspects (application, Kubernetes, runtime ...).
• Object oriented approach.
• Simplified yaml as an interface, where to hide the complexity of Kubernetes.
• A clear understanding of what is happening and why.
• Reusing values ​​in different components.
• And scripts should have access to values.

We do not like :

• Kubectl contexts
• Text template engines for creating yaml.
• Counting indent: {{ toYaml .Values.resources | indent 10 }}.
• Magic: everything should be clear and clear. No tricks.
• Manual management of passwords and secrets of the application.
• Tiller Approach: We wanted to control the use of manifests.
• Git-crypt approach: secrets on the disk are not encrypted.
• Template pipeline directly to kubectl.
• Passing command line options.

And then two things happened :

1. We discovered jsonnet Dave Cunningham ( by Dave Cunningham ) for standardization yaml / json object-oriented language.
2. Gustavo Buriola ( of Gustavo Buriola ) showed us reclass , and we have not gone far without him.

Ricardo Amaro got to work, and soon the whole team sat down at Kapitan - some worked on the basic functionality, others on its use in our internal projects. Secrets management, gpg \ kms support, user-defined functions: now Kapitan is a full-fledged product that does more than promised.

Who is Kapitan?

Kapitan is trying to solve all (or almost all) of the problems that I talked about.

From a technical point of view, Kapitan is very simple:

• Inventory : A hierarchical collection of values ​​describing deploy based on yaml. Based on reclass. Like hiera.
• Template engines : now it is Jinja2, Jsonnet, Kadet. They take inventory and create files (yaml, json, documentation or bash scripts).
• Secrets : template secrets, and Kapitan will deal with them.

We're using jsonnet to template manifestos and Jinja to do the rest.

Sometimes people complain that the jsonnet file is completely different from the same yaml, so it’s hard for them to switch to jsonnet.

We tried to solve this problem with Kadet by wrapping yaml in Python. Take your favorite yaml as a basis and add Python to it.

Consider this a Python exoskeleton for yaml! Somehow I’ll talk about this.

In the workflow, Kapitan immediately shows the character:

• Freedom of choice : we do not impose any work processes and technologies, but usually work on the principles described below. In fact, Kapitan can be used as you like. You are not required to use git, you must not compile files in it, and you can even do without jsonnet! Do what you want.
• GitOps to the bone: everything in git, everything in a master- light that reflects our intent.
• Declarability : Kapitan welcomes the compilation of manifest templates in specific representations. And you compile your scripts.
• Controlled context : we use compiled scripts to simplify our work, for example, when setting contexts and configuring clusters.
  Kubernetes Configuration: compiled/target_A/setup/setup.sh
  Apply Changes:compiled/target_A/setup/apply.sh
• Idempotency : Kapitan allows you to change templates and code refactoring tools. Compiled manifests and code will not change without your command, so you have nothing to fear when refactoring.
• Cause and effect : we are for a workflow where changes to inventory or templates and compiled files are included in one merge request. So the reviewer will be able to evaluate your changes and their actual consequences. It’s good to know if a change in the template will affect one, two or more targets.
• And finally : Kapitan is not attached to Kubernetes. It just creates files. Deployment of changes involved kubectl . We only give a shell for commands to be executed in a consistent manner.

Do I need it?

Let's make it clear : you probably don't need Kapitan (yet).

But it all depends on what you are trying to do and how complicated your system is.

Kapitan is a powerful investment requiring tool . Use it in complex scenarios where you have to deploy a bunch of applications on a bunch of clusters.

If you have standard applications, you are just learning Kubernetes or are already happy with your workflow, then Helm or its current alternative will work.

I imagine Helm as apt-get for Kubernetes , and Kapitan is something like Puppet .

In the next post I will give specific examples and describe in detail the inventory. Write about what you want to know or what you agree / disagree with in this post.

Thanks to Jacek Gruzewski .