Is Kubernetes Cluster easy and convenient to prepare? Announce addon-operator
Following the shell-operator, we introduce his older brother - addon-operator . This is an Open Source project that is used to install system components in the Kubernetes cluster, which can be called a common word - add-ons.
Why do any additions at all?
It's no secret that Kubernetes is not an all-in-one finished product, and various additions will be needed to build an “adult” cluster. Addon-operator will help to install, configure and keep these add-ons up to date.
The need for additional components in the cluster is disclosed in a report by colleagues driusha. In short, the situation with Kubernetes at the moment is that for a simple installation you can “play around” with components out of the box, for developers and testing you can add Ingress, but for a full installation, which you can say “your production is ready,” you need to add a dozen different add-ons: something for monitoring, something for logs, do not forget ingress and cert-manager, select host groups, add network policies, season with sysctl and pod autoscaler settings ...
What are the specifics of working with them?
As practice shows, the case is not limited to one installation. For comfortable work with the cluster, the add-ons will need to be updated, disabled (deleted from the cluster), and you will want to test something before installing it in the production cluster.
So maybe Ansible is enough here? Maybe. But full-fledged additions in the general case do not live without settings . These settings may vary depending on the cluster option (aws, gce, azure, bare-metal, do, ...). Some settings cannot be set in advance - they must be obtained from the cluster. And the cluster is not static: for some settings you will have to follow the changes. And here Ansible is already missing: we need a program that lives in a cluster, i.e. Kubernetes Operator.
Those who have tried shell-operator, they will say that the tasks of installing and updating add-ons and tracking settings can be completely solved using hooks for the shell-operator. You can write a script that will do a conditional
kubectl applyand monitor, for example, ConfigMap, where the settings will be stored. This is approximately what is implemented in addon-operator.How is this organized in addon-operator?
Creating a new solution, we proceeded from the following principles:
- The add-on installer must support templating and declarative configuration . We do not do magic scripts that install add-ons. Addon-operator uses Helm to install add-ons. To install, you need to create a chart and highlight the values that will be used to configure.
- Settings can be generated during installation , they can be obtained from the cluster , or receive updates by monitoring cluster resources. These operations can be implemented using hooks.
- Settings can be stored in a cluster . To store the settings in the cluster, a ConfigMap / addon-operator is created and Addon-operator monitors the changes of this ConfigMap. Addon-operator gives hooks access to settings using simple conventions.
- Addition depends on settings . If the settings have changed, then Addon-operator rolls out the Helm-chart with new values. The union of the Helm chart, the values for it and the hooks we called the module (see below for more details).
- Staging . No magic release scripts. The update mechanism is similar to a regular application - collect add-ons and addon-operator into an image, test and roll out.
- Control of the result . Addon-operator can give metrics for Prometheus.
What is the addon in addon-operator?
An addition can be considered everything that adds new functions to the cluster. For example, installing Ingress is a great add-on. It can be any operator or controller with its own CRD: prometheus-operator, cert-manager, kube-controller-manager, etc. Or something small, but simplifying operation - for example, secret copier, copying registry secrets to new namespaces, or sysctl tuner, configuring sysctl parameters on new nodes.
Addon-operator provides several concepts for implementing add-ons:
- The Helm chart is used to install various software into the cluster - for example, Prometheus, Grafana, nginx-ingress. If the required component has a Helm chart, then installing it using the Addon-operator will be very simple.
- Store of values . Helm charts usually have many different settings that can change over time. Addon-operator supports the storage of these settings and is able to monitor their changes in order to reset the Helm-chart with new values.
- Hooks are executable files that Addon-operator runs on events and that access the values store. The hook can monitor changes in the cluster and update values in the values store. Those. with the help of hooks, you can make discovery to collect values from the cluster at startup or according to a schedule, or continuous discovery, by collecting values from the cluster by changes in the cluster.
- A module is a union of a Helm chart, values storage and hooks. Modules can be turned on and off. Disabling a module is the removal of all releases of the Helm chart. Modules can turn on themselves dynamically, for example, if all the modules he needs are included or if discovery found the necessary parameters in hooks - this is done using an auxiliary enabled script.
- Global hooks . These are “on their own” hooks, they are not included in the modules and have access to the global values storage, the values of which are available to all hooks in the modules.
How do these parts work together? Consider the picture from the documentation:
Two work scenarios:
- A global hook is triggered by an event - for example, when a resource in a cluster changes. This hook processes the changes and writes the new values to the global values storage. Addon-operator notices that the global storage has changed and launches all modules. Each module, using its hooks, determines whether it needs to be turned on and updates its values store. If the module is enabled, then Addon-operator starts the installation of the Helm-chart. In this case, the values from the module storage and from the global storage are accessible to the Helm-chart.
- The second scenario is simpler: a module hook is triggered by an event, changes the values in the module values storage. Addon-operator notices this and launches the Helm chart with updated values.
The add-on can be implemented as a single hook or as one Helm-chart, or even as several dependent modules - this depends on the complexity of the component installed in the cluster and on the desired level of configuration flexibility. For example, in the repository ( / examples ) there is a sysctl-tuner add-on, which is implemented both as a simple module with a hook and a Helm chart, and using the values storage, which makes it possible to add settings through editing ConfigMap.
Update delivery
A few words about the organization of component updates that Addon-operator installs.
To launch the Addon-operator in the cluster, you need to collect an image with additions in the form of hooks files and Helm-charts, add a binary file
addon-operatorand all that you will need to hook: bash, kubectl, jq, python, etc. Further, this image can be rolled out to the cluster as a regular application, and most likely you will want to organize this or that tagging scheme. If there are few clusters, the same approach as with applications may be suitable: a new release, a new version, go across all clusters and correct the image for Pods. However, in the case of a rollout to a noticeable number of clusters, the concept of self-renewal from the channel was more suitable for us. We have it arranged like this:
- A channel is essentially an identifier that can be set by anyone (for example, dev / stage / ea / stable).
- The channel name is the image tag. When you need to roll out updates to the channel, a new image is collected and tagged with the name of the channel.
- When a new image appears in the registry, Addon-operator restarts and starts with the new image.
This is not a best practice, as described in the Kubernetes documentation . This is not recommended, but we are talking about a regular application that lives in one cluster . In the case of Addon-operator, an application is a lot of Deployments scattered across clusters, and self-updating is very helpful and simplifies life.
Channels help in testing : if there is an auxiliary cluster, you can configure it on a channel
stageand roll updates to it before rolling out into channels eaand stable. If eaan error has occurred with the cluster on the channel , you can switch it to stablewhile the investigation of the problem with this cluster is ongoing. If the cluster is withdrawn from active support, it switches to its “frozen” channel - for example,freeze-2019-03-20. In addition to updating hooks and Helm charts, you may need to update a third-party component . For example, you noticed an error in the conditional node-exporter and even figured out how to patch it. Then we opened PR and wait for a new release to go through all the clusters and increase the version of the image. In order not to wait indefinitely, you can collect your node-exporter and switch to it before accepting the PR.
In general, this can be done without Addon-operator, but with Addon-operator the module for installing node-exporter will be visible in one repository, you can keep the Dockerfile to build your image right there, it becomes easier for all participants in the process to understand that happens ... And if there are several clusters, it becomes easier both to test your PR and roll up a new version!
This organization of component updates works successfully with us, but you can implement any other suitable scheme, because in this case Addon-operator is a simple binary file .
Conclusion
The principles implemented in Addon-operator allow you to build a transparent process of creating, testing, installing and updating add-ons in a cluster, similar to the processes of developing ordinary applications.
Add-ons for Addon-operator in the format of modules (Helm-chart + hooks) can be uploaded to the public. We, the company Flant, plan to lay out our achievements during the summer in the form of such additions. Join the development on GitHub ( shell-operator , addon-operator ), try to make your addition based on examples and documentation , wait for the news on the Habré and on our YouTube channel !
UPDATED (June 14): If you have English-speaking colleagues who may be interested in addon-operator, then the corresponding announcement for them is available on our blog on Medium .
PS
Read also in our blog:
