Grafana panels for Kubernetes administration

Hello! Today we continue to share material translated specifically for students of the Kubernetes Based Infrastructure Platform course . Enjoy reading.

Introduction

For some time now I’ve been experimenting with creating Grafana dashboards for administering Kubernetes. As the owner and administrator of several Kubernetes clusters, I wanted to get a visual representation of some of the metrics. First, I need normal Grafana dashboards for the Kubernetes Control Plane and Node components: kube-api, kube-scheduler, kube-controller-manager, as well as kubelet and kube-proxy . Secondly, I would like to post the resulting panels on GitHub so that the rest can leave suggestions and error reports (issues), send requests for acceptance of changes (pull requests), update the panels (git pull) and help me work on them.

Publish Grafana Dashboards

Grafana panels are now mostly published on  grafana.com/dashboards . To get the right panel, you need to find it on the page, copy the identifier and import it into the local Grafana instance. Although importing dashboards is relatively simple, experience shows that most users find it inconvenient to work with them. There are several reasons for this.

First, people use different tag selectors to indicate monitoring goals in Prometheus. Secondly, in new versions of Prometheus exporters and applications for accessing metrics, the names of metrics are constantly changing and deleted. Thirdly, it’s extremely difficult to find really high-quality dashboards - you never know what you will ultimately get after import. Having selected the panel, you have to comb through all the graphs and edit the metrics along with the selectors of the labels - and this is a large amount of manual work.

Monitoring Mixins Project

I found a very cool project called Monitoring Mixins. In fact, this is a kind of ecosystem, designed to overcome all sore panel problems. The idea occurred to Frederick Branchiku (Frederic Branczyk), and it was he who wrote the original design document , which I strongly recommend to read. Moreover, he thought out a real mechanism for updating dashboards and extracting them from various GitHub repositories. This mechanism is very similar to  go get , and I really liked it. Here you can read a brief description of the jsonnet-bundler package manager using this mechanism. Many thanks to Frederick for useful developments!

Kubernetes Grafana Mixin Package

If you read to this place, you can take a break and enjoy the beautiful pictures.

API Server

Kubernetes API Server Dashboard

Controller manager

Kubernetes Controller Manager Dashboard

Scheduler

Kubernetes Scheduler Dashboard

Kublet (node ​​agent)

Kubernetes Cubelet


Dashboard Kubernetes Cubelet Dashboard

Kube proxy

Kube Proxy Dashboard

Using Kubernetes Grafana Mixin

First you need to install  jsonnet and  jsonnet-bundler.

Jsonnet

If you are running macOS, you can use the command  brew install jsonnet. Otherwise, it is better to compile this component yourself.

git clone https://github.com/google/jsonnet.git jsonnet_git
cd jsonnet_git
make
sudo mv jsonnet /usr/local/bin/

Jsonnet bundler

go get -u github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb

Import panels

Create a new directory for the project and enter it with the team  cd .

Install mixin :

jb init
jb install https://github.com/povilasv/kubernetes-grafana-mixin

Now create a new file  config.libsonnet.

local kubedashboards = import 'kubernetes-grafana-mixin/mixin.libsonnet';
kubedashboards {
  _config+:: {
    kubeletSelector: 'job="kubernetes-nodes2"',
    kubeSchedulerSelector: 'job="kube-scheduler2"',
    kubeControllerManagerSelector: 'job="kube-controller-manager2"',
    kubeApiserverSelector: 'job="kube-apiserver2"',
    kubeProxySelector: 'job="kube-proxy2"',
  },
}

After that, the information panel is imported and the job selectors are overwritten.

Correct the Prometheus tag selectors to suit your environment.

Now create a directory  dashboards . Finally, run  jsonnetto compile  config.libsonnet:

jsonnet -J vendor -m dashboards -e '(import "config.libsonnet").grafanaDashboards'

You will see the following list:

dashboards/kube-apiserver.json
dashboards/kube-controller-manager.json
dashboards/kube-proxy.json
dashboards/kube-scheduler.json
dashboards/kubelet.json

Result

Display the contents of the dashboard catalog.

ls -l dashboards

-rw-r--r-- 1 povilasv povilasv 35746 Apr 26 08:29 kube-apiserver.json
-rw-r--r-- 1 povilasv povilasv 34790 Apr 26 08:29 kube-controller-manager.json
-rw-r--r-- 1 povilasv povilasv 62845 Apr 26 08:29 kubelet.json
-rw-r--r-- 1 povilasv povilasv 27673 Apr 26 08:29 kube-proxy.json
-rw-r--r-- 1 povilasv povilasv 25650 Apr 26 08:29 kube-scheduler.json

Adding dashboards to Grafana

I have already said that I highly recommend placing panels through configuration files. You can learn more about how this is done in the Grafana documentation .

On the other hand, nothing prevents you from opening the Grafana interface and adding panels through it.

Update dashboards

Over time, the panels will change. To update them, the following command is enough:

jb update 

That's all for today.

I hope you enjoy my dashboards. In a separate article I will talk about how I created them and why I made them just like that, so be sure to subscribe to updates. Subscribers see my work first, even before publication.

I am also extremely interested in your feedback. If you used my panels, evaluate how useful they were to you, or share if they helped you out in a real working situation. Just leave a comment!

By the way, if you want to thank me for my efforts, you can see my open wish list and buy me, for example, a good book.

I really appreciate quality open source projects. Kubernetes-grafana-mixin package available under license  Apache 2 . If you liked the project, check it out on Github!