Monitoring with Prometheus in Kubernetes in 15 minutes

Original author: Giancarlo Rubio
  • Transfer
  • Tutorial


Note perev. : The author of the article, Giancarlo Rubio - DevOps engineer from the IT company LINKIT (Netherlands) - through the online resource ITNEXT shares a concise recipe for setting up monitoring with Prometheus in Kubernetes using the Prometheus Operator. The instruction appeared as a result of recent experience in the selection and implementation of a proactive monitoring system after the project migrated from bare metal to the cloud infrastructure. The recipe is perfect for quick theoretical (first half of the article) and practical (second half) acquaintance. For some teams URLs were fixed, which in the original material, most likely, were converted by the medium engine.

K8s cluster monitoring will help your team to implement:

  • proactive monitoring
  • displaying cluster visibility and planning its employment / performance,
  • notifications about triggering of triggers and warnings (alerts),
  • panels with metrics.

Not so long ago, CoreOS launched the Prometheus Operator , which perfectly matched my needs, simplifying the entire initial configuration.

Kubernetes Operator


“Operators, we call a new class of software. An operator is an application-specific controller that extends the Kubernetes API with the ability to create, configure, and manage instances of complex stateful applications on behalf of the Kubernetes user. The operator is based on the concepts of a resource and a controller in Kubernetes and adds to them knowledge from a specific area or application-specific to automate basic tasks. ”
- Brandon Philips from CoreOS.

Operators for Kubernetes were introduced in 2016 and offer the ability to abstract the deployment and configuration of applications. I myself managed to actively work with elasticsearch-operator and prometheus-operator . A detailed description of the operators for Kubernetes is beyond the scope of this article, and I would like to invite everyone interested to read the list of other available operators here .

Note perev. : We wrote more about operators for Kubernetes and their use for working with stateful applications in this article .

Prometheus


Prometheus is a set of open source utilities for monitoring and notifications (alerts), inspired by Google Borg Monitor. Its development began at SoundCloud, and then transferred to the CNCF fund. ( Note : We recently wrote about the history of Prometheus in this article .)

Prometheus supports applications in many programming languages . Being an excellent solution for monitoring infrastructure and applications, it also offers ready-made integration for connecting well-known applications (including PostgreSQL, MySQL, AWS Cloudwatch, etcd, K8s, etc.) as exporters .

Prometheus operator


“The mission of the Prometheus Operator is to make launching Prometheus on top of Kubernetes as simple as possible, while maintaining its configurability and configuring it in its native form for Kubernetes.”
- Getting Started Guide by Prometheus Operator.

The Prometheus Operator offers easy monitoring for services and deployments in K8s, as well as managing the settings of Prometheus, Alertmanager and Grafana (see below for more details) .

How it works?



Architecture Prometheus Operator. Source: GitHub

When you deploy a new version of the application, K8s creates a new one and, when this one is ready, destroys the old one. Prometheus constantly monitors the Kubernetes API and, when it detects changes, creates a new Prometheus configuration, guided by changes in services (sub).

Servicemonitor


Prometheus Operator uses a Custom Resource Definition (CRD) called ServiceMonitor to abstract the configuration. In the example below, you can see how to use ServiceMonitor to monitor under with nginx. ServiceMonitor selects under with nginx using the matchLabels selector . The Prometheus Operator looks for pods matching the label selector and creates the target from which Prometheus will collect metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  namespaceSelector:
    matchNames:
    - default
  endpoints:
  - port: web
interval: 30s

We try it ourselves


Further instructions can be completed in less than 15 minutes if you already have:

  1. Raised and working Kubernetes cluster.
  2. Installed and working Helm .
  3. helm list.

Installation


1. Create a namespace and clone the prometheus-operator Git repository:

$ kubectl create ns monitoring
$ git clone https://github.com/coreos/prometheus-operator.git
$ cd prometheus-operator

2. Install prometheus-operator deployment:

$ helm install --name prometheus-operator \ 
  --set rbacEnable=true --namespace=monitoring helm/prometheus-operator

3. Install Prometheus and Alertmanager specs, as well as Grafana deployment:

$ helm install --name prometheus --set serviceMonitorsSelector.app=prometheus \ 
  --set ruleSelector.app=prometheus --namespace=monitoring helm/prometheus
$ helm install --name alertmanager --namespace=monitoring helm/alertmanager
$ helm install --name grafana --namespace=monitoring helm/grafana

4. Install kube-prometheus to download the predefined k8s exporters and serviceMonitors:

$ helm install --name kube-prometheus --namespace=monitoring helm/kube-prometheus

If everything went well, you can run this command to list the applications:

$ kubectl get pods -n monitoring
NAME                                                      READY     STATUS    RESTARTS   AGE
alertmanager-alertmanager-0                               2/2       Running   0          3m
grafana-grafana-3066287131-brj8n                          2/2       Running   0          4m
kube-prometheus-exporter-kube-state-2696859725-s8m56      2/2       Running   0          3m
kube-prometheus-exporter-node-029w0                       1/1       Running   0          3m
kube-prometheus-exporter-node-n3txz                       1/1       Running   0          3m
kube-prometheus-exporter-node-q2rk3                       1/1       Running   0          3m
prometheus-operator-prometheus-operator-514889780-qm3fp   1/1       Running   0          4m
prometheus-prometheus-0                                   2/2       Running   0          3m

Prometheus


Forward the Prometheus server to your computer to access the panel through http://localhost:9090:

$ kubectl port-forward -n monitoring prometheus-prometheus-0 9090



In the Prometheus panel, you can make requests for metrics, view the predefined notifications and targets of Prometheus.

Note : If any targets return an unavailability error, check the security groups and firewall rules. If you do not have the goals shown in the screenshot above, check the K8s hearth labels, as sometimes the utilities used to deploy the cluster do not install them.

Please note (No. 2) : In the prometheus-operator project, they are working on packing standard notifications for K8s in the Helm chart. However, now to load them you need to execute the sequence of commands below (in the future this need will disappear):

$ sed -ie 's/role: prometheus-rulefiles/app: prometheus/g' contrib/kube-prometheus/manifests/prometheus/prometheus-k8s-rules.yaml
$ sed -ie 's/prometheus: k8s/prometheus: prometheus/g' contrib/kube-prometheus/manifests/prometheus/prometheus-k8s-rules.yaml
$ sed -ie 's/job=\"kube-controller-manager/job=\"kube-prometheus-exporter-kube-controller-manager/g' contrib/kube-prometheus/manifests/prometheus/prometheus-k8s-rules.yaml
$ sed -ie 's/job=\"apiserver/job=\"kube-prometheus-exporter-kube-api/g' contrib/kube-prometheus/manifests/prometheus/prometheus-k8s-rules.yaml
$ sed -ie 's/job=\"kube-scheduler/job=\"kube-prometheus-exporter-kube-scheduler/g' contrib/kube-prometheus/manifests/prometheus/prometheus-k8s-rules.yaml
$ sed -ie 's/job=\"node-exporter/job=\"kube-prometheus-exporter-node/g' contrib/kube-prometheus/manifests/prometheus/prometheus-k8s-rules.yaml
$ kubectl apply -n monitoring -f contrib/kube-prometheus/manifests/prometheus/prometheus-k8s-rules.yaml

Grafana


For debugging purposes, Prometheus has an expression browser . To get a beautiful dashboard, use Grafana with the built-in ability to fulfill requests in Prometheus.

Note : In the prometheus-operator project, they are working on creating a simple deployment for Grafana, probably using the new CRD . At the moment, to configure it, you need to run the following commands (in the future, this need will disappear):

$ sed -ie 's/grafana-dashboards-0/grafana-grafana/g' https://raw.githubusercontent.com/coreos/prometheus-operator/master/contrib/kube-prometheus/manifests/grafana/grafana-dashboards.yaml
$ sed -ie 's/prometheus-k8s.monitoring/prometheus-prometheus.monitoring/g' https://raw.githubusercontent.com/coreos/prometheus-operator/master/contrib/kube-prometheus/manifests/grafana/grafana-dashboards.yaml
$ kubectl apply -n monitoring -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/contrib/kube-prometheus/manifests/grafana/grafana-dashboards.yaml
$ kubectl port-forward -n monitoring $(kubectl get pods --selector=app=grafana-grafana -n monitoring --output=jsonpath={.items..metadata.name})  3000

Wait a few seconds for Grafana to load the data, open it http://localhost:3000in a browser and learn great graphics!


Grafana: Affordable Dashboards for Kubernetes Grafana: Schedules for Kubernetes


Employment / Performance Planning

Alertmanager


Alertmanager serves notifications sent by client applications like the Prometheus server. It provides elimination of duplicates, grouping, sending to the correct recipient service such as email, PagerDuty or OpsGenie. He is also responsible for silence and inhibit notifications.

We already installed Alertmanager with the commands above, and it remains to forward the service port to your computer, after which it can be opened http://localhost:9093in a web browser:

$ kubectl port-forward -n monitoring alertmanager-alertmanager-0 9093

Done


All K8s components are added to monitoring. I also recommend watching this video on YouTube in order to better understand how the prometheus-operator works.

PS from the translator


The author also promises to tell in his next article about setting up notifications using Alertmanager. Her appearance can be expected here .

Read also in our blog:


Also popular now: