How to Monitor Kubernetes with Grafana and Prometheus

Kubernetes Monitoring with Grafana & Prometheus: Step-by-Step

Kubernetes Monitoring with Grafana & Prometheus: Step-by-Step

Kubernetes has become the foundational layer for modern application deployment, yet its dynamic, distributed nature makes comprehensive observability a non-negotiable requirement for operational success. You need to understand the health of your nodes, the performance of your pods, and the state of your cluster to ensure reliability and avoid costly downtime. This guide provides a definitive, step-by-step path to mastering how to monitor Kubernetes with Grafana and Prometheus, transforming raw metrics into actionable insights that allow you to proactively manage your infrastructure.

What You'll Learn

By the end of this guide, you'll understand how to deploy a complete, production-ready monitoring stack on your Kubernetes cluster using the industry-standard kube-prometheus-stack. You'll be able to connect Grafana to Prometheus, import pre-built dashboards for immediate visibility, and walk away with a clear plan for customizing alerts and visualizations to suit your specific needs. The single most important takeaway is that the kube-prometheus-stack Helm chart is the fastest and most reliable method to achieve full-stack Kubernetes monitoring.

The Industry-Standard Monitoring Stack for Kubernetes

The combination of Prometheus and Grafana is the de facto standard for Kubernetes monitoring. According to industry data, an estimated 75% of organizations using Kubernetes adopt Prometheus and Grafana for their observability needs . This isn't just a trend; it's a testament to the power and flexibility of this open-source toolkit.

• Prometheus is a Cloud Native Computing Foundation (CNCF) graduated project designed for reliability and scalability. It works by scraping metrics from various targets (like your application pods or Kubernetes nodes) at specified intervals and storing them in a time-series database. You can then query this data using PromQL (Prometheus Query Language), a powerful and flexible language for aggregating and analyzing metrics .
• Grafana is the visualization layer. It integrates seamlessly with Prometheus, transforming the raw, scraped data into beautiful, real-time dashboards and charts. This visual representation makes it significantly easier to spot trends, detect anomalies, and troubleshoot performance issues at a glance . Grafana can also serve as a unified UI for other data sources, but for Kubernetes, Prometheus is its most crucial partner.

Prerequisites for Your Setup

Before you begin, ensure you have the following tools installed and configured. This guide assumes you have a working Kubernetes cluster, whether it's a local development environment like Minikube or a production cluster in the cloud.

• A Running Kubernetes Cluster: You need a cluster to deploy the monitoring stack onto. For local testing, Minikube or Kind works perfectly .
• Helm: The Kubernetes package manager is essential for deploying the kube-prometheus-stack. Ensure you have Helm v3.2+ installed on your local machine .
• kubectl: The Kubernetes command-line tool must be configured to communicate with your cluster .

Step-by-Step Deployment: Using the kube-prometheus-stack

The most efficient and recommended way to deploy Prometheus and Grafana for Kubernetes monitoring is by using the kube-prometheus-stack Helm chart. This chart is maintained by the Prometheus Community and bundles all the necessary components into a single, easy-to-manage package . It includes:

• Prometheus Operator: Manages the Prometheus and Alertmanager instances.
• Prometheus: The core metrics server.
• Alertmanager: Handles alerts sent by Prometheus.
• Grafana: The visualization platform.
• Node Exporter: Collects node-level metrics (CPU, memory, disk) from your Kubernetes nodes.
• kube-state-metrics: Generates metrics about the state of Kubernetes objects like pods, deployments, and services .

1. Add the Prometheus Community Helm Repository

First, add the official Helm repository and update it to ensure you get the latest chart versions .

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

2. Install the Stack

You can install the stack with a single Helm command, customizing it with various parameters. This example creates a monitoring namespace and enables persistent storage for Prometheus and Grafana data .

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.service.type=NodePort \
  --set grafana.service.nodePort=30094 \
  --set grafana.adminPassword=admin \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.accessModes=["ReadWriteOnce"] \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi \
  --set prometheus.prometheusSpec.retention=30d

• grafana.adminPassword=admin: Sets the admin password for Grafana. Change this in production.
• --set grafana.service.type=NodePort: Exposes Grafana outside the cluster via a NodePort for easy access.
• prometheus.prometheusSpec.retention=30d: Sets the Prometheus data retention period to 30 days, which is essential for long-term trend analysis .

3. Verify the Installation

Check that all pods in the monitoring namespace are running.

kubectl get pods -n monitoring

The output should show several pods with a status of Running, including those for Grafana, Prometheus, Alertmanager, node-exporter, and kube-state-metrics .

Connecting Grafana to Your Prometheus Data Source

The kube-prometheus-stack Helm chart often configures Prometheus as a data source in Grafana automatically. However, it's best to verify the connection .

Accessing the Grafana Interface

To access the Grafana web interface, you can use port-forwarding for local testing or expose the service with a LoadBalancer or Ingress for production use .

Option A: Port-Forwarding (For Quick Local Access)

This is the simplest way to access your dashboard temporarily .

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

Now, open your browser and go to http://localhost:3000.

⚠️ Important: Port-forwarding is only suitable for local testing and development. For a permanent, production setup, you should expose Grafana using a Service of type LoadBalancer or an Ingress Controller .

Option B: LoadBalancer (Recommended for Cloud Environments)

If you're running on a cloud provider like UpCloud, AWS, or GCP, you can simply patch the Grafana service to use a LoadBalancer .

kubectl patch svc prometheus-grafana -n monitoring -p '{"spec": {"type": "LoadBalancer"}}'

Then, retrieve the external IP address:

kubectl get svc -n monitoring prometheus-grafana

Access Grafana at http://<EXTERNAL-IP>:80.

Logging In

Use the default credentials to log in:

• Username: admin
• Password: The password you set during installation (e.g., admin or prom-operator) .

Once logged in, navigate to Configuration > Data Sources to confirm that a Prometheus data source exists and is working. Click on it and use the Test button to ensure connectivity .

Building Dashboards: From Pre-Built to Custom

With Grafana connected to Prometheus, you can start visualizing your Kubernetes metrics. The fastest way to get valuable insights is to import pre-built dashboards from the Grafana community.

Importing Pre-Built Dashboards

Grafana offers a vast library of community dashboards that you can import using their unique IDs .

1. Hover over the + icon in the sidebar and select Import.
2. Enter a dashboard ID in the "Find and import dashboards" field. Two highly recommended dashboards are:
   • 1860: Node Exporter Full – provides detailed metrics about your cluster's nodes, including CPU, memory, disk, and network .
   • 8588: Kubernetes Deployment Statefulset Daemonset – offers a view of your Kubernetes workloads .
3. Click Load, then select your Prometheus data source and click Import.

These dashboards will give you an immediate, comprehensive view of your cluster health without writing any PromQL queries.

Creating Custom Dashboards with PromQL

While pre-built dashboards are a great starting point, you'll often need to create custom views tailored to your specific applications. This is where PromQL comes in.

To create a new dashboard, click the + icon and select New Dashboard. Then, click Add visualization and choose your Prometheus data source .

Here are a few essential PromQL queries for Kubernetes monitoring:

• CPU Usage per Pod:

  sum(rate(container_cpu_usage_seconds_total{namespace!="kube-system", pod!=""}[5m])) by (pod)

• Memory Usage per Pod:

  sum(container_memory_working_set_bytes{namespace!="kube-system", pod!=""}) by (pod) / 1024 / 1024 / 1024

• Pod Status by Phase:

  sum(kube_pod_status_phase) by (phase)

  This query is perfect for a Stat panel, allowing you to see at a glance if all your pods are running .

Configuring Alerts and Notifications

Monitoring is only half the battle; you also need to be notified when things go wrong. This is the role of Alertmanager, which is included in the kube-prometheus-stack.

Setting Up Alert Rules

You can configure Alertmanager to send notifications to various channels, such as Slack, email, or PagerDuty. A common and effective setup is to send alerts to a dedicated Slack channel .

1. First, create an incoming webhook in your Slack workspace.
2. Then, you need to modify the Alertmanager configuration. You can do this by providing a custom values.yaml file during a Helm upgrade or by editing the secret directly.
3. Update the alertmanager section in your values.yaml file with a route and receiver pointing to your Slack webhook URL :

alertmanager:
  config:
    route:
      receiver: 'slack-notifications'
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
    receivers:
      - name: 'slack-notifications'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/your/webhook/url'
            send_resolved: true
            title: '{{ .CommonLabels.alertname }} ({{ .Status | toUpper }})'
            text: >-
              {{ range .Alerts }}
              *Alert:* {{ .Labels.alertname }}
              *Severity:* {{ if .Labels.severity }}{{ .Labels.severity }}{{ else }}n/a{{ end }}
              *Summary:* {{ if .Annotations.summary }}{{ .Annotations.summary }}{{ else }}n/a{{ end }}
              *Description:* {{ if .Annotations.description }}{{ .Annotations.description }}{{ else }}n/a{{ end }}
              {{- end }}

After applying these changes with a helm upgrade, alerts will be sent to your Slack channel .

Best Practices for Production Monitoring

To ensure your monitoring stack is robust and scalable in a production environment, follow these key best practices.

• Configure Persistent Storage: When deploying in production, always enable persistent storage for both Prometheus and Grafana . This ensures your metrics data and dashboard configurations survive pod restarts and cluster upgrades. In the kube-prometheus-stack, this is controlled by the persistence and storageSpec settings in the Helm chart .
• Optimize Query Performance: High-cardinality queries can slow down your dashboards and Prometheus itself. To maintain performance, always aggregate your PromQL queries at the source. For example, instead of querying container_cpu_usage_seconds_total, aggregate it by namespace: sum by (namespace) (rate(container_cpu_usage_seconds_total[5m])). This reduces the amount of data Grafana needs to process .
• Set Appropriate Retention Periods: The default Prometheus retention period is 15 days . Adjust this based on your needs. A longer retention period (e.g., 30-60 days) is useful for long-term trend analysis, but it will require more storage space .
• Manage Configuration as Code: As your monitoring needs evolve, you'll likely customize dashboards and alert rules. Export your dashboards as JSON and store them in version control (e.g., Git) alongside your alert rules. This practice, known as GitOps, makes your monitoring setup reproducible and auditable .
• Tune Scrape Intervals: The default scrape interval for Prometheus is 30 seconds . For critical, high-velocity metrics, you might want to scrape more often, but be aware that this increases the load on Prometheus and your cluster's API server. You can customize these intervals in the Helm values .

Frequently Asked Questions

1. What is the easiest way to monitor Kubernetes with Grafana and Prometheus? The easiest and most recommended method is to use the kube-prometheus-stack Helm chart. It bundles Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics into a single package, and you can deploy it with just a few commands .

2. Is it necessary to use kube-state-metrics for monitoring Kubernetes? Yes, kube-state-metrics is a critical component that generates metrics about the state of your Kubernetes objects, like the number of ready pods or the status of a deployment . Without it, you wouldn't have visibility into the health of your higher-level Kubernetes resources.

3. How can I get my Kubernetes dashboards to load faster? Slow dashboards are often caused by inefficient PromQL queries. A key fix is to aggregate your queries to reduce cardinality. For example, instead of displaying data for every pod, summarize it at the namespace level using queries like sum by (namespace) (rate(...)) .

4. What's the best way to get alerts from my Kubernetes monitoring? Prometheus works with Alertmanager, which can send notifications to various channels. A popular and simple method is to configure Alertmanager to send alerts to a Slack channel via an incoming webhook . This provides real-time, actionable notifications.

5. Do I need to manually install Node Exporter on every node? No. When you deploy the kube-prometheus-stack Helm chart, it automatically deploys a Node Exporter DaemonSet. This ensures that a Node Exporter pod runs on every node in your cluster, allowing Prometheus to collect node-level metrics like CPU, memory, and disk usage without any manual intervention .

Sources

1. UpCloud Global. (2025). Monitoring on UpCloud with Prometheus: Part 3.
2. Ubuntu Documentation. (2026). How to use Prometheus with Canonical Kubernetes.
3. Sfeir Institute. (2026). Create Effective Grafana Dashboards for Kubernetes Monitoring.
4. Ory. (2025). Monitoring.
5. GitHub - AkramGalal. (2025). deploying-prometheus-grafana-K8s-using-helm.
6. Go Packages. (2025). kube-prometheus.
7. GitHub - dotdc. (2026). grafana-dashboards-kubernetes.
8. Scaleway. (2025). Kubernetes Kapsule cluster monitoring with Prometheus & Grafana.
9. Mirantis. (2025). How to Set Up Prometheus & Grafana in K8s with Alertmanager & Slack Alerts.
10. Microsoft Learn. (2025). Using ConfigMap to customize Prometheus metric collection in Azure Monitor.
11. GitHub - Rurutia1027. (2025). Hands-on-Lab.K8s-Monitoring.

— Editorial Team