Monitoring Kubernetes with Prometheus and Grafana: Complete Setup Guide

Kubernetes is dynamic by design. Pods appear and disappear, workloads scale horizontally, nodes become unavailable, and deployments continuously modify the state of the cluster. That dynamism makes observability essential. Without reliable monitoring, engineers often discover problems only after users experience them. CPU saturation, memory pressure, failed deployments, container restarts, and application latency can all develop quietly until they become production incidents. Prometheus and Grafana form one of the most widely adopted open-source combinations for addressing this challenge. Prometheus collects and stores time-series metrics, while Grafana turns those metrics into interactive dashboards and visualizations. Together, they provide a practical foundation for understanding Kubernetes health and application performance. Why Prometheus and Grafana? Prometheus is a metrics and monitoring system designed around time-series data. It periodically collects metrics from configured targets and stores them with labels that make multidimensional querying possible. Grafana sits on top of that data. Instead of examining raw metric values, engineers can create dashboards showing CPU utilization, memory consumption, request rates, latency, pod restarts, and other operational indicators. The combination is powerful because it supports both real-time troubleshooting and longer-term analysis. For example, a sudden memory increase might initially look like an isolated incident. Historical Grafana data can reveal whether it is actually part of a gradual memory leak. Kubernetes Monitoring Architecture A typical Kubernetes monitoring stack contains several components: Prometheus — collects and stores metrics. Grafana — visualizes metrics and creates dashboards. Alertmanager — routes and manages Prometheus alerts. Node Exporter — exposes operating-system and node-level metrics. kube-state-metrics — exposes metrics representing Kubernetes object state. Prometheus Operator — simplifies the configuration and management of Prometheus resources. A simplified flow looks like this: Kubernetes workloads → Metrics endpoints → Prometheus → Grafana Alerts generally follow another path: Prometheus → Alertmanager → Notification channel This separation keeps collection, visualization, and alert routing logically distinct. Prerequisites and Cluster Preparation Before installing the monitoring stack, ensure the Kubernetes cluster is operational and accessible through kubectl. Helm should also be installed because it provides a convenient mechanism for deploying the monitoring components. Verify the cluster: kubectl get nodes Confirm that the nodes are healthy and that workloads can be scheduled. Then verify Helm: helm version A dedicated namespace is useful for separating observability components from application workloads: kubectl create namespace monitoring Namespace isolation makes resource management, access control, and troubleshooting more straightforward. Installing Prometheus and Grafana with Helm The Prometheus Community Helm repository provides a convenient way to install a Kubernetes monitoring stack. Add the repository: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update Install the kube-prometheus-stack chart: helm install monitoring prometheus-community/kube-prometheus-stack \ --namespace monitoring Check the resulting workloads: kubectl get pods -n monitoring You should see components associated with Prometheus, Grafana, Alertmanager, exporters, and the Prometheus Operator. If the pods are not becoming ready, inspect them individually: kubectl describe pod -n monitoring Logs are often the fastest route to identifying configuration, scheduling, image-pull, or permission problems. Configuring Prometheus for Kubernetes Metrics Prometheus needs to know which endpoints should be scraped. The Prometheus Operator makes this easier through Kubernetes custom resources such as ServiceMonitor and PodMonitor. A ServiceMonitor can define which services expose metrics and how Prometheus should collect them. For example, an application might expose: /metrics on port 8080. A corresponding monitoring configuration can instruct Prometheus to periodically scrape that endpoint. This model is particularly useful in Kubernetes because services and workloads are ephemeral. Monitoring configuration can follow Kubernetes resources rather than relying exclusively on static infrastructure addresses. Understanding Kubernetes Metrics Kubernetes monitoring becomes much more useful when metrics are categorized. Infrastructure metrics These describe the underlying compute environment: CPU utilization Memory utilization Disk usage Network traffic Filesystem capacity Kubernetes object metrics These describe cluster state: Pod availability Deployment replicas DaemonSet status StatefulSet replicas Job completion Container restart counts Application metrics These describe the application itself: Requests per second Error rate Request latency Queue depth Database connection usage The distinction matters because infrastructure can appear healthy while an application is failing. Building Grafana Dashboards After installation, Grafana can be exposed through a service. Check available services: kubectl get svc -n monitoring For temporary local access, port forwarding is convenient: kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring Grafana can then be accessed locally. A useful Kubernetes dashboard should answer operational questions quickly: Are nodes healthy? Which workloads are consuming the most CPU? Which pods are restarting? Are memory limits being exceeded? Are error rates increasing? Is network traffic abnormal? Are workloads properly replicated? Dashboards should prioritize actionable information rather than becoming a collage of every metric available. Creating Prometheus Alerts Dashboards tell engineers what is happening. Alerts tell them when intervention may be necessary. Prometheus alert rules can detect conditions such as: High CPU utilization High memory consumption Persistent pod restarts Disk exhaustion Unavailable replicas Increased application error rates High request latency A simplified alert rule might look like: groups: - name: kubernetes-alerts rules: - alert: HighNodeCPU expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 10m labels: severity: warning annotations: summary: "High node CPU utilization" The for clause is important. It prevents transient spikes from immediately generating noisy alerts. Alert quality matters more than alert quantity. Monitoring Nodes and Cluster Resources Node-level monitoring helps identify resource exhaustion before workloads become unstable. Important indicators include: CPU utilization Memory utilization Disk capacity Disk I/O Network throughput Filesystem usage Node readiness A node approaching memory exhaustion may begin evicting pods. A filesystem approaching capacity can cause unexpected application failures. Monitoring trends is particularly valuable. A node consuming 70% of disk space today may not appear problematic, but if utilization increases by several percent every day, it deserves attention. Monitoring Pods, Deployments, and Services At the workload level, monitor both desired and actual state. Useful indicators include: Available replicas Desired replicas Pod restarts Pending pods CrashLoopBackOff conditions Container readiness Deployment rollout status A deployment configured for five replicas but running only three represents a reliability concern even if the remaining pods appear healthy. Similarly, repeated container restarts may indicate application crashes, resource constraints, configuration problems, or failing dependencies. Metrics should therefore be interpreted alongside Kubernetes events and logs. Monitoring Application Metrics Infrastructure metrics alone cannot explain application behavior. Applications should expose meaningful metrics where possible. For an HTTP service, useful metrics include: Request count Request duration HTTP status codes Error rate Active requests Dependency latency Prometheus can scrape these application metrics when they are exposed through compatible endpoints. A common pattern is to combine infrastructure metrics with application telemetry. For example, increased CPU usage becomes much more meaningful when correlated with a sudden increase in request volume. Monitoring Kubernetes Control Plane Components Control-plane observability becomes increasingly important as clusters grow. Depending on the Kubernetes environment and access model, monitoring can include: API server performance Scheduler activity Controller manager behavior etcd health API request latency API error rates Managed Kubernetes services may restrict direct access to certain control-plane components. In those environments, use the metrics and monitoring capabilities exposed by the cloud provider. Control-plane degradation can manifest as slow deployments, delayed scheduling, failed API requests, or inconsistent cluster behavior. Troubleshooting Common Monitoring Problems Prometheus is not scraping a target Check the target configuration and inspect Prometheus targets. Verify: Service labels ServiceMonitor selectors Namespace configuration Metrics endpoint Port names Network connectivity Grafana shows no data First verify that Prometheus contains the expected metrics. If Prometheus has data but Grafana does not, check the Grafana data-source configuration and PromQL queries. Pods continuously restart Inspect: kubectl get pods -n kubectl describe pod -n kubectl logs -n Then correlate the restart behavior with CPU, memory, and application metrics. Alerts are too noisy Review thresholds and durations. Alerts should represent conditions that require meaningful action, not every minor fluctuation. Production Best Practices A production monitoring system should itself be treated as critical infrastructure. Define meaningful alerts Avoid creating alerts merely because a metric exists. Every alert should have a clear operational response. Establish retention policies Prometheus storage requirements can grow rapidly. Configure retention according to operational needs and available capacity. Protect Grafana Use proper authentication and role-based access instead of exposing administrative interfaces publicly. Monitor the monitoring stack Prometheus, Grafana, and Alertmanager need their own health metrics. An observability system that silently fails creates a dangerous blind spot. Use recording rules Frequently executed expensive PromQL expressions can be precomputed with recording rules, improving query efficiency. Preserve useful historical data Long-term metrics can support capacity planning, performance analysis, incident investigation, and reliability engineering. Integrate alerts with incident workflows Alerts should reach the appropriate operational channels and contain enough contextual information for engineers to begin troubleshooting. Prometheus and Grafana provide a flexible foundation for Kubernetes observability. Prometheus handles metric collection and time-series analysis, while Grafana turns those measurements into operationally useful visualizations. The real value, however, comes from how the monitoring system is designed. A strong Kubernetes observability strategy combines node metrics, Kubernetes object state, application telemetry, meaningful alerts, historical analysis, and disciplined incident response. The goal is not to collect every possible metric. It is to create enough reliable signal that engineers can understand what is happening, identify why it is happening, and respond before a small anomaly becomes a major production incident.