Setting Up Kubernetes Monitoring with Prometheus & Grafana

If your Kubernetes cluster doesn't have monitoring, you don't have a cluster — you have a mystery box that occasionally serves traffic. This guide walks through the stack we deploy for every client: Prometheus for metrics, Grafana for dashboards, and AlertManager for routing pages to humans.

What we're building

• Prometheus scraping node, pod, and control-plane metrics
• Grafana with the dashboards that matter (not 400 panels nobody reads)
• AlertManager routing to Slack with sane severity levels

The whole thing runs inside the cluster via the kube-prometheus-stack Helm chart, which bundles the Prometheus Operator, exporters, and a starter set of alerting rules.

Step 1 — Install the stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  -f values.yaml

A minimal values.yaml that avoids the usual surprises:

prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 50Gi
grafana:
  adminPassword: changeme-use-a-secret
  persistence:
    enabled: true
    size: 5Gi

Two things people skip and regret: retention (the default keeps metrics for 10 days with no storage request, which evicts your Prometheus pod under pressure) and persistence for Grafana (lose the pod, lose your dashboards).

Step 2 — Verify scraping

Port-forward and check the targets page:

kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090

Open http://localhost:9090/targets. Everything should be UP. If kube-proxy targets are down (common on EKS), bind its metrics address:

kubectl -n kube-system edit configmap kube-proxy-config
# set metricsBindAddress: 0.0.0.0:10249

Step 3 — Dashboards that earn their place

The chart ships with dozens of dashboards. We keep four pinned:

1. Kubernetes / Compute Resources / Cluster — is the cluster itself healthy?
2. Node Exporter / Nodes — disk, memory, and CPU per node
3. Kubernetes / Compute Resources / Namespace (Pods) — who is eating the cluster
4. One per-service dashboard built from your app's RED metrics (rate, errors, duration)

If a dashboard hasn't answered a question in a month, delete it. Dashboard sprawl is how real signals get lost.

Step 4 — Alerts that wake the right person

AlertManager config routed to Slack, with severity doing actual work:

route:
  receiver: slack-warnings
  routes:
    - matchers: [severity="critical"]
      receiver: slack-critical
      repeat_interval: 1h
receivers:
  - name: slack-critical
    slack_configs:
      - channel: '#oncall'
        api_url: <webhook>
        title: '{{ .CommonAnnotations.summary }}'
  - name: slack-warnings
    slack_configs:
      - channel: '#alerts-feed'
        api_url: <webhook>

The rule we enforce everywhere: critical means a human acts now; warning means a human reads it tomorrow. The moment warnings page people at night, alert fatigue sets in and the real pages get ignored.

The three alerts that matter most

Beyond the bundled rules, add these:

- alert: PodCrashLooping
  expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
  for: 5m
  labels: { severity: critical }

- alert: NodeDiskPressure
  expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.10
  for: 10m
  labels: { severity: critical }

- alert: CertExpiringSoon
  expr: certmanager_certificate_expiration_timestamp_seconds - time() < 7 * 86400
  labels: { severity: warning }

Crash loops, full disks, and expired certificates cause more 2 AM incidents than everything else combined.

Where this falls short

Prometheus inside the cluster monitors the cluster — but when the cluster itself dies, your monitoring dies with it. For the "is the whole thing up" question you want something outside: a hosted check, a second small Prometheus, or an agent-based tool like our own Pulse, which we built precisely because of this gap.

Set this stack up once, keep it boring, and your future self at 2 AM will thank you.