Grafana

Mon, 01 Jan 2024 00:00:00 +0000

Prometheus shows you the spike. It tells you memory climbed at 14:32, error rate crossed 5% at 14:35, and latency hit 2 seconds at 14:37. But raw PromQL results are numbers in a table. You cannot see the shape of an incident in a table. You cannot hand a table to a product manager and explain what happened.

So you use Grafana. It connects to Prometheus (and Loki, and a dozen other data sources) and turns those numbers into dashboards. You see the spike, the timeline, the correlation between services — all on one screen.

Data sources

Grafana is a visualisation layer, not a storage layer. It queries data sources and renders the results. In a Kubernetes observability stack, the typical setup:

Data source	What it provides
Prometheus	Metrics — CPU, memory, request rates, error rates, latency
Loki	Logs — searchable, filterable, correlated with metrics by time
Jaeger / Tempo	Traces — individual request journeys across services

Adding a data source is a few fields in the UI or a ConfigMap if you manage Grafana as code.

Dashboards

A dashboard is a collection of panels. Each panel runs a query against a data source and renders the result as a graph, gauge, stat, table, or heatmap.

The fastest way to get useful dashboards is grafana.com/grafana/dashboards — a library of community dashboards for almost every common component. Import by ID:

1860 — Node Exporter Full (host metrics: CPU, memory, disk, network)
6417 — Kubernetes cluster overview
7362 — MySQL overview
9628 — Postgres overview

Import these on day one and you have coverage before writing a single PromQL query.

Variables

Dashboard variables make panels reusable across namespaces, clusters, or services. A variable populated from a Prometheus label query:

label_values(kube_pod_info{namespace=~".+"}, namespace)

Now every panel can use $namespace in its query, and a dropdown at the top of the dashboard filters the whole view.

Alerting

Grafana has its own alert engine that evaluates queries on a schedule and routes alerts through contact points (Slack, PagerDuty, email). For Kubernetes setups already using Alertmanager, it is usually cleaner to define alert rules in Prometheus and use Grafana purely for visualisation — one place for alert rules, not two.

Managing Grafana as code

Dashboards built in the UI are fragile — they live in a database and disappear if you rebuild the stack. Two better approaches:

Grafana provisioning — mount dashboard JSON files via ConfigMap. Grafana loads them on startup and they survive restarts.

Grafonnet / Jsonnet — generate dashboard JSON programmatically. Verbose but version-controllable and reviewable in pull requests.

The observability trio

Grafana is the front end for the full observability stack:

Prometheus — something is wrong, here are the numbers
Loki — here are the log lines from that time window
Jaeger — here is the exact request that failed and where it slowed down

Each answers a different question. Grafana is where you look at all three in one place.

Resources

Grafana documentation
Grafana dashboard library
kube-prometheus-stack — installs Prometheus + Grafana together

Loki

Mon, 01 Jan 2024 00:00:00 +0000

Prometheus tells you that something is wrong and when it started. Loki tells you what happened — it is the log aggregation layer of the observability stack. Logs from every pod across every node are collected, indexed, and made searchable in one place. Grafana is the front end for both.

How it works

Loki stores logs as compressed chunks, indexed only by labels (not by content). This makes it cheap to store and fast to query by label — namespace, pod name, app — but slower for full-text search than something like Elasticsearch. The trade-off is intentional: label-scoped queries cover the vast majority of real operational use, and the storage cost is dramatically lower.

Promtail runs as a DaemonSet on every node, tails log files from /var/log/pods/, attaches Kubernetes labels, and ships to Loki. Grafana queries Loki directly.

Deployment modes

SingleBinary — ingestion, querying, and management all run in a single instance. Simple to deploy, minimal operational overhead. A single point of failure: if it goes down, ingestion stops and logs are lost. The right starting point for most clusters.

SimpleScalable — responsibilities split into separate pods, each running a minimum of two instances for HA. Ingestion, querying, and the compactor can be scaled independently. Significantly more operational overhead, but fault-tolerant and tunable under load. The right move for production once you have volume and reliability requirements.

Getting started

The fastest path to a working stack is deploying Loki alongside kube-prometheus-stack, which brings up Prometheus, Grafana, and Alertmanager together. See the Prometheus note for the kube-prometheus-stack setup and the ArgoCD CRD workaround.

Loki and Promtail are installed as a separate ArgoCD Application, using multiple Helm sources with values pulled from the cluster config repo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: log-ingestion
 namespace: argo-cd
spec:
 project: default
 sources:

 # Loki
 - repoURL: "https://grafana.github.io/helm-charts"
 chart: loki
 targetRevision: 6.55.0
 helm:
 releaseName: loki
 valueFiles:
 - $values/cluster/testing/overlay/monitoring/helm/loki-values.yaml

 # Promtail
 - repoURL: "https://grafana.github.io/helm-charts"
 chart: promtail
 targetRevision: 6.17.1
 helm:
 releaseName: promtail
 valueFiles:
 - $values/cluster/testing/overlay/monitoring/helm/promtail-values.yaml

 # Values source — cluster config repo
 - repoURL: 'git@github.com:example-org/cluster-config.git'
 targetRevision: HEAD
 ref: values

 destination:
 server: https://kubernetes.default.svc
 namespace: monitoring
 syncPolicy:
 automated:
 selfHeal: true
 prune: true
 syncOptions:
 - CreateNamespace=true
 - ServerSideApply=true

Note: targetRevision: HEAD is fine for testing environments. Pin to a tag for staging and production.

Promtail deprecation

Promtail is deprecated as of February 2025 and in LTS — security fixes only, no new features. Expected EOL is end of 2026.

The Grafana-recommended replacement is Grafana Alloy, a more capable collector that handles metrics, logs, and traces in a single agent. The migration path is not yet settled enough for a confident recommendation — worth waiting for clear community consensus before moving. Until then, Promtail continues to work and the LTS window gives time to plan.

Grafana integration

Add Loki as a data source in Grafana and logs become queryable alongside metrics. A useful starting point is a simple app-oriented logs dashboard — filter by namespace and pod, tail in near-real-time, correlate timestamps with Prometheus spikes.

LogQL, Loki’s query language, mirrors PromQL in style:

# All error logs from a namespace
{namespace="production"} |= "error"

# Parse and filter structured logs
{app="my-api"} | json | status >= 500

# Rate of error log lines over time
rate({namespace="production"} |= "error" [5m])

Resources

Loki documentation
Grafana Alloy documentation — future Promtail replacement
loki-stack Helm chart
kube-prometheus-stack

Grafana on Backend Engineering Strategy Tools

Grafana

Data sources

Dashboards

Variables

Alerting

Managing Grafana as code

The observability trio

Resources

Loki

How it works

Deployment modes

Getting started

Promtail deprecation

Grafana integration

Resources