Observability on Backend Engineering Strategy Tools

Elasticsearch & Kibana

Mon, 01 Jan 2024 00:00:00 +0000

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. Kibana is its web UI for querying, visualising, and exploring the data stored in Elasticsearch. Together they form the search and analysis layer of the ELK stack — typically with Logstash or Beats collecting and shipping data into Elasticsearch, and Kibana on top for humans to interact with it.

Elasticsearch

A document store where every document is JSON and every field is indexed by default. Queries are also JSON, using a rich query DSL that supports full-text search, structured filters, aggregations, and geospatial queries. Elasticsearch is horizontally scalable — an index is split into shards, shards are distributed across nodes, and replicas provide redundancy. Adding nodes increases both capacity and query throughput.

# Index a document
POST /logs/_doc
{
 "timestamp": "2026-06-04T12:00:00Z",
 "level": "error",
 "service": "api",
 "message": "connection refused"
}

# Search with filter
GET /logs/_search
{
 "query": {
 "bool": {
 "filter": [
 { "term": { "level": "error" } },
 { "term": { "service": "api" } }
 ]
 }
 }
}

At scale, index lifecycle management (ILM) policies handle the hot-warm-cold tiering automatically — recent indices stay on fast nodes, older indices roll to cheaper storage, and expired indices are deleted.

Kibana

The interface to Elasticsearch. Kibana’s core is Discover — a time-series log explorer with free-text search and field filtering — and Dashboards — composable visualisations (time series, bar charts, pie charts, data tables, maps) that query Elasticsearch directly. For log aggregation and observability use cases, a typical workflow is: ship logs into Elasticsearch via Filebeat or Logstash, explore them in Discover, build dashboards for the signals that matter, set up alerting rules on those patterns.

Kibana also hosts the Elastic APM UI (application performance monitoring), the SIEM app (security event correlation), and the Lens visual editor for building dashboards without writing aggregation queries by hand.

ELK vs the Grafana stack

The Grafana stack (Loki + Prometheus + Grafana) has become the common alternative for cloud-native environments. The key difference: Loki indexes only log metadata (labels), not the full log content — it is cheaper to run and query at scale, but full-text search across log bodies is slower. Elasticsearch indexes everything and full-text search is fast, but the storage and memory cost is significantly higher. For log volumes in the hundreds of GB/day and above, the operational cost of Elasticsearch becomes the dominant factor. For environments that need fast full-text search across structured and unstructured data — logs, documents, events — Elasticsearch earns its cost.

Resources

Grafana

Mon, 01 Jan 2024 00:00:00 +0000

Prometheus shows you the spike. It tells you memory climbed at 14:32, error rate crossed 5% at 14:35, and latency hit 2 seconds at 14:37. But raw PromQL results are numbers in a table. You cannot see the shape of an incident in a table. You cannot hand a table to a product manager and explain what happened.

So you use Grafana. It connects to Prometheus (and Loki, and a dozen other data sources) and turns those numbers into dashboards. You see the spike, the timeline, the correlation between services — all on one screen.

Data sources

Grafana is a visualisation layer, not a storage layer. It queries data sources and renders the results. In a Kubernetes observability stack, the typical setup:

Data source	What it provides
Prometheus	Metrics — CPU, memory, request rates, error rates, latency
Loki	Logs — searchable, filterable, correlated with metrics by time
Jaeger / Tempo	Traces — individual request journeys across services

Adding a data source is a few fields in the UI or a ConfigMap if you manage Grafana as code.

Dashboards

A dashboard is a collection of panels. Each panel runs a query against a data source and renders the result as a graph, gauge, stat, table, or heatmap.

The fastest way to get useful dashboards is grafana.com/grafana/dashboards — a library of community dashboards for almost every common component. Import by ID:

1860 — Node Exporter Full (host metrics: CPU, memory, disk, network)
6417 — Kubernetes cluster overview
7362 — MySQL overview
9628 — Postgres overview

Import these on day one and you have coverage before writing a single PromQL query.

Variables

Dashboard variables make panels reusable across namespaces, clusters, or services. A variable populated from a Prometheus label query:

label_values(kube_pod_info{namespace=~".+"}, namespace)

Now every panel can use $namespace in its query, and a dropdown at the top of the dashboard filters the whole view.

Alerting

Grafana has its own alert engine that evaluates queries on a schedule and routes alerts through contact points (Slack, PagerDuty, email). For Kubernetes setups already using Alertmanager, it is usually cleaner to define alert rules in Prometheus and use Grafana purely for visualisation — one place for alert rules, not two.

Managing Grafana as code

Dashboards built in the UI are fragile — they live in a database and disappear if you rebuild the stack. Two better approaches:

Grafana provisioning — mount dashboard JSON files via ConfigMap. Grafana loads them on startup and they survive restarts.

Grafonnet / Jsonnet — generate dashboard JSON programmatically. Verbose but version-controllable and reviewable in pull requests.

The observability trio

Grafana is the front end for the full observability stack:

Prometheus — something is wrong, here are the numbers
Loki — here are the log lines from that time window
Jaeger — here is the exact request that failed and where it slowed down

Each answers a different question. Grafana is where you look at all three in one place.

Resources

Grafana documentation
Grafana dashboard library
kube-prometheus-stack — installs Prometheus + Grafana together

Istio

Mon, 01 Jan 2024 00:00:00 +0000

Istio is a service mesh for Kubernetes. It injects a sidecar proxy (Envoy) into every pod, and all traffic between pods flows through these proxies rather than directly between containers. This gives the mesh control over traffic routing, security, and observability without any changes to application code.

What it solves

In a large microservice deployment, every service needs to handle retries, timeouts, circuit breaking, mutual TLS, and metrics collection — or skip them and accept the risk. Without a mesh, each team implements this differently, or not at all. Istio moves these concerns out of the application and into the infrastructure layer, where they are configured once and applied uniformly.

Traffic management

Istio’s VirtualService and DestinationRule CRDs give fine-grained control over how traffic is routed:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
 name: reviews
spec:
 hosts:
 - reviews
 http:
 - match:
 - headers:
 end-user:
 exact: test-user
 route:
 - destination:
 host: reviews
 subset: v2
 - route:
 - destination:
 host: reviews
 subset: v1

This routes a specific user to v2 of a service while everyone else gets v1 — canary testing without a load balancer rule or code change.

mTLS

Istio issues and rotates certificates for every workload and enforces mutual TLS between services automatically. Services authenticate each other’s identity, not just encrypt the connection. A PeerAuthentication policy can enforce strict mTLS across a namespace, ensuring no plaintext traffic is accepted.

Observability

Because all traffic flows through Envoy sidecars, Istio generates L7 metrics (request rate, error rate, latency percentiles), distributed traces, and access logs for every service-to-service call — without instrumentation in the services themselves. This integrates with Prometheus, Grafana, and Jaeger.

Cost

Istio adds latency (two extra proxy hops per call) and resource overhead (a sidecar per pod). For clusters with tens of services, the operational benefit is clear. For small clusters or teams early in a microservices journey, the complexity may outweigh the gains.

Resources

Jaeger

Mon, 01 Jan 2024 00:00:00 +0000

Metrics tell you something is wrong. Logs tell you what happened on one service. Distributed tracing tells you what happened across all the services involved in a single request — where the time went, which service called which, and where a failure or latency spike originated.

Jaeger is an open source distributed tracing system originally from Uber, now a CNCF graduated project. It collects trace data from instrumented services, stores it, and provides a UI for querying and visualising traces. A trace is a tree of spans — each span represents one operation (an HTTP request, a database query, a cache lookup), with start time, duration, and metadata. Jaeger assembles the spans from all services involved in a request into a single trace and lets you follow a request from the frontend through every microservice it touched.

How it works

Services emit spans using the OpenTelemetry SDK (the modern standard) or the legacy Jaeger client libraries. Spans are sent to a Jaeger Collector, stored in a backend (Elasticsearch, Cassandra, or in-memory for development), and queried via the Jaeger UI or API.

The typical deployment in Kubernetes uses the Jaeger Operator:

kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/latest/download/jaeger-operator.yaml

A minimal all-in-one instance (suitable for development):

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
 name: jaeger
spec:
 strategy: allInOne

For production, use the production strategy with a separate Collector, Query, and storage backend.

OpenTelemetry

Jaeger is increasingly used as a backend rather than as the instrumentation library. OpenTelemetry (OTel) is the standard for instrumenting applications — language SDKs, auto-instrumentation agents, and a Collector that receives, processes, and exports telemetry. OTel exports traces to Jaeger (or Tempo, Zipkin, or any OTLP-compatible backend) via the OTLP protocol. The practical consequence: instrument once with OTel, route to whichever backend fits.

In the observability stack

Jaeger covers the tracing pillar alongside Prometheus (metrics) and Loki (logs). Grafana can query Jaeger directly — a trace ID in a log line becomes a clickable link that opens the trace in Grafana’s trace explorer, connecting all three pillars in a single investigation workflow. Grafana Tempo is an alternative tracing backend that integrates more tightly with the Grafana stack, but Jaeger remains the standalone tracing solution with the longest track record.

Resources

Loki

Mon, 01 Jan 2024 00:00:00 +0000

Prometheus tells you that something is wrong and when it started. Loki tells you what happened — it is the log aggregation layer of the observability stack. Logs from every pod across every node are collected, indexed, and made searchable in one place. Grafana is the front end for both.

How it works

Loki stores logs as compressed chunks, indexed only by labels (not by content). This makes it cheap to store and fast to query by label — namespace, pod name, app — but slower for full-text search than something like Elasticsearch. The trade-off is intentional: label-scoped queries cover the vast majority of real operational use, and the storage cost is dramatically lower.

Promtail runs as a DaemonSet on every node, tails log files from /var/log/pods/, attaches Kubernetes labels, and ships to Loki. Grafana queries Loki directly.

Deployment modes

SingleBinary — ingestion, querying, and management all run in a single instance. Simple to deploy, minimal operational overhead. A single point of failure: if it goes down, ingestion stops and logs are lost. The right starting point for most clusters.

SimpleScalable — responsibilities split into separate pods, each running a minimum of two instances for HA. Ingestion, querying, and the compactor can be scaled independently. Significantly more operational overhead, but fault-tolerant and tunable under load. The right move for production once you have volume and reliability requirements.

Getting started

The fastest path to a working stack is deploying Loki alongside kube-prometheus-stack, which brings up Prometheus, Grafana, and Alertmanager together. See the Prometheus note for the kube-prometheus-stack setup and the ArgoCD CRD workaround.

Loki and Promtail are installed as a separate ArgoCD Application, using multiple Helm sources with values pulled from the cluster config repo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: log-ingestion
 namespace: argo-cd
spec:
 project: default
 sources:

 # Loki
 - repoURL: "https://grafana.github.io/helm-charts"
 chart: loki
 targetRevision: 6.55.0
 helm:
 releaseName: loki
 valueFiles:
 - $values/cluster/testing/overlay/monitoring/helm/loki-values.yaml

 # Promtail
 - repoURL: "https://grafana.github.io/helm-charts"
 chart: promtail
 targetRevision: 6.17.1
 helm:
 releaseName: promtail
 valueFiles:
 - $values/cluster/testing/overlay/monitoring/helm/promtail-values.yaml

 # Values source — cluster config repo
 - repoURL: 'git@github.com:example-org/cluster-config.git'
 targetRevision: HEAD
 ref: values

 destination:
 server: https://kubernetes.default.svc
 namespace: monitoring
 syncPolicy:
 automated:
 selfHeal: true
 prune: true
 syncOptions:
 - CreateNamespace=true
 - ServerSideApply=true

Note: targetRevision: HEAD is fine for testing environments. Pin to a tag for staging and production.

Promtail deprecation

Promtail is deprecated as of February 2025 and in LTS — security fixes only, no new features. Expected EOL is end of 2026.

The Grafana-recommended replacement is Grafana Alloy, a more capable collector that handles metrics, logs, and traces in a single agent. The migration path is not yet settled enough for a confident recommendation — worth waiting for clear community consensus before moving. Until then, Promtail continues to work and the LTS window gives time to plan.

Grafana integration

Add Loki as a data source in Grafana and logs become queryable alongside metrics. A useful starting point is a simple app-oriented logs dashboard — filter by namespace and pod, tail in near-real-time, correlate timestamps with Prometheus spikes.

LogQL, Loki’s query language, mirrors PromQL in style:

# All error logs from a namespace
{namespace="production"} |= "error"

# Parse and filter structured logs
{app="my-api"} | json | status >= 500

# Rate of error log lines over time
rate({namespace="production"} |= "error" [5m])

Resources

Loki documentation
Grafana Alloy documentation — future Promtail replacement
loki-stack Helm chart
kube-prometheus-stack

Prometheus

Mon, 01 Jan 2024 00:00:00 +0000

Something is wrong. Pods are restarting, latency is climbing, and a request that usually takes 50ms is now taking 2 seconds. You know something happened — users are complaining — but you have no numbers, no history, and no way to know when it started or which service caused it.

The instinct is to grep the logs. And with one service on one server, that works. But once you have 20 services running across 40 pods, grepping logs to understand system behaviour does not scale. You are looking at individual events trying to infer aggregate trends — the wrong tool for the question. Logs tell you what happened in one place at one moment. Metrics tell you how the system is behaving across all of it, over time.

So you use Prometheus. It scrapes metrics from every pod, node, and cluster component on a regular interval and stores them as time series. Now you have the spike, the exact minute it started, and a number attached to every symptom.

How it works

Prometheus is pull-based: it reaches out to your services and scrapes a /metrics endpoint on a schedule. Services expose metrics in a simple text format; Prometheus stores them and makes them queryable.

# EXPOSE from your app
http://my-service:8080/metrics

# Prometheus scrapes this every 15s and stores:
http_requests_total{method="POST", status="500"} 42
http_request_duration_seconds{quantile="0.99"} 1.847

Most infrastructure components (Kubernetes itself, NGINX, Postgres, Redis, JVM) either expose Prometheus metrics natively or have an exporter that does it for them.

In Kubernetes — kube-prometheus-stack

Running Prometheus in Kubernetes manually is fiddly. The fastest path to a working Prometheus + Grafana + Alertmanager stack is the kube-prometheus-stack Helm chart — it installs the Prometheus Operator, Grafana, Alertmanager, node exporters, and a set of default dashboards and alert rules in one go. Add Loki on top for logs.

CRD workaround

kube-prometheus-stack ships with a large set of CRDs. When managed through ArgoCD, applying CRDs and the chart in the same sync can cause ordering failures. The standard workaround is skipCrds: true in the ArgoCD Application, with CRDs applied via a separate kustomize source in the same Application:

- repoURL: "oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack"
 chart: kube-prometheus-stack
 targetRevision: 82.14.1
 helm:
 releaseName: kube-prometheus-stack
 valueFiles:
 - $values/cluster/development/overlay/monitoring/helm/kube-prometheus-stack.yaml
 skipCrds: true

This keeps the CRDs in Git and lets ArgoCD manage them, while avoiding the race condition on first install.

ServiceMonitor

The Prometheus Operator (installed by the stack) manages scrape config via CRDs. The key one is ServiceMonitor — it tells Prometheus which services to scrape without editing Prometheus config directly:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: my-app
spec:
 selector:
 matchLabels:
 app: my-app
 endpoints:
 - port: metrics
 interval: 30s

Deploy this alongside your app and Prometheus picks it up automatically.

PromQL

Prometheus Query Language lets you slice and aggregate metrics. A few patterns worth knowing:

# Request rate over last 5 minutes
rate(http_requests_total[5m])

# 99th percentile latency
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# Error ratio
rate(http_requests_total{status=~"5.."}[5m])
 /
rate(http_requests_total[5m])

# Memory usage per pod
container_memory_working_set_bytes{namespace="production"}

Alerting

Prometheus evaluates alert rules continuously and fires them to Alertmanager, which handles routing, grouping, and silencing:

- alert: HighErrorRate
 expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
 for: 5m
 labels:
 severity: critical
 annotations:
 summary: "Error rate above 5% for {{ $labels.service }}"

for: 5m means the condition must hold for 5 minutes before firing — avoids noisy alerts on brief spikes.

Metrics Server

Metrics Server is a separate, lightweight component that provides the real-time resource metrics (kubectl top pods, kubectl top nodes) that HPA uses to make scaling decisions. It is not Prometheus — it does not store history and is not queryable. It exists purely to feed the Kubernetes control plane.

Install via ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: metrics-server
 namespace: argo-cd
spec:
 project: default
 sources:
 - repoURL: "https://kubernetes-sigs.github.io/metrics-server"
 chart: metrics-server
 targetRevision: 3.13.0
 helm:
 releaseName: metrics-server
 destination:
 server: https://kubernetes.default.svc
 namespace: kube-system
 syncPolicy:
 automated:
 selfHeal: true
 prune: true
 syncOptions:
 - CreateNamespace=true

Install this early — HPA does nothing without it, and kubectl top is the first thing you reach for when something looks wrong.

What Prometheus is not

Prometheus stores metrics — numbers over time. It does not store logs (see Loki) and it does not trace individual requests across services (see Jaeger). Metrics tell you that something is wrong and when it started. The other two tell you what happened and where.