Containers on Backend Engineering Strategy Tools

Dagger

Tue, 02 Jun 2026 00:00:00 +0000

Your build script works on your laptop. It breaks in CI because a tool version differs. It breaks for a colleague because they’re on a different OS. You’re writing apt-get install steps in YAML and maintaining a separate script for local builds.

Dagger solves this the same way Docker solved “works on my machine” for applications: run everything in containers. Every pipeline step executes inside a defined container image — identically on your laptop, in GitHub Actions, or inside a Kubernetes pod. The Dagger Engine is a containerised daemon your pipeline calls into. Where it runs is an operational detail; what it does is defined once in code.

The other thing Dagger gives you that YAML-based CI systems don’t: your pipeline is real code. Write it in Go (or TypeScript or Python), build a library of reusable functions, share it across projects, and test it the same way you’d test any other code.

Module

A Dagger module is a Go package that exposes pipeline functions. Initialise one in your repo:

dagger init --sdk go --name my-pipeline
dagger develop

dagger init creates the scaffolding. dagger develop generates the Go bindings and drops you into a module ready to edit. A minimal function:

package main

import (
 "context"
 "dagger/my-pipeline/internal/dagger"
)

type MyPipeline struct{}

func (m *MyPipeline) Build(ctx context.Context, src *dagger.Directory) *dagger.Container {
 return dag.Container().
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
}

Call it from anywhere — local terminal, CI step, Argo workflow:

dagger call build --src .

Library pattern

Because a module is just Go code, you build it like any other Go library: shared types, helper functions, unit tests. The pattern that works well in practice is a single internal module that codifies your organisation’s conventions — base images, registry credentials, standard lint and test steps — and each project imports it.

A fix to the shared module propagates everywhere. You’re not updating 40 YAML files.

Testing pipeline logic with go test works normally. A test that calls dagger call exercises the real container execution, not a mock. This is the thing YAML pipelines can’t give you: a fast feedback loop on the pipeline logic itself before it hits CI.

Multi-arch builds

Dagger has first-class support for multi-architecture container builds. Pass the platform as an argument:

func (m *MyPipeline) BuildMultiArch(ctx context.Context, src *dagger.Directory) []*dagger.Container {
 platforms := []dagger.Platform{"linux/amd64", "linux/arm64"}
 containers := make([]*dagger.Container, len(platforms))
 for i, platform := range platforms {
 containers[i] = dag.Container(dagger.ContainerOpts{Platform: platform}).
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
 }
 return containers
}

Same function, multiple targets, no extra tooling.

Running in Kubernetes with Argo Workflows

Dagger Engine runs as a container. In Kubernetes, you run it as a sidecar or dedicated pod and point dagger call at it via the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable. Each Argo Workflows step calls dagger call <function> — Argo handles DAG orchestration, retries, and observability; Dagger handles the build execution inside containers.

The split is clean: Argo owns workflow-level concerns, Dagger owns build reproducibility.

Key commands

Command	What it does
`dagger init --sdk go`	Initialise a new module with the Go SDK
`dagger develop`	Generate bindings, enter the development loop
`dagger functions`	List available functions in the current module
`dagger call <function> [args]`	Invoke a pipeline function
`dagger run <command>`	Run an arbitrary command with Dagger Engine available

Dagger Cloud

Dagger Cloud is a hosted observability layer for Dagger pipelines. Free tier available.

Every pipeline run — local or CI — produces a trace: a visualisation of the container graph with per-step timing, cache hit/miss, and log output. The trace is linked from the terminal output and browsable at cloud.dagger.io.

Enable it by setting one environment variable:

export DAGGER_CLOUD_TOKEN=<token>
dagger call build --src .
# → Run URL: https://app.dagger.cloud/runs/<id>

In GitHub Actions, add DAGGER_CLOUD_TOKEN as an organisation-level secret (see GitHub Actions). The dagger/dagger-action picks it up automatically — no workflow change needed beyond the env var being present.

Resources

Shared Tooling Images — One Image, Three Contexts

Tue, 02 Jun 2026 00:00:00 +0000

The problem with per-context tooling: CI uses one version of a linter, a developer’s machine has another, a colleague has a third. Someone upgrades locally, the pipeline fails. Someone pins the pipeline, local runs drift. The maintenance surface is the number of places you install tools multiplied by the number of people on the team.

The fix is one Docker image that travels across all three contexts:

CI/CD — the pipeline pulls the image, runs the same tools
Local development — developers run the image instead of installing tools natively
Colleagues — same image, same versions, no setup docs to go stale

When you need to upgrade a tool, you change one Dockerfile and cut a new tag. Everyone gets it on next pull. No coordination required.

Why Docker?

There are other ways to solve this — dotfiles, Nix, OS package managers, VMs, cloud-init. All valid in different contexts.

The reason Docker makes sense here is that we are already working in a Kubernetes ecosystem where containers are the starting point. Most CI systems support Docker and OCI-compatible images natively. So the question becomes: is it worth introducing a different tool management approach alongside containers, or is it less costly to stay consistent?

Personally, I lean toward staying consistent. Your mileage may vary depending on your stack.

What Goes in the Image

The principle: include anything that needs to be consistent across contexts. Linters, formatters, build tools, CLI utilities.

Not the runtime — that is the application’s own image.

What specifically goes in depends on the target platform. A Kubernetes-only image looks different from one that also includes a cloud provider CLI. Adding tools increases the maintenance burden, so keep each image focused. That is also why there are multiple images — see The Repos below.

Typical candidates:

kubectl, helm, kustomize
Linters and formatters for the languages in use
jq, yq, curl and similar utilities
Cloud provider CLI where needed (aws, openstack)

Versioning

Follow SemVer — image tag equals version. Merge to main triggers a release. No need to complicate it beyond trunk-based development.

Before merging, build and publish an image from the branch so the change can be tested and verified before it lands on mainline. The branch image is short-lived and not promoted to latest until the merge.

Usage Patterns

A shell alias or Makefile target wraps the docker run so nobody has to remember the full invocation:

alias toolbox="docker run -it --rm -v $(pwd):/work -w /work image:latest"

Then toolbox helm lint . or toolbox kubectl apply -f . works the same locally as it does in CI.

Interactive sessions are where this pattern pays off beyond CI. Drop into a persistent terminal with the right context already loaded — kubeconfig mounted in, cloud credentials available, working directory set:

docker run -it --rm \
 -v $(pwd):/work \
 -v ~/.kube:/root/.kube:ro \
 -w /work \
 image:latest bash

You get a shell that looks the same for everyone on the team, regardless of what is installed on their machine.

For interactive use it is worth including a help script that runs on entry — either via ENTRYPOINT or by sourcing it from .bashrc inside the image. It should print what tools are available, their versions, and any common usage examples. Keeps the image self-documenting: a new colleague drops in and immediately knows what they have to work with without reading a README.

# example /usr/local/bin/toolbox-help
echo "=== Toolbox ==="
echo "kubectl $(kubectl version --client -o json | jq -r '.clientVersion.gitVersion')"
echo "helm $(helm version --short)"
echo "aws $(aws --version 2>&1)"
echo ""
echo "Run 'toolbox-help' at any time to see this again."

In CI the step just references the image directly — no installation step, no version pinning in the pipeline config beyond the image tag.

The Repos

Three tooling images, each a superset of the previous, plus two supporting images:

GitHub repo	Docker Hub	Contents
`image-tooling`	`best-tools/tooling-k8s`	kubectl, helm, kustomize, argocd, k9s, jq, yq
`image-tooling`	`best-tools/tooling-k8s-aws`	`tooling-k8s` + AWS CLI
`image-tooling`	`best-tools/tooling-k8s-openstack`	`tooling-k8s` + OpenStack CLI
`image-buildx`	`best-tools/buildx`	CI builder — Docker buildx, AWS CLI, Dagger CLI
`image-pandoc`	`best-tools/pandoc`	PDF generation — pandoc + TeX Live

Repo names use image-<purpose> in the Backend-Engineering-Strategy-Tools org. Docker Hub names drop the prefix and just describe the tool. The three tooling flavours share one image-tooling monorepo so dependency updates and scanning are configured once.

Pipelines are written in Go using Dagger — the same pipeline runs locally and in CI, publishing multi-arch images (amd64 + arm64) triggered by a version tag.

Docker & OCI

Mon, 01 Jan 2024 00:00:00 +0000

Docker packages applications and their dependencies into portable, reproducible units called containers. Unlike virtual machines, containers share the host kernel — they’re isolated processes, not emulated hardware. This makes them fast to start, light on resources, and consistent across environments: the same image runs on a developer’s laptop, in CI, and in production.

Docker popularised containers, but the underlying standard is now open. The OCI (Open Container Initiative) defines three specifications:

Image spec — the format of a container image: layers, config, manifest
Runtime spec — how a container is run: namespaces, cgroups, lifecycle
Distribution spec — how images are pushed and pulled from registries

Any tool that produces an OCI image can run on any OCI-compliant runtime. Docker is one implementation. It is still the most natural entry point and the docker CLI remains the most familiar interface, but it is worth knowing that the ecosystem is broader than Docker Inc.

OCI images and containers

An image is a read-only, layered filesystem snapshot built from a Dockerfile — each layer is a diff on top of the previous one. A container is a running instance of an image — an isolated process with its own filesystem, network interface, and process space, sharing the host kernel.

docker build -t myapp:1.0 . # build OCI image from Dockerfile
docker run -p 8080:8080 myapp:1.0 # start container
docker ps # list running containers
docker exec -it <id> bash # shell into a running container

Images are stored in registries — Docker Hub, GitHub Container Registry, ECR, Nexus. All speak the OCI distribution spec, so images built with any tool push and pull the same way.

Dockerfile

The Dockerfile defines how an image is built — each instruction adds a layer:

FROM golang:1.22-alpine AS build
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o /app/server .

FROM alpine:3.19
COPY --from=build /app/server /server
EXPOSE 8080
ENTRYPOINT ["/server"]

Multi-stage builds keep the final image lean: the first stage compiles using the full toolchain, the second copies only the binary. No compiler, no source, no build cache in the image you ship.

Order matters for layer caching — put things that change rarely (dependency downloads) before things that change often (source code). A cache miss invalidates all subsequent layers.

Volumes and bind mounts

Containers have ephemeral filesystems — anything written inside is lost when the container stops. Persist data with volumes:

docker volume create pgdata
docker run -v pgdata:/var/lib/postgresql/data postgres:16

For local development, bind mounts map a host directory into the container:

docker run -v $(pwd):/app -w /app node:20 npm test

Networking

Containers on the same Docker network can reach each other by name. Docker Compose creates a default network automatically; named networks can be created explicitly:

docker network create backend
docker run --network backend --name db postgres:16
docker run --network backend myapp # can reach 'db' by hostname

Podman

Podman is a drop-in Docker replacement that runs without a daemon and without root. The CLI is intentionally compatible:

alias docker=podman # usually just works

Rootless containers mean a compromised container process cannot escalate to host root. Daemonless means no long-running background service with broad system access. On RHEL and Fedora, Podman is the default. For CI environments and security-conscious setups it is the better choice.

Podman also supports pods — groups of containers sharing a network namespace, mirroring the Kubernetes pod model. Useful for local development that needs to mirror how things will run in the cluster.

Buildah

Buildah builds OCI images without a Docker daemon. It can build from a Dockerfile or construct images programmatically using shell commands — useful in CI pipelines where running a privileged Docker daemon is undesirable:

buildah bud -t myapp:1.0 . # build from Dockerfile
buildah push myapp:1.0 registry/myapp:1.0

Buildah and Podman share the same underlying storage, so images built with Buildah are immediately available to Podman.

Docker Compose

Compose manages multi-container applications defined in compose.yml:

services:
 app:
 build: .
 ports:
 - "8080:8080"
 environment:
 DATABASE_URL: postgres://app:secret@db/appdb
 depends_on:
 - db

 db:
 image: postgres:16
 volumes:
 - pgdata:/var/lib/postgresql/data
 environment:
 POSTGRES_PASSWORD: secret

volumes:
 pgdata:

docker compose up -d # start in background
docker compose logs -f # stream logs
docker compose down # stop and remove containers

Compose is useful for local development environments. It is a shame it exists as a separate abstraction — it taught people to think in multi-container terms without teaching them Kubernetes, and then left them with a gap to cross when they needed to go to production. That said, it is practical for what it does and is not going away.

For production orchestration, see Kubernetes.

Skopeo

Skopeo works with OCI images directly — copy, inspect, and convert — without pulling them to local storage. Useful in pipelines and for auditing registries:

# Inspect an image without pulling it
skopeo inspect docker://registry.example.com/myapp:1.0

# Copy between registries without touching local disk
skopeo copy docker://source-registry/myapp:1.0 docker://dest-registry/myapp:1.0

# Copy to a local OCI layout
skopeo copy docker://myapp:1.0 oci:myapp-local:1.0

skopeo inspect is particularly useful for checking image metadata, digest, and labels in CI before deciding whether to promote an image.

ORAS

ORAS (OCI Registry As Storage) pushes and pulls arbitrary artifacts to OCI registries — not just container images. Helm charts, SBOMs, attestations, Terraform modules, binary releases — anything can be stored in a registry that speaks OCI distribution spec:

# Push a file as an OCI artifact
oras push registry.example.com/myapp-sbom:1.0 sbom.json:application/spdx+json

# Pull it back
oras pull registry.example.com/myapp-sbom:1.0

This matters because it means a single registry can become the distribution mechanism for the entire software supply chain — image, SBOM, signature, attestation — all with the same access controls and audit trail.

Useful practices

Use specific image tags (postgres:16.2, not postgres:latest) — latest changes under you
Reference images by digest in production (myapp@sha256:abc123) — tags are mutable, digests are not
Run as a non-root user: USER appuser in the Dockerfile
Add a .dockerignore to exclude .git, node_modules, build artefacts from the build context
Keep images small — large images are slow to push, pull, and scan

Resources

GitHub Actions

Mon, 01 Jan 2024 00:00:00 +0000

For a small project or proof of concept, the cost of building a CI environment often exceeds the cost of the project itself. Spinning up a Tekton cluster, installing Argo Workflows, or maintaining a Jenkins server is real infrastructure — with real setup time, real maintenance overhead, and real dependencies to keep running.

GitHub Actions is already there. If your code is on GitHub, you have CI. No extra infrastructure, no additional accounts, a marketplace of pre-built integrations for almost everything you need. For open source projects and small teams it is the pragmatic default: free, integrated, and good enough for the standard build → test → publish pipeline.

Workflow anatomy

Workflows live in .github/workflows/ as YAML files. A workflow has triggers, jobs, and steps:

name: Build and publish

on:
 push:
 branches: [main]
 pull_request:
 branches: [main]

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4

 - name: Build
 run: make build

 - name: Test
 run: make test

 - name: Push image
 if: github.ref == 'refs/heads/main'
 run: |
 echo "${{ secrets.DOCKER_HUB_TOKEN }}" | docker login -u ${{ secrets.DOCKER_HUB_USER }} --password-stdin
 docker push myorg/myimage:latest

on: defines what triggers the workflow. runs-on: selects the runner. uses: pulls a pre-built action from the marketplace. run: executes shell commands directly.

The if: condition on the push step is a common pattern: run tests on every pull request, but only publish on merge to main.

Secrets

GitHub Secrets store credentials without putting them in the repository. Add them under Settings → Secrets and variables → Actions, then reference them in workflows as ${{ secrets.MY_SECRET }}.

For pushing to external services — Docker Hub, a package registry, a cloud provider — this is the integration point. The workflow authenticates using the secret, the secret itself never appears in logs or code.

Organisation secrets

Secrets can be set at the organisation level and inherited by all repositories — no per-repo configuration needed. Useful for shared CI credentials reused across many repos.

Path: github.com/<org> → Settings → Secrets and variables → Actions → New organisation secret

Set Repository access to “All repositories” or a selected subset once you know which repos need it.

A worked example — shared image publishing credentials:

DOCKERHUB_TOKEN # hub.docker.com → Account → Security → Access Tokens
DAGGER_CLOUD_TOKEN # cloud.dagger.io → Organisation → Tokens

Set once at org level; every image-* repo inherits them. Workflows reference them identically to repo secrets: ${{ secrets.DOCKERHUB_TOKEN }}. See Dagger for DAGGER_CLOUD_TOKEN context.

Environments

Environments add a protection layer on top of secrets. Define named environments (e.g. staging, production) and configure:

Required reviewers — a human must approve before the job runs
Environment secrets — secrets scoped to that environment only, not available to other jobs
Wait timer — a mandatory delay before deployment proceeds

A deployment job targets an environment by name:

jobs:
 deploy:
 runs-on: ubuntu-latest
 environment: production
 steps:
 - name: Deploy
 run: ./deploy.sh
 env:
 API_KEY: ${{ secrets.PROD_API_KEY }}

The job pauses for approval before running. Useful for open source projects where the pipeline is public but deployment credentials are not.

Marketplace actions

Most common tasks have a pre-built action. A few worth knowing:

Action	What it does
`actions/checkout`	Clone the repository
`actions/setup-go`	Install a specific Go version
`docker/setup-buildx-action`	Enable multi-arch Docker builds
`docker/build-push-action`	Build and push a container image
`helm/kind-action`	Spin up a local Kubernetes cluster for tests

Using marketplace actions trades control for speed. For a POC or small project that’s the right trade. For anything critical, pin the action to a specific commit SHA rather than a tag — tags are mutable.

Self-hosted runners

GitHub-hosted runners (ubuntu-latest, windows-latest) cover most cases. Self-hosted runners make sense when you need access to an internal network, specific hardware (a GPU, specialised build machine), or want to avoid per-minute billing at scale.

At that point you are closer to running a full CI platform, which is where tools like Tekton or Argo Workflows become relevant — they give you the same kind of pipeline orchestration but running entirely on your own infrastructure.

Resources

Kubernetes

Mon, 01 Jan 2024 00:00:00 +0000

Kubernetes (K8s) is the de facto standard for container orchestration and the second largest open source project after the Linux kernel. It has well and truly reached the plateau of productivity — the ecosystem is mature and it genuinely delivers.

That said, the honest take: K8s is ridiculously hard to deploy and manage (day 2 operations especially). Docker Swarm is equally ridiculously easy to get started with. For raw scale, Mesos/DC/OS wins — clusters of 80k+ nodes have been documented in the wild, versus K8s master’s practical ceiling of around 5k nodes.

So the real question is whether the ecosystem justifies the complexity for your situation. For most teams doing cloud-native work, it does.

Core concepts

The main building blocks:

Pods — smallest deployable unit, wrapping one or more containers that share network and storage.

Deployments — declare desired state; K8s handles rolling updates and self-healing.

Secrets — store sensitive data (passwords, tokens, keys) separately from application config.

DaemonSets — run a pod on every node. Typical use: log collectors, monitoring agents.

ReplicaSets — ensure N copies of a pod are running at any given time.

Ingress — HTTP/S routing rules at layer 7. Your load balancer config, declarative.

CronJobs — scheduled jobs, K8s-native.

Custom Resource Definitions (CRDs) — extend the K8s API with your own resource types. The foundation of most K8s operators.

Architecture

How the pieces fit together internally:

Containers vs virtual machines

Not an either/or — they solve different problems and are frequently combined.

Local clusters for development

When you need K8s without a full cluster:

Tool	Best for
MicroK8s	Ubuntu, snap-based, batteries included
Minikube	The classic, broad driver support
Kind	K8s in Docker, great for CI pipelines
K3D	K3s in Docker, fast startup
K3S	Lightweight K8s, edge and IoT use cases

Resources

kubernetes.io
CNCF Landscape — map of the cloud-native ecosystem
TGI Kubernetes intro (YouTube)
Setting up MicroK8s with RBAC and Storage