Cicd on Backend Engineering Strategy Tools

Image Tooling

Thu, 18 Jun 2026 00:00:00 +0000

Versioned, multi-arch Docker images for Kubernetes workflows — built with Dagger, published to Docker Hub, triggered by a version tag.

The motivation is in Shared Tooling Images: one image, consistent versions, three contexts — CI, local, colleagues.

Images

GitHub repo	Docker Hub	Contents
`image-tooling`	`best-tools/tooling-k8s`	kubectl, helm, kustomize, argocd CLI, k9s, jq, yq
`image-tooling`	`best-tools/tooling-k8s-aws`	`tooling-k8s` + AWS CLI
`image-tooling`	`best-tools/tooling-k8s-openstack`	`tooling-k8s` + OpenStack CLI
`image-buildx`	`best-tools/buildx`	CI builder — Docker buildx, AWS CLI, Dagger CLI
`image-pandoc`	`best-tools/pandoc`	PDF generation — pandoc + TeX Live

All images publish as multi-arch manifests: linux/amd64 + linux/arm64.

Quick start

Interactive shell with kubeconfig mounted:

docker run -it --rm \
 -v ~/.kube:/mnt/kube:ro \
 -v $(pwd):/work \
 -w /work \
 docker.io/best-tools/tooling-k8s:latest

The image entry point symlinks /mnt/kube → /root/.kube on startup, so kubectl picks it up immediately.

Shell alias for daily use:

alias k8s='docker run -it --rm \
 -v ~/.kube:/mnt/kube:ro \
 -v $(pwd):/work -w /work \
 docker.io/best-tools/tooling-k8s:latest'

k8s helm lint .
k8s kubectl get pods -n argocd

In CI (GitHub Actions):

- name: Lint chart
 run: docker run --rm -v ${{ github.workspace }}:/work -w /work docker.io/best-tools/tooling-k8s:latest helm lint .

Or reference the image directly as the job container — no install step needed.

Setup (contributors / maintainers)

Credentials are set once as GitHub org-level secrets and inherited by all image-* repos automatically.

Secret	Where to get it
`DOCKERHUB_TOKEN`	hub.docker.com → Account → Security → Access Tokens (Read, Write, Delete)
`DAGGER_CLOUD_TOKEN`	cloud.dagger.io → Organisation → Tokens

Path: github.com/Backend-Engineering-Strategy-Tools → Settings → Secrets and variables → Actions → New organisation secret.

Releasing

git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0

The GitHub Actions workflow triggers on v*.*.* tags, calls dagger call publish-multi-arch, and pushes both best-tools/<image>:v1.0.0 and best-tools/<image>:latest to Docker Hub. Pipeline trace at cloud.dagger.io.

This Site

Thu, 18 Jun 2026 00:00:00 +0000

This started as an effort to clean up and publish existing notes — things accumulated over years of work that were sitting in various raw forms and never properly written up. The act of publishing forces a level of clarity that private notes rarely get: you have to explain the context, fill the gaps, and decide what actually matters.

It doubles as deliberate practice in documentation. Getting into the habit of writing things up properly — not just as a record of what was done, but as something useful to come back to — is a skill that compounds. The site is both the output and the exercise.

Practically it also serves as a personal reference: a place to look things up that I have actually done and know the details of, rather than re-deriving them from scratch.

Stack: Hugo static site using the Stack theme v4 via Go modules. Deployed to GitHub Pages via GitHub Actions.

Repo: Backend-Engineering-Strategy-Tools/site

Why Hugo

Static is the right default for content that does not need dynamic rendering. Hugo is fast — full site builds in milliseconds — and its template model maps well to a Go-centric mindset. The theme ecosystem is good and the Stack theme in particular supports the layout and content model needed here without significant customisation.

Build Pipeline

The GHA workflow runs on push to main:

Fetch private assets — CV and cover letter PDFs are stored in a private release on a separate repo. The workflow fetches them via GitHub API using a scoped BEST_SITE_PAT secret and writes them into static/cv/ before the build.
Hugo build — hugo --minify via the official ghcr.io/gohugoio/hugo Docker image, producing public/.
Deploy — actions/deploy-pages pushes public/ to GitHub Pages.

This keeps CV content separate from site code — the PDF repo can be updated independently without touching the site repo, and a site redeploy picks up the latest version.

Theme Customisation

Stack theme v4 is imported via go.mod. Hugo’s lookup order means local files in layouts/ or assets/ override the theme without forking it.

The main override is layouts/_default/single.html — the site uses layout: single on all content pages to get full-width rendering (no sidebar). All pages also set showReadingTime: false in front matter.

Domain

Currently served at backend-engineering-strategy-tools.github.io/site/. A redirect from best.mjnet.info is set up via an S3 static website bucket — no files in the bucket, just a redirect rule. DNS managed in Route53.

S3 bucket setup (bucket name must match domain):

aws s3api create-bucket \
 --bucket best.mjnet.info \
 --region eu-central-1 \
 --create-bucket-configuration LocationConstraint=eu-central-1

aws s3api put-public-access-block \
 --bucket best.mjnet.info \
 --public-access-block-configuration BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false

aws s3api put-bucket-website \
 --bucket best.mjnet.info \
 --website-configuration '{"IndexDocument":{"Suffix":"index.html"},"RoutingRules":[{"Redirect":{"Protocol":"https","HostName":"backend-engineering-strategy-tools.github.io","ReplaceKeyPrefixWith":"site/"}}]}'

Route53 ALIAS record pointing best.mjnet.info at the S3 website endpoint:

aws route53 change-resource-record-sets \
 --hosted-zone-id <ZONE_ID> \
 --change-batch '{
 "Changes": [{
 "Action": "CREATE",
 "ResourceRecordSet": {
 "Name": "best.mjnet.info",
 "Type": "A",
 "AliasTarget": {
 "HostedZoneId": "Z21DNDUVLTQW6Q",
 "DNSName": "s3-website.eu-central-1.amazonaws.com",
 "EvaluateTargetHealth": false
 }
 }
 }]
 }'

Z21DNDUVLTQW6Q is the fixed AWS hosted zone ID for S3 website endpoints in eu-central-1.

Content Model

Two sections with distinct purposes:

Section	Tone	What goes here
`homelab/`	Narrative, first-person	What actually happened — the sequence, dead ends, workarounds
`public-notes/`	Reference, factual	How things work — distilled, structured, without the personal journey

The separation keeps the narrative and reference writing from muddying each other. A reader looking for a quick reference does not want to read about what went wrong first; a reader following the homelab story wants the context.

projects/ and thinking/ round out the structure — projects are documented outcomes, thinking is longer-form analysis.

What’s Next

The current setup — Hugo on GitHub Pages — is the right foundation for a content-heavy site. A few things are planned or in progress:

Custom domain — best.mjnet.info redirect via S3 + Route53 (see Domain section above). Proper custom domain with HTTPS as a follow-up.
Better build pipeline — replacing the Makefile with a Dagger Go module for local/CI parity and pinned Hugo versions.
More content — the backlog of notes and projects is long. The site grows as things get cleaned up and published.

Longer term, the intent is to grow this into a more multi-faceted site — login-protected sections, dynamic content, interactive tooling. Hugo handles the static part well; the approach is to add separate paths alongside it for authentication and dynamic content rather than replacing the static foundation. The static site stays fast and low-cost, dynamic parts layer in where they are actually needed.

Conftest

Mon, 08 Jun 2026 00:00:00 +0000

Conftest is a CLI tool that runs OPA policies against structured config files — Kubernetes manifests, Terraform plans, Helm output, Dockerfiles, GitHub Actions workflows, anything that can be parsed. It is the CI/CD enforcement layer on top of Rego.

Write a policy once in Rego, run conftest test in your pipeline, fail the build if violations are found.

Install

brew install conftest

Basic usage

# Test a Kubernetes manifest
conftest test deployment.yaml

# Test all YAML in a directory
conftest test k8s/

# Test a Terraform plan (JSON output)
terraform show -json tfplan > tfplan.json
conftest test tfplan.json --parser json

# Test Helm-rendered output
helm template my-app ./chart | conftest test -

By default Conftest looks for policies in ./policy/. Change with --policy.

Policy structure

Policies are Rego files in the policy/ directory. Conftest checks for deny, warn, and violation rules.

# policy/k8s.rego
package main

# deny blocks the pipeline
deny contains msg if {
 input.kind == "Deployment"
 not input.spec.template.spec.securityContext.runAsNonRoot
 msg := sprintf("Deployment %v must set runAsNonRoot", [input.metadata.name])
}

# warn prints a warning but does not fail
warn contains msg if {
 input.kind == "Deployment"
 not input.metadata.labels["app.kubernetes.io/version"]
 msg := sprintf("Deployment %v is missing version label", [input.metadata.name])
}

$ conftest test deployment.yaml
FAIL - deployment.yaml - main - Deployment nginx must set runAsNonRoot
WARN - deployment.yaml - main - Deployment nginx is missing version label

2 tests, 0 passed, 1 warning, 1 failure

Namespaces

Use --namespace to scope which Rego package Conftest evaluates. Useful when you have per-resource-type policies in separate packages.

conftest test deployment.yaml --namespace kubernetes.deployments

Multiple parsers

Conftest supports many input formats. Common ones:

Flag	Parses
`--parser yaml`	YAML (default for `.yaml`/`.yml`)
`--parser json`	JSON, Terraform plan
`--parser hcl2`	Terraform HCL
`--parser dockerfile`	Dockerfiles
`--parser toml`	TOML

Policies can be distributed as OCI artifacts and pulled with conftest pull.

conftest pull ghcr.io/myorg/policies:latest
conftest test deployment.yaml

The Conftest policy hub documents the format. Useful for sharing a common policy library across repos without copy-pasting Rego.

In CI

# GitHub Actions example
- name: Policy check
 run: |
 conftest test k8s/ --policy policy/

Exits non-zero on deny violations. warn violations print but do not fail the build — useful for phased enforcement (warn first, deny later).

Resources

These Are Not the Pipelines You Are Looking For

Thu, 04 Jun 2026 00:00:00 +0000

There are a lot of CI/CD platforms. GitHub Actions, Tekton, Jenkins, Jenkins X, Harness, Bitbucket Pipelines, GitLab CI, CircleCI, Argo Workflows. The choice between them is a perennial argument — and mostly the wrong one.

The mistake is putting your build logic inside the pipeline. Once you do that, the pipeline owns your build. Reproducing a failure locally means setting up the CI environment. Switching platforms means rewriting all the steps. Testing the pipeline means pushing a commit and waiting. The platform becomes load-bearing.

The fix is straightforward: keep all logic in Make or Dagger. The pipeline calls one command. make build. make test. dagger call publish. That’s it.

The pipeline shrinks to thin orchestration: trigger on a git event, check out the code, call the command, report the result. It does not know what the build does. It does not care. You can swap GitHub Actions for Tekton or Tekton for Argo Workflows and the only thing that changes is the YAML wrapper around the same command.

More importantly: you can run the same command on your laptop. No CI environment to replicate, no “it works locally but fails in CI” gap to debug. The build is the build, wherever it runs.

This matters more the longer a project lives. Build tools consolidate and diverge. Platforms get acquired, deprecated, or repriced. Jenkins pipelines from five years ago are maintained by people who weren’t there when they were written and can’t run them outside of Jenkins. The projects that survive these transitions cleanly are the ones where the build logic was never in the pipeline to begin with — it was in a Makefile or a build tool that could run anywhere.

Gradle, Maven, Make — these predate CI/CD platforms by decades and will outlast the current generation of them. Put your logic there. The pipeline is a trigger and a reporter. Keep it that way.

The pattern in practice:

build:
 docker build -t myimage:$(VERSION) .

test:
 go test ./...

publish:
 docker push myimage:$(VERSION)

# GitHub Actions — the entire pipeline
steps:
 - uses: actions/checkout@v4
 - run: make build
 - run: make test
 - run: make publish

# Tekton — same command, different wrapper
steps:
 - name: build-test-publish
 image: golang:1.22
 script: |
 make build
 make test
 make publish

Same logic. One command. Any pipeline.

Make — Task Runner Pattern

Wed, 03 Jun 2026 00:00:00 +0000

Make predates most of the tooling in this notes collection by decades. Originally built to manage C compilation — track which source files changed, recompile only what’s needed. The dependency graph and incremental execution model are genuinely elegant.

In modern use, Make is rarely used for what it was designed for. It is used as a task runner: a consistent interface that wraps whatever the project actually needs to do.

.PHONY: build test lint deploy

build:
	docker build -t myapp .

test:
	docker run --rm myapp pytest

lint:
	docker run --rm myapp ruff check .

deploy:
	helm upgrade --install myapp ./chart

make build, make test, make lint — same interface regardless of what’s underneath. New team member, CI pipeline, colleague’s machine: one place to look.

Why it still works

Installed everywhere. No setup, no version to pin.
Self-documenting entry point — cat Makefile tells you how to work with the project.
Trivially wraps Docker, shell scripts, language-specific tools, cloud CLIs.
Low overhead — it is just shell execution with a thin layer on top.

The fact that it is old is not a weakness. It is stable, understood, and available.

The `.PHONY` problem

Make’s native model assumes targets are files. make build checks if a file named build exists and skips the recipe if it does. .PHONY declares that a target is not a file, forcing it to always run.

Forgetting .PHONY is a classic Make footgun. Always declare task targets phony.

.PHONY: build test lint clean

Modern alternatives

Just (Justfile) — purpose-built task runner, not a build system. Cleaner syntax than Make, no file-dependency model to work around, better error messages. Cross-platform. Requires installation.

Task (Taskfile.yml) — YAML-based, similar goals to Just. More readable for people uncomfortable with Make syntax.

Why still use Make then: zero install cost, ubiquity in CI environments, and for many projects the .PHONY quirk is the only real friction. If the team already knows Make and the project isn’t complex, switching to Just adds a dependency without solving a real problem.

If starting fresh on a project where the team is comfortable with the tooling choice, Just is the cleaner option.

It is there, it works. That is not faint praise — ubiquity and stability are real value. Every project gets a Makefile. New person joins, CI pipeline runs, colleague needs to build something: make is the answer and it requires no explanation.

Pattern: shared interface over varied internals

The real value is not Make specifically — it is the pattern of having one consistent entry point regardless of what’s underneath. Whether that is Make, Just, or a scripts/ directory with a run script, the goal is the same: anyone (person or pipeline) can run the project without knowing its internals.

See also Shared Tooling Images — the same principle applied to the tools themselves.

Where this pattern applies

The signal that logic is leaking into the wrong place: when a tool’s config format starts looking like a scripting language. That is the point to extract it into a script and wrap in Make.

Build systems Gradle compiles, tests, and packages. Deployment logic, Docker builds, release steps → scripts. make build calls Gradle. make docker calls a script. make deploy calls Helm. See Build Systems for the full pattern.

Infrastructure

Terraform: terraform apply does the infra. Environment selection, var file injection, state backend config, plan review → scripts. make plan, make apply.
Helm: Helm packages and deploys. Cluster selection, secret fetching, values overrides → scripts. make deploy.
Ansible: Ansible provisions. Inventory management, vault decryption, pre-flight checks → scripts. make provision.

Testing Test runners run tests. Environment setup, test data seeding, coverage reporting → scripts. make test is the entry point regardless of whether it is pytest, jest, or go test underneath.

Package managers A common npm/yarn trap: stuffing 20 scripts into package.json until it becomes a second build system. Keep package.json for dependency management. Orchestration belongs in Make.

Static site / Hugo Hugo builds the site. PDF fetching, Docker wrapping, deploy steps → Makefile. The pattern this site already uses.

SLSA — Supply-chain Levels for Software Artifacts

Wed, 03 Jun 2026 00:00:00 +0000

SLSA (pronounced “salsa”) is a framework for securing the software supply chain. Developed by Google, now under the OpenSSF. The core question it answers: how do you know the artifact you are deploying is actually what was built from the source you think it was?

The answer is provenance: cryptographically signed metadata that records what built an artifact, from what source, when, and under what conditions. SLSA defines levels of increasing rigour around how that provenance is generated and verified.

Why it matters

The software supply chain is an attack surface. SolarWinds, XZ Utils, Log4Shell — different attack vectors but the same underlying problem: something got into the build or the dependency that should not have been there, and nobody noticed until it was too late.

SLSA does not prevent all supply chain attacks. It makes certain classes of attack verifiably harder and gives consumers of an artifact a way to check that it was built the way it claims to have been.

The levels

SLSA defines three build levels (reorganised from the original four in SLSA v1.0):

L1 — Provenance exists The build produces signed provenance documenting what produced the artifact. Any build system, any format. Protects against accidental tampering and gives a starting point for verification.

L2 — Hosted build, signed provenance Build runs on a hosted CI platform (GitHub Actions, Cloud Build, etc.). Provenance is generated and signed by the build platform, not the developer’s machine. Harder to falsify — an attacker needs to compromise the build platform, not just a developer laptop.

L3 — Hardened build Ephemeral, isolated build environment. Non-falsifiable provenance — the build platform itself signs the provenance in a way that even a compromised build script cannot forge. Parameterless builds where the inputs are fully declared.

Provenance in practice

Provenance is a JSON document (SLSA uses the in-toto Attestation Framework) that records:

The source repository and commit
The build platform and workflow
The artifact digest(s)
The builder identity

It is signed using Sigstore / cosign, which provides keyless signing backed by OIDC identity. No key management required for the common case.

Tooling

SLSA GitHub Generator: reusable GitHub Actions workflows that generate SLSA L3 provenance for common artifact types (Go binaries, container images, Maven/Gradle packages). The easiest path to L3 on GitHub Actions.

cosign: signs and verifies container images and provenance attestations. Part of the Sigstore project.

slsa-verifier: CLI tool to verify SLSA provenance against an artifact. Used by consumers to check that an artifact meets a required SLSA level.

Dependency Track / Grype / Trivy: SBOM and vulnerability scanning tools that complement SLSA (SLSA is about build integrity, SBOM is about knowing what is in the artifact — related but distinct).

SLSA and container images

Container images are a natural fit. The image digest is the artifact, provenance attests to the build that produced it, cosign attaches the attestation to the registry.

# verify an image has SLSA provenance
slsa-verifier verify-image \
 ghcr.io/myorg/myimage@sha256:abc123 \
 --source-uri github.com/myorg/myrepo \
 --source-tag v1.0.0

For the tooling images pattern — images published to a registry that teams depend on — SLSA provenance is worth adding. It gives consumers a verifiable chain from source to image.

Practical adoption

L1 is low friction. Generate provenance as part of the build, attach it to the release. Most CI platforms make this straightforward.

L2 requires a hosted build platform (most teams already use one) and using the platform’s signing rather than rolling your own.

L3 requires workflow changes — ephemeral environments, parameterless builds, using the SLSA GitHub Generator or equivalent. More investment but achievable for a standard GitHub Actions project.

Start at L1. Provenance exists, the chain is documented, the habit is established. L2 and L3 follow naturally as the pipeline matures — you are not making a big architectural decision, you are incrementally tightening what you already have.

In practice: never seen L1, L2, or L3 required explicitly by a client or platform. It is still “good idea in theory” territory for most teams. That said, the underlying practices — signed builds, traceable artifacts, reproducible pipelines — show up as requirements all the time, just not always under the SLSA name.

Which points to what SLSA actually is: a name and a framework for practices that are already good ideas. Automate the build, sign the output, record the provenance. Quality follows from automation — it frees up time to iterate, makes review easier, and ensures consistency without manual discipline. SLSA formalises that and gives you a vocabulary to talk about it with others. The levels are a ladder, not a checklist to complete before you can ship.

Shared Tooling Images: SLSA provenance applies directly to published tooling images
GitHub Actions: where SLSA GitHub Generator workflows run

Dagger

Tue, 02 Jun 2026 00:00:00 +0000

Your build script works on your laptop. It breaks in CI because a tool version differs. It breaks for a colleague because they’re on a different OS. You’re writing apt-get install steps in YAML and maintaining a separate script for local builds.

Dagger solves this the same way Docker solved “works on my machine” for applications: run everything in containers. Every pipeline step executes inside a defined container image — identically on your laptop, in GitHub Actions, or inside a Kubernetes pod. The Dagger Engine is a containerised daemon your pipeline calls into. Where it runs is an operational detail; what it does is defined once in code.

The other thing Dagger gives you that YAML-based CI systems don’t: your pipeline is real code. Write it in Go (or TypeScript or Python), build a library of reusable functions, share it across projects, and test it the same way you’d test any other code.

Module

A Dagger module is a Go package that exposes pipeline functions. Initialise one in your repo:

dagger init --sdk go --name my-pipeline
dagger develop

dagger init creates the scaffolding. dagger develop generates the Go bindings and drops you into a module ready to edit. A minimal function:

package main

import (
 "context"
 "dagger/my-pipeline/internal/dagger"
)

type MyPipeline struct{}

func (m *MyPipeline) Build(ctx context.Context, src *dagger.Directory) *dagger.Container {
 return dag.Container().
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
}

Call it from anywhere — local terminal, CI step, Argo workflow:

dagger call build --src .

Library pattern

Because a module is just Go code, you build it like any other Go library: shared types, helper functions, unit tests. The pattern that works well in practice is a single internal module that codifies your organisation’s conventions — base images, registry credentials, standard lint and test steps — and each project imports it.

A fix to the shared module propagates everywhere. You’re not updating 40 YAML files.

Testing pipeline logic with go test works normally. A test that calls dagger call exercises the real container execution, not a mock. This is the thing YAML pipelines can’t give you: a fast feedback loop on the pipeline logic itself before it hits CI.

Multi-arch builds

Dagger has first-class support for multi-architecture container builds. Pass the platform as an argument:

func (m *MyPipeline) BuildMultiArch(ctx context.Context, src *dagger.Directory) []*dagger.Container {
 platforms := []dagger.Platform{"linux/amd64", "linux/arm64"}
 containers := make([]*dagger.Container, len(platforms))
 for i, platform := range platforms {
 containers[i] = dag.Container(dagger.ContainerOpts{Platform: platform}).
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
 }
 return containers
}

Same function, multiple targets, no extra tooling.

Running in Kubernetes with Argo Workflows

Dagger Engine runs as a container. In Kubernetes, you run it as a sidecar or dedicated pod and point dagger call at it via the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable. Each Argo Workflows step calls dagger call <function> — Argo handles DAG orchestration, retries, and observability; Dagger handles the build execution inside containers.

The split is clean: Argo owns workflow-level concerns, Dagger owns build reproducibility.

Key commands

Command	What it does
`dagger init --sdk go`	Initialise a new module with the Go SDK
`dagger develop`	Generate bindings, enter the development loop
`dagger functions`	List available functions in the current module
`dagger call <function> [args]`	Invoke a pipeline function
`dagger run <command>`	Run an arbitrary command with Dagger Engine available

Dagger Cloud

Dagger Cloud is a hosted observability layer for Dagger pipelines. Free tier available.

Every pipeline run — local or CI — produces a trace: a visualisation of the container graph with per-step timing, cache hit/miss, and log output. The trace is linked from the terminal output and browsable at cloud.dagger.io.

Enable it by setting one environment variable:

export DAGGER_CLOUD_TOKEN=<token>
dagger call build --src .
# → Run URL: https://app.dagger.cloud/runs/<id>

In GitHub Actions, add DAGGER_CLOUD_TOKEN as an organisation-level secret (see GitHub Actions). The dagger/dagger-action picks it up automatically — no workflow change needed beyond the env var being present.

Resources

Shared Tooling Images — One Image, Three Contexts

Tue, 02 Jun 2026 00:00:00 +0000

The problem with per-context tooling: CI uses one version of a linter, a developer’s machine has another, a colleague has a third. Someone upgrades locally, the pipeline fails. Someone pins the pipeline, local runs drift. The maintenance surface is the number of places you install tools multiplied by the number of people on the team.

The fix is one Docker image that travels across all three contexts:

CI/CD — the pipeline pulls the image, runs the same tools
Local development — developers run the image instead of installing tools natively
Colleagues — same image, same versions, no setup docs to go stale

When you need to upgrade a tool, you change one Dockerfile and cut a new tag. Everyone gets it on next pull. No coordination required.

Why Docker?

There are other ways to solve this — dotfiles, Nix, OS package managers, VMs, cloud-init. All valid in different contexts.

The reason Docker makes sense here is that we are already working in a Kubernetes ecosystem where containers are the starting point. Most CI systems support Docker and OCI-compatible images natively. So the question becomes: is it worth introducing a different tool management approach alongside containers, or is it less costly to stay consistent?

Personally, I lean toward staying consistent. Your mileage may vary depending on your stack.

What Goes in the Image

The principle: include anything that needs to be consistent across contexts. Linters, formatters, build tools, CLI utilities.

Not the runtime — that is the application’s own image.

What specifically goes in depends on the target platform. A Kubernetes-only image looks different from one that also includes a cloud provider CLI. Adding tools increases the maintenance burden, so keep each image focused. That is also why there are multiple images — see The Repos below.

Typical candidates:

kubectl, helm, kustomize
Linters and formatters for the languages in use
jq, yq, curl and similar utilities
Cloud provider CLI where needed (aws, openstack)

Versioning

Follow SemVer — image tag equals version. Merge to main triggers a release. No need to complicate it beyond trunk-based development.

Before merging, build and publish an image from the branch so the change can be tested and verified before it lands on mainline. The branch image is short-lived and not promoted to latest until the merge.

Usage Patterns

A shell alias or Makefile target wraps the docker run so nobody has to remember the full invocation:

alias toolbox="docker run -it --rm -v $(pwd):/work -w /work image:latest"

Then toolbox helm lint . or toolbox kubectl apply -f . works the same locally as it does in CI.

Interactive sessions are where this pattern pays off beyond CI. Drop into a persistent terminal with the right context already loaded — kubeconfig mounted in, cloud credentials available, working directory set:

docker run -it --rm \
 -v $(pwd):/work \
 -v ~/.kube:/root/.kube:ro \
 -w /work \
 image:latest bash

You get a shell that looks the same for everyone on the team, regardless of what is installed on their machine.

For interactive use it is worth including a help script that runs on entry — either via ENTRYPOINT or by sourcing it from .bashrc inside the image. It should print what tools are available, their versions, and any common usage examples. Keeps the image self-documenting: a new colleague drops in and immediately knows what they have to work with without reading a README.

# example /usr/local/bin/toolbox-help
echo "=== Toolbox ==="
echo "kubectl $(kubectl version --client -o json | jq -r '.clientVersion.gitVersion')"
echo "helm $(helm version --short)"
echo "aws $(aws --version 2>&1)"
echo ""
echo "Run 'toolbox-help' at any time to see this again."

In CI the step just references the image directly — no installation step, no version pinning in the pipeline config beyond the image tag.

The Repos

Three tooling images, each a superset of the previous, plus two supporting images:

GitHub repo	Docker Hub	Contents
`image-tooling`	`best-tools/tooling-k8s`	kubectl, helm, kustomize, argocd, k9s, jq, yq
`image-tooling`	`best-tools/tooling-k8s-aws`	`tooling-k8s` + AWS CLI
`image-tooling`	`best-tools/tooling-k8s-openstack`	`tooling-k8s` + OpenStack CLI
`image-buildx`	`best-tools/buildx`	CI builder — Docker buildx, AWS CLI, Dagger CLI
`image-pandoc`	`best-tools/pandoc`	PDF generation — pandoc + TeX Live

Repo names use image-<purpose> in the Backend-Engineering-Strategy-Tools org. Docker Hub names drop the prefix and just describe the tool. The three tooling flavours share one image-tooling monorepo so dependency updates and scanning are configured once.

Pipelines are written in Go using Dagger — the same pipeline runs locally and in CI, publishing multi-arch images (amd64 + arm64) triggered by a version tag.

ArgoCD

Mon, 01 Jan 2024 00:00:00 +0000

You deploy with kubectl apply from your laptop. It works. Then a colleague edits a deployment directly on the cluster to fix something urgent. Now what is running no longer matches what is in Git. That is drift, and it is silent — until something breaks in production and nobody can explain why the live state differs from the last known good config.

So you use ArgoCD. Git becomes the single source of truth. Every change flows through a pull request, gets reviewed, and syncs to the cluster automatically. If anyone touches a resource directly, ArgoCD detects the divergence and overrides it back. The cluster converges to Git, always.

This is GitOps: the deployment pipeline is driven by Git state, not by humans running commands.

CI vs CD

A useful mental separation: CI and CD are different concerns and should be handled by different tools.

CI (Continuous Integration) is about code — build, test, produce an artifact (a container image). A pipeline in GitHub Actions, Tekton, or Jenkins owns this. It ends with an image pushed to a registry.

CD (Continuous Delivery) is about cluster state — take that artifact and make sure the right version is running in the right environment. ArgoCD owns this. It watches Git, not the CI pipeline.

Keeping them separate means your deployment logic is not buried inside a CI pipeline that developers need to understand and maintain. ArgoCD runs in the cluster and continuously reconciles state. It is always on.

Applications

ArgoCD manages Applications — a CRD that maps a Git source to a cluster destination:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: my-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/myorg/my-app-config
 targetRevision: main
 path: manifests/
 destination:
 server: https://kubernetes.default.svc
 namespace: my-app
 syncPolicy:
 automated:
 prune: true
 selfHeal: true

prune: true — resources removed from Git are deleted from the cluster. selfHeal: true — any manual change to the cluster is immediately reverted.

App of Apps

Managing dozens of Applications individually gets unwieldy. The App of Apps pattern solves this: one root Application whose source is a directory of other Application manifests. ArgoCD applies the root, which creates all the child Applications, which in turn sync their own workloads. One repo, one sync, everything deployed.

Sync strategies

Strategy	Behaviour
Automated	ArgoCD syncs on every Git change automatically
Manual	Changes are detected and shown as OutOfSync — a human triggers the sync

Automated sync with selfHeal is the purest GitOps posture. Manual sync is useful for production environments where you want a human approval step before changes roll out.

Rollback

Because every state the cluster has ever been in corresponds to a Git commit, rollback is a git revert — or clicking “Sync to previous revision” in the ArgoCD UI. No special tooling, no runbooks, just Git history.

Repo structure

A layout that works well in practice separates ArgoCD’s own installation from the workloads it manages:

cluster/<cluster>/
 cfg/argo-cd/ # ArgoCD install only — CRDs and Helm values
 app-of-apps/ # Root Application, Projects, app definitions
 overlay/<app>/ # Per-cluster Kustomize patches, secret/config overrides

external/ # Reusable base manifests shared across clusters
internal/ # Internal app base manifests

The key separations:

ArgoCD install is isolated in cfg/argo-cd to avoid recursive install loops and make upgrades predictable. ArgoCD is not managing its own installation yet at this point — that comes later.
App-of-Apps lives separately from the install. Once ArgoCD is running, applying app-of-apps/ bootstraps the entire cluster in one step.
Base vs overlay — external/ and internal/ define what an app is. The cluster overlay defines how it runs in this environment. Cluster-specific concerns (resource limits, replica counts, secret refs) stay in the cluster directory and never bleed into the base.

Bootstrapping a cluster

There is a chicken-and-egg problem: ArgoCD manages everything, but something has to install ArgoCD first. The two-step bootstrap solves it cleanly.

Step 1 — Install ArgoCD manually (once):

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd \
 -n argo-cd --create-namespace \
 -f cluster/staging/cfg/argo-cd/values.yaml

Step 2 — Apply the App-of-Apps root:

kubectl apply -k cluster/<cluster>/app-of-apps/

From this point ArgoCD reconciles the entire cluster. Every subsequent change goes through Git — you never run helm install or kubectl apply for workloads again.

Self-management

The final step is making ArgoCD manage its own upgrades. Create an Application that points at cluster/<cluster>/cfg/argo-cd:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: argocd
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/myorg/cluster-config
 targetRevision: main
 path: cluster/staging/cfg/argo-cd
 destination:
 server: https://kubernetes.default.svc
 namespace: argocd
 syncPolicy:
 automated:
 prune: false # be cautious pruning ArgoCD's own resources
 selfHeal: true

Now ArgoCD upgrades itself when you update the Helm values in Git. No more manual helm upgrade — the cluster is fully self-managing. Changes to ArgoCD config go through the same PR review process as everything else.

Resources

CI/CD Platforms

Mon, 01 Jan 2024 00:00:00 +0000

There are many CI/CD platforms and the choice between them matters less than it appears. All of them are thin orchestration wrappers — trigger on a git event, run some steps, report the result. The build logic itself should live in Make or Dagger, not inside the pipeline definition. See One Command, Any Pipeline.

CruiseControl

The early pioneer, released in 2001. CruiseControl introduced continuous integration as a practice — polling a source repository, building on every change, sending email on failure. Configuration was XML, the dashboard was a web page, and it ran on a server you managed yourself. Most of the concepts in modern CI trace back here. Largely historical today but worth knowing as the origin point.

Hudson

Hudson was Sun Microsystems’ take on CI — a Java application with a plugin ecosystem that made it far more extensible than CruiseControl. It gained wide adoption in enterprise Java shops during the late 2000s. When Oracle acquired Sun, the project forked. The community fork became Jenkins; the Oracle-maintained branch kept the Hudson name and eventually faded. Hudson is effectively dead.

Jenkins

The fork that won. Jenkins took Hudson’s plugin architecture and ran with it — today it has over 1800 plugins covering almost every tool in the ecosystem. A Jenkinsfile defines the pipeline as Groovy DSL and lives in the repository alongside the code. Jenkins is the most widely deployed self-hosted CI server and the default answer in many enterprises. The flip side: it is heavyweight, the Groovy DSL has sharp edges, and complex pipelines are difficult to test outside of Jenkins itself.

Jenkins X

Jenkins X is a cloud-native reimagining of Jenkins for Kubernetes. It imposes a strongly opinionated GitOps workflow — pull requests promote through environments, preview environments spin up automatically, everything is driven by Git events. Built on Tekton under the hood. If you want opinionated Kubernetes CI/CD without building the conventions yourself, Jenkins X is one answer. If you want more control over the pipeline structure, raw Tekton gives you the primitives without the opinions.

Tekton

Kubernetes-native CI/CD where pipelines, tasks, and triggers are all Kubernetes CRDs — defined in YAML and applied to a cluster, running as pods. No separate CI server to maintain, no external SaaS dependency. CI runs in the same cluster as your workloads using the same RBAC, secrets, and storage primitives. The core primitives are Task (a sequence of container steps), Pipeline (an ordered set of tasks), PipelineRun (an execution), and Trigger (an event listener that creates runs). The pattern that works well: keep Tasks thin and have them call make <target> — Tekton handles orchestration, Make handles logic.

GitHub Actions

GitHub’s built-in CI/CD, available to any repository on GitHub. Zero infrastructure, zero setup — if your code is on GitHub, Actions is already there. Workflows are YAML files in .github/workflows/, triggered by git events, running on GitHub-managed runners. A large marketplace of pre-built actions covers most common tasks. The zero-friction default for open source projects and small teams. See the GitHub Actions note for more detail.

Argo Workflows

A Kubernetes-native workflow engine from the Argo project. Where Tekton models CI primitives (Tasks, Pipelines), Argo Workflows is a general-purpose DAG executor — it can run any containerised workload as a directed acyclic graph of steps, with fan-out, fan-in, conditionals, and retry logic. Widely used as the execution layer under other tools (including Dagger on Kubernetes). Pairs well with ArgoCD for a fully Argo-based GitOps stack. See the Argo note for coverage of the full Argo ecosystem including Rollouts, Events, and Kargo.

Bitbucket Pipelines

Bitbucket’s built-in CI/CD, integrated directly with Atlassian’s hosting. If your code is already in Bitbucket, Pipelines is the zero-infrastructure option — the same position GitHub Actions occupies for GitHub users. Workflows are YAML, steps run in Docker containers, and Atlassian handles the runner infrastructure. Tightly integrated with Jira for deployment tracking. The right choice when you’re already in the Atlassian ecosystem and don’t want to introduce a separate CI tool.

Harness

A commercial platform with a broader scope than most CI/CD tools — it covers CI, CD, feature flags, cloud cost management, and security testing under one roof. Enterprise-focused, with AI-assisted pipeline generation and strong support for policy and governance across large engineering organisations. Harness is the answer when the organisation needs a managed platform with SLAs, support, and audit trails rather than self-hosted infrastructure. Pricing reflects that positioning.

Resources

GitHub Actions

Mon, 01 Jan 2024 00:00:00 +0000

For a small project or proof of concept, the cost of building a CI environment often exceeds the cost of the project itself. Spinning up a Tekton cluster, installing Argo Workflows, or maintaining a Jenkins server is real infrastructure — with real setup time, real maintenance overhead, and real dependencies to keep running.

GitHub Actions is already there. If your code is on GitHub, you have CI. No extra infrastructure, no additional accounts, a marketplace of pre-built integrations for almost everything you need. For open source projects and small teams it is the pragmatic default: free, integrated, and good enough for the standard build → test → publish pipeline.

Workflow anatomy

Workflows live in .github/workflows/ as YAML files. A workflow has triggers, jobs, and steps:

name: Build and publish

on:
 push:
 branches: [main]
 pull_request:
 branches: [main]

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4

 - name: Build
 run: make build

 - name: Test
 run: make test

 - name: Push image
 if: github.ref == 'refs/heads/main'
 run: |
 echo "${{ secrets.DOCKER_HUB_TOKEN }}" | docker login -u ${{ secrets.DOCKER_HUB_USER }} --password-stdin
 docker push myorg/myimage:latest

on: defines what triggers the workflow. runs-on: selects the runner. uses: pulls a pre-built action from the marketplace. run: executes shell commands directly.

The if: condition on the push step is a common pattern: run tests on every pull request, but only publish on merge to main.

Secrets

GitHub Secrets store credentials without putting them in the repository. Add them under Settings → Secrets and variables → Actions, then reference them in workflows as ${{ secrets.MY_SECRET }}.

For pushing to external services — Docker Hub, a package registry, a cloud provider — this is the integration point. The workflow authenticates using the secret, the secret itself never appears in logs or code.

Organisation secrets

Secrets can be set at the organisation level and inherited by all repositories — no per-repo configuration needed. Useful for shared CI credentials reused across many repos.

Path: github.com/<org> → Settings → Secrets and variables → Actions → New organisation secret

Set Repository access to “All repositories” or a selected subset once you know which repos need it.

A worked example — shared image publishing credentials:

DOCKERHUB_TOKEN # hub.docker.com → Account → Security → Access Tokens
DAGGER_CLOUD_TOKEN # cloud.dagger.io → Organisation → Tokens

Set once at org level; every image-* repo inherits them. Workflows reference them identically to repo secrets: ${{ secrets.DOCKERHUB_TOKEN }}. See Dagger for DAGGER_CLOUD_TOKEN context.

Environments

Environments add a protection layer on top of secrets. Define named environments (e.g. staging, production) and configure:

Required reviewers — a human must approve before the job runs
Environment secrets — secrets scoped to that environment only, not available to other jobs
Wait timer — a mandatory delay before deployment proceeds

A deployment job targets an environment by name:

jobs:
 deploy:
 runs-on: ubuntu-latest
 environment: production
 steps:
 - name: Deploy
 run: ./deploy.sh
 env:
 API_KEY: ${{ secrets.PROD_API_KEY }}

The job pauses for approval before running. Useful for open source projects where the pipeline is public but deployment credentials are not.

Marketplace actions

Most common tasks have a pre-built action. A few worth knowing:

Action	What it does
`actions/checkout`	Clone the repository
`actions/setup-go`	Install a specific Go version
`docker/setup-buildx-action`	Enable multi-arch Docker builds
`docker/build-push-action`	Build and push a container image
`helm/kind-action`	Spin up a local Kubernetes cluster for tests

Using marketplace actions trades control for speed. For a POC or small project that’s the right trade. For anything critical, pin the action to a specific commit SHA rather than a tag — tags are mutable.

Self-hosted runners

GitHub-hosted runners (ubuntu-latest, windows-latest) cover most cases. Self-hosted runners make sense when you need access to an internal network, specific hardware (a GPU, specialised build machine), or want to avoid per-minute billing at scale.

At that point you are closer to running a full CI platform, which is where tools like Tekton or Argo Workflows become relevant — they give you the same kind of pipeline orchestration but running entirely on your own infrastructure.

Cicd on Backend Engineering Strategy Tools

Image Tooling

Images

Quick start

Setup (contributors / maintainers)

Releasing

Links

This Site

Why Hugo

Build Pipeline

Theme Customisation

Domain

Content Model

What’s Next

Conftest

Install

Basic usage

Policy structure

Namespaces

Multiple parsers

Sharing policies

In CI

Resources

These Are Not the Pipelines You Are Looking For

Make — Task Runner Pattern

Why it still works

The .PHONY problem

Modern alternatives

Pattern: shared interface over varied internals

Where this pattern applies

SLSA — Supply-chain Levels for Software Artifacts

Why it matters

The levels

Provenance in practice

Tooling

SLSA and container images

Practical adoption

Related

Dagger

Module

Library pattern

Multi-arch builds

Running in Kubernetes with Argo Workflows

Key commands

Dagger Cloud

Resources

Shared Tooling Images — One Image, Three Contexts

Why Docker?

What Goes in the Image

Versioning

Usage Patterns

The Repos

ArgoCD

CI vs CD

Applications

App of Apps

Sync strategies

Rollback

Repo structure

Bootstrapping a cluster

Self-management

Resources

CI/CD Platforms

CruiseControl

Hudson

Jenkins

Jenkins X

Tekton

GitHub Actions

Argo Workflows

Bitbucket Pipelines

Harness

Resources

GitHub Actions

Workflow anatomy

Secrets

Organisation secrets

Environments

Marketplace actions

Self-hosted runners

The `.PHONY` problem