CI/CD on Backend Engineering Strategy Tools

Build Systems — Ant, Maven, Gradle, Bazel

Wed, 03 Jun 2026 00:00:00 +0000

The Java ecosystem has cycled through several generations of build tooling. Each generation solved real problems with the previous one and introduced new ones of its own.

Ant

XML-based, imperative. You describe exactly what to do and in what order — compile these files, copy to this directory, package this jar. No conventions, no opinions.

Strengths: total control, predictable, easy to understand what any given build does.

Weaknesses: verbose, no built-in dependency management (Ivy was a separate add-on), and every project reinvents the same targets from scratch. Large Ant builds become hard to maintain.

Still found in older enterprise Java projects. Worth knowing to read and modify, less worth starting from scratch.

Used it. It was what it was — at least you always knew exactly what the build was doing.

Maven

Convention over configuration. If your project follows the standard layout (src/main/java, src/test/java, etc.) most of the build is declared, not scripted. Dependency management built in via the POM and central repository.

Maven 1: the original. Repository model and POM concept introduced. Plugin system was limited and the build lifecycle was rigid.

Maven 2: major redesign. The lifecycle phases (validate → compile → test → package → install → deploy) that most people know. Dependency resolution significantly improved. This is the version that won widespread adoption.

Maven 3: incremental improvements over Maven 2. Better performance, improved parallel builds, polyglot POM support. Most Maven projects today run on 3.x.

Strengths: standardised project layout means any Maven project is immediately navigable; dependency management and central repository model that the whole ecosystem built on.

Weaknesses: XML is verbose for expressing logic; the lifecycle is powerful but opaque when things go wrong; plugin configuration gets unwieldy; multi-module builds are better than Ant but still awkward at scale.

Not a fan. All three weaknesses compound each other: the XML is painful to write, the lifecycle hides what is actually executing and where, and parent POM inheritance chains in multi-module projects make it genuinely hard to answer “what does this build actually do?” You end up cargo-culting POM snippets from Stack Overflow and hoping the lifecycle does what you think it does.

Gradle

Groovy (and later Kotlin) DSL instead of XML. Keeps Maven’s dependency management and repository model, drops the rigid lifecycle in favour of a task graph. Incremental builds — only re-runs tasks whose inputs changed.

Became the default for Android builds and has significant adoption in JVM projects that outgrew Maven.

Strengths: expressive build scripts; incremental and cached builds make large projects faster; Kotlin DSL gives type safety and IDE support; flexible enough to model non-standard builds.

Weaknesses: the flexibility is also the trap — Gradle builds vary wildly and can be hard to read; the Groovy DSL is easy to write badly; build times on cold caches can be slow; debugging task dependencies is non-trivial.

Gradle vs Maven: Maven wins on standardisation and predictability; Gradle wins on performance and flexibility. For a standard Java/Kotlin service with no unusual requirements, Maven is often the lower-maintenance choice. For Android, large monorepos, or complex multi-language builds, Gradle.

Gradle solves the XML problem but introduces a different one: the flexibility makes it very easy to get wrong. Every team ends up with a slightly different Gradle build, and reading someone else’s is often just as opaque as Maven’s lifecycle — just for different reasons. The problem of “what is this build actually doing” doesn’t go away, it just changes shape.

The discipline that keeps Gradle manageable: don’t stuff everything into the Gradle build. Gradle should compile, test, and package — that’s it. Deployment logic, environment setup, Docker builds, release steps — those belong in scripts or other tools that are good at those things. Then wrap the whole thing in a Makefile that gives a single consistent entry point. make build calls Gradle. make docker calls a shell script. make deploy calls Helm. Gradle stays focused and readable; the Makefile is the seam that holds it together.

SBT

The Scala Build Tool. Reactive, incremental compilation at the core. Not just a build tool — the REPL and interactive session are part of the workflow.

Heavy dependency on Scala itself; the build definition is Scala code. Powerful but a significant learning curve for anyone not already in the Scala ecosystem.

Notes to follow.

Bazel & Bazelisk

Originally Google’s internal build system (Blaze), open-sourced as Bazel. Hermetic builds — each action declares its inputs and outputs explicitly, no implicit filesystem access. Correctness and reproducibility guaranteed by construction.

Language-agnostic: rules exist for Java, Go, Python, C++, and others. Built for monorepos at scale — incremental and distributed builds, remote caching, remote execution.

Bazelisk: version manager for Bazel. Pin the Bazel version in .bazelversion, run bazelisk instead of bazel — it downloads and uses the pinned version automatically. Essential for team consistency.

Strengths: genuinely reproducible builds; scales to very large codebases; remote caching makes CI fast once warm.

Weaknesses: steep learning curve; rules ecosystem outside Google’s own languages is less mature; significant setup cost; overkill for anything smaller than a large monorepo.

Opinions to follow.

Choosing

	Small project	Standard service	Large monorepo	Polyglot
Ant	Possible	Legacy only	No	No
Maven	Fine	Good default	Struggles	No
Gradle	Fine	Good	Better	Partial
Bazel	Overkill	Overkill	Strong	Strong

Both Maven and Gradle have the same core problem: you end up not really knowing what your build is doing. Ant at least was honest about it. If forced to choose, Gradle is the least bad option — the DSL flexibility is a trap but it is a trap you can avoid with discipline, whereas Maven’s lifecycle opacity and XML are just the deal regardless. For anything in the JVM ecosystem today the choice is usually made for you by the project or the framework — if you get to choose, Gradle with a simple build file and keep it that way.

Make — Task Runner Pattern

Wed, 03 Jun 2026 00:00:00 +0000

Make predates most of the tooling in this notes collection by decades. Originally built to manage C compilation — track which source files changed, recompile only what’s needed. The dependency graph and incremental execution model are genuinely elegant.

In modern use, Make is rarely used for what it was designed for. It is used as a task runner: a consistent interface that wraps whatever the project actually needs to do.

.PHONY: build test lint deploy

build:
	docker build -t myapp .

test:
	docker run --rm myapp pytest

lint:
	docker run --rm myapp ruff check .

deploy:
	helm upgrade --install myapp ./chart

make build, make test, make lint — same interface regardless of what’s underneath. New team member, CI pipeline, colleague’s machine: one place to look.

Why it still works

Installed everywhere. No setup, no version to pin.
Self-documenting entry point — cat Makefile tells you how to work with the project.
Trivially wraps Docker, shell scripts, language-specific tools, cloud CLIs.
Low overhead — it is just shell execution with a thin layer on top.

The fact that it is old is not a weakness. It is stable, understood, and available.

The `.PHONY` problem

Make’s native model assumes targets are files. make build checks if a file named build exists and skips the recipe if it does. .PHONY declares that a target is not a file, forcing it to always run.

Forgetting .PHONY is a classic Make footgun. Always declare task targets phony.

.PHONY: build test lint clean

Modern alternatives

Just (Justfile) — purpose-built task runner, not a build system. Cleaner syntax than Make, no file-dependency model to work around, better error messages. Cross-platform. Requires installation.

Task (Taskfile.yml) — YAML-based, similar goals to Just. More readable for people uncomfortable with Make syntax.

Why still use Make then: zero install cost, ubiquity in CI environments, and for many projects the .PHONY quirk is the only real friction. If the team already knows Make and the project isn’t complex, switching to Just adds a dependency without solving a real problem.

If starting fresh on a project where the team is comfortable with the tooling choice, Just is the cleaner option.

It is there, it works. That is not faint praise — ubiquity and stability are real value. Every project gets a Makefile. New person joins, CI pipeline runs, colleague needs to build something: make is the answer and it requires no explanation.

Pattern: shared interface over varied internals

The real value is not Make specifically — it is the pattern of having one consistent entry point regardless of what’s underneath. Whether that is Make, Just, or a scripts/ directory with a run script, the goal is the same: anyone (person or pipeline) can run the project without knowing its internals.

See also Shared Tooling Images — the same principle applied to the tools themselves.

Where this pattern applies

The signal that logic is leaking into the wrong place: when a tool’s config format starts looking like a scripting language. That is the point to extract it into a script and wrap in Make.

Build systems Gradle compiles, tests, and packages. Deployment logic, Docker builds, release steps → scripts. make build calls Gradle. make docker calls a script. make deploy calls Helm. See Build Systems for the full pattern.

Infrastructure

Terraform: terraform apply does the infra. Environment selection, var file injection, state backend config, plan review → scripts. make plan, make apply.
Helm: Helm packages and deploys. Cluster selection, secret fetching, values overrides → scripts. make deploy.
Ansible: Ansible provisions. Inventory management, vault decryption, pre-flight checks → scripts. make provision.

Testing Test runners run tests. Environment setup, test data seeding, coverage reporting → scripts. make test is the entry point regardless of whether it is pytest, jest, or go test underneath.

Package managers A common npm/yarn trap: stuffing 20 scripts into package.json until it becomes a second build system. Keep package.json for dependency management. Orchestration belongs in Make.

Static site / Hugo Hugo builds the site. PDF fetching, Docker wrapping, deploy steps → Makefile. The pattern this site already uses.

SLSA — Supply-chain Levels for Software Artifacts

Wed, 03 Jun 2026 00:00:00 +0000

SLSA (pronounced “salsa”) is a framework for securing the software supply chain. Developed by Google, now under the OpenSSF. The core question it answers: how do you know the artifact you are deploying is actually what was built from the source you think it was?

The answer is provenance: cryptographically signed metadata that records what built an artifact, from what source, when, and under what conditions. SLSA defines levels of increasing rigour around how that provenance is generated and verified.

Why it matters

The software supply chain is an attack surface. SolarWinds, XZ Utils, Log4Shell — different attack vectors but the same underlying problem: something got into the build or the dependency that should not have been there, and nobody noticed until it was too late.

SLSA does not prevent all supply chain attacks. It makes certain classes of attack verifiably harder and gives consumers of an artifact a way to check that it was built the way it claims to have been.

The levels

SLSA defines three build levels (reorganised from the original four in SLSA v1.0):

L1 — Provenance exists The build produces signed provenance documenting what produced the artifact. Any build system, any format. Protects against accidental tampering and gives a starting point for verification.

L2 — Hosted build, signed provenance Build runs on a hosted CI platform (GitHub Actions, Cloud Build, etc.). Provenance is generated and signed by the build platform, not the developer’s machine. Harder to falsify — an attacker needs to compromise the build platform, not just a developer laptop.

L3 — Hardened build Ephemeral, isolated build environment. Non-falsifiable provenance — the build platform itself signs the provenance in a way that even a compromised build script cannot forge. Parameterless builds where the inputs are fully declared.

Provenance in practice

Provenance is a JSON document (SLSA uses the in-toto Attestation Framework) that records:

The source repository and commit
The build platform and workflow
The artifact digest(s)
The builder identity

It is signed using Sigstore / cosign, which provides keyless signing backed by OIDC identity. No key management required for the common case.

Tooling

SLSA GitHub Generator: reusable GitHub Actions workflows that generate SLSA L3 provenance for common artifact types (Go binaries, container images, Maven/Gradle packages). The easiest path to L3 on GitHub Actions.

cosign: signs and verifies container images and provenance attestations. Part of the Sigstore project.

slsa-verifier: CLI tool to verify SLSA provenance against an artifact. Used by consumers to check that an artifact meets a required SLSA level.

Dependency Track / Grype / Trivy: SBOM and vulnerability scanning tools that complement SLSA (SLSA is about build integrity, SBOM is about knowing what is in the artifact — related but distinct).

SLSA and container images

Container images are a natural fit. The image digest is the artifact, provenance attests to the build that produced it, cosign attaches the attestation to the registry.

# verify an image has SLSA provenance
slsa-verifier verify-image \
 ghcr.io/myorg/myimage@sha256:abc123 \
 --source-uri github.com/myorg/myrepo \
 --source-tag v1.0.0

For the tooling images pattern — images published to a registry that teams depend on — SLSA provenance is worth adding. It gives consumers a verifiable chain from source to image.

Practical adoption

L1 is low friction. Generate provenance as part of the build, attach it to the release. Most CI platforms make this straightforward.

L2 requires a hosted build platform (most teams already use one) and using the platform’s signing rather than rolling your own.

L3 requires workflow changes — ephemeral environments, parameterless builds, using the SLSA GitHub Generator or equivalent. More investment but achievable for a standard GitHub Actions project.

Start at L1. Provenance exists, the chain is documented, the habit is established. L2 and L3 follow naturally as the pipeline matures — you are not making a big architectural decision, you are incrementally tightening what you already have.

In practice: never seen L1, L2, or L3 required explicitly by a client or platform. It is still “good idea in theory” territory for most teams. That said, the underlying practices — signed builds, traceable artifacts, reproducible pipelines — show up as requirements all the time, just not always under the SLSA name.

Which points to what SLSA actually is: a name and a framework for practices that are already good ideas. Automate the build, sign the output, record the provenance. Quality follows from automation — it frees up time to iterate, makes review easier, and ensures consistency without manual discipline. SLSA formalises that and gives you a vocabulary to talk about it with others. The levels are a ladder, not a checklist to complete before you can ship.

Shared Tooling Images: SLSA provenance applies directly to published tooling images
GitHub Actions: where SLSA GitHub Generator workflows run

Dagger

Tue, 02 Jun 2026 00:00:00 +0000

Your build script works on your laptop. It breaks in CI because a tool version differs. It breaks for a colleague because they’re on a different OS. You’re writing apt-get install steps in YAML and maintaining a separate script for local builds.

Dagger solves this the same way Docker solved “works on my machine” for applications: run everything in containers. Every pipeline step executes inside a defined container image — identically on your laptop, in GitHub Actions, or inside a Kubernetes pod. The Dagger Engine is a containerised daemon your pipeline calls into. Where it runs is an operational detail; what it does is defined once in code.

The other thing Dagger gives you that YAML-based CI systems don’t: your pipeline is real code. Write it in Go (or TypeScript or Python), build a library of reusable functions, share it across projects, and test it the same way you’d test any other code.

Module

A Dagger module is a Go package that exposes pipeline functions. Initialise one in your repo:

dagger init --sdk go --name my-pipeline
dagger develop

dagger init creates the scaffolding. dagger develop generates the Go bindings and drops you into a module ready to edit. A minimal function:

package main

import (
 "context"
 "dagger/my-pipeline/internal/dagger"
)

type MyPipeline struct{}

func (m *MyPipeline) Build(ctx context.Context, src *dagger.Directory) *dagger.Container {
 return dag.Container().
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
}

Call it from anywhere — local terminal, CI step, Argo workflow:

dagger call build --src .

Library pattern

Because a module is just Go code, you build it like any other Go library: shared types, helper functions, unit tests. The pattern that works well in practice is a single internal module that codifies your organisation’s conventions — base images, registry credentials, standard lint and test steps — and each project imports it.

A fix to the shared module propagates everywhere. You’re not updating 40 YAML files.

Testing pipeline logic with go test works normally. A test that calls dagger call exercises the real container execution, not a mock. This is the thing YAML pipelines can’t give you: a fast feedback loop on the pipeline logic itself before it hits CI.

Multi-arch builds

Dagger has first-class support for multi-architecture container builds. Pass the platform as an argument:

func (m *MyPipeline) BuildMultiArch(ctx context.Context, src *dagger.Directory) []*dagger.Container {
 platforms := []dagger.Platform{"linux/amd64", "linux/arm64"}
 containers := make([]*dagger.Container, len(platforms))
 for i, platform := range platforms {
 containers[i] = dag.Container(dagger.ContainerOpts{Platform: platform}).
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
 }
 return containers
}

Same function, multiple targets, no extra tooling.

Running in Kubernetes with Argo Workflows

Dagger Engine runs as a container. In Kubernetes, you run it as a sidecar or dedicated pod and point dagger call at it via the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable. Each Argo Workflows step calls dagger call <function> — Argo handles DAG orchestration, retries, and observability; Dagger handles the build execution inside containers.

The split is clean: Argo owns workflow-level concerns, Dagger owns build reproducibility.

Key commands

Command	What it does
`dagger init --sdk go`	Initialise a new module with the Go SDK
`dagger develop`	Generate bindings, enter the development loop
`dagger functions`	List available functions in the current module
`dagger call <function> [args]`	Invoke a pipeline function
`dagger run <command>`	Run an arbitrary command with Dagger Engine available

Dagger Cloud

Dagger Cloud is a hosted observability layer for Dagger pipelines. Free tier available.

Every pipeline run — local or CI — produces a trace: a visualisation of the container graph with per-step timing, cache hit/miss, and log output. The trace is linked from the terminal output and browsable at cloud.dagger.io.

Enable it by setting one environment variable:

export DAGGER_CLOUD_TOKEN=<token>
dagger call build --src .
# → Run URL: https://app.dagger.cloud/runs/<id>

In GitHub Actions, add DAGGER_CLOUD_TOKEN as an organisation-level secret (see GitHub Actions). The dagger/dagger-action picks it up automatically — no workflow change needed beyond the env var being present.

Resources

Argo

Mon, 01 Jan 2024 00:00:00 +0000

The Argo project is a suite of Kubernetes-native tools for running and managing workloads and deployments. Each tool solves a distinct problem and they compose well together, but they are independent — you can use any one without the others. All four are CNCF graduated or incubating projects.

ArgoCD

GitOps continuous delivery. ArgoCD watches a Git repository and continuously reconciles the cluster state to match it — any drift is detected and corrected automatically. It is the CD half of a modern Kubernetes delivery pipeline: a CI system builds and pushes an image, ArgoCD detects the new tag and rolls it out. See the ArgoCD note for a full walkthrough including App of Apps, bootstrapping, and self-management.

Argo Workflows

A general-purpose workflow execution engine for Kubernetes. Workflows are CRDs that define DAGs or sequential step graphs — each step runs in a container, with outputs passed as artifacts or parameters to downstream steps. Used for CI pipelines, ML training jobs, data processing, and batch workloads. Where Tekton models CI-specific primitives (Tasks, Pipelines), Argo Workflows is lower-level and more flexible: any containerised workload that has dependencies between steps fits the model.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
 entrypoint: build-test
 templates:
 - name: build-test
 dag:
 tasks:
 - name: build
 template: run-step
 arguments:
 parameters: [{name: cmd, value: "make build"}]
 - name: test
 dependencies: [build]
 template: run-step
 arguments:
 parameters: [{name: cmd, value: "make test"}]
 - name: run-step
 inputs:
 parameters:
 - name: cmd
 container:
 image: golang:1.22
 command: [sh, -c]
 args: ["{{inputs.parameters.cmd}}"]

Argo Rollouts

Progressive delivery for Kubernetes. Where a standard Kubernetes Deployment does a rolling update (replace pods gradually), Argo Rollouts adds canary and blue-green strategies with analysis gates. A canary rollout shifts a percentage of traffic to the new version, runs automated analysis (checking metrics from Prometheus, Datadog, or similar), and either promotes fully or rolls back based on the result. This makes deployments measurably safer — a bad release fails the analysis gate before it reaches 100% of traffic.

Argo Events

Event-driven automation. Argo Events defines EventSources (sensors that listen for events — git pushes, S3 uploads, Kafka messages, webhooks, cron schedules) and Sensors (triggers that respond to those events by creating Argo Workflows, sending notifications, or calling other systems). It is the event bus that ties the rest of the Argo stack together: a git push fires an EventSource, a Sensor detects it and creates a Workflow, the Workflow builds and tests, ArgoCD picks up the new image and rolls it out.

Kargo

A newer tool from Akuity (the company behind ArgoCD) that solves multi-stage GitOps promotion. ArgoCD is good at keeping one environment in sync with a Git ref — but promoting a release through dev → staging → production requires updating that ref in each environment and coordinating the sequence. Kargo models this as Stages with FreightRequests — a release is a piece of freight that must pass through each stage in order, with optional approval gates between them. It sits above ArgoCD in the stack and handles the promotion logic that ArgoCD deliberately leaves out.

Resources

ArgoCD

Mon, 01 Jan 2024 00:00:00 +0000

You deploy with kubectl apply from your laptop. It works. Then a colleague edits a deployment directly on the cluster to fix something urgent. Now what is running no longer matches what is in Git. That is drift, and it is silent — until something breaks in production and nobody can explain why the live state differs from the last known good config.

So you use ArgoCD. Git becomes the single source of truth. Every change flows through a pull request, gets reviewed, and syncs to the cluster automatically. If anyone touches a resource directly, ArgoCD detects the divergence and overrides it back. The cluster converges to Git, always.

This is GitOps: the deployment pipeline is driven by Git state, not by humans running commands.

CI vs CD

A useful mental separation: CI and CD are different concerns and should be handled by different tools.

CI (Continuous Integration) is about code — build, test, produce an artifact (a container image). A pipeline in GitHub Actions, Tekton, or Jenkins owns this. It ends with an image pushed to a registry.

CD (Continuous Delivery) is about cluster state — take that artifact and make sure the right version is running in the right environment. ArgoCD owns this. It watches Git, not the CI pipeline.

Keeping them separate means your deployment logic is not buried inside a CI pipeline that developers need to understand and maintain. ArgoCD runs in the cluster and continuously reconciles state. It is always on.

Applications

ArgoCD manages Applications — a CRD that maps a Git source to a cluster destination:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: my-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/myorg/my-app-config
 targetRevision: main
 path: manifests/
 destination:
 server: https://kubernetes.default.svc
 namespace: my-app
 syncPolicy:
 automated:
 prune: true
 selfHeal: true

prune: true — resources removed from Git are deleted from the cluster. selfHeal: true — any manual change to the cluster is immediately reverted.

App of Apps

Managing dozens of Applications individually gets unwieldy. The App of Apps pattern solves this: one root Application whose source is a directory of other Application manifests. ArgoCD applies the root, which creates all the child Applications, which in turn sync their own workloads. One repo, one sync, everything deployed.

Sync strategies

Strategy	Behaviour
Automated	ArgoCD syncs on every Git change automatically
Manual	Changes are detected and shown as OutOfSync — a human triggers the sync

Automated sync with selfHeal is the purest GitOps posture. Manual sync is useful for production environments where you want a human approval step before changes roll out.

Rollback

Because every state the cluster has ever been in corresponds to a Git commit, rollback is a git revert — or clicking “Sync to previous revision” in the ArgoCD UI. No special tooling, no runbooks, just Git history.

Repo structure

A layout that works well in practice separates ArgoCD’s own installation from the workloads it manages:

cluster/<cluster>/
 cfg/argo-cd/ # ArgoCD install only — CRDs and Helm values
 app-of-apps/ # Root Application, Projects, app definitions
 overlay/<app>/ # Per-cluster Kustomize patches, secret/config overrides

external/ # Reusable base manifests shared across clusters
internal/ # Internal app base manifests

The key separations:

ArgoCD install is isolated in cfg/argo-cd to avoid recursive install loops and make upgrades predictable. ArgoCD is not managing its own installation yet at this point — that comes later.
App-of-Apps lives separately from the install. Once ArgoCD is running, applying app-of-apps/ bootstraps the entire cluster in one step.
Base vs overlay — external/ and internal/ define what an app is. The cluster overlay defines how it runs in this environment. Cluster-specific concerns (resource limits, replica counts, secret refs) stay in the cluster directory and never bleed into the base.

Bootstrapping a cluster

There is a chicken-and-egg problem: ArgoCD manages everything, but something has to install ArgoCD first. The two-step bootstrap solves it cleanly.

Step 1 — Install ArgoCD manually (once):

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd \
 -n argo-cd --create-namespace \
 -f cluster/staging/cfg/argo-cd/values.yaml

Step 2 — Apply the App-of-Apps root:

kubectl apply -k cluster/<cluster>/app-of-apps/

From this point ArgoCD reconciles the entire cluster. Every subsequent change goes through Git — you never run helm install or kubectl apply for workloads again.

Self-management

The final step is making ArgoCD manage its own upgrades. Create an Application that points at cluster/<cluster>/cfg/argo-cd:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: argocd
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/myorg/cluster-config
 targetRevision: main
 path: cluster/staging/cfg/argo-cd
 destination:
 server: https://kubernetes.default.svc
 namespace: argocd
 syncPolicy:
 automated:
 prune: false # be cautious pruning ArgoCD's own resources
 selfHeal: true

Now ArgoCD upgrades itself when you update the Helm values in Git. No more manual helm upgrade — the cluster is fully self-managing. Changes to ArgoCD config go through the same PR review process as everything else.

Resources

CI/CD Platforms

Mon, 01 Jan 2024 00:00:00 +0000

There are many CI/CD platforms and the choice between them matters less than it appears. All of them are thin orchestration wrappers — trigger on a git event, run some steps, report the result. The build logic itself should live in Make or Dagger, not inside the pipeline definition. See One Command, Any Pipeline.

CruiseControl

The early pioneer, released in 2001. CruiseControl introduced continuous integration as a practice — polling a source repository, building on every change, sending email on failure. Configuration was XML, the dashboard was a web page, and it ran on a server you managed yourself. Most of the concepts in modern CI trace back here. Largely historical today but worth knowing as the origin point.

Hudson

Hudson was Sun Microsystems’ take on CI — a Java application with a plugin ecosystem that made it far more extensible than CruiseControl. It gained wide adoption in enterprise Java shops during the late 2000s. When Oracle acquired Sun, the project forked. The community fork became Jenkins; the Oracle-maintained branch kept the Hudson name and eventually faded. Hudson is effectively dead.

Jenkins

The fork that won. Jenkins took Hudson’s plugin architecture and ran with it — today it has over 1800 plugins covering almost every tool in the ecosystem. A Jenkinsfile defines the pipeline as Groovy DSL and lives in the repository alongside the code. Jenkins is the most widely deployed self-hosted CI server and the default answer in many enterprises. The flip side: it is heavyweight, the Groovy DSL has sharp edges, and complex pipelines are difficult to test outside of Jenkins itself.

Jenkins X

Jenkins X is a cloud-native reimagining of Jenkins for Kubernetes. It imposes a strongly opinionated GitOps workflow — pull requests promote through environments, preview environments spin up automatically, everything is driven by Git events. Built on Tekton under the hood. If you want opinionated Kubernetes CI/CD without building the conventions yourself, Jenkins X is one answer. If you want more control over the pipeline structure, raw Tekton gives you the primitives without the opinions.

Tekton

Kubernetes-native CI/CD where pipelines, tasks, and triggers are all Kubernetes CRDs — defined in YAML and applied to a cluster, running as pods. No separate CI server to maintain, no external SaaS dependency. CI runs in the same cluster as your workloads using the same RBAC, secrets, and storage primitives. The core primitives are Task (a sequence of container steps), Pipeline (an ordered set of tasks), PipelineRun (an execution), and Trigger (an event listener that creates runs). The pattern that works well: keep Tasks thin and have them call make <target> — Tekton handles orchestration, Make handles logic.

GitHub Actions

GitHub’s built-in CI/CD, available to any repository on GitHub. Zero infrastructure, zero setup — if your code is on GitHub, Actions is already there. Workflows are YAML files in .github/workflows/, triggered by git events, running on GitHub-managed runners. A large marketplace of pre-built actions covers most common tasks. The zero-friction default for open source projects and small teams. See the GitHub Actions note for more detail.

Argo Workflows

A Kubernetes-native workflow engine from the Argo project. Where Tekton models CI primitives (Tasks, Pipelines), Argo Workflows is a general-purpose DAG executor — it can run any containerised workload as a directed acyclic graph of steps, with fan-out, fan-in, conditionals, and retry logic. Widely used as the execution layer under other tools (including Dagger on Kubernetes). Pairs well with ArgoCD for a fully Argo-based GitOps stack. See the Argo note for coverage of the full Argo ecosystem including Rollouts, Events, and Kargo.

Bitbucket Pipelines

Bitbucket’s built-in CI/CD, integrated directly with Atlassian’s hosting. If your code is already in Bitbucket, Pipelines is the zero-infrastructure option — the same position GitHub Actions occupies for GitHub users. Workflows are YAML, steps run in Docker containers, and Atlassian handles the runner infrastructure. Tightly integrated with Jira for deployment tracking. The right choice when you’re already in the Atlassian ecosystem and don’t want to introduce a separate CI tool.

Harness

A commercial platform with a broader scope than most CI/CD tools — it covers CI, CD, feature flags, cloud cost management, and security testing under one roof. Enterprise-focused, with AI-assisted pipeline generation and strong support for policy and governance across large engineering organisations. Harness is the answer when the organisation needs a managed platform with SLAs, support, and audit trails rather than self-hosted infrastructure. Pricing reflects that positioning.

Resources

Git

Mon, 01 Jan 2024 00:00:00 +0000

Git is the distributed version control system behind virtually all modern software development. Every clone is a full copy of the history — fast, local, and cryptographically integrity-checked.

Branches

Branches in Git are cheap: creating, merging, and deleting them is fast because a branch is just a pointer to a commit, not a copy of the files.

By convention, one branch is singled out and called main (historically master).

Workflows

How teams structure collaboration varies. Three common models:

Benevolent dictator — used by the Linux kernel. A hierarchy of trusted lieutenants, with one maintainer having final merge authority.

Integration manager — common in open source. Each contributor has their own public fork; a maintainer pulls from forks into a blessed repository.

Shared repository — the most common in companies. Everyone pushes to the same remote (origin). All instances are equal; origin is just a convention.

Data integrity

Git’s data model uses SHA-1 checksums for every file and commit. This means:

Every object is verified on checkout
History cannot be silently altered — changing any commit changes all IDs after it
You can’t push without first pulling (unless you force-push and overwrite)

Merge vs rebase

Two ways to integrate branches:

Merge preserves history exactly as it happened — a merge commit ties the two branches together.

Rebase replays your commits on top of the target branch — produces a linear history but rewrites commit SHAs. Never rebase shared/public branches.

Branch strategies

Git Flow — a structured model with main, develop, feature/*, release/*, and hotfix/* branches. Good for software with versioned releases.

Git Flow original post

GitHub Flow / GitLab Flow — simpler: branch from main, open a pull request, merge when reviewed. Works well for continuously deployed web apps.

GitHub Flow · GitLab Flow

Which strategy fits

Project type	Recommendation
Library / SDK	Git Flow — versioned releases, may need to support multiple versions simultaneously
Web application	GitHub Flow — continuously delivered, no need for release branches
Client / installable app	Git Flow — users run specific versions you need to patch and maintain

Advanced Features

Git Worktree

git worktree allows you to check out multiple branches of the same repository into separate directories simultaneously. This is particularly useful for:

Working on multiple features or bug fixes in parallel without stashing changes.
Quickly switching contexts for code reviews or testing different versions of your code.

All worktrees share the same Git objects, minimizing disk space usage.

Resources

Gitea / Forgejo

Mon, 01 Jan 2024 00:00:00 +0000

Gogs came first — a self-hosted Git service, lightweight, single binary, runs anywhere. Think GitHub but on your own infrastructure. Simple to deploy, easy to understand, and that was the point.

Gitea forked from Gogs in 2016 and has moved considerably faster since — more features, more active development, more community. For any new self-hosted Git deployment, Gitea became the natural choice over Gogs. Unless you look at where Gitea itself is heading.

Forgejo

Gitea has been drifting toward the enterprise and commercial end. The governance has shifted, cloud services have been prioritised, and the community-first ethos that made it attractive has weakened.

Forgejo is the community response — a fork of Gitea maintained by the Codeberg team, focused on staying open, community-governed, and free. It is a drop-in replacement: same API, same data format, same deployment model. Migrating from Gitea to Forgejo is straightforward.

For new deployments today, Forgejo is the better choice. Gitea is still solid and maintained, but the trajectory matters.

Codeberg — the public hosting platform run by the Forgejo maintainers — is worth knowing about as a GitHub alternative for open source projects.

Features

SSH and HTTPS repository access
Issues, pull requests, code review, wiki, protected branches
Webhooks and Git hooks
Deploy keys and fine-grained access tokens
LDAP, SMTP, OAuth, OIDC authentication
PostgreSQL, MySQL, SQLite support
Built-in container registry
Gitea Actions / Forgejo Actions — GitHub Actions-compatible CI

Issues

The issue tracker covers the basics well:

Numeric ID, description, comment thread
Assignable to users, file attachments, labels, milestones
Open/closed state per repository

Default labels — fully customizable:

Organisation-wide issue overview:

Resources

Forgejo — community fork, recommended for new deployments
Gitea
Gogs — where it all started
Codeberg — public Forgejo hosting

GitHub Actions

Mon, 01 Jan 2024 00:00:00 +0000

For a small project or proof of concept, the cost of building a CI environment often exceeds the cost of the project itself. Spinning up a Tekton cluster, installing Argo Workflows, or maintaining a Jenkins server is real infrastructure — with real setup time, real maintenance overhead, and real dependencies to keep running.

GitHub Actions is already there. If your code is on GitHub, you have CI. No extra infrastructure, no additional accounts, a marketplace of pre-built integrations for almost everything you need. For open source projects and small teams it is the pragmatic default: free, integrated, and good enough for the standard build → test → publish pipeline.

Workflow anatomy

Workflows live in .github/workflows/ as YAML files. A workflow has triggers, jobs, and steps:

name: Build and publish

on:
 push:
 branches: [main]
 pull_request:
 branches: [main]

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4

 - name: Build
 run: make build

 - name: Test
 run: make test

 - name: Push image
 if: github.ref == 'refs/heads/main'
 run: |
 echo "${{ secrets.DOCKER_HUB_TOKEN }}" | docker login -u ${{ secrets.DOCKER_HUB_USER }} --password-stdin
 docker push myorg/myimage:latest

on: defines what triggers the workflow. runs-on: selects the runner. uses: pulls a pre-built action from the marketplace. run: executes shell commands directly.

The if: condition on the push step is a common pattern: run tests on every pull request, but only publish on merge to main.

Secrets

GitHub Secrets store credentials without putting them in the repository. Add them under Settings → Secrets and variables → Actions, then reference them in workflows as ${{ secrets.MY_SECRET }}.

For pushing to external services — Docker Hub, a package registry, a cloud provider — this is the integration point. The workflow authenticates using the secret, the secret itself never appears in logs or code.

Organisation secrets

Secrets can be set at the organisation level and inherited by all repositories — no per-repo configuration needed. Useful for shared CI credentials reused across many repos.

Path: github.com/<org> → Settings → Secrets and variables → Actions → New organisation secret

Set Repository access to “All repositories” or a selected subset once you know which repos need it.

A worked example — shared image publishing credentials:

DOCKERHUB_TOKEN # hub.docker.com → Account → Security → Access Tokens
DAGGER_CLOUD_TOKEN # cloud.dagger.io → Organisation → Tokens

Set once at org level; every image-* repo inherits them. Workflows reference them identically to repo secrets: ${{ secrets.DOCKERHUB_TOKEN }}. See Dagger for DAGGER_CLOUD_TOKEN context.

Environments

Environments add a protection layer on top of secrets. Define named environments (e.g. staging, production) and configure:

Required reviewers — a human must approve before the job runs
Environment secrets — secrets scoped to that environment only, not available to other jobs
Wait timer — a mandatory delay before deployment proceeds

A deployment job targets an environment by name:

jobs:
 deploy:
 runs-on: ubuntu-latest
 environment: production
 steps:
 - name: Deploy
 run: ./deploy.sh
 env:
 API_KEY: ${{ secrets.PROD_API_KEY }}

The job pauses for approval before running. Useful for open source projects where the pipeline is public but deployment credentials are not.

Marketplace actions

Most common tasks have a pre-built action. A few worth knowing:

Action	What it does
`actions/checkout`	Clone the repository
`actions/setup-go`	Install a specific Go version
`docker/setup-buildx-action`	Enable multi-arch Docker builds
`docker/build-push-action`	Build and push a container image
`helm/kind-action`	Spin up a local Kubernetes cluster for tests

Using marketplace actions trades control for speed. For a POC or small project that’s the right trade. For anything critical, pin the action to a specific commit SHA rather than a tag — tags are mutable.

Self-hosted runners

GitHub-hosted runners (ubuntu-latest, windows-latest) cover most cases. Self-hosted runners make sense when you need access to an internal network, specific hardware (a GPU, specialised build machine), or want to avoid per-minute billing at scale.

At that point you are closer to running a full CI platform, which is where tools like Tekton or Argo Workflows become relevant — they give you the same kind of pipeline orchestration but running entirely on your own infrastructure.

CI/CD on Backend Engineering Strategy Tools

Build Systems — Ant, Maven, Gradle, Bazel

Ant

Maven

Gradle

SBT

Bazel & Bazelisk

Choosing

Make — Task Runner Pattern

Why it still works

The .PHONY problem

Modern alternatives

Pattern: shared interface over varied internals

Where this pattern applies

SLSA — Supply-chain Levels for Software Artifacts

Why it matters

The levels

Provenance in practice

Tooling

SLSA and container images

Practical adoption

Related

Dagger

Module

Library pattern

Multi-arch builds

Running in Kubernetes with Argo Workflows

Key commands

Dagger Cloud

Resources

Argo

ArgoCD

Argo Workflows

Argo Rollouts

Argo Events

Kargo

Resources

ArgoCD

CI vs CD

Applications

App of Apps

Sync strategies

Rollback

Repo structure

Bootstrapping a cluster

Self-management

Resources

CI/CD Platforms

CruiseControl

Hudson

Jenkins

Jenkins X

Tekton

GitHub Actions

Argo Workflows

Bitbucket Pipelines

Harness

Resources

Git

Branches

Workflows

Data integrity

Merge vs rebase

Branch strategies

Which strategy fits

Advanced Features

Git Worktree

Resources

Gitea / Forgejo

Forgejo

Features

Issues

Resources

GitHub Actions

Workflow anatomy

Secrets

Organisation secrets

Environments

Marketplace actions

Self-hosted runners

The `.PHONY` problem