Pipelines on Backend Engineering Strategy Tools

These Are Not the Pipelines You Are Looking For

Thu, 04 Jun 2026 00:00:00 +0000

There are a lot of CI/CD platforms. GitHub Actions, Tekton, Jenkins, Jenkins X, Harness, Bitbucket Pipelines, GitLab CI, CircleCI, Argo Workflows. The choice between them is a perennial argument — and mostly the wrong one.

The mistake is putting your build logic inside the pipeline. Once you do that, the pipeline owns your build. Reproducing a failure locally means setting up the CI environment. Switching platforms means rewriting all the steps. Testing the pipeline means pushing a commit and waiting. The platform becomes load-bearing.

The fix is straightforward: keep all logic in Make or Dagger. The pipeline calls one command. make build. make test. dagger call publish. That’s it.

The pipeline shrinks to thin orchestration: trigger on a git event, check out the code, call the command, report the result. It does not know what the build does. It does not care. You can swap GitHub Actions for Tekton or Tekton for Argo Workflows and the only thing that changes is the YAML wrapper around the same command.

More importantly: you can run the same command on your laptop. No CI environment to replicate, no “it works locally but fails in CI” gap to debug. The build is the build, wherever it runs.

This matters more the longer a project lives. Build tools consolidate and diverge. Platforms get acquired, deprecated, or repriced. Jenkins pipelines from five years ago are maintained by people who weren’t there when they were written and can’t run them outside of Jenkins. The projects that survive these transitions cleanly are the ones where the build logic was never in the pipeline to begin with — it was in a Makefile or a build tool that could run anywhere.

Gradle, Maven, Make — these predate CI/CD platforms by decades and will outlast the current generation of them. Put your logic there. The pipeline is a trigger and a reporter. Keep it that way.

The pattern in practice:

build:
 docker build -t myimage:$(VERSION) .

test:
 go test ./...

publish:
 docker push myimage:$(VERSION)

# GitHub Actions — the entire pipeline
steps:
 - uses: actions/checkout@v4
 - run: make build
 - run: make test
 - run: make publish

# Tekton — same command, different wrapper
steps:
 - name: build-test-publish
 image: golang:1.22
 script: |
 make build
 make test
 make publish

Same logic. One command. Any pipeline.

Dagger

Tue, 02 Jun 2026 00:00:00 +0000

Your build script works on your laptop. It breaks in CI because a tool version differs. It breaks for a colleague because they’re on a different OS. You’re writing apt-get install steps in YAML and maintaining a separate script for local builds.

Dagger solves this the same way Docker solved “works on my machine” for applications: run everything in containers. Every pipeline step executes inside a defined container image — identically on your laptop, in GitHub Actions, or inside a Kubernetes pod. The Dagger Engine is a containerised daemon your pipeline calls into. Where it runs is an operational detail; what it does is defined once in code.

The other thing Dagger gives you that YAML-based CI systems don’t: your pipeline is real code. Write it in Go (or TypeScript or Python), build a library of reusable functions, share it across projects, and test it the same way you’d test any other code.

Module

A Dagger module is a Go package that exposes pipeline functions. Initialise one in your repo:

dagger init --sdk go --name my-pipeline
dagger develop

dagger init creates the scaffolding. dagger develop generates the Go bindings and drops you into a module ready to edit. A minimal function:

package main

import (
 "context"
 "dagger/my-pipeline/internal/dagger"
)

type MyPipeline struct{}

func (m *MyPipeline) Build(ctx context.Context, src *dagger.Directory) *dagger.Container {
 return dag.Container().
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
}

Call it from anywhere — local terminal, CI step, Argo workflow:

dagger call build --src .

Library pattern

Because a module is just Go code, you build it like any other Go library: shared types, helper functions, unit tests. The pattern that works well in practice is a single internal module that codifies your organisation’s conventions — base images, registry credentials, standard lint and test steps — and each project imports it.

A fix to the shared module propagates everywhere. You’re not updating 40 YAML files.

Testing pipeline logic with go test works normally. A test that calls dagger call exercises the real container execution, not a mock. This is the thing YAML pipelines can’t give you: a fast feedback loop on the pipeline logic itself before it hits CI.

Multi-arch builds

Dagger has first-class support for multi-architecture container builds. Pass the platform as an argument:

func (m *MyPipeline) BuildMultiArch(ctx context.Context, src *dagger.Directory) []*dagger.Container {
 platforms := []dagger.Platform{"linux/amd64", "linux/arm64"}
 containers := make([]*dagger.Container, len(platforms))
 for i, platform := range platforms {
 containers[i] = dag.Container(dagger.ContainerOpts{Platform: platform}).
 From("golang:1.22").
 WithDirectory("/src", src).
 WithWorkdir("/src").
 WithExec([]string{"go", "build", "./..."})
 }
 return containers
}

Same function, multiple targets, no extra tooling.

Running in Kubernetes with Argo Workflows

Dagger Engine runs as a container. In Kubernetes, you run it as a sidecar or dedicated pod and point dagger call at it via the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable. Each Argo Workflows step calls dagger call <function> — Argo handles DAG orchestration, retries, and observability; Dagger handles the build execution inside containers.

The split is clean: Argo owns workflow-level concerns, Dagger owns build reproducibility.

Key commands

Command	What it does
`dagger init --sdk go`	Initialise a new module with the Go SDK
`dagger develop`	Generate bindings, enter the development loop
`dagger functions`	List available functions in the current module
`dagger call <function> [args]`	Invoke a pipeline function
`dagger run <command>`	Run an arbitrary command with Dagger Engine available

Dagger Cloud

Dagger Cloud is a hosted observability layer for Dagger pipelines. Free tier available.

Every pipeline run — local or CI — produces a trace: a visualisation of the container graph with per-step timing, cache hit/miss, and log output. The trace is linked from the terminal output and browsable at cloud.dagger.io.

Enable it by setting one environment variable:

export DAGGER_CLOUD_TOKEN=<token>
dagger call build --src .
# → Run URL: https://app.dagger.cloud/runs/<id>

In GitHub Actions, add DAGGER_CLOUD_TOKEN as an organisation-level secret (see GitHub Actions). The dagger/dagger-action picks it up automatically — no workflow change needed beyond the env var being present.

Resources

CI/CD Platforms

Mon, 01 Jan 2024 00:00:00 +0000

There are many CI/CD platforms and the choice between them matters less than it appears. All of them are thin orchestration wrappers — trigger on a git event, run some steps, report the result. The build logic itself should live in Make or Dagger, not inside the pipeline definition. See One Command, Any Pipeline.

CruiseControl

The early pioneer, released in 2001. CruiseControl introduced continuous integration as a practice — polling a source repository, building on every change, sending email on failure. Configuration was XML, the dashboard was a web page, and it ran on a server you managed yourself. Most of the concepts in modern CI trace back here. Largely historical today but worth knowing as the origin point.

Hudson

Hudson was Sun Microsystems’ take on CI — a Java application with a plugin ecosystem that made it far more extensible than CruiseControl. It gained wide adoption in enterprise Java shops during the late 2000s. When Oracle acquired Sun, the project forked. The community fork became Jenkins; the Oracle-maintained branch kept the Hudson name and eventually faded. Hudson is effectively dead.

Jenkins

The fork that won. Jenkins took Hudson’s plugin architecture and ran with it — today it has over 1800 plugins covering almost every tool in the ecosystem. A Jenkinsfile defines the pipeline as Groovy DSL and lives in the repository alongside the code. Jenkins is the most widely deployed self-hosted CI server and the default answer in many enterprises. The flip side: it is heavyweight, the Groovy DSL has sharp edges, and complex pipelines are difficult to test outside of Jenkins itself.

Jenkins X

Jenkins X is a cloud-native reimagining of Jenkins for Kubernetes. It imposes a strongly opinionated GitOps workflow — pull requests promote through environments, preview environments spin up automatically, everything is driven by Git events. Built on Tekton under the hood. If you want opinionated Kubernetes CI/CD without building the conventions yourself, Jenkins X is one answer. If you want more control over the pipeline structure, raw Tekton gives you the primitives without the opinions.

Tekton

Kubernetes-native CI/CD where pipelines, tasks, and triggers are all Kubernetes CRDs — defined in YAML and applied to a cluster, running as pods. No separate CI server to maintain, no external SaaS dependency. CI runs in the same cluster as your workloads using the same RBAC, secrets, and storage primitives. The core primitives are Task (a sequence of container steps), Pipeline (an ordered set of tasks), PipelineRun (an execution), and Trigger (an event listener that creates runs). The pattern that works well: keep Tasks thin and have them call make <target> — Tekton handles orchestration, Make handles logic.

GitHub Actions

GitHub’s built-in CI/CD, available to any repository on GitHub. Zero infrastructure, zero setup — if your code is on GitHub, Actions is already there. Workflows are YAML files in .github/workflows/, triggered by git events, running on GitHub-managed runners. A large marketplace of pre-built actions covers most common tasks. The zero-friction default for open source projects and small teams. See the GitHub Actions note for more detail.

Argo Workflows

A Kubernetes-native workflow engine from the Argo project. Where Tekton models CI primitives (Tasks, Pipelines), Argo Workflows is a general-purpose DAG executor — it can run any containerised workload as a directed acyclic graph of steps, with fan-out, fan-in, conditionals, and retry logic. Widely used as the execution layer under other tools (including Dagger on Kubernetes). Pairs well with ArgoCD for a fully Argo-based GitOps stack. See the Argo note for coverage of the full Argo ecosystem including Rollouts, Events, and Kargo.

Bitbucket Pipelines

Bitbucket’s built-in CI/CD, integrated directly with Atlassian’s hosting. If your code is already in Bitbucket, Pipelines is the zero-infrastructure option — the same position GitHub Actions occupies for GitHub users. Workflows are YAML, steps run in Docker containers, and Atlassian handles the runner infrastructure. Tightly integrated with Jira for deployment tracking. The right choice when you’re already in the Atlassian ecosystem and don’t want to introduce a separate CI tool.

Harness

A commercial platform with a broader scope than most CI/CD tools — it covers CI, CD, feature flags, cloud cost management, and security testing under one roof. Enterprise-focused, with AI-assisted pipeline generation and strong support for policy and governance across large engineering organisations. Harness is the answer when the organisation needs a managed platform with SLAs, support, and audit trails rather than self-hosted infrastructure. Pricing reflects that positioning.

Resources

GitHub Actions

Mon, 01 Jan 2024 00:00:00 +0000

For a small project or proof of concept, the cost of building a CI environment often exceeds the cost of the project itself. Spinning up a Tekton cluster, installing Argo Workflows, or maintaining a Jenkins server is real infrastructure — with real setup time, real maintenance overhead, and real dependencies to keep running.

GitHub Actions is already there. If your code is on GitHub, you have CI. No extra infrastructure, no additional accounts, a marketplace of pre-built integrations for almost everything you need. For open source projects and small teams it is the pragmatic default: free, integrated, and good enough for the standard build → test → publish pipeline.

Workflow anatomy

Workflows live in .github/workflows/ as YAML files. A workflow has triggers, jobs, and steps:

name: Build and publish

on:
 push:
 branches: [main]
 pull_request:
 branches: [main]

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4

 - name: Build
 run: make build

 - name: Test
 run: make test

 - name: Push image
 if: github.ref == 'refs/heads/main'
 run: |
 echo "${{ secrets.DOCKER_HUB_TOKEN }}" | docker login -u ${{ secrets.DOCKER_HUB_USER }} --password-stdin
 docker push myorg/myimage:latest

on: defines what triggers the workflow. runs-on: selects the runner. uses: pulls a pre-built action from the marketplace. run: executes shell commands directly.

The if: condition on the push step is a common pattern: run tests on every pull request, but only publish on merge to main.

Secrets

GitHub Secrets store credentials without putting them in the repository. Add them under Settings → Secrets and variables → Actions, then reference them in workflows as ${{ secrets.MY_SECRET }}.

For pushing to external services — Docker Hub, a package registry, a cloud provider — this is the integration point. The workflow authenticates using the secret, the secret itself never appears in logs or code.

Organisation secrets

Secrets can be set at the organisation level and inherited by all repositories — no per-repo configuration needed. Useful for shared CI credentials reused across many repos.

Path: github.com/<org> → Settings → Secrets and variables → Actions → New organisation secret

Set Repository access to “All repositories” or a selected subset once you know which repos need it.

A worked example — shared image publishing credentials:

DOCKERHUB_TOKEN # hub.docker.com → Account → Security → Access Tokens
DAGGER_CLOUD_TOKEN # cloud.dagger.io → Organisation → Tokens

Set once at org level; every image-* repo inherits them. Workflows reference them identically to repo secrets: ${{ secrets.DOCKERHUB_TOKEN }}. See Dagger for DAGGER_CLOUD_TOKEN context.

Environments

Environments add a protection layer on top of secrets. Define named environments (e.g. staging, production) and configure:

Required reviewers — a human must approve before the job runs
Environment secrets — secrets scoped to that environment only, not available to other jobs
Wait timer — a mandatory delay before deployment proceeds

A deployment job targets an environment by name:

jobs:
 deploy:
 runs-on: ubuntu-latest
 environment: production
 steps:
 - name: Deploy
 run: ./deploy.sh
 env:
 API_KEY: ${{ secrets.PROD_API_KEY }}

The job pauses for approval before running. Useful for open source projects where the pipeline is public but deployment credentials are not.

Marketplace actions

Most common tasks have a pre-built action. A few worth knowing:

Action	What it does
`actions/checkout`	Clone the repository
`actions/setup-go`	Install a specific Go version
`docker/setup-buildx-action`	Enable multi-arch Docker builds
`docker/build-push-action`	Build and push a container image
`helm/kind-action`	Spin up a local Kubernetes cluster for tests

Using marketplace actions trades control for speed. For a POC or small project that’s the right trade. For anything critical, pin the action to a specific commit SHA rather than a tag — tags are mutable.

Self-hosted runners

GitHub-hosted runners (ubuntu-latest, windows-latest) cover most cases. Self-hosted runners make sense when you need access to an internal network, specific hardware (a GPU, specialised build machine), or want to avoid per-minute billing at scale.

At that point you are closer to running a full CI platform, which is where tools like Tekton or Argo Workflows become relevant — they give you the same kind of pipeline orchestration but running entirely on your own infrastructure.