Talos on Backend Engineering Strategy Tools

ASGARD — the blade cluster

Fri, 15 May 2026 00:00:00 +0000

ASGARD (SYS-007) is the HP BladeSystem C7000 with 16× BL460c Gen8 blades. The reason to use it is profile switching: boot a blade as a Slurm compute node, run the experiment, reimage it as a Talos worker, run the next one. The same iPXE boot menu already set up for ODEN works here — the C7000 Onboard Administrator lets you configure boot order per blade slot, so switching roles is a BIOS setting and a PXE entry, not a reinstall.

Power reality

Before committing to blades as the permanent always-on platform, it’s worth being honest about the enclosure overhead. The C7000 has fixed costs regardless of how many blades are populated: 10 fans, dual OA modules, 2 interconnect switches, backplane management. It doesn’t scale down gracefully.

Setup	Approx power
C7000 enclosure alone (no blades)	200–400W
C7000 + 1 blade	350–550W
C7000 + 3 blades	500–800W
ODEN alone (1U M3, Talos)	100–150W
HEIMDAL alone (Sun X4150, router)	150–200W
ODEN + HEIMDAL	250–350W

Two pizza boxes beat three blades in the enclosure on power. The overhead only amortises at 8+ populated slots. For a permanent minimal setup, the 1U rack servers win. For experiments where you want to run 8–16 nodes at once, ASGARD earns its place.

What each role actually needs

Role	RAM	Disk	Network	Limiting factor
Talos / K8s worker	32–64GB	1× OSD disk	1GbE fine	RAM — current blades too thin
OpenStack compute	32–64GB	local ephemeral	1GbE fine	RAM
OpenStack control	32GB+	small	1GbE fine	RAM
Slurm compute	as much as possible	fast scratch	1GbE mediocre	network
Ceph OSD	16–32GB	more / bigger disks	1GbE	disk count

The network note matters for Slurm: blade LOM connects to the enclosure switch backplane at 1GbE, not 10GbE. The switch has 10GbE uplinks going out, but blade-to-blade traffic inside the enclosure goes through the switch at 1GbE. For Talos and OpenStack this is fine. For MPI jobs exchanging large datasets between Slurm nodes it’s a real bottleneck — HPC wants InfiniBand, which the empty interconnect bays 5–8 could take (plus matching mezzanine cards in each blade), but that’s a separate cost. For learning Slurm, 1GbE is workable.

Current blade state

Most blades are underpowered for any of the roles above. CPUs are also unknown across all 16 slots — the OA web GUI reports CPU model and core count per blade and should be checked first. The E5-2600 v1 range runs from E5-2603 (4c, 80W) to E5-2690 (8c/16t, 135W), which matters significantly for role assignment.

Slot	RAM	Disk
BLD-001	4GB	2× 146GB SAS
BLD-002	14GB (mixed, odd count)	—
BLD-003	32GB	2× 300GB SAS
BLD-004	8GB	—
BLD-005	8GB	1× 146GB + 1× 300GB SAS
BLD-006	8GB	2× 300GB SAS
BLD-007	8GB	2× 900GB SAS
BLD-008	16GB	2× 300GB SAS
BLD-009	8GB	—
BLD-010	8GB	2× 300GB SAS
BLD-011	8GB	2× 300GB SAS
BLD-012	8GB	2× 300GB SAS
BLD-013	32GB	—
BLD-014	8GB	—
BLD-015	8GB	2× 300GB SAS
BLD-016	8GB	—

BLD-003 and BLD-013 are already at 32GB and are natural candidates for control-plane or master roles once CPUs are confirmed.

Suggested configuration from existing stock

Available spare hardware:

14× RAM-007 (8GB DDR3 1600MHz ECC Reg) — unassigned
2× HDD-004 (120GB SATA SSD) — spare
6× HDD-002 (146GB 10K SAS) — spare
Embedded P220i on each blade (can be set to JBOD/passthrough for Ceph)

“Fat” nodes × 2 — Talos control plane, OpenStack control, Slurm master: Add 4× RAM-007 to each blade. From a base of 8–16GB that gives ~40GB. Candidates: BLD-006 and BLD-010, both have 2× 300GB SAS for local storage. Costs 8 of 14 spare sticks. Install a spare 120GB SSD as boot disk in each.

“Medium” nodes × 3 — Talos workers, OpenStack compute, Slurm compute: Add 2× RAM-007 to each → 24GB from the 8GB base. Candidates: BLD-008 (already 16GB, gets to 32GB), BLD-011, BLD-012. All three have 300GB SAS for scratch or Ceph OSDs. Costs the remaining 6 spare sticks.

Rest — thin compute, storage expansion, or powered off: Leave at current RAM. BLD-007’s 900GB SAS pair is better used elsewhere (see below). BLD-003 and BLD-013 at 32GB can step up to fat-node role once CPUs are confirmed.

That leaves 5 blades properly kitted and 11 available for experiments or idle.

BL460c Gen8 DIMM rule: populate per-CPU symmetrically — pairs or quads per memory channel — for best throughput. Don’t mix odd counts.

Storage — what moves where

Pull the 900GB SAS drives from BLD-007 now. HDD-013 (HGST 900GB) and HDD-014 (Toshiba 900GB) are the two largest drives in the blade pool and they’re sitting in a blade that may end up as a thin compute worker. Move them into ODEN or LOKE as permanent Ceph OSDs. This immediately gives the always-on cluster substantially more storage than the current 120GB SSDs.

MIMIR (SYS-004, 15× 1TB SAS) is the Ceph expansion story for later. To connect it: install CTRL-006 (ServeRAID-8e, have 2 unplaced) into a server with a free PCIe slot, then cable it with a SFF-8470 → SFF-8088 cable (not currently owned, inexpensive). TOR is the natural host — it already has CTRL-003 in HBA mode and free PCIe slots. Not urgent, but the hardware is almost all there.

What	Goes to	When
900GB SAS ×2 from BLD-007	ODEN or LOKE, permanent Ceph OSDs	Now
120GB SSD ×2 spare	BLD fat node boot disks	Before Talos on blades
300GB SAS in blades	Local scratch or blade Ceph OSDs	During ASGARD experiments
MIMIR 15× 1TB SAS	TOR via CTRL-006, Ceph expansion	Later (needs cable)

Three things to do before blades can boot anything

Identify CPUs. Connect to the OA management port, open the web GUI, check CPU model per slot. Ten minutes. Everything else depends on this.
Network uplink. The blade switches in bays 1 and 2 have 4× RJ45 1GbE uplinks (ports 22–25). Run a patch cable from one to any available switch — MODI, MAGNI, whatever’s reachable from the cable box. That’s enough for blades to reach DHCP and iPXE.
RAM redistribution. Pull the 14 spare RAM-007 sticks and install into the chosen fat and medium nodes per the profile above.

The permanent vs experiment split

Always on (~300–400W total):
 HEIMDAL → OPNsense router, Sun X4150, ~150–200W
 ODEN → Talos, Minecraft + small services, ~100–150W
 LOKE → 2nd Talos node (needs RAM-007 × 8 + SSD boot), ~100–150W

Experiments (fire up, learn, power off):
 ASGARD → 3–16 blades for Slurm / OpenStack / larger Talos cluster
 TYR+TOR+FREJA → Proxmox cluster (M1 DDR2, temporary)

Once the Proxmox experiment wraps, TYR, TOR, and FREJA can be powered down permanently. If ASGARD blades eventually become the long-term compute platform, OPNsense can move to a VM on a blade at that point — but not before the blades are stable and trusted. Don’t consolidate the router onto experimental infrastructure.

PXE Booting with OPNSense + iPXE

Thu, 14 May 2026 00:00:00 +0000

How to configure OPNSense as a PXE boot server using its built-in DHCP and TFTP services, and how to write an iPXE boot menu that can chainload Talos Linux (or anything else).

OPNSense DHCP — Network Boot Fields

Services → ISC DHCPv4 → [LAN] → Network Booting

Field	Value
Enable network booting	✓
Next-server IP	`192.168.1.1` (OPNSense LAN address)
Default BIOS filename	`undionly.kpxe`
x86 UEFI (32-bit) filename	`ipxe.efi`
x64 UEFI/EBC (64-bit) filename	`ipxe.efi`
iPXE boot filename	`default.ipxe`

The DHCP server serves the correct boot file based on client architecture. BIOS clients get undionly.kpxe; UEFI clients get ipxe.efi. Both then chainload default.ipxe.

TFTP — Downloading the Boot Files

OPNSense runs a TFTP server rooted at /usr/local/tftp. SSH in and fetch the iPXE binaries:

fetch -o /usr/local/tftp/undionly.kpxe https://boot.ipxe.org/undionly.kpxe
fetch -o /usr/local/tftp/ipxe.efi https://boot.ipxe.org/ipxe.efi

iPXE Boot Script

Save as /usr/local/tftp/default.ipxe. This example has a boot menu with options for netboot.xyz, a Talos Omni boot, and a debug shell.

#!ipxe

dhcp
set menu-timeout 5000

:start
menu PXE Boot Menu
item --gap -- ----------------------------
item netboot Boot netboot.xyz
item talos Boot Talos (Omni)
item shell iPXE Shell
item --gap -- ----------------------------
choose target && goto ${target}

:netboot
chain http://boot.netboot.xyz || goto failed
goto start

:talos
echo Booting Talos via Omni...

set api https://<your-omni-instance>.omni.siderolabs.io
set token <join-token>
set event [<siderolink-ipv6>]:8090
set log tcp://[<siderolink-ipv6>]:8092

kernel tftp://${next-server}/talos/vmlinuz-omni \
 ima_template=ima-ng \
 ima_appraise=fix \
 ima_hash=sha512 \
 selinux=1 \
 consoleblank=0 \
 nvme_core.io_timeout=4294967295 \
 initrd=initramfs.xz \
 init_on_alloc=1 \
 slab_nomerge \
 pti=on \
 console=tty0 \
 console=ttyS0 \
 printk.devkmsg=on \
 talos.platform=metal \
 siderolink.api=${api}?jointoken=${token} \
 talos.events.sink=${event} \
 talos.logging.kernel=${log}

initrd tftp://${next-server}/talos/initramfs-omni.xz
boot || goto failed

:shell
shell

:failed
echo Boot failed, press Enter to continue...
read fake
goto start

The api, token, event, and log values come from the Omni console when you generate a join link.

Talos Kernel and Initramfs — Image Factory

The standard Talos release binaries do not include firmware for all hardware. Since Talos 1.6, several older NIC drivers (including Broadcom BNX2 / BCM5709) were removed from the mainline image and made available as extensions via the image factory.

Generate a custom image at factory.talos.dev with the extensions you need (e.g. siderolabs/bnx2), then download the PXE artifacts:

mkdir -p /usr/local/tftp/talos

fetch -o /usr/local/tftp/talos/vmlinuz-omni \
 "https://pxe.factory.talos.dev/image/<IMAGE_ID>/v1.10.1/kernel-amd64"

fetch -o /usr/local/tftp/talos/initramfs-omni.xz \
 "https://pxe.factory.talos.dev/image/<IMAGE_ID>/v1.10.1/initramfs-amd64.xz"

Replace <IMAGE_ID> with the schematic ID from the image factory, and adjust the version tag as needed.

Gotchas

UEFI boot and NIC memory limits — ipxe.efi can be too large to fit in the NIC’s PXE memory buffer on some older hardware. If the UEFI chain fails silently, switch to BIOS/legacy mode and use undionly.kpxe instead.

DHCP options 66/67 conflict — If you have previously set DHCP options 66 (next-server) and 67 (boot file) as raw additional options, remove them. OPNSense’s built-in network boot fields handle this; having both causes conflicts.

BIOS boot order after first boot — Talos writes its own bootloader on first boot. If the BIOS is set to PXE as the primary device, the machine will fall back to the PXE menu on every subsequent reboot. Set disk as the primary boot device once the cluster is up.

Talos Linux + Omni

Thu, 14 May 2026 00:00:00 +0000

Talos Linux is an immutable, minimal operating system designed specifically for running Kubernetes. There is no shell, no SSH, no package manager. The entire OS is read-only and managed via a gRPC API (talosctl). Node configuration is declarative YAML applied over the API; changes that require a reboot take effect on the next boot.

The tradeoff is rigidity for operational simplicity. You cannot log into a Talos node and fix something by hand. In return, nodes are deterministic, reproducible, and there is no configuration drift.

Comparison to other installs

Method	OS	Config	Mutable
kubeadm	Ubuntu / RHEL / etc	Manual + scripts	Yes
k3s	Any Linux	Minimal	Yes
Talos	Talos Linux	Declarative API	No

k3s and kubeadm give you more flexibility and a familiar Linux environment. Talos is the right choice when you want the cluster nodes to behave like appliances — provisioned, never touched.

Omni

Omni is a cluster management platform by Sidero Labs built on top of Talos. It handles:

Node registration (nodes boot and phone home to the Omni API)
Cluster creation and machine assignment
Kubernetes upgrades (one action in the UI)
talosctl and kubeconfig access via the Omni CLI

Nodes register via a join token embedded in the kernel command line at PXE boot time. The cluster runs on your hardware; Omni only manages the control plane.

Hobby tier: 10 nodes, non-commercial use, free. Sidero Labs also offers a self-hosted version.

Image Factory

factory.talos.dev generates custom Talos images with hardware extensions included. Notable extensions:

siderolabs/bnx2 — Broadcom NetXtreme II (BCM5708/BCM5709) NIC firmware, required on some enterprise hardware (IBM x3550 M3, HP Gen 6/7 blades)
siderolabs/intel-ucode — Intel microcode updates
siderolabs/nvidia-* — NVIDIA GPU support

The factory produces both ISO and PXE artifacts (kernel + initramfs). See the OPNSense + iPXE reference for how to serve these over TFTP.

Supporting Sidero Labs

Talos and Omni are built by Sidero Labs — good people doing good work. I sponsor them via GitHub Sponsors at the fanboi tier.

Relevant links

Talos Linux in the homelab via Omni

Thu, 14 May 2026 00:00:00 +0000

Getting Talos Linux running in the homelab via PXE boot and Omni — starting with ODEN (SYS-005), an IBM System x3550 M3. The full OPNSense + iPXE configuration lives in the reference note; this covers what actually happened, in order.

Setup

Hardware: ODEN (SYS-005) — IBM x3550 M3, Broadcom BNX2 NICs (BCM5709)
Network: OPNSense router on LAN; ODEN connected via one NIC (start with one — removes variables)
Target: Single-node Talos cluster registered in Omni

Step 1 — OPNSense DHCP and TFTP

Enable network booting on the LAN DHCP server and download the iPXE binaries to the TFTP root. Full field values in the iPXE reference note.

One thing to check first: if you previously set DHCP options 66 and 67 as raw additional options, remove them. OPNSense’s built-in network boot fields do the same job and having both causes conflicts.

Step 2 — iPXE boot script

Write default.ipxe to /usr/local/tftp/. Include a boot menu with at minimum a Talos option and a shell fallback — the shell is genuinely useful when something fails and you need to debug from the boot prompt. Full script in the reference note.

The Talos entry in the menu needs the Omni join token from your Omni console. Generate a join link in Omni; it provides the API endpoint, token, and SideroLink addresses.

Step 3 — Talos kernel and initramfs

The standard Talos release binaries do not include BNX2 firmware. Since around Talos 1.6 those drivers are available as extensions but not in the mainline image. Without them, the node boots, fails to initialise the NIC, and produces can't load firmware bnx2 errors — everything else looks fine until you notice the node never gets an IP and never appears in Omni.

Fix: generate a custom image at factory.talos.dev with the siderolabs/bnx2 extension included, then download the PXE kernel and initramfs from the factory URL. Commands in the reference note.

Step 4 — First boot

Go into BIOS and set the boot device to PXE. On the M3, UEFI boot with ipxe.efi fails silently — the image is too large for the NIC’s PXE memory buffer. Switch to legacy/BIOS mode and use undionly.kpxe instead.

The machine takes a while to POST and boot. This is normal for old enterprise hardware. It is also why demos typically use virtual machines.

Step 5 — Static IP

After the BNX2 fix the node boots Talos successfully but still does not appear in Omni. The DHCP assignment for the node is not being picked up during early boot. Workaround: add a static IP via kernel params in the iPXE script:

ip=192.168.1.171::192.168.1.1:255.255.255.0::eth0:off

Add this to the kernel line in the Talos iPXE entry. The format is ip=<client-ip>::<gateway>:<netmask>::<iface>:off.

Step 6 — Omni registration

With a working NIC and an IP, the node contacts the Omni API using the join token. It appears in the Omni console as an unallocated machine. Create a cluster, assign the machine, and let Omni configure it. The initial cluster bootstrap takes a few minutes.

Step 7 — Fix the BIOS boot order

After the cluster is up, change the BIOS boot order so the disk is first. If PXE remains the primary boot device, every reboot drops the machine back to the iPXE menu instead of booting the installed Talos. Discovered on first reboot. Worth noting it here so you don’t make the same trip to the garage.

Upgrade

Omni makes single-node upgrades straightforward: open the cluster in the Omni console, select a new Talos version, apply. The node reboots once. Single-node means the cluster has downtime during the reboot; that is expected. Nothing else to do.

Result

Single-node Kubernetes cluster running on ODEN, managed via Omni. kubectl and talosctl access via the Omni CLI. Next experiment: Rook + Ceph for persistent storage.

Kubernetes Across the Stack

Mon, 16 Mar 2026 00:00:00 +0000

A documented comparison of running Kubernetes across every major hosting model — cloud managed, self-managed on cloud, private cloud, and bare metal at home. The goal is a honest, practical reference for each environment: what it costs you in time and money, where the rough edges are, and how the networking story differs between them.

The thread running through all of it is Talos Linux — an immutable, API-driven OS built specifically for Kubernetes. No SSH, no shell, no config drift. The same OS everywhere means the operational model stays consistent regardless of what is running underneath.

Environment	Approach
OpenStack — Cleura	Talos & Terraform	draft exists
OpenStack — Cleura	Talos, with Omni	maybe ?
OpenStack — ElastX	Talos & Terraform	draft exists
OpenStack — ElastX	Talos, with Omni	maybe ?
Homelab — bare metal	Talos + Pixieboot + Omni	draft exists
Homelab — bare metal	Talos + Pixieboot without Omni	maybe ?
Homelab — OpenStack	OpenStack on bare metal, Talos running on top	(stretch)
Homelab — OpenStack	Talos on bare metal, OpenStack inside cluster	(stretch)
AWS	Talos on EC2	(stretch)
Azure	Talos on VMs	(stretch)
GCP	Talos on Compute Engine	(stretch)

Stretch goals

AWS, Azure, GCP — same Talos approach, different underlying infrastructure. Interesting eventually, but not the priority.

Omni

Omni is Sidero’s managed control plane for Talos clusters — worth documenting both with and without it. Without Omni gives you the full picture of what Talos management looks like manually; with Omni shows what the managed layer buys you.

Homelab provisioning

Nodes provisioned via Pixieboot — no USB sticks, no manual installations. A node powers on, boots from the network, and registers. The goal is a fully reproducible cluster from scratch with minimal human steps.

Scope

Cluster provisioning and bootstrap for each environment
Networking — CNI choices, ingress, cross-cluster connectivity
Storage — what you get managed vs what you have to bring yourself
Operational differences — upgrades, node management, observability
Cost and trade-off summary across environments

Making it usable

Getting a cluster running is the easy part. Making it usable is where environments diverge. Each environment needs an answer for ingress, DNS, and storage — and the answer varies significantly depending on what the underlying platform provides.

On managed cloud you can lean on load balancers and block storage from the provider. On OpenStack you have those options if the provider exposes them. On bare metal at home you are on your own — MetalLB or similar for load balancer IPs, a local DNS solution, and either local storage or something like Rook/Ceph. Same Kubernetes, very different operational story underneath.

Notes exist in various states — pulling them together, testing, and documenting properly is the work.