Homelab on Backend Engineering Strategy Tools

BIFROST — Raspberry Pi jump node

Fri, 05 Jun 2026 00:00:00 +0000

The homelab needed a permanent always-on entry point — something low power, always reachable, a stable first hop. A first-gen Raspberry Pi in the rack fills that role. BIFROST.

Hardware

Raspberry Pi 1 Model B running Raspbian, mounted in the rack with a 3D printed 1U mount. Draws under 2W at idle. Nothing runs on it except sshd.

How it works

ssh -p 22222 user@bifrost.mjnet.info

The chain:

bifrost.mjnet.info — Route53 CNAME pointing to router.mjnet.info
router.mjnet.info — HEIMDAL (SYS-009), kept current via DDNS
OPNsense port forward: external TCP 22222 → Pi:22
OPNsense DNS override: bifrost.mjnet.info → Pi’s internal IP

The DNS override means the same hostname resolves to the internal IP when used inside the network — no split config needed in ~/.ssh/config.

OPNsense config

Port forward (Firewall → NAT → Port Forward):

Interface: WAN
Protocol: TCP
Destination port: 22222
Redirect target: Pi internal IP, port 22

DNS override (Services → Unbound DNS → Host Overrides):

Host: bifrost
Domain: mjnet.info
IP: Pi internal IP

SSH config

Add to ~/.ssh/config on any client:

Host bifrost
 HostName bifrost.mjnet.info
 Port 22222
 User pi

Then ssh bifrost from anywhere.

If a more robust solution becomes necessary later (no open ports, survives CGNAT), the options doc covers Tailscale, Cloudflare Tunnel, and WireGuard.

Rack Support Brace — Programmatic 3D Print

Tue, 02 Jun 2026 00:00:00 +0000

220×150×8mm support brace for the server rack. 40 through-holes in an alternating-spacing grid, B.E.S.T engraved into the top face, exported to STL — all generated by a Python script running in Blender. No manual modelling.

The same script that builds the geometry now also sets up cameras, runs EEVEE renders from four angles, and saves PNGs alongside the STL. The renders above are the direct output.

Downloads

↓rack_support_brace.py ↓rack_support_brace_220x150.stl

The script

Flip EXPORT_STL / EXPORT_RENDER at the top of the file, point EXPORT_DIR at your folder, press Alt+P in the Blender Script Editor.

The config block at the top of the file is the only place dimensions live:

PLATE_X = 220.0 # mm
PLATE_Y = 150.0 # mm
HOLE_DIAM = 7.0 # mm
N_COLS = 10
ROW_SPACING = 20.0 # mm

Iterating means editing a constant and re-running — not touching a model.

See Blender Python for 3D Printing for the general pattern behind this.

Step 1 — Render from script ✓

Done. Script builds geometry → exports STL → places cameras at four angles (top, front, side, iso) → EEVEE renders → PNGs saved to EXPORT_DIR/renders/.

Pipeline next steps (headless render, CI/CD, AI feedback loop) are tracked in the Procedural Mesh project.

The Printed Part

Finished rack support brace installed in the rack.

ASGARD — the blade cluster

Fri, 15 May 2026 00:00:00 +0000

ASGARD (SYS-007) is the HP BladeSystem C7000 with 16× BL460c Gen8 blades. The reason to use it is profile switching: boot a blade as a Slurm compute node, run the experiment, reimage it as a Talos worker, run the next one. The same iPXE boot menu already set up for ODEN works here — the C7000 Onboard Administrator lets you configure boot order per blade slot, so switching roles is a BIOS setting and a PXE entry, not a reinstall.

Power reality

Before committing to blades as the permanent always-on platform, it’s worth being honest about the enclosure overhead. The C7000 has fixed costs regardless of how many blades are populated: 10 fans, dual OA modules, 2 interconnect switches, backplane management. It doesn’t scale down gracefully.

Setup	Approx power
C7000 enclosure alone (no blades)	200–400W
C7000 + 1 blade	350–550W
C7000 + 3 blades	500–800W
ODEN alone (1U M3, Talos)	100–150W
HEIMDAL alone (Sun X4150, router)	150–200W
ODEN + HEIMDAL	250–350W

Two pizza boxes beat three blades in the enclosure on power. The overhead only amortises at 8+ populated slots. For a permanent minimal setup, the 1U rack servers win. For experiments where you want to run 8–16 nodes at once, ASGARD earns its place.

What each role actually needs

Role	RAM	Disk	Network	Limiting factor
Talos / K8s worker	32–64GB	1× OSD disk	1GbE fine	RAM — current blades too thin
OpenStack compute	32–64GB	local ephemeral	1GbE fine	RAM
OpenStack control	32GB+	small	1GbE fine	RAM
Slurm compute	as much as possible	fast scratch	1GbE mediocre	network
Ceph OSD	16–32GB	more / bigger disks	1GbE	disk count

The network note matters for Slurm: blade LOM connects to the enclosure switch backplane at 1GbE, not 10GbE. The switch has 10GbE uplinks going out, but blade-to-blade traffic inside the enclosure goes through the switch at 1GbE. For Talos and OpenStack this is fine. For MPI jobs exchanging large datasets between Slurm nodes it’s a real bottleneck — HPC wants InfiniBand, which the empty interconnect bays 5–8 could take (plus matching mezzanine cards in each blade), but that’s a separate cost. For learning Slurm, 1GbE is workable.

Current blade state

Most blades are underpowered for any of the roles above. CPUs are also unknown across all 16 slots — the OA web GUI reports CPU model and core count per blade and should be checked first. The E5-2600 v1 range runs from E5-2603 (4c, 80W) to E5-2690 (8c/16t, 135W), which matters significantly for role assignment.

Slot	RAM	Disk
BLD-001	4GB	2× 146GB SAS
BLD-002	14GB (mixed, odd count)	—
BLD-003	32GB	2× 300GB SAS
BLD-004	8GB	—
BLD-005	8GB	1× 146GB + 1× 300GB SAS
BLD-006	8GB	2× 300GB SAS
BLD-007	8GB	2× 900GB SAS
BLD-008	16GB	2× 300GB SAS
BLD-009	8GB	—
BLD-010	8GB	2× 300GB SAS
BLD-011	8GB	2× 300GB SAS
BLD-012	8GB	2× 300GB SAS
BLD-013	32GB	—
BLD-014	8GB	—
BLD-015	8GB	2× 300GB SAS
BLD-016	8GB	—

BLD-003 and BLD-013 are already at 32GB and are natural candidates for control-plane or master roles once CPUs are confirmed.

Suggested configuration from existing stock

Available spare hardware:

14× RAM-007 (8GB DDR3 1600MHz ECC Reg) — unassigned
2× HDD-004 (120GB SATA SSD) — spare
6× HDD-002 (146GB 10K SAS) — spare
Embedded P220i on each blade (can be set to JBOD/passthrough for Ceph)

“Fat” nodes × 2 — Talos control plane, OpenStack control, Slurm master: Add 4× RAM-007 to each blade. From a base of 8–16GB that gives ~40GB. Candidates: BLD-006 and BLD-010, both have 2× 300GB SAS for local storage. Costs 8 of 14 spare sticks. Install a spare 120GB SSD as boot disk in each.

“Medium” nodes × 3 — Talos workers, OpenStack compute, Slurm compute: Add 2× RAM-007 to each → 24GB from the 8GB base. Candidates: BLD-008 (already 16GB, gets to 32GB), BLD-011, BLD-012. All three have 300GB SAS for scratch or Ceph OSDs. Costs the remaining 6 spare sticks.

Rest — thin compute, storage expansion, or powered off: Leave at current RAM. BLD-007’s 900GB SAS pair is better used elsewhere (see below). BLD-003 and BLD-013 at 32GB can step up to fat-node role once CPUs are confirmed.

That leaves 5 blades properly kitted and 11 available for experiments or idle.

BL460c Gen8 DIMM rule: populate per-CPU symmetrically — pairs or quads per memory channel — for best throughput. Don’t mix odd counts.

Storage — what moves where

Pull the 900GB SAS drives from BLD-007 now. HDD-013 (HGST 900GB) and HDD-014 (Toshiba 900GB) are the two largest drives in the blade pool and they’re sitting in a blade that may end up as a thin compute worker. Move them into ODEN or LOKE as permanent Ceph OSDs. This immediately gives the always-on cluster substantially more storage than the current 120GB SSDs.

MIMIR (SYS-004, 15× 1TB SAS) is the Ceph expansion story for later. To connect it: install CTRL-006 (ServeRAID-8e, have 2 unplaced) into a server with a free PCIe slot, then cable it with a SFF-8470 → SFF-8088 cable (not currently owned, inexpensive). TOR is the natural host — it already has CTRL-003 in HBA mode and free PCIe slots. Not urgent, but the hardware is almost all there.

What	Goes to	When
900GB SAS ×2 from BLD-007	ODEN or LOKE, permanent Ceph OSDs	Now
120GB SSD ×2 spare	BLD fat node boot disks	Before Talos on blades
300GB SAS in blades	Local scratch or blade Ceph OSDs	During ASGARD experiments
MIMIR 15× 1TB SAS	TOR via CTRL-006, Ceph expansion	Later (needs cable)

Three things to do before blades can boot anything

Identify CPUs. Connect to the OA management port, open the web GUI, check CPU model per slot. Ten minutes. Everything else depends on this.
Network uplink. The blade switches in bays 1 and 2 have 4× RJ45 1GbE uplinks (ports 22–25). Run a patch cable from one to any available switch — MODI, MAGNI, whatever’s reachable from the cable box. That’s enough for blades to reach DHCP and iPXE.
RAM redistribution. Pull the 14 spare RAM-007 sticks and install into the chosen fat and medium nodes per the profile above.

The permanent vs experiment split

Always on (~300–400W total):
 HEIMDAL → OPNsense router, Sun X4150, ~150–200W
 ODEN → Talos, Minecraft + small services, ~100–150W
 LOKE → 2nd Talos node (needs RAM-007 × 8 + SSD boot), ~100–150W

Experiments (fire up, learn, power off):
 ASGARD → 3–16 blades for Slurm / OpenStack / larger Talos cluster
 TYR+TOR+FREJA → Proxmox cluster (M1 DDR2, temporary)

Once the Proxmox experiment wraps, TYR, TOR, and FREJA can be powered down permanently. If ASGARD blades eventually become the long-term compute platform, OPNsense can move to a VM on a blade at that point — but not before the blades are stable and trusted. Don’t consolidate the router onto experimental infrastructure.

OPNsense in the homelab

Thu, 14 May 2026 00:00:00 +0000

OPNsense running on the Sun Fire X4150 — perimeter gateway for both the homelab and the home network.

Current state: inherited setup, not clean, not properly documented. It works, but it was not built with intention. A redo is on the todo list.

Dual role is intentional and will stay that way — OPNsense on the Sun Fire handles both the homelab and the house. One box, one gateway.

The Sun Fire X4150 handles it without breaking a sweat. Old, but solid.

TODO: Clean reinstall

Fresh OPNsense install on the Sun Fire
Rename from router.mjnet.info → heimdal.mjnet.info
Document the full configuration: firewall rules, DHCP, DNS (Unbound), TFTP/PXE, BGP peering with VyOS

On the hostname rename: making the router’s hostname publicly resolvable is a mild information disclosure — it signals which host is the perimeter device. In practice the IP is what matters, and if it’s already reachable the name adds little. Still worth being deliberate about what resolves publicly.

Intended role

ISP/WAN → OPNsense (heimdal.mjnet.info) → LAN switch → rack

Perimeter firewall and NAT gateway
DHCP for the LAN
DNS via Unbound
TFTP/PXE for bare-metal provisioning
eBGP peer upstream of VyOS (future — see VyOS + BGP)

Proxmox Cluster in the homelab

Thu, 14 May 2026 00:00:00 +0000

Getting a three-node Proxmox VE cluster running in the homelab.

The goal is a shared virtualization platform for running VMs and LXC containers across the rack. Also, a good excuse to kick the tires on Proxmox itself so, naturally, let’s needlessly complicate things with some self-imposed constraints:

run it clustered
don’t use any hardware already earmarked for other projects

Hardware

I am going try and use three IBM rack servers from the inventory.

Asset ID	Hostname	Model	Form Factor	RAM	CPU
SYS-001	FREJA	IBM System x3550 M1 (7978)	1U	24GB	single
SYS-002	TYR	IBM System x3650 M1 (7979)	2U	64GB	dual
SYS-003	TOR	IBM System x3650 M1 (7979)	2U	64GB	dual

Three nodes satisfies Corosync quorum without needing a qdevice — losing one node still leaves a majority.

Installation

In progress.

Cluster formation

In progress.

Rook + Ceph on ODEN

Thu, 14 May 2026 00:00:00 +0000

Attempting to add persistent block storage to the ODEN single-node Talos cluster using Rook and Ceph. This did not fully succeed — the setup reached the point of a bound PVC and a working write test, but the cluster was not left in a clean stable state. Notes are here for completeness.

This builds on the Talos cluster setup on ODEN.

Hardware

ODEN has five storage devices:

Device	Type	Size	Role
`/dev/sdb`	Kingston SA400S3 SSD (SATA)	120 GB	Boot disk — leave alone
`/dev/nvme0n1`	Samsung 970 EVO NVMe	500 GB	OSD
`/dev/sdc`	Kingston SA400S3 SSD (SATA)	120 GB	OSD
`/dev/sdd`	Kingston SA400S3 SSD (SATA)	120 GB	OSD
`/dev/sde`	Kingston SA400S3 SSD (SATA)	120 GB	OSD

Do not add /dev/sdb to Ceph. It is the boot disk.

Step 1 — Install the Rook operator

kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/common.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/operator.yaml

Wait for the operator pod to be running in rook-ceph namespace before continuing.

Step 2 — CephCluster (single-node)

Single-node requires allowMultiplePerNode: true and explicit disk selection. The cluster-test example from the Rook repo is a reasonable starting point:

storage:
 useAllNodes: false
 nodes:
 - name: "192.168.1.171"
 devices:
 - name: "nvme0n1"
 - name: "sdc"
 - name: "sdd"
 - name: "sde"

Reference: cluster-test.yaml

Step 3 — CephBlockPool and StorageClass

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
 name: replicapool
 namespace: rook-ceph
spec:
 replicated:
 size: 1

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
 clusterID: rook-ceph
 pool: replicapool
 imageFormat: "2"
 imageFeatures: layering
reclaimPolicy: Delete

Step 4 — PVC test

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: test-pvc
spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: rook-ceph-block
 resources:
 requests:
 storage: 10Gi

PVC reached Bound. A BusyBox pod mounting it could write to /mnt. The Ceph dashboard (kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 7000:7000) showed OSDs active and the pool present.

What did not work

The cluster ran but was not left stable. Single-node Ceph produces health warnings by design (no redundancy, no failure domain separation). More importantly, the setup was not revisited after initial testing and there are unresolved questions about:

CSI driver behaviour on Talos (Talos has specific requirements for CSI socket paths)
Whether the dashboard warnings were cosmetic or indicated real issues
Long-term stability under actual workloads

This is left as a draft until there is time to run it properly — ideally on more than one node.

Talos Linux in the homelab via Omni

Thu, 14 May 2026 00:00:00 +0000

Getting Talos Linux running in the homelab via PXE boot and Omni — starting with ODEN (SYS-005), an IBM System x3550 M3. The full OPNSense + iPXE configuration lives in the reference note; this covers what actually happened, in order.

Setup

Hardware: ODEN (SYS-005) — IBM x3550 M3, Broadcom BNX2 NICs (BCM5709)
Network: OPNSense router on LAN; ODEN connected via one NIC (start with one — removes variables)
Target: Single-node Talos cluster registered in Omni

Step 1 — OPNSense DHCP and TFTP

Enable network booting on the LAN DHCP server and download the iPXE binaries to the TFTP root. Full field values in the iPXE reference note.

One thing to check first: if you previously set DHCP options 66 and 67 as raw additional options, remove them. OPNSense’s built-in network boot fields do the same job and having both causes conflicts.

Step 2 — iPXE boot script

Write default.ipxe to /usr/local/tftp/. Include a boot menu with at minimum a Talos option and a shell fallback — the shell is genuinely useful when something fails and you need to debug from the boot prompt. Full script in the reference note.

The Talos entry in the menu needs the Omni join token from your Omni console. Generate a join link in Omni; it provides the API endpoint, token, and SideroLink addresses.

Step 3 — Talos kernel and initramfs

The standard Talos release binaries do not include BNX2 firmware. Since around Talos 1.6 those drivers are available as extensions but not in the mainline image. Without them, the node boots, fails to initialise the NIC, and produces can't load firmware bnx2 errors — everything else looks fine until you notice the node never gets an IP and never appears in Omni.

Fix: generate a custom image at factory.talos.dev with the siderolabs/bnx2 extension included, then download the PXE kernel and initramfs from the factory URL. Commands in the reference note.

Step 4 — First boot

Go into BIOS and set the boot device to PXE. On the M3, UEFI boot with ipxe.efi fails silently — the image is too large for the NIC’s PXE memory buffer. Switch to legacy/BIOS mode and use undionly.kpxe instead.

The machine takes a while to POST and boot. This is normal for old enterprise hardware. It is also why demos typically use virtual machines.

Step 5 — Static IP

After the BNX2 fix the node boots Talos successfully but still does not appear in Omni. The DHCP assignment for the node is not being picked up during early boot. Workaround: add a static IP via kernel params in the iPXE script:

ip=192.168.1.171::192.168.1.1:255.255.255.0::eth0:off

Add this to the kernel line in the Talos iPXE entry. The format is ip=<client-ip>::<gateway>:<netmask>::<iface>:off.

Step 6 — Omni registration

With a working NIC and an IP, the node contacts the Omni API using the join token. It appears in the Omni console as an unallocated machine. Create a cluster, assign the machine, and let Omni configure it. The initial cluster bootstrap takes a few minutes.

Step 7 — Fix the BIOS boot order

After the cluster is up, change the BIOS boot order so the disk is first. If PXE remains the primary boot device, every reboot drops the machine back to the iPXE menu instead of booting the installed Talos. Discovered on first reboot. Worth noting it here so you don’t make the same trip to the garage.

Upgrade

Omni makes single-node upgrades straightforward: open the cluster in the Omni console, select a new Talos version, apply. The node reboots once. Single-node means the cluster has downtime during the reboot; that is expected. Nothing else to do.

Result

Single-node Kubernetes cluster running on ODEN, managed via Omni. kubectl and talosctl access via the Omni CLI. Next experiment: Rook + Ceph for persistent storage.

3D Printed Rack Parts

Sun, 12 Apr 2026 00:00:00 +0000

Filling gaps in the rack build with printed parts — ears, blanks, and a modular tray system. Mix of sourced models from Printables and geometry scripted in Blender Python.

Sourced Models

Model	What it is	Status
1U Universal Rack Ears	Rack ears for gear that ships without them	Printed
1U–4U Spacer / Blank Panel	Blank panels to fill empty rack units	Printed
RackMod 1U Slide-A	Modular 1U tray system with slide-in modules	Testing
19" 1U/2U/3U Cover Plates	Solid cover/blanking plates for 19" rack	Queued
1U Rack Cable Ears	Rack ears with integrated cable management	Queued
Raspberry Pi 1U rack mount	1U mount for RPi in 19" rack (BIFROST)	Printed
Zip Tie Clip 45mm T-Slot	Cable management clips for 45mm extrusion	Queued

Scripted Parts

Geometry generated by a Python script running inside Blender rather than modeled by hand. The immediate need was a support brace for the rack — a 220×150×8mm plate with 40 through-holes in a specific alternating-spacing pattern, the B.E.S.T label engraved into the top face, exported directly to STL.

Writing it in Python means the layout lives in a config block at the top of the file. Changing hole count, plate dimensions, or row spacing is a constant, not a modeling operation.

Rack Support Brace — 220×150mm plate, 10×4 hole grid, engraved text, automated renders | script

More on the approach: Blender Python for 3D printing

Print setup: 3D Printing — Garage