Ceph on Backend Engineering Strategy Tools

Ceph

Thu, 14 May 2026 00:00:00 +0000

Ceph is an open-source distributed storage platform providing object, block, and file storage in a single unified system. It runs across multiple nodes and has no single point of failure.

The core idea: data is not stored on specific disks on specific nodes. Instead, the CRUSH algorithm distributes data across all available OSDs (Object Storage Daemons) based on a placement map. Add nodes and the cluster rebalances automatically. Lose a node and Ceph re-replicates from surviving copies without operator intervention.

Storage types

Type	Interface	Typical use
Block (RBD)	Kernel block device / iSCSI	Kubernetes PVCs, VM disks
Object (RGW)	S3-compatible API	Backups, artifacts, media
File (CephFS)	POSIX filesystem / NFS	Shared filesystems, home dirs

For Kubernetes workloads, RBD block storage via a StorageClass is the common path.

Components

MON (Monitor) — maintains the cluster map; quorum-based, needs an odd number (typically 3 or 5). Not a data path.

OSD (Object Storage Daemon) — one per disk; handles actual data reads/writes and replication.

MGR (Manager) — collects metrics, hosts the dashboard, runs modules (balancer, alertmanager, etc.).

MDS (Metadata Server) — only required for CephFS; manages the filesystem namespace.

Single-node constraint

A single-node Ceph cluster can be made to run (allowMultiplePerNode: true in Rook, replication size: 1), but it provides no actual redundancy. There is nothing to replicate to. This is fine for testing concepts; it is not a valid storage setup for anything you care about.

Ceph documentation
Rook — Kubernetes operator that manages Ceph clusters inside K8s
Proxmox — Ceph is a native storage backend in Proxmox clusters
Rook + Ceph in the homelab

Rook

Thu, 14 May 2026 00:00:00 +0000

Rook is a Kubernetes operator that deploys and manages storage systems — primarily Ceph — as native Kubernetes resources. The distinction: Ceph is the storage system; Rook is the Kubernetes wiring around it.

Without Rook you would run Ceph manually (or via cephadm) and then configure the Kubernetes CSI driver separately. Rook collapses that into CRDs and handles the full lifecycle: deployment, configuration, expansion, upgrades, and failure recovery.

How it works

Rook introduces several CRDs:

CephCluster — declares the cluster: which nodes, which disks to use as OSDs, replication settings.

CephBlockPool — defines a Ceph pool (replication factor, failure domain). Maps to an RBD pool.

StorageClass — references a CephBlockPool and enables dynamic PVC provisioning. Kubernetes workloads request storage; Rook/Ceph fulfils it.

CephFilesystem — deploys CephFS + MDS for POSIX shared filesystem access.

CephObjectStore — deploys the Ceph RGW S3-compatible object storage gateway.

Typical install sequence

kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/common.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/operator.yaml

Then apply a CephCluster manifest declaring your storage topology, followed by CephBlockPool and StorageClass for PVC support.

Single-node considerations

A single-node setup requires allowMultiplePerNode: true in the CephCluster spec (MONs, MGR, and OSDs all land on the same node). Replication size must be set to 1 — there is nowhere else to replicate. This works for experimentation; it is not a production configuration. See Ceph for details on the replication model.

Rook documentation
Ceph — the underlying storage system
Rook + Ceph in the homelab

Rook + Ceph on ODEN

Thu, 14 May 2026 00:00:00 +0000

Attempting to add persistent block storage to the ODEN single-node Talos cluster using Rook and Ceph. This did not fully succeed — the setup reached the point of a bound PVC and a working write test, but the cluster was not left in a clean stable state. Notes are here for completeness.

This builds on the Talos cluster setup on ODEN.

Hardware

ODEN has five storage devices:

Device	Type	Size	Role
`/dev/sdb`	Kingston SA400S3 SSD (SATA)	120 GB	Boot disk — leave alone
`/dev/nvme0n1`	Samsung 970 EVO NVMe	500 GB	OSD
`/dev/sdc`	Kingston SA400S3 SSD (SATA)	120 GB	OSD
`/dev/sdd`	Kingston SA400S3 SSD (SATA)	120 GB	OSD
`/dev/sde`	Kingston SA400S3 SSD (SATA)	120 GB	OSD

Do not add /dev/sdb to Ceph. It is the boot disk.

Step 1 — Install the Rook operator

kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/common.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/tags/v1.17.9/deploy/examples/operator.yaml

Wait for the operator pod to be running in rook-ceph namespace before continuing.

Step 2 — CephCluster (single-node)

Single-node requires allowMultiplePerNode: true and explicit disk selection. The cluster-test example from the Rook repo is a reasonable starting point:

storage:
 useAllNodes: false
 nodes:
 - name: "192.168.1.171"
 devices:
 - name: "nvme0n1"
 - name: "sdc"
 - name: "sdd"
 - name: "sde"

Reference: cluster-test.yaml

Step 3 — CephBlockPool and StorageClass

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
 name: replicapool
 namespace: rook-ceph
spec:
 replicated:
 size: 1

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
 clusterID: rook-ceph
 pool: replicapool
 imageFormat: "2"
 imageFeatures: layering
reclaimPolicy: Delete

Step 4 — PVC test

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: test-pvc
spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: rook-ceph-block
 resources:
 requests:
 storage: 10Gi

PVC reached Bound. A BusyBox pod mounting it could write to /mnt. The Ceph dashboard (kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 7000:7000) showed OSDs active and the pool present.

What did not work

The cluster ran but was not left stable. Single-node Ceph produces health warnings by design (no redundancy, no failure domain separation). More importantly, the setup was not revisited after initial testing and there are unresolved questions about:

CSI driver behaviour on Talos (Talos has specific requirements for CSI socket paths)
Whether the dashboard warnings were cosmetic or indicated real issues
Long-term stability under actual workloads

This is left as a draft until there is time to run it properly — ideally on more than one node.

OpenShift Data Foundation

Mon, 01 Jan 2024 00:00:00 +0000

OpenShift Data Foundation (ODF) is Red Hat’s enterprise Kubernetes storage platform, built on Ceph orchestrated by Rook. Where Rook-Ceph is the open source upstream, ODF packages it with an operator, a validated configuration, enterprise support, and integration with the OpenShift console. It provides block (RBD), file (CephFS), and object (S3-compatible via Ceph RGW) storage as Kubernetes StorageClasses on the same hardware.

What it provides

Three storage modes from one cluster:

Mode	StorageClass	Use case
Block (RBD)	`ocs-storagecluster-ceph-rbd`	Databases, stateful apps needing a single-writer disk
File (CephFS)	`ocs-storagecluster-cephfs`	Shared filesystems, multiple pods reading/writing the same volume
Object	S3-compatible endpoint	Buckets via `ObjectBucketClaim`, backup targets, artifact storage

Installation

ODF installs via the ODF operator from OperatorHub. The operator creates a StorageCluster CR that drives the Ceph deployment:

apiVersion: ocs.openshift.io/v1
kind: StorageCluster
metadata:
 name: ocs-storagecluster
 namespace: openshift-storage
spec:
 storageDeviceSets:
 - name: ocs-deviceset
 count: 1
 replica: 3
 dataPVCTemplate:
 spec:
 storageClassName: local-storage
 volumeMode: Block
 resources:
 requests:
 storage: 1Ti

Requires at minimum three nodes with dedicated block devices. The operator handles Ceph cluster formation, monitors, MGRs, and OSDs.

vs Rook-Ceph

ODF IS Rook-Ceph under the hood. The difference is packaging and support: ODF is tested and supported on OpenShift, includes the NooBaa multi-cloud gateway for object storage federation, and integrates with the OpenShift UI. For self-managed Kubernetes outside OpenShift, raw Rook-Ceph is the equivalent path.

Ceph on Backend Engineering Strategy Tools

Ceph

Storage types

Components

Single-node constraint

Related

Rook

How it works

Typical install sequence

Single-node considerations

Related

Rook + Ceph on ODEN

Hardware

Step 1 — Install the Rook operator

Step 2 — CephCluster (single-node)

Step 3 — CephBlockPool and StorageClass

Step 4 — PVC test

What did not work

OpenShift Data Foundation

What it provides

Installation

vs Rook-Ceph

Resources