Ceph

Thu, 14 May 2026 00:00:00 +0000

Ceph is an open-source distributed storage platform providing object, block, and file storage in a single unified system. It runs across multiple nodes and has no single point of failure.

The core idea: data is not stored on specific disks on specific nodes. Instead, the CRUSH algorithm distributes data across all available OSDs (Object Storage Daemons) based on a placement map. Add nodes and the cluster rebalances automatically. Lose a node and Ceph re-replicates from surviving copies without operator intervention.

Storage types

Type	Interface	Typical use
Block (RBD)	Kernel block device / iSCSI	Kubernetes PVCs, VM disks
Object (RGW)	S3-compatible API	Backups, artifacts, media
File (CephFS)	POSIX filesystem / NFS	Shared filesystems, home dirs

For Kubernetes workloads, RBD block storage via a StorageClass is the common path.

Components

MON (Monitor) — maintains the cluster map; quorum-based, needs an odd number (typically 3 or 5). Not a data path.

OSD (Object Storage Daemon) — one per disk; handles actual data reads/writes and replication.

MGR (Manager) — collects metrics, hosts the dashboard, runs modules (balancer, alertmanager, etc.).

MDS (Metadata Server) — only required for CephFS; manages the filesystem namespace.

Single-node constraint

A single-node Ceph cluster can be made to run (allowMultiplePerNode: true in Rook, replication size: 1), but it provides no actual redundancy. There is nothing to replicate to. This is fine for testing concepts; it is not a valid storage setup for anything you care about.

Ceph documentation
Rook — Kubernetes operator that manages Ceph clusters inside K8s
Proxmox — Ceph is a native storage backend in Proxmox clusters
Rook + Ceph in the homelab

Distributed on Backend Engineering Strategy Tools

Ceph

Storage types

Components

Single-node constraint

Related