Skip to content

Instantly share code, notes, and snippets.

@morrismusumi

morrismusumi/external_ceph_cluster_for_kubernetes.md

Last active August 6, 2025 07:58
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save morrismusumi/16d926b3ec86da1088d00b7f9076f3ed to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save morrismusumi/16d926b3ec86da1088d00b7f9076f3ed to your computer and use it in GitHub Desktop.
Download ZIP
External Ceph Cluster for Kubernetes
Raw
external_ceph_cluster_for_kubernetes.md

External Ceph Cluster for Kubernetes

Create an external Ceph Cluster for all your Kubernetes clusters.

INSTALL CEPHADM

  • Prerequisites:

    • 3 VMs, with unused disks
    • Python 3, Cephadm is a Python-based utility
    • Systemd, runs Ceph daemons like mons, mgrs, mds
    • Podman or Docker some Cephadmn tasks are executed in containers
    • Time synchronization is important for distributed systems ensure Chrony or ntpd
    • LVM2 Ceph for provisioning and managing storage devices

  • Install Cephadm on first server which will be the Ceph manager

  • Set Ceph version and download the cephadm binary

    CEPH_RELEASE=19.2.2
curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm

  • Add the Ceph repo and install Cephadm to the system and verify version

    ./cephadm add-repo --release squid
./cephadm install
cephadm --version

CREATE CEPH CLUSTER

  • Bootstrap a new Ceph cluster

    cephadm bootstrap --mon-ip 172.30.0.61 --ssh-user debian

===Output Truncated===
Ceph Dashboard is now available at:

             URL: https://ceph-1:8443/
            User: admin
        Password: ek1sfbgrl2
===Output Truncated===

  • Login to Ceph dashboard via https://ceph-1:8443/

  • Install Ceph tools to manage cluster

    cephadm install ceph-common

ceph status

  cluster:
    id:     e376a9b4-4696-11f0-8594-2264c6d97862
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum ceph-1 (age 10m)
    mgr: ceph-1.jbqokr(active, since 9m)
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

  • Allow Ceph admin node to SSH into other nodes without a password

    ssh-copy-id -f -i /etc/ceph/ceph.pub debian@172.30.0.62
ssh-copy-id -f -i /etc/ceph/ceph.pub debian@172.30.0.63

  • Add new hosts to cluster, admin label ensures node can run ceph mgt taks, also allows for HA and fault tolerance.

    ceph orch host add ceph-3 172.30.0.62 --labels _admin
ceph orch host add ceph-3 172.30.0.63 --labels _admin

ceph status

  • Add OSDs, to fix the error with OSDs

    ceph status
ceph orch apply osd --all-available-devices

ceph osd ls
ceph osd df

ENABLE BLOCK STORAGE

  • Create a pool in the Ceph dashboards, from which to provision block storage
  • Create a pool(name rbd) set replication size to 3 and application to RBD or RADOS Block Device which is Ceph's block storage application

ENABLE FS STORAGE

  • To create Filesystem storage MDS or Metadata Server is required

  • MDS is a Ceph daemon that manages metadata for CephFS, Ceph's distributed file system

  • First set MDS labels on servers that can be used to run mds daemons

  • Create a cephfs filesystem, which will auto create the cephfs data and metadata pools as well as the mds daemons

    ceph fs volume create cephfs --placement="label:mds"

ceph -s
  cluster:
    id:     e376a9b4-4696-11f0-8594-2264c6d97862
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 17m)
    mgr: ceph-1.jbqokr(active, since 33m), standbys: ceph-2.vjlwim
    mds: 1/1 daemons up, 1 standby
    osd: 3 osds: 3 up (since 14m), 3 in (since 14m)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 35 pgs
    objects: 24 objects, 579 KiB
    usage:   82 MiB used, 120 GiB / 120 GiB avail
    pgs:     35 active+clean

ceph osd pool ls
.mgr
rbd
cephfs.cephfs.meta
cephfs.cephfs.data

ENABLE OBJECT STORAGE

  • Create a RADOS Gateway (radosgw or RGW)

  • A RADOS Gateway is Ceph's object storage gateway that provides RESTful APIs compatible with Amazon S3

  • Create and use S3 buckets with the Ceph cluster

  • Add 'rgw' label to GW eligible nodes from the dashboard

  • Create a RADOS Gateway on selected ceph nodes

    ceph orch apply rgw radosgw '--placement=label:rgw count-per-host:1' --port=8001

ceph osd pool ls
.mgr
rbd
cephfs.cephfs.meta
cephfs.cephfs.data
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta

INSTALL ROOK IN KUBERNETES

  • Rook is Open-Source, Cloud-Native Storage for Kubernetes and is typically how you install Ceph in a Kubernetes cluster.

  • Clone the Rook repository to your local machine, node ceph node

    git clone --single-branch --branch master https://github.com/rook/rook.git

  • Install the Rook CRDs and Rook Operator

    cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

kubectl get pods -n rook-ceph

NAME                                           READY   STATUS    RESTARTS   AGE
rook-ceph-operator-59dcf6d55b-p7hk5            1/1     Running   0          66m

  • Install a Ceph cluster of type external, meaning Rook connects to an external ceph cluster

    cd ../charts/rook-ceph-cluster
helm repo add rook-release https://charts.rook.io/release
helm install --namespace rook-ceph rook-ceph-cluster \
--set operatorNamespace=rook-ceph rook-release/rook-ceph-cluster -f values-external.yaml

kubectl get cephcluster

kubectl --namespace rook-ceph get cephcluster
NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE        MESSAGE                                             HEALTH   EXTERNAL   FSID
rook-ceph   /var/lib/rook     3          27s   Connecting   Attempting to connect to an external Ceph cluster            true

kubectl get pods -n rook-ceph
NAME                                           READY   STATUS    RESTARTS   AGE
csi-cephfsplugin-4vb4w                         3/3     Running   0          58m
csi-cephfsplugin-provisioner-57467b74d-df826   6/6     Running   0          58m
csi-rbdplugin-7wsx9                            3/3     Running   0          58m
csi-rbdplugin-provisioner-7c47cc896b-tl2zj     6/6     Running   0          58m
rook-ceph-operator-59dcf6d55b-p7hk5            1/1     Running   0          66m

  • Import provider cluster data that will initiate the connection of the ceph cluster with Rook

  • We need to extract some information form the ceph cluster

  • On ceph manager node clone the Rook git repo

git clone --single-branch --branch master https://github.com/rook/rook.git

cd rook/deploy/examples/external

ls
cluster-external.yaml
common-external.yaml
create-external-cluster-resources-tests.py
create-external-cluster-resources.py
dashboard-external-http.yaml
dashboard-external-https.yaml
import-external-cluster.sh
object-bucket-claim-delete.yaml
object-external.yaml
storageclass-bucket-delete.yaml

python3 create-external-cluster-resources.py --rbd-data-pool-name rbd --cephfs-filesystem-name cephfs --rgw-endpoint 172.30.0.61:8000 --namespace rook-ceph --format bash

  • Copy the output and save it in a ceph-cluster.env file

  • Import the data into Rook

  • On your local machine with kuberentes cluster context

    cd rook/deploy/examples/external

ls
cluster-external.yaml                      dashboard-external-https.yaml
common-external.yaml                       import-external-cluster.sh
create-external-cluster-resources-tests.py object-bucket-claim-delete.yaml
create-external-cluster-resources.py       object-external.yaml
dashboard-external-http.yaml               storageclass-bucket-delete.yaml

source ceph-cluster.env

./import-external-cluster.sh
cluster namespace rook-ceph already exists
secret/rook-ceph-mon created
configmap/rook-ceph-mon-endpoints created
configmap/external-cluster-user-command created
secret/rook-csi-rbd-node created
secret/rook-csi-rbd-provisioner created
secret/rgw-admin-ops-user created
secret/rook-csi-cephfs-node created
secret/rook-csi-cephfs-provisioner created
storageclass.storage.k8s.io/ceph-rbd created
storageclass.storage.k8s.io/cephfs created

kubectl -n rook-ceph  get CephCluster
NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE       MESSAGE                          HEALTH      EXTERNAL   FSID
rook-ceph   /var/lib/rook     3          10m   Connected   Cluster connected successfully   HEALTH_OK   true       e376a9b4-4696-11f0-8594-2264c6d97862


kubectl -n rook-ceph get sc
NAME                   PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ceph-rbd               rook-ceph.rbd.csi.ceph.com      Delete          Immediate              true                   56s
cephfs                 rook-ceph.cephfs.csi.ceph.com   Delete          Immediate              true                   56s
local-path (default)   rancher.io/local-path           Delete          WaitForFirstConsumer   false                  28m

USE STORAGE

  • Deploy statefulset to consume block storage

    apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-deploy
  namespace: default
spec:
  selector:
    matchLabels:
      app: test-deploy 
  serviceName: "test-deploy"
  replicas: 3
  template:
    metadata:
      labels:
        app: test-deploy
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: test-deploy
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "ceph-rbd"
      resources:
        requests:
          storage: 2Gi

kubectl apply -f k8s-test-deploy.yml -n default

kubectl get pods -n default
kubectl get pvc -n default

  • Create a CephObjectStore connect to rados gateway and Storage Class to provision object storage in the k8s cluster

  • Create ObjectBucketClaim to create an S3 bucket

    apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: external-object-store
  namespace: rook-ceph
spec:
  gateway:
    port: 8001
    externalRgwEndpoints:
      - ip: 172.30.0.61
      - ip: 172.30.0.63

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-bucket
provisioner: rook-ceph.ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
  objectStoreName: external-object-store
  objectStoreNamespace: rook-ceph

---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-bucket
spec:
  generateBucketName: ceph-bkt
  storageClassName: rook-ceph-bucket


k get sc,ObjectBucketClaim
NAME                                               PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/ceph-rbd               rook-ceph.rbd.csi.ceph.com      Delete          Immediate              true                   6m26s
storageclass.storage.k8s.io/cephfs                 rook-ceph.cephfs.csi.ceph.com   Delete          Immediate              true                   6m26s
storageclass.storage.k8s.io/local-path (default)   rancher.io/local-path           Delete          WaitForFirstConsumer   false                  33m
storageclass.storage.k8s.io/rook-ceph-bucket       rook-ceph.ceph.rook.io/bucket   Delete          Immediate              false                  63s

NAME                                            AGE
objectbucketclaim.objectbucket.io/ceph-bucket   63s

  • Test bucket

    export AWS_HOST=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_HOST}')
export PORT=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_PORT}')
export BUCKET_NAME=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_NAME}')
export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)


❯ mc alias set ceph-rgw http://$AWS_HOST:$PORT $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY
Added `ceph-rgw` successfully.
❯ mc ls ceph-rgw
[2025-06-11 10:50:06 CEST]     0B ceph-bkt-33aaa50b-cfa0-4d7b-94a8-fb62cca60ca0/
❯ mc cp ceph-cluster.env ceph-rgw/ceph-bkt-33aaa50b-cfa0-4d7b-94a8-fb62cca60ca0/
.../ceph-cluster.env: 1.51 KiB / 1.51 KiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 506 B/s 3s❯ mc ls ceph-rgw/ceph-bkt-33aaa50b-cfa0-4d7b-94a8-fb62cca60ca0/
[2025-06-11 10:54:51 CEST] 1.5KiB STANDARD ceph-cluster.env
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment