Rook

December 9, 2017 | Docker Ceph Kubernetes

Rook is an open source orchestrator for distributed storage systems that currently focuses on orchestrating Ceph on top of Kubernetes. Deploying Rook on a Kubernetes cluster is as easy as just running 2 simple commands:

$ kubectl create -f rook-operator.yaml
$ kubectl create -f rook-cluster.yaml

Rook has been deployed on the PRP Kubernetes cluster. We’ll explore its configurations in this post. Unless otherwise noted, we’ll execute all kubectl commands using the admin Kubeconfig:

$ kubectl config use-context admin
Switched to context "admin".

rook-operator.yaml

We can find a sample rook-operator.yaml file at https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/rook-operator.yaml. We see that rook-operator.yaml defines:

  • The rook-system Namespace
  • The rook-operator ClusterRole
  • The rook-operator ServiceAccount in the rook-system Namespace
  • The rook-operator ClusterRoleBinding in the rook-system Namespace
  • The rook-operator Deployment in the rook-system Namespace

We can list the details of these resources on the PRP Kubernetes cluster with:

$ kubectl get namespace rook-system -o yaml
$ kubectl get clusterrole rook-operator -o yaml
$ kubectl get serviceaccount rook-operator -n rook-system -o yaml
$ kubectl get clusterrolebinding rook-operator -n rook-system -o yaml
$ kubectl get deployment rook-operator -n rook-system -o yaml

It appears that no customization of rook-operator.yaml has been done for the PRP Kubernetes cluster.

List all Replica Sets in the rook-system namespace:

$ kubectl get replicasets -n rook-system
NAME                       DESIRED   CURRENT   READY     AGE
rook-operator-766689445f   1         1         1         41d

List all Pods in the rook-system namespace:

$ kubectl get pods -n rook-system
NAME                             READY     STATUS    RESTARTS   AGE
rook-agent-5dzwj                 1/1       Running   3          41d
rook-agent-6lpb6                 1/1       Running   0          41d
rook-agent-7dflp                 1/1       Running   2          32d
rook-agent-8pxth                 1/1       Running   2          33d
rook-agent-9sjcq                 1/1       Running   0          41d
rook-agent-gl58x                 1/1       Running   3          41d
rook-agent-l5ck4                 1/1       Running   0          38d
rook-agent-nghrz                 1/1       Running   1          41d
rook-agent-p9qvv                 1/1       Running   0          33d
rook-agent-r2c27                 1/1       Running   0          41d
rook-agent-shhb5                 1/1       Running   0          41d
rook-agent-wzhbc                 1/1       Running   0          41d
rook-agent-xdxj6                 1/1       Running   0          41d
rook-agent-zm6w2                 1/1       Running   0          41d
rook-operator-766689445f-mxl8k   1/1       Running   0          31d

List all Daemon Sets in the rook-system namespace:

$ kubectl get daemonsets -n rook-system
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
rook-agent   14        14        14        14           14          <none>          41d

Or we can list all resource in the rook-system namespace:

$ kubectl get all -n rook-system

rook-cluster.yaml

We can find a sample rook-cluster.yaml file at https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/rook-cluster.yaml. We see that rook-cluster.yaml defines:

  • The rook Namespace
  • The rook Cluster in the rook Namespace

We can list the details of these resources on the PRP Kubernetes cluster with:

$ kubectl get ns rook -o yaml
$ kubectl get cluster rook -n rook -o yaml
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  clusterName: ""
  creationTimestamp: 2017-11-06T20:09:15Z
  deletionGracePeriodSeconds: null
  deletionTimestamp: null
  name: rook
  namespace: rook
  resourceVersion: "14432980"
  selfLink: /apis/rook.io/v1alpha1/namespaces/rook/clusters/rook
  uid: 5d35ea69-c32e-11e7-ba59-0cc47a6a1e1e
spec:
  dataDirHostPath: /var/lib/rook
  storage:
    deviceFilter: null
    location: null
    metadataDevice: null
    nodes:
    - devices:
      - name: nvme0n1
      name: coreos-01.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      name: coreos-02.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      name: coreos-03.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      name: coreos-04.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      name: coreos-05.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      name: coreos-06.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      name: coreos-07.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      name: coreos-08.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      - name: nvme1n1
      - name: nvme2n1
      - name: nvme3n1
      - name: nvme4n1
      - name: nvme5n1
      - name: nvme6n1
      - name: nvme7n1
      name: ps-100g.sdsu.edu
    - devices:
      - name: nvme0n1
      - name: nvme1n1
      - name: nvme2n1
      - name: nvme3n1
      - name: nvme4n1
      - name: nvme5n1
      - name: nvme6n1
      - name: nvme7n1
      name: siderea.ucsc.edu
    - devices:
      - name: nvme0n1
      - name: nvme1n1
      - name: nvme2n1
      - name: nvme3n1
      - name: nvme4n1
      - name: nvme5n1
      - name: nvme6n1
      - name: nvme7n1
      name: k8s-nvme-01.ultralight.org
    - devices:
      - name: nvme2n1
      - name: nvme3n1
      - name: nvme4n1
      - name: nvme5n1
      - name: nvme6n1
      - name: nvme7n1
      name: k8s-nvme-01.sdsc.optiputer.net
    - devices:
      - name: sdd
      - name: sde
      - name: sdf
      name: k8s-gpu-01.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      - name: nvme1n1
      - name: sdd
      - name: sde
      - name: sdf
      name: k8s-gpu-02.calit2.optiputer.net
    - devices:
      - name: nvme0n1
      - name: nvme1n1
      - name: nvme2n1
      - name: nvme3n1
      - name: nvme4n1
      - name: nvme5n1
      - name: nvme6n1
      - name: nvme7n1
      - name: sde
      - name: sdf
      - name: sdg
      - name: sdh
      - name: sdi
      - name: sdj
      - name: sdk
      - name: sdl
      - name: sdm
      - name: sdn
      name: k8s-epyc-01.sdsc.optiputer.net
    storeConfig:
      storeType: bluestore
    useAllDevices: false
    useAllNodes: false
  versionTag: master

List all Pods in the rook namespace:

$ kubectl get po -n rook
NAME                                                  READY     STATUS    RESTARTS   AGE
rook-api-5f95f65577-5cn87                             1/1       Running   0          41d
rook-ceph-mds-calogan-fs-694446745-cc5hp              1/1       Running   0          41d
rook-ceph-mds-calogan-fs-694446745-kvp2r              1/1       Running   0          39d
rook-ceph-mgr0-cbb9df9d4-w6jgb                        1/1       Running   0          41d
rook-ceph-mgr1-cf647cfcd-wq4n4                        1/1       Running   0          41d
rook-ceph-mon0-hwxjc                                  1/1       Running   1          41d
rook-ceph-mon1-8sjc7                                  1/1       Running   3          41d
rook-ceph-mon2-pbf8g                                  1/1       Running   0          41d
rook-ceph-osd-coreos-01.calit2.optiputer.net-vlmxs    1/1       Running   4          41d
rook-ceph-osd-coreos-02.calit2.optiputer.net-n98fs    1/1       Running   6          41d
rook-ceph-osd-coreos-03.calit2.optiputer.net-8bczl    1/1       Running   2          31d
rook-ceph-osd-coreos-04.calit2.optiputer.net-8fp9w    1/1       Running   2          32d
rook-ceph-osd-coreos-05.calit2.optiputer.net-25qqt    1/1       Running   2          32d
rook-ceph-osd-coreos-06.calit2.optiputer.net-7lbtc    1/1       Running   1          21d
rook-ceph-osd-coreos-07.calit2.optiputer.net-zchfm    1/1       Running   2          30d
rook-ceph-osd-coreos-08.calit2.optiputer.net-rtwcz    1/1       Running   3          41d
rook-ceph-osd-k8s-epyc-01.sdsc.optiputer.net-sj2ct    0/1       Pending   0          31d
rook-ceph-osd-k8s-gpu-01.calit2.optiputer.net-m4flt   1/1       Running   2          31d
rook-ceph-osd-k8s-gpu-02.calit2.optiputer.net-dwtjv   1/1       Running   3          39d
rook-ceph-osd-k8s-nvme-01.sdsc.optiputer.net-llgjl    1/1       Running   0          11d
rook-ceph-osd-k8s-nvme-01.ultralight.org-dpng7        0/1       Pending   0          38d
rook-ceph-osd-ps-100g.sdsu.edu-tk644                  1/1       Running   2          32d
rook-ceph-osd-siderea.ucsc.edu-p7djb                  1/1       Running   2          31d

List all Services in the rook namespace:

$ kubectl get svc -n rook
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
rook-api         ClusterIP   10.97.201.235    <none>        8124/TCP   41d
rook-ceph-mon0   ClusterIP   10.105.88.218    <none>        6790/TCP   41d
rook-ceph-mon1   ClusterIP   10.96.181.144    <none>        6790/TCP   41d
rook-ceph-mon2   ClusterIP   10.105.145.177   <none>        6790/TCP   41d

List all Deployments in the rook namespace:

$ kubectl get deploy -n rook
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
rook-api                   1         1         1            1           41d
rook-ceph-mds-calogan-fs   2         2         2            2           41d
rook-ceph-mgr0             1         1         1            1           41d
rook-ceph-mgr1             1         1         1            1           41d

List all Replica Sets in the rook namespace:

$ kubectl get rs -n rook
NAME                                            DESIRED   CURRENT   READY     AGE
rook-api-5f95f65577                             1         1         1         41d
rook-ceph-mds-calogan-fs-694446745              2         2         2         41d
rook-ceph-mgr0-cbb9df9d4                        1         1         1         41d
rook-ceph-mgr1-cf647cfcd                        1         1         1         41d
rook-ceph-mon0                                  1         1         1         41d
rook-ceph-mon1                                  1         1         1         41d
rook-ceph-mon2                                  1         1         1         41d
rook-ceph-osd-coreos-01.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-coreos-02.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-coreos-03.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-coreos-04.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-coreos-05.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-coreos-06.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-coreos-07.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-coreos-08.calit2.optiputer.net    1         1         1         41d
rook-ceph-osd-k8s-epyc-01.sdsc.optiputer.net    1         1         0         41d
rook-ceph-osd-k8s-gpu-01.calit2.optiputer.net   1         1         1         41d
rook-ceph-osd-k8s-gpu-02.calit2.optiputer.net   1         1         1         41d
rook-ceph-osd-k8s-nvme-01.sdsc.optiputer.net    1         1         1         41d
rook-ceph-osd-k8s-nvme-01.ultralight.org        1         1         0         41d
rook-ceph-osd-ps-100g.sdsu.edu                  1         1         1         41d
rook-ceph-osd-siderea.ucsc.edu                  1         1         1         41d

Or we can list all resource in the rook namespace:

$ kubectl get all -n rook

Block Storage

3 types of storage can be exposed by Rook:

  • Block: block storage to be consumed by a pod
  • Shared File System: a file system to be shared across multiple pods
  • Object: an object store that is accessible inside or outside the Kubernetes cluster

As of this writing, a block storage and a shared file system have been exposed by Rook on the PRP Kubernetes cluster. The block storage is defined by rook-storageclass.yaml. We can find a sample rook-storageclass file at https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/rook-storageclass.yaml. A rook-storageclass.yaml defines:

  • A Pool in the rook Namespace
  • A StorageClass

We can find the details of these resources on the PRP Kubernetes cluster with:

$ kubectl get pools -n rook
NAME      AGE
rbd       42d
$ kubectl get pool rbd -n rook -o yaml
apiVersion: rook.io/v1alpha1
kind: Pool
metadata:
  clusterName: ""
  creationTimestamp: 2017-11-06T20:11:29Z
  deletionGracePeriodSeconds: null
  deletionTimestamp: null
  name: rbd
  namespace: rook
  resourceVersion: "14434030"
  selfLink: /apis/rook.io/v1alpha1/namespaces/rook/pools/rbd
  uid: aca25212-c32e-11e7-ba59-0cc47a6a1e1e
spec:
  replicated:
    size: 3

$ kubectl get storageclasses
NAME                   PROVISIONER
rook-block (default)   rook.io/block
$ kubectl get storageclass rook-block -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2017-11-06T20:11:29Z
  name: rook-block
  resourceVersion: "15494156"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/rook-block
  uid: aca51a93-c32e-11e7-ba59-0cc47a6a1e1e
parameters:
  fstype: xfs
  pool: rbd
provisioner: rook.io/block
reclaimPolicy: Delete

Shared File System

A shared file system can be mounted read-write from multiple pods. The block storage is defined by rook-filesystem.yaml. We can find a sample rook-filesystem file at https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/rook-filesystem.yaml. A rook-filesystem.yaml defines:

  • A Filesystem in the rook Namespace

We can find the details of the shared file system on the PRP Kubernetes cluster with:

$ kubectl get filesystems -n rook
NAME         AGE
calogan-fs   42d

$ kubectl get filesystem calogan-fs -n rook -o yaml
apiVersion: rook.io/v1alpha1
kind: Filesystem
metadata:
  clusterName: ""
  creationTimestamp: 2017-11-06T21:56:13Z
  deletionGracePeriodSeconds: null
  deletionTimestamp: null
  name: calogan-fs
  namespace: rook
  resourceVersion: "14451836"
  selfLink: /apis/rook.io/v1alpha1/namespaces/rook/filesystems/calogan-fs
  uid: 4e32437a-c33d-11e7-ba59-0cc47a6a1e1e
spec:
  dataPools:
  - erasureCoded:
      codingChunks: 1
      dataChunks: 2
  metadataPool:
    replicated:
      size: 3
  metadataServer:
    activeCount: 1
    activeStandby: true

Rook Toolbox

The Rook toolbox is a container with common tools used for rook debugging and testing. The toolbox is based on Ubuntu.

Download the tools spec:

$ wget https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/rook-tools.yaml

$ cat rook-tools.yaml
apiVersion: v1
kind: Pod
metadata:
  name: rook-tools
  namespace: rook
spec:
  dnsPolicy: ClusterFirstWithHostNet
  containers:
  - name: rook-tools
    image: rook/toolbox:master
    imagePullPolicy: IfNotPresent
    env:
      - name: ROOK_ADMIN_SECRET
        valueFrom:
          secretKeyRef:
            name: rook-ceph-mon
            key: admin-secret
    securityContext:
      privileged: true
    volumeMounts:
      - mountPath: /dev
        name: dev
      - mountPath: /sys/bus
        name: sysbus
      - mountPath: /lib/modules
        name: libmodules
      - name: mon-endpoint-volume
        mountPath: /etc/rook
  hostNetwork: false
  volumes:
    - name: dev
      hostPath:
        path: /dev
    - name: sysbus
      hostPath:
        path: /sys/bus
    - name: libmodules
      hostPath:
        path: /lib/modules
    - name: mon-endpoint-volume
      configMap:
        name: rook-ceph-mon-endpoints
        items:
        - key: data
          path: mon-endpoints

Launch the rook-tools pod:

$ kubectl create -f rook-tools.yaml

Check the status of the pod:

$ kubectl -n rook get pod rook-tools
NAME         READY     STATUS    RESTARTS   AGE
rook-tools   1/1       Running   2          44m

Connect to the pod with:

$ kubectl -n rook exec -it rook-tools bash
root@rook-tools:/# echo $ROOK_ADMIN_SECRET
[REDACTED]

root@rook-tools:/# rookctl status
OVERALL STATUS: OK

USAGE:
TOTAL       USED       DATA       AVAILABLE
40.73 TiB   4.75 TiB   2.65 TiB   35.98 TiB

MONITORS:
NAME             ADDRESS                 IN QUORUM   STATUS
rook-ceph-mon1   10.96.181.144:6790/0    true        OK
rook-ceph-mon0   10.105.88.218:6790/0    true        OK
rook-ceph-mon2   10.105.145.177:6790/0   true        OK

MGRs:
NAME             STATUS
rook-ceph-mgr0   Active
rook-ceph-mgr1   Standby

OSDs:
TOTAL     UP        IN        FULL      NEAR FULL
60        34        34        false     false

PLACEMENT GROUPS (2148 total):
STATE          COUNT
active+clean   2148

root@rook-tools:/# ceph status
  cluster:
    id:     830408ae-bab3-4405-a697-00073152231c
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum rook-ceph-mon1,rook-ceph-mon0,rook-ceph-mon2
    mgr: rook-ceph-mgr0(active), standbys: rook-ceph-mgr1
    mds: calogan-fs-1/1/1 up  {0=mkvp2r=up:active}, 1 up:standby-replay
    osd: 60 osds: 34 up, 34 in

  data:
    pools:   3 pools, 2148 pgs
    objects: 683k objects, 2712 GB
    usage:   4868 GB used, 36843 GB / 41712 GB avail
    pgs:     2148 active+clean

  io:
    client:   851 B/s rd, 61955 B/s wr, 2 op/s rd, 1 op/s wr

root@rook-tools:/# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    41712G     36843G        4868G         11.67
POOLS:
    NAME                    ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd                     1      52417M      0.15        11156G       14240
    calogan-fs-metadata     2      63044k         0        11156G         145
    calogan-fs-data0        3       2661G      7.37        22312G      685793

root@rook-tools:/# rados df
POOL_NAME           USED   OBJECTS CLONES COPIES  MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS  RD    WR_OPS   WR
calogan-fs-data0     2661G  685793      0 2057379                  0       0        0 1163965 3217G  1617699  3549G
calogan-fs-metadata 63044k     145      0     435                  0       0        0 3794812 2851M   282641   915M
rbd                 52417M   14240      0   42720                  0       0        0 3110097  528G 47831276 10241G

total_objects    700178
total_used       4868G
total_avail      36843G
total_space      41712G

root@rook-tools:/# exit
exit

We can verify that the value of the environment variable ROOK_ADMIN_SECRET inside the container is the value of the admin-secret key of the rook-ceph-mon secret:

$ kubectl get secret rook-ceph-mon -n rook -o yaml
apiVersion: v1
data:
  admin-secret: [REDACTED]
  cluster-name: [REDACTED]
  fsid: [REDACTED]
  mon-secret: [REDACTED]
kind: Secret
metadata:
  creationTimestamp: 2017-11-06T20:09:21Z
  name: rook-ceph-mon
  namespace: rook
  resourceVersion: "14432999"
  selfLink: /api/v1/namespaces/rook/secrets/rook-ceph-mon
  uid: 60d46d73-c32e-11e7-ba59-0cc47a6a1e1e
type: kubernetes.io/rook