kubernetes的ceph RBD volume(1)：使用Ceph RBD作为后端Volume

Kubernetes使用Ceph RBD作为后端Volume。 Kubernetes的官方源码 的examples/volumes/rbd目录下，就有一个使用cephrbd作为kubernetes pod volume的例子，我们可以参考

1. 当ceph集群安装完成以后，我们就要创建相应的rbd块用于kubernetes存储。创建块设备之前，需要先创建存储池

ceph osd pool create kube 256 256    #后面两个256分别为pg-num和pgp-num

ceph osd pool ls detail

pool 4 'kube' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 2410 flags hashpspool stripe_width 0

    removed_snaps [1~3]

2. 在kube存储池创建一个映像文件，就叫vol50，该映像文件的大小为50GB：

rbd create kube/vol50 --size 50000

 rbd -p kube info vol50

rbd image 'vol50':

    size 51200 MB in 12800 objects

    order 22 (4096 kB objects)

    block_name_prefix: rb.0.754829.238e1f29

    format: 1

3. 创建用户client.kube 用admin用户

ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read class-write object_prefix rbd_children, allow rwx pool=kube' -o ceph.client.kube.keyring

通常我们在ceph install时在ceph.conf中使用默认的安全验证协议 cephx – The Ceph authentication protocol 了。

4.  生成secret

得到key(base64)

grep key /etc/ceph/ceph.client.kube.keyring |awk '{printf "%s", $NF}'|base64

QVFCK0l4RlpqK0xDTkJBQTRKYVBPTWx6WkFIVVhLK2ozM2lQdUE9PQo=

写secret.yaml：

[root@testnew rbd]# cat ceph-secret.yaml

apiVersion: v1

kind: Secret

metadata:

  name: ceph-secret

type: "kubernetes.io/rbd"

data:

  key: QVFCK0l4RlpqK0xDTkJBQTRKYVBPTWx6WkFIVVhLK2ozM2lQdUE9PQo=

 copy /etc/ceph/ceph.client.kube.keyring和ceph.conf到kubernetes的所有节点

kubectl create -f ceph-secret.yaml

[root@testnew ~]# kubectl get secret

NAME          TYPE                DATA      AGE

ceph-secret   kubernetes.io/rbd   1         13d

5. 格式化一个空image

格式化一个空image那样对其进行格式化了，这里格成ext4文件系统（格式化这一步可以不需要）

rbd map kube/vol50

rbd info kube/vol50

mkfs.ext4 /dev/rbd0

rbd unmap /dev/rbd0

6. 创建pod with RBD

[root@testnew kube]# cat frontend-rbd-controller.yaml

apiVersion: v1

kind: ReplicationController

metadata:

  name: frontendrbd1

  labels:

    name: frontendrbd1

spec:

  replicas: 1

  selector:

    name: frontendrbd1

  template:

    metadata:

      labels:

        name: frontendrbd1

    spec:

      containers:

      - name: frontendrbd1

        image: kubeguide/guestbook-php-frontend

        env:

        - name: GET_HOSTS_FROM

          value: env

        ports:

        - containerPort: 80

        volumeMounts:

        - mountPath: /mnt/rbd

          name: rbdpb

      volumes:

      - name: rbdpb

        rbd:

          monitors:

          - 10.0.200.11:6789

          - 10.0.200.13:6789

          - 10.0.200.14:6789

          pool: kube

          image: vol50

          user: kube

          secretRef:

              name: ceph-secret

          fsType: ext4

          readOnly: false

kubectl create -f frontend-rbd-controller.yaml

[root@testnew ~]# kubectl get rc

NAME            DESIRED   CURRENT   READY     AGE

frontendrbd1    1         1         1         13d

[root@testnew ~]# kubectl get pods

NAME                  READY     STATUS    RESTARTS   AGE

frontendrbd1-h9z78    1/1       Running   1          13d

7. 验证volume在container里。

[root@testnew ~]# kubectl exec frontendrbd1-h9z78 -it bash

root@frontendrbd1-h9z78:/var/www/html# df -k

Filesystem                                                                                       1K-blocks     Used Available Use% Mounted on

/dev/mapper/docker-253:1-530097-861967a5b3b1a5f40b4880db1921a52af2656a10bf5ce9d1727c40845a4aa9c2  10474496   623084   9851412   6% /

tmpfs                                                                                              4087712        0   4087712   0% /dev

tmpfs                                                                                              4087712        0   4087712   0% /sys/fs/cgroup
/dev/rbd0                                                                                         51474912 16840936  31996152  35% /mnt/rbd

/dev/vda1                                                                                         19593296 15144980   3577460  81% /etc/hosts

shm                                                                                                  65536        0     65536   0% /dev/shm