最新版Ceph( Reef版本)块存储简单对接k8s

当前ceph

你的ceph集群上执行

1.创建名为k8s-rbd 的存储池
ceph osd pool create k8s-rbd  64 64

2.初始化
rbd pool init k8s-rbd

3 创建k8s访问块设备的认证用户
ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=k8s-rbd'

部署 ceph-rbd-csi

cat <<EOF > csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "76e8c1f6-0baa-11ef-ae7a-bc24115bcbdc",
        "monitors": [
          "192.168.0.26:6789",
          "192.168.0.27:6789",
          "192.168.0.28:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
EOF


###说明###
"clusterID": "76e8c1f6-0baa-11ef-ae7a-bc24115bcbdc"  通过 ceph mon dump 中的 fsid 获取

创建名为"ceph-csi-encryption-kms-config"的ConfigMap，因没有使用KMS，配置内容为空（不能省略，否则后面容器启动会报错

 cat <<EOF > csi-kms-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {}
metadata:
  name: ceph-csi-encryption-kms-config
EOF

最新版本的Ceph - CSI还需要另一个ConfigMap对象来定义Ceph配置，以添加到CSI容器内的Ceph .conf文件中:

 cat <<EOF > ceph-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  ceph.conf: |
    [global]
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx
  keyring: |
metadata:
  name: ceph-config
EOF

创建 secret对象，存储访问ceph的 key

cat <<EOF > csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: AQAnEjpmMbBeOhAA8SDvxVDYCGcOWtiQjhS+sg==
EOF

###说明###
userID 为上面在 ceph 集群中创建的 kubernetes
userKey 通过下面获取，这里无需通过 base64 加密，切记！

# ceph auth get client.kubernetes
[client.kubernetes]
	key = AQAnEjpmMbBeOhAA8SDvxVDYCGcOWtiQjhS+sg==
	caps mon = "profile rbd"
	caps osd = "profile rbd pool=k8s-rbd"

再部署官网提供的yaml文件

$ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
$ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml

$ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
$ kubectl apply -f csi-rbdplugin-provisioner.yaml

$ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
$ kubectl apply -f csi-rbdplugin.yaml

创建SC

 cat <<EOF > csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 76e8c1f6-0baa-11ef-ae7a-bc24115bcbdc
   pool: k8s-rbd <<<<<<<<<<<<<<<<<---------------（注意你的池）
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: default
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard
EOF

####创建一个pvc 试一试

cat <<EOF > raw-block-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF

helm在部署一个redis