cephFS的使用以及K8S对接cephFS

1 cephFS快速搭建

1.创建两个存储池分别用于存储mds的元数据和数据

[root@ceph141ceph]# ceph osd pool create cephfs_data
pool 'cephfs_data' created
[root@ceph141ceph]# ceph osd pool create cephfs_metadata
pool 'cephfs_metadata' created

2.创建一个文件系统，名称为"zhiyong18-cephfs"

[root@ceph141ceph]# ceph fs new zhiyong18-cephfs cephfs_metadata cephfs_data
  Pool 'cephfs_data' (id '10') has pg autoscale mode 'on' but is not marked as bulk.
  Consider setting the flag by running
    # ceph osd pool set cephfs_data bulk true
new fs with metadata pool 11 and data pool 10

3.查看创建的存储池

[root@ceph141ceph]# ceph fs ls
name: zhiyong18-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[root@ceph141ceph]# ceph mds stat
zhiyong18-cephfs:0
[root@ceph141ceph]# ceph -s
  cluster:
    id:     12fad866-9aa0-11ef-8656-6516a17ad6dd
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 pool(s) do not have an application enabled
 
  services:
    mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 73m)
    mgr: ceph141.yvswvf(active, since 73m), standbys: ceph142.gtcikx
    mds: 0/0 daemons up
    osd: 7 osds: 7 up (since 73m), 7 in (since 38h)
 
  data:
    volumes: 1/1 healthy
    pools:   6 pools, 129 pgs
    objects: 50 objects, 18 MiB
    usage:   335 MiB used, 3.3 TiB / 3.3 TiB avail
    pgs:     129 active+clean

发现：存储池的状态无法正常使用，而且集群是有错误的（HEALTH_ERR），因此我们需要先解决这个问题

在web页面中也有提示：All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline.

所以先开启mds服务，元数据服务

4.应用mds的文件系统

[root@ceph141ceph]# ceph orch apply mds zhiyong18-cephfs
Scheduled mds.zhiyong18-cephfs update...

5.添加第1个mds服务器，ceph143节点

[root@ceph141ceph]# ceph orch daemon add mds zhiyong18-cephfs ceph143
Deployed mds.zhiyong18-cephfs.ceph143.eltfff on host 'ceph143'
[root@ceph141ceph]# ceph mds stat
zhiyong18-cephfs:1 {0=zhiyong18-cephfs.ceph143.eltfff=up:active} 1 up:standby

添加第2个mds，ceph141节点

[root@ceph141ceph]# ceph orch daemon add mds zhiyong18-cephfs ceph141
Deployed mds.zhiyong18-cephfs.ceph141.geohlx on host 'ceph141'

6.目前活跃提供服务是ceph143，备用的是ceph141

[root@ceph141ceph]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK  STATE                 MDS                   ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  zhiyong18-cephfs.ceph143.eltfff  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs_metadata  metadata  96.0k  1083G  
  cephfs_data      data       0   1083G  
          STANDBY MDS            
zhiyong18-cephfs.ceph141.geohlx  
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

7.再次查看集群状态ERR状态消失。此处ERR状态是因为，之前的rbd问题（可以忽略）

[root@ceph141ceph]# ceph -s
  cluster:
    id:     12fad866-9aa0-11ef-8656-6516a17ad6dd
    health: HEALTH_WARN
            1 pool(s) do not have an application enabled
 
  services:
    mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 84m)
    mgr: ceph141.yvswvf(active, since 84m), standbys: ceph142.gtcikx
    mds: 1/1 daemons up, 1 standby
    osd: 7 osds: 7 up (since 84m), 7 in (since 39h)
 
  data:
    volumes: 1/1 healthy
    pools:   6 pools, 129 pgs
    objects: 72 objects, 18 MiB
    usage:   335 MiB used, 3.3 TiB / 3.3 TiB avail
    pgs:     129 active+clean

8.验证mds的高可用性

直接给ceph143节点关机：init 0，然后查看cephfs的状态还是正常的

[root@ceph141ceph]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK  STATE                 MDS                   ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  zhiyong18-cephfs.ceph141.geohlx  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs_metadata  metadata  96.0k  1625G  
  cephfs_data      data       0   1083G  
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

2 cephFS配置多活

1.查看之前的状态

[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK  STATE                 MDS                   ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  zhiyong18-cephfs.ceph141.geohlx  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs_metadata  metadata   104k  1083G  
  cephfs_data      data       0   1083G  
          STANDBY MDS            
zhiyong18-cephfs.ceph143.eltfff  
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

2.查看默认的mds数量为1，同时仅有一个对外提供服务。修改为2，以让2个mds同时对外提供服务

[root@ceph141~]# ceph fs get zhiyong18-cephfs | grep  max_mds
max_mds	1
[root@ceph141~]# ceph fs set zhiyong18-cephfs  max_mds 2
[root@ceph141~]# ceph fs get zhiyong18-cephfs | grep  max_mds
max_mds	2

3.再次查看cephFS 。此时ceph141和ceph143同时对外提供服务，任何一个mds挂掉，都导致集群不可用！

[root@ceph141~]# ceph orch daemon add mds zhiyong18-cephfs ceph142
[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK  STATE                 MDS                   ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  zhiyong18-cephfs.ceph141.geohlx  Reqs:    0 /s    10     13     12      0   
 1    active  zhiyong18-cephfs.ceph143.eltfff  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs_metadata  metadata   176k  1083G  
  cephfs_data      data       0   1083G  
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

4.添加一个备用的mds，ceph142充当。添加后再次查看cephFS状态

[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK  STATE                 MDS                   ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  zhiyong18-cephfs.ceph141.geohlx  Reqs:    0 /s    10     13     12      0   
 1    active  zhiyong18-cephfs.ceph143.eltfff  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs_metadata  metadata   176k  1083G  
  cephfs_data      data       0   1083G  
          STANDBY MDS            
zhiyong18-cephfs.ceph142.igigvp  
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

5.验证是否生效。关闭ceph143节点：init 0。然后发现ceph142节点切换过去。

[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK  STATE                 MDS                   ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  zhiyong18-cephfs.ceph141.geohlx  Reqs:    0 /s    10     13     12      0   
 1    active  zhiyong18-cephfs.ceph142.igigvp  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs_metadata  metadata   183k  1625G  
  cephfs_data      data       0   1083G  
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
[root@ceph141~]# ceph -s
  cluster:
    id:     12fad866-9aa0-11ef-8656-6516a17ad6dd
    health: HEALTH_WARN
            failed to probe daemons or devices
            insufficient standby MDS daemons available
            1/3 mons down, quorum ceph141,ceph142
            2 osds down
            1 host (2 osds) down
            Degraded data redundancy: 91/273 objects degraded (33.333%), 50 pgs degraded
            1 pool(s) do not have an application enabled
 
  services:
    mon: 3 daemons, quorum ceph141,ceph142 (age 36s), out of quorum: ceph143
    mgr: ceph141.yvswvf(active, since 7h), standbys: ceph142.gtcikx
    mds: 2/2 daemons up
    osd: 7 osds: 5 up (since 35s), 7 in (since 80m)
 
  data:
    volumes: 1/1 healthy
    pools:   6 pools, 129 pgs
    objects: 91 objects, 18 MiB
    usage:   367 MiB used, 3.3 TiB / 3.3 TiB avail
    pgs:     91/273 objects degraded (33.333%)
             79 active+undersized
             50 active+undersized+degraded

6.再把ceph143添加到集群

[root@ceph141~]# ceph orch daemon add mds zhiyong18-cephfs ceph143

最后发现添加不成功，可能是mds不支持3个，最后把max_mds改为了1

3 客户端使用cephFS

01 服务端准备工作

1.确认cephFS的可用

[root@ceph141~]# ceph fs ls
name: zhiyong18-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK  STATE                 MDS                   ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  zhiyong18-cephfs.ceph141.geohlx  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs_metadata  metadata   249k  1083G  
  cephfs_data      data       0   1083G  
          STANDBY MDS            
zhiyong18-cephfs.ceph142.oszjfe  
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
[root@ceph141~]# ceph fs get zhiyong18-cephfs | grep max_mds
max_mds	1

2.管理节点创建用户并导出钥匙环和key文件，发到其他节点

[root@ceph141~]# ceph auth add client.wenzhiyongfs mon 'allow r' mds 'allow rw' osd 'allow rwx'
added key for client.wenzhiyongfs
[root@ceph141~]# ceph auth get client.wenzhiyongfs
[client.wenzhiyongfs]
	key = AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
	caps mds = "allow rw"
	caps mon = "allow r"
	caps osd = "allow rwx"
[root@ceph141~]# ceph auth get client.wenzhiyongfs > ceph.client.wenzhiyongfs.keyring
[root@ceph141~]# ceph auth  print-key client.wenzhiyongfs > wenzhiyongfs.key
[root@ceph141~]# more wenzhiyongfs.key 
AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==


[root@ceph141~]# scp ceph.client.wenzhiyongfs.keyring wenzhiyongfs.key ceph143:/etc/ceph/
[root@ceph141~]# scp ceph.client.wenzhiyongfs.keyring  ceph142:/etc/ceph/

3.客户端确认本地的认证文件

[root@ceph143ceph]# ll | grep wen
-rw-r--r-- 1 root root 139 Nov  6 22:53 ceph.client.wenzhiyongfs.keyring
-rw-r--r-- 1 root root  40 Nov  6 22:53 wenzhiyongfs.key

02 客户端使用cephFS

1.客户端ceph141基于key文件进行挂载并尝试写入数据

[root@ceph143ceph]# df -h | grep mnt
[root@ceph143ceph]# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt \
-o name=wenzhiyongfs,secretfile=/etc/ceph/wenzhiyongfs.key

[root@ceph143ceph]# df -h | grep mnt
10.0.0.141:6789,10.0.0.142:6789,127.0.1.1:6789,10.0.0.143:6789:/  1.1T     0  1.1T   0% /mnt

[root@ceph143ceph]# cp /etc/os-release /etc/fstab /etc/hosts /mnt/
[root@ceph143ceph]# ls /mnt/
fstab  hosts  os-release

2.启动另一个客户端ceph142测试，测试时发现数据和ceph143同步了：都能看到所有节点上传的4个文件

[root@ceph142~]# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt \
-o name=wenzhiyongfs,secret=AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
[root@ceph142~]# df -h | grep mnt
10.0.0.141:6789,127.0.1.1:6789,10.0.0.142:6789,10.0.0.143:6789:/  1.1T     0  1.1T   0% /mnt
[root@ceph142~]# cp /etc/hostname /mnt/
[root@ceph142~]# ls /mnt/
fstab  hostname  hosts  os-release

ceph143节点查看：

[root@ceph143~]# ls /mnt/
fstab  hostname  hosts  os-release

3.把认证文件删除后ceph.client.wenzhiyongfs.keyring，虽然不影响挂载，但是会有提示。

[root@ceph142ceph]# ls
ceph.client.admin.keyring  ceph.client.wenzhiyongfs.keyring  ceph.conf  rbdmap
[root@ceph142ceph]# rm -rf ceph.client.wenzhiyongfs.keyring 
[root@ceph142ceph]# umount /mnt 
[root@ceph142ceph]# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt -o name=wenzhiyongfs,secret=AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
2024-11-06T23:35:58.145+0800 7f21d1629100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.wenzhiyongfs.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
[root@ceph142ceph]# df -h | grep mnt
10.0.0.141:6789,127.0.1.1:6789,10.0.0.142:6789,10.0.0.143:6789:/  1.1T     0  1.1T   0% /mnt

4.ceph142节点依旧可以正常写入。用起来就和NFS差不多

[root@ceph142ceph]# cp /etc/netplan/00-installer-config.yaml /mnt/

4 K8S对接cephFS

4.1 cephFS作为volumes

1.Ubuntu默认安装的最新版本是17.2.7，也可以使用拷贝ceph的APT源，实现安装最新版。

所有K8S节点安装ceph-common：

apt -y install ceph-common

ceph节点把认证文件传输到所有K8S节点：

scp admin.secret 10.0.0.231:/etc/ceph/
scp admin.secret 10.0.0.232:/etc/ceph/
scp admin.secret 10.0.0.233:/etc/ceph/

1.编写资源清单

[root@master23104-cephfs]# cat 01-k8s-use-cephfs.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-volume-cephfs-admin-secretfile
spec:
  replicas: 3
  selector:
    matchLabels:
      apps: ceph-fs
  template:
    metadata:
      labels:
        apps: ceph-fs
    spec:
      volumes:
      - name: data
        # 指定存储卷的类型是cephFS
        cephfs:
          monitors:
          - 10.0.0.141:6789
          - 10.0.0.142:6789
          - 10.0.0.143:6789
          # 指定引用的cephFS的路径，若不指定默认为"/"
          path: /
          # 对于Pod而言，无需修改文件，因此设置为true
          readOnly: true
          # 指定连接ceph集群的用户，若不指定，默认为admin
          user: admin
          # 指定admin用户对应的认证文件所在路径
          secretFile: "/etc/ceph/admin.secret"
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
        volumeMounts:
        - name: data
          mountPath: /wenzhiyong-data
        ports:
        - containerPort: 80
---

apiVersion: v1
kind: Service
metadata:
  name: svc-cephfs-secretfile
spec:
  type: NodePort
  selector:
    apps: ceph-fs
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30011

2.进入容器测试，可以看到之前ceph节点写入的数据；因为设置了只读，所以不能写入数据。pod如果要有写入权限，就把volumes中改为：readOnly: false

kubectl exec deploy-volume-cephfs-admin-secretfile-57f6cf7df5-wmfg5 -- sh -c 'ls /wenzhiyong-data'

00-installer-config.yaml
fstab
hostname
hosts
os-release

# 进入pod测试
/ # cd /wenzhiyong-data/
/wenzhiyong-data # ls
00-installer-config.yaml  fstab                     hostname                  hosts                     os-release
/wenzhiyong-data # echo 11 >> pods.txt
sh: can't create pods.txt: Read-only file system

4.2 cephFS作为PVC

apiVersion: v1
kind: PersistentVolume
metadata:
  name: zhiyong18-cephfs-pv
  labels:
    school: zhiyong18
spec:
  accessModes:
  - ReadWriteMany
  cephfs:
    monitors:
    - 10.0.0.141:6789
    - 10.0.0.141:6789
    - 10.0.0.141:6789
    path: /
    readOnly: false
    user: admin
    secretFile: "/etc/ceph/admin.secret"
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 2Gi
---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zhiyong18-cephfs-pvc
spec:
  # 选择上一步创建的PV
  volumeName: zhiyong18-cephfs-pv
  accessModes:
  - ReadWriteMany
  resources:
    limits:
       storage: 2Gi
    requests:
       storage: 1Gi
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-pvc-cephfs-secretfile
spec:
  replicas: 3
  selector:
    matchLabels:
      apps: cephfs
  template:
    metadata:
      labels:
        apps: cephfs
    spec:
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: zhiyong18-cephfs-pvc
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
        volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
---

apiVersion: v1
kind: Service
metadata:
  name: svc-cephfs-secretfile
spec:
  type: NodePort
  selector:
    apps: cephfs
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30012

• PV 的 accessModes 表示该卷支持的访问模式范围。ReadWriteMany` 表示该 PV 可以支持多个节点同时读写该卷
• PVC 的 accessModes 表示 PVC 向PV请求的访问模式。如果 PVC 请求的是 ReadWriteMany，那么它会尝试绑定一个支持该模式的 PV

2.运用后查看pod目录下的文件，还有之前ceph集群测试时留下来的文件。在nginx pod内创建目录和写入文件。

[root@master23104-cephfs]# kubectl exec -it deploy-pvc-cephfs-secretfile-6cfcdbfcf-r6sbt -- sh
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # ls
00-installer-config.yaml     fstab         hostname        hosts         os-release

/usr/share/nginx/html # mkdir nginx
/usr/share/nginx/html # echo wzywzy.fun > nginx/index.html

3.回到宿主机测试，访问任意一个pod都能看到返回正确的内容，说明：3个pod在共享这个cephFS

[root@master23104-cephfs]# kubectl get pods -o wide
NAME                                           READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
deploy-pvc-cephfs-secretfile-6cfcdbfcf-r6sbt   1/1     Running   0          7m57s   10.100.2.37   worker232   <none>           <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-v55zb   1/1     Running   0          7m57s   10.100.2.38   worker232   <none>           <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-vzzbf   1/1     Running   0          7m57s   10.100.1.19   worker233   <none>           <none>
[root@master23104-cephfs]# curl 10.100.2.37/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.2.38/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.1.19/nginx/index.html
wzywzy.fun
ning   0          7m57s   10.100.2.37   worker232   <none>           <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-v55zb   1/1     Running   0          7m57s   10.100.2.38   worker232   <none>           <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-vzzbf   1/1     Running   0          7m57s   10.100.1.19   worker233   <none>           <none>
[root@master23104-cephfs]# curl 10.100.2.37/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.2.38/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.1.19/nginx/index.html
wzywzy.fun