1 cephFS快速搭建
1.创建两个存储池分别用于存储mds的元数据和数据
bash
[root@ceph141ceph]# ceph osd pool create cephfs_data
pool 'cephfs_data' created
[root@ceph141ceph]# ceph osd pool create cephfs_metadata
pool 'cephfs_metadata' created2.创建一个文件系统，名称为"zhiyong18-cephfs"
bash
[root@ceph141ceph]# ceph fs new zhiyong18-cephfs cephfs_metadata cephfs_data
Pool 'cephfs_data' (id '10') has pg autoscale mode 'on' but is not marked as bulk.
Consider setting the flag by running
# ceph osd pool set cephfs_data bulk true
new fs with metadata pool 11 and data pool 103.查看创建的存储池
bash
[root@ceph141ceph]# ceph fs ls
name: zhiyong18-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[root@ceph141ceph]# ceph mds stat
zhiyong18-cephfs:0
[root@ceph141ceph]# ceph -s
cluster:
id: 12fad866-9aa0-11ef-8656-6516a17ad6dd
health: HEALTH_ERR
1 filesystem is offline
1 filesystem is online with fewer MDS than max_mds
1 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 73m)
mgr: ceph141.yvswvf(active, since 73m), standbys: ceph142.gtcikx
mds: 0/0 daemons up
osd: 7 osds: 7 up (since 73m), 7 in (since 38h)
data:
volumes: 1/1 healthy
pools: 6 pools, 129 pgs
objects: 50 objects, 18 MiB
usage: 335 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 129 active+clean发现：存储池的状态无法正常使用，而且集群是有错误的（HEALTH_ERR），因此我们需要先解决这个问题
在web页面中也有提示：All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline.
所以先开启mds服务，元数据服务
4.应用mds的文件系统
bash
[root@ceph141ceph]# ceph orch apply mds zhiyong18-cephfs
Scheduled mds.zhiyong18-cephfs update...5.添加第1个mds服务器，ceph143节点
bash
[root@ceph141ceph]# ceph orch daemon add mds zhiyong18-cephfs ceph143
Deployed mds.zhiyong18-cephfs.ceph143.eltfff on host 'ceph143'
[root@ceph141ceph]# ceph mds stat
zhiyong18-cephfs:1 {0=zhiyong18-cephfs.ceph143.eltfff=up:active} 1 up:standby添加第2个mds，ceph141节点
bash
[root@ceph141ceph]# ceph orch daemon add mds zhiyong18-cephfs ceph141
Deployed mds.zhiyong18-cephfs.ceph141.geohlx on host 'ceph141'6.目前活跃提供服务是ceph143，备用的是ceph141
bash
[root@ceph141ceph]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active zhiyong18-cephfs.ceph143.eltfff Reqs: 0 /s 10 13 12 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 96.0k 1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph141.geohlx
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)7.再次查看集群状态ERR状态消失。此处ERR状态是因为，之前的rbd问题（可以忽略）
bash
[root@ceph141ceph]# ceph -s
cluster:
id: 12fad866-9aa0-11ef-8656-6516a17ad6dd
health: HEALTH_WARN
1 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 84m)
mgr: ceph141.yvswvf(active, since 84m), standbys: ceph142.gtcikx
mds: 1/1 daemons up, 1 standby
osd: 7 osds: 7 up (since 84m), 7 in (since 39h)
data:
volumes: 1/1 healthy
pools: 6 pools, 129 pgs
objects: 72 objects, 18 MiB
usage: 335 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 129 active+clean8.验证mds的高可用性
直接给ceph143节点关机：init 0，然后查看cephfs的状态还是正常的
bash
[root@ceph141ceph]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active zhiyong18-cephfs.ceph141.geohlx Reqs: 0 /s 10 13 12 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 96.0k 1625G
cephfs_data data 0 1083G
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)2 cephFS配置多活
1.查看之前的状态
bash
[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active zhiyong18-cephfs.ceph141.geohlx Reqs: 0 /s 10 13 12 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 104k 1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph143.eltfff
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)2.查看默认的mds数量为1，同时仅有一个对外提供服务。修改为2，以让2个mds同时对外提供服务
bash
[root@ceph141~]# ceph fs get zhiyong18-cephfs | grep max_mds
max_mds 1
[root@ceph141~]# ceph fs set zhiyong18-cephfs max_mds 2
[root@ceph141~]# ceph fs get zhiyong18-cephfs | grep max_mds
max_mds 23.再次查看cephFS 。此时ceph141和ceph143同时对外提供服务，任何一个mds挂掉，都导致集群不可用！
bash
[root@ceph141~]# ceph orch daemon add mds zhiyong18-cephfs ceph142
[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active zhiyong18-cephfs.ceph141.geohlx Reqs: 0 /s 10 13 12 0
1 active zhiyong18-cephfs.ceph143.eltfff Reqs: 0 /s 10 13 11 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 176k 1083G
cephfs_data data 0 1083G
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)4.添加一个备用的mds，ceph142充当。添加后再次查看cephFS状态
bash
[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active zhiyong18-cephfs.ceph141.geohlx Reqs: 0 /s 10 13 12 0
1 active zhiyong18-cephfs.ceph143.eltfff Reqs: 0 /s 10 13 11 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 176k 1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph142.igigvp
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)5.验证是否生效。关闭ceph143节点：init 0。然后发现ceph142节点切换过去。
bash
[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active zhiyong18-cephfs.ceph141.geohlx Reqs: 0 /s 10 13 12 0
1 active zhiyong18-cephfs.ceph142.igigvp Reqs: 0 /s 10 13 11 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 183k 1625G
cephfs_data data 0 1083G
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
[root@ceph141~]# ceph -s
cluster:
id: 12fad866-9aa0-11ef-8656-6516a17ad6dd
health: HEALTH_WARN
failed to probe daemons or devices
insufficient standby MDS daemons available
1/3 mons down, quorum ceph141,ceph142
2 osds down
1 host (2 osds) down
Degraded data redundancy: 91/273 objects degraded (33.333%), 50 pgs degraded
1 pool(s) do not have an application enabled
services:
mon: 3 daemons, quorum ceph141,ceph142 (age 36s), out of quorum: ceph143
mgr: ceph141.yvswvf(active, since 7h), standbys: ceph142.gtcikx
mds: 2/2 daemons up
osd: 7 osds: 5 up (since 35s), 7 in (since 80m)
data:
volumes: 1/1 healthy
pools: 6 pools, 129 pgs
objects: 91 objects, 18 MiB
usage: 367 MiB used, 3.3 TiB / 3.3 TiB avail
pgs: 91/273 objects degraded (33.333%)
79 active+undersized
50 active+undersized+degraded6.再把ceph143添加到集群
bash
[root@ceph141~]# ceph orch daemon add mds zhiyong18-cephfs ceph143最后发现添加不成功，可能是mds不支持3个，最后把max_mds改为了1
3 客户端使用cephFS
01 服务端准备工作
1.确认cephFS的可用
bash
[root@ceph141~]# ceph fs ls
name: zhiyong18-cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[root@ceph141~]# ceph fs status zhiyong18-cephfs
zhiyong18-cephfs - 0 clients
================
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active zhiyong18-cephfs.ceph141.geohlx Reqs: 0 /s 10 13 12 0
POOL TYPE USED AVAIL
cephfs_metadata metadata 249k 1083G
cephfs_data data 0 1083G
STANDBY MDS
zhiyong18-cephfs.ceph142.oszjfe
MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
[root@ceph141~]# ceph fs get zhiyong18-cephfs | grep max_mds
max_mds 12.管理节点创建用户并导出钥匙环和key文件，发到其他节点
bash
[root@ceph141~]# ceph auth add client.wenzhiyongfs mon 'allow r' mds 'allow rw' osd 'allow rwx'
added key for client.wenzhiyongfs
[root@ceph141~]# ceph auth get client.wenzhiyongfs
[client.wenzhiyongfs]
key = AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
caps mds = "allow rw"
caps mon = "allow r"
caps osd = "allow rwx"
[root@ceph141~]# ceph auth get client.wenzhiyongfs > ceph.client.wenzhiyongfs.keyring
[root@ceph141~]# ceph auth print-key client.wenzhiyongfs > wenzhiyongfs.key
[root@ceph141~]# more wenzhiyongfs.key
AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
[root@ceph141~]# scp ceph.client.wenzhiyongfs.keyring wenzhiyongfs.key ceph143:/etc/ceph/
[root@ceph141~]# scp ceph.client.wenzhiyongfs.keyring ceph142:/etc/ceph/3.客户端确认本地的认证文件
[root@ceph143ceph]# ll | grep wen
-rw-r--r-- 1 root root 139 Nov 6 22:53 ceph.client.wenzhiyongfs.keyring
-rw-r--r-- 1 root root 40 Nov 6 22:53 wenzhiyongfs.key02 客户端使用cephFS
1.客户端ceph141基于key文件进行挂载并尝试写入数据
bash
[root@ceph143ceph]# df -h | grep mnt
[root@ceph143ceph]# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt \
-o name=wenzhiyongfs,secretfile=/etc/ceph/wenzhiyongfs.key
[root@ceph143ceph]# df -h | grep mnt
10.0.0.141:6789,10.0.0.142:6789,127.0.1.1:6789,10.0.0.143:6789:/ 1.1T 0 1.1T 0% /mnt
[root@ceph143ceph]# cp /etc/os-release /etc/fstab /etc/hosts /mnt/
[root@ceph143ceph]# ls /mnt/
fstab hosts os-release2.启动另一个客户端ceph142测试，测试时发现数据和ceph143同步了：都能看到所有节点上传的4个文件
bash
[root@ceph142~]# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt \
-o name=wenzhiyongfs,secret=AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
[root@ceph142~]# df -h | grep mnt
10.0.0.141:6789,127.0.1.1:6789,10.0.0.142:6789,10.0.0.143:6789:/ 1.1T 0 1.1T 0% /mnt
[root@ceph142~]# cp /etc/hostname /mnt/
[root@ceph142~]# ls /mnt/
fstab hostname hosts os-releaseceph143节点查看：
bash
[root@ceph143~]# ls /mnt/
fstab hostname hosts os-release3.把认证文件删除后ceph.client.wenzhiyongfs.keyring，虽然不影响挂载，但是会有提示。
bash
[root@ceph142ceph]# ls
ceph.client.admin.keyring ceph.client.wenzhiyongfs.keyring ceph.conf rbdmap
[root@ceph142ceph]# rm -rf ceph.client.wenzhiyongfs.keyring
[root@ceph142ceph]# umount /mnt
[root@ceph142ceph]# mount -t ceph ceph141:6789,ceph142:6789,ceph143:6789:/ /mnt -o name=wenzhiyongfs,secret=AQCOfitnDobhOBAAdJT9D0JVIjRch+kOQl8q3g==
2024-11-06T23:35:58.145+0800 7f21d1629100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.wenzhiyongfs.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
[root@ceph142ceph]# df -h | grep mnt
10.0.0.141:6789,127.0.1.1:6789,10.0.0.142:6789,10.0.0.143:6789:/ 1.1T 0 1.1T 0% /mnt4.ceph142节点依旧可以正常写入。用起来就和NFS差不多
bash
[root@ceph142ceph]# cp /etc/netplan/00-installer-config.yaml /mnt/4 K8S对接cephFS
4.1 cephFS作为volumes
1.Ubuntu默认安装的最新版本是17.2.7，也可以使用拷贝ceph的APT源，实现安装最新版。
所有K8S节点安装ceph-common：
bash
apt -y install ceph-commonceph节点把认证文件传输到所有K8S节点：
bash
scp admin.secret 10.0.0.231:/etc/ceph/
scp admin.secret 10.0.0.232:/etc/ceph/
scp admin.secret 10.0.0.233:/etc/ceph/1.编写资源清单
yaml
[root@master23104-cephfs]# cat 01-k8s-use-cephfs.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-volume-cephfs-admin-secretfile
spec:
replicas: 3
selector:
matchLabels:
apps: ceph-fs
template:
metadata:
labels:
apps: ceph-fs
spec:
volumes:
- name: data
# 指定存储卷的类型是cephFS
cephfs:
monitors:
- 10.0.0.141:6789
- 10.0.0.142:6789
- 10.0.0.143:6789
# 指定引用的cephFS的路径，若不指定默认为"/"
path: /
# 对于Pod而言，无需修改文件，因此设置为true
readOnly: true
# 指定连接ceph集群的用户，若不指定，默认为admin
user: admin
# 指定admin用户对应的认证文件所在路径
secretFile: "/etc/ceph/admin.secret"
containers:
- name: c1
image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
volumeMounts:
- name: data
mountPath: /wenzhiyong-data
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: svc-cephfs-secretfile
spec:
type: NodePort
selector:
apps: ceph-fs
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 300112.进入容器测试，可以看到之前ceph节点写入的数据；因为设置了只读，所以不能写入数据。pod如果要有写入权限，就把volumes中改为：readOnly: false
bash
kubectl exec deploy-volume-cephfs-admin-secretfile-57f6cf7df5-wmfg5 -- sh -c 'ls /wenzhiyong-data'
00-installer-config.yaml
fstab
hostname
hosts
os-release
# 进入pod测试
/ # cd /wenzhiyong-data/
/wenzhiyong-data # ls
00-installer-config.yaml fstab hostname hosts os-release
/wenzhiyong-data # echo 11 >> pods.txt
sh: can't create pods.txt: Read-only file system4.2 cephFS作为PVC
yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: zhiyong18-cephfs-pv
labels:
school: zhiyong18
spec:
accessModes:
- ReadWriteMany
cephfs:
monitors:
- 10.0.0.141:6789
- 10.0.0.141:6789
- 10.0.0.141:6789
path: /
readOnly: false
user: admin
secretFile: "/etc/ceph/admin.secret"
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: zhiyong18-cephfs-pvc
spec:
# 选择上一步创建的PV
volumeName: zhiyong18-cephfs-pv
accessModes:
- ReadWriteMany
resources:
limits:
storage: 2Gi
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-pvc-cephfs-secretfile
spec:
replicas: 3
selector:
matchLabels:
apps: cephfs
template:
metadata:
labels:
apps: cephfs
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: zhiyong18-cephfs-pvc
containers:
- name: c1
image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
volumeMounts:
- name: data
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: svc-cephfs-secretfile
spec:
type: NodePort
selector:
apps: cephfs
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 30012
- PV 的
accessModes表示该卷支持的访问模式范围。ReadWriteMany` 表示该 PV 可以支持多个节点同时读写该卷- PVC 的
accessModes表示 PVC 向PV请求的访问模式。如果 PVC 请求的是ReadWriteMany，那么它会尝试绑定一个支持该模式的 PV
2.运用后查看pod目录下的文件，还有之前ceph集群测试时留下来的文件。在nginx pod内创建目录和写入文件。
bash
[root@master23104-cephfs]# kubectl exec -it deploy-pvc-cephfs-secretfile-6cfcdbfcf-r6sbt -- sh
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # ls
00-installer-config.yaml fstab hostname hosts os-release
/usr/share/nginx/html # mkdir nginx
/usr/share/nginx/html # echo wzywzy.fun > nginx/index.html3.回到宿主机测试，访问任意一个pod都能看到返回正确的内容，说明：3个pod在共享这个cephFS
bash
[root@master23104-cephfs]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deploy-pvc-cephfs-secretfile-6cfcdbfcf-r6sbt 1/1 Running 0 7m57s 10.100.2.37 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-v55zb 1/1 Running 0 7m57s 10.100.2.38 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-vzzbf 1/1 Running 0 7m57s 10.100.1.19 worker233 <none> <none>
[root@master23104-cephfs]# curl 10.100.2.37/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.2.38/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.1.19/nginx/index.html
wzywzy.fun
ning 0 7m57s 10.100.2.37 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-v55zb 1/1 Running 0 7m57s 10.100.2.38 worker232 <none> <none>
deploy-pvc-cephfs-secretfile-6cfcdbfcf-vzzbf 1/1 Running 0 7m57s 10.100.1.19 worker233 <none> <none>
[root@master23104-cephfs]# curl 10.100.2.37/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.2.38/nginx/index.html
wzywzy.fun
[root@master23104-cephfs]# curl 10.100.1.19/nginx/index.html
wzywzy.fun