不重启删除rook-ceph
一、背景说明
客户环境中使用了rook-ceph来做ceph存储。在安装后存在问题，需要卸载。但按照官网方式卸载后还存在残留，安装还会报失败或者功能使用后各种卷重复。于是总结如下：
【注意】在不保留数据的情况下执行该步骤
正常三节点安装后的pod如下：
root@master1:/opt/ceph# kubectl get pod -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-8jb9v 3/3 Running 1 (3m40s ago) 4m14s
csi-cephfsplugin-dp78k 3/3 Running 1 (3m6s ago) 4m14s
csi-cephfsplugin-nw8nj 3/3 Running 0 4m14s
csi-cephfsplugin-provisioner-78d847f5d4-2pchq 6/6 Running 1 (3m38s ago) 4m14s
csi-cephfsplugin-provisioner-78d847f5d4-jpcgk 6/6 Running 1 (3m2s ago) 4m14s
csi-rbdplugin-9cjzq 3/3 Running 1 (3m40s ago) 4m14s
csi-rbdplugin-gm5qf 3/3 Running 1 (3m6s ago) 4m14s
csi-rbdplugin-provisioner-6dd5684d4b-hfm66 6/6 Running 1 (3m37s ago) 4m14s
csi-rbdplugin-provisioner-6dd5684d4b-r8cqb 6/6 Running 1 (3m4s ago) 4m14s
csi-rbdplugin-ws45k 3/3 Running 0 4m14s
rook-ceph-crashcollector-master1-7c6c8f578-nftw8 1/1 Running 0 2m34s
rook-ceph-crashcollector-master2-76c677787-tvv98 1/1 Running 0 2m6s
rook-ceph-crashcollector-master3-7f5c4666dd-f7j6g 1/1 Running 0 2m6s
rook-ceph-exporter-master1-67f4c7bdbc-7zh8g 1/1 Running 0 2m34s
rook-ceph-exporter-master2-677bb9b494-7cb2c 1/1 Running 0 2m3s
rook-ceph-exporter-master3-6fdfd54b45-l7jx6 1/1 Running 0 2m3s
rook-ceph-mgr-a-5cbcf885f6-pqh5j 3/3 Running 0 2m52s
rook-ceph-mgr-b-6db674b769-sx82w 3/3 Running 0 2m50s
rook-ceph-mon-a-b46b766d4-vpj7l 2/2 Running 0 3m21s
rook-ceph-mon-b-df5ff469d-k8ql8 2/2 Running 0 3m15s
rook-ceph-mon-c-c764d44b5-8fgd9 2/2 Running 0 3m4s
rook-ceph-operator-78dcf875cf-6cmnk 1/1 Running 0 4m45s
rook-ceph-osd-0-6b4fb8c65b-4b88n 2/2 Running 0 2m2s
rook-ceph-osd-1-5c69f97568-j2kjt 2/2 Running 0 2m7s
rook-ceph-osd-2-564d496c7b-27wzb 2/2 Running 0 2m6s
rook-ceph-osd-3-7c65b8f59-hhlsb 2/2 Running 0 2m2s
rook-ceph-osd-4-55c49f94fc-rwvrt 2/2 Running 0 2m6s
rook-ceph-osd-5-65fc76fc77-p6jjn 2/2 Running 0 2m6s
rook-ceph-osd-6-54c7798545-qnjl9 2/2 Running 0 2m2s
rook-ceph-osd-prepare-master1-tdln2 0/1 Completed 0 2m29s
rook-ceph-osd-prepare-master2-ntqdh 0/1 Completed 0 2m28s
rook-ceph-osd-prepare-master3-d5x5p 0/1 Completed 0 2m27s
rook-ceph-tools-5c5cbb7454-845m5 1/1 Running 0 4m43s二、删除前清理
1、删除使用该存储的应用，pv，pvc，sc。否则需要重启节点
root@master1:/opt/ceph# kuebctl get pv
kuebctl: command not found
root@master1:/opt/ceph# kubectl get pv -A
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-98a6f72f-822e-42bd-8f2b-e9e91f0af926 500Gi RWO Delete Bound venus-plugin/registry-data openebs-hostpath <unset> 7d5h
root@master1:/opt/ceph# kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
venus-plugin registry-data Bound pvc-98a6f72f-822e-42bd-8f2b-e9e91f0af926 500Gi RWO openebs-hostpath <unset> 7d5h
root@master1:/opt/ceph# kubectl get sc -A
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-hostpath (default) openebs.io/local Delete WaitForFirstConsumer false 7d5h2、删除images
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash查看replicapool
bash-5.1$ rbd --pool replicapool ls
csi-vol-0f45adae-b107-48a0-ab82-87b2e1c2468a
csi-vol-3dbfc2eb-a683-4b7a-acb9-be657b110981
csi-vol-679164e6-8c38-4a1e-a7f6-ab8621e685d4
csi-vol-90311708-c111-4d23-bbb0-fc049136b743
csi-vol-90311708-c111-4d23-bbb0-fc049136b745
csi-vol-e81c09bf-922c-4d0c-a52b-82f2664cd441若存在，则需要删除
bash-5.1$ rbd rm replicapool/csi-vol-e81c09bf-922c-4d0c-a52b-82f2664cd441
Removing image: 100% complete...done.3、删除pool
kubectl delete -f pool.yaml4、添加清理主机路径
kubectl -n rook-ceph patch cephcluster rook-ceph --type merge -p '{"spec":{"cleanupPolicy":{"confirmation":"yes-really-destroy-data"}}}'三、执行删除
bash uninstall-rook-ceph.sh四、删除后处理
**【注意】**以下步骤每个rook-ceph节点都执行
lsblk -f
# 清理物理磁盘（如果不需要数据）
wipefs -a /dev/sdb1
wipefs -a /dev/sdc1
wipefs -a /dev/sdd1
sgdisk --zap-all /dev/sdb1
sgdisk --zap-all /dev/sdc1
sgdisk --zap-all /dev/sdd1
#擦除数据，防止卷卡主，耗时较久
blkdiscard /dev/sdb1
blkdiscard /dev/sdc1
blkdiscard /dev/sdd1
#重新注册次哦按
partprobe /dev/sdb
partprobe /dev/sdc
partprobe /dev/sdd删除残留目录
rm -rf /var/lib/rook/*
rm -rf /var/lib/kubelet/plugins/rook-ceph.*
rm -rf /var/lib/kubelet/plugins_registry/rook-ceph.*删除LVM映射
# This command hangs on some systems: with caution, 'dmsetup remove_all --force' can be used
ls /dev/mapper/ceph-* | xargs -I% -- dmsetup remove %
rm -rf /dev/ceph-*
rm -rf /dev/mapper/ceph--*五、删除卡住处理
1、查看
ps aux | grep uninstall-rook-ceph.sh
ps -ef | grep 51651
lsof -p 516512、查看pool
kubectl get cephcluster -n rook-ceph
kubectl get cephblockpools.ceph.rook.io -n rook-ceph3、删除cm和secret
kubectl -n rook-ceph patch secret rook-ceph-mon --type merge -p '{"metadata":{"finalizers": []}}'
kubectl -n rook-ceph patch configmap rook-ceph-mon-endpoints --type merge -p '{"metadata":{"finalizers": []}}'4、 Ceph 集群核心状态
进入工具箱，检查最基本的 Ceph 组件状态：
# 进入工具箱
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
# 检查集群整体状态
ceph -s
ceph status
# 检查 MON 状态
ceph mon stat
ceph mon dump
# 检查 OSD 状态
ceph osd stat
ceph osd tree
ceph osd dump
# 检查 PG 状态
ceph pg stat