Linux学习37-rook-ceph部署

官网：Quickstart - Rook Ceph Documentation

Rook 是 Kubernetes 的存储编排工具；rook-ceph = Rook + Ceph，就是在 K8s 集群里面，用 Rook 自动部署、管理 Ceph 分布式存储

Ceph ：开源分布式存储，把多节点磁盘整合为一个大存储池，支持块存储、文件存储、对象存储

Rook ：Ceph 原生很复杂，Rook 以 Operator 方式跑在 k8s 里，自动部署、监控、扩缩容、故障自愈 Ceph，不用手动敲大量 ceph 命令

部署ceph集群

集群的状态需要为一master两worker
在实验之前，在k8s三个节点上各添加一块50G磁盘

部署rook-ceph

添加 rook-ceph chart库

再将value.yml文件中的镜像拉取，打标签，上传至私有仓库

笔者这里编辑了一个脚本来自动拉取，打标签，上传

复制代码

#!/bin/bash
set -e

# 定义镜像数组
images=(
"quay.io/cephcsi/cephcsi:v3.17.1"
"registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0"
"registry.k8s.io/sig-storage/csi-provisioner:v6.2.0"
"registry.k8s.io/sig-storage/csi-snapshotter:v8.5.0"
"registry.k8s.io/sig-storage/csi-attacher:v4.12.0"
"registry.k8s.io/sig-storage/csi-resizer:v2.1.0"
"quay.io/csiaddons/k8s-sidecar:v0.14.0"
)

# 目标仓库前缀
TARGET_PREFIX="reg.westos.org/ceph"

for img in "${images[@]}"; do
  echo -e "\n========== Processing $img =========="
  # 拉取原始镜像
  echo "docker pull $img"
  docker pull "$img"

  # 提取镜像名:tag
  img_name_tag=${img##*/}
  new_img="${TARGET_PREFIX}/${img_name_tag}"

  # 打新标签
  echo "docker tag $img $new_img"
  docker tag "$img" "$new_img"

  # 推送到私有仓库
  echo "docker push $new_img"
  docker push "$new_img"

  echo "==== Finished $img ===="
done

echo -e "\n✅ 全部镜像拉取、打标签、推送完成！"

还缺少了两个镜像，可以自行拉取

最后再替换掉value.yaml文件中的镜像路径

image:

repository: reg.westos.org/rook/ceph

tag: v1.20.7

csi:

cephcsi:

repository: reg.westos.org/ceph/cephcsi

tag: v3.17.1

registrar:

repository: reg.westos.org/ceph/csi-node-driver-registrar

tag: v2.17.0

provisioner:

repository: reg.westos.org/ceph/csi-provisioner

tag: v6.2.0

snapshotter:

repository: reg.westos.org/ceph/csi-snapshotter

tag: v8.5.0

attacher:

repository: reg.westos.org/ceph/csi-attacher

tag: v4.12.0

resizer:

repository: reg.westos.org/ceph/csi-resizer

tag: v2.1.0

csiAddons:

repository: reg.westos.org/ceph/k8s-sidecar

tag: v0.14.0

tolerations:

key: node-role.kubernetes.io/control-plane

effect: NoSchedule

ceph-csi-operator:

controllerManager:

manager:

image:

repository: reg.westos.org/ceph/ceph-csi-operator

tag: v1.0.4

再将k8s1的admin.conf拷贝到该节点，写入kubeconfig

复制代码

mkdir -p ~/.kube
cp /etc/kubernetes/admin.conf ~/.kube/config
chown $(id -u):$(id -g) ~/.kube/config

验证

部署ceph集群

下载rook-ceph的部署文件，注意匹配版本

修改文件

这里的磁盘必须要是新添加的未使用的盘

参考：
apiVersion: ceph.rook.io/v1

kind: CephCluster

metadata:

namespace: rook-ceph # namespace:cluster

spec:

cephVersion:

image: reg.westos.org/rook/ceph:v20.2.4

allowUnsupported: false

security:

cephx:

csi:

keyType: aes # required when Kubernetes nodes don't run Linux kernel 7.0+

dataDirHostPath: /var/lib/rook

skipUpgradeChecks: false

continueUpgradeAfterChecksEvenIfNotHealthy: false

waitTimeoutForHealthyOSDInMinutes: 10

upgradeOSDRequiresHealthyPGs: false

mon:

allowMultiplePerNode: false

mgr:

allowMultiplePerNode: false

modules:

name: rook

enabled: false

dashboard:

enabled: true

ssl: true

monitoring:

enabled: false

metricsDisabled: false

exporter:

perfCountersPrioLimit: 5

statsPeriodSeconds: 5

network:

connections:

encryption:

enabled: false

compression:

enabled: false

requireMsgr2: false

provider: host

crashCollector:

disable: false

logCollector:

enabled: true

periodicity: daily # one of: hourly, daily, weekly, monthly

maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.

cleanupPolicy:

confirmation: ""

sanitizeDisks:

method: quick

dataSource: zero

iteration: 1

allowUninstallWithVolumes: false

placement:

all:

tolerations:

key: node-role.kubernetes.io/control-plane

effect: NoSchedule

annotations:

labels:

resources:

removeOSDsIfOutAndSafeToRemove: false

priorityClassNames:

mon: system-node-critical

osd: system-node-critical

mgr: system-cluster-critical

storage: # cluster level storage configuration and selection

useAllNodes: false

useAllDevices: false

config:

allowDeviceClassUpdate: false # whether to allow changing the device class of an OSD after it is created

allowOsdCrushWeightUpdate: false # whether to allow resizing the OSD crush weight after osd pvc is increased

nodes:

name: "k8s1"

devices:

name: "nvme0n2"
name: "k8s2"

devices:

name: "nvme0n2"
name: "k8s3"

devices:

name: "nvme0n2"

scheduleAlways: false

onlyApplyOSDPlacement: false

disruptionManagement:

managePodBudgets: true

osdMaintenanceTimeout: 30

csi:

readAffinity:

enabled: false

cephfs:

healthCheck:

daemonHealth:

mon:

disabled: false

interval: 45s

osd:

disabled: false

interval: 60s

status:

disabled: false

interval: 60s

livenessProbe:

mon:

disabled: false

mgr:

disabled: false

osd:

disabled: false

startupProbe:

mon:

disabled: false

mgr:

disabled: false

osd:

disabled: false

最终出现三个osd-n的pod即为成功

部署storageclass

创建pvc

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

spec:

storageClassName: rook-ceph-block

accessModes:

ReadWriteOnce

resources:

requests:

storage: 1Gi

Rook 1.20+ 用 helm 部署时： rook-ceph chart 只部署 rook operator + ceph-csi-operator ，不再自动部署 RBD/cephfs CSI 驱动（provisioner、nodeplugin、CSIDriver） 。必须额外独立安装 ceph-csi-drivers helm chart ，否则：