Kubernetes 动态存储实战：从 NFS Provisioner 部署到 Redis 集群完整指南

一、环境准备

1.1 集群信息

组件	版本/信息
Kubernetes	v1.28+
NFS Server	192.168.194.11
NFS 共享目录	/opt/nfs
StorageClass	nfs-client
Redis 镜像	hb.reg.com/k8s/redis:8.0.0

1.2 前置条件

• ✅ Kubernetes 集群已部署
• ✅ NFS 服务端已安装
• ✅ 所有 Node 节点已安装 nfs-utils

---

二、NFS 服务端配置

2.1 配置 NFS 共享目录

# 编辑 /etc/exports
vim /etc/exports

添加以下内容：

/opt/nfs 192.168.194.0/24(rw,no_root_squash,sync,no_subtree_check)

2.2 创建目录并设置权限

mkdir -p /opt/nfs
chown -R nobody:nobody /opt/nfs
chmod 755 /opt/nfs

2.3 启动 NFS 服务

systemctl restart nfs-server
systemctl enable nfs-server

2.4 验证 NFS 共享

showmount -e 192.168.194.11

预期输出：

Export list for 192.168.194.11:
/opt/nfs 192.168.194.0/24

---

三、部署 NFS Provisioner

3.1 创建命名空间

kubectl create ns nfs-storageclass

3.2 创建 Deployment

文件名：deployment.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
  namespace: nfs-storageclass
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: hb.reg.com/k8s/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.194.11
            - name: NFS_PATH
              value: /opt/nfs
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.194.11
            path: /opt/nfs/

3.3 创建 RBAC

文件名：rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: nfs-storageclass
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: nfs-storageclass
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: leader-locking-nfs-client-provisioner
  namespace: nfs-storageclass
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: leader-locking-nfs-client-provisioner
  namespace: nfs-storageclass
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: nfs-storageclass
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

3.4 创建 StorageClass

文件名：storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
  onDelete: delete
reclaimPolicy: Delete
volumeBindingMode: Immediate

3.5 部署所有资源

kubectl apply -f deployment.yaml
kubectl apply -f rbac.yaml
kubectl apply -f storageclass.yaml

3.6 验证部署

# 查看 Provisioner Pod
kubectl get pod -n nfs-storageclass

# 查看 StorageClass
kubectl get storageclass

预期输出：

NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-7d6f5ddd49-d2npd   1/1     Running   0          14m

NAME         PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   AGE
nfs-client   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           13m

---

四、StorageClass 配置详解

4.1 参数说明

参数	值	说明
provisioner	k8s-sigs.io/nfs-subdir-external-provisioner	指定 Provisioner
onDelete	delete	PVC 删除时自动清理 NFS 数据
reclaimPolicy	Delete	PV 回收策略
volumeBindingMode	Immediate	PVC 创建后立即绑定

4.2 默认命名规则

当 不设置 pathPattern 时，Provisioner 使用默认命名规则：

{namespace}-{pvcName}-{pvName}

实际目录示例：

/opt/nfs/default-redis-cluster-data-redis-cluster-0-pvc-8c4246ad-6cad-4a8d-8ffc-83ae6f5e4321/

4.3 自定义命名规则（可选）

如果需要自定义目录结构，可以使用 pathPattern 参数：

parameters:
  pathPattern: "${.PVC.namespace}/${.PVC.name}"
  onDelete: delete

⚠️ 注意 ：v4.0.2 版本支持的语法是 ${.PVC.namespace}/${.PVC.name}，而不是 {``{ .PVC.Namespace }}/{``{ .PVC.Name }}。

---

五、测试动态存储

5.1 创建测试 PVC 和 Pod

文件名：test-pod.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
  storageClassName: nfs-client
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: hb.reg.com/library/myapp:1.0
    volumeMounts:
    - name: nfs-pvc
      mountPath: "/usr/local/nginx/html"
  restartPolicy: Never
  volumes:
  - name: nfs-pvc
    persistentVolumeClaim:
      claimName: test-claim

5.2 部署测试资源

kubectl apply -f test-pod.yaml

5.3 验证

# 查看 Pod
kubectl get pod

# 查看 PVC
kubectl get pvc

# 查看 PV（会自动创建）
kubectl get pv

预期输出：

NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          15s

NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
test-claim   Bound    pvc-2338abf3-cc29-4130-abd5-63fbadfdfab8   1Mi        RWX            nfs-client

5.4 验证 NFS 数据持久化

进入 Pod 创建测试文件：

kubectl exec -it test-pod -- /bin/bash
cd /usr/local/nginx/html/
echo "Hello NFS" > index.html
exit

在 NFS 服务端查看：

ls -la /opt/nfs/default-test-claim-pvc-xxxxx/
# 可以看到 index.html 文件

---

六、部署 Redis 集群

6.1 创建 ConfigMap

文件名：redis-cluster-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-config
data:
  redis-config: |
    appendonly yes
    protected-mode no
    dir /data
    port 6379
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 5000
    masterauth Chengke2026
    requirepass Chengke2026

6.2 创建 StatefulSet

文件名：redis-cluster-sts.yaml

apiVersion: v1
kind: Service
metadata:
  name: redis-headless
  labels:
    app.kubernetes.io/name: redis-cluster
spec:
  ports:
    - name: redis-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
  selector:
    app.kubernetes.io/name: redis-cluster
  clusterIP: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
  labels:
    app.kubernetes.io/name: redis-cluster
spec:
  serviceName: redis-headless
  replicas: 6
  selector:
    matchLabels:
      app.kubernetes.io/name: redis-cluster
  template:
    metadata:
      labels:
        app.kubernetes.io/name: redis-cluster
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app.kubernetes.io/name
                      operator: In
                      values:
                        - redis-cluster
                topologyKey: kubernetes.io/hostname
      containers:
        - name: redis
          image: hb.reg.com/k8s/redis:8.0.0
          imagePullPolicy: IfNotPresent
          command:
            - "redis-server"
          args:
            - "/etc/redis/redis.conf"
            - "--protected-mode"
            - "no"
            - "--cluster-announce-ip"
            - "$(POD_IP)"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: redis-6379
              containerPort: 6379
              protocol: TCP
          volumeMounts:
            - name: config
              mountPath: /etc/redis
            - name: redis-cluster-data
              mountPath: /data
          resources:
            requests:
              cpu: 100m
              memory: 500Mi
            limits:
              cpu: "200m"
              memory: 1Gi
      volumes:
        - name: config
          configMap:
            name: redis-cluster-config
            items:
              - key: redis-config
                path: redis.conf
  volumeClaimTemplates:
    - metadata:
        name: redis-cluster-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: nfs-client
        resources:
          requests:
            storage: 5Gi

如果这个hb.reg.com/k8s/redis:8.0.0镜像拉取不下来就直接在每个node节点上直接使用docker pull redis:8.0.0拉取即可，因为本集群使用的是cri-docker作为容器运行接口、底层容器运行时是docker来安装k8s的

6.3 创建外部访问 Service

文件名：redis-cluster-svc-external.yaml

kind: Service
apiVersion: v1
metadata:
  name: redis-cluster-external
  labels:
    app: redis-cluster-external
spec:
  ports:
    - protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 31379
  selector:
    app.kubernetes.io/name: redis-cluster
  type: NodePort

6.4 部署 Redis

kubectl apply -f redis-cluster-cm.yaml
kubectl apply -f redis-cluster-sts.yaml
kubectl apply -f redis-cluster-svc-external.yaml

6.5 验证部署

kubectl get pod
kubectl get pvc
kubectl get svc

预期输出：

NAME              READY   STATUS    RESTARTS   AGE
redis-cluster-0   1/1     Running   0          81s
redis-cluster-1   1/1     Running   0          53s
redis-cluster-2   1/1     Running   0          16s
redis-cluster-3   1/1     Running   0          15s
redis-cluster-4   1/1     Running   0          13s
redis-cluster-5   1/1     Running   0          11s

NAME                                      STATUS   VOLUME                                     CAPACITY   STORAGECLASS
redis-cluster-data-redis-cluster-0        Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   5Gi        nfs-client
redis-cluster-data-redis-cluster-1        Bound    pvc-yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy   5Gi        nfs-client
...

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
redis-cluster-external  NodePort    10.8.157.240   <none>        6379:31379/TCP   27s
redis-headless          ClusterIP   None           <none>        6379/TCP         9m52s

6.6 初始化 Redis 集群

# 进入任意一个 Redis Pod
kubectl exec -it redis-cluster-0 -- bash

# 执行集群初始化
redis-cli --cluster create \
  redis-cluster-0.redis-headless:6379 \
  redis-cluster-1.redis-headless:6379 \
  redis-cluster-2.redis-headless:6379 \
  redis-cluster-3.redis-headless:6379 \
  redis-cluster-4.redis-headless:6379 \
  redis-cluster-5.redis-headless:6379 \
  --cluster-replicas 1 -a Chengke2026

看到以下输出表示成功：

[OK] All 16384 slots covered.

6.7 验证集群状态

kubectl exec -it redis-cluster-0 -- redis-cli -a Chengke2026 cluster info
kubectl exec -it redis-cluster-0 -- redis-cli -a Chengke2026 cluster nodes

---

七、踩坑与排错

7.1 问题：Pod 一直 Pending，PVC 无法绑定

现象：

kubectl get pvc
NAME                                      STATUS    VOLUME   STORAGECLASS
redis-cluster-data-redis-cluster-0        Pending            nfs-client

原因： StorageClass 的 volumeBindingMode 设置为 WaitForFirstConsumer，PVC 要等 Pod 调度后才绑定。

解决： 修改为 Immediate 模式。

volumeBindingMode: Immediate

---

7.2 问题：Pod 挂载失败，报 No such file or directory

错误日志：

MountVolume.SetUp failed for volume "pvc-xxx" : 
mount failed: exit status 32
mount.nfs: mounting 192.168.194.11:/opt/nfs/{{.PVC.namespace}}/{{.PVC.name}} failed, 
reason given by server: No such file or directory

原因： pathPattern 模板变量未被解析，被当成了字面量字符串。

解决： 删除 pathPattern 参数，使用 Provisioner 默认命名规则。

parameters:
  onDelete: delete
  # 删除 pathPattern

---

7.3 问题：Redis Pod CrashLoopBackOff

错误日志：

Sorry, the cluster configuration file /data/nodes.conf is already used by 
a different Redis Cluster node.

原因： 所有 Pod 的 PVC 挂载到了同一个 NFS 目录，nodes.conf 文件冲突。

解决： 确保每个 Pod 有独立的存储目录。删除 pathPattern 后，Provisioner 使用默认命名规则，自动为每个 PVC 创建独立目录。

---

7.4 问题：pathPattern 语法不生效

尝试的语法	结果
{``{.PVC.namespace}}/{``{.PVC.name}}	❌ 被当字面量
{``{ .PVC.Namespace }}/{``{ .PVC.Name }}	❌ 被当字面量
${.PVC.namespace}/${.PVC.name}	❌ 被当字面量

结论： nfs-subdir-external-provisioner:v4.0.2 的 pathPattern 参数可能存在兼容性问题。

建议： 不使用 pathPattern，采用默认命名规则，简单可靠。

---

7.5 问题：StorageClass 参数不可修改

错误：

The StorageClass "nfs-client" is invalid: parameters: field is immutable

原因： StorageClass 的 parameters 一旦创建就不可修改。

解决： 删除并重建 StorageClass。

kubectl delete storageclass nfs-client
kubectl apply -f storageclass.yaml

---

八、总结

8.1 最终正确的 StorageClass 配置

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
  onDelete: delete
reclaimPolicy: Delete
volumeBindingMode: Immediate

8.2 NFS 目录结构

删除 pathPattern 后，Provisioner 自动创建的目录结构：

/opt/nfs/
├── default-redis-cluster-data-redis-cluster-0-pvc-8c4246ad-6cad-4a8d-8ffc-83ae6f5e4321/
├── default-redis-cluster-data-redis-cluster-1-pvc-687b1762-1f06-4cd7-8728-0c328e566203/
├── default-redis-cluster-data-redis-cluster-2-pvc-6bdf1389-4a9d-4838-9208-62f5c6b32acb/
├── default-redis-cluster-data-redis-cluster-3-pvc-ca521771-8805-413b-b15d-3a975a681419/
├── default-redis-cluster-data-redis-cluster-4-pvc-98001c61-176d-4b7c-ac96-207998def745/
├── default-redis-cluster-data-redis-cluster-5-pvc-b8477ef4-8849-455a-a60e-d3fa81ddf9c9/
├── mysql-replica/
└── mysql-source/

8.3 关键经验

1. 尽量不用 pathPattern：默认命名规则足够使用，避免语法兼容性问题
2. 使用 Immediate 绑定模式：StatefulSet 场景下更可靠
3. StorageClass 参数不可修改：配置前确认好，修改需要删除重建
4. 每个 Pod 必须有独立存储目录 ：避免 nodes.conf 等配置文件冲突

---

🔗 参考资料

• nfs-subdir-external-provisioner GitHub
• Kubernetes StorageClass 官方文档
• Redis 集群部署文档