提示：本文原创作品，良心制作，干货为主，简洁清晰，一看就会
文章目录
- 前言
- 一、整体流程概述
- 二、NFS+K8S部署redis-cluster集群实战
-
- [2.1 安装NFS](#2.1 安装NFS)
- [2.2 创建SC](#2.2 创建SC)
- [2.3 创建SA并授权](#2.3 创建SA并授权)
- [2.4 部署自动创建PV的pod服务](#2.4 部署自动创建PV的pod服务)
- [2.5 创建Redis的Sts](#2.5 创建Redis的Sts)
- [2.6 测试集群内Redis域名解析连通性](#2.6 测试集群内Redis域名解析连通性)
- [2.7 安装redis-trib](#2.7 安装redis-trib)
- [2.8 配置Redis三主三从](#2.8 配置Redis三主三从)
- [2.9 创建用于访问Redis的SVC](#2.9 创建用于访问Redis的SVC)
前言
在Kubernetes中搭建Redis Cluster，是实现高可用、可扩展缓存服务的关键实践。本文从底层存储配置出发，完整演示基于NFS动态供给的Redis集群部署流程，帮助你快速构建一套生产可用的分布式缓存系统
本文实验基于 K8s PV/PVC 动态存储前置知识，对此尚不熟悉的同学可先参阅此篇文章https://blog.csdn.net/m0_63756214/article/details/160648532?spm=1001.2014.3001.5502
一、整体流程概述
1，准备一台 NFS 服务器，为 K8s 提供统一后端存储，供 PV/PVC 挂载使用
2，创建 StorageClass、ServiceAccount 并授权，部署自动供给 PV/PVC 的 NFS Provisioner Pod
3，创建 ConfigMap 存储自定义 redis.conf 配置文件，创建 Headless Service，为 Redis 有状态应用提供固定域名解析
4，部署 StatefulSet，引用 ConfigMap 与 Service，配置 6 副本，通过 StorageClass 自动申请 NFS 持久化存储
5，如果是centos系统，则创建一个ubuntu的pod，在 Ubuntu 容器内安装集群管理工具redis-trib，将 6 个 Redis 节点配置为三主三从高可用集群；如果是ubuntu系统直接安装集群管理工具redis-trib，执行相关命令
二、NFS+K8S部署redis-cluster集群实战
2.1 安装NFS
yaml
## 准备一台机器安装NFS，后续的PV/PVC会到NFS中拿空间
## 我这台nfs的ip是192.168.13.137
root@NFS:~# apt -y install nfs-kernel-server
root@NFS:~# systemctl start nfs-kernel-server
root@NFS:~# mkdir /data
root@NFS:~# vim /etc/exports
/data *(rw,sync,no_root_squash,no_subtree_check)
root@NFS:~# chmod 777 /data
root@NFS:~# systemctl restart nfs-kernel-server
root@NFS:~# systemctl enable nfs-kernel-server
## 在K8S的每个node节点都安装nfs客户端，不然会挂载失败
root@k8s-node1:~# apt -y install nfs-common
root@k8s-node2:~# apt -y install nfs-common2.2 创建SC
yaml
root@k8s-master1:~# vim storageclass-nfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
## 指定外部NFS供给器的名字
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
root@k8s-master1:~# kubectl apply -f storageclass-nfs.yaml 2.3 创建SA并授权
yaml
## 创建RBAC
root@k8s-master1:~# vim rbac.yaml
## ServiceAccount：NFS provisioner pod 使用的身份
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
# 需要指定命名空间
namespace: default
---
## ClusterRole：定义集群维度的权限
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nfs-client-provisioner-ClusterRole
rules:
- apiGroups: [""]
resources: ["nodes"] # 需要获取节点信息，用于拓扑感知
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["persistentvolumes"] # 需要全量操作PV
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"] # 需要监控PVC，并更新其状态
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"] # 需要获取storageclasses信息
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"] # 需要记录事件
verbs: ["create", "update", "patch"]
---
## ClusterRoleBinding：将ServiceAccount绑定到ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: nfs-clinet-provisioner-ClusterRoleBinding
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# 必须添加：ServiceAccount所在命名空间
namespace: default
roleRef:
kind: ClusterRole
name: nfs-clinet-provisioner-ClusterRole
apiGroup: rbac.authorization.k8s.io
---
## Role: 命名空间维度的权限，可以和clusterrole合并
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: nfs-clinet-provisioner-Role
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"] # 某些provisioner实现会使用endpoints做leader election
verbs: ["get","list","watch","create","update","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: nfs-clinet-provisioner-RoleBinding
namespace: default
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# 必须添加：ServiceAccount所在命名空间
namespace: default
roleRef:
kind: Role
name: nfs-clinet-provisioner-Role
apiGroup: rbac.authorization.k8s.io
yaml
root@k8s-master1:~# kubectl apply -f rbac.yaml 2.4 部署自动创建PV的pod服务
yaml
root@k8s-master1:~# vim deploy-nfs.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy: # 升级策略
type: Recreate # 当需要更新Pod时，K8S会先删除现有的Pod，再创建新的Pod；这适用于不能同时运行多个副本的有状态服务
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
# 指定pod运行时使用的sa名称,其绑定的RBAC权限决定了provisioner能够访问哪些K8S API资源
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
# 这里使用阿里云镜像仓库中的nfs-subdir-external-provisioner镜像，版本 v4.0.0；该镜像实现了nfs动态供给器
image: registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0
volumeMounts:
- name: nfs-root
# 将nfs共享挂载到容器内的 /persistentvolumes 路径,不建议改；provisioner 会在该目录下为每个pvc创建子目录
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
# 必须与StorageClass中的 provisioner 字段完全一致，这样StorageClass才能调用该供给器
value: k8s-sigs.io/nfs-subdir-external-provisioner
- name: NFS_SERVER
value: 192.168.13.137 # nfs服务器的ip地址
- name: NFS_PATH
value: /data # nfs服务器上导出的共享目录路径
volumes:
- name: nfs-root
nfs:
server: 192.168.13.137
path: /data
yaml
root@k8s-master1:~# kubectl apply -f deploy-nfs.yaml 2.5 创建Redis的Sts
yaml
root@k8s-master1:~# cat redis-cm-svc-pod.yaml
# 1. Redis 配置文件 ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-conf
data:
redis.conf: |
port 6379
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /var/lib/redis # 数据存放目录
protected-mode no
daemonize no
pidfile /var/run/redis.pid
logfile ""
---
# 2. Redis Headless Service（集群必须）
apiVersion: v1
kind: Service
metadata:
name: redis-service
labels:
app: redis
spec:
ports:
- name: redis-port
port: 6379
clusterIP: None # 不分配clusterIP
selector:
app: redis
appCluster: redis-cluster
---
# 3. Redis StatefulSet 有状态部署
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
# 指定关联的Headless Service名称（必须与Service同名）
serviceName: redis-service
# 创建6个Redis实例（3主3从）
replicas: 6
selector:
matchLabels:
app: redis
appCluster: redis-cluster
template:
metadata:
labels:
app: redis
appCluster: redis-cluster
spec:
containers:
- name: redis
image: redis:latest
# 启动命令：执行redis-server
command:
- "redis-server"
# 启动参数：指定配置文件 + 关闭保护模式
args:
- "/etc/redis/redis.conf" # 加载自定义配置
- "--protected-mode" # 关闭保护模式（允许集群通信）
- "no"
# 资源限制：CPU与内存申请
resources:
requests:
cpu: "100m"
memory: "200Mi"
# 声明容器内部监听了什么端口,不提供流量转发与访问暴露
ports:
- containerPort: 6379 # redis客户端连接端口
name: redis
protocol: "TCP"
- containerPort: 16379 # redis 集群节点间通信的端口
name: cluster
protocol: "TCP"
volumeMounts:
- name: redis-config # 挂载配置文件
mountPath: /etc/redis/redis.conf
subPath: redis.conf
- name: redis-data # 挂载数据文件
mountPath: /var/lib/redis
# 挂载ConfigMap配置
volumes:
- name: redis-config
configMap:
name: redis-conf
# 动态存储申请（配合之前的 NFS StorageClass）
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: [ "ReadWriteOnce" ] # 访问模式：单节点读写
resources:
requests:
storage: 1Gi # 申请大小
# 这里使用上面创建的 StorageClass
storageClassName: "managed-nfs-storage"
yaml
root@k8s-master1:~# kubectl apply -f redis-cm-svc-pod.yaml
root@k8s-master1:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 37d
redis-service ClusterIP None <none> 6379/TCP 17s
## 通过创建时间可以看出，sts在创建pod的时候不是同时启动的，而是一个一个的顺序启动
root@k8s-master1:~# kubectl get pod -o wide | grep redis
redis-0 1/1 Running 0 3m53s 10.244.36.108 k8s-node1 <none> <none>
redis-1 1/1 Running 0 3m31s 10.244.169.183 k8s-node2 <none> <none>
redis-2 1/1 Running 0 2m51s 10.244.36.109 k8s-node1 <none> <none>
redis-3 1/1 Running 0 2m26s 10.244.169.184 k8s-node2 <none> <none>
redis-4 1/1 Running 0 2m1s 10.244.36.110 k8s-node1 <none> <none>
redis-5 1/1 Running 0 98s 10.244.169.185 k8s-node2 <none> <none>
## 查看nfs的/data目录，会产生数据
root@NFS:~# ls /data/
default-redis-data-redis-0-pvc-992e88f2-1460-4b48-8096-491dc31fa274 default-redis-data-redis-3-pvc-57b4f9bf-50b6-4057-a844-fed705b1ec8f
default-redis-data-redis-1-pvc-d9414620-ec81-4464-906a-473e54882199 default-redis-data-redis-4-pvc-eab88346-8fc9-4467-ad71-9f044ec412c2
default-redis-data-redis-2-pvc-299b0bc2-c6c8-4a3f-b068-f65b365d9e2d default-redis-data-redis-5-pvc-644ffafe-9d26-4852-81ea-e46e9082f481
root@NFS:~# ls /data/default-redis-data-redis-0-pvc-992e88f2-1460-4b48-8096-491dc31fa274/
appendonlydir nodes.conf
root@NFS:~# 2.6 测试集群内Redis域名解析连通性
StatefulSet 中的每个 Pod 会获得形如 <pod-name>.<service-name>.<namespace>.svc.cluster.local 的稳定 DNS 名称
例如，redis-0.redis.default.svc.cluster.local，即使 Pod 重新调度，该域名也不会改变
我们可以进入 busybox 测试容器，对 Redis 各节点域名进行连通性验证，确保 Kubernetes 集群内部 DNS 解析正常、Pod 网络通信正，为后续 Redis 集群组建提供稳定的网络环境
yaml
root@k8s-master1:~# kubectl run busybox --image=busybox -it -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # ping redis-0.redis.default.svc.cluster.local
ping: bad address 'redis-0.redis.default.svc.cluster.local'
/ # ping redis-0.redis-service.default.svc.cluster.local # ping该域名解析出来的就是redis-0对应的ip 10.244.36.108
PING redis-0.redis-service.default.svc.cluster.local (10.244.36.108): 56 data bytes
64 bytes from 10.244.36.108: seq=0 ttl=63 time=0.648 ms
64 bytes from 10.244.36.108: seq=1 ttl=63 time=0.084 ms
64 bytes from 10.244.36.108: seq=2 ttl=63 time=0.086 ms
^C
--- redis-0.redis-service.default.svc.cluster.local ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.084/0.272/0.648 ms
/ # ping redis-4.redis-service.default.svc.cluster.local # ping该域名解析出来的就是redis-4对应的ip 10.244.36.110
PING redis-4.redis-service.default.svc.cluster.local (10.244.36.110): 56 data bytes
64 bytes from 10.244.36.110: seq=0 ttl=63 time=0.099 ms
64 bytes from 10.244.36.110: seq=1 ttl=63 time=0.107 ms
64 bytes from 10.244.36.110: seq=2 ttl=63 time=0.169 ms
^C
--- redis-4.redis-service.default.svc.cluster.local ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.099/0.125/0.169 ms
/ # 2.7 安装redis-trib
如果是centos系统，需要先运行一个ubuntu的pod，然后在ubuntu中安装redis-trib命令
如果是ubuntu系统，直接安装下列命令即可，不需要运行ubuntu的pod
我的是ubuntu系统，为了方便演示，我这里还是创建了一个ubuntu的pod
yaml
root@k8s-master1:~# kubectl run -it ubuntu --image=ubuntu:18.04 --restart=Never /bin/bash
If you don't see a command prompt, try pressing enter.
## 把软件下载源替换成阿里云镜像源
root@ubuntu:/# cat > /etc/apt/sources.list << EOF
> deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
> deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
>
> deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
> deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
>
> deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
> deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
>
> deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
> deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
>
> deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
> deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
> EOF
root@ubuntu:/# apt update
## 安装相关命令
root@ubuntu:/# apt install -y python2.7 python-pip redis-tools dnsutils
root@ubuntu:/# pip install redis-trib==0.5.1
## 查这个域名在 K8s 里的ip地址，可以测试一下是否正确
root@ubuntu:/# dig +short redis-0.redis-service.default.svc.cluster.local
10.244.36.108
root@ubuntu:/# dig +short redis-5.redis-service.default.svc.cluster.local
10.244.169.185
root@ubuntu:/# 2.8 配置Redis三主三从
在Ubuntu pod中设置redis三主三从
yaml
## 设置redis-0 redis-1 redis-2 为master
root@ubuntu:/# redis-trib.py create \
> `dig +short redis-0.redis-service.default.svc.cluster.local`:6379 \
> `dig +short redis-1.redis-service.default.svc.cluster.local`:6379 \
> `dig +short redis-2.redis-service.default.svc.cluster.local`:6379
## 设置redis-3为redis-0的slave
root@ubuntu:/# redis-trib.py replicate --master-addr `dig +short redis-0.redis-service.default.svc.cluster.local`:6379 --slave-addr `dig +short redis-3.redis-service.default.svc.cluster.local`:6379
## 设置redis-4为redis-1的slave
root@ubuntu:/# redis-trib.py replicate --master-addr `dig +short redis-1.redis-service.default.svc.cluster.local`:6379 --slave-addr `dig +short redis-4.redis-service.default.svc.cluster.local`:6379
## 设置redis-5为redis-2的slave
root@ubuntu:/# redis-trib.py replicate --master-addr `dig +short redis-2.redis-service.default.svc.cluster.local`:6379 --slave-addr `dig +short redis-5.redis-service.default.svc.cluster.local`:6379
yaml
## 退出ubuntu pod，然后随意进入一个redis pod，查看集群信息
root@k8s-master1:~# kubectl exec -it redis-0 /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@redis-0:/data# redis-cli -c
127.0.0.1:6379> cluster nodes
de0431085ec24f3445be041d33911d1ada4d7bda 10.244.169.184:6379@16379 slave c7f02678e65c8a5100dd3a96f82840efff27a3fc 0 1778838395000 3 connected
e193aed56f924e576e6788050b7d285e9756e480 10.244.36.110:6379@16379 slave aac66b559d8f6b738861eff8e5bcdf7ee49789e7 0 1778838396871 1 connected
104beb94505d8a6ef14995f1759dd1db7204f2f0 10.244.169.185:6379@16379 slave 6232737d86561370f931c3b18a3f1170f7173b08 0 1778838396368 2 connected
c7f02678e65c8a5100dd3a96f82840efff27a3fc 10.244.36.108:6379@16379 myself,master - 0 0 3 connected 0-5461
aac66b559d8f6b738861eff8e5bcdf7ee49789e7 10.244.169.183:6379@16379 master - 0 1778838395000 1 connected 10923-16383
6232737d86561370f931c3b18a3f1170f7173b08 10.244.36.109:6379@16379 master - 0 1778838396000 2 connected 5462-10922
127.0.0.1:6379> 2.9 创建用于访问Redis的SVC
在上面我们创建了用于实现sts的headless service，但该svc没有cluster ip，因此不能用于外接访问，所以我们还需要创建一个svc来为redis集群提供访问和负载均衡
yaml
root@k8s-master1:~# vim redis-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: redis-access-service
labels:
app: redis
spec:
type: NodePort
ports:
- name: redis-port
protocol: "TCP"
port: 6379 # 集群内访问端口
targetPort: 6379 # 映射Pod的6379端口
nodePort: 30079 # 外部固定访问端口（范围30000-32767）
selector:
# 多标签为AND关系，精准匹配当前Redis集群Pod
app: redis
appCluster: redis-cluster
yaml
root@k8s-master1:~# kubectl apply -f redis-svc.yaml
root@k8s-master1:~# kubectl get svc | grep redis
redis-access-service NodePort 10.106.169.154 <none> 6379:30079/TCP 3m7s
redis-service ClusterIP None <none> 6379/TCP 50m
root@k8s-master1:~# kubectl describe endpoints redis-access-service
Name: redis-access-service
Namespace: default
Labels: app=redis
Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2026-05-18T00:16:02Z
Subsets:
Addresses: 10.244.169.129,10.244.169.132,10.244.169.133,10.244.36.116,10.244.36.117,10.244.36.120
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
redis-port 6379 TCP
Events: <none>
yaml
## pod内部登录redis，K8s Service 域名只在集群内部 Pod 中生效，宿主机无法直接解析
## 进入业务 Pod 使用服务名 redis-access-service:6379
root@k8s-master1:~# kubectl exec -it redis-0 /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@redis-0:/data# redis-cli -h redis-access-service -p 6379 # 或者redis-cli -h redis-access-service.default.svc.cluster.local -p 6379
redis-access-service:6379> ping
PONG
redis-access-service:6379> quit
root@redis-0:/data# exit
exit
root@k8s-master1:~#
## 宿主机访问redis，使用 NodePort 端口 30079
root@k8s-master1:~# redis-cli -h 127.0.0.1 -p 30079
127.0.0.1:30079> ping
PONG
127.0.0.1:30079> quit
## K8S集群外部访问redis，使用 NodePort 端口 30079
root@NFS:~# redis-cli -h 192.168.13.139 -p 30079
192.168.13.139:30079> ping
PONG
192.168.13.139:30079> quit
root@NFS:~# 至此，K8S部署redis-cluster集群完成！
注：
文中若有疏漏，欢迎大家指正赐教。
本文为100%原创，转载请务必标注原创作者，尊重劳动成果。
求赞、求关注、求评论！你的支持是我更新的最大动力，评论区等你～