Kubernetes 构建高可用、高性能 Redis 集群实战指南

**1.**部署方案规划

1.1 部署架构图

1.2****前提说明

• 本实战环境使用 NFS 作为 k8s 集群的持久化存储
• Redis 集群所有资源部署在命名空间 chengke 内。

2.创建ConfigMap

2.1创建ConfigMap

2.1.1创建Redis****配置文件

[root@k8s-m1 1]# cat redis-cluster-cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-config
data:
  redis-config: |
    appendonly yes
    protected-mode no
    dir /data
    port 6379
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 5000
    masterauth Chengke2025
    requirepass Chengke2025

2.1.2****创建资源

[root@k8s-m1 1]# kubectl apply -f redis-cluster-cm.yaml 

2.1.3****验证资源

[root@k8s-m1 1]# kubectl get cm
NAME                   DATA   AGE
redis-cluster-config   1      68m

2.2创建Redis

本文使用 StatefulSet 部署 Redis 服务，需要创建 StatefulSet 和 HeadLess 两种资源。

2.2.1****创建资源清单

[root@k8s-m1 1]# cat redis-cluster-sts.yaml 
apiVersion: v1
kind: Service
metadata:
  name: redis-headless
  labels:
    app.kubernetes.io/name: redis-cluster
spec:
  ports:
    - name: redis-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
  selector:
    app.kubernetes.io/name: redis-cluster
  clusterIP: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
  labels:
    app.kubernetes.io/name: redis-cluster
spec:
  serviceName: redis-headless
  replicas: 6
  selector:
    matchLabels:
      app.kubernetes.io/name: redis-cluster
  template:
    metadata:
      labels:
        app.kubernetes.io/name: redis-cluster
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app.kubernetes.io/name
                      operator: In
                      values:
                        - redis-cluster
                topologyKey: kubernetes.io/hostname
      containers:
        - name: redis
          image: redis:8.0.2
          imagePullPolicy: IfNotPresent
          command:
            - "redis-server"
          args:
            - "/etc/redis/redis.conf"
            - "--protected-mode"
            - "no"
            - "--cluster-announce-ip"
            - "$(POD_IP)"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: redis-6379
              containerPort: 6379
              protocol: TCP
          volumeMounts:
            - name: config
              mountPath: /etc/redis
            - name: redis-cluster-data
              mountPath: /data
          resources:
            requests:
              cpu: 50m
              memory: 500Mi
            limits:
              cpu: "2"
              memory: 4Gi
      volumes:
        - name: config
          configMap:
            name: redis-cluster-config
            items:
              - key: redis-config
                path: redis.conf
  volumeClaimTemplates:
    - metadata:
        name: redis-cluster-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: nfs-client
        resources:
          requests:
            storage: 5Gi

注意： POD_IP 是重点，如果不配置会导致线上的 POD 重启换 IP 后，集群状态无法自动同步

2.2.2****创建资源

[root@k8s-master01 1]# kubectl apply -f redis-cluster-sts.yaml
service/redis-headless created
statefulset.apps/redis-cluster created

2.2.3****验证资源

执行下面的命令，查看 StatefulSet、Pod、Service 创建结果

[root@k8s-m1 1]# kubectl get pod,svc,sts
NAME                                READY   STATUS    RESTARTS   AGE
pod/redis-cluster-0                 1/1     Running   0          69m
pod/redis-cluster-1                 1/1     Running   0          67m
pod/redis-cluster-2                 1/1     Running   0          48m
pod/redis-cluster-3                 1/1     Running   0          65m
pod/redis-cluster-4                 1/1     Running   0          65m
pod/redis-cluster-5                 1/1     Running   0          65m
pod/redisinsight-5cb887744f-79h5k   1/1     Running   0          42m

NAME                             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/kubernetes               ClusterIP   10.0.0.1     <none>        443/TCP          8d
service/redis-cluster-external   NodePort    10.5.96.53   <none>        6379:31379/TCP   62m
service/redis-headless           ClusterIP   None         <none>        6379/TCP         69m
service/redisinsight-external    NodePort    10.9.143.0   <none>        5540:31380/TCP   42m

NAME                             READY   AGE
statefulset.apps/redis-cluster   6/6     69m

2.3创建k8s****集群外部访问服务

2.3.1****编写资源

我们采用 NodePort 方式在 Kubernetes 集群外发布 Redis 服务，指定的端口为 31379。

[root@k8s-m1 1]# cat redis-cluster-svc-external.yaml 
kind: Service
apiVersion: v1
metadata:
  name: redis-cluster-external
  labels:
    app: redis-cluster-external
spec:
  ports:
    - protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 31379
  selector:
    app.kubernetes.io/name: redis-cluster
  type: NodePort

2.3.2****创建资源

[root@k8s-master01 1]# kubectl apply -f redis-cluster-svc-external.yaml
service/redis-cluster-external created

2.3.3****验证资源

执行下面的命令，查看 Service 创建结果

[root@k8s-m1 1]# kubectl get endpointslice
NAME                           ADDRESSTYPE   PORTS   ENDPOINTS                                                 AGE
kubernetes                     IPv4          6443    192.168.10.11                                             24d
my-service-noselector-1        IPv4          80      192.168.10.12,192.168.10.13                               18d
redis-cluster-external-qvl7h   IPv4          6379    10.244.215.78,10.244.215.111,10.244.111.244 + 3 more...   65m
redis-headless-j2x96           IPv4          6379    10.244.111.251,10.244.215.111,10.244.215.73 + 3 more...   73m
redisinsight-external-b2vks    IPv4          5540    10.244.111.252                                            45m
[root@k8s-m1 1]# kubectl get svc
NAME                     TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
kubernetes               ClusterIP   10.0.0.1     <none>        443/TCP          8d
redis-cluster-external   NodePort    10.5.96.53   <none>        6379:31379/TCP   65m
redis-headless           ClusterIP   None         <none>        6379/TCP         73m
redisinsight-external    NodePort    10.9.143.0   <none>        5540:31380/TCP   46m

3.创建Redis****集群

Redis POD 创建完成后，不会自动创建 Redis 集群，需要手工执行集群初始化的命令，有自动创建和手工创建两种方式，二选一，建议选择自动。

3.1自动创建Redis****集群

执行下面的命令，自动创建 3 个 master 和 3 个 slave 的集群，中间需要输入一次 yes。

[root@k8s-m1 1]# kubectl exec -it redis-cluster-0 -- redis-cli -a Chengke2025 --cluster create --cluster-replicas 1 $(kubectl get pods -l app.kubernetes.io/name=redis-cluster -o jsonpath='{range.items[*]} {.status.podIP}:6379 {end}')
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.244.111.244:6379 to 10.244.111.251:6379
Adding replica 10.244.215.78:6379 to 10.244.215.111:6379
Adding replica 10.244.215.73:6379 to 10.244.111.253:6379
M: 80be000a17cc1b2d1d7180e722876d501c90c9a3 10.244.111.251:6379
   slots:[0-5460] (5461 slots) master
M: 01a21392ef5998e7da5d382a1ed6d0ec35bef845 10.244.215.111:6379
   slots:[5461-10922] (5462 slots) master
M: 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31 10.244.111.253:6379
   slots:[10923-16383] (5461 slots) master
S: 48978e7c86fe9bf341880eb9cf1fac3aa8e48dc4 10.244.215.73:6379
   replicates 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31
S: 0d8bab081a2125b7a83c67c588c2a9f93482d69a 10.244.111.244:6379
   replicates 80be000a17cc1b2d1d7180e722876d501c90c9a3
S: d7f1dc037259b2d2c155ac5a5c5c5b4b8f7818c4 10.244.215.78:6379
   replicates 01a21392ef5998e7da5d382a1ed6d0ec35bef845
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 10.244.111.251:6379)
M: 80be000a17cc1b2d1d7180e722876d501c90c9a3 10.244.111.251:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 48978e7c86fe9bf341880eb9cf1fac3aa8e48dc4 10.244.215.73:6379
   slots: (0 slots) slave
   replicates 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31
S: 0d8bab081a2125b7a83c67c588c2a9f93482d69a 10.244.111.244:6379
   slots: (0 slots) slave
   replicates 80be000a17cc1b2d1d7180e722876d501c90c9a3
M: 8e009ec3ccc37b0810db6b1ffc5bd6c241c0cd31 10.244.111.253:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 01a21392ef5998e7da5d382a1ed6d0ec35bef845 10.244.215.111:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: d7f1dc037259b2d2c155ac5a5c5c5b4b8f7818c4 10.244.215.78:6379
   slots: (0 slots) slave
   replicates 01a21392ef5998e7da5d382a1ed6d0ec35bef845
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

3.2****验证集群状态

[root@k8s-m1 1]# kubectl exec -it redis-cluster-0 -- redis-cli -p 6379 -a Chengke2025 cluster info
E0802 16:18:19.072387   92106 websocket.go:297] Unknown stream id 1, discarding message
                                                                                       Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:6868
cluster_stats_messages_pong_sent:6767
cluster_stats_messages_sent:13635
cluster_stats_messages_ping_received:6762
cluster_stats_messages_pong_received:6867
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:13634
total_cluster_links_buffer_limit_exceeded:0

**4.**集群功能测试

4.1****压力测试

使用 Redis 自带的压力测试工具，测试 Redis 集群是否可用，并简单测试性能。
使用 set 和get命令，发送100000次请求，每个请求包含一个键值对，其中键是随机生成的，值的大小是100字节，同时有20个客户端并发执行。

测试set场景：

kubectl exec -it redis-cluster-0 -- redis-benchmark -h 192.168.10.11 -p 31379 -a Chengke2025 -t set -n 100000 -c 20 -d 100 --cluster

测试get场景：

[root@k8s-master01 1]# kubectl exec -it redis-cluster-0 -- redis-benchmark -h
192.168.10.11 -p 31379 -a Chengke2025 -t get -n 100000 -c 20 -d 100 --cluster

4.2****故障切换测试

4.2.1. 测试场景1：

测试：手动删除一个 Master 的 Slave，观察 Slave Pod 是否会自动重建并加入原有 Master。

**结果：**原有 Slave IP 删除后自动重建，IP 变更为 **一个新的IP，**并自动加入原有的Master。

4.2.2. 测试场景2：

测试：手动删除 Master ，观察 Master Pod 是否会自动重建并重新变成 Master。
结果：原有 Master IP 删除后自动重建， IP变更为一个新的IP，并重新变成****Master

以上测试内容，仅是简单的故障切换测试，生产环境请增加更多的测试场景！！！