Helm部署高可用redis k8s分片集群

安装部署

添加bitnami仓库并查找redis

helm repo add bitnami https://charts.bitnami.com/bitnami

helm repo update
[kmning@k8s-register-node ~]$ helm search repo redis
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/redis           17.10.1         7.0.11          Redis(R) is an open source, advanced key-value ...
bitnami/redis-cluster   8.4.4           7.0.11          Redis(R) is an open source, scalable, distribut...

复制代码

拉取chat到本地

helm pull bitnami/redis-cluster --version 8.4.4

tar -zxvf redis-cluster-8.4.4.tgz

cp redis-cluster/values.yaml ./values-test.yaml

对本地values-test.yaml进行修改，配置非常多，根据实际情况进行修改，比如我主要修改了如下内容

global:
  imageRegistry: "xxx.com:443"
  ## E.g.
  ## imagePullSecrets:
  ##   - myRegistryKeySecretName
  ##
  imagePullSecrets: []
  storageClass: "managed-nfs-storage"
  redis:
    password: "pwd"

所有用到image的地方改成私服
image:
  registry: xxx:443
  repository: lib-proxy/bitnami/redis-cluster
  ## Bitnami Redis&reg; image tag
  ## ref: https://github.com/bitnami/containers/tree/main/bitnami/redis#supported-tags-and-respective-dockerfile-links
  ##
  tag: 7.0.11-debian-11-r0

persistence:
  ## @param persistence.enabled Enable persistence on Redis&reg;
  ## If enabled, nodes are using Persistent Volume Claims
  ## If disabled, an emptyDir volume is used
  ##
  enabled: true
  ## @param persistence.path Path to mount the volume at, to use other images Redis&reg; images.
  ##
  path: /bitnami/redis/data
  ## @param persistence.subPath The subdirectory of the volume to mount to, useful in dev environments and one PV for multiple services
  ##
  subPath: ""
  ## @param persistence.storageClass Storage class of backing PVC
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "managed-nfs-storage"

helm安装redis集群

kubectl create ns redis
helm -n redis install redis-cluster  redis-cluster-8.4.4.tgz  -f values-test.yaml \
--set useBundledSystemChart=true

安装后打印

kmning@k8s-master-1:~/redis-k8s-cluster$ helm -n redis install redis-cluster  redis-cluster-8.4.4.tgz  -f values-test.yaml \
> --set useBundledSystemChart=true
NAME: redis-cluster
LAST DEPLOYED: Thu Apr 27 08:44:02 2023
NAMESPACE: redis
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 8.4.4
APP VERSION: 7.0.11** Please be patient while the chart is being deployed **


To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "redis" redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

You have deployed a Redis&reg; Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis&reg; cluster:

1. Run a Redis&reg; pod that you can use as a client:
kubectl run --namespace redis redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image k8s-register-node.com:443/lib-proxy/bitnami/redis-cluster:7.0.11-debian-11-r0 -- bash

2. Connect using the Redis&reg; CLI:

redis-cli -c -h redis-cluster -a $REDIS_PASSWORD

查看已安装chat

kmning@k8s-master-1:~/redis-k8s-cluster$ helm -n redis list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
redis-cluster   redis           1               2023-04-27 08:33:55.88999883 +0000 UTC  deployed        redis-cluster-8.4.4     7.0.11

查看服务部署情况

kmning@k8s-master-1:~/redis-k8s-cluster$ kubectl get sts -n redis
NAME            READY   AGE
redis-cluster   6/6     4m27s
euht@k8s-master-1:~/redis-k8s-cluster$ kubectl get pods -n redis
NAME              READY   STATUS    RESTARTS        AGE
redis-cluster-0   1/1     Running   1 (3m26s ago)   4m33s
redis-cluster-1   1/1     Running   1 (3m18s ago)   4m33s
redis-cluster-2   1/1     Running   0               4m33s
redis-cluster-3   1/1     Running   0               4m33s
redis-cluster-4   1/1     Running   1 (3m25s ago)   4m33s
redis-cluster-5   1/1     Running   1 (3m6s ago)    4m33s
euht@k8s-master-1:~/redis-k8s-cluster$ kubectl get svc -n redis
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
redis-cluster            ClusterIP   10.43.252.164   <none>        6379/TCP             5m23s
redis-cluster-headless   ClusterIP   None            <none>        6379/TCP,16379/TCP   5m24s
kmning@k8s-master-1:~/redis-k8s-cluster$ kubectl get pvc -n redis
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
redis-data-redis-cluster-0   Bound    pvc-42ac9dea-3dda-4c43-90c6-4cd59f5af8b9   8Gi        RWO            managed-nfs-storage   15m
redis-data-redis-cluster-1   Bound    pvc-3c5cc893-dc6d-4f2a-b485-00544564a5b5   8Gi        RWO            managed-nfs-storage   15m
redis-data-redis-cluster-2   Bound    pvc-064448ee-240e-4658-b161-c96da01c977d   8Gi        RWO            managed-nfs-storage   15m
redis-data-redis-cluster-3   Bound    pvc-198311a6-2c25-41d6-ae0f-6e6c055b1a20   8Gi        RWO            managed-nfs-storage   15m
redis-data-redis-cluster-4   Bound    pvc-419b04c2-ff80-4907-aafc-3adaf33242a2   8Gi        RWO            managed-nfs-storage   15m
redis-data-redis-cluster-5   Bound    pvc-92a0de8f-d073-4430-9c64-a06e86aa3041   8Gi        RWO            managed-nfs-storage   15m

看起来一切正常。默认集群主节点为redis-cluster-0、redis-cluster-2、redis-cluster-4，而集群从节点为redis-cluster-1、redis-cluster-3、redis-cluster-5。

服务验证

kubectl exec -it redis-cluster-0 bash -n redis


I have no name!@redis-cluster-client:/$  redis-cli -c -h redis-cluster -a $REDIS_PASSWORD
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-cluster:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:1022
cluster_stats_messages_pong_sent:1006
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:2029
cluster_stats_messages_ping_received:1006
cluster_stats_messages_pong_received:1023
cluster_stats_messages_received:2029
total_cluster_links_buffer_limit_exceeded:0
redis-cluster:6379> cluster nodes
dcc21275c8dc404d2fd3d6c18c719face4640e07 10.42.2.38:6379@16379 slave b118e1f8eb353849abe08749540235264887918c 0 1682586093120 3 connected
358ddf185fe7b9f3c88745a15edeb7f513c6e695 10.42.4.15:6379@16379 slave 10eb0915585400a27b725f721d67b7bb11e8babf 0 1682586094125 2 connected
922c7d7ce319276b4693192372d8b217b608d733 10.42.3.17:6379@16379 slave 012396047e02fc8c890dfb838dbb6e3d8e497eb9 0 1682586088098 1 connected
012396047e02fc8c890dfb838dbb6e3d8e497eb9 10.42.0.38:6379@16379 master - 0 1682586092117 1 connected 0-5460
10eb0915585400a27b725f721d67b7bb11e8babf 10.42.1.30:6379@16379 master - 0 1682586095129 2 connected 5461-10922
b118e1f8eb353849abe08749540235264887918c 10.42.5.13:6379@16379 myself,master - 0 1682586045000 3 connected 10923-16383
redis-cluster:6379>

从日志可以看出，三主三从的分片集群已完成搭建。

客户端使用

通过上面的安装，我们已经将Service暴露出来

kmning@k8s-master-1:~/redis-k8s-cluster$ kubectl get svc -n redis
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
redis-cluster            ClusterIP   10.43.252.164   <none>        6379/TCP             5m23s
redis-cluster-headless   ClusterIP   None            <none>        6379/TCP,16379/TCP   5m24s

可以看出，暴露了一个可负载均衡访问的Service和一个无头Service，可根据需要去使用。正常来讲，我们只需要以分片集群的方式连接可负载均衡的Service即可。

如本文的情况，redis cluster的每个节点都是一个跑在k8s里面的pod，这些pod并不能被外部直接访问，而是通过ingress等方法对外暴露一个访问接口，即只有一个统一的ip：port给外部访问。经由k8s的调度，对这个统一接口的访问会被发送到redis集群的某个节点。这时候对redis的用户来说，看起来这就像是一个单节点的redis。但是，此时无论是直接使用命令行工具redis-cli，还是某种语言的sdk，还是需要按照集群来配置redis的连接信息，才能正确连接，例如

spring:
  redis:
    cluster:
      nodes: redis-cluster.redis.svc.cluster.local:6379
    password: yourpwd

当然，这里也可以使用无头Service去灵活连接，把nodes的内容改下如下即可

redis-cluster-0.redis-cluster-headless.redis.svc.cluster.local,redis-cluster-1.redis-cluster-headless.redis.svc.cluster.local,redis-cluster-2.redis-cluster-headless.redis.svc.cluster.local,redis-cluster-3.redis-cluster-headless.redis.svc.cluster.local,redis-cluster-4.redis-cluster-headless.redis.svc.cluster.local,redis-cluster-5.redis-cluster-headless.redis.svc.cluster.local