K8S Redis-Cluster安装（含redis-cluster-proxy、RedisInsight）

参考：在 K8S 中快速部署 Redis Cluster & Redisinsight

helm redis-cluster

[root@k8s-master01 redis-cluster]# helm search repo redis-cluster
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/redis-cluster   9.1.3           7.2.3           Redis(R) is an open source, scalable, distribut...

[root@k8s-master01 redis-cluster]# helm pull bitnami/redis-cluster

解压，查看使用到的镜像文件，提前下载并上传私有仓库

[root@k8s-master01 redis-cluster]# cat Chart.yaml
annotations:
category: Database
images: |
- name: os-shell
image: docker.io/bitnami/os-shell:11-debian-11-r91
- name: redis-cluster
image: docker.io/bitnami/redis-cluster:7.2.3-debian-11-r1
- name: redis-exporter
image: docker.io/bitnami/redis-exporter:1.55.0-debian-11-r2

配置私有仓库

docker tag docker.io/bitnami/redis-exporter:1.55.0-debian-11-r2  harbor.david.org/bitnami/redis-exporter:1.55.0-debian-11-r2
docker tag docker.io/bitnami/redis-cluster:7.2.3-debian-11-r1  harbor.david.org/bitnami/redis-cluster:7.2.3-debian-11-r1
docker tag docker.io/bitnami/os-shell:11-debian-11-r91  harbor.david.org/bitnami/os-shell:11-debian-11-r91
docker tag docker.io/redislabs/redisinsight:latest  harbor.david.org/redislabs/redisinsight:latest

helm安装redis-cluster

使用之前的storyageClass

StorageClass
helm install redis-cluster bitnami/redis-cluster -n develop \
 --values values.yaml \
 --set persistence.storageClass=nfs-sc \
 --set image.registry=harbor.david.org \
 --set volumePermissions.image.registry=harbor.david.org \
 --set metrics.image.registry=harbor.david.org \
 --set sysctlImage.image.registry=harbor.david.org

安装完成

NAME: redis-cluster
LAST DEPLOYED: Fri Nov 24 14:05:41 2023
NAMESPACE: develop
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 9.1.3
APP VERSION: 7.2.3** Please be patient while the chart is being deployed **


To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "develop" redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

You have deployed a Redis&reg; Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis&reg; cluster:

1. Run a Redis&reg; pod that you can use as a client:
kubectl run --namespace develop redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image harbor.david.org/bitnami/redis-cluster:7.2.3-debian-11-r1 -- bash

2. Connect using the Redis&reg; CLI:

redis-cli -c -h redis-cluster -a $REDIS_PASSWORD

查看密码

[root@k8s-master01 redis-cluster]# kubectl get secret --namespace "develop" redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d

导出所有节点的地址为ClusterIp（如果需要）

参考： VMware Tanzu Application Catalog (Applications Tutorials) Documentation

 helm upgrade redis-cluster --set "password=xxxxxxxxxx,cluster.externalAccess.enabled=true bitnami/redis-cluster -n develop

[root@k8s-master01 redis]# helm upgrade redis-cluster bitnami/redis-cluster --set cluster.externalAccess.enabled=true --set password=An1y6Hh0Cv -n develop
Release "redis-cluster" has been upgraded. Happy Helming!
NAME: redis-cluster
LAST DEPLOYED: Sun Nov 26 18:10:15 2023
NAMESPACE: develop
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 9.1.3
APP VERSION: 7.2.3** Please be patient while the chart is being deployed **


To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "develop" redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

To connect to your Redis&reg; server from outside the cluster check the following information:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace develop -w redis-cluster'

    You will have a different external IP for each Redis&reg; node. Get the external ip from `-external` suffixed services: `kubectl get svc`.
    Redis&reg; port: 6379
    Once the LoadBalancerIPs are ready, you need to provide them and perform a Helm Upgrade:

    helm upgrade --namespace develop redis-cluster --set "cluster.externalAccess.enabled=true,cluster.externalAccess.service.type=LoadBalancer,cluster.externalAccess.service.loadBalancerIP[0]=load-balancerip-0,cluster.externalAccess.service.loadBalancerIP[1]=load-balancerip-1,cluster.externalAccess.service.loadBalancerIP[2]=load-balancerip-2,cluster.externalAccess.service.loadBalancerIP[3]=load-balancerip-3,cluster.externalAccess.service.loadBalancerIP[4]=load-balancerip-4,cluster.externalAccess.service.loadBalancerIP[5]=load-balancerip-5" oci://registry-1.docker.io/bitnamicharts/redis-cluster
    Where loadbalancer-ip-i are the LoadBalancerIPs provided by the cluster.

helm upgrade ingress-nginx导出redis 6379端口

参考：K8s and Redis; a tale of Layer 4 Ingress

helm upgrade ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx \
  --values values.yaml \
  --set tcp.26379="develop/redis-cluster:6379" --debug

修改端口为你想要的端口

 [root@k8s-master01 ~]# kubectl edit svc -n ingress-nginx ingress-nginx-controller
 service/ingress-nginx-controller edited
 [root@k8s-master01 ~]# kubectl get svc -n ingress-nginx

也可以使用port-forward进行临时转发端口，不过得一直挂着

参考：使用端口转发来访问集群中的应用

参考： kubectl port-forward 踩坑记录

 [root@k8s-master01 redis]# kubectl port-forward service/redis-cluster -n develop 30009:6379 --address 0.0.0.0
 Forwarding from 0.0.0.0:30009 -> 6379
 Handling connection for 30009

使用ingress-nginx进行转发（集群方式下不推荐）

因为集群会涉及到不同的key在不同的slot上，

Redis会进行重定向，将内网IP返回给客户端

但客户端并不能访问到内网地址

参考：浅谈Redis Cluster 集群的MOVED、ASK重定向

ingress-nginx导出TCP端口

配置redis-cluster-proxy代理（外网客户端及程序可连接）

参考：【Redis】------引入redis-cluster-proxy使得Redis Cluster对Kubernetes外部可提供服务

参考： Redis系列(十五)、Redis6新特性之集群代理(Cluster Proxy)

linux上安装redis-cluster-proxy

依赖gcc需要提前安装

gcc version 9.3.1

参考： CentOS/Ubuntu安装最新的gcc-9

参考：

【已解决】Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist

或者参考下面的dockerfile

make redis-cluster-proxy

# Make Install 
git clone https://github.com/artix75/redis-cluster-proxy
cd redis-cluster-proxy
make PREFIX=/usr/local/redis_cluster_proxy install 
 dockerfile
 在目录/usr/local/redis_cluster_proxy/bin/ 下复制redis-cluster-proxy到当前目录，然后创建dockerfile 内容如下
 redis_cluster_proxy使用C编写，需要gcc version 9.3.1环境
 https://blog.csdn.net/weixin_43252521/article/details/124409151
[root@master01 redis-cluster-proxy]# vi dockerfile
FROM centos:8

WORKDIR /data

RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
RUN yum makecache
#RUN yum update -y

RUN rpm -ivh https://cbs.centos.org/kojifiles/packages/centos-release-scl-rh/2/3.el7.centos/noarch/centos-release-scl-rh-2-3.el7.centos.noarch.rpm
RUN rpm -ivh https://cbs.centos.org/kojifiles/packages/centos-release-scl/2/3.el7.centos/noarch/centos-release-scl-2-3.el7.centos.noarch.rpm

RUN yum install -y centos-release-scl
RUN yum install -y devtoolset-9-gcc devtoolset-9-gcc-c++

RUN echo "scl enable devtoolset-9 bash" >> /etc/profile
RUN echo "source /opt/rh/devtoolset-9/enable" >> /etc/profile

RUN source /etc/profile

RUN yum install -y wget
RUN wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
RUN sed -i  's/$releasever/8/g' /etc/yum.repos.d/CentOS-Base.repo
RUN yum repolist

RUN yum install -y  gcc
RUN gcc -v

ADD redis-cluster-proxy /usr/local/bin/

EXPOSE 7777

 docker build . -t redis-cluster-proxy:v1.0.0

 [root@k8s-master01 ~]# docker tag redis-cluster-proxy:v1.0.0  harbor.david.org/redislabs/redis-cluster-proxy:v1.0.0
 [root@k8s-master01 ~]# docker push  harbor.david.org/redislabs/redis-cluster-proxy:v1.0.0

配置redis-cluster-proxy configmap

[root@master01 redis-cluster-proxy]# vi configMap.yaml
---
# Redis-Proxy Config
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-proxy
  namespace: develop
data:
  proxy.conf: |
    cluster redis-cluster:6379     # 配置为Redis Cluster Service
    bind 0.0.0.0
    port 7777   # redis-cluster-proxy 对外暴露端口
    threads 8   # 线程数量
    daemonize no
    enable-cross-slot yes
    auth An1y6Hh0Cv     # 配置Redis Cluster 认证密码
    log-level error
[root@master01 redis-cluster-proxy]#
kubectl apply -f configMap.yaml  -n develop

配置deployment.yaml

[root@master01 redis-cluster-proxy]# vi deployment.yaml
---
# Redis-Proxy NodePort
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-proxy
  namespace: develop
spec:
  type: NodePort # 对K8S外部提供服务
  ports:
  - name: redis-cluster-proxy
    nodePort: 30009   # 对外提供的端口
    port: 7777
    protocol: TCP
    targetPort: 7777
  selector:
    app: redis-cluster-proxy
---
# Redis-Proxy Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-cluster-proxy
  namespace: develop
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-proxy
  template:
    metadata:
      labels:
        app: redis-cluster-proxy
    spec:
      #imagePullSecrets:
      #  - name: harbor
      containers:
        - name: redis-cluster-proxy
          image: harbor.david.org/redislabs/redis-cluster-proxy:v1.0.0
          imagePullPolicy: Always
          command: ["redis-cluster-proxy"]
          args:
            - -c
            - /data/proxy.conf   # 指定启动配置文件
          ports:
            - name: redis-7777
              containerPort: 7777
              protocol: TCP
          volumeMounts:
            - name: redis-proxy-conf
              mountPath: /data/
      volumes:   # 挂载proxy配置文件
        - name: redis-proxy-conf
          configMap:
            name: redis-cluster-proxy

[root@master01 redis-cluster-proxy]#
kubectl apply -f deployment.yaml  -n develop

[root@master01 redis-cluster-proxy]# kubectl get pods -n develop
NAME                                   READY   STATUS    RESTARTS        AGE
redis-cluster-proxy-6679cb8f78-nbgr4   1/1     Running   0               167m

[root@master01 redis-cluster-proxy]# kubectl get svc -n develop |grep redis
redis-cluster                   ClusterIP   10.102.144.124   <none>        6379/TCP                        5d21h
redis-cluster-headless          ClusterIP   None             <none>        6379/TCP,16379/TCP              5d21h
redis-cluster-proxy             NodePort    10.103.13.20     <none>        7777:30009/TCP                  167m

安装redisinsight

参考：redisinsight

打开网页右上角的 install

 helm repo add redisinsight https://spy86.github.io/redisinsight

[root@k8s-master01 ~]# helm search hub redisinsight -o json
[{"url":"https://artifacthub.io/packages/helm/redisinsight/redisinsight","version":"0.1.0","app_version":"0.1.0","description":"A Helm chart for Redis Insight.","repository":{"url":"https://spy86.github.io/redisinsight","name":"redisinsight"}},{"url":"https://artifacthub.io/packages/helm/truecharts/redisinsight","version":"3.0.13","app_version":"1.14.0","description":"GUI for Redis.","repository":{"url":"https://charts.truecharts.org/","name":"truecharts"}},{"url":"https://artifacthub.io/packages/helm/riftbit/redisinsight","version":"0.1.0","app_version":"v1.0.0","description":"RedisInsight - The GUI for Redis","repository":{"url":"https://charts.riftbit.com/","name":"riftbit"}},{"url":"https://artifacthub.io/packages/helm/heywood8-helm-charts/redisinsight","version":"0.4.5","app_version":"2.28.0","description":"A Redisinsight Helm chart for kubernetes","repository":{"url":"https://heywood8.github.io/helm-charts/","name":"heywood8-helm-charts"}},{"url":"https://artifacthub.io/packages/helm/eximiait/openshift-secured-redisinsight","version":"0.9.0","app_version":"","description":"A secured version of RedisInsight for OpenShift","repository":{"url":"https://charts.eximiait.com.ar","name":"eximiait"}}]
[root@k8s-master01 ~]#

[root@k8s-master01 ~]# helm search repo redisinsight
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
redisinsight/redisinsight       0.1.0           0.1.0           A Helm chart for Redis Insight.
[root@k8s-master01 ~]#

[root@k8s-master01 ~]# helm pull redisinsight/redisinsight

解压，查看values.yaml

helm install redisinsight redisinsight/redisinsight -n develop \
--values values.yaml \
--set namespace.name=develop \
--set image.repository=harbor.david.org/redislabs/redisinsight

NAME: redisinsight
LAST DEPLOYED: Fri Nov 24 14:36:26 2023
NAMESPACE: develop
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace develop -l "app.kubernetes.io/name=redisinsight,app.kubernetes.io/instance=redisinsight" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8001 to use your application"
  kubectl --namespace develop port-forward $POD_NAME 8001:8001

配置ingress-nginx转发

vi ingress-nginx.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: redisinsight-ingress
  namespace: develop
spec:
  ingressClassName: nginx
  rules: # 一个ingress可以配置多个rules
    - host: redisinsight.david.org # 域名配置，可以不写，匹配*，或者写 *.bar.com
      http:        
        paths: # 相当于nginx的location，同一个host可以配置多个path
          - pathType: Prefix
            backend:
              service:
                name: redisinsight  # 代理到哪个svc
                port:
                  number: 80 # svc的端口
            path: /

应用

kubectl apply -f ingress-nginx.yaml

配置hosts

192.168.221.132 redisinsight.david.org

访问测试

http://redisinsight.david.org