K8S下redis哨兵集群使用secret隐藏configmap内明文密码方案详解

斯普信专业组2025-02-21 17:00

#作者：朱雷

文章目录

  • 一、背景环境及方案说明
  • [二、redis secret configmap deployment参考](#二、redis secret configmap deployment参考)
    • [2.1 创建secret-redis.yaml参考](#2.1 创建secret-redis.yaml参考)
    • [2.2 修改configmap配置参考](#2.2 修改configmap配置参考)
      • [2.2.1 哨兵节点修改（每个节点都修改）](#2.2.1 哨兵节点修改（每个节点都修改）)
      • [2.2.2 主从节点配置修改](#2.2.2 主从节点配置修改)
      • [2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）](#2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）)
    • [2.3 修改deployment配置参考](#2.3 修改deployment配置参考)
      • [2.3.1 master&slaves节点的deployment的yaml修改](#2.3.1 master&slaves节点的deployment的yaml修改)
      • [2.3.2 哨兵节点的deployment的yaml修改](#2.3.2 哨兵节点的deployment的yaml修改)
      • [2.3.4 镜像环境变量参考](#2.3.4 镜像环境变量参考)
  • 三、测试是否生效
  • [四、 注意事项](#四、 注意事项)

一、背景环境及方案说明

Redis 哨兵configmap里包含明文密码，需要处理不在configmap内显示明文密码。

1.1、环境说明

方案基于Redis-5.0.14 哨兵版本（5.x，6.x版本兼容）

方案基于redis-sentinel-exporter-5.0.8版本

方案基于容器环境变量

使用两种方案的其中任意一种均可实现

1.2、方案一：使用配置文件设置密码

参照 2.2.1 和2.2.2 修改configmap

参照 2.3.1 和2.3.2 修改deployment

1.3、方案二：使用args 的命令行传参设置密码

参照 2.2.3.1 和2.2.3.2 修改configmap

参照 2.3.3.1 和2.3.3.2 修改deployment

二、redis secret configmap deployment参考

2.1 创建secret-redis.yaml参考

复制代码 
${}内为redis密码的base64的编码，若认证和同步密码不一致分别定义即可
apiVersion: v1
data:
  password: ${aGFyYm9yMjM0NSM=}
kind: Secret
metadata:
  name: redis-auth-secret
  namespace: paas-middleware

2.2 修改configmap配置参考

2.2.1 哨兵节点修改（每个节点都修改）

复制代码 
apiVersion: v1
data:
  redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
        cp /etc/redis/redis.conf /redis-conf/redis.conf
        **echo -e "sentinel auth-pass mymaster ${REDIS_PASSWORD}" >> /redis-conf/redis.conf
    fi**
    redis-sentinel /redis-conf/redis.conf $@
  redis.conf: |
    port 26379
    protected-mode no
    daemonize no
    sentinel monitor mymaster 169.169.164.253 6379 2
    sentinel down-after-milliseconds mymaster 15000
    sentinel failover-timeout mymaster 60000
    sentinel deny-scripts-reconfig yes
    sentinel parallel-syncs mymaster 2
    sentinel auth-pass mymaster somepassword   # 删除这行配置
kind: ConfigMap
metadata:
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
  每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

2.2.2 主从节点配置修改

复制代码 
apiVersion: v1
data:
  redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
        cp /etc/redis/redis.conf /redis-conf/redis.conf
        **echo -e "masterauth ${REDIS_MASTER_PASSWORD}" >> /redis-conf/redis.conf
        echo -e "requirepass ${REDIS_PASSWORD}" >> /redis-conf/redis.conf**
    fi
    redis-server /redis-conf/redis.conf $@
  redis.conf: |
    bind 0.0.0.0 ::
    port 6379
    daemonize no
    protected-mode no
    timeout 300
    tcp-keepalive 300
    replica-read-only yes
    replica-serve-stale-data yes
    maxclients 20000
    maxmemory 0
    maxmemory-policy noeviction
    masterauth somepassword  # 删除此行配置
    requirepass somepassword  # 删除此行配置
    rename-command FLUSHALL ""
    dir "/data/"
    pidfile "/data/redis.pid"
    logfile "/data/redis.log"
kind: ConfigMap
metadata:
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
 所有主从configmap配置文件都修改， 有****不带删除线的为新增行

2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）

  1. 以下为哨兵节点configmap 修改

    apiVersion: v1
    data:
    redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
    cp /etc/redis/redis.conf /redis-conf/redis.conf
    fi
    redis-sentinel /redis-conf/redis.conf $@
    redis.conf: |
    port 26379
    protected-mode no
    daemonize no
    sentinel monitor mymaster 169.169.164.253 6379 2
    sentinel down-after-milliseconds mymaster 15000
    sentinel failover-timeout mymaster 60000
    sentinel deny-scripts-reconfig yes
    sentinel parallel-syncs mymaster 2
    sentinel auth-pass mymaster somepassword # 删除这行配置
    kind: ConfigMap
    metadata:
    labels:
    app: redis-base-1
    type: redis
    name: redis-base-1-sentinel-1
    namespace: paas-middleware
    每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

  2. 以下为主从节点configmap 修改

    下面为主从节点实例configmap修改，有****不带删除线为新增行
    apiVersion: v1
    data:
    redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
    cp /etc/redis/redis.conf /redis-conf/redis.conf
    fi
    redis-server /redis-conf/redis.conf $@
    redis.conf: |
    bind 0.0.0.0 ::
    port 6379
    daemonize no
    protected-mode no
    timeout 300
    tcp-keepalive 300
    replica-read-only yes
    replica-serve-stale-data yes
    maxclients 20000
    maxmemory 0
    maxmemory-policy noeviction
    masterauth somepassword # 删除此行配置
    requirepass somepassword # 删除此行配置
    rename-command FLUSHALL ""
    dir "/data/"
    pidfile "/data/redis.pid"
    logfile "/data/redis.log"
    kind: ConfigMap
    metadata:
    labels:
    app: redis-base-1
    type: redis
    name: redis-base-1-master
    namespace: paas-middleware

2.3 修改deployment配置参考

2.3.1 master&slaves节点的deployment的yaml修改

复制代码 
所有主从节点配置文件都修改，  有**xxx**为新增行
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-master
      servicename: redis-base-1
      type: redis
      withexporter: "yes"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9121"
        prometheus.io/scrape: "true"
      labels:
        app: redis-base-1
        name: redis-base-1-master
        servicename: redis-base-1
        type: redis
        withexporter: "yes"
    spec:
      containers:
      - args:
        - --replica-announce-ip
        - 169.169.164.253
        - --replica-announce-port
        - "6379"
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 6379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      - args:
        - --redis.addr
        - redis://localhost:6379
        - --redis.password
        - somepassword  #密码替换成 $(REDIS_PASSWORD) 变量
        - **$(REDIS_PASSWORD)**
        - --web.listen-address
        - 0.0.0.0:9121
        image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest
        imagePullPolicy: Always
        name: redis-exporter
		**env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**
        ports:
        - containerPort: 9121
          name: redis-exporter
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-master
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-master/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-master/redis-conf
          type: ""
        name: actual-config

2.3.2 哨兵节点的deployment的yaml修改

复制代码 
所有哨兵节点配置文件都修改， 有****为新增行
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-sentinel-1
      role: sentinel
      type: redis
      withexporter: "no"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-base-1
        name: redis-base-1-sentinel-1
        role: sentinel
        type: redis
        withexporter: "no"
    spec:
      containers:
      - args:
        - --sentinel
        - announce-ip
        - 169.169.196.242
        - --replica-announce-port
        - "26379"
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 26379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-sentinel-1
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/redis-conf
          type: ""
        name: actual-config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-11-09T03:25:41Z"
    lastUpdateTime: "2023-11-09T03:25:43Z"
    message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-07-21T16:48:34Z"
    lastUpdateTime: "2024-07-21T16:48:34Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

2.3.3 使用命令行参数指定密码（本小节与上面两小节互斥）

  1. 哨兵节点deployment 修改
    所有哨兵节点配置文件都修改， 有****不带删除线的为新增行

    apiVersion: apps/v1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    labels:
    app: redis-base-1
    type: redis
    name: redis-base-1-sentinel-1
    namespace: paas-middleware
    spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
    matchLabels:
    app: redis-base-1
    name: redis-base-1-sentinel-1
    role: sentinel
    type: redis
    withexporter: "no"
    strategy:
    rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
    type: RollingUpdate
    template:
    metadata:
    creationTimestamp: null
    labels:
    app: redis-base-1
    name: redis-base-1-sentinel-1
    role: sentinel
    type: redis
    withexporter: "no"
    spec:
    containers:
    - args:
    - --sentinel
    - announce-ip
    - 169.169.196.242
    - --replica-announce-port
    - "26379"
    - --sentinel
    - auth-pass
    - mymaster
    - $(REDIS_PASSWORD)
    command:
    - /etc/redis/redis-docker-entrypoint.sh
    image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
    imagePullPolicy: Always
    name: redis
    env:
    - name: REDIS_PASSWORD
    valueFrom:
    secretKeyRef:
    name: redis-auth-secret
    key: password
    - name: REDIS_MASTER_PASSWORD
    valueFrom:
    secretKeyRef:
    name: redis-auth-secret
    key: password

    复制代码 
         ports:
     - containerPort: 26379
       name: client
       protocol: TCP
     resources: {}
     terminationMessagePath: /dev/termination-log
     terminationMessagePolicy: File
     volumeMounts:
     - mountPath: /etc/redis/
       name: config
     - mountPath: /data
       name: data
     - mountPath: /redis-conf
       name: actual-config
   dnsPolicy: ClusterFirst
   nodeSelector:
     kubernetes.io/hostname: 10.179.75.111
   restartPolicy: Always
   schedulerName: default-scheduler
   securityContext: {}
   terminationGracePeriodSeconds: 30
   volumes:
   - configMap:
       defaultMode: 509
       name: redis-base-1-sentinel-1
     name: config
   - hostPath:
       path: /data/redis/redis-base-1-sentinel-1/data
       type: ""
     name: data
   - hostPath:
       path: /data/redis/redis-base-1-sentinel-1/redis-conf
       type: ""
     name: actual-config

    status:
    availableReplicas: 1
    conditions:

    • lastTransitionTime: "2023-11-09T03:25:41Z"
      lastUpdateTime: "2023-11-09T03:25:43Z"
      message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
    • lastTransitionTime: "2024-07-21T16:48:34Z"
      lastUpdateTime: "2024-07-21T16:48:34Z"
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
      observedGeneration: 3
      readyReplicas: 1
      replicas: 1
      updatedReplicas: 1

2、以下为主从实例deployment 配置修改

所有主从节点配置文件都修改， 有****不带删除线的为新增行

复制代码 
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-master
      servicename: redis-base-1
      type: redis
      withexporter: "yes"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9121"
        prometheus.io/scrape: "true"
      labels:
        app: redis-base-1
        name: redis-base-1-master
        servicename: redis-base-1
        type: redis
        withexporter: "yes"
    spec:
      containers:
      - args:
        - --replica-announce-ip
        - 169.169.164.253
        - --replica-announce-port
        - "6379"
        - --**requirepass
        - $(REDIS_PASSWORD)
        - -- masterauth
        - $(REDIS_MASTER_PASSWORD)**
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 6379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      - args:
        - --redis.addr
        - redis://localhost:6379
        - --redis.password
        - somepassword  #密码替换成 $(REDIS_PASSWORD) 变量
        - **$(REDIS_PASSWORD)**
        - --web.listen-address
        - 0.0.0.0:9121
        image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest
        imagePullPolicy: Always
        name: redis-exporter
		**env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**
        ports:
        - containerPort: 9121
          name: redis-exporter
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-master
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-master/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-master/redis-conf
          type: ""
        name: actual-config

2.3.4 镜像环境变量参考

https://hub.docker.com/r/bitnami/redis#configuration

https://github.com/oliver006/redis_exporter#flags

三、测试是否生效

Master节点

Slave节点

哨兵节点

测试redis-sentinel-exporter 指标抓取

四、 注意事项

  1. 所有节点configmap和deployment yaml 配置文件都按照上面修改别遗漏
  2. 修改完先在测试环境验证没有问题，再连接到连接哨兵集群进行读写测试
相关推荐
嘻哈baby1 天前
Redis高可用部署与集群管理实战
数据库·redis·bootstrap
ascarl20101 天前
Kubernetes 环境 NFS 卡死问题排查与解决纪要
云原生·容器·kubernetes
Java爱好狂.1 天前
Java面试Redis核心知识点整理！
java·数据库·redis·分布式锁·java面试·后端开发·java八股文
谷隐凡二1 天前
etcd在Kubernetes中的作用简单介绍
数据库·kubernetes·etcd
阿杆1 天前
如何在 Spring Boot 中接入 Amazon ElastiCache
java·数据库·redis
m0_569531011 天前
《K8s 网络入门到进阶：Service 与 Ingress 原理、部署方案及核心区别对比》
网络·容器·kubernetes
新手小白*2 天前
K8s 中的 CoreDNS 组件
云原生·容器·kubernetes
一周困⁸天.2 天前
K8S-CoreDNS组件
网络·kubernetes
此生只爱蛋2 天前
【Redis】String 字符串
java·数据库·redis
青云交2 天前
Java 大视界 -- 基于 Java+Flink 构建实时电商交易风控系统实战（436）
java·redis·flink·规则引擎·drools·实时风控·电商交易
热门推荐
01GitHub 镜像站点02UV安装并设置国内源03Linux下V2Ray安装配置指南04在VSCode配置Java开发环境的保姆级教程（适配各类AI编程IDE）05BongoCat - 跨平台键盘猫动画工具06Neo4j（一） - Neo4j安装教程（Windows）07【AutoGLM部署】本地私有化部署AI手机Agent08安娜的档案(Anna’s Archive) 镜像网站/国内最新可访问入口（持续更新）09Cursor 又偷偷更新，这个功能太实用：Visual Editor for Cursor Browser10Open-AutoGLM Windows 安装部署教程