K8S下redis哨兵集群使用secret隐藏configmap内明文密码方案详解

斯普信专业组2025-02-21 17:00

#作者：朱雷

文章目录

  • 一、背景环境及方案说明
  • [二、redis secret configmap deployment参考](#二、redis secret configmap deployment参考)
    • [2.1 创建secret-redis.yaml参考](#2.1 创建secret-redis.yaml参考)
    • [2.2 修改configmap配置参考](#2.2 修改configmap配置参考)
      • [2.2.1 哨兵节点修改（每个节点都修改）](#2.2.1 哨兵节点修改（每个节点都修改）)
      • [2.2.2 主从节点配置修改](#2.2.2 主从节点配置修改)
      • [2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）](#2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）)
    • [2.3 修改deployment配置参考](#2.3 修改deployment配置参考)
      • [2.3.1 master&slaves节点的deployment的yaml修改](#2.3.1 master&slaves节点的deployment的yaml修改)
      • [2.3.2 哨兵节点的deployment的yaml修改](#2.3.2 哨兵节点的deployment的yaml修改)
      • [2.3.4 镜像环境变量参考](#2.3.4 镜像环境变量参考)
  • 三、测试是否生效
  • [四、 注意事项](#四、 注意事项)

一、背景环境及方案说明

Redis 哨兵configmap里包含明文密码，需要处理不在configmap内显示明文密码。

1.1、环境说明

方案基于Redis-5.0.14 哨兵版本（5.x，6.x版本兼容）

方案基于redis-sentinel-exporter-5.0.8版本

方案基于容器环境变量

使用两种方案的其中任意一种均可实现

1.2、方案一：使用配置文件设置密码

参照 2.2.1 和2.2.2 修改configmap

参照 2.3.1 和2.3.2 修改deployment

1.3、方案二：使用args 的命令行传参设置密码

参照 2.2.3.1 和2.2.3.2 修改configmap

参照 2.3.3.1 和2.3.3.2 修改deployment

二、redis secret configmap deployment参考

2.1 创建secret-redis.yaml参考

${}内为redis密码的base64的编码，若认证和同步密码不一致分别定义即可
apiVersion: v1
data:
  password: ${aGFyYm9yMjM0NSM=}
kind: Secret
metadata:
  name: redis-auth-secret
  namespace: paas-middleware

2.2 修改configmap配置参考

2.2.1 哨兵节点修改（每个节点都修改）

apiVersion: v1
data:
  redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
        cp /etc/redis/redis.conf /redis-conf/redis.conf
        **echo -e "sentinel auth-pass mymaster ${REDIS_PASSWORD}" >> /redis-conf/redis.conf
    fi**
    redis-sentinel /redis-conf/redis.conf $@
  redis.conf: |
    port 26379
    protected-mode no
    daemonize no
    sentinel monitor mymaster 169.169.164.253 6379 2
    sentinel down-after-milliseconds mymaster 15000
    sentinel failover-timeout mymaster 60000
    sentinel deny-scripts-reconfig yes
    sentinel parallel-syncs mymaster 2
    sentinel auth-pass mymaster somepassword   # 删除这行配置
kind: ConfigMap
metadata:
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
  每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

2.2.2 主从节点配置修改

apiVersion: v1
data:
  redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
        cp /etc/redis/redis.conf /redis-conf/redis.conf
        **echo -e "masterauth ${REDIS_MASTER_PASSWORD}" >> /redis-conf/redis.conf
        echo -e "requirepass ${REDIS_PASSWORD}" >> /redis-conf/redis.conf**
    fi
    redis-server /redis-conf/redis.conf $@
  redis.conf: |
    bind 0.0.0.0 ::
    port 6379
    daemonize no
    protected-mode no
    timeout 300
    tcp-keepalive 300
    replica-read-only yes
    replica-serve-stale-data yes
    maxclients 20000
    maxmemory 0
    maxmemory-policy noeviction
    masterauth somepassword  # 删除此行配置
    requirepass somepassword  # 删除此行配置
    rename-command FLUSHALL ""
    dir "/data/"
    pidfile "/data/redis.pid"
    logfile "/data/redis.log"
kind: ConfigMap
metadata:
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
 所有主从configmap配置文件都修改， 有****不带删除线的为新增行

2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）

  1. 以下为哨兵节点configmap 修改

    apiVersion: v1
    data:
    redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
    cp /etc/redis/redis.conf /redis-conf/redis.conf
    fi
    redis-sentinel /redis-conf/redis.conf $@
    redis.conf: |
    port 26379
    protected-mode no
    daemonize no
    sentinel monitor mymaster 169.169.164.253 6379 2
    sentinel down-after-milliseconds mymaster 15000
    sentinel failover-timeout mymaster 60000
    sentinel deny-scripts-reconfig yes
    sentinel parallel-syncs mymaster 2
    sentinel auth-pass mymaster somepassword # 删除这行配置
    kind: ConfigMap
    metadata:
    labels:
    app: redis-base-1
    type: redis
    name: redis-base-1-sentinel-1
    namespace: paas-middleware
    每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

  2. 以下为主从节点configmap 修改

    下面为主从节点实例configmap修改，有****不带删除线为新增行
    apiVersion: v1
    data:
    redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
    cp /etc/redis/redis.conf /redis-conf/redis.conf
    fi
    redis-server /redis-conf/redis.conf $@
    redis.conf: |
    bind 0.0.0.0 ::
    port 6379
    daemonize no
    protected-mode no
    timeout 300
    tcp-keepalive 300
    replica-read-only yes
    replica-serve-stale-data yes
    maxclients 20000
    maxmemory 0
    maxmemory-policy noeviction
    masterauth somepassword # 删除此行配置
    requirepass somepassword # 删除此行配置
    rename-command FLUSHALL ""
    dir "/data/"
    pidfile "/data/redis.pid"
    logfile "/data/redis.log"
    kind: ConfigMap
    metadata:
    labels:
    app: redis-base-1
    type: redis
    name: redis-base-1-master
    namespace: paas-middleware

2.3 修改deployment配置参考

2.3.1 master&slaves节点的deployment的yaml修改

所有主从节点配置文件都修改，  有**xxx**为新增行
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-master
      servicename: redis-base-1
      type: redis
      withexporter: "yes"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9121"
        prometheus.io/scrape: "true"
      labels:
        app: redis-base-1
        name: redis-base-1-master
        servicename: redis-base-1
        type: redis
        withexporter: "yes"
    spec:
      containers:
      - args:
        - --replica-announce-ip
        - 169.169.164.253
        - --replica-announce-port
        - "6379"
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 6379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      - args:
        - --redis.addr
        - redis://localhost:6379
        - --redis.password
        - somepassword  #密码替换成 $(REDIS_PASSWORD) 变量
        - **$(REDIS_PASSWORD)**
        - --web.listen-address
        - 0.0.0.0:9121
        image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest
        imagePullPolicy: Always
        name: redis-exporter
		**env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**
        ports:
        - containerPort: 9121
          name: redis-exporter
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-master
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-master/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-master/redis-conf
          type: ""
        name: actual-config

2.3.2 哨兵节点的deployment的yaml修改

所有哨兵节点配置文件都修改， 有****为新增行
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-sentinel-1
      role: sentinel
      type: redis
      withexporter: "no"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-base-1
        name: redis-base-1-sentinel-1
        role: sentinel
        type: redis
        withexporter: "no"
    spec:
      containers:
      - args:
        - --sentinel
        - announce-ip
        - 169.169.196.242
        - --replica-announce-port
        - "26379"
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 26379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-sentinel-1
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/redis-conf
          type: ""
        name: actual-config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-11-09T03:25:41Z"
    lastUpdateTime: "2023-11-09T03:25:43Z"
    message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-07-21T16:48:34Z"
    lastUpdateTime: "2024-07-21T16:48:34Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

2.3.3 使用命令行参数指定密码（本小节与上面两小节互斥）

  1. 哨兵节点deployment 修改
    所有哨兵节点配置文件都修改， 有****不带删除线的为新增行

    apiVersion: apps/v1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    labels:
    app: redis-base-1
    type: redis
    name: redis-base-1-sentinel-1
    namespace: paas-middleware
    spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
    matchLabels:
    app: redis-base-1
    name: redis-base-1-sentinel-1
    role: sentinel
    type: redis
    withexporter: "no"
    strategy:
    rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
    type: RollingUpdate
    template:
    metadata:
    creationTimestamp: null
    labels:
    app: redis-base-1
    name: redis-base-1-sentinel-1
    role: sentinel
    type: redis
    withexporter: "no"
    spec:
    containers:
    - args:
    - --sentinel
    - announce-ip
    - 169.169.196.242
    - --replica-announce-port
    - "26379"
    - --sentinel
    - auth-pass
    - mymaster
    - $(REDIS_PASSWORD)
    command:
    - /etc/redis/redis-docker-entrypoint.sh
    image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
    imagePullPolicy: Always
    name: redis
    env:
    - name: REDIS_PASSWORD
    valueFrom:
    secretKeyRef:
    name: redis-auth-secret
    key: password
    - name: REDIS_MASTER_PASSWORD
    valueFrom:
    secretKeyRef:
    name: redis-auth-secret
    key: password

         ports:
     - containerPort: 26379
       name: client
       protocol: TCP
     resources: {}
     terminationMessagePath: /dev/termination-log
     terminationMessagePolicy: File
     volumeMounts:
     - mountPath: /etc/redis/
       name: config
     - mountPath: /data
       name: data
     - mountPath: /redis-conf
       name: actual-config
   dnsPolicy: ClusterFirst
   nodeSelector:
     kubernetes.io/hostname: 10.179.75.111
   restartPolicy: Always
   schedulerName: default-scheduler
   securityContext: {}
   terminationGracePeriodSeconds: 30
   volumes:
   - configMap:
       defaultMode: 509
       name: redis-base-1-sentinel-1
     name: config
   - hostPath:
       path: /data/redis/redis-base-1-sentinel-1/data
       type: ""
     name: data
   - hostPath:
       path: /data/redis/redis-base-1-sentinel-1/redis-conf
       type: ""
     name: actual-config

    status:
    availableReplicas: 1
    conditions:

    • lastTransitionTime: "2023-11-09T03:25:41Z"
      lastUpdateTime: "2023-11-09T03:25:43Z"
      message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
    • lastTransitionTime: "2024-07-21T16:48:34Z"
      lastUpdateTime: "2024-07-21T16:48:34Z"
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
      observedGeneration: 3
      readyReplicas: 1
      replicas: 1
      updatedReplicas: 1

2、以下为主从实例deployment 配置修改

所有主从节点配置文件都修改， 有****不带删除线的为新增行

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-master
      servicename: redis-base-1
      type: redis
      withexporter: "yes"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9121"
        prometheus.io/scrape: "true"
      labels:
        app: redis-base-1
        name: redis-base-1-master
        servicename: redis-base-1
        type: redis
        withexporter: "yes"
    spec:
      containers:
      - args:
        - --replica-announce-ip
        - 169.169.164.253
        - --replica-announce-port
        - "6379"
        - --**requirepass
        - $(REDIS_PASSWORD)
        - -- masterauth
        - $(REDIS_MASTER_PASSWORD)**
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 6379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      - args:
        - --redis.addr
        - redis://localhost:6379
        - --redis.password
        - somepassword  #密码替换成 $(REDIS_PASSWORD) 变量
        - **$(REDIS_PASSWORD)**
        - --web.listen-address
        - 0.0.0.0:9121
        image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest
        imagePullPolicy: Always
        name: redis-exporter
		**env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**
        ports:
        - containerPort: 9121
          name: redis-exporter
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-master
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-master/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-master/redis-conf
          type: ""
        name: actual-config

2.3.4 镜像环境变量参考

https://hub.docker.com/r/bitnami/redis#configuration

https://github.com/oliver006/redis_exporter#flags

三、测试是否生效

Master节点

Slave节点

哨兵节点

测试redis-sentinel-exporter 指标抓取

四、 注意事项

  1. 所有节点configmap和deployment yaml 配置文件都按照上面修改别遗漏
  2. 修改完先在测试环境验证没有问题，再连接到连接哨兵集群进行读写测试
相关推荐
CT随2 分钟前
Redis 存在线程安全问题吗？为什么？
数据库·redis·安全
TravisBytes4 分钟前
Redis如何解决热Key问题
数据库·redis·缓存
ITPUB-微风8 分钟前
云原生监控体系建设：Kubernetes架构下的全面监控策略
云原生·架构·kubernetes
hjnjmjkj14 分钟前
基于windows的docker-desktop安装kubenetes以及dashboard
docker·容器·kubernetes
HUNAG-DA-PAO2 小时前
Redis存在线程安全吗？为什么？
redis·安全·php
jay丿2 小时前
Redis 中列表（List）常见命令详解
数据库·redis·list
青灯文案14 小时前
如何在 SpringBoot 项目使用 Redis 的 Pipeline 功能
spring boot·redis·后端
Karoku0665 小时前
【CI/CD】Jenkinsfile管理+参数化构建+邮件通知以及Jenkins + SonarQube 代码审查
运维·ci/cd·容器·kubernetes·jenkins·rancher
明明跟你说过7 小时前
【Kubernetes】常用命令全解析：从入门到实战（下）
运维·云原生·容器·kubernetes·k8s
mikey棒棒棒13 小时前
Redis——优惠券秒杀问题（分布式id、一人多单超卖、乐悲锁、CAS、分布式锁、Redisson）
数据库·redis·lua·redisson·watchdog·cas·并发锁
热门推荐
01本地部署DeepSeek教程（Mac版本）02如何在WPS和Word/Excel中直接使用DeepSeek功能03DeepSeek本地部署详细指南04太炸裂了！清华大学deepseek从入门到精通使用手册又出第三版了，《普通人如何抓住DeepSeek红利》（无套路，直接下载）05DeepSeek各版本说明与优缺点分析06DeepSeek r1本地安装全指南07本地化部署AI知识库：基于Ollama+DeepSeek+AnythingLLM保姆级教程08DeepSeek R1本地化部署 Ollama + Chatbox 打造最强 AI 工具09在Windows下安装Ollama并体验DeepSeek r1大模型10Page Assist - 本地Deepseek模型 Web UI 的安装和使用