华为云kubernetes基于keda自动伸缩deployment副本（监听redis队列长度）

1 概述

KEDA（Kubernetes-based Event-Driven Autoscaler，网址是https://keda.sh）是在 Kubernetes 中事件驱动的弹性伸缩器，功能非常强大。不仅支持根据基础的CPU和内存指标进行伸缩，还支持根据各种消息队列中的长度、数据库中的数据统计、QPS、Cron 定时计划以及您可以想象的任何其他指标进行伸缩。KEDA支持的所有scaler，可从如下网址里查询：

复制代码

https://keda.sh/docs/2.16/scalers

该项目于2020年3月被 CNCF 接收，并于2021年8月开始孵化，最终在2023年8月宣布毕业，目前已经非常成熟，可放心在生产环境中使用。

本文介绍在华为云kubernetes中部署keda v2.16.1，keda监听redis队列中的长度来伸缩目标deployment的副本数。

2 架构图

KEDA 并不是要替代HPA，而是作为HPA的补充或者增强。实际上，KEDA 经常与 HPA 一起协同工作 。以下是 KEDA 官方的架构图

当要将工作负载的副本数缩到闲时副本数，或从闲时副本数开始扩容时，由KEDA通过修改工作负载的副本数实现（闲时副本数小于 minReplicaCount，包括0，即可以缩到0）。
其他情况下的扩缩容过程都由Kubernetes HPA实现，HPA被KEDA管理，HPA使用External Metrics作为数据源，而External Metrics实际的数据由KEDA提供。
KEDA 各种Scalers 的目的其实就是为HPA 暴露External Metrics 格式的数据，KEDA会将各种外部事件转换为所需的External Metrics数据，最终实现HPA读取这些External Metrics数据进行自动伸缩，因此KEDA直接复用了HPA 已有的能力，如果需要控制扩缩容的行为细节（例如快速扩容、缓慢缩容），可以直接通过配置 HPA 的 behavior 字段来实现（要求 Kubernetes 版本 ≥1.18）。

3 环境准备

3.1 华为云kubernetes集群

准备一个kubernetes集群，如下图所示：

如果你需要KEDA基于传统的CPU和内存来伸缩工作负载，则需要为kubernetes集群开启metrics插件 。

如果你需要KEDA基于传统的CPU和内存来伸缩工作负载，则需要为kubernetes集群开启metrics插件 。
开启metrics插件 只需要在华为云kubernetes控制台的插件中心里安装即可，如下图：

3.2 redis服务

准备一个redis实例，如下图所示：

4 部署

4.1 部署keda

为helm添加新的repo，命令如下：

复制代码

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

下载values.yaml，命令如下：

复制代码

helm show values kedacore/keda > values.yaml

修改values.yaml文件中的容器镜像，方便在国内环境拉取，如下所示：

复制代码

image:
  keda:
    registry: docker.io
    repository: imroc/keda
    tag: "2.16.1"
  metricsApiServer:
    registry: docker.io
    repository: imroc/keda-metrics-apiserver 
    tag: "2.16.1"
  webhooks:
    registry: docker.io
    repository: imroc/keda-admission-webhooks
    tag: "2.16.1"

部署keda，命令如下所示：

复制代码

helm upgrade --install keda kedacore/keda \
--namespace keda --create-namespace \
-f values.yaml

4.2 部署待被伸缩的目标deployment

创建一个零副本的服务，如下所示：

复制代码

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
  namespace: default
spec:
  replicas: 0
  selector:
    matchLabels:
      run: php-apache
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - image: deis/hpa-example
        imagePullPolicy: Always
        name: php-apache
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          limits:
            cpu: 100m
          requests:
            cpu: 20m

5 创建伸缩对象ScaledObject

创建如下对象：

复制代码

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: hpa-app
  namespace: default
spec:
  scaleTargetRef:                           # 扩容目标对象
    name: php-apache
    kind: Deployment
    apiVersion: apps/v1
  minReplicaCount: 1                        # 最小副本数
  maxReplicaCount: 6                        # 最大副本数
  triggers:                                 # 触发规则
    - metadata:
        address: redis-****.cn-south-1.dcs.myhuaweicloud.com:6379    # Redis地址
        listName: keda-hpa-demo-list                               # Redis的列表的key名称
        listLength: "10"                                        # 触发伸缩的队列长度
        password: "********your-redis-password********"
      type: redis                                             # 事件源类型为redis

6 往redis中的队列添加元素

队列名称为keda-hpa-demo-list，往里面添加20个元素，命令如下：

复制代码

RPUSH keda-hpa-demo-list value1 value2 value3 value4 value5 value6 value7 value8 value9 value10
RPUSH keda-hpa-demo-list value11 value12 value13 value14 value15 value16 value17 value18 value19 value20
LRANGE keda-hpa-demo-list 0 -1

7 伸缩现象

查看kubernetes event，可见副本数量伸缩为2了。

复制代码

kubectl get event

31m         Normal    SuccessfulRescale         horizontalpodautoscaler/keda-hpa-hpa-app   New size: 2; reason: external metric s0-redis-keda-hpa-demo-list(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: hpa-app,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

仔细发现，keda自动创建了HPA对象，这个对象里的一些字段是来自，如下所示：

复制代码

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"keda.sh/v1alpha1","kind":"ScaledObject","metadata":{"annotations":{},"name":"hpa-app","namespace":"default"},"spec":{"maxReplicaCount":6,"minReplicaCount":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"php-apache"},"triggers":[{"metadata":{"address":"redis-6482e5d4-6f37-4138-ab01-64f7ca6a4a6d.cn-south-1.dcs.myhuaweicloud.com:6379","listLength":"10","listName":"keda-hpa-demo-list","password":"iloveredis@2025"},"type":"redis"}]}}
  creationTimestamp: "2025-02-12T14:28:56Z"
  labels:
    app.kubernetes.io/managed-by: keda-operator
    app.kubernetes.io/name: keda-hpa-hpa-app
    app.kubernetes.io/part-of: hpa-app
    app.kubernetes.io/version: 2.16.1
    scaledobject.keda.sh/name: hpa-app
  name: keda-hpa-hpa-app
  namespace: default
  ownerReferences:
  - apiVersion: keda.sh/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ScaledObject
    name: hpa-app
    uid: 795e92f1-da13-4281-a460-9cfe750ec753
  resourceVersion: "99768"
  uid: 629c14c2-b0a0-4007-8c38-eaa104dc20a2
spec:
  maxReplicas: 6
  metrics:
  - external:
      metric:
        name: s0-redis-keda-hpa-demo-list
        selector:
          matchLabels:
            scaledobject.keda.sh/name: hpa-app
      target:
        averageValue: "10"
        type: AverageValue
    type: External
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
status:
  conditions:
  - lastTransitionTime: "2025-02-12T14:29:11Z"
    message: the HPA controller was able to get the target's current scale
    reason: SucceededGetScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2025-02-12T15:34:36Z"
    message: 'the HPA was unable to compute the replica count: unable to get external
      metric default/s0-redis-keda-hpa-demo-list/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name:
      hpa-app,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics
      from external metrics API: rpc error: code = Unknown desc = error when getting
      metric values error getting scalers connection to redis failed: dial tcp: lookup
      redis-6482e5d4-6f37-4138-ab01-64f7ca6a4a6d.cn-south-1.dcs.myhuaweicloud.com
      on 169.254.1.1:53: no such host'
    reason: FailedGetExternalMetric
    status: "False"
    type: ScalingActive
  - lastTransitionTime: "2025-02-12T14:29:11Z"
    message: the desired count is within the acceptable range
    reason: DesiredWithinRange
    status: "False"
    type: ScalingLimited
  currentMetrics:
  - type: ""
  currentReplicas: 2
  desiredReplicas: 2
  lastScaleTime: "2025-02-12T14:30:11Z"

同时，还创建了external metrics对象，如下所示：

复制代码

kubectl get apiservices | grep external.metrics.k8s.io

复制代码

kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    meta.helm.sh/release-name: keda
    meta.helm.sh/release-namespace: keda
  creationTimestamp: "2025-02-12T07:52:48Z"
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/instance: keda
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
    app.kubernetes.io/part-of: keda-operator
    app.kubernetes.io/version: 2.16.1
    helm.sh/chart: keda-2.16.1
  name: v1beta1.external.metrics.k8s.io
  resourceVersion: "47073"
  uid: 0428df85-50ed-4363-9799-f3c03dee88e9
spec:
  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0************************DdBPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  service:
    name: keda-operator-metrics-apiserver
    namespace: keda
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2025-02-12T13:26:19Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/s0-redis-keda-hpa-demo-list?labelSelector=scaledobject.keda.sh%2Fname%3Dhpa-app" | jq .

可见keda namespace下的keda-operator-metrics-apiserver服务是kube-apiserver的指标类接口的后端，也是keda各个scaler的前端网关。

8 小结

通过内置很多scaler的伸缩神器keda，以redis队列的长度为依据对目标工作负载deployment进行伸缩，开箱即用，解决了以往的HPA Controller + Prometheus Adaptor + Prometheus + Exporter的自定义流程。