根据Nginx Ingress指标对指定后端进行HPA

本文分享自华为云社区《根据Nginx Ingress指标对指定后端进行HPA》，作者： 可以交个朋友。

背景

生产场景下，Nginx Ingress的流量会通过域名和path路径最终转发至不同的应用，而有时候cpu和内存并不是nginx的性能瓶颈，此时可以基于nginx_ingress_controller_requests指标，为其对应的应用配置HPA，以实现基于不同域名和path的请求量弹性指定后端工作负载

简介

环境准备

• nginx-ingress已部署
• 云原生监控插件kube-prometheus-stack已安装（server模式)，插件默认监控nginx-ingress，开源环境请自行配置监控。
• 已配置kubectl命令或使用cloudshell

注意：由于HPA规则中scaleTargetRef和describedObject两个字段都无法指定命名空间，所以指标来源、HPA和弹性目标需在同一命名空间，而nginx-ingress和业务工作负载一般处在不同命名空间；本次方案采用external类型的HPA，可以忽略指标来源的命名空间

操作步骤

创建演示需要的弹性目标工作负载，service以及ingress

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: test-hpa 
  labels: 
    app: test-app 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: test-app 
  template: 
    metadata: 
      labels: 
        app: test-app 
    spec: 
      containers: 
      - image: skto/sample-app:v2 
        name: metrics-provider 
        ports: 
        - name: http 
          containerPort: 8080 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: test-app 
  namespace: default 
  labels: 
    app: test-app 
spec: 
  ports: 
    - port: 8080 
      name: http 
      protocol: TCP 
      targetPort: 8080 
  selector: 
    app: test-app 
  type: ClusterIP 

--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: sample-app 
  labels: 
    app: sample-app 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: sample-app 
  template: 
    metadata: 
      labels: 
        app: sample-app 
    spec: 
      containers: 
      - image: skto/sample-app:v2 
        name: metrics-provider 
        ports: 
        - name: http 
          containerPort: 8080 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: sample-app 
  namespace: default 
  labels: 
    app: sample-app 
spec: 
  ports: 
    - port: 80 
      name: http 
      protocol: TCP 
      targetPort: 8080 
  selector: 
    app: sample-app 
  type: ClusterIP 
--- 
apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
  name: test-ingress 
  namespace: default 
spec: 
  ingressClassName: nginx 
  rules: 
    - host: test.example.com 
      http: 
        paths: 
          - backend: 
              service: 
                name: sample-app 
                port: 
                  number: 80 
            path: / 
            pathType: ImplementationSpecific 
          - backend: 
              service: 
                name: test-app 
                port: 
                  number: 8080 
            path: /home 
            pathType: ImplementationSpecific

分别查询test.example.com/和test.example.com/home的nginx_ingress_controller_requests指标，指标正常

创建external类型的apiservices资源；创建后apiservices的状态为false是正常现象，添加externalRules后状态变为true

apiVersion: apiregistration.k8s.io/v1 
kind: APIService 
metadata: 
  name: v1beta1.external.metrics.k8s.io 
spec: 
  group: external.metrics.k8s.io 
  groupPriorityMinimum: 100 
  insecureSkipTLSVerify: true 
  service:         #指定prometheus-adapter对应的service，华为CCE插件中adapter名称为custom-metrics-apiserver 
    name: custom-metrics-apiserver 
    namespace: monitoring 
    port: 443 
  version: v1beta1 
  versionPriority: 100

将externalRules规则添加到adapter的configmap中，修改后需要重启prometheus-adapter服务

kubectl -n monitoring edit configmap user-adapter-config

externalRules: 
- metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>) 
  name: 
    as: ${1}_per_second 
    matches: ^(.*) 
  resources: 
    namespaced: false  #忽略指标来源的命名空间，该配置不适用rules规则 
  seriesQuery: nginx_ingress_controller_requests

seriesQuery：原始指标；可以直接写指标名称，也可以使用{labelKey=labelValue}的方式筛选出原始指标

metricsQuery ：用PromQL对指标进行筛选汇聚；.Series表示原始指标，.LabelMatchers表示对指标进行标签筛选，hpa中可以配置具体的筛选规则

name：将指标重命名

resources：hpa查询指标时通过api的方式调用，调用路径为：

而resources的作用就是将指标中命名空间标签的值替换路径中的${namespace}，而我们本次方案需要忽略指标来源命名空间。

custom-metrics-apiserver服务重启后需要等待1分钟左右，执行命令查看指标是否正常

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/*/nginx_ingress_controller_requests_per_second

创建HPA规则

apiVersion: autoscaling/v2 
kind: HorizontalPodAutoscaler 
metadata: 
  name: sample-hpa 
spec: 
  scaleTargetRef: 
    apiVersion: apps/v1 
    kind: Deployment 
    name: sample-app 
  minReplicas: 1 
  maxReplicas: 10 
  metrics: 
    - type: External 
      external: 
        metric: 
          name: nginx_ingress_controller_requests_per_second 
          selector: 
            matchLabels:    #可以通过该字段对指标进行过滤，这里标签筛选条件会加入到externalRules的<<.LabelMatchers>>中。 
              exported_service: sample-app  #筛选后端服务为sample-app的请求 
              host: test.example.com        #筛选访问域名为test.example.com的请求 
        target: 
          type: AverageValue   #External指标类型下只支持Value和AverageValue类型的目标值。 
          averageValue: 30 
--- 
apiVersion: autoscaling/v2 
kind: HorizontalPodAutoscaler 
metadata: 
  name: test-hpa 
spec: 
  scaleTargetRef: 
    apiVersion: apps/v1 
    kind: Deployment 
    name: test-app 
  minReplicas: 1 
  maxReplicas: 10 
  metrics: 
    - type: External 
      external: 
        metric: 
          name: nginx_ingress_controller_requests_per_second 
          selector: 
            matchLabels: 
              exported_service: test-app 
              host: test.example.com 
        target: 
          type: AverageValue 
          averageValue: 30

弹性演示

使用命令压测sample-app对应的访问域名和路径，正常触发弹性；请自行配置域名与ELB地址的映射

ab -c 50 -n 5000 http://test.example.com/

用同样的方式压测test-app，正常触发弹性

ab -c 50 -n 5000 http://test.example.com/home

点击关注，第一时间了解华为云新鲜技术~