K8S Gateway AB测试、蓝绿发布、金丝雀(灰度)发布

假设有如下三个节点的 K8S 集群：

k8s31master 是控制节点

k8s31node1、k8s31node2 是工作节点

容器运行时是 containerd

一、场景分析

阅读本文，默认您已经安装了 K8S Gateway。

关于 AB 测试、金丝雀发布，可以看这篇文章。

二、实验准备

镜像下载

bash 复制代码

# 在各个工作节点下载
[root@k8s31node1 ~]# ctr -n=k8s.io images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/openresty/openresty:latest
[root@k8s31node1 ~]# ctr -n=k8s.io images tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/openresty/openresty:latest  docker.io/openresty/openresty:latest

[root@k8s31node2 ~]# ctr -n=k8s.io images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/openresty/openresty:latest
[root@k8s31node2 ~]# ctr -n=k8s.io images tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/openresty/openresty:latest  docker.io/openresty/openresty:latest

部署 v1

javascript 复制代码

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      version: v1
  template:
    metadata:
      labels:
        app: nginx
        version: v1
    spec:
      containers:
      - name: nginx
        image: "openresty/openresty:latest"
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          protocol: TCP
          containerPort: 80
        volumeMounts:
        - mountPath: /usr/local/openresty/nginx/conf/nginx.conf
          name: config
          subPath: nginx.conf
      volumes:
      - name: config
        configMap:
          name: nginx-v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: nginx
    version: v1
  name: nginx-v1
data:
  nginx.conf: |-
    worker_processes  1;
    events {
        accept_mutex on;
        multi_accept on;
        use epoll;
        worker_connections  1024;
    }
    http {
        ignore_invalid_headers off;
        server {
            listen 80;
            location / {
                access_by_lua '
                    local header_str = ngx.say("nginx-v1")
                ';
            }
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-v1
spec:
  type: ClusterIP
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: nginx
    version: v1

该 yml 定义了三个资源 ConfigMap、Deployment、Service。

ConfigMap 定义了一个 nginx.conf 配置文件，使用 lua 脚本输出 nginx-v1。

Deployment 定义了一个 Pod，里面运行 openresty 它是一个封装了 nginx+lua 的 web 服务器。Pod 有两个标签 app: nginx、version: v1。

Service 代理了 Deployment 运行的 Pod。

部署 v2

javascript 复制代码

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      version: v2
  template:
    metadata:
      labels:
        app: nginx
        version: v2
    spec:
      containers:
      - name: nginx
        image: "openresty/openresty:latest"
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          protocol: TCP
          containerPort: 80
        volumeMounts:
        - mountPath: /usr/local/openresty/nginx/conf/nginx.conf
          name: config
          subPath: nginx.conf
      volumes:
      - name: config
        configMap:
          name: nginx-v2
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: nginx
    version: v2
  name: nginx-v2
data:
  nginx.conf: |-
    worker_processes  1;
    events {
        accept_mutex on;
        multi_accept on;
        use epoll;
        worker_connections  1024;
    }
    http {
        ignore_invalid_headers off;
        server {
            listen 80;
            location / {
                access_by_lua '
                    local header_str = ngx.say("nginx-v2")
                ';
            }
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-v2
spec:
  type: ClusterIP
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: nginx
    version: v2

三、AB 测试

1）创建 gateway

javascript 复制代码

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: nginx-gateway
  namespace: default
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    protocol: HTTP
    port: 80

2）创建 httproute

javascript 复制代码

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: abtest-route
  namespace: default
spec:
  parentRefs:
    - name: nginx-gateway
  hostnames:
    - "abtest.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: / 
          headers:
            - name: "version"
              value: "v1"
      backendRefs:
        - name: nginx-v1
          kind: Service
          port: 80
    - matches:
        - path:
            type: PathPrefix
            value: /
          headers:
            - name: "version"
              value: "v2"  
      backendRefs:
        - name: nginx-v2
          kind: Service
          port: 80

parentRefs：绑定我们新建的 gateway。

hostnames：定义访问的主机名。

rules.matches：定义路由规则，PathPrefix 表示路径前缀匹配。headers 对请求头进行匹配。

backendRefs：定义后端服务以及服务端口。

3）测试

bash 复制代码

curl -H "Host: abtest.example.com" -H "version: v1" http://192.168.40.20:30185/
curl -H "Host: abtest.example.com" -H "version: v2" http://192.168.40.20:30185/

30185 为 nginx-gateway-controller 80 映射端口。

四、金丝雀发布

1）创建 httproute

javascript 复制代码

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary-route
  namespace: default
spec:
  parentRefs:
    - name: nginx-gateway
  hostnames:
    - "canary.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: / 
      backendRefs:
        - name: nginx-v1
          kind: Service
          port: 80
          weight: 10
        - name: nginx-v2
          kind: Service
          port: 80
          weight: 90

backendRefs.weight 定义流量分发的权重。

2）测试

bash 复制代码

for i in {1..20}; do curl -H "Host: canary.example.com" http://192.168.40.20:30185/; done;