Kubernetes WebHook 入门 -- 入门案例: apiserver 接入 github

文章目录

- [k8s 集群配置](#k8s 集群配置)
- 介绍
- [Admission Webhook](#Admission Webhook)
- [WebHook 入门实践: github 认证接入](#WebHook 入门实践: github 认证接入)
- - [web 服务器](#web 服务器)
  - [Dockerfile 镜像制作](#Dockerfile 镜像制作)
  - Makefile
  - [webhook 接入 apiserver](#webhook 接入 apiserver)
  - - webhook.yaml
    - [apiserver 挂载 webconfig](#apiserver 挂载 webconfig)
    - [在 github 中创建认证 token](#在 github 中创建认证 token)
    - [将 token 添加到 kubeconfig](#将 token 添加到 kubeconfig)
  - 验证
  - - 授权验证
- 参考

k8s 集群配置

IP	Host	配置
11.0.1.150	master1 (keepalived+haproxy)	2C 4G 30G
11.0.1.151	master2 (keepalived+haproxy)	2C 4G 30G
11.0.1.152	node1	2C 4G 30G

vip 地址: https://11.0.1.100:16443

介绍

Webhook就是一种HTTP回调，用于在某种情况下执行某些动作，Webhook不是K8S独有的，很多场景下都可以进行Webhook，比如在提交完代码后调用一个Webhook自动构建docker镜像

K8S中提供了自定义资源类型 和自定义控制器 来扩展功能，还提供了动态准入控制，其实就是通过Webhook来实现准入控制，分为两种：验证性质的准入 Webhook （Validating Admission Webhook）和修改性质的准入 Webhook （Mutating Admission Webhook）

Admission Webhook有哪些使用场景？如下

在资源持久化到ETCD之前进行修改（Mutating Webhook），比如增加init Container或者sidecar Container
在资源持久化到ETCD之前进行校验（Validating Webhook），不满足条件的资源直接拒绝并给出相应信息

现在非常火热的的 Service Mesh 应用istio就是通过 mutating webhooks 来自动将Envoy这个 sidecar 容器注入到 Pod 中去的：

https://istio.io/docs/setup/kubernetes/sidecar-injection/。

Admission Webhook

上面提到K8S的动态准入控制是通过Webhook来实现的，那么它到底是在哪个环节执行的？请看下图

综上, k8s webhook 可分为两类:

Mutating Webhook: 这类 webhook 可以更改请求对象
Validating Webhook: 这类 webhook 不能去更改请求对象, 就算更改也会被忽略, 用于写入 etcd 前的验证逻辑

WebHook 入门实践: github 认证接入

源码地址: https://github.com/Ai-feier/k8s-webhook/tree/main/auth-github

写一个用于认证校验的 mutating webhook

请求流程:

拦截 apiserver 请求
将请求(TokenReview)中的 token 发送给 github 用于验证
将 github 返回的用户信息填入 TokenReview

web 服务器

go 复制代码

package main

import (
	"context"
	"encoding/json"
	"github.com/google/go-github/github"
	"golang.org/x/oauth2"
	authentication "k8s.io/api/authentication/v1beta1"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/authenticate", func(writer http.ResponseWriter, request *http.Request) {
		// 获取请求中的 TokonReview
		decoder := json.NewDecoder(request.Body)
		var tokenReview authentication.TokenReview
		if err := decoder.Decode(&tokenReview); err != nil {
			log.Println("decode error: ", err)
			// 响应返回错误
			writer.WriteHeader(http.StatusBadRequest)
			_ = json.NewEncoder(writer).Encode(map[string]interface{}{
				"apiVersion": "authentication.k8s.io/v1beta1",
				"kind":       "TokenReview",
				"status": authentication.TokenReviewStatus{
					Authenticated: false,
				},
			})
			return
		}

		log.Println("receive request")

		// 检查用户
		// 创建 oauth2 对象, 发送给 github
		oauthToken := oauth2.StaticTokenSource(&oauth2.Token{
			AccessToken: tokenReview.Spec.Token, // tokenReview 的 token
		})
		oauthClient := oauth2.NewClient(context.Background(), oauthToken)
		client := github.NewClient(oauthClient)
		user, _, err := client.Users.Get(context.Background(), "")
		if err != nil {
			log.Println("github auth error: ", err)
			writer.WriteHeader(http.StatusUnauthorized)
			_ = json.NewEncoder(writer).Encode(map[string]any{
				"apiVersion": "authentication.k8s.io/v1beta1",
				"kind":       "TokenReview",
				"status": authentication.TokenReviewStatus{
					Authenticated: false, // 返回认证失败
				},
			})
			return
		}

		// 认证成功
		log.Println("github login success, as: ", *user.Login)
		status := authentication.TokenReviewStatus{
			Authenticated: true,
			User: authentication.UserInfo{
				Username: *user.Login,
				UID: *user.Login,
			},
		}
		json.NewEncoder(writer).Encode(map[string]any{
			"apiVersion": "authentication.k8s.io/v1beta1",
			"kind":       "TokenReview",
			"status": status,
		})
	})
	log.Println(http.ListenAndServe(":3000", nil))
}

Dockerfile 镜像制作

本项目是使用 go 作为 web 服务器, go 提供一种非常好用的编译特性 -- 交叉编译

在 dockerfile 中我们选择的 busybox 作为基础镜像, 但默认 busybox 架构为 x86_64(uname -m查看), 所以可以使用 amd64/busybox 或交叉编译 go 代码

当出现这类问题 exec /usr/bin/sh: exec format error 多半是架构问题

推荐: Golang交叉编译详解 -- CSDN

amd64

dockerfile 复制代码

FROM golang:1.20 as builder

WORKDIR /app

COPY . .

# go 交叉编译: amd64
RUN go env -w  GOPROXY=https://goproxy.cn,direct && go mod tidy && \
    CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o main main.go 

RUN chmod +x /app/main

FROM amd64/busybox

COPY --from=builder /app/main .

EXPOSE 3000/tcp

CMD ["/main"]

x86_64

dockerfile 复制代码

FROM golang:1.20 as builder

WORKDIR /app

COPY . .

# go 交叉编译: x86
RUN go env -w  GOPROXY=https://goproxy.cn,direct && go mod tidy && \
    CGO_ENABLED=0 GOOS=linux GOARCH=386 go build -o main main.go 

RUN chmod +x /app/main

FROM busybox

COPY --from=builder /app/main .

EXPOSE 3000/tcp

CMD ["/main"]

构造镜像

sh 复制代码

$ docker build -t aifeierwithinmkt/auth-github-webhook:amd64 -f Dockerfile_amd64 .

$ docker build -t aifeierwithinmkt/auth-github-webhook:x86 -f Dockerfile_x86 .

$ docker tag aifeierwithinmkt/auth-github-webhook:amd64 aifeierwithinmkt/auth-github-webhook

检验镜像

sh 复制代码

$ docker run --rm -p 3000:3000 -d aifeierwithinmkt/auth-github-webhook:amd64
$ docker run --rm -p 3001:3000 -d aifeierwithinmkt/auth-github-webhook:x86

$ curl 127.0.0.1:3000/authenticate
{"apiVersion":"authentication.k8s.io/v1beta1","kind":"TokenReview","status":{"user":{}}}
$ curl 127.0.0.1:3001/authenticate
{"apiVersion":"authentication.k8s.io/v1beta1","kind":"TokenReview","status":{"user":{}}}

镜像服务正常

Makefile

makefile 复制代码

build_amd64:
	mkdir -p bin/amd64
	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/amd64/main .

build_x86:
	mkdir -p bin/x86
	CGO_ENABLED=0 GOOS=linux GOARCH=386 go build -o bin/x86/main .

webhook 接入 apiserver

webhook.yaml

yaml 复制代码

apiVersion: v1
clusters:
- cluster:
    server: http://192.168.1.7:3000/authenticate
  name: github-authn
contexts:
- context:
    cluster: github-authn
    user: authn-apiserver
  name: webhook
current-context: webhook
kind: Config
preferences: {}
users:
- name: authn-apiserver
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJRlYrSjRRNm1heUF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpFeU16QXdPVE16TVRkYUZ3MHlOREV5TWprd09UTTRNVGxhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTQ2NTQzOWp6cVpqcldXd3EKMkloeWtQcjREb3MySnNTM0JrUlhJTzY2UnhhbmxqVGszWlBka21oaUFvRldHTHI5VitwTUdwTytVYzhmbHhDKwoyZVc3QmVwelhzVjd2Y1RRZ25RMVlnVGh2bldRdkxVVFJuUXhoUHhYS2RGNzBEL2RISkxjMGxyVHhjZ2JNUUVTCmdPZWZZQm03dWdBdEJ0RDg3MitrdWVuV3RIc201a09nd0l0N3EzbjVvQ1huZWd4VHBFTHFtcmhtQTYrUXcrNXkKbEg1bG5LdzlFdUJmYmVqY0ZOaEd2UVNFSmFNZ2s0Z2h4bUNQWHJUcFZ0bmRkMHJQMGc4dzkvQ0NKU1owdDV6TApHMk5idFlqQzRRK2tyQXZoSllIUGRzVkRtWTFpMkZsSmsyZGgyU0RzbzJLMGNldmMvRGpUaTdhbGNpTlQwU05ICjFoVXpsd0lEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JRU1oxaVlWKzlQNTRiNHB3UU9QVkI0TklXYgp3REFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBV3pmWXJMWUd5ZGJoM0hpRmxVMWhuR2ZYeGxDMGk3VDBNMmZ1CnZMUkJBM1lYelRMQmNJeFhjT3J2R2Z1OGp3ZE1qY1VmZ2tuQ0JOL2VrN2lqSDF6ZmxmRkVwK0JiWHE2WFpUQWQKTkh1aGdKVG5SRUV4OURHNldtLy9wNnpZOExEVU1ZekZwZGYyYkt2ZWdkQ0NaT3R5OTJ2eFZmbUJ5THJnUHYwRQpGSEZhR1ZCQ083Z012VnFCN05rMGxVN3Axdjg3YzVBTmJxUHI2YjYyZmMzbE9XcGQ2bUN0bDFjdms1QWROczJlCnBRVlFBRnhGR1o3WGFDaU9waFM3L2M1Vm1MUk9lWERHZ1BjUnNjdEtZL0VybXNqQ2JrYnpobnUydk85dnJPRVQKZEh0QlBYRlBYV29PWTVsZWlQa29WRUs4S1FFMmhPWVBpZDhIVHNhZzl1ZGc2Zkwvenc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBNDY1NDM5anpxWmpyV1d3cTJJaHlrUHI0RG9zMkpzUzNCa1JYSU82NlJ4YW5salRrCjNaUGRrbWhpQW9GV0dMcjlWK3BNR3BPK1VjOGZseEMrMmVXN0JlcHpYc1Y3dmNUUWduUTFZZ1Rodm5XUXZMVVQKUm5ReGhQeFhLZEY3MEQvZEhKTGMwbHJUeGNnYk1RRVNnT2VmWUJtN3VnQXRCdEQ4NzIra3Vlbld0SHNtNWtPZwp3SXQ3cTNuNW9DWG5lZ3hUcEVMcW1yaG1BNitRdys1eWxINWxuS3c5RXVCZmJlamNGTmhHdlFTRUphTWdrNGdoCnhtQ1BYclRwVnRuZGQwclAwZzh3OS9DQ0pTWjB0NXpMRzJOYnRZakM0UStrckF2aEpZSFBkc1ZEbVkxaTJGbEoKazJkaDJTRHNvMkswY2V2Yy9EalRpN2FsY2lOVDBTTkgxaFV6bHdJREFRQUJBb0lCQUJIaWRxUSt5b1ViK2dEQQpPbTFmNm4vdzl1Tk5sQ2RmZEhFTmxUcUZCaVRuWnFxcDVRQnl5UWpqSWkvSU1SY29PUlphMVRlUk8zWDVxeVdXCnJ5YzJvSVpLY0YyVmJhN3VjdUtNZGxVSXhTTE00VjJ4YTU0eEttS2ozOFR0SzZpa0c1NU8rd0diR04rRVpINW8KOHljbENxUGw0WlV1eGxxdXQrK20rVzJSTE1ob2JoOG1mb1RDNGExczJlMEdpM3lkdE16SjNXZnczcUUvUlRGbQoyYm0rMHd6bFlUaXB2cmo4OEJ0aHBQUDlrTmJyVEs4WFNROXUzMnNjS3VWVllpcUJpeStMVUhMek5GY09ZbkpPCjBtdUFsTTFhTmh2WXJQMnI4QThkLzZtays5eDN6THVlM1NVYm1BdTJFUFdMKzZneUxiRHIrV1RJeHYxRVNQS0EKZW5TcWRRRUNnWUVBNVZZMGloY3I2Y0pGbkZZUnBJTjRSUjVrejBhdE12RUZmQWhIbERoUVQrUmpCTXRkVHlFSApNQVFqdXlZaDhReEh0TmxEQzBJTFJTdWpPSWdhZHFsMlBmbzNwR1RmM0xlbExkbGx4amRsVTZJc0lLYVVDdWw5CndPcTBmbkVTbUVUdGdxZS9nazFMS01ZMXUyOXg2S2FkZ25RSlBybFlQT21idW1SY3pydHc3N2NDZ1lFQS9pY0EKcnhDODcvc3VDQS9HUXpRdWlvVWphd0NXSEgyVnE2eUlSNDRVWXJaNWMwRXl1OU5jOFhoQVhTbGJxNzF6NkNDMgpFQVE4cm8weWFWTFB2TDFWeDB2aDVKUjVnaHVkMkcvMFdPbzFVR0l1SFhPaGpSeWR3UDlzZ2NZRGpUaitHYlJNCjlkSWJrWi9YcWx3Uzh2aVJHQ1BaN3hjVm5pTlJ5RzJ5N3BxeUd5RUNnWUIxNDdlRVdONzQvaVc4ZEw0Qy9KWWgKcWJzV2xmVkluMzg3UUNKVGZoTkN6bHRjUnBJRHNDMjZzQllTQ1VzZlZ6bXhMSkg3UW9yNmxyRUR5V3NaSG9tcQoyR29yOXJMaENnSStMR2ZWMmZvYllOMGdONkVZYnVoMjkrK3FvOE4wUk5KMi9IWkVyQ2o3bjlCVk5yZXVhWi9FClJKUFFDNFRoWXhEcll0WVdhMkpseHdLQmdRRFh3SjU0LzNtVk5DTkFucnVOZzNmYkNla21SZ29veDRmT2hCbncKdkxHYmx4S0ZBQjBraStyRDVuU2xZWjI3cm9uOXpmOGdtNmd6K2hPSWk4OWtoMHFSZEY2Z29GYUNXQlZvanFuYwo3WDR5N2hYOTFKS1phMmlVVllGMHJYZUlaSkI1bTdFVm9iYmJxZGo0ZTA5dXlncktkbXprNWpEbzNVenBIQThoCk5WdnJZUUtCZ1FDNDRreXNsRlBUVkNlbkp5SDFHLy82VmQ0aXRFaFlYQVllYWJUdU15eVlkZFlxcnJYTTFUS1QKL24wRHQrQXFIa0dmU2pvUExCa3NITkNDZWZnZSt0dDRxVmFPNjJyWkhRRTQ0TXVueHhxRXBNdmxFeUNEVUM4cApTaUF6bW9XaDR4R2tUdkpLUlBLWGlPNEdjbk5wZVNuUVY5RjJSY01ocllCUVZjTzJLb2N1Umc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

该配置与 kubeconfig 配置相同, 直接在 kubeconfig 的基础上修改就行, 重要的是 cluster 的 server (即提供服务的地址

apiserver 挂载 webconfig

yaml 复制代码

# 修改 apiserver 配置参数, 增加:
- --authentication-token-webhook-config-file=/etc/config/webhook.yaml

# 相应增加挂载卷
    volumes:
    - hostPath:
        path: /etc/config
        type: DirectoryOrCreate
      name: auth-webhook
    
    volumeMounts:
    - mountPath: /etc/config
      name: auth-webhook
      readOnly: true

修改 manifest 后 apiserver 理论上会自动重启, 如果没有则 systemctl restart kubelet

本文实验主题为认证 webhook 可直接使用 --authentication-token-webhook-config-file, 如需自定义 mutatingwebhook 请参考: mutatingwebhook

在 github 中创建认证 token

setting -> Developer Settings -> Personal access tokens (classic)

将 token 添加到 kubeconfig

yaml 复制代码

# 在 kubeconfig 下增加一个 user
- name: githubuser
  user:
    token: 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

验证

sh 复制代码

$ k get po --user githubuser
Error from server (Forbidden): pods is forbidden: User "Ai-feier" cannot list resource "pods" in API group "" in the namespace "default"

授权验证

sh 复制代码

$ k create clusterrolebinding testview --clusterrole view --user Ai-feier
$ k get po --user githubuser
NAME                         READY   STATUS    RESTARTS      AGE
nginx-kusc00401              0/1     Pending   0             4d23h
testnginx-5cdb847cd9-7nm4k   1/1     Running   1 (29h ago)   43h
testnginx-5cdb847cd9-llmxf   1/1     Running   1 (29h ago)   43h
testnginx-5cdb847cd9-qfsdn   1/1     Running   1 (29h ago)   43h
web-server                   0/1     Pending   0             4d22h

参考

https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/extensible-admission-controllers/