Kubernetes 审计(Auditing)

目录

  • 一.系统环境
  • 二.前言
  • [三.Kubernetes 审计简介](#三.Kubernetes 审计简介)
  • 四.审计策略简介
  • 五.启用审计
    • [5.1 引入审计](#5.1 引入审计)
    • [5.2 启用审计](#5.2 启用审计)
  • 六.审计策略
    • [6.1 记录审计阶段为:ResponseStarted,审计级别为Metadata,apiVersion为group: "" 的日志](#6.1 记录审计阶段为:ResponseStarted,审计级别为Metadata,apiVersion为group: "" 的日志)
    • [6.2 只记录audit命名空间里的日志](#6.2 只记录audit命名空间里的日志)
    • [6.3 只记录audit命名空间的pods操作日志](#6.3 只记录audit命名空间的pods操作日志)
    • [6.4 只记录audit命名空间的pods,services,deployments操作日志](#6.4 只记录audit命名空间的pods,services,deployments操作日志)
    • [6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse](#6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse)
    • [6.6 只记录audit命名空间下的tom用户的pods操作,其他用户操作不记录](#6.6 只记录audit命名空间下的tom用户的pods操作,其他用户操作不记录)
    • [6.7 rules规则是从上往下匹配的,第一条规则已经匹配了,第二条就不匹配了](#6.7 rules规则是从上往下匹配的,第一条规则已经匹配了,第二条就不匹配了)
    • [6.8 在 Metadata 级别为所有请求生成日志](#6.8 在 Metadata 级别为所有请求生成日志)
  • 七.总结

一.系统环境

本文主要基于Kubernetes1.22.2和Linux操作系统Ubuntu 18.04。

服务器版本 docker软件版本 Kubernetes(k8s)集群版本 CPU架构
Ubuntu 18.04.5 LTS Docker version 20.10.14 v1.22.2 x86_64

Kubernetes集群架构:k8scludes1作为master节点,k8scludes2,k8scludes3作为worker节点。

服务器 操作系统版本 CPU架构 进程 功能描述
k8scludes1/192.168.110.128 Ubuntu 18.04.5 LTS x86_64 docker,kube-apiserver,etcd,kube-scheduler,kube-controller-manager,kubelet,kube-proxy,coredns,calico k8s master节点
k8scludes2/192.168.110.129 Ubuntu 18.04.5 LTS x86_64 docker,kubelet,kube-proxy,calico k8s worker节点
k8scludes3/192.168.110.130 Ubuntu 18.04.5 LTS x86_64 docker,kubelet,kube-proxy,calico k8s worker节点

二.前言

在当今的云计算时代,容器编排技术已成为企业部署和管理应用程序的关键组成部分。Kubernetes 作为最流行的容器编排平台,提供了强大的功能来部署、管理和扩展容器化应用程序。然而,随着应用程序的日益复杂和部署环境的扩大,确保 Kubernetes 集群的安全性变得越来越重要。在这篇博客中,我们将深入了解 Kubernetes 审计的概念,并探讨如何在 Kubernetes 环境中实施审计策略。

使用Kubernetes 审计(Auditing)的前提 是已经有一套可以正常运行的Kubernetes集群,关于Kubernetes(k8s)集群的安装部署,可以查看博客《Ubuntu 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/17632858.html。

三.Kubernetes 审计简介

Kubernetes 审计(Auditing) 功能提供了与安全相关的、按时间顺序排列的记录集, 记录每个用户、使用 Kubernetes API 的应用以及控制面自身引发的活动。

审计功能使得集群管理员能够回答以下问题:

  • 发生了什么?
  • 什么时候发生的?
  • 谁触发的?
  • 活动发生在哪个(些)对象上?
  • 在哪观察到的?
  • 它从哪触发的?
  • 活动的后续处理行为是什么?

审计记录最初产生于 kube-apiserver 内部。每个请求在不同执行阶段都会生成审计事件;这些审计事件会根据特定策略被预处理并写入后端。 策略确定要记录的内容和用来存储记录的后端,当前的后端支持日志文件webhook

每个请求都可被记录其相关的阶段(stage),阶段(stage)可以理解为什么时候记录。。已定义的阶段有:

  • RequestReceived - 此阶段对应审计处理器接收到请求后, 并且在委托给其余处理器之前生成的事件。
  • ResponseStarted - 在响应消息的头部发送后,响应消息体发送前生成的事件。 只有长时间运行的请求(例如 watch)才会生成这个阶段。
  • ResponseComplete - 当响应消息体完成并且没有更多数据需要传输的时候。
  • Panic - 当 panic 发生时生成。

审计日志记录功能会增加 API server 的内存消耗,因为需要为每个请求存储审计所需的某些上下文。 内存消耗取决于审计日志记录的配置。

Kubernetes 审计是一种监控和记录 Kubernetes 集群中资源操作的方法,用于确保集群的安全性和符合性。通过审计,管理员可以跟踪对集群资源的访问和修改,以便在发生安全事件时进行调查和响应。Kubernetes 提供了审计日志记录的框架,允许管理员自定义审计策略,以确定哪些资源操作应该被记录。

四.审计策略简介

审计策略定义了关于应记录哪些事件以及应包含哪些数据的规则。 审计策略对象结构定义在 audit.k8s.io API 组。 处理事件时,将按顺序与规则列表进行比较。第一个匹配规则设置事件的审计级别(Audit Level),审计级别(Audit Level)可以理解为记录什么? 已定义的审计级别有:

  • None - 符合这条规则的日志将不会记录。
  • Metadata - 记录请求的元数据(请求的用户、时间戳、资源、动词等等), 但是不记录请求或者响应的消息体。
  • Request - 记录事件的元数据和请求的消息体,但是不记录响应的消息体。 这不适用于非资源类型的请求。
  • RequestResponse - 记录事件的元数据,请求和响应的消息体。这不适用于非资源类型的请求。

你可以使用 --audit-policy-file 标志将包含策略的文件传递给 kube-apiserver。 如果不设置该标志,则不记录事件。 注意 rules 字段必须在审计策略文件中提供。没有(0)规则的策略将被视为非法配置。

审计策略定义了哪些资源操作应该被审计以及审计记录的格式。在 Kubernetes 中,审计策略通过 Admission Controllers 实现,可以通过 Webhook 的方式进行集成。审计策略可以根据资源的类型、操作的类型和用户身份等信息进行过滤,以满足不同场景下的审计需求。

五.启用审计

5.1 引入审计

本次使用etcd2机器作为kubernetes集群的客户端,kctom是kubeconfig文件,etcd2机器使用kctom连接kubernetes集群,现在用户tom没有权限,需要先授权。kubeconfig文件在博客《Kubernetes(k8s)访问控制:身份认证》里已经详细介绍过了,这里不再赘述。

shell 复制代码
[root@etcd2 ~]# ls kc* -lh
-rw------- 1 root root 5.5K 5月   5 17:46 kctom

[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
Error from server (Forbidden): nodes is forbidden: User "tom" cannot list resource "nodes" in API group "" at the cluster scope

去kubernetes集群给tom用户授予cluster-admin权限,也可以授予部分权限。关于授权的详细信息,请查看博客《Kubernetes(k8s)访问控制:权限管理之RBAC授权/鉴权》。

shell 复制代码
root@k8scludes1:~# kubectl create clusterrolebinding crbindtom --clusterrole=cluster-admin --user=tom
clusterrolebinding.rbac.authorization.k8s.io/crbindtom created

授予cluster-admin权限之后,客户端就具有权限了。

shell 复制代码
[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 65d   v1.22.2
k8scludes3   Ready    <none>                 65d   v1.22.2

编辑pod配置文件,表示使用nginx镜像创建pod。

yaml 复制代码
root@k8scludes1:~# vim pod.yaml 

root@k8scludes1:~# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时,立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent:表示如果本地已经存在该镜像,则不重新下载;否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

在admissioncontr命名空间创建一个pod。

shell 复制代码
root@k8scludes1:~# kubectl apply -f pod.yaml 
pod/podtest created

在客户端etcd2机器可以看到这个pod。

shell 复制代码
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n admissioncontr
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          34s

在客户端删除pod。

shell 复制代码
[root@etcd2 ~]# kubectl delete pod podtest --kubeconfig=kctom -n admissioncontr
pod "podtest" deleted

kubectl get ev (查看事件),可以看到pod被删除掉了,但是看不到pod被谁删除掉了!

shell 复制代码
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
LAST SEEN   TYPE     REASON    OBJECT        SUBOBJECT                  SOURCE                MESSAGE                                                                            FIRST SEEN   COUNT   NAME
3m37s       Normal   Pulled    pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Container image "hub.c.163.com/library/nginx:latest" already present on machine   3m37s        1       podtest.16fab43e14342a3f
3m37s       Normal   Created   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Created container podtest                                                          3m37s        1       podtest.16fab43e1db484f4
3m36s       Normal   Started   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Started container podtest                                                          3m36s        1       podtest.16fab43e3315ada7
2m47s       Normal   Killing   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Stopping container podtest                                                         2m47s        1       podtest.16fab449c468f9af

我们的k8s集群会被各种用户连接使用,我们想知道连接k8s集群的各个用户做了什么操作?应该怎么办?通过审计解决!

5.2 启用审计

创建audit目录存放yaml文件。

shell 复制代码
root@k8scludes1:~# mkdir audit

root@k8scludes1:~# cd audit/

root@k8scludes1:~/audit# pwd
/root/audit

创建命名空间audit。

shell 复制代码
root@k8scludes1:~/audit# kubectl create ns audit
namespace/audit created

切换到命名空间audit。

shell 复制代码
root@k8scludes1:~/audit# kubens audit
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "audit".

root@k8scludes1:~/audit# kubectl get pod 
No resources found in audit namespace.

kubernetes默认并没有启用审计,需要启用审计功能。指定审计策略文件的路径:--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml

shell 复制代码
root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:~# grep audit-policy-file /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml

创建审计策略文件,文件内容可以先为空。

shell 复制代码
root@k8scludes1:~# mkdir /etc/kubernetes/audit/

root@k8scludes1:~# cd /etc/kubernetes/audit/

root@k8scludes1:/etc/kubernetes/audit# touch audit-policy.yaml

root@k8scludes1:/etc/kubernetes/audit# ls /etc/kubernetes/audit/audit-policy.yaml
/etc/kubernetes/audit/audit-policy.yaml

指定审计日志存放的路径:--audit-log-path=/var/log/kubernetes/audit/audit.log 。审计日志格式为JSONlines 格式

shell 复制代码
root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml

root@k8scludes1:/etc/kubernetes/audit# grep audit-log-path /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log

创建审计日志文件。

shell 复制代码
root@k8scludes1:/etc/kubernetes/audit# mkdir -p /var/log/kubernetes/audit/

root@k8scludes1:/etc/kubernetes/audit# touch /var/log/kubernetes/audit/audit.log

root@k8scludes1:/etc/kubernetes/audit# ls /var/log/kubernetes/audit/audit.log
/var/log/kubernetes/audit/audit.log

审计日志参数如下:

  • --audit-log-path 指定用来写入审计事件的日志文件路径。不指定此标志会禁用日志后端。- 意味着标准化
  • --audit-log-maxage 定义保留旧审计日志文件的最大天数
  • --audit-log-maxbackup 定义要保留的审计日志文件的最大数量
  • --audit-log-maxsize 定义审计日志文件轮转之前的最大大小(兆字节)

配置审计日志的参数。

yaml 复制代码
root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml

root@k8scludes1:/etc/kubernetes/audit# grep audit /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=365
    - --audit-log-maxbackup=730
    - --audit-log-maxsize=250

如果你的集群控制面以 Pod 的形式运行 kube-apiserver,需要通过 hostPath 数据卷来访问策略文件和日志文件所在的目录,这样审计记录才会持久保存下来。

现在/etc/kubernetes/audit/audit-policy.yaml和/var/log/kubernetes/audit/audit.log在宿主机里是存在的,但是kube-apiserver是以pod的方式运行的,pod里并不存在这两个文件,需要通过hostPath 数据卷进行映射。

/etc/kubernetes/manifests/kube-apiserver.yaml所有的修改如下截图:

注意:/var/log/kubernetes/audit/的readOnly为false,否则不能写入日志!

自此,成功启用审计了,下面开始配置审计策略。

六.审计策略

6.1 记录审计阶段为:ResponseStarted,审计级别为Metadata,apiVersion为group: "" 的日志

先不用重启kubelet,因为审计策略文件还没写好。

现在编写审计策略文件,该审计策略表示什么都不记录。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: None
    resources:
    - group: ""

重启kubelet使配置生效。

shell 复制代码
root@k8scludes1:~# systemctl restart kubelet

root@k8scludes1:~# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 66d   v1.22.2
k8scludes3   Ready    <none>                 66d   v1.22.2

在客户端etcd2机器执行命令:kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide。

shell 复制代码
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

查看审计日志,没有任何日志记录。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log 

审计级别修改为Metadata,表示记录请求的元数据(请求的用户、时间戳、资源、动词等等), 但是不记录请求或者响应的消息体。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""

重启kubelet使配置生效。

shell 复制代码
root@k8scludes1:~# systemctl daemon-reload ; systemctl restart kubelet

在客户端etcd2机器执行命令:kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide 。

shell 复制代码
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

查看审计日志,发现还是没有日志产生,看来重启kubelet还不行,现在重启机器。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log

现在直接重启k8scludes1机器,k8s worker节点不用重启。

shell 复制代码
root@k8scludes1:~# reboot

重启k8scludes1机器之后,现在/var/log/kubernetes/audit/audit.log里有很多数据,先删除数据,然后客户端执行命令,查看审计日志。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
3317

清空日志。

shell 复制代码
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

客户端执行kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide命令。

shell 复制代码
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

现在不停的产生日志。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
32

这是最新的日志。

json 复制代码
root@k8scludes1:~# tail -1 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"72736d33-508f-4279-89a5-d95aea7b98c5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader","verb":"update","user":{"username":"system:serviceaccount:ingress-nginx:ingress-nginx","uid":"70a36f2c-225f-450d-849e-2432db224f40","groups":["system:serviceaccounts","system:serviceaccounts:ingress-nginx","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["ingress-nginx-controller-684bbc4b45-jmpxk"],"authentication.kubernetes.io/pod-uid":["308fdf30-854b-49f7-b041-4e5d770c0667"]}},"sourceIPs":["192.168.110.130"],"userAgent":"nginx-ingress-controller/v1.0.0 (linux/amd64) ingress-nginx/041eb167c7bfccb1d1653f194924b0c5fd885e10","objectRef":{"resource":"configmaps","namespace":"ingress-nginx","name":"ingress-controller-leader","uid":"db625974-8e33-4bd6-a2e2-1816301af942","apiVersion":"v1","resourceVersion":"2739119"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:03:36.751232Z","stageTimestamp":"2022-06-22T13:03:36.759442Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ingress-nginx/ingress-nginx\" of Role \"ingress-nginx\" to ServiceAccount \"ingress-nginx/ingress-nginx\"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"}}

客户端不执行命令就产生了太多审计日志了,需要修改审计策略。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
1174

6.2 只记录audit命名空间里的日志

修改审计策略,现在配置只记录某个命名空间里的审计日志,namespaces: ["audit"]表示只记录audit命名空间里的日志。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
    namespaces: ["audit"]

重启kubelet使其生效,但是审计策略没有生效,重启机器。

shell 复制代码
root@k8scludes1:~# systemctl restart kubelet

重启k8s master节点。

shell 复制代码
root@k8scludes1:~# reboot

root@k8scludes1:~# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 66d   v1.22.2
k8scludes3   Ready    <none>                 66d   v1.22.2

重启k8s master节点之后,清空日志。

shell 复制代码
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

客户端执行kubectl get pod --kubeconfig=kctom -n default -o wide。

shell 复制代码
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n default -o wide
No resources found in default namespace.

客户端执行 kubectl get pod --kubeconfig=kctom -n audit -o wide。

shell 复制代码
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

查看审计日志,只记录了audit命名空间的操作,default命名空间的操作没有记录。

json 复制代码
root@k8scludes1:~# tail -2 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.732751Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.741665Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}

6.3 只记录audit命名空间的pods操作日志

修改审计策略,该审计策略表示只记录audit命名空间的pods操作。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使审计策略生效。

shell 复制代码
root@k8scludes1:~# reboot

重启之后,清空日志。

shell 复制代码
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

在客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide。

shell 复制代码
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

在客户端执行 kubectl get svc --kubeconfig=kctom -n audit -o wide 。

shell 复制代码
[root@etcd2 ~]# kubectl get svc --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

查看日志,发现只记录了audit命名空间的pod操作,svc操作没有记录。

json 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.351036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.434147Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}

6.4 只记录audit命名空间的pods,services,deployments操作日志

编辑审计策略文件,表示只记录audit命名空间的pods,services,deployments操作,因为deployments的apiVersion的父级为apps,所以需要group: "apps" 。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods","services"]
    - group: "apps"
      resources: ["deployments"]
    namespaces: ["audit"]

6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse

编辑审计策略文件,表示只记录audit命名空间的pods操作,审计级别为RequestResponse,记录事件的元数据,请求和响应的消息体。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使审计策略生效。

shell 复制代码
root@k8scludes1:~# reboot

创建一个pod。

shell 复制代码
root@k8scludes1:~# kubectl apply -f pod.yaml 
pod/podtest created

root@k8scludes1:~# kubectl get pod
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          5s

清空审计日志。

shell 复制代码
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide 。

shell 复制代码
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          48s   10.244.1.97   k8scludes3   <none>           <none>

查看审计日志,记录的内容很丰富,元数据,请求和响应的消息体都显示出来了。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"d6836f9b-4918-49f2-80dc-ad02e246949b","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.782241Z"}
......
23T00:42:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PodScheduled\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.97\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]}}}]},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.785883Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}

6.6 只记录audit命名空间下的tom用户的pods操作,其他用户操作不记录

编辑审计策略文件,设置审计策略:只记录audit命名空间下的tom用户的pods操作,其他用户的操作不记录。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    users: ["tom"]
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使审计策略生效。

shell 复制代码
root@k8scludes1:~# reboot 

清空日志。

shell 复制代码
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

使用管理员用户查看pod。

shell 复制代码
root@k8scludes1:~# kubectl get pod -n audit
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          29m

使用tom用户在客户端查看pod 。

shell 复制代码
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          29m   10.244.1.97   k8scludes3   <none>           <none>

查看日志,可以发现只记录了tom用户的pod查询操作。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.103036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.107872Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}

6.7 rules规则是从上往下匹配的,第一条规则已经匹配了,第二条就不匹配了

编辑审计策略文件,有两条策略,一条是:apiVersion为group: ""的操作不记录日志,另外一条是:只记录audit命名空间下tom用户的pod操作。

yaml 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  #apiVersion为group: ""的操作不记录日志
  - level: None
    resources:
    - group: ""
  #只记录audit命名空间下tom用户的pod操作
  - level: Metadata
    users: ["tom"]
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使配置生效。

shell 复制代码
root@k8scludes1:~# reboot 

清空日志。

shell 复制代码
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

使用管理员用户查询audit命名空间的pod。

shell 复制代码
root@k8scludes1:~# kubectl get pod -n audit
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          37m

使用tom用户在客户端机器etcd2上查询audit命名空间的pod。

shell 复制代码
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          37m   10.244.1.97   k8scludes3   <none>           <none>

查看审计日志,可以发现在客户端使用tom用户查询pod和使用管理员用户查看pod都没有生成审计日志,rules规则是从上往下匹配的,第一条规则已经匹配了,第二条就不匹配了。

shell 复制代码
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log

6.8 在 Metadata 级别为所有请求生成日志

编辑审计策略文件,在 Metadata 级别为所有请求生成日志。

shell 复制代码
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata

七.总结

在 Kubernetes 1.22.2 环境中实施审计策略可以帮助管理员监控和记录集群中的资源操作,确保集群的安全性和符合性。通过启用审计 Admission Controller 和配置相应的审计策略,我们可以灵活地控制审计记录的格式和范围。