K8S 的持久化
K8S 实现持久化存储的方法有很多种
例如 卷 (Volume), 持久卷(PV), 临时卷(EV) 等, 还有很多不常用的选项上图没有列出来
其中Volume 本身也分很多种
包括
Secret, configMap(之前的文章covered了), hostPath, emptyDir等
本文主要focus on hostPath
HostPath 的简介
官方定义:
hostPath 卷能将主机节点文件系统上的文件或目录挂载到你的 Pod 中。 虽然这不是大多数 Pod 需要的,但是它为一些应用提供了强大的逃生舱。
简单来讲就是让k8s node 的目录or 文件map在你的POD 容器里
至于为什么上面提到的逃生舱, 是因为假如你的POD 崩溃了or 被shutdown, 我们仍然可以在node对应的path上找到我们想要的数据(前提是把相应的数据output 到hostpath)
但是 官方并不推荐使用hostpath 把数据输出到node上
建议用local PersisentVolume代替, 主要是安全风险的原因
原因:
如果你通过准入时的验证来限制对节点上特定目录的访问,这种限制只有在你额外要求所有 hostPath 卷的挂载都是只读的情况下才有效。如果你允许不受信任的 Pod 以读写方式挂载任意主机路径, 则该 Pod 中的容器可能会破坏可读写主机挂载卷的安全性。
无论 hostPath 卷是以只读还是读写方式挂载,使用时都需要小心,这是因为:
- 访问主机文件系统可能会暴露特权系统凭证(例如 kubelet 的凭证)或特权 API(例如容器运行时套接字), 这些可以被用于容器逃逸或攻击集群的其他部分。
- 具有相同配置的 Pod(例如基于 PodTemplate 创建的 Pod)可能会由于节点上的文件不同而在不同节点上表现出不同的行为。
- hostPath 卷的用量不会被视为临时存储用量。 你需要自己监控磁盘使用情况,因为过多的 hostPath 磁盘使用量会导致节点上的磁盘压力。
HostPath 的类型
除了必需的 path 属性外,你还可以选择为 hostPath 卷指定 type。
value | desc |
---|---|
"" | 空字符串(默认)用于向后兼容,这意味着在安装 hostPath 卷之前不会执行任何检查。 |
DirectoryOrCreate | 如果在给定路径上什么都不存在,那么将根据需要创建空目录,权限设置为 0755,具有与 kubelet 相同的组和属主信息。 |
Directory | 在给定路径上必须存在的目录。 |
FileOrCreate | 如果在给定路径上什么都不存在,那么将在那里根据需要创建空文件,权限设置为 0644,具有与 kubelet 相同的组和所有权。 |
File | 在给定路径上必须存在的文件。 |
Socket | 在给定路径上必须存在的 UNIX 套接字。 |
CharDevice | (仅 Linux Node) 在给定路径上必须存在的字符设备。 |
BlockDevice | (仅 Linux Node) 在给定路径上必须存在的块设备。 |
例子
这个例子是测试多个pod同时往1个hostpath 写日志
添加filter 让其可以输出api 调用日志
我们先为springboot cloud-order service 增加1个filter
java
@Component
@WebFilter
@Order(1)
@Slf4j
public class InfoFilter implements Filter {
@Autowired
private String hostname;
@Override
public void init(FilterConfig filterConfig) throws ServletException {
// Initialization code
}
@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
// Log the request information
log.info("API of hostname: {} has been called, Request Method: {}, Request URI: {}, Requested from : {}", hostname, httpRequest.getMethod(), httpRequest.getRequestURI(), httpRequest.getRemoteAddr());
// You can log more information as needed
// Call the next filter in the chain
chain.doFilter(request, response);
}
@Override
public void destroy() {
// Cleanup code
}
}
为k8s-node1 加上label
这个例子测试 多个POD 同时为1个hostpath 写日志
所以为了方便测试, 需要把所有PODs 都部署在同1个node上。
所以我们要为k8s-node1 加上label
bash
gateman@MoreFine-S500:/app/logs$ kubectl label nodes k8s-node1 cloud-order=enabled
node/k8s-node1 labeled
gateman@MoreFine-S500:/app/logs$ kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
k8s-master Ready control-plane,master 190d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
k8s-node0 Ready <none> 190d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node0,kubernetes.io/os=linux
k8s-node1 Ready <none> 190d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cloud-order=enabled,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node1,kubernetes.io/os=linux
k8s-node3 Ready <none> 169d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node3,kubernetes.io/os=linux
可以见到, cloud-order=enabled 这个label 已经加上到了k8s-node1
在k8s-node1 提前创建1个folder
这个例子用的类型是Directory
当然如果用DirectoryOrCreate 是可以skip 这个步骤
bash
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ gcloud compute ssh k8s-node1
WARNING: This command is using service account impersonation. All API calls will be executed as [terraform@jason-hsbc.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [terraform@jason-hsbc.iam.gserviceaccount.com].
No zone specified. Using zone [europe-west2-c] for instance: [k8s-node1].
External IP address was not found; defaulting to using IAP tunneling.
WARNING:
To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth
WARNING: This command is using service account impersonation. All API calls will be executed as [terraform@jason-hsbc.iam.gserviceaccount.com].
Warning: Permanently added 'compute.7230880099421476498' (ED25519) to the list of known hosts.
Linux k8s-node1 5.10.0-32-cloud-amd64 #1 SMP Debian 5.10.223-1 (2024-08-10) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Mar 9 19:39:42 2024 from 192.168.0.35
gateman@k8s-node1:~$ sudo su -
root@k8s-node1:~# mkdir /k8s-shared
root@k8s-node1:~# cd /k8s-shared/
root@k8s-node1:/k8s-shared# ls
root@k8s-node1:/k8s-shared# mkdir logs
root@k8s-node1:/k8s-shared# ls
logs
编写deployment yaml
deployment-cloud-order-hostpath.yaml
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels: # label of this deployment
app: cloud-order # custom defined
author: nvd11
name: deployment-cloud-order # name of this deployment
namespace: default
spec:
replicas: 3 # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.
revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollback
selector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error
# error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..
matchLabels:
app: cloud-order
strategy: # Strategy of upodate
type: RollingUpdate # RollingUpdate or Recreate
rollingUpdate:
maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the update
maxUnavailable: 25% # The maximum number of Pods that can be unavailable during the update
template: # Pod template
metadata:
labels:
app: cloud-order # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`
spec: # specification of the Pod
containers:
- image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.1.0 # image of the container
imagePullPolicy: Always
name: container-cloud-order
command: ["bash"]
args:
- "-c"
- |
java -jar -Dserver.port=8080 app.jar --spring.profiles.active=$APP_ENVIRONMENT --logging.file.name=/app/logs/cloud-order.log
env: # set env varaibles
- name: APP_ENVIRONMENT # name of the environment variable
value: prod # value of the environment variable
volumeMounts:
- name: volume-log
mountPath: /app/logs/
readOnly: false # read only is set to false
ports:
- containerPort: 8080
name: cloud-order
nodeSelector:
cloud-order: enabled
volumes:
- name: volume-log
hostPath:
path: /k8s-shared/logs # the path on the host (k8s node)
type: Directory
restartPolicy: Always # Restart policy for all containers within the Pod
terminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully
注意几点,
- 启动命令加上 --logging.file.name=/app/logs/cloud-order.log 让日志输出到指定位置
- 加上volume-log 这个hostpath, 映射的主机路径是 /k8s-shared/logs
- volumemount 那里把 volume-log mount在了 /app/logs
- 使用nodeSelector 让pods 都部署在1个node
部署
bash
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order/volumes$ kubectl delete deployment deployment-cloud-order
deployment.apps "deployment-cloud-order" deleted
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order/volumes$ kubectl apply -f deployment-cloud-order-hostpath.yaml
deployment.apps/deployment-cloud-order created
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order/volumes$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deployment-bq-api-service-6f6ffc7866-8djx9 1/1 Running 3 (13d ago) 18d
deployment-bq-api-service-6f6ffc7866-g4854 1/1 Running 12 (13d ago) 61d
deployment-bq-api-service-6f6ffc7866-lwxt7 1/1 Running 14 (13d ago) 64d
deployment-bq-api-service-6f6ffc7866-mxwcq 1/1 Running 11 (13d ago) 61d
deployment-cloud-order-ff4989c97-h8ktj 1/1 Running 0 7s
deployment-cloud-order-ff4989c97-xg2x2 1/1 Running 0 7s
deployment-cloud-order-ff4989c97-zc2dq 1/1 Running 0 7s
测试
多次调用cloud-order的某个api
bash
curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info
curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info
curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info
curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info
curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info
进入k8s-node1 查看相应的日志文件:
bash
root@k8s-node1:/k8s-shared/logs# tail -f cloud-order.log
...
2024-09-01T14:28:14.298Z INFO 1 --- [http-nio-8080-exec-1] com.home.clouduser.filter.InfoFilter : API of hostname: deployment-cloud-order-ff4989c97-h8ktj has been called, Request Method: GET, Request URI: /actuator/info, Requested from : 192.168.0.35
2024-09-01T14:28:15.384Z INFO 1 --- [http-nio-8080-exec-2] com.home.clouduser.filter.InfoFilter : API of hostname: deployment-cloud-order-ff4989c97-h8ktj has been called, Request Method: GET, Request URI: /actuator/info, Requested from : 192.168.0.35
2024-09-01T14:28:15.856Z INFO 1 --- [http-nio-8080-exec-2] com.home.clouduser.filter.InfoFilter : API of hostname: deployment-cloud-order-ff4989c97-zc2dq has been called, Request Method: GET, Request URI: /actuator/info, Requested from : 192.168.0.35
2024-09-01T14:28:17.211Z INFO 1 --- [http-nio-8080-exec-3] com.home.clouduser.filter.InfoFilter : API of hostname: deployment-cloud-order-ff4989c97-h8ktj has been called, Request Method: GET, Request URI: /actuator/info, Requested from : 192.168.0.35
2024-09-01T14:28:18.183Z INFO 1 --- [http-nio-8080-exec-4] com.home.clouduser.filter.InfoFilter : API of hostname: deployment-cloud-order-ff4989c97-h8ktj has been called, Request Method: GET, Request URI: /actuator/info, Requested from : 192.168.0.35
2024-09-01T14:28:19.128Z INFO 1 --- [http-nio-8080-exec-2] com.home.clouduser.filter.InfoFilter : API of hostname: deployment-cloud-order-ff4989c97-xg2x2 has been called, Request Method: GET, Request URI: /actuator/info, Requested from : 192.168.0.35
可以见到 , 由于loadbalancer 的关系, 多个pods轮流被调用, 但是都可以写进同1个hostpath内的日志文件
所以如果用append 方式(例如写日子) 多个pod 写同1个host path 是不会由文件冲突问题的!