文章目录
前言
使用nfs-client-provisioner这个应用，利用nfs server给kubernets提供作为持久化后端，并且动态提供pv。所有节点需要安装nfs-utils组件，并且nfs服务器与kubernets worker节点都能网络连通。
部署NFS服务器
注：这里使用master节点作为nfs服务器，根据自己资源情况进行调整部署。
安装nfs组件及创建数据目录
[root@k8s-master ~]# yum install -y nfs-utils rpcbind
[root@k8s-master ~]# mkdir -p /data/nfs_data
编辑exportfs文件
# cat /etc/exports
/data/nfs_data 10.255.82.0/24(rw,no_root_squash)
配置生效
# exportfs -rav
启动rpcbind、nfs服务
[root@k8s-master ~]# systemctl restart rpcbind && systemctl enable rpcbind
[root@k8s-master ~]# systemctl restart nfs && systemctl enable nfs
showmount测试
[root@k8s-master ~]# showmount -e 10.255.82.25
Export list for 10.255.82.25
/data/nfs_data 10.255.82.0/24
[root@k8s-master ~]#
K8S部署NFS
nfs部署资源文件如下
[root@k8s-master nfs]# ls -l
总用量 12
-rw-r--r-- 1 root root 181 12月 8 17:47 class.yaml
-rw-r--r-- 1 root root 987 12月 8 17:43 deployment.yaml
-rw-r--r-- 1 root root 1509 12月 8 17:34 rbac.yaml
[root@k8s-master nfs]#
创建nfs角色权限
[root@k8s-master nfs]# kubectl apply -f rbac.yaml
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
[root@k8s-master nfs]#
部署nfs-client-provisioner
[root@k8s-master nfs]# kubectl apply -f deployment.yaml
serviceaccount/nfs-client-provisioner created
deployment.apps/nfs-client-provisioner created
[root@k8s-master nfs]#
创建nfs storageclass
[root@k8s-master nfs]# kubectl apply -f class.yaml
storageclass.storage.k8s.io/managed-nfs-storage created
[root@k8s-master nfs]#
验证nfs部署成功
[root@k8s-master nfs]# kubectl get pod -l app=nfs-client-provisioner
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-647d8f5c7b-5lk5b 1/1 Running 3 (13h ago) 15h
[root@k8s-master nfs]#
查看nfs storageclass
[root@k8s-master nfs]# kubectl get storageclasses.storage.k8s.io
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage fuseim.pri/ifs Delete Immediate true 15h
部署nginx服务使用nfs存储持久化
[root@k8s-master nfs]# cat nfs-ngx-sts.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 1 # by default is 1
minReadySeconds: 10 # by default is 0
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 1Gi
[root@k8s-master nfs]#
[root@k8s-master nfs]# kubectl apply -f nfs-ngx-sts.yaml
查看ngx-pod运行状态
[root@k8s-master nfs]# kubectl get pod | grep web
web-0 1/1 Running 0 13h
[root@k8s-master nfs]#
查看pvc状态
[root@k8s-master nfs]# kubectl get pvc -l app=nginx
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pvc-782925cf-307b-4602-a226-871825fa86ac 1Gi RWO managed-nfs-storage 13h
[root@k8s-master nfs]#
验证ngx服务存储nfs持久化
[root@k8s-master nfs]# kubectl exec -it web-0 -- bash
root@web-0:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 17G 5.1G 12G 30% /
tmpfs 64M 0 64M 0% /dev
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda2 17G 5.1G 12G 30% /etc/hosts
shm 64M 0 64M 0% /dev/shm
192.168.31.102:/data/nfs_data/default-www-web-0-pvc-782925cf-307b-4602-a226-871825fa86ac 17G 7.5G 9.6G 45% /usr/share/nginx/html
tmpfs 3.8G 12K 3.8G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 2.0G 0 2.0G 0% /proc/acpi
tmpfs 2.0G 0 2.0G 0% /proc/scsi
tmpfs 2.0G 0 2.0G 0% /sys/firmware
root@web-0:/# ls -lh /usr/share/nginx/html/
total 0
root@web-0:/# echo "nfs storage" > /usr/share/nginx/html/index.html
root@web-0:/# cat /usr/share/nginx/html/index.html
nfs storage
root@web-0:/# exit
exit
nfs服务器端验证数据
[root@k8s-master nfs]# cat /data/nfs_data/default-www-web-0-pvc-782925cf-307b-4602-a226-871825fa86ac/index.html
nfs storage
[root@k8s-master nfs]#
ngx资源删除验证数据是否持久化nfs存储
[root@k8s-master nfs]# kubectl delete -f nfs-ngx-sts.yaml
service "nginx-svc" deleted
statefulset.apps "web" deleted
[root@k8s-master nfs]# cat /data/nfs_data/default-www-web-0-pvc-782925cf-307b-4602-a226-871825fa86ac/index.html
nfs storage
[root@k8s-master nfs]#
问题记录
pvc创建处于pending状态事件描述：Normal ExternalProvisioning 2s (x4 over 48s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "fuseim.pri/ifs" or manually created by system administrator
原因：
Kubernetes v1.20 (opens new window)开始，默认删除了 metadata.selfLink 字段，然而，部分应用仍然依赖于这个字段，例如 nfs-client-provisioner。如果仍然要继续使用这些应用，您将需要重新启用该字段。
解决方法：
通过配置 apiserver 启动参数中的 --feature-gates 中的 RemoveSelfLink=false，可以重新启用 metadata.selfLink 字段。
-
如果使用kubeadm安装Kubernetes，请修改/etc/kubernetes/manifests/kube-apiserver.yaml文件，并在其启动参数中增加一行- --feature-gates=RemoveSelfLink=false，如下所示：
然后更新kube-apiserver.yaml即可
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
-
② 如果是通过二进制部署kube-apiserver
通过systemctl status kube-apiserver 这个命令找到 kube-apiserver.service文件，
最好先备份，然后再修改，即在 kube-apiserver.service中添加 --feature-gates=RemoveSelfLink=false。
systemctl daemon-reload
systemctl restart kube-apiserver
