k8s 集群搭建的一些坑

k8s集群部署的时候会遇到很多的坑，即使看网上的文档也可能遇到各种的坑。

安装准备

1、虚拟机两台（ip按自己的网络环境相应配置）(master/node)

192.168.100.215 k8s-master

192.168.100.216 k8s-node1

2、关闭防火墙(master/node)

systemctl stop firewalld

systemctl disable firewalld

3、关闭selinux(master/node)

setenforce 0 # 临时关闭

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config # 永久关闭

4、关闭swap(master/node)

swapoff -a # 临时关闭；关闭swap主要是为了性能考虑

free # 可以通过这个命令查看swap是否关闭了

sed -ri 's/.swap. /#&/' /etc/fstab # 永久关闭

5、添加主机名与IP对应的关系(master/node)

$ vim /etc/hosts

#添加如下内容：

192.168.100.215 k8s-master

192.168.100.216 k8s-node1

#保存退出

6、修改主机名(master/node)

#k8s-master

$root@localhost \~$ hostname

localhost.localdomain

$root@localhost \~$ hostname k8s-master ##临时生效

$root@localhost \~$ hostnamectl set-hostname k8s-master ##重启后永久生效

#k8s-node1

$root@localhost \~$ hostname

localhost.localdomain

$root@localhost \~$ hostname k8s-node1 ##临时生效

$root@localhost \~$ hostnamectl set-hostname k8s-node1 ##重启后永久生效

7、桥接设置(master/node)

$ cat > /etc/sysctl.d/k8s.conf << EOF

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

$ sysctl --system

p.s.

以上几步最好照着都执行一下，以免后面报一大堆错。

安装docker

这一步需要注意docker版本与k8s版本对照关系，否则执行master初始化的时候会报错。

部署Kubernetes (master)

kubeadm init

--apiserver-advertise-address=192.168.100.215

--image-repository registry.aliyuncs.com/google_containers

--kubernetes-version v1.18.0

--service-cidr=10.1.0.0/16

--pod-network-cidr=10.244.0.0/16

执行命令会出现可能一些告警，需要处理一下

cgroupfs切换为systemd

$WARNING IsDockerSystemdCheck$ : detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

解决方案：

新建daemon.json切换为systemd

mkdir /etc/docker/

cat > /etc/docker/daemon.json << EOF

{

"registry-mirrors": $"https://gqs7xcfd.mirror.aliyuncs.com","https://hub-mirror.c.163.com"$ ,

"exec-opts": $"native.cgroupdriver=systemd"$ ,

"log-driver": "json-file",

"log-opts": {

"max-size": "100m"

"storage-driver": "overlay2"

}

EOF

Docker 与k8s版本不兼容

$WARNING SystemVerification$ : this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 19.03

解决方案：

删除旧版本的docker

sudo yum remove docker-ce

sudo rm -rf /var/lib/docker

安装兼容版本的docker

yum -y install docker-ce-19.03.9-3.el7 docker-ce-cli-19.03.9-3.el7

systemctl enable docker

systemctl start docker

在两台Node节点上执行join命令

kubeadm join 192.168.200.128:6443 --token 29f2jw.4lczl18ymfnzcax7

--discovery-token-ca-cert-hash sha256:57e05025b3a40415cd9297e089f0439f265a6e5df46ff1b431e017f326b313

cannot stat '/etc/kubernetes/admin.conf'

从master节点拷贝过来。

bash 复制代码

# master节点执行
[root@master ~]# scp /etc/kubernetes/admin.conf root@192.168.247.137:/etc/kubernetes/admin.conf
The authenticity of host '192.168.247.137 (192.168.247.137)' can't be established.
ECDSA key fingerprint is SHA256:oj2gG+nDvkzbS+IggsOxKVwMTVP2zxClBoNEPXmHvzk.
ECDSA key fingerprint is MD5:fa:f6:9a:89:1a:4a:54:63:fe:3d:70:88:c4:a4:4d:ea.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.247.137' (ECDSA) to the list of known hosts.
root@192.168.247.137's password: 
admin.conf                                                                    100% 5451     3.5MB/s   00:00    
 
# node节点执行
[root@slave2 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@slave2 ~]# source ~/.bash_profile
[root@slave2 ~]# chown $(id -u):$(id -g) $HOME/.kube/config
 
# 验证
[root@slave2 ~]# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6867cdf567-vkf99   1/1     Running   0          6m38s

node节点 kubectl get nodes 报错

error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

bash 复制代码

[root@localhost ~]# kubectl get nodes
error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

解决方案

bash 复制代码

[root@localhost ~]# find / -name  admin.conf
/etc/kubernetes/admin.conf
[root@localhost ~]# mkdir -p $HOME/.kube
[root@localhost ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@localhost ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

删除pod service deployment

复制代码

删除service
kubectl delete deployment grafana -n kube-system
kubectl delete service grafana -n kube-system
kubectl delete serviceaccount grafana -n kube-system
kubectl delete serviceaccount grafana -n kube-system --force --grace-period=0

查看
kubectl get pods,deployments,services,serviceaccounts -n kube-system | grep grafana

删除 deployment
kubectl delete deployment monitoring-grafana -n kube-system
kubectl delete pod monitoring-grafana-b46f4c778-z74wg -n kube-system