目录
[(四)部署kubelet kubeadm kubectl工具](#(四)部署kubelet kubeadm kubectl工具)
[1、使用 YUM 方式安装Kubernetes时,推荐使用阿里的yum。](#1、使用 YUM 方式安装Kubernetes时,推荐使用阿里的yum。)
[2、安装kubelet kubeadm kubectl](#2、安装kubelet kubeadm kubectl)
[(五)部署Kubernetes Master](#(五)部署Kubernetes Master)
1)修改kube-controller-manager.yaml文件
1、在k8s-master2和k8s-master3节点创建文件夹
[(八)加入Kubernetes Node](#(八)加入Kubernetes Node)
2、在Kubernetes集群中创建一个pod,验证是否正常运行。
[3、创建完 Deployment 的资源清单之后,使用 create 执行资源清单来创建容器。](#3、创建完 Deployment 的资源清单之后,使用 create 执行资源清单来创建容器。)
【kubernetes】k8s高可用集群搭建(三主三从)
一、服务器设置
准备6台虚拟机,3台master节点,3台node节点,保证master节点数为>=3的奇数。
****网络:****所有机器网络互通、可以访问外网
****硬件:****2核CPU+、2G内存+、硬盘20G+
|-----------------|--------|---------------|---------------------------------------------------|
| IP地址 | 角色 | 主机名 | 需要用到的服务 |
| 192.168.100.131 | master | k8s-master1 | keepalived haproxy docker kubectl kubeadm kubelet |
| 192.168.100.132 | master | k8s-master2 | keepalived haproxy docker kubectl kubeadm kubelet |
| 192.168.100.133 | master | k8s-master3 | keepalived haproxy docker kubectl kubeadm kubelet |
| 192.168.100.134 | node | k8s-node1 | docker kubectl kubeadm kubelet |
| 192.168.100.135 | node | k8s-node2 | docker kubectl kubeadm kubelet |
| 192.168.100.136 | node | k8s-node3 | docker kubectl kubeadm kubelet |
| 192.168.100.154 | VIP | master.k8s.io | 代理master1/2/3 |
二、环境配置
1、关闭防火墙
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
2、关闭selinux
[root@localhost ~]# sed -i 's/enforcing/disabled/' /etc/selinux/config
[root@localhost ~]# setenforce 0
3、关闭swap
[root@localhost ~]# swapoff -a
[root@localhost ~]# sed -ri 's/.*swap.*/#&/' /etc/fstab
![](https://file.jishuzhan.net/article/1692538624114954242/39b065a6eca346c0ac03befdac22bc3a.png)
4、修改主机名(根据主机角色不同,做相应修改)
hostname k8s-node1 建议做永久主机名配置 hostnamectl set-hostname k8s-master1
bash
5、主机名映射
[root@k8s-master1 ~]# cat >> /etc/hosts << EOF
192.168.100.131 master1.k8s.io k8s-master1
192.168.100.132 master2.k8s.io k8s-master2
192.168.100.133 master3.k8s.io k8s-master3
192.168.100.134 node1.k8s.io k8s-node1
192.168.100.135 node2.k8s.io k8s-node2
192.168.100.136 node3.k8s.io k8s-node3
192.168.100.154 master.k8s.io k8s-vip
EOF
![](https://file.jishuzhan.net/article/1692538624114954242/16f78f99e15a436caa0ee96522782fde.png)
6、将桥接的IPv4流量传递到iptables的链
[root@k8s-master1 ~]# cat << EOF >> /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@k8s-master1 ~]# modprobe br_netfilter
[root@k8s-master1 ~]# sysctl -p
![](https://file.jishuzhan.net/article/1692538624114954242/cadaed9d790649e9b18a586d7fa4219d.png)
7、时间同步
[root@k8s-master1 ~]# yum install ntpdate -y
[root@k8s-master1 ~]# ntpdate time.windows.com
![](https://file.jishuzhan.net/article/1692538624114954242/573df92a654e47ba81fd5d879eccab7b.png)
8、master之间进行免密登录设置
[root@k8s-master1 ~]# ssh-keygen
[root@k8s-master1 ~]# ssh-cpoy-id 192.168.100.131
[root@k8s-master1 ~]# ssh-cpoy-id 192.168.100.132
[root@k8s-master1 ~]# ssh-cpoy-id 192.168.100.133
同理:
master2/3操作,互相之间免密登录
![](https://file.jishuzhan.net/article/1692538624114954242/830ec80e85604199bb6c20e2f831031c.png)
![](https://file.jishuzhan.net/article/1692538624114954242/c99d0c280c0742c8938d2b0f922e94e1.png)
![](https://file.jishuzhan.net/article/1692538624114954242/1299011ca30a4a7292858fc72296b8ec.png)
三、安装部署
(一)配置部署keepalived服务
1、安装Keepalived(所有master主机)
[root@k8s-master1 ~]# yum install -y keepalived
![](https://file.jishuzhan.net/article/1692538624114954242/24dc736e4a7249a69fb68af3aadc3359.png)
2、k8s-master1/2/3节点配置
[root@k8s-master1 ~]#
cat /etc/keepalived/keepalived.conf
![](https://file.jishuzhan.net/article/1692538624114954242/8d8fce9098c34b94bb6ebf731aae1d35.png)
k8s-master2节点配置
[root@k8s-master2 ~]#
cat /etc/keepalived/keepalived.conf
![](https://file.jishuzhan.net/article/1692538624114954242/ec58b4e2a15c47c5809ac49d9fb07dfc.png)
k8s-master3节点配置
[root@k8s-master3 ~]#
cat /etc/keepalived/keepalived.conf
![](https://file.jishuzhan.net/article/1692538624114954242/fe7590927637426e8e0cd050a9529fae.png)
3、启动和检查
所有master节点都要执行
[root@k8s-master1 ~]# systemctl start keepalived
[root@k8s-master1 ~]# systemctl enable keepalived
4、查看启动状态
[root@k8s-master1 ~]# systemctl status keepalived
![](https://file.jishuzhan.net/article/1692538624114954242/1115be52bff9426889a961bdb344a56b.png)
5、启动完成后在master1查看网络信息
[root@k8s-master1 ~]# ip a s ens33
![](https://file.jishuzhan.net/article/1692538624114954242/4545140c5dbb461e88954b4e4e7813d5.png)
(二)配置部署haproxy服务
1、所有master主机安装haproxy
[root@k8s-master1 ~]# yum install -y haproxy
![](https://file.jishuzhan.net/article/1692538624114954242/ed3d67deacef4fbcb8864512e087d8d7.png)
每台master节点中的配置均相同,配置中声明了后端代理的每个master节点服务器,指定了haproxy的端口为16443,因此16443端口为集群的入口。
2、修改配置文件
[root@k8s-master1 ~]# cat /etc/haproxy/haproxy.cfg
![](https://file.jishuzhan.net/article/1692538624114954242/08983c02bddc45229c48fbbd622d6acf.png)
3、启动和检查
所有master节点都要执行
[root@k8s-master1 ~]# systemctl start haproxy
[root@k8s-master1 ~]# systemctl enable haproxy
4、查看启动状态
[root@k8s-master1 ~]# systemctl status haproxy
![](https://file.jishuzhan.net/article/1692538624114954242/f7d1b4e79f5a48e7a243ae29b7c03d03.png)
5、检查端口
[root@k8s-master1 ~]# netstat -lntup|grep haproxy
![](https://file.jishuzhan.net/article/1692538624114954242/7f1e5588ba3b447fac66789d4b15ef25.png)
(三)配置部署Docker服务(所有主机)
所有主机上分别部署 Docker 环境,因为 Kubernetes 对容器的编排需要 Docker 的支持。
[root@k8s-master ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[root@k8s-master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
使用 YUM 方式安装 Docker 时,推荐使用阿里的 YUM 源。
[root@k8s-master ~]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@k8s-master ~]# yum clean all && yum makecache fast
[root@k8s-master ~]# yum -y install docker-ce
[root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker
镜像加速器(所有主机配置)
[root@k8s-master ~]# cat /etc/docker/daemon.json
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart docker
![](https://file.jishuzhan.net/article/1692538624114954242/4154fa2532424ad48df32ff7909c46c8.png)
![](https://file.jishuzhan.net/article/1692538624114954242/d7015530904542d0a415309089119d59.png)
(四)部署kubelet kubeadm kubectl工具
1、使用 YUM 方式安装Kubernetes时,推荐使用阿里的yum。
所有主机配置
[root@k8s-master ~]#
cat /etc/yum.repos.d/kubernetes.repo
[root@k8s-master ~]# ls /etc/yum.repos.d/
backup Centos-7.repo CentOS-Media.repo CentOS-x86_64-kernel.repo docker-ce.repo kubernetes.repo
![](https://file.jishuzhan.net/article/1692538624114954242/b90350fba0454c54b942c0fadf949cc1.png)
2、安装kubelet kubeadm kubectl
所有主机配置
[root@k8s-master ~]# yum install -y kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0
[root@k8s-master ~]# systemctl enable kubelet
![](https://file.jishuzhan.net/article/1692538624114954242/4c373e59dbd24c019cf2fb069ffa21f9.png)
(五)部署Kubernetes Master
在具有vip的master上操作。此处的vip节点为k8s-master1。
1、创建kubeadm-config.yaml文件
[root@k8s-master1 ~]# cat kubeadm-config.yaml
![](https://file.jishuzhan.net/article/1692538624114954242/802f3ce4421e4830b7ed627dcda1f434.png)
2、查看所需镜像信息
[root@k8s-master1 ~]# kubeadm config images list --config kubeadm-config.yaml
![](https://file.jishuzhan.net/article/1692538624114954242/fc3f3c79ad904fe3b0133e1aa7d0a607.png)
3、下载k8s所需的镜像(所有master主机)
[root@k8s-master1 ~]# kubeadm config images pull --config kubeadm-config.yaml
![](https://file.jishuzhan.net/article/1692538624114954242/f0be7753de5d4303ace3d71155a784a2.png)
4、使用kubeadm命令初始化k8s
[root@k8s-master1 ~]# kubeadm init --config kubeadm-config.yaml
![](https://file.jishuzhan.net/article/1692538624114954242/31606a235f844cd1b7965ac9c80a6033.png)
初始化中的错误:
![](https://file.jishuzhan.net/article/1692538624114954242/99ab6e59acdb46398bfa6abfdbc16531.png)
执行以下命令后重新执行初始化命令
[root@k8s-master1 ~]# echo "1" > /proc/sys/net/ipv4/ip_forward
[root@k8s-master1 ~]# kubeadm init --config kubeadm-config.yaml
![](https://file.jishuzhan.net/article/1692538624114954242/4ebcb43920c34695976a9528d18e2a42.png)
5、根据初始化的结果操作
[root@k8s-master1 ~]# mkdir -p $HOME/.kube
[root@k8s-master1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master1 ~]# sudo chown (id -u):(id -g) $HOME/.kube/config
![](https://file.jishuzhan.net/article/1692538624114954242/ee13dfbfe5f049958579b773ca514ff1.png)
6、查看集群状态
[root@k8s-master1 manifests]# kubectl get cs
![](https://file.jishuzhan.net/article/1692538624114954242/d82f57386d1c447497813e67678a2e29.png)
注意:出现以上错误情况,是因为/etc/kubernetes/manifests/下的kube-controller-manager.yaml和kube-scheduler.yaml设置的默认端口为0导致的,解决方式是注释掉对应的port即可
1)修改kube-controller-manager.yaml文件
![](https://file.jishuzhan.net/article/1692538624114954242/0bf078fcf57e413fa0bc74e62ec5bf34.png)
2)修改kube-scheduler.yaml文件
![](https://file.jishuzhan.net/article/1692538624114954242/11daa5b8b8424656820a7ee3b945fe3c.png)
3)查看集群状态
[root@k8s-master1 ~]# kubectl get cs
![](https://file.jishuzhan.net/article/1692538624114954242/e8830736253f4e69be93d9fbbb508a80.png)
4)查看pod信息
[root@k8s-master1 ~]# kubectl get pods -n kube-system
![](https://file.jishuzhan.net/article/1692538624114954242/8cd329aa15d64ade834ee226f4b6eacb.png)
5)查看节点信息
[root@k8s-master1 ~]# kubectl get nodes
![](https://file.jishuzhan.net/article/1692538624114954242/dce398eb51b34135afe842c4a8870faf.png)
(六)安装集群网络
在k8s-master1节点执行
[root@k8s-master1 ~]# docker load < flannel_v0.12.0-amd64.tar
![](https://file.jishuzhan.net/article/1692538624114954242/49ce1e0a1c46447a8e644f7b7e19d30b.png)
没有变成ready:(原因是网络插件缺失)
![](https://file.jishuzhan.net/article/1692538624114954242/764a0a4804fc439bbc44a557b25b7e1a.png)
上传插件:
[root@k8s-master1 ~]# tar xf cni-plugins-linux-amd64-v0.8.6.tgz
[root@k8s-master1 ~]# cp flannel /opt/cni/bin/
[root@k8s-master1 ~]# kubectl delete -f kube-flannel.yml 删除之前的apply操作
![](https://file.jishuzhan.net/article/1692538624114954242/8f14a31a98b94f8fada4399213fda9fc.png)
再次查看节点信息:
[root@k8s-master1 ~]# kubectl apply -f kube-flannel.yml
[root@k8s-master1 ~]# kubectl get nodes
![](https://file.jishuzhan.net/article/1692538624114954242/ae6ba589e495473f98e3ab248e117752.png)
(七)添加master节点
1、在k8s-master2和k8s-master3节点创建文件夹
[root@k8s-master2 ~]# mkdir -p /etc/kubernetes/pki/etcd
[root@k8s-master3 ~]# mkdir -p /etc/kubernetes/pki/etcd
2、在k8s-master1节点执行
从k8s-master1复制秘钥和相关文件到k8s-master2和k8s-master3
[root@k8s-master1 ~]# scp /etc/kubernetes/admin.conf root@192.168.100.132:/etc/kubernetes
[root@k8s-master1 ~]# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@192.168.100.132:/etc/kubernetes/pki
[root@k8s-master1 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@192.168.100.132:/etc/kubernetes/pki/etcd
[root@k8s-master1 ~]# scp /etc/kubernetes/admin.conf root@192.168.100.133:/etc/kubernetes[root@k8s-master1 ~]# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@192.168.100.133:/etc/kubernetes/pki
[root@k8s-master1 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@192.168.100.133:/etc/kubernetes/pki/etcd
![](https://file.jishuzhan.net/article/1692538624114954242/1faff18643344bba96f30b0be8d96c3d.png)
3、将其他master节点加入集群
****注意:****kubeadm init生成的token有效期只有1天,生成不过期token
[root@k8s-master1 manifests]# kubeadm token create --ttl 0 --print-join-command
[root@k8s-master1 manifests]# kubeadm token list
![](https://file.jishuzhan.net/article/1692538624114954242/6ecac97bc7784d03925490ccd46afb9b.png)
k8s-master2和k8s-master3都需要加入
[root@k8s-master2 ~]# kubeadm join master.k8s.io:6443 --token pj2haa.zf72tyum7uiyeamx --discovery-token-ca-cert-hash sha256:aaec80f6efa10581c329034bef7e2c2f2f1cb2ef4228f8ddcfcbbb44df55aae3 --control-plane
![](https://file.jishuzhan.net/article/1692538624114954242/ece456c6c6e14039aaf9c152c01cb6c9.png)
[root@k8s-master2 ~]# mkdir -p $HOME/.kube
[root@k8s-master2 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master2 ~]# sudo chown (id -u):(id -g) $HOME/.kube/config
![](https://file.jishuzhan.net/article/1692538624114954242/3279044e6c52429198d9876628cf5640.png)
[root@k8s-master2 ~]# docker load < flannel_v0.12.0-amd64.tar
[root@k8s-master2 ~]# tar xf cni-plugins-linux-amd64-v0.8.6.tgz
[root@k8s-master2 ~]# cp flannel /opt/cni/bin/
![](https://file.jishuzhan.net/article/1692538624114954242/c6363e1bf438472f9628d8bd37bc3dad.png)
查看节点信息
[root@k8s-master1 ~]# kubectl get nodes
![](https://file.jishuzhan.net/article/1692538624114954242/2d208c5f64394e0e9f2b09aec067f0be.png)
[root@k8s-master1 manifests]# kubectl get pods --all-namespaces
![](https://file.jishuzhan.net/article/1692538624114954242/cdcbc7745a4e4b24b62b31ce7f23efed.png)
(八)加入Kubernetes Node
直接在node节点服务器上执行k8s-master1初始化成功后的消息即可:
[root@k8s-node1 ~]# kubeadm join master.k8s.io:6443 --token o3j9wj.58io4u28r6q8o9lj --discovery-token-ca-cert-hash sha256:6ad29ff932b12680844e140938eaeaaca120d6020c273b6b56d69d256fbc44b0
![](https://file.jishuzhan.net/article/1692538624114954242/3a30c9b0723f4c719396a48f172be34a.png)
[root@k8s-node1 ~]# docker load < flannel_v0.12.0-amd64.tar
![](https://file.jishuzhan.net/article/1692538624114954242/c89b4e5a2667435e9afc78d9bd71fc78.png)
查看节点信息
[root@k8s-master1 ~]# kubectl get nodes
![](https://file.jishuzhan.net/article/1692538624114954242/926ae100d6894fbabbf737804e6c48e2.png)
查看pod信息
[root@k8s-master1 ~]# kubectl get pods -n kube-system
![](https://file.jishuzhan.net/article/1692538624114954242/273e8fe96cc74aeaa87e64976261284a.png)
(九)测试Kubernetes集群
1、所有node主机下载测试镜像
[root@k8s-node1 ~]# docker pull nginx:1.19.0
![](https://file.jishuzhan.net/article/1692538624114954242/86365b48ff1b47eba1e249a2f22dfe92.png)
2、在Kubernetes集群中创建一个pod,验证是否正常运行。
[root@k8s-master1 ~]# mkdir demo
[root@k8s-master1 ~]# cd demo
[root@k8s-master1 demo]# vim nginx-deployment.yaml
![](https://file.jishuzhan.net/article/1692538624114954242/483272275435436594eb2910ef22737e.png)
3、创建完 Deployment 的资源清单之后,使用 create 执行资源清单来创建容器。
通过 get pods 可以查看到 Pod 容器资源已经自动创建完成。
[root@k8s-master1 demo]# kubectl create -f nginx-deployment.yaml
[root@k8s-master1 demo]# kubectl get pods
![](https://file.jishuzhan.net/article/1692538624114954242/d0b52d0532ff48a0b1e31f55a6d20da3.png)
[root@k8s-master1 ~]# kubectl get pods -o wide
![](https://file.jishuzhan.net/article/1692538624114954242/32a1a5f3e6e24d188da53e9cde0c4fbb.png)
4、创建Service资源清单
在创建的 nginx-service 资源清单中,定义名称为 nginx-service 的 Service、标签选择器为 app: nginx、type 为 NodePort 指明外部流量可以访问内部容器。在 ports 中定义暴露的端口库号列表,对外暴露访问的端口是 80,容器内部的端口也是 80。
[root@k8s-master1 demo]# vim nginx-service.yaml
![](https://file.jishuzhan.net/article/1692538624114954242/8093d82db25c4f6d8c815bd6488903f5.png)
[root@k8s-master1 demo]# kubectl create -f nginx-service.yaml
[root@k8s-master1 demo]# kubectl get svc
![](https://file.jishuzhan.net/article/1692538624114954242/74fb4055490347eea28a180b5a5f19be.png)
5、访问测试
通过浏览器访问nginx:http://master.k8s.io:31350 域名或者VIP地址
[root@k8s-master1 demo]# elinks --dump http://master.k8s.io:31350
![](https://file.jishuzhan.net/article/1692538624114954242/f9cbb895acb74ed8a0da38582e197937.png)
浏览器测试:192.168.100.154:31350
![](https://file.jishuzhan.net/article/1692538624114954242/dfb592c47f5d4af48e6e2f9d24bd9e6f.png)
四、拓展试验
(一)宕机master1,验证服务
挂起k8s-master1节点,刷新页面还是能访问nginx,说明高可用集群部署成功。
![](https://file.jishuzhan.net/article/1692538624114954242/655738758d694d33895c7f26b7feb39a.png)
![](https://file.jishuzhan.net/article/1692538624114954242/8ff6a631aa134ee9a8482b6cb800d8fb.png)
检查会发现VIP已经转移到k8s-master2节点上
[root@k8s-master2 ~]# ip a s ens33
![](https://file.jishuzhan.net/article/1692538624114954242/8edbe2ff02284a0ab200176a841963c0.png)
验证操作master
[root@k8s-master2 ]#kubectl get nodes
![](https://file.jishuzhan.net/article/1692538624114954242/dd59b3aac5d74f1fbd6979bc283dade7.png)
至此Kubernetes企业级高可用环境完美实现。
(二)宕机master1/2,验证服务
以此类推,停掉master2的服务,vip跳转至master3,服务仍保持
![](https://file.jishuzhan.net/article/1692538624114954242/1cee0cfa673d4c559d91649a93573048.png)
![](https://file.jishuzhan.net/article/1692538624114954242/83632fbef87d44e582efaa17cfe8fddf.png)
检查会发现VIP已经转移到k8s-master3节点上
[root@k8s-master3 ~]# ip a s ens33
![](https://file.jishuzhan.net/article/1692538624114954242/a6141ecc00c846459c7eb2d1c180ed7f.png)
验证操作master
[root@k8s-master3 ]#kubectl get nodes
![](https://file.jishuzhan.net/article/1692538624114954242/a4d8fd57e1c6448cb4537b982b312582.png)
(三)两台宕机主机恢复,验证服务
Master1:
![](https://file.jishuzhan.net/article/1692538624114954242/558e4dd96b3b4bdfb9114ca75f39cb13.png)
Master2:
![](https://file.jishuzhan.net/article/1692538624114954242/b04302c8698d480bb5f242d60e41a38c.png)
Master3:
![](https://file.jishuzhan.net/article/1692538624114954242/38d9cc1c846d4d6a89a6221985e746c5.png)
访问服务:
![](https://file.jishuzhan.net/article/1692538624114954242/5695c01cb912440fa8b601c9802ac225.png)
五、实验总结
1、集群中只要有一个master节点正常运行就可以正常对外提供业务服务。
2、如果需要在master节点使用kubectl相关的命令,必须保证至少有2个master节点正常运行才可以使用,不然会有 Unable to connect to the server: net/http: TLS handshake timeout 这样的错误。
3、当一台可以查看nodes节点的master宕机之后,其余两台随机一台获取vip,然后可以观察nodes节点,但是当超过两台master宕机之后,集群需重建才可以观察nodes节点,但服务未停止;当两台宕机主机回复之后,服务停止,node节点不可观察,集群停止,需重建!
4、Node节点故障时pod自动转移:当pod所在的Node节点宕机后,根据 controller-manager的--pod-eviction-timeout 配置,默认是5分钟,5分钟后k8s会把pod状态设置为unkown, 然后在其它节点启动pod。当故障节点恢复后,k8s会删除故障节点上面的unkown pod。如果你想立即强制迁移,可以用 kubectl drain nodename
5、为了保证集群的高可用性,建议master节点和node节点至少分别部署3台及以上,且master节点应该部署基数个实例(3、5、7、9)。