基于containerd容器运行时,kubeadmin部署k8s 1.28集群

一.主机准备

1.1主机配置与操作系统说明

centos7u9

1.2主机硬件配置说明

序号	主机名	ip地址	CPU	内存	硬盘
1	k8s-master1	192.168.1.200	2C	2G	100G
2	k8s-worker1	192.168.1.201	2C	2G	100G
3	k8s-worker2	192.168.1.202	2C	2G	100G

1.3主机配置

1.3.1主机名配置

bash 复制代码

hostnamectl set-hostname k8s-master01

1.3.2主机IP地址配置

vi /etc/sysconfig/network-scripts/ifcfg-ens33

bash 复制代码

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="f2c1c981-22ee-4e43-8209-2d6bf20ac6ca"
DEVICE="ens33"
ONBOOT="yes"
IPV6_PRIVACY="no"
IPADDR="192.168.1.200"
PREFIX="24"
GATEWAY="192.168.1.54"
DNS1=119.29.29.29

bash 复制代码

systemctl restart network

1.3.3主机名与IP地址解析(hosts)

vi /etc/hosts

bash 复制代码

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.200 master1
192.168.1.201 worker1
192.168.1.202 worker2

1.3.4防火墙配置

关闭防火墙firewalld

bash 复制代码

systemctl disable firewalld
systemctl stop firewalld
systemctl status firewalld

1.3.5SELINUX配置

修改后需要重启

sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

ba 复制代码

sestatus

1.3.6时间同步配置

bash 复制代码

yum -y install ntpdate

bash 复制代码

crontab -e

0 */1 * * * ntpdate time1.aliyun.com

crontab -l

bash 复制代码

ntpdate time1.aliyun.com

1.3.7升级操作系统内核

bash 复制代码

不要一下子升太高,也不要使用很低的内核版本
导入elrepo gpg key [软件安装验证密钥]
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

bash 复制代码

安装elrepo YUM源仓库
yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

bash 复制代码

安装kernel-lt版本,ml为长期稳定版本,lt为长期维护版本
yum --enablerepo="elrepo-kernel" -y install kernel-lt.x86_64

bash 复制代码

设置更新的内核为默认引导
grub2-set-default 0

bash 复制代码

重新生成grub2引导文件
grub2-mkconfig -o /boot/grub2/grub.cfg

更新后,需要重启,使升级的内核生效

bash 复制代码

验证内核是否为更新的版本
uname -r

1.3.8配置内核转发及网桥过滤

bash 复制代码

添加网桥过滤及内核转发配置文件
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

加载br_netfilter模块
modprobe br_netfilter 
查看是否加载
lsmod |grep br_netfilter

vim /etc/sysconfig/modules/br_netfilter.modules 设置环境变量

bash 复制代码

#!/bin/bash
modprobe br_netfilter
chmod 755 /etc/sysconfig/modules/br_netfilter.modules

1.3.9安装ipset及ipvsadm

bash 复制代码

安装ipset及ipvsadm
yum -y install ipset ipvsadm

bash 复制代码

cat > /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

bash 复制代码

授权、运行、检查是否加载
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

1.3.10关闭SWAP分区

vim /etc/fstab

bash 复制代码

临时关闭
swapoff -a

bash 复制代码

永久关闭SWAP分区
#
# /etc/fstab
# Created by anaconda on Sun Nov 12 23:08:40 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=90946160-91fa-4b5c-afbb-97e6e82341ea /boot                   xfs     defaults        0 0
/dev/mapper/centos-home /home                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0  #注释掉

二.容器运行时Containerd准备

Containerd准备

2.1.1Containerd部署文件获取

bash 复制代码

下载cri-containerd
wget https://github.com/containerd/containerd/releases/download/v1.7.8/cri-containerd-1.7.8-linux-amd64.tar.gz

解压到根目录
tar xf -cri-containerd-1.7.8-linux-amd64.tar.gz -C /

2.1.2Containerd配置文件生成并修改

bash 复制代码

mkdir /etc/containerd

containerd config default > /etc/containerd/config.toml

vim /etc/containerd/config.toml

sandibox_image = "registry.k8s.io/pause:3.9" #k8s 1.28默认使用3.9

#2.1.3Containerd启动及开机自启动

bash 复制代码

systemctl enable --now containerd #--now是启动选项

验证版本
containerd --version

crictl images

runc替换

2.2.1libseccomp准备

bash 复制代码

wget https://github.com/opencontainers/runc/releases/download/v1.1.10/libseccomp-2.5.4.tar.gz

tar xf libseccomp-2.5.4.tar.gz 

cd libseccomp-2.5.4

yum -y install gperf  

./configure      

make &&  make install       

find / -name "libseccomp.so"

2.2.2runc替换

bash 复制代码

wget https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64

which runc      

 rm -rf 'which runc'

mv runc.amd64 /usr/local/sbin/runc

chmod +x runc

runc

runc --version

三.kubeadm部署k8s集群

k8sYUM源准备

官方源

bash 复制代码

# 此操作会覆盖 /etc/yum.repos.d/kubernetes.repo 中现存的所有配置
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

3.2k8s集群软件安装

3.2.1安装及设置自启动

bash 复制代码

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

3.2.1配置kubelet

vim /etc/sysconfig/kubelet

KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"

3.3k8s集群初始化

master节点操作

bash 复制代码

kubeadm init --pod-network-cidr=10.244.0.0/16 -apiserver-advertise-address=192.168.1.200

3.3.1kubectl api配置

bash 复制代码

mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
   
kubectl get nodes

3.3.2工作节点加入集群

不一定相同，请复制实际中输出的内容

在工作节点执行命令

bash 复制代码

kubeadm join 192.168.1.200:6443 --token 5jz1bl.dt6etzj1m49646vd \
        --discovery-token-ca-cert-hash sha256:5e15bad72866d0b00b3f287f21d7140831ee97f665dd5d63226f50bdca0894fa

结果

bash 复制代码

 kubectl get nodes
NAME          STATUS     ROLES           AGE    VERSION
k8s-master1   NotReady   control-plane   110s   v1.28.3
k8s-worker1   NotReady   <none>          7s     v1.28.3
k8s-worker2   NotReady   <none>          4s     v1.28.3

bash 复制代码

kubectl get pods -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-5dd5756b68-9g67g              0/1     Pending   0          2m30s
coredns-5dd5756b68-nv68x              0/1     Pending   0          2m30s
etcd-k8s-master1                      1/1     Running   0          2m44s
kube-apiserver-k8s-master1            1/1     Running   0          2m44s
kube-controller-manager-k8s-master1   1/1     Running   0          2m44s
kube-proxy-77m25                      1/1     Running   0          2m30s
kube-proxy-hlk5h                      1/1     Running   0          65s
kube-proxy-zjg9r                      1/1     Running   0          62s
kube-scheduler-k8s-master1            1/1     Running   0          2m44s

四.calico网络插件

4.1安装calico operator

bash 复制代码

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/tigera-operator.yaml

bash 复制代码

kubectl get ns  #查看命名空间
#NAME              STATUS   AGE
#default           Active   10m
#kube-node-lease   Active   10m
#kube-public       Active   10m
#kube-system       Active   10m
#！！tigera-operator   Active   26s

一定要等待tigera-operator这个pod状态为运行（作用是对calico进行部署）

bash 复制代码

kubectl get pods -n tigera-operator
NAME                               READY   STATUS    RESTARTS   AGE
tigera-operator-597bf4ddf6-nstgk   1/1     Running   0          45s

4.2下载并编辑calico配置文件

bash 复制代码

wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/custom-resources.yaml

vim custom-resources.yaml

	#cidr: 10.244.0.0/16
	
kubectl create -f custom-resources.yaml

kubectl get ns

kubectl get pods -n calico-system

watch kubectl get pods -n calico-system#监视pods直至全部ready

检查coredns是否已经获得ip地址

bash 复制代码

 kubectl get pods -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-5dd5756b68-9g67g              1/1     Running   0          23m
coredns-5dd5756b68-nv68x              1/1     Running   0          23m
etcd-k8s-master1                      1/1     Running   0          23m
kube-apiserver-k8s-master1            1/1     Running   0          23m
kube-controller-manager-k8s-master1   1/1     Running   0          23m
kube-proxy-77m25                      1/1     Running   0          23m
kube-proxy-hlk5h                      1/1     Running   0          21m
kube-proxy-zjg9r                      1/1     Running   0          21m
kube-scheduler-k8s-master1            1/1     Running   0          23m

bash 复制代码

kubectl get pods -n kube-system -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP               NODE          NOMINATED NODE   READINESS GATES
coredns-5dd5756b68-9g67g              1/1     Running   0          24m   10.244.159.131   k8s-master1   <none>           <none>
coredns-5dd5756b68-nv68x              1/1     Running   0          24m   10.244.159.129   k8s-master1   <none>           <none>
etcd-k8s-master1                      1/1     Running   0          24m   192.168.1.200    k8s-master1   <none>           <none>
kube-apiserver-k8s-master1            1/1     Running   0          24m   192.168.1.200    k8s-master1   <none>           <none>
kube-controller-manager-k8s-master1   1/1     Running   0          24m   192.168.1.200    k8s-master1   <none>           <none>
kube-proxy-77m25                      1/1     Running   0          24m   192.168.1.200    k8s-master1   <none>           <none>
kube-proxy-hlk5h                      1/1     Running   0          22m   192.168.1.201    k8s-worker1   <none>           <none>
kube-proxy-zjg9r                      1/1     Running   0          22m   192.168.1.202    k8s-worker2   <none>           <none>
kube-scheduler-k8s-master1            1/1     Running   0          24m   192.168.1.200    k8s-master1   <none>           <none>

检查域名解析服务

bash 复制代码

kubectl get svc -n kube-system

 yum install bind-utils#安装dig命令

dig -t a www.baidu.com @10.96.0.10
	;; ANSWER SECTION:
	www.baidu.com.          5       IN      A       198.18.0.169
    解析成功

4.3检查集群是否可用

添加nginx-pod.yml文件创建pod

bash 复制代码

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx:alpine
    ports:
    - containerPort: 80

使用命令 kubectl apply 创建 pod：

bash 复制代码

kubectl apply -f nginx-pod.yml

kubectl get pods nginx -o wide

watch kubectl get pods nginx -o wide
	NAME    READY   STATUS    RESTARTS   AGE    IP              NODE          NOMINATED NODE   READINESS GATES
	nginx   1/1     Running   0          3m3s   10.244.194.67   k8s-worker1   <none>           <none>

ready后使用curl工具测试

bash 复制代码

curl 10.244.194.67
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

参考部署nginx

https://leehao.me/k8s-部署-nginx-入门/

4.4查看集群健康状态

bash 复制代码

kubectl cluster-info
Kubernetes control plane is running at https://192.168.1.200:6443
CoreDNS is running at https://192.168.1.200:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

bash 复制代码

mkdir /.kube && cd /.kube

scp master1:/root/.kube/config .

有了这个文件就可以在目标节点使用kubectl连接到apiserver,即可以使用kubectl命令管理集群