（k8s）kubernetes集群基于Containerd部署

资源列表
基础环境
一、基础环境准备
- 1.1、关闭Swap分区
- 1.2、添加hosts解析
- 1.3、桥接的IPv4流量传递给iptables的链
二、准备Containerd容器运行时
- 2.1、安装Containerd
- 2.2、配置Containerd
- 2.3、启动Containerd
三、部署Kubernetes集群
- 3.1、安装Kubeadm工具
- 3.2、配置crictl工具
- 3.3、配置Containerd镜像加速器
- 3.4、测试crictl工具是否可用
- 3.5、初始化Master节点
- 3.6、Node节点加入集群
四、部署网络插件（CNI）
- 4.1、拉取必要镜像
- 4.2、部署网络插件
- 4.2、查看节点状态
- 4.3、查看组件状态
- 4.4、查看所有Pod状态
五、Containerd基本操作
- 4.1、Crictl镜像管理
- 4.2、Crictl容器管理
- 4.3、Crictl Pod管理

资源列表

操作系统	配置	主机名	IP
CentOS 7.9	2C4G	master	192.168.93.101
CentOS 7.9	2C4G	node1	192.168.93.102
CentOS 7.9	2C4G	node2	192.168.93.103

基础环境

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
关闭内核安全机制

setenforce 0
sed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config
修改主机名

hostnamectl set-hostname master
hostnamectl set-hostname node1
hostnamectl set-hostname node2

一、基础环境准备

三台主机都要操作，以master节点为例进行演示

1.1、关闭Swap分区

# 临时关闭
[root@master ~]# swapoff -a
# 永久关闭
[root@master ~]# sed -i 's/.*swap.*/#&/g' /etc/fstab

1.2、添加hosts解析

[root@master ~]# cat >> /etc/hosts << EOF
192.168.93.101 master
192.168.93.102 node1
192.168.93.103 node2
EOF

1.3、桥接的IPv4流量传递给iptables的链

# 加载 overlay 内核模块
[root@master ~]# modprobe overlay
# 加载 br_netfilter 模块
[root@master ~]# modprobe br_netfilter
[root@master ~]# cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
[root@master ~]# sysctl --system

二、准备Containerd容器运行时

三台主机都要操作，以master节点为例进行演示

2.1、安装Containerd

# 添加 docker 源，containerd也在docker源内的
[root@master ~]# cat <<EOF | sudo tee /etc/yum.repos.d/docker-ce.repo
[docker]
name=docker-ce
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF


# 快速建立 yum 缓存
[root@master ~]# yum makecache fast


# 安装containerd
# 列出所有containerd版本
[root@master ~]# yum list containerd.io --showduplicates
[root@master ~]# yum -y install containerd.io-1.6.6-3.1.el7.x86_64

2.2、配置Containerd

# 生成配置文件
[root@master ~]# mkdir -p /etc/containerd
[root@master ~]# containerd config default | sudo tee /etc/containerd/config.toml 


# 修改/etc/containerd/config.toml文件中sandbox_image的值
[root@master ~]# grep 'sandbox_image' /etc/containerd/config.toml 
# 在原有的基础上进行修改，
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

2.3、启动Containerd

[root@master ~]# systemctl enable containerd
[root@master ~]# systemctl start containerd

三、部署Kubernetes集群

3.1、安装Kubeadm工具

所有节点都要操作

[root@master ~]# cat << EOF >> /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

快速建立yum缓存

[root@master ~]# yum makecache fast

安装 kubectl：命令行管理工具、kubeadm：安装K8S集群工具、kubelet管理容器工具

[root@master ~]# yum install -y kubelet-1.27.0 kubeadm-1.27.0 kubectl-1.27.0

切记这个时候不要启动，只需要设置为开机自启

[root@master ~]# systemctl enable kubelet.service

3.2、配置crictl工具

crictl是CRI兼容的容器运行时命令行接口。你可以使用它来检查和调试Kubernetes节点上的容器运行时和应用程序。crictl和它的源代码在 cri-tools 代码库
更好 Containerd后，以上我们常用的docker命令也不再使用了，取而代之的分别是 crictl 和 ctr 两个命令行客户端
crictl是遵循CRI接口规范的一个命令行工具，通常用它来检查和管理kubelet节点上的容器运行时和镜像
ctr是containerd的一个客户端工具

所有节点都要操作

[root@master ~]# cat << EOF >> /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: false
EOF

3.3、配置Containerd镜像加速器

所有节点都要操作

在/etc/containerd/config.toml找到[plugins."io.containerd.grpc.v1.cri".registry.mirrors] ，默认在153行左右，在此行下面添加两行配置，阿里云的镜像加速器

[root@master ~]# vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://i9h06ghu.mirror.aliyuncs.com"]

保存退出，刷新服务

[root@master ~]# systemctl restart containerd

3.4、测试crictl工具是否可用

# 以拉取一个Nginx镜像进行演示是否可用
[root@master ~]# crictl pull nginx:latest
Image is up to date for sha256:605c77e624ddb75e6110f997c58876baa13f8754486b461117934b24a9dc3a85


# 查看镜像
[root@master ~]# crictl images
IMAGE                     TAG                 IMAGE ID            SIZE
docker.io/library/nginx   latest              605c77e624ddb       56.7MB

3.5、初始化Master节点

在master节点上操作

生成配置文件

[root@master ~]# kubeadm config print init-defaults > kubeadm-init.yaml

修改kubeadm-init.yaml文件的advertiseAddress、name、imageRepository，添加Pod网络podSubnet

[root@master ~]# vim kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
    token: abcdef.0123456789abcdef
    ttl: 24h0m0s
    usages:
  - signing
  - authentication
    kind: InitConfiguration
    localAPIEndpoint:
    advertiseAddress: 192.168.93.101
    bindPort: 6443
    nodeRegistration:
    criSocket: unix:///var/run/containerd/containerd.sock
    imagePullPolicy: IfNotPresent
    name: master
    taints: null
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.27.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
podSubnet: 10.244.0.0/16
scheduler: {}

初始化集群

[root@master ~]# kubeadm init --config=kubeadm-init.yaml
[init] Using Kubernetes version: v1.27.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0629 11:07:42.117214 8814 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
W0629 11:08:11.311847 8814 checks.go:835] detected that the sandbox image "registry.aliyuncs.com/google_containers/pause:3.9" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.93.101]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.93.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.93.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
W0629 11:08:41.485522 8814 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.001513 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
#####################################################################
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$ (id -g) $HOME/.kube/config
#####################################################################
Alternatively, if you are the root user, you can run:
```
export KUBECONFIG=/etc/kubernetes/admin.conf
```
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:
#####################################################################
kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef
--discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d
#####################################################################

初始化成功以后要根据提示执行以下3个命令，才可以操作集群

[root@master ~]# mkdir -p $HOME/.kube
[root@master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master ~]# sudo chown $(id -u):$ (id -g) $HOME/.kube/config

3.6、Node节点加入集群

所有Node节点都要操作

node1节点

[root@node1 ~]# kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef \

--discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
- Certificate signing request was sent to apiserver and a response was received.
- The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

node2节点

[root@node2 ~]# kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef \

--discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
- Certificate signing request was sent to apiserver and a response was received.
- The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

如果加入集群的命令找不到了可以在master节点生成一个

[root@master ~]# kubeadm token create --print-join-command
kubeadm join 192.168.93.101:6443 --token 2d6hku.7esvsdqjfrvgkewg --discovery-token-ca-cert-hash sha256:aa55ff35b12c120ff5a12098622122c77eff343d033a54d069d3e55364b4184d

四、部署网络插件（CNI）

4.1、拉取必要镜像

所有节点都需要拉取以下两个镜像
在部署网络插件的过程中，实测是可能会有两个镜像拉取不来下，所以手动拉取一下，不过要使用魔法，拉取不到了评论或私信（开源免费）

以下是拉取镜像的命令，所有节点都需要有这两个镜像

[root@master ~]# crictl pull docker.io/flannel/flannel-cni-plugin:v1.1.2
[root@master ~]# crictl pull docker.io/flannel/flannel:v0.21.5
如果有镜像的话，使用以下命令把镜像导入即可，我的镜像名字是自己打的标签

[root@master ~]# ctr -n k8s.io images import flannel.tar
unpacking docker.io/flannel/flannel:v0.21.5 (sha256:6f2d991efb758c5530e7de90761dfb29637b7604a807d431312e20189e09f9e6)...done
[root@master ~]# ctr -n k8s.io images import flannel-cni-plugin-v1.1.2.tar
unpacking docker.io/flannel/flannel-cni-plugin:v1.1.2 (sha256:539d3bf046c8581557f0747dbad9d3b78a4de112d3c0bf9d291651593060fc9f)...done

查看镜像

[root@master ~]# crictl images | grep flannel
docker.io/flannel/flannel-cni-plugin v1.1.2 7a2dcab94698c 8.25MB
docker.io/flannel/flannel v0.21.5 a6c0cb5dbd211 69.9MB
4.2、部署网络插件

4.2、部署网络插件

master节点操作即可
没有网络插件的私信或评论（免费开源）

[root@master ~]# kubectl apply -f kube-flannel.yaml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

4.3、查看节点状态

[root@master ~]# kubectl get node
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   15m   v1.27.0
node1    Ready    <none>          12m   v1.27.0
node2    Ready    <none>          12m   v1.27.0

4.4、查看组件状态

[root@master ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}

4.5、查看所有Pod状态

[root@master ~]# kubectl get pod -A
NAMESPACE      NAME                             READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-bkp7z            1/1     Running   0          2m5s
kube-flannel   kube-flannel-ds-q2565            1/1     Running   0          2m5s
kube-flannel   kube-flannel-ds-vz72p            1/1     Running   0          2m5s
kube-system    coredns-65dcc469f7-6x74z         1/1     Running   0          16m
kube-system    coredns-65dcc469f7-j8266         1/1     Running   0          16m
kube-system    etcd-master                      1/1     Running   0          16m
kube-system    kube-apiserver-master            1/1     Running   0          16m
kube-system    kube-controller-manager-master   1/1     Running   0          16m
kube-system    kube-proxy-2npvg                 1/1     Running   0          16m
kube-system    kube-proxy-8kqps                 1/1     Running   0          13m
kube-system    kube-proxy-zbc4h                 1/1     Running   0          13m
kube-system    kube-scheduler-master            1/1     Running   0          16m

五、Containerd基本操作

5.1、Crictl镜像管理

# 下载镜像
[root@master ~]# crictl pull nginx:latest

# 查看所有镜像
[root@master ~]# crictl images

# 查看镜像详细信息
[root@master ~]# crictl inspecti nginx:latest

# 删除镜像
[root@master ~]# crictl rmi nginx:latest

5.2、Crictl容器管理

# 列出运行中的容器，查看所有容器加-a选项
[root@master ~]# crictl ps

# 查看容器的详细信息
[root@master ~]# crictl inspect 07376dc402441

# 启动容器
[root@master ~]# crictl start 07376dc402441

# 停止容器
[root@master ~]# crictl stop 07376dc402441

# 删除容器
[root@master ~]# crictl rm 07376dc402441

# 打印日志
crictl logs <container-id 或 container-name>

# 进入容器执行命令
crictl exec -it <container-id 或 container-name> sh

5.3、Crictl Pod管理

# 打印所有 Pod
[root@node2 ~]# crictl pods
# 根据pod名称打印pod信息
[root@master ~]# crictl pods --name kube-proxy-trnjn

# 根据标签打印 Pod，crictl pods --label key=value
[root@master ~]# crictl pods --label k8s-app=kube-proxy

（k8s）kubernetes集群基于Containerd部署

资源列表

基础环境

一、基础环境准备

1.1、关闭Swap分区

二、准备Containerd容器运行时

2.1、安装Containerd

2.2、配置Containerd

2.3、启动Containerd

三、部署Kubernetes集群

3.1、安装Kubeadm工具

快速建立yum缓存

安装 kubectl：命令行管理工具、kubeadm：安装K8S集群工具、kubelet管理容器工具

切记这个时候不要启动，只需要设置为开机自启

3.2、配置crictl工具

所有节点都要操作

3.3、配置Containerd镜像加速器

在/etc/containerd/config.toml找到[plugins."io.containerd.grpc.v1.cri".registry.mirrors] ，默认在153行左右，在此行下面添加两行配置，阿里云的镜像加速器

保存退出，刷新服务

3.4、测试crictl工具是否可用

3.5、初始化Master节点

生成配置文件

修改kubeadm-init.yaml文件的advertiseAddress、name、imageRepository，添加Pod网络podSubnet

初始化集群

初始化成功以后要根据提示执行以下3个命令，才可以操作集群

3.6、Node节点加入集群

node1节点

node2节点

如果加入集群的命令找不到了可以在master节点生成一个

四、部署网络插件（CNI）

4.1、拉取必要镜像

以下是拉取镜像的命令，所有节点都需要有这两个镜像

查看镜像

4.2、部署网络插件

4.3、查看节点状态

4.4、查看组件状态

4.5、查看所有Pod状态

五、Containerd基本操作

5.1、Crictl镜像管理

5.2、Crictl容器管理

5.3、Crictl Pod管理