目录
- 一、实验环境规划
- 二、部署流程总览
- [三、第一步：构建 harbor 镜像仓库](#三、第一步：构建 harbor 镜像仓库)
- 四、第二步：准备集群主机
- 五、第三步：对接容器运行时（cri-dockerd）
- [六、第四步：初始化控制面（kubeadm init）](#六、第四步：初始化控制面（kubeadm init）)
- 七、第五步：工作节点加入
- 八、第六步：部署网络插件（flannel）
- 九、部署完成后的验证
- 十、自动化部署：Ansible
Kubernetes（简称 k8s）是容器编排平台，用"声明期望状态"的方式管理容器：告诉它要几个副本、用什么镜像，剩下的由它自己收敛。
一、实验环境规划
本次实验一共四台虚拟机：
| 主机名 | IP | 配置 | 角色 |
|---|---|---|---|
| harbor.timinglee.org | 172.25.254.254 |
cpu1、mem 1G | harbor 仓库 |
| master | 172.25.254.100 |
cpu4、mem>4G | 控制面节点 |
| node1 | 172.25.254.10 |
cpu2、mem 2G | 工作节点 |
| node2 | 172.25.254.20 |
cpu2、mem 2G | 工作节点 |
二、部署流程总览
整个部署分六步，前两步是环境准备，中间三步是集群本体，最后是验证：
#mermaid-svg-OWSYJJou4LQoxRSW{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-OWSYJJou4LQoxRSW .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-OWSYJJou4LQoxRSW .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-OWSYJJou4LQoxRSW .error-icon{fill:#552222;}#mermaid-svg-OWSYJJou4LQoxRSW .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-OWSYJJou4LQoxRSW .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-OWSYJJou4LQoxRSW .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-OWSYJJou4LQoxRSW .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-OWSYJJou4LQoxRSW .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-OWSYJJou4LQoxRSW .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-OWSYJJou4LQoxRSW .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-OWSYJJou4LQoxRSW .marker{fill:#333333;stroke:#333333;}#mermaid-svg-OWSYJJou4LQoxRSW .marker.cross{stroke:#333333;}#mermaid-svg-OWSYJJou4LQoxRSW svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-OWSYJJou4LQoxRSW p{margin:0;}#mermaid-svg-OWSYJJou4LQoxRSW .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-OWSYJJou4LQoxRSW .cluster-label text{fill:#333;}#mermaid-svg-OWSYJJou4LQoxRSW .cluster-label span{color:#333;}#mermaid-svg-OWSYJJou4LQoxRSW .cluster-label span p{background-color:transparent;}#mermaid-svg-OWSYJJou4LQoxRSW .label text,#mermaid-svg-OWSYJJou4LQoxRSW span{fill:#333;color:#333;}#mermaid-svg-OWSYJJou4LQoxRSW .node rect,#mermaid-svg-OWSYJJou4LQoxRSW .node circle,#mermaid-svg-OWSYJJou4LQoxRSW .node ellipse,#mermaid-svg-OWSYJJou4LQoxRSW .node polygon,#mermaid-svg-OWSYJJou4LQoxRSW .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-OWSYJJou4LQoxRSW .rough-node .label text,#mermaid-svg-OWSYJJou4LQoxRSW .node .label text,#mermaid-svg-OWSYJJou4LQoxRSW .image-shape .label,#mermaid-svg-OWSYJJou4LQoxRSW .icon-shape .label{text-anchor:middle;}#mermaid-svg-OWSYJJou4LQoxRSW .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-OWSYJJou4LQoxRSW .rough-node .label,#mermaid-svg-OWSYJJou4LQoxRSW .node .label,#mermaid-svg-OWSYJJou4LQoxRSW .image-shape .label,#mermaid-svg-OWSYJJou4LQoxRSW .icon-shape .label{text-align:center;}#mermaid-svg-OWSYJJou4LQoxRSW .node.clickable{cursor:pointer;}#mermaid-svg-OWSYJJou4LQoxRSW .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-OWSYJJou4LQoxRSW .arrowheadPath{fill:#333333;}#mermaid-svg-OWSYJJou4LQoxRSW .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-OWSYJJou4LQoxRSW .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-OWSYJJou4LQoxRSW .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OWSYJJou4LQoxRSW .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-OWSYJJou4LQoxRSW .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OWSYJJou4LQoxRSW .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-OWSYJJou4LQoxRSW .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-OWSYJJou4LQoxRSW .cluster text{fill:#333;}#mermaid-svg-OWSYJJou4LQoxRSW .cluster span{color:#333;}#mermaid-svg-OWSYJJou4LQoxRSW div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-OWSYJJou4LQoxRSW .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-OWSYJJou4LQoxRSW rect.text{fill:none;stroke-width:0;}#mermaid-svg-OWSYJJou4LQoxRSW .icon-shape,#mermaid-svg-OWSYJJou4LQoxRSW .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OWSYJJou4LQoxRSW .icon-shape p,#mermaid-svg-OWSYJJou4LQoxRSW .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-OWSYJJou4LQoxRSW .icon-shape .label rect,#mermaid-svg-OWSYJJou4LQoxRSW .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OWSYJJou4LQoxRSW .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-OWSYJJou4LQoxRSW .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-OWSYJJou4LQoxRSW :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} ① 部署 harbor 镜像仓库证书 + 离线安装
② 准备集群主机关 swap、装 docker、配仓库、写 hosts
③ 安装 cri-dockerd容器运行时对接 CRI
④ kubeadm init初始化控制面
⑤ kubeadm joinnode1 / node2 入群
⑥ 部署 flannel节点从 NotReady 变 Ready
验证：kubectl get nodes三台全部 Ready
三、第一步：构建 harbor 镜像仓库
为什么需要私有仓库
k8s 组件镜像都放在 registry.k8s.io，这个地址在国内基本拉不动。更麻烦的是，node 节点创建 Pod 时要拉业务镜像，如果每次都走公网，既慢又不稳定。解决办法是在内网搭一个 harbor，所有镜像先推上去，节点统一从内网拉。
harbor 本身是跑在 docker 里的一堆容器（nginx、core、registry、jobservice），装 harbor 之前先要把 docker 装好。
装 docker：先解决离线安装的问题
离线环境装 docker-ce 没有 yum 源，思路是：找一台能上网的机器下载好 rpm 包，用 httpd 搭一个本地 yum 源，其他主机通过这个源安装。
bash
[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/
gpgcheck = 0
EOF
[root@harbor ~]# dnf install httpd createrepo -y
[root@harbor ~]# mkdir /var/www/html/docker/ -p
[root@harbor ~]# dnf install docker-ce --downloadonly --destdir /mnt/ -y
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# createrepo -v /var/www/html/docker/然后把这个源地址改成内网地址，其他主机就能从 http://172.25.254.254:4444/docker 装 docker 了。
docker 装完，有两处配置是给 k8s 打基础：
bash
# 1. 加载 br_netfilter 模块，开启网桥流量经过 iptables
# kubelet 依赖 iptables 处理容器间流量，网桥流量默认不走 iptables，必须显式开启
[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter
[root@harbor ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
# 2. dockerd 显式开启 iptables
[root@harbor ~]# vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true
[root@harbor ~]# systemctl daemon-reload
[root@harbor ~]# systemctl enable --now dockernet.ipv4.ip_forward 让主机能转发数据包，k8s 的 Pod 跨节点通信依赖它。
生成证书
harbor 默认走 HTTPS，需要一张证书。这里用 openssl 生成自签名证书，域名填 reg.timinglee.org：
bash
[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# openssl req -newkey rsa:4096 \
-nodes -sha256 -keyout /data/certs/timinglee.org.key \
-addext "subjectAltName = DNS:reg.timinglee.org" \
-x509 -days 365 -out /data/certs/timinglee.org.crt填证书信息的时候，Common Name 必须写成 reg.timinglee.org，否则后面 docker login 会报证书不匹配。
部署 harbor
bash
[root@harbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml
hostname: reg.timinglee.org
certificate: /data/certs/timinglee.org.crt
private_key: /data/certs/timinglee.org.key
harbor_admin_password: lee
[root@harbor harbor]# ./install.sh --with-chartmuseuminstall.sh 会按 harbor.yml 把各个容器拉起来。为了让 harbor 开机自启，写一个 systemd 服务托管 docker compose：
bash
[root@harbor ~]# vim /lib/systemd/system/harbor.service
[Unit]
Description=harbor with Docker Compose
After=docker.service network-online.target
Requires=docker.service
[Service]
Type=oneshot
WorkingDirectory=/opt/harbor
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
[root@harbor ~]# systemctl enable --now harbor验证
bash
# docker 要信任这张自签证书，放到 certs.d 目录下
[root@harbor harbor]# mkdir -p /etc/docker/certs.d/reg.timinglee.org/
[root@harbor harbor]# cp /data/certs/timinglee.org.crt /etc/docker/certs.d/reg.timinglee.org/ca.crt
[root@harbor harbor]# vim /etc/hosts
172.25.254.200 harbor reg.timinglee.org
[root@harbor harbor]# systemctl restart docker
[root@harbor harbor]# docker login reg.timinglee.org -u admin
Password:
Login Succeeded能登录就说明仓库通了。
四、第二步：准备集群主机
master 和两个 node 的准备工作完全一样：
| 事项 | 命令要点 | 原因 |
|---|---|---|
| 关闭 swap | systemctl mask swap.target + 注释 /etc/fstab 里的 swap 行 |
kubelet 要精确计算节点可用内存，swap 会让它算不准 |
| 安装 docker | 走内网 yum 源 | 所有节点都要跑容器 |
| 信任 harbor 证书 | 证书分发到 /etc/docker/certs.d/ |
节点才能从仓库拉镜像 |
| 配置镜像加速器 | daemon.json 里 registry-mirrors 指向 https://reg.timinglee.org |
让 docker 默认从内网仓库拉取 |
| 主机解析 + k8s 安装源 | /etc/hosts 写四台主机；配 kubernetes.repo |
节点间互访、安装 kubeadm 用 |
证书分发用一条循环命令搞定：
bash
[root@harbor ~]# for i in 100 10 20
> do
> scp /data/certs/timinglee.org.crt root@172.25.254.$i:/etc/docker/certs.d/reg.timinglee.org/ca.crt
> done镜像加速器其实是把 reg.timinglee.org 当成 mirror 用，这样 docker pull 不带域名前缀的镜像（比如 myapp:v1）时，会直接去内网仓库找：
bash
[root@master ~]# cat >/etc/docker/daemon.json <<EOF
{
"registry-mirrors":["https://reg.timinglee.org"]
}
EOF
[root@master ~]# systemctl restart docker
[root@master ~]# docker info
Registry Mirrors:
https://reg.timinglee.org/kubernetes 安装源用的是阿里云镜像站，装 kubelet 之前先确认能查到包：
bash
[root@master ~]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name = kubernetes
baseurl = https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm/
gpgcheck = 0
[root@master ~]# dnf list kubelet全部配完重启主机，swapon -s 没有输出就说明 swap 关干净了。
五、第三步：对接容器运行时（cri-dockerd）
k8s 不直接跟 docker 说话。kubelet 通过 CRI（容器运行时接口）跟运行时通信，docker 早先靠内嵌的 dockershim 走这个接口，但 dockershim 在 k8s 1.24 被移除了。现在想让 kubelet 继续用 docker，就得在中间加一个 cri-dockerd，它把 CRI 的请求翻译成 docker 能懂的操作：
#mermaid-svg-C6NUo3Qf0or6HyMY{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-C6NUo3Qf0or6HyMY .error-icon{fill:#552222;}#mermaid-svg-C6NUo3Qf0or6HyMY .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-C6NUo3Qf0or6HyMY .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-C6NUo3Qf0or6HyMY .marker{fill:#333333;stroke:#333333;}#mermaid-svg-C6NUo3Qf0or6HyMY .marker.cross{stroke:#333333;}#mermaid-svg-C6NUo3Qf0or6HyMY svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-C6NUo3Qf0or6HyMY p{margin:0;}#mermaid-svg-C6NUo3Qf0or6HyMY .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-C6NUo3Qf0or6HyMY .cluster-label text{fill:#333;}#mermaid-svg-C6NUo3Qf0or6HyMY .cluster-label span{color:#333;}#mermaid-svg-C6NUo3Qf0or6HyMY .cluster-label span p{background-color:transparent;}#mermaid-svg-C6NUo3Qf0or6HyMY .label text,#mermaid-svg-C6NUo3Qf0or6HyMY span{fill:#333;color:#333;}#mermaid-svg-C6NUo3Qf0or6HyMY .node rect,#mermaid-svg-C6NUo3Qf0or6HyMY .node circle,#mermaid-svg-C6NUo3Qf0or6HyMY .node ellipse,#mermaid-svg-C6NUo3Qf0or6HyMY .node polygon,#mermaid-svg-C6NUo3Qf0or6HyMY .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-C6NUo3Qf0or6HyMY .rough-node .label text,#mermaid-svg-C6NUo3Qf0or6HyMY .node .label text,#mermaid-svg-C6NUo3Qf0or6HyMY .image-shape .label,#mermaid-svg-C6NUo3Qf0or6HyMY .icon-shape .label{text-anchor:middle;}#mermaid-svg-C6NUo3Qf0or6HyMY .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-C6NUo3Qf0or6HyMY .rough-node .label,#mermaid-svg-C6NUo3Qf0or6HyMY .node .label,#mermaid-svg-C6NUo3Qf0or6HyMY .image-shape .label,#mermaid-svg-C6NUo3Qf0or6HyMY .icon-shape .label{text-align:center;}#mermaid-svg-C6NUo3Qf0or6HyMY .node.clickable{cursor:pointer;}#mermaid-svg-C6NUo3Qf0or6HyMY .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-C6NUo3Qf0or6HyMY .arrowheadPath{fill:#333333;}#mermaid-svg-C6NUo3Qf0or6HyMY .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-C6NUo3Qf0or6HyMY .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-C6NUo3Qf0or6HyMY .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-C6NUo3Qf0or6HyMY .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-C6NUo3Qf0or6HyMY .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-C6NUo3Qf0or6HyMY .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-C6NUo3Qf0or6HyMY .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-C6NUo3Qf0or6HyMY .cluster text{fill:#333;}#mermaid-svg-C6NUo3Qf0or6HyMY .cluster span{color:#333;}#mermaid-svg-C6NUo3Qf0or6HyMY div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-C6NUo3Qf0or6HyMY .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-C6NUo3Qf0or6HyMY rect.text{fill:none;stroke-width:0;}#mermaid-svg-C6NUo3Qf0or6HyMY .icon-shape,#mermaid-svg-C6NUo3Qf0or6HyMY .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-C6NUo3Qf0or6HyMY .icon-shape p,#mermaid-svg-C6NUo3Qf0or6HyMY .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-C6NUo3Qf0or6HyMY .icon-shape .label rect,#mermaid-svg-C6NUo3Qf0or6HyMY .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-C6NUo3Qf0or6HyMY .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-C6NUo3Qf0or6HyMY .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-C6NUo3Qf0or6HyMY :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} CRI 接口
docker API
kubelet
cri-dockerd翻译层
dockerd
容器 1
容器 2
安装 cri-dockerd
bash
[root@master ~]# rpm -ivh cri-dockerd-0.3.14-3.el8.x86_64.rpm libcgroup-0.41-19.el8.x86_64.rpm二进制包方式，解压后三个文件，可执行文件放到 /usr/local/bin，service 文件拷到 systemd 目录：
bash
[root@docker-node1 ~]# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.4.4/cri-dockerd-0.4.4.amd64.tgz
[root@docker-node1 ~]# tar zxf cri-dockerd-0.4.4.amd64.tgz
[root@docker-node1 cri-dockerd]# install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
[root@docker-node1 cri-dockerd]# cp cri-docker.s* /lib/systemd/system不管哪种方式，有一处配置必须改：--pod-infra-container-image 指向内网仓库的 pause 镜像。pause 是每个 Pod 里"占坑"的基础容器，负责持有网络命名空间，Pod 里其他容器都共享它的网络。不指定的话 cri-dockerd 会去公网拉，拉不动 Pod 就起不来：
bash
[root@docker-node1 cri-dockerd]# vim /lib/systemd/system/cri-docker.service
[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni \
--pod-infra-container-image=reg.timinglee.org/k8s/pause:3.10.1 \
--container-runtime-endpoint fd://
Restart=always
[root@docker-node1 cri-dockerd]# systemctl daemon-reload
[root@docker-node1 cri-dockerd]# systemctl enable --now cri-docker.service装完检查 socket 文件在不在，kubelet 后面就靠它跟 cri-dockerd 通信：ll /var/run/cri-dockerd.sock。
六、第四步：初始化控制面（kubeadm init）
装软件
master 装三个，node 只装两个（node 不需要 kubectl）：
bash
# master
[root@master ~]# dnf install kubelet kubeadm kubectl -y
# node
[root@node1 ~]# dnf install kubelet kubeadm -y三台都 systemctl enable --now kubelet.service。此时 kubelet 是起不来的，因为没有集群配置，属正常现象。
先把组件镜像搬进 harbor
k8s 控制面组件本身也是容器（apiserver、etcd、coredns......），init 的时候要从镜像仓库拉。：
bash
[root@k8s-master ~]# kubeadm config images list
registry.k8s.io/kube-apiserver:v1.35.7
registry.k8s.io/kube-controller-manager:v1.35.7
registry.k8s.io/kube-scheduler:v1.35.7
registry.k8s.io/kube-proxy:v1.35.7
registry.k8s.io/coredns/coredns:v1.13.1
registry.k8s.io/pause:3.10.1
registry.k8s.io/etcd:3.6.6-0从阿里云镜像站拉下来，然后批量打 tag、批量 push 到 harbor 的 k8s 项目下：
bash
[root@master ~]# kubeadm config images pull \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.35.7 \
--cri-socket=unix:///var/run/cri-dockerd.sock
[root@master ~]# docker images --format "{{.Repository}}:{{.Tag}}" \
| awk -F "/" '/google/{system("docker tag "$0" reg.timinglee.org/k8s/"$3)}'
[root@master ~]# docker images --format "{{.Repository}}:{{.Tag}}" \
| awk -F "/" '/timinglee/{system("docker push "$0)}'执行 init
bash
[root@master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \
--image-repository reg.timinglee.org/k8s \
--kubernetes-version v1.35.7 \
--cri-socket=unix:///var/run/cri-dockerd.sock三个参数缺一不可：
| 参数 | 作用 |
|---|---|
--pod-network-cidr=10.244.0.0/16 |
声明 Pod 网段，flannel 就按这个网段分配 IP，写错网络插件会对不上 |
--image-repository reg.timinglee.org/k8s |
让 kubeadm 从内网仓库拉组件镜像 |
--cri-socket=unix:///var/run/cri-dockerd.sock |
告诉 kubeadm 用 cri-dockerd 而不是默认的 containerd |
命令成功后会打印一串 join 凭证，这是其他节点入群的唯一凭据，务必保存：
kubeadm join 172.25.254.100:6443 --token tdwjoc.8d1yw3wl4r4tm6c4 \
--discovery-token-ca-cert-hash sha256:6b5950ef2cdba85d6dfdb564ee90d4187fa3d341767dc9852cbdd5c9dee4f927#mermaid-svg-N6dK4fopXgOHExeM{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-N6dK4fopXgOHExeM .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-N6dK4fopXgOHExeM .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-N6dK4fopXgOHExeM .error-icon{fill:#552222;}#mermaid-svg-N6dK4fopXgOHExeM .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-N6dK4fopXgOHExeM .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-N6dK4fopXgOHExeM .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-N6dK4fopXgOHExeM .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-N6dK4fopXgOHExeM .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-N6dK4fopXgOHExeM .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-N6dK4fopXgOHExeM .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-N6dK4fopXgOHExeM .marker{fill:#333333;stroke:#333333;}#mermaid-svg-N6dK4fopXgOHExeM .marker.cross{stroke:#333333;}#mermaid-svg-N6dK4fopXgOHExeM svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-N6dK4fopXgOHExeM p{margin:0;}#mermaid-svg-N6dK4fopXgOHExeM .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-N6dK4fopXgOHExeM .cluster-label text{fill:#333;}#mermaid-svg-N6dK4fopXgOHExeM .cluster-label span{color:#333;}#mermaid-svg-N6dK4fopXgOHExeM .cluster-label span p{background-color:transparent;}#mermaid-svg-N6dK4fopXgOHExeM .label text,#mermaid-svg-N6dK4fopXgOHExeM span{fill:#333;color:#333;}#mermaid-svg-N6dK4fopXgOHExeM .node rect,#mermaid-svg-N6dK4fopXgOHExeM .node circle,#mermaid-svg-N6dK4fopXgOHExeM .node ellipse,#mermaid-svg-N6dK4fopXgOHExeM .node polygon,#mermaid-svg-N6dK4fopXgOHExeM .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-N6dK4fopXgOHExeM .rough-node .label text,#mermaid-svg-N6dK4fopXgOHExeM .node .label text,#mermaid-svg-N6dK4fopXgOHExeM .image-shape .label,#mermaid-svg-N6dK4fopXgOHExeM .icon-shape .label{text-anchor:middle;}#mermaid-svg-N6dK4fopXgOHExeM .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-N6dK4fopXgOHExeM .rough-node .label,#mermaid-svg-N6dK4fopXgOHExeM .node .label,#mermaid-svg-N6dK4fopXgOHExeM .image-shape .label,#mermaid-svg-N6dK4fopXgOHExeM .icon-shape .label{text-align:center;}#mermaid-svg-N6dK4fopXgOHExeM .node.clickable{cursor:pointer;}#mermaid-svg-N6dK4fopXgOHExeM .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-N6dK4fopXgOHExeM .arrowheadPath{fill:#333333;}#mermaid-svg-N6dK4fopXgOHExeM .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-N6dK4fopXgOHExeM .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-N6dK4fopXgOHExeM .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-N6dK4fopXgOHExeM .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-N6dK4fopXgOHExeM .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-N6dK4fopXgOHExeM .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-N6dK4fopXgOHExeM .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-N6dK4fopXgOHExeM .cluster text{fill:#333;}#mermaid-svg-N6dK4fopXgOHExeM .cluster span{color:#333;}#mermaid-svg-N6dK4fopXgOHExeM div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-N6dK4fopXgOHExeM .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-N6dK4fopXgOHExeM rect.text{fill:none;stroke-width:0;}#mermaid-svg-N6dK4fopXgOHExeM .icon-shape,#mermaid-svg-N6dK4fopXgOHExeM .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-N6dK4fopXgOHExeM .icon-shape p,#mermaid-svg-N6dK4fopXgOHExeM .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-N6dK4fopXgOHExeM .icon-shape .label rect,#mermaid-svg-N6dK4fopXgOHExeM .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-N6dK4fopXgOHExeM .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-N6dK4fopXgOHExeM .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-N6dK4fopXgOHExeM :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} kubeadm init
生成 CA 证书apiserver / etcd / 各组件证书
部署 etcd集群数据落地
部署 kube-apiserver控制面入口就绪
部署 controller-manager 和 scheduler
生成 admin.conf拷到 ~/.kube/config
部署 coredns 和 kube-proxy
打印 join 命令（token + CA hash）
bash
[root@master ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" > ~/.bash_profile
[root@master ~]# source ~/.bash_profile
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 102s v1.35.3现在能看到 master 自己了，但状态是 NotReady。网络插件还没装，节点之间还连不通。
七、第五步：工作节点加入
node 上执行 init 时打印的那条 join 命令（注意 node 要用自己的 cri-socket）：
bash
[root@node1 ~]# kubeadm join 172.25.254.100:6443 --token jl4ztx.cax3iysvu7onsh5s \
--discovery-token-ca-cert-hash sha256:6b5950ef2cdba85d6dfdb564ee90d4187fa3d341767dc9852cbdd5c9dee4f927 \
--cri-socket=unix:///var/run/cri-dockerd.sockjoin 的过程本质是一次带认证的注册：
kube-apiserver node 节点 kube-apiserver node 节点 #mermaid-svg-xWEddQNfDECguTPp{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-xWEddQNfDECguTPp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-xWEddQNfDECguTPp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-xWEddQNfDECguTPp .error-icon{fill:#552222;}#mermaid-svg-xWEddQNfDECguTPp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-xWEddQNfDECguTPp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-xWEddQNfDECguTPp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-xWEddQNfDECguTPp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-xWEddQNfDECguTPp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-xWEddQNfDECguTPp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-xWEddQNfDECguTPp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-xWEddQNfDECguTPp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-xWEddQNfDECguTPp .marker.cross{stroke:#333333;}#mermaid-svg-xWEddQNfDECguTPp svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-xWEddQNfDECguTPp p{margin:0;}#mermaid-svg-xWEddQNfDECguTPp .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-xWEddQNfDECguTPp text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-xWEddQNfDECguTPp .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-xWEddQNfDECguTPp .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-xWEddQNfDECguTPp .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-xWEddQNfDECguTPp .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-xWEddQNfDECguTPp #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-xWEddQNfDECguTPp .sequenceNumber{fill:white;}#mermaid-svg-xWEddQNfDECguTPp #sequencenumber{fill:#333;}#mermaid-svg-xWEddQNfDECguTPp #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-xWEddQNfDECguTPp .messageText{fill:#333;stroke:none;}#mermaid-svg-xWEddQNfDECguTPp .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-xWEddQNfDECguTPp .labelText,#mermaid-svg-xWEddQNfDECguTPp .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-xWEddQNfDECguTPp .loopText,#mermaid-svg-xWEddQNfDECguTPp .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-xWEddQNfDECguTPp .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-xWEddQNfDECguTPp .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-xWEddQNfDECguTPp .noteText,#mermaid-svg-xWEddQNfDECguTPp .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-xWEddQNfDECguTPp .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-xWEddQNfDECguTPp .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-xWEddQNfDECguTPp .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-xWEddQNfDECguTPp .actorPopupMenu{position:absolute;}#mermaid-svg-xWEddQNfDECguTPp .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-xWEddQNfDECguTPp .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-xWEddQNfDECguTPp .actor-man circle,#mermaid-svg-xWEddQNfDECguTPp line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-xWEddQNfDECguTPp :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 用 token 生成引导凭证 请求加入（携带 token） 校验 token + CA hash 下发 kubelet 证书与集群配置 启动 kubelet，开始上报节点状态 心跳 + 节点信息（NotReady）
bash
[root@master ~]# kubeadm token create --print-join-command三台主机都就位后，kubectl get nodes 能看到三台机器，但全部 NotReady------节点能注册，互相通信还不行。
八、第六步：部署网络插件（flannel）
为什么需要网络插件
k8s 要求每个 Pod 都有独立的 IP，而且跨节点也能互通。这个"每个节点一个 Pod 网段 + 跨节点路由"的能力，k8s 自己不实现，交给 CNI 插件。本实验用 flannel。
flannel 的核心思路是给每台节点划一个子网（node1 是 10.244.1.0/24，node2 是 10.244.2.0/24），节点之间用 VXLAN 隧道把数据包封起来传。对 Pod 来说，大家在一个大网段里，好像都在同一台机器上：
#mermaid-svg-MIwF43o7TYIAcjHb{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-MIwF43o7TYIAcjHb .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-MIwF43o7TYIAcjHb .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-MIwF43o7TYIAcjHb .error-icon{fill:#552222;}#mermaid-svg-MIwF43o7TYIAcjHb .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-MIwF43o7TYIAcjHb .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-MIwF43o7TYIAcjHb .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-MIwF43o7TYIAcjHb .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-MIwF43o7TYIAcjHb .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-MIwF43o7TYIAcjHb .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-MIwF43o7TYIAcjHb .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-MIwF43o7TYIAcjHb .marker{fill:#333333;stroke:#333333;}#mermaid-svg-MIwF43o7TYIAcjHb .marker.cross{stroke:#333333;}#mermaid-svg-MIwF43o7TYIAcjHb svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-MIwF43o7TYIAcjHb p{margin:0;}#mermaid-svg-MIwF43o7TYIAcjHb .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-MIwF43o7TYIAcjHb .cluster-label text{fill:#333;}#mermaid-svg-MIwF43o7TYIAcjHb .cluster-label span{color:#333;}#mermaid-svg-MIwF43o7TYIAcjHb .cluster-label span p{background-color:transparent;}#mermaid-svg-MIwF43o7TYIAcjHb .label text,#mermaid-svg-MIwF43o7TYIAcjHb span{fill:#333;color:#333;}#mermaid-svg-MIwF43o7TYIAcjHb .node rect,#mermaid-svg-MIwF43o7TYIAcjHb .node circle,#mermaid-svg-MIwF43o7TYIAcjHb .node ellipse,#mermaid-svg-MIwF43o7TYIAcjHb .node polygon,#mermaid-svg-MIwF43o7TYIAcjHb .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-MIwF43o7TYIAcjHb .rough-node .label text,#mermaid-svg-MIwF43o7TYIAcjHb .node .label text,#mermaid-svg-MIwF43o7TYIAcjHb .image-shape .label,#mermaid-svg-MIwF43o7TYIAcjHb .icon-shape .label{text-anchor:middle;}#mermaid-svg-MIwF43o7TYIAcjHb .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-MIwF43o7TYIAcjHb .rough-node .label,#mermaid-svg-MIwF43o7TYIAcjHb .node .label,#mermaid-svg-MIwF43o7TYIAcjHb .image-shape .label,#mermaid-svg-MIwF43o7TYIAcjHb .icon-shape .label{text-align:center;}#mermaid-svg-MIwF43o7TYIAcjHb .node.clickable{cursor:pointer;}#mermaid-svg-MIwF43o7TYIAcjHb .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-MIwF43o7TYIAcjHb .arrowheadPath{fill:#333333;}#mermaid-svg-MIwF43o7TYIAcjHb .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-MIwF43o7TYIAcjHb .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-MIwF43o7TYIAcjHb .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MIwF43o7TYIAcjHb .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-MIwF43o7TYIAcjHb .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MIwF43o7TYIAcjHb .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-MIwF43o7TYIAcjHb .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-MIwF43o7TYIAcjHb .cluster text{fill:#333;}#mermaid-svg-MIwF43o7TYIAcjHb .cluster span{color:#333;}#mermaid-svg-MIwF43o7TYIAcjHb div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-MIwF43o7TYIAcjHb .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-MIwF43o7TYIAcjHb rect.text{fill:none;stroke-width:0;}#mermaid-svg-MIwF43o7TYIAcjHb .icon-shape,#mermaid-svg-MIwF43o7TYIAcjHb .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-MIwF43o7TYIAcjHb .icon-shape p,#mermaid-svg-MIwF43o7TYIAcjHb .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-MIwF43o7TYIAcjHb .icon-shape .label rect,#mermaid-svg-MIwF43o7TYIAcjHb .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-MIwF43o7TYIAcjHb .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-MIwF43o7TYIAcjHb .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-MIwF43o7TYIAcjHb :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} node2
node1
VXLAN 隧道封装传输
Pod 10.244.1.10
flannel.1
flannel.1
Pod 10.244.2.3
部署 flannel
flannel 以 DaemonSet 方式运行，每台节点自动跑一个副本。镜像同样先进 harbor：
bash
[root@k8s-master ~]# tar zxf kube-flannel-v0.28.9.tar.gz
[root@k8s-master ~]# docker load -i kube-flannel-v0.28.9.tarflannel 的 yaml 文件里引用的镜像是公网地址，要改成内网仓库的路径：
bash
[root@k8s-master flannel]# grep -n "image:" kube-flannel.yml
148: image: flannel-io/flannel:v0.28.9
175: image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3
186: image: flannel-io/flannel:v0.28.9
[root@k8s-master ~]# vim kube-flannel.yml
# 把 image 前缀改成 reg.timinglee.org/flannel-io/
[root@master ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created等几十秒，三台节点陆续变 Ready：
bash
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 13m v1.35.3
node1 Ready <none> 9m44s v1.35.3
node2 Ready <none> 9m36s v1.35.3到这里，集群部署的本体就完成了。flannel 用的网段 10.244.0.0/16 必须和 init 时 --pod-network-cidr 一致，这是本实验最容易踩的坑。
九、部署完成后的验证
bash
[root@k8s-master ~]# kubectl run lee --image nginx:latest
pod/lee created
[root@k8s-master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
lee 1/1 Running 0 25s 10.244.1.10 k8s-node1 <none> <none>Pod 被调度到 node1，拿到了网段里的 IP（10.244.1.10），说明调度、镜像拉取、flannel 网络全部正常。注意这个 IP 正好落在 node1 的 10.244.1.0/24 子网里，flannel 的网段划分和预期一致。
十、自动化部署：Ansible
前面的步骤都是手动一台台敲的。步骤固定、重复性高，正好适合用 Ansible 批量执行。实验里把前三步环境准备做成了 playbook，在 harbor 主机上统一管控。
先配好管控端
bash
[root@harbor ~]# dnf install ansible-core.x86_64 ansible.noarch -y
[root@harbor ~]# useradd devops
[root@harbor ~]# echo lee | passwd --stdin devopsinventory 里写三台集群主机，ansible.cfg 里指定用 devops 用户提权执行：
yaml
[devops@harbor ansible]$ vim inventory
[servers]
172.25.254.100
172.25.254.20
172.25.254.10
[devops@harbor ansible]$ vim ansible.cfg
[defaults]
inventory = /home/devops/ansible/inventory
remote_user = devops
host_key_checking = false
[privilege_escalation]
become = true
become_method = sudo
become_user = root先验证连通性：ansible all -m ping，三台都回 pong 就往下走。
三个 playbook 对应三步环境准备
| playbook | 做的事 | 对应手动步骤 |
|---|---|---|
| 部署 docker | 配 yum 源、装 docker-ce、改 docker.service、配加速器、分发证书、加载内核模块 | 第四步主机准备 |
| 部署 cri-dockerd | 拷贝二进制和 service 文件、改 pause 镜像参数、启动服务 | 第五步运行时对接 |
| 部署 k8s 软件 | 配 kubernetes.repo、master 装三个包、node 装两个包、启动 kubelet | 第六步装软件 |
playbook 用的是 yum_repository、dnf、copy、replace、service 这些标准模块，核心逻辑和手动敲的命令一一对应：
yaml
- name: setup docker repo
yum_repository:
name: docker
baseurl: http://172.25.254.254:4444/docker
gpgcheck: no
- name: install docker
dnf:
name: docker-ce
state: present
- name: setup docker.service
replace:
path: /lib/systemd/system/docker.service
regexp: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"
replace: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true"执行就一条命令：
bash
[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml#mermaid-svg-10eMtok4RMESCLtP{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-10eMtok4RMESCLtP .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-10eMtok4RMESCLtP .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-10eMtok4RMESCLtP .error-icon{fill:#552222;}#mermaid-svg-10eMtok4RMESCLtP .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-10eMtok4RMESCLtP .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-10eMtok4RMESCLtP .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-10eMtok4RMESCLtP .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-10eMtok4RMESCLtP .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-10eMtok4RMESCLtP .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-10eMtok4RMESCLtP .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-10eMtok4RMESCLtP .marker{fill:#333333;stroke:#333333;}#mermaid-svg-10eMtok4RMESCLtP .marker.cross{stroke:#333333;}#mermaid-svg-10eMtok4RMESCLtP svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-10eMtok4RMESCLtP p{margin:0;}#mermaid-svg-10eMtok4RMESCLtP .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-10eMtok4RMESCLtP .cluster-label text{fill:#333;}#mermaid-svg-10eMtok4RMESCLtP .cluster-label span{color:#333;}#mermaid-svg-10eMtok4RMESCLtP .cluster-label span p{background-color:transparent;}#mermaid-svg-10eMtok4RMESCLtP .label text,#mermaid-svg-10eMtok4RMESCLtP span{fill:#333;color:#333;}#mermaid-svg-10eMtok4RMESCLtP .node rect,#mermaid-svg-10eMtok4RMESCLtP .node circle,#mermaid-svg-10eMtok4RMESCLtP .node ellipse,#mermaid-svg-10eMtok4RMESCLtP .node polygon,#mermaid-svg-10eMtok4RMESCLtP .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-10eMtok4RMESCLtP .rough-node .label text,#mermaid-svg-10eMtok4RMESCLtP .node .label text,#mermaid-svg-10eMtok4RMESCLtP .image-shape .label,#mermaid-svg-10eMtok4RMESCLtP .icon-shape .label{text-anchor:middle;}#mermaid-svg-10eMtok4RMESCLtP .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-10eMtok4RMESCLtP .rough-node .label,#mermaid-svg-10eMtok4RMESCLtP .node .label,#mermaid-svg-10eMtok4RMESCLtP .image-shape .label,#mermaid-svg-10eMtok4RMESCLtP .icon-shape .label{text-align:center;}#mermaid-svg-10eMtok4RMESCLtP .node.clickable{cursor:pointer;}#mermaid-svg-10eMtok4RMESCLtP .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-10eMtok4RMESCLtP .arrowheadPath{fill:#333333;}#mermaid-svg-10eMtok4RMESCLtP .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-10eMtok4RMESCLtP .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-10eMtok4RMESCLtP .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-10eMtok4RMESCLtP .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-10eMtok4RMESCLtP .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-10eMtok4RMESCLtP .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-10eMtok4RMESCLtP .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-10eMtok4RMESCLtP .cluster text{fill:#333;}#mermaid-svg-10eMtok4RMESCLtP .cluster span{color:#333;}#mermaid-svg-10eMtok4RMESCLtP div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-10eMtok4RMESCLtP .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-10eMtok4RMESCLtP rect.text{fill:none;stroke-width:0;}#mermaid-svg-10eMtok4RMESCLtP .icon-shape,#mermaid-svg-10eMtok4RMESCLtP .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-10eMtok4RMESCLtP .icon-shape p,#mermaid-svg-10eMtok4RMESCLtP .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-10eMtok4RMESCLtP .icon-shape .label rect,#mermaid-svg-10eMtok4RMESCLtP .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-10eMtok4RMESCLtP .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-10eMtok4RMESCLtP .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-10eMtok4RMESCLtP :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} harbor 主机安装 ansible + devops 用户
配置 inventory3 台集群主机
playbook 1部署 docker 环境
playbook 2部署 cri-dockerd
playbook 3部署 kubeadm / kubelet
手动：kubeadm init
手动：分发 join 命令
手动：kubeadm join × 2