一、Kubernetes 简单介绍
Kubernetes 简称 K8s,是 Google 开源容器编排平台,用于自动化容器部署、扩缩容、故障自愈。
K8s 集群分为控制平面(Master) 和工作节点(Node):
- Master:apiserver、etcd、scheduler、controller‑manager,负责集群调度、存储集群所有资源状态
- Node:kubelet、kube‑proxy、容器运行时,真正执行业务 Pod 运行
- Pod:K8s 最小调度单元,一个 Pod 内部可以运行多个容器
二、环境规划
| 主机名 | IP | 配置 | 角色 |
|---|---|---|---|
| harbor.timinglee.org | 172.25.254.250 | 1C1G | Harbor 私有镜像仓库 |
| master‑node | 172.25.254.100 | 4C4G+ | K8s 控制节点 master |
| node1 | 172.25.254.10 | 2C2G | worker 工作节点 |
| node2 | 172.25.254.20 | 2C2G | worker 工作节点 |
三、部署 Harbor 私有镜像仓库
3.1 部署 Docker 本地 yum 源(harbor 节点)
[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
> [docker]
> name = docker
> baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/
> gpgcheck = 0
> EOF
[root@harbor ~]# dnf install httpd createrepo -y
正在更新 Subscription Management 软件仓库。
无法读取客户身份
本系统尚未在权利服务器中注册。可使用 "rhc" 或 "subscription-manager" 进行注册。
已杀死
[root@harbor ~]# mkdir /var/www/html/docker/ -p
#修改httpd监听端口
[root@harbor ~]# vim /etc/httpd/conf/httpd.conf
Listen 4444
[root@harbor ~]# systemctl enable --now httpd
[root@harbor ~]# dnf install docker-ce --downloadonly --destdir /mnt/ -y
#下载docker‑ce rpm包到本地
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# createrepo -v /var/www/html/docker/
#所有节点使用这个内网源
[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = http://172.25.254.250:4444/docker
gpgcheck = 0
EOF
3.2 harbor 节点安装 docker,内核参数调优
[root@harbor ~]# dnf install docker-ce -y
#加载br_netfilter模块
[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter
[root@harbor ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
[root@harbor ~]# systemctl restart systemd-modules-load.service
[root@harbor ~]# sysctl --system
#修改docker.service参数
[root@harbor ~]# vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true
[root@harbor ~]# systemctl daemon-reload
[root@harbor ~]# systemctl enable --now docker
3.3 生成自签 TLS 证书
[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# openssl req -newkey rsa:4096 \-nodes -sha256 -keyout /data/certs/timinglee.org.key \-addext "subjectAltName = DNS:reg.timinglee.org" \-x509 -days 365 -out /data/certs/timinglee.org.crt
交互填写信息:
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Shangxi
Locality Name (eg, city) [Default City]:xi'an
Organization Name (eg, company) [Default Company Ltd]:kubernetes
Organizational Unit Name (eg, section) []:harbor
Common Name (eg, your name or your server's hostname) []:reg.timinglee.org
Email Address []:admin@timinglee.org
3.4 修改 harbor 配置,安装 harbor
[root@harbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/
[root@harbor harbor]# ls
common common.sh docker-compose.yml harbor.v2.5.4.tar.gz harbor.yml.tmpl install.sh LICENSE prepare
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml
hostname: reg.timinglee.org
certificate: /data/certs/timinglee.org.crt
private_key: /data/certs/timinglee.org.key
harbor_admin_password: lee
#安装开启chartmuseum支持helm仓库
[root@harbor harbor]# ./install.sh --with-chartmuseum

3.5 systemd 管理 harbor 服务
[root@harbor harbor]# vim /lib/systemd/system/harbor.service
[Unit]
Description=harbor with Docker Compose
Documentation=https://reg.timinglee.com/compose/
After=docker.service network-online.target
Requires=docker.service
[Service]
Type=oneshot
WorkingDirectory=/opt/harbor
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
[root@harbor harbor]# docker compose down
[root@harbor harbor]# systemctl enable --now harbor
[root@harbor harbor]# docker compose ps

3.6 验证 harbor 登录
[root@harbor harbor]# mkdir /etc/docker/certs.d/reg.timinglee.org/ -p
[root@harbor harbor]# cp /data/certs/timinglee.org.crt /etc/docker/certs.d/reg.timinglee.org/ca.crt
[root@harbor harbor]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.250 harbor reg.timinglee.org
[root@harbor harbor]# systemctl restart docker
[root@harbor harbor]# docker compose up -d
[root@harbor harbor]# docker login reg.timinglee.org -u admin
Password:
Login Succeeded

harbor 节点执行,预先把证书推送到三台 k8s 节点,后面 ansible 也会二次同步证书
[root@harbor harbor]# for i in 100 10 20
> do
> scp /data/certs/timinglee.org.crt root@172.25.254.$i:/etc/docker/certs.d/reg.timinglee.org/ca.crt
> done
方案一:手动部署 k8s 集群(适合新手理解每一步原理)
四、所有 k8s 节点统一环境初始化
下面操作master、node1、node2 三台机器全部执行
4.1 关闭 swap
[root@k8s-master ~]# systemctl disable --now swap.target
[root@k8s-master ~]# systemctl mask swap.target
[root@k8s-master ~]# sed '/swap/s/^/#/g' -i /etc/fstab
#验证,无输出代表成功
[root@k8s-master ~]# swapon -s
4.2 docker 配置镜像加速器指向内网 harbor
[root@k8s-master ~]# cat >/etc/docker/daemon.json <<EOF
> {
> "registry-mirrors":["https://reg.timinglee.org"]
> }
> EOF
[root@k8s-master ~]# systemctl enable docker
[root@k8s-master ~]# systemctl restart docker
[root@k8s-master ~]# docker info
可以看到
Registry Mirrors:
https://reg.timinglee.org/
4.3 hosts 主机解析
[root@k8s-master ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100 k8s-master
172.25.254.10 k8s-nade1
172.25.254.20 k8s-node2
172.25.254.250 harbor reg.timinglee.org
4.4 kubernetes yum 源
[root@k8s-master ~]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name = kubernetes
baseurl = https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm/
gpgcheck = 0
[root@k8s-master ~]# dnf list kubelet

五、所有节点部署 cri‑dockerd(适配 docker 运行时)
二进制部署方式:
[root@k8s-master ~]# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.4.4/cri-dockerd-0.4.4.amd64.tgz
[root@k8s-master ~]# tar zxf cri-dockerd-0.4.4.amd64.tgz
[root@k8s-master ~]# cd cri-dockerd/
[root@k8s-master cri-dockerd]# ls
cri-dockerd cri-docker.service cri-docker.socket
[root@k8s-master cri-dockerd]# install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
[root@k8s-master cri-dockerd]# ls -l /usr/local/bin/cri-dockerd
-rwxr-xr-x 1 root root 51638434 8月 22 19:38 /usr/local/bin/cri-dockerd
[root@k8s-master cri-dockerd]# cp cri-docker.s* /lib/systemd/system
[root@k8s-master cri-dockerd]# chmod +x /lib/systemd/system/cri-docker.s*
[root@k8s-master cri-dockerd]# vim /lib/systemd/system/cri-docker.service
[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=reg.timinglee.org/k8s/pause:3.10.1 --container-runtime-endpoint fd://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
[root@k8s-master cri-dockerd]# systemctl daemon-reload
[root@k8s-master cri-dockerd]# systemctl enable --now cri-docker.service
[root@k8s-master cri-dockerd]# systemctl status cri-docker.service
● cri-docker.service - CRI Interface for Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/cri-docker.service; enabled; preset: disabled)
Active: active (running) since Sat 2026-08-22 18:04:30 CST; 1h 35min ago
TriggeredBy: ● cri-docker.socket
Docs: https://docs.mirantis.com
Main PID: 2856 (cri-dockerd)
Tasks: 10
Memory: 49.5M (peak: 92.0M)
CPU: 37.468s
CGroup: /system.slice/cri-docker.service
└─2856 /usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=reg.timinglee.org/k8s/pause:3.10
验证 socket 文件存在
[root@k8s-master cri-dockerd]# ll /var/run/cri-dockerd.sock
srw-rw---- 1 root docker 0 8月 22 18:04 /var/run/cri-dockerd.sock
六、安装 k8s 组件
master 节点
[root@k8s-master ~]# dnf install kubelet kubeadm kubectl -y
[root@k8s-master ~]# systemctl enable --now kubelet.service
[root@k8s-master ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@k8s-master ~]# echo "source <(kubeadm completion bash)" >> ~/.bashrc
[root@k8s-master ~]# source ~/.bashrc
node 节点
[root@k8s-node2 ~]# dnf install kubelet kubeadm -y
[root@k8s-node2 ~]# systemctl enable --now kubelet.service
七、master 节点拉取镜像并推送到内网 Harbor
#查看需要哪些镜像
[root@k8s-master ~]# kubeadm config images list
#拉取镜像,指定阿里源,指定cri‑socket
[root@k8s-master ~]# kubeadm config images pull \--image-repository registry.aliyuncs.com/google_containers \--kubernetes-version v1.35.7 \--cri-socket=unix:///var/run/cri-dockerd.sock
#登录harbor
[root@k8s-master ~]# docker login reg.timinglee.org -u admin
#打标签推送到私有harbor仓库
[root@k8s-master ~]# docker images --format "{{.Repository}}:{{.Tag}}" | awk -F "/" '/google/{system("docker tag "$0" reg.timinglee.org/k8s/"$3)}'
[root@k8s-master ~]# docker images --format "{{.Repository}}:{{.Tag}}" | awk -F "/" '/timinglee/{system("docker push "$0)}'
八、kubeadm 初始化集群(master 节点)
[root@k8s-master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \--image-repository reg.timinglee.org/k8s \--kubernetes-version v1.35.7 \--cri-socket=unix:///var/run/cri-dockerd.sock
执行成功后输出 join 命令,类似:
kubeadm join 172.25.254.100:6443 --token h9725j.s6q4mrz25fpcua3q --discovery-token-ca-cert-hash sha256:bb480a79316ad10f529e7cd4627508daca68c39e6c5a3fe1671bbdb92a9ff87a
如果忘了
[root@k8s-master ~]# kubeadm token create --print-join-command
kubeadm join 172.25.254.100:6443 --token h9725j.s6q4mrz25fpcua3q --discovery-token-ca-cert-hash sha256:bb480a79316ad10f529e7cd4627508daca68c39e6c5a3fe1671bbdb92a9ff87a
如果初始化集群失败
[root@master ~]# kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock
九、node 节点加入集群
复制 master 输出的 kubeadm join 命令,每个 node 节点执行,务必带上 cri‑socket 参数
添加kubernets环境变量到本机
[root@k8s-master ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" > ~/.bash_profile
[root@k8s-master ~]# source ~/.bash_profile
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 102s v1.35.3
添加node节点到本集群
[root@k8s-node2 ~]# kubeadm join 172.25.254.100:6443 --token h9725j.s6q4mrz25fpcua3q --discovery-token-ca-cert-hash sha256:bb480a79316ad10f529e7cd4627508daca68c39e6c5a3fe1671bbdb92a9ff87a --cri-socket=unix:///var/run/cri-dockerd.sock
测试
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady control-plane 2d9h v1.35.7
k8s-node1 NotReady <none> 2d4h v1.35.7
k8s-node2 NotReady <none> 2d9h v1.35.7
十、部署 flannel 网络插件
pod 网络必须安装,否则节点不会 Ready
[root@k8s-master ~]# tar zxf kube-flannel-v0.28.9.tar.gz [root@k8s-master ~]# cd flannel/ [root@k8s-master flannel]# docker load -i kube-flannel-v0.28.9.tar Loaded image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 Loaded image: ghcr.io/flannel-io/flannel:v0.28.9 [root@k8s-master ~]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 [root@k8s-master ~]# docker push reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.1-flanne3 [root@k8s-master ~]# docker tag ghcr.io/flannel-io/flannel:v0.28.9 reg.timinglee.org/flannel-io/flannel:v0.28.9 [root@k8s-master ~]# docker push reg.timinglee.org/flannel-io/flannel:v0.28.9 [root@k8s-master ~]# vim kube-flannel.yml image: flannel-io/flannel:v0.28.9 image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3 image: flannel-io/flannel:v0.28.9 [root@k8s-master flannel]# grep -n image: kube-flannel.yml 148: image: ghcr.io/flannel-io/flannel:v0.28.9 175: image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 186: image: ghcr.io/flannel-io/flannel:v0.28.9 [root@k8s-master flannel]# kubectl apply -f kube-flannel.yml namespace/kube-flannel unchanged serviceaccount/flannel unchanged clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged configmap/kube-flannel-cfg unchanged daemonset.apps/kube-flannel-ds unchanged [root@k8s-master flannel]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane 2d9h v1.35.7 k8s-node1 Ready <none> 2d4h v1.35.7 k8s-node2 Ready <none> 2d9h v1.35.7方案二:Ansible 自动化初始化 K8s 节点环境
🎯适用场景:多台 k8s 节点重复配置,解放双手。只做环境预处理,集群初始化、join、网络插件依旧沿用方案一第八、九、十章节手动
Ansible 自动化部署(适合多节点,批量做环境初始化,提高部署效率)
步骤 1:部署 Ansible(在 harbor 节点执行,作为 ansible 控制机)
[root@harbor ~]# vim /etc/yum.repos.d/epel.repo [epel] name = epel baseurl = https://mirrors.aliyun.com/epel/9/Everything/x86_64/ gpgcheck = 0 [root@harbor ~]# dnf install ansible-core.x86_64 ansible.noarch -y步骤 2:创建 ansible 工作目录与用户
root@harbor ~]# useradd devops [root@harbor ~]# echo lee | passwd --stdin devops 更改用户 devops 的密码 。 passwd:所有的身份验证令牌已经成功更新。 [root@harbor ~]# su - devops [devops@harbor ~]$ mkdir ansible [devops@harbor ~]$ cd ansible/2.1 主机清单 inventory
文件名:
inventory
[devops@harbor ansible]$ vim inventory [servers] 172.25.254.100 172.25.254.20 172.25.254.102.2 ansible.cfg 配置文件
文件名:
ansible.cfg
[devops@harbor ansible]$ vim ansible.cfg [defaults] inventory = /home/devops/ansible/inventory remote_user = devops ask_pass = false host_key_checking = false [privilege_escalation] become = true become_method = sudo become_user = root become_ask_pass = false2.3 推送 devops 用户与 sudo 权限到所有 k8s 节点
执行 ping 连通性测试,确认网络与 ssh 正常
[devops@harbor ansible]$ ansible all -m ping --ask-pass SSH password: 172.25.254.100 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" } 172.25.254.20 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" } 172.25.254.10 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" }批量创建 devops 用户,免密 sudo
[devops@harbor ansible]$ ansible all -m user -a "name=devops state=present" [devops@harbor ansible]$ ansible all -m shell -a 'echo "lee" | passwd --stdin devops' [devops@harbor ansible]$ ansible all -m lineinfile -a 'path=/etc/sudoers line="devops ALL=(ALL) NOPASSWD: ALL" insertafter=EOF'步骤 3:准备 cri‑dockerd 二进制文件
把 cri‑dockerd 压缩包上传到 ansible 工作目录,解压,playbook 会把二进制、service 文件分发到远端主机
[devops@harbor ansible]$ wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.4.4/cri-dockerd-0.4.4.amd64.tgz [devops@harbor ansible]$ tar zxf cri-dockerd-0.4.4.amd64.tgz修改
cri-dockerd/cri-docker.service启动参数,指定内网 pause 镜像
[devops@harbor ansible]$ cd cri-dockerd/ [devops@harbor cri-dockerd]$ ls cri-dockerd cri-docker.service cri-docker.socket [devops@harbor cri-dockerd]$ vim cri-docker.service [Service] Type=notify ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=reg.timinglee.org/k8s/pause:3.10.1 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always步骤 4:完整 playbook k8s-init.yml
playbook 完成工作:关闭 swap、配置 docker 源、安装 docker、修改 docker 启动参数、daemon.json 镜像加速器、分发 harbor 证书、内核模块与 sysctl 参数、推送 cri‑dockerd 二进制与 service 文件、配置 k8s yum 源、区分 master/node 安装不同 k8s 软件、配置 kubectl 命令补全、启动 docker 与 cri‑dockerd、kubelet。
[devops@harbor ansible]$ vim kubernetes-devops.yml - name: k8s nodes environment init hosts: all tasks: # 1.关闭swap - name: disable swap service shell: | systemctl disable --now swap.target systemctl mask swap.target sed -i '/swap/s/^/#/' /etc/fstab ignore_errors: yes # 2.配置docker yum源 - name: setup docker repo yum_repository: name: docker description: docker repo baseurl: http://172.25.254.254:4444/docker file: docker gpgcheck: no #3.安装docker-ce - name: install docker-ce dnf: name: docker-ce state: present #4.修改docker.service启动参数 - name: modify docker service ExecStart replace: path: /lib/systemd/system/docker.service regexp: 'ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock' replace: 'ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true' #5.docker daemon.json配置镜像加速器 - name: config docker daemon.json copy: content: | { "registry-mirrors":["https://reg.timinglee.org"] } dest: /etc/docker/daemon.json #6.创建harbor证书目录,拷贝ca证书 - name: create cert dir file: path: /etc/docker/certs.d/reg.timinglee.org state: directory mode: '0755' - name: copy harbor ca cert copy: src: /data/certs/timinglee.org.crt dest: /etc/docker/certs.d/reg.timinglee.org/ca.crt remote_src: no - name: copy cert to system trust copy: src: /data/certs/timinglee.org.crt dest: /etc/pki/ca-trust/source/anchors/ca.crt #7.内核模块 br_netfilter - name: load br_netfilter module config copy: dest: /etc/modules-load.d/docker_mod.conf content: br_netfilter - name: sysctl bridge and ip_forward copy: dest: /etc/sysctl.d/docker.conf content: | net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 #8.分发cri‑dockerd二进制和systemd单元 - name: copy cri‑dockerd binary & service file copy: src: "{{item.src}}" dest: "{{item.dest}}" mode: "{{item.mode | default('0644')}}" loop: - { src: "./cri-dockerd/cri-dockerd", dest: "/usr/local/bin/cri-dockerd", mode: "0755" } - { src: "./cri-dockerd/cri-docker.service", dest: "/lib/systemd/system/cri-docker.service" } - { src: "./cri-dockerd/cri-docker.socket", dest: "/lib/systemd/system/cri-docker.socket" } #9.更新ca信任、sysctl、重载systemd - name: apply sysctl and ca trust shell: | update-ca-trust extract sysctl --system systemctl daemon-reload #10.配置kubernetes yum源 - name: setup kubernetes repo yum_repository: name: kubernetes description: kubernetes repo baseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm file: kubernetes gpgcheck: no #11.区分master和node安装组件 - name: install k8s component for master(172.25.254.100) dnf: name: "{{item}}" state: present loop: - kubelet - kubeadm - kubectl when: inventory_hostname == "172.25.254.100" - name: install k8s component for worker node dnf: name: "{{item}}" state: present loop: - kubelet - kubeadm when: inventory_hostname in ["172.25.254.10","172.25.254.20"] #12.master节点配置命令补齐 - name: kubectl kubeadm bash completion lineinfile: path: /root/.bashrc line: | source <(kubectl completion bash) source <(kubeadm completion bash) when: inventory_hostname == "172.25.254.100" #13.启动服务 - name: enable and start service service: name: "{{item}}" state: restarted enabled: yes loop: - docker - systemd-modules-load.service - cri-docker - kubelet执行 playbook
[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml