K8s集群部署的方法原理

一、环境部署

|--------|----------------|--------------|------------------|
| 主机 | IP | 配置 | 角色 |
| harbor | 172.25.254.250 | cpu1、mem 1G | harbor仓库 |
| master | 172.25.254.100 | cpu4、mem>4G | master，k8s集群控制节点 |
| node1 | 172.25.254.10 | cpu2、mem 2G | k8s集群工作节点 |
| node2 | 172.25.254.20 | cpu2、mem 2G | k8s集群工作节点 |

bash 复制代码

#所有主机关闭swap
[root@harbor harbor]# systemctl disable --now swap.target
                      systemctl mask swap.target
                      sed '/swap/s/^/#/g' -i /etc/fstab

swapon -s  #如果还有输出
swapoff -a #直接关闭

二、构建harbor的镜像仓库

1.部署docker本地仓库

bash 复制代码

[root@harbor ~]# vim /etc/yum.repos.d/docker.repo

[docker]
name = docker
baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/
gpgcheck = 0

[root@harbor ~]# dnf install httpd createrepo -y
[root@harbor ~]# mkdir /var/www/html/docker/ -p
[root@harbor ~]# vim /etc/httpd/conf/httpd.conf 

Listen 4444

[root@harbor ~]# systemctl enable --now httpd
[root@harbor ~]# dnf install docker-ce --downloadonly --destdir /mnt/ -y
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# createrepo -v /var/www/html/docker/

[root@harbor ~]# vim /etc/yum.repos.d/docker.repo 

[docker]
name = docker
baseurl = http://172.25.254.250:4444/docker
gpgcheck = 0

[root@harbor ~]# dnf install docker-ce -y

2.安装docker在harbor仓库节点

bash 复制代码

[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter 

[root@harbor ~]# vim /etc/sysctl.d/docker.conf

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

[root@harbor ~]# systemctl restart systemd-modules-load.service 
[root@harbor ~]# sysctl --system

[root@harbor ~]# vim /lib/systemd/system/docker.service 

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true

[root@harbor ~]# systemctl daemon-reload 
[root@harbor ~]# systemctl enable --now docker

3.生成key

bash 复制代码

[root@harbor ~]# mkdir /data/certs -p

[root@harbor ~]# openssl req -newkey  rsa:4096 \
> -nodes -sha256 -keyout /data/certs/foldwin.org.key \
> -addext "subjectAltName = DNS:reg.foldwin.org" \
> -x509 -days 365 -out /data/certs/foldwin.org.crt

Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Shannxi
Locality Name (eg, city) [Default City]:Xi'an
Organization Name (eg, company) [Default Company Ltd]:kubernetes
Organizational Unit Name (eg, section) []:harbor
Common Name (eg, your name or your server's hostname) []:reg.foldwin.org
Email Address []:admin@foldwin.org

4.编辑harbor配置文件

bash 复制代码

[root@harbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml

  5 hostname: reg.foldwin.org
 17   certificate: /data/certs/foldwin.org.crt
 18   private_key: /data/certs/foldwin.org.key
 34 harbor_admin_password: 123

[root@harbor harbor]# ./install.sh --with-chartmuseum

[root@harbor harbor]# vim /lib/systemd/system/harbor.service

[Unit]
Description=harbor with Docker Compose
Documentation=https://reg.foldwin.com/compose/
After=docker.service network-online.target
Requires=docker.service

[Service]
Type=oneshot
WorkingDirectory=/opt/harbor
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
RemainAfterExit=yesv

[Install]
WantedBy=multi-user.target

[root@harbor harbor]# docker compose down
[root@harbor harbor]# systemctl enable --now harbor
[root@harbor harbor]# docker compose ps

5.启动并验证

bash 复制代码

[root@harbor harbor]# mkdir /etc/docker/certs.d/reg.foldwin.org/ -p
[root@harbor harbor]# cp /data/certs/foldwin.org.crt /etc/docker/certs.d/reg.foldwin.org/ca.crt
[root@harbor harbor]# vim /etc/hosts

172.25.254.250  harbor reg.foldwin.org

[root@harbor harbor]# systemctl restart docker
[root@harbor harbor]# docker compose up -d
[root@harbor harbor]# docker login reg.foldwin.org -u admin
Password: 

WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded

三、利用Ansible构建其他主机docker

1.安装ansible

bash 复制代码

[root@harbor ~]# vim /etc/yum.repos.d/epel.repo   --配置镜像仓库

[epel]
name = epel
baseurl = https://mirrors.aliyun.com/epel/9/Everything/x86_64/
gpgcheck = 0

[root@harbor ~]# dnf install ansible-core.x86_64 ansible.noarch -y --安装核心及扩展包

2.建立普通用户来运行ansible

如用root来运行ansible脚本，出现改错文件时，由于权限太高，可能会导致集群直接崩盘，所以采用普通用户，也更贴合企业真实工作环境

bash 复制代码

[root@harbor ~]# useradd devops
[root@harbor ~]# echo 123 | passwd --stdin devops
更改用户 devops 的密码 。
passwd：所有的身份验证令牌已经成功更新。
[root@harbor ~]# su - devops
[devops@harbor ~]$ mkdir ansible
[devops@harbor ~]$ cd ansible/
[devops@harbor ansible]$

3.生成ansible的临时配置文件并在每个主机上建立普通用户devops

bash 复制代码

[devops@harbor ansible]$ vim inventory

[servers]
172.25.254.100
172.25.254.20
172.25.254.10

[devops@harbor ansible]$ ssh-keygen -t ed25519
#后面全部回车跳过

[devops@harbor ansible]$ for i in 100 10 20
> do
> ssh-copy-id root@172.25.254.$i
> done

[devops@harbor ansible]$ ansible all -m user -a "name=devops state=present"

[devops@harbor ansible]$ ansible all -m shell -a 'echo "123" | passwd --stdin devops'

[devops@harbor ansible]$ ansible all -m lineinfile -a 'path=/etc/sudoers line="devops  ALL=(ALL)  NOPASSWD:ALL" insertafter=EOF'

4.生成ansible的主配置文件并测试

bash 复制代码

[defaults]
inventory = /home/devops/ansible/inventory
remote_user = devops
ask_pass = false
host_key_checking = false

[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ask_pass = false

[devops@harbor ansible]$ ansible all -m shell -a 'whoami'
172.25.254.100 | CHANGED | rc=0 >>
root
172.25.254.20 | CHANGED | rc=0 >>
root
172.25.254.10 | CHANGED | rc=0 >>
root

5.利用ansible剧本部署docker

bash 复制代码

[devops@harbor ansible]$ vim kubernetes-devops.yml 


- name: devops k8s from ansible
  hosts: all
  tasks:
  - name: setup docker repo
    yum_repository:
      name: docker
      description: docker
      baseurl: http://172.25.254.250:4444/docker
      file: docker
      gpgcheck: no

  - name: install docker
    dnf:
      name: docker-ce
      state: present

  - name: setup docker.service
    replace:
      path: /lib/systemd/system/docker.service
      regexp: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"
      replace: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock  --iptables=true"

  - name: setup docker registry
    copy:
      content: |
        {
          "registry-mirrors":["https://reg.foldwin.org"]
        }
      dest: /etc/docker/daemon.json

  - name: setup docker certs
    file:
      path: /etc/docker/certs.d/reg.foldwin.org
      state: directory

  - name: cp certs file
    copy:
      src: "{{item.src}}"
      dest: "{{item.dest}}"
    loop:
      - src: /etc/docker/certs.d/reg.foldwin.org/ca.crt
        dest: /etc/docker/certs.d/reg.foldwin.org/ca.crt

      - src: /etc/docker/certs.d/reg.foldwin.org/ca.crt
        dest: /etc/pki/ca-trust/source/anchors/ca.crt

  - name: load module
    copy:
      dest: "{{item.dest}}"
      content: "{{item.value}}"
    loop:
      - dest: /etc/modules-load.d/docker_mod.conf
        value: br_netfilter
      - dest: /etc/sysctl.d/docker.conf
        value: |
          net.bridge.bridge-nf-call-iptables = 1
          net.bridge.bridge-nf-call-ip6tables = 1
          net.ipv4.ip_forward = 1

  - name: update ca cert
    shell: "update-ca-trust extract;sysctl --system"

  - name: start services
    service:
      name: "{{item}}"
      state: restarted
      enabled: yes
    loop:
      - docker
      - systemd-modules-load.service

[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

6.利用ansible安装cri-dockerd

bash 复制代码

[devops@harbor ansible]$ tar zxf cri-dockerd-0.4.4.amd64.tgz 
[devops@harbor ansible]$ ls
ansible.cfg  cri-dockerd  cri-dockerd-0.4.4.amd64.tgz  inventory  kubernetes-devops.yml

[devops@harbor ansible]$ cd cri-dockerd/
[devops@harbor cri-dockerd]$ ls
cri-dockerd  cri-docker.service  cri-docker.socket

[devops@harbor cri-dockerd]$ vim cri-docker.service
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=reg.foldwin.org/k8s/pause:3.10.1
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

[devops@harbor ansible]$ vim kubernetes-devops.yml 
  - name: stup cri-dockerd
    copy:
      src: "{{item.src}}"
      dest: "{{item.dest}}"
      mode: '0755'
    loop:
      - src: ./cri-dockerd/cri-dockerd
        dest: /usr/bin/cri-dockerd

      - src: "./cri-dockerd/cri-docker.service"
        dest: "/lib/systemd/system/cri-docker.service"

      - src: "./cri-dockerd/cri-docker.socket"
        dest: "/lib/systemd/system/cri-docker.socket"

  - name: update ca cert
    shell: "update-ca-trust extract;sysctl --system;systemctl daemon-reload"

  - name: start services
    service:
      name: "{{item}}"
      state: restarted
      enabled: yes
    loop:
      - docker
      - systemd-modules-load.service
      - cri-docker


[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

7.利用ansible部署k8s软件

bash 复制代码

[devops@harbor ansible]$ vim kubernetes-devops.yml

- name: install kubernetes from ansible
  hosts: all
  tasks:
  - name: setup k8s repo
    yum_repository:
      name: kubernetes
      description: kubernetes
      baseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm
      file: kubernetes
      gpgcheck: no

  - name: install software in master
    dnf:
      name: "{{item}}"
      state: present
    loop:
      - kubeadm-1.35.7-150500.1.1
      - kubelet-1.35.7-150500.1.1
      - kubectl-1.35.7-150500.1.1
    when: inventory_hostname == "172.25.254.100"

  - name: install software in work node
    dnf:
      name: "{{item}}"
      state: present
    loop:
      - kubeadm-1.35.7-150500.1.1
      - kubelet-1.35.7-150500.1.1
    when: inventory_hostname == "172.25.254.10" or inventory_hostname == "172.25.254.20"
  
  - name: setup kubectl kubeadm complication
    lineinfile:
      path: /root/.bashrc
      line: |
        source <(kubectl completion bash)
        source <(kubeadm completion bash)
    when: inventory_hostname == "172.25.254.100"

  - name: start kubelet
    service:
      name: kubelet
      state: started
      enabled: yes

[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

[root@master ~]# source ~/.bashrc
[root@master ~]# kubectl 

alpha          (Commands for features in alpha)
annotate       (更新一个资源的注解)
api-resources  (Print the supported API resources on the server)
api-versions   (Print the supported API versions on the server, in the form of "group/version")
apply          (Apply a configuration to a resource by file name or stdin)
attach         (挂接到一个运行中的容器)
auth           (Inspect authorization)
autoscale      (Auto-scale a deployment, replica set, stateful set, or replication controller)
certificate    (Modify certificate resources)
cluster-info   (Display cluster information)
completion     (Output shell completion code for the specified shell (bash, zsh, fish, or powershell))
config         (修改 kubeconfig 文件)
cordon         (标记节点为不可调度)
cp             (Copy files and directories to and from containers)
create         (Create a resource from a file or from stdin)
debug          (Create debugging sessions for troubleshooting workloads and nodes)
delete         (Delete resources by file names, stdin, resources and names, or by resources and label...)
describe       (显示特定资源或资源组的详细信息)
diff           (Diff the live version against a would-be applied version)
drain          (清空节点以准备维护)
edit           (编辑服务器上的资源)
events         (List events)
exec           (在某个容器中执行一个命令)
explain        (Get documentation for a resource)
expose         (Take a replication controller, service, deployment or pod and expose it as a new Kube...)
get            (显示一个或多个资源)
help           (Help about any command)
kustomize      (Build a kustomization target from a directory or URL)
label          (更新某资源上的标签)
logs           (打印 Pod 中容器的日志)
options        (输出所有命令的层级关系)
patch          (Update fields of a resource)
plugin         (Provides utilities for interacting with plugins)
port-forward   (将一个或多个本地端口转发到某个 Pod)
proxy          (运行一个指向 Kubernetes API 服务器的代理)
replace        (Replace a resource by file name or stdin)
rollout        (Manage the rollout of a resource)
run            (在集群上运行特定镜像)
scale          (Set a new size for a deployment, replica set, or replication controller)
set            (为对象设置指定特性)
taint          (更新一个或者多个节点上的污点)
top            (Display resource (CPU/memory) usage)
uncordon       (标记节点为可调度)
version        (输出客户端和服务端的版本信息)
wait           (Wait for a specific condition on one or many resources)

[root@master ~]# kubeadm 
certs       config      init        kubeconfig  token       version     
completion  help        join        reset       upgrade 

[root@master ~]# systemctl status kubelet.service

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Sat 2026-08-22 20:42:57 CST; 4s ago
       Docs: https://kubernetes.io/docs/
    Process: 51855 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_K>
   Main PID: 51855 (code=exited, status=1/FAILURE)
        CPU: 20ms

8月 22 20:42:57 master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
8月 22 20:42:57 master systemd[1]: kubelet.service: Failed with result 'exit-code'.

四、完成初始化kubernetes集群

1.下载kubernetes集群所需镜像

bash 复制代码

#查看镜像
[root@master ~]# kubeadm config images list
I0822 20:52:23.434259   52150 version.go:260] remote version is much newer: v1.36.4; falling back to: stable-1.35
registry.k8s.io/kube-apiserver:v1.35.8
registry.k8s.io/kube-controller-manager:v1.35.8
registry.k8s.io/kube-scheduler:v1.35.8
registry.k8s.io/kube-proxy:v1.35.8
registry.k8s.io/coredns/coredns:v1.13.1
registry.k8s.io/pause:3.10.1
registry.k8s.io/etcd:3.6.6-0

#下载镜像
[root@master ~]# kubeadm config images pull \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.35.7 \
--cri-socket=unix:///var/run/cri-dockerd.sock
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.35.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.35.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.35.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.35.3
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.13.1
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.10.1
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.6.6-0

#上传镜像到本地
[root@master ~]# docker login reg.foldwin.org -u admin
Password: 

WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded

[root@master ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/google/{system("docker tag "$0" reg.foldwin.org/k8s/"$3)}'
[root@master ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/foldwin/{system("docker push "$0)}'

2.在master中初始化kubernetes集

bash 复制代码

[root@master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \
--image-repository reg.timinglee.org/k8s \
--kubernetes-version v1.35.7 \
--cri-socket=unix:///var/run/cri-dockerd.sock

#每个人不一样，其他主机加入本集群的凭证
kubeadm join 172.25.254.100:6443 --token 8ug1qv.syb84mf0fckkwge3 \

	--discovery-token-ca-cert-hash sha256:362b85dd22ed50a7cf62b944acb77a33b4b200021f293f7850d843631d976fe7 

#如果忘记
[root@master ~]#  kubeadm token create --print-join-command
kubeadm join 172.25.254.100:6443 --token 71whpo.ieo4x5itivtpahyy --discovery-token-ca-cert-hash sha256:362b85dd22ed50a7cf62b944acb77a33b4b200021f293f7850d843631d976fe7 

#如果初始化出问题
[root@master ~]# kubeadm reset  --cri-socket=unix:///var/run/cri-dockerd.sock #可以重置集群设定


#添加kubernets环境变量到本机
[root@master ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" > ~/.bash_profile
[root@master ~]# source ~/.bash_profile
[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES           AGE     VERSION
master   NotReady   control-plane   2m43s   v1.35.7

#添加node节点到本集群
[root@node1 ~]# kubeadm join 172.25.254.100:6443 --token 71whpo.ieo4x5itivtpahyy --discovery-token-ca-cert-hash sha256:362b85dd22ed50a7cf62b944acb77a33b4b200021f293f7850d843631d976fe7 --cri-socket=unix:///var/run/cri-dockerd.sock

#查看，因为没有装网络插件所以还是notready
[root@master ~]#  kubectl get nodes     
NAME     STATUS     ROLES           AGE    VERSION
master   NotReady   control-plane   26m    v1.35.7
node1    NotReady   <none>          2m1s   v1.35.7
node2    NotReady   <none>          113s   v1.35.7

3.安装网络插件

bash 复制代码

[root@master ~]# tar zxf kube-flannel-v0.28.9.tar.gz
[root@master ~]# ls
flannel  kube-flannel-v0.28.9.tar.gz
[root@master ~]# cd flannel/

[root@master flannel]# docker load  -i kube-flannel-v0.28.9.tar
Loaded image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3
Loaded image: ghcr.io/flannel-io/flannel:v0.28.9


[root@master ~]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 reg.foldwin.org/flannel-io/flannel-cni-plugin:v1.9.1-flannel3
[root@master ~]# docker push  reg.foldwin.org/flannel-io/flannel-cni-plugin:v1.9.1-flannel3
The push refers to repository [reg.foldwin.org/flannel-io/flannel-cni-plugin]
2c8aa52d4746: Pushed
5aa68bbbc67e: Pushed
v1.9.0-flannel1: digest: sha256:b3d30c221113b30fea3e8a7fccb145e929b097d0319b9eeb6b5a591b10b5c671 size: 739

[root@master ~]# docker tag ghcr.io/flannel-io/flannel:v0.28.9 reg.foldwin.org/flannel-io/flannel:v0.28.9
[root@master ~]# docker push reg.foldwin.org/flannel-io/flannel:v0.28.9
The push refers to repository [reg.foldwin.org/flannel-io/flannel]
        
[root@master flannel]# vim kube-flannel.yml  
148:        image: flannel-io/flannel:v0.28.9
175:        image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3
186:        image: flannel-io/flannel:v0.28.9


[root@master ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

#测试
[root@master ~]# kubectl get  nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   13m     v1.35.7
node1    Ready    <none>          9m44s   v1.35.7
node2    Ready    <none>          9m36s   v1.35.7