1.构建部署kubernetes所需主机
|--------|----------------|-------------|-------------------|
| harbor | 172.25.254.200 | cpu1,内存2G | harbor仓库 |
| master | 172.25.254.100 | cpu2,内存>4G | maseter,k8s集群控制节点 |
| node1 | 172.25.254.10 | cpu2,内存2G | worker,k8s集群工作节点 |
| node2 | 172.25.254.20 | cpu2,内存2G | worker,k8s集群工作节点 |
2.构建harbor镜像仓库
部署docker本地仓库
cpp
[root@harbor ~]# dnf remove podman -y
正在更新 Subscription Management 软件仓库。
无法读取客户身份
[root@harbor ~]# dnf install httpd -y
正在更新 Subscription Management 软件仓库。
无法读取客户身份
[root@harbor harbor]# vim /etc/httpd/conf/httpd.conf
Listen 1111
[root@harbor docker]# systemctl enable --now httpd
[root@harbor ~]# mkdir /var/www/html/docker -p


复制这个下载地址

cs
[root@harbor ~]# cd /etc/yum.repos.d/
[root@harbor yum.repos.d]# vim docker.repo
[docker]
name = docker
baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.8/x86_64/stable/
gpgcheck = 0
[root@harbor ~]# dnf install docker-ce --downloadonly --destdir /mnt/
正在更新 Subscription Management 软件仓库。
无法读取客户身份
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# cd /var/www/html/docker/
[root@harbor docker]# dnf install createrepo
[root@harbor docker]# createrepo -v /var/www/html/docker/
[root@harbor docker]# vim /etc/yum.repos.d/docker.repo
[docker]
name = docker
baseurl = http://172.25.254.200:1111/docker/
gpgcheck = 0
[root@harbor ~]# dnf install docker-ce -y
cpp
[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter
[root@harbor ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
[root@harbor ~]# systemctl restart systemd-modules-load.service
[root@harbor ~]# sysctl --system
[root@harbor ~]# vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
--iptables=true #加入 --iptables=true
[root@harbor ~]# systemctl daemon-reload
[root@harbor ~]# systemctl enable docker.service
生成仓库密钥key
cpp
[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# openssl req -newkey rsa:4096 \
> -nodes -sha256 -keyout /data/certs/xiao.org.key \
> -addext "subjectAltName = DNS:reg.xiao.org" \
> -x509 -days 365 -out /data/certs/xiao.org.crt
Country Name (2 letter code) [XX]:cn
State or Province Name (full name) []:cq
Locality Name (eg, city) [Default City]:ps
Organization Name (eg, company) [Default Company Ltd]:k8s
Organizational Unit Name (eg, section) []:harbor
Common Name (eg, your name or your server's hostname) []:reg.xiao.org
Email Address []:
编辑harbor配置文件
cpp
[root@harbor ~]# ls
anaconda-ks.cfg harbor-offline-installer-v2.5.4.tgz
[root@harbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/
[root@harbor harbor]# ls
common.sh harbor.v2.5.4.tar.gz harbor.yml.tmpl install.sh LICENSE prepare
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml
hostname: reg.xiao.org
certificate: /data/certs/xiao.org.crt
private_key: /data/certs/xiao.org.key
harbor_admin_password: 123
[root@harbor harbor]# ./install.sh --with-chartmuseum
cpp
#编写启动脚本
[root@harbor ~]# vim /lib/systemd/system/harbor.service
[Unit]
Description=harbor with Docker Compose
Documentation=https://reg.timinglee.com/compose/
After=docker.service network-online.target
Requires=docker.service
[Service]
Type=oneshot
WorkingDirectory=/opt/harbor
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
[root@harbor ~]# systemctl daemon-reload
cpp
[root@harbor ~]# systemctl stop harbor.service
[root@harbor harbor]# docker compose down
#当关闭harbor时,会自动删除容器,开启时又自动启动容器
[root@harbor ~]# systemctl restart harbor.service
[root@harbor harbor]# docker compose ps
启动并验证
cpp
[root@harbor harbor]# mkdir /etc/docker/certs.d/reg.xiao.org/ -p
[root@harbor harbor]# cp /data/certs/xiao.org.crt /etc/docker/certs.d/reg.xiao. org/ca.crt
[root@harbor harbor]# vim /etc/hosts
[root@harbor harbor]# systemctl restart docker
[root@harbor harbor]# docker compose up -d
[root@harbor harbor]# docker login reg.xiao.org -u admin
Password:
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
Login Succeeded

#将swap注释掉,然后重启生效
cpp
[root@master ~]# vim /etc/hosts
172.25.254.100 master
172.25.254.10 node1
172.25.254.20 node2
172.25.254.200 reg.xiao.org
[root@master ~]# for i in 10 20
> do
> scp /etc/hosts root@172.25.254.$i:/etc/hosts
> done
[root@master ~]# vim /etc/fstab
#/dev/mapper/rhel-swap none swap defaults 0 0
#注释 swap
[root@master ~]# reboot #重启
3.利用Ansible部署kubernetes运行环境
cpp
[root@harbor ~]# vim /etc/yum.repos.d/docker.repo
[root@harbor ~]# cd /etc/yum.repos.d/
[root@harbor yum.repos.d]# ls
aliyun.repo redhat.repo rhel9.repo
[root@harbor yum.repos.d]# vim epel.repo
[epel]
name = epel
baseurl = https://mirrors.aliyun.com/epel/9/Everything/x86_64/
gpgcheck = 0
[root@harbor yum.repos.d]# dnf install ansible-core.x86_64 ansible.noarch -y
正在更新 Subscription Management 软件仓库。
无法读取客户身份
配置所需用户
cpp
[root@harbor yum.repos.d]# useradd devops
[root@harbor yum.repos.d]# echo 123 | passwd --stdin devops
更改用户 devops 的密码 。
passwd:所有的身份验证令牌已经成功更新。
[root@harbor yum.repos.d]# su - devops
[devops@harbor ~]$ mkdir ansible
[devops@harbor ~]$ cd ansible/
生成ansible的临时配置文件并建立运行环境
cpp
[devops@harbor ansible]$ vim inventory
[servers]
172.25.254.100
172.25.254.10
172.25.254.20
#编辑主机清单文件 `inventory`,用来定义 Ansible 要管理哪些主机。
[devops@harbor ansible]$ vim ansible.cfg
[defaults]
inventory = /home/devops/ansible/inventory # 指定主机清单文件路径
remote_user = root # 连接远端机器使用root用户
ask_pass = false # 不交互式询问ssh密码(使用密钥登录)
host_key_checking = false # 关闭主机指纹检查,避免第一次连接确认yes/no卡住
[devops@harbor ansible]$ ansible all -m ping
172.25.254.20 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
172.25.254.10 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
172.25.254.100 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
#- `ansible all`:对清单中所有主机执行任务
#- `-m ping`:调用`ping`模块,ansible 最基础连通性测试
#全部返回 pong,代表环境完全通了
cpp
#1 创建devops用户
ansible all -m user -a "name=devops state=present"
#2 设置密码123(RHEL9兼容)
ansible all -m shell -a 'echo "devops:123" | chpasswd'
#3 安全配置sudo免密,放到sudoers.d目录
ansible all -m lineinfile -a 'path=/etc/sudoers.d/devops line="devops ALL=(ALL) NOPASSWD: ALL" create=yes '
生成ansible的主配置文件并测试
cpp
[devops@harbor ansible]$ vim ansible.cfg
[defaults]
inventory = /home/devops/ansible/inventory
remote_user = devops # SSH登录远端机器先用 devops 用户登录
ask_pass = false # 不询问ssh登录密码(依赖ssh密钥)
host_key_checking = false # 关闭主机指纹确认
[privilege_escalation] # ✅权限提升(提权)配置段
become = true # 开启提权,相当于命令行加 -b
become_method = sudo # 提权方式使用 sudo
become_user = root # 提权到 root 用户
become_ask_pass = false # sudo不需要输入密码(依赖NOPASSWD sudo权限)
[devops@harbor ansible]$ ansible all -m shell -a 'whoami '
172.25.254.100 | CHANGED | rc=0 >>
root
172.25.254.10 | CHANGED | rc=0 >>
root
172.25.254.20 | CHANGED | rc=0 >>
root
利用ansible剧本部署docker
cpp
[devops@harbor ansible]$ vim docker-devops.yml
---
- name: devops k8s from ansible
hosts: all
tasks:
- name: setup docker repo
yum_repository:
name: docker
description: docker
baseurl: http://172.25.254.200:1111/docker
file: docker
gpgcheck: no
- name: install docker
dnf:
name: docker-ce
state: present
- name: setup docker.service
replace:
path: /lib/systemd/system/docker.service
regexp: 'ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock'
replace: 'ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true'
- name: setup docker registry
copy:
dest: /etc/docker/daemon.json
content: '{"registry-mirrors": ["https://reg.xiao.org"]}'
- name: setup docker certs
file:
path: /etc/docker/certs.d/reg.timinglee.org
state: directory
- name: cp certs file
copy:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
loop:
- { src: /etc/docker/certs.d/reg.xiao.org/ca.crt, dest: /etc/docker/certs.d/reg.xiao.org/ca.crt }
- { src: /etc/docker/certs.d/reg.xiao.org/ca.crt, dest: /etc/pki/ca-trust/source/anchors/ca.crt }
- name: load module br_netfilter
copy:
dest: /etc/modules-load.d/docker_mod.conf
content: br_netfilter
- name: load module sysctl
copy:
dest: /etc/sysctl.d/docker.conf
content: |
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
- name: update ca cert
shell: update-ca-trust extract && sysctl --system
- name: start services
service:
name: "{{ item }}"
state: restarted
enabled: yes
loop:
- docker
- systemd-modules-load.service
[devops@harbor ansible]$ ansible-playbook docker-devops.yml
利用ansible安装cri-dockerd

cpp
[devops@harbor ansible]$ ls
ansible.cfg cri-dockerd-0.4.4.amd64.tgz docker-devops.yml inventory
[devops@harbor ansible]$ tar zxf cri-dockerd-0.4.4.amd64.tgz
[devops@harbor ansible]$ ls
ansible.cfg cri-dockerd-0.4.4.amd64.tgz inventory
cri-dockerd docker-devops.yml
[devops@harbor ansible]$ cd cri-dockerd/
[devops@harbor cri-dockerd]$ ls
cri-dockerd cri-docker.service cri-docker.socket
[devops@harbor cri-dockerd]$ vim cri-docker.service
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=reg.xiao.org/k8s/pause:3.10.1 #加入后面这一段
cpp
[devops@harbor ansible]$ vim kubernetes-devops.yml
1 ---
2 - name: devops k8s from ansible
3 hosts: all
4 tasks:
5 - name: setup docker repo
6 yum_repository:
7 name: docker
8 description: docker
9 baseurl: http://172.25.254.200:1111/docker
10 file: docker
11 gpgcheck: no
12
13 - name: install docker
14 dnf:
15 name: docker-ce
16 state: present
17
18 - name: setup docker.service
19 replace:
20 path: /lib/systemd/system/docker.service
21 regexp: 'ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock'
22 replace: 'ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true'
23
24 - name: setup docker registry
25 copy:
26 dest: /etc/docker/daemon.json
27 content: '{"registry-mirrors": ["https://reg.xiao.org"]}'
28
29 - name: setup docker certs
30 file:
31 path: /etc/docker/certs.d/reg.xiao.org
32 state: directory
33
34 - name: cp certs file
35 copy:
36 src: "{{ item.src }}"
37 dest: "{{ item.dest }}"
38 loop:
39 - { src: /etc/docker/certs.d/reg.xiao.org/ca.crt, dest: /etc/docker/certs.d/reg.xiao.org/ca.crt }
40 - { src: /etc/docker/certs.d/reg.xiao.org/ca.crt, dest: /etc/pki/ca-trust/source/anchors/ca.crt }
41
42 - name: load module br_netfilter
43 copy:
44 dest: /etc/modules-load.d/docker_mod.conf
45 content: br_netfilter
46
47 - name: load module sysctl
48 copy:
49 dest: /etc/sysctl.d/docker.conf
50 content: |
51 net.bridge.bridge-nf-call-iptables=1
52 net.bridge.bridge-nf-call-ip6tables=1
53 net.ipv4.ip_forward=1
54
55 - name: update ca cert
56 shell: update-ca-trust extract && sysctl --system
57
58 - name: start services
59 service:
60 name: "{{ item }}"
61 state: restarted
62 enabled: yes
63 loop:
64 - docker
65 - systemd-modules-load.service
66
67 - name: setup cri-docker
68 copy:
69 src: "{{item.src}}"
70 dest: "{{item.dest}}"
71 mode: '0755'
72 loop:
73 - src: ./cri-dockerd/cri-dockerd
74 dest: /usr/bin/cri-dockerd
75 - src: ./cri-dockerd/cri-docker.service
76 dest: /lib/systemd/system/cri-docker.service
77 - src: ./cri-dockerd/cri-docker.socket
78 dest: /lib/systemd/system/cri-docker.socket
79
80 - name: update ca cert
81 shell: "update-ca-trust extract;sysctl --system;systemctl daemon-reload"
82
83 - name: start services
84 service:
85 name: "{{item}}"
86 state: restarted
87 enabled: yes
88 loop:
89 - docker
90 - systemd-modules-load.service
91 - cri-docker
[devops@harbor ansible]$ ansible-playbook docker-devops.yml
用ansible部署k8s软件
cpp
[root@harhor ~]# cd /etc/yum.repos.d/
[root@harhor yum.repos.d]# ls
aliyun.repo docker.repo epel.repo redhat.repo rhel9.repo
[root@harhor yum.repos.d]# vim kubernetes.repo
[k8s]
name = k8s
baseurl = https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm/
gpgcheck =0
[root@harhor yum.repos.d]# dnf list kubeadm --showduplicates
kubeadm.x86_64 1.35.7-150500.1.1 k8s
[root@harhor yum.repos.d]# dnf list kubeadm-1.35.7-150500.1.1 --showduplicates
正在更新 Subscription Management 软件仓库。
[root@harhor yum.repos.d]# dnf list kubeadm-1.35.7-150500.1.1 kubelet-1.35.7-150500.1.1 kubectl-1.35.7-150500.1.1 --showduplicates
正在更新 Subscription Management 软件仓库。
cpp
[devops@harbor ansible]$ vim kubernetes-devops.yml
- name: install kubernetes from ansible
hosts: all
tasks:
- name: setup k8s repo
yum_repository:
name: kubernetes
description: kubernetes
baseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm
file: kubernetes
gpgcheck: no
- name: install software in master
dnf:
name: "{{item}}"
state: present
loop:
- kubeadm-1.35.7-150500.1.1
- kubelet-1.35.7-150500.1.1
- kubectl-1.35.7-150500.1.1
when: inventory_hostname == "172.25.254.100"
- name: install software in work node
dnf:
name: "{{item}}"
state: present
loop:
- kubeadm-1.35.7-150500.1.1
- kubelet-1.35.7-150500.1.1
when: inventory_hostname == "172.25.254.10" or inventory_hostname == "172.25.254.20"
- name: setup kubectl kubeadm complication
lineinfile:
path: /root/.bashrc
line: |
source <(kubectl completion bash)
source <(kubeadm completion bash)
when: inventory_hostname == "172.25.254.100"
- name: start kubelet
service:
name: kubelet
state: started
enabled: yes
cpp
#让补齐功能生效
[root@master mnt]# source ~/.bashrc
[root@k8s-master mnt]# systemctl status kubelet.service
#注意:此时kubelet 不能启动是正常的
4.下载kubernetes集群所需镜像
cpp
查看镜像
[root@master ~]# kubeadm config images list
I0820 23:18:27.955971 66565 version.go:260] remote version is much newer: v1.36.4; falling back to: stable-1.35
registry.k8s.io/kube-apiserver:v1.35.7
registry.k8s.io/kube-controller-manager:v1.35.7
registry.k8s.io/kube-scheduler:v1.35.7
registry.k8s.io/kube-proxy:v1.35.7
registry.k8s.io/coredns/coredns:v1.13.1
registry.k8s.io/pause:3.10.1
registry.k8s.io/etcd:3.6.6-0
下载镜像
[root@master ~]# kubeadm config images pull \
> --image-repository registry.aliyuncs.com/google_containers \
> --kubernetes-version v1.35.7 \
> --cri-socket=unix:///var/run/cri-dockerd.sock
上传镜像到本地harbor
cpp
#登录认证
[root@master ~]# docker login reg.timinglee.org -u admin
Password:
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
Login Succeeded
#上传镜像, 注意你的harbor里面必须有k8s才会成功
[root@master ~]# docker images --format "{{.Repository}}:{{.Tag}}" | awk -F "/" '/google/{system("docker tag "$0" reg.timinglee.org/k8s/"$3)}'
[root@master ~]# docker images --format "{{.Repository}}:{{.Tag}}" | awk -F "/" '/timinglee/{system("docker push "$0)}'
上传成功

5.在master中初始化kubernetes集群
在master中完成集群初始化
cpp
[root@master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \
> --image-repository reg.xiao.org/k8s \
> --kubernetes-version v1.35.7 \
> --cri-socket=unix:///var/run/cri-dockerd.sock
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.25.254.100:6443 --token nc59pb.vcn5hbqak2rphr7r \
--discovery-token-ca-cert-hash sha256:97be669e0881a04f8b3970fd0c504ee2b4 0932d6ebea1a2e49f06d796cd32262
#注意:上面这一长串是自动生成的,每个人都不一样。
添加kubernets环境变量到本机
cpp
[root@master ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" > ~/.bash_profile
[root@master ~]# source ~/.bash_profile
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 2m58s v1.35.7
添加node节点到本集群
cpp
[root@node2 ~]# kubeadm join 172.25.254.100:6443 --token nc59pb.vcn5hbqak2rphr7r --discovery-token-ca-cert-hash sha256:97be669e0881a04f8b3970fd0c504ee2b40932d6ebea1a2e49f06d796cd32262 --cri-socket=unix:///var/run/cri-dockerd.sock
#在两台node主机上加入上面的命令,使他们加入master 集群
#加入成功后,在master主机上可以看到他们已经加入集群
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 10m v1.35.7
node1 NotReady <none> 43s v1.35.7
node2 NotReady <none> 5s v1.35.7
6.安装网络插件

cpp
[root@master ~]# ls
anaconda-ks.cfg kube-flannel-v0.28.9.tar.gz
[root@master ~]# tar zxf kube-flannel-v0.28.9.tar.gz
[root@master ~]# ls
anaconda-ks.cfg flannel kube-flannel-v0.28.9.tar.gz
[root@master ~]# cd flannel/
[root@master flannel]# ls
kube-flannel-v0.28.9.tar kube-flannel.yml
#导入镜像
[root@master flannel]# docker load -i kube-flannel-v0.28.9.tar
Loaded image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3
Loaded image: ghcr.io/flannel-io/flannel:v0.28.9
#上传镜像
[root@master flannel]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-f lannel3 reg.xiao.org/flannel-io/flannel-cni-plugin:v1.9.0-flannel3
[root@master flannel]# docker push reg.xiao.org/flannel-io/flannel-cni-plugin :v1.9.0-flannel3
[root@master flannel]# docker tag ghcr.io/flannel-io/flannel:v0.28.9 reg.xiao.o rg/flannel-io/flannel:v0.28.9
[root@master flannel]# docker push reg.xiao.org/flannel-io/flannel:v0.28.9
cpp
[root@master ~]# vim kube-flannel.yml
image: flannel-io/flannel:v0.28.1
image: flannel-io/flannel-cni-plugin:v1.9.0-flannel1
image: flannel-io/flannel:v0.28.1
[root@master flannel]# grep -n image: kube-flannel.yml
148: image: ghcr.io/flannel-io/flannel:v0.28.9
175: image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3
186: image: ghcr.io/flannel-io/flannel:v0.28.9
[root@master flannel]# ls
kube-flannel-v0.28.9.tar kube-flannel.yml
[root@master flannel]# vim kube-flannel.yml
[root@master flannel]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@master flannel]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 36m v1.35.7
node1 Ready <none> 26m v1.35.7
node2 Ready <none> 25m v1.35.7