⼀、编排分类
单机容器编排: docker-compose
容器集群编排: docker swarm、mesos+marathon、kubernetes
应⽤编排: ansible(模块，剧本，⻆⾊)
⼆、系统管理进化史
1. 传统部署时代
早期，各个组织是在物理服务器上运⾏应⽤程序。 由于⽆法限制在
物理服务器中运⾏的应⽤程序资源使⽤，因此会导致资源分配问
题。
例如，如果在同⼀台物理服务器上运⾏多个应⽤程序， 则可能会出
现⼀个应⽤程序占⽤⼤部分资源的情况，⽽导致其他应⽤程序的性
能下降。
⽽⼀种解决⽅案是将每个应⽤程序都运⾏在不同的物理服务器上，
但是当某个应⽤程序资源利⽤率不⾼时，剩余资源⽆法被分配给其
他应⽤程序，⽽且维护许多物理服务器的成本很⾼。
2. 虚拟化部署时代⽽后，虚拟化技术被引⼊了。虚拟化技术允许在单个物理服务器的
CPU 上运⾏多台虚拟机(VirtureMachine)。 虚拟化能使应⽤程序在不
同 VM 之间被彼此隔离，且能提供⼀定程度的安全性， 因为⼀个应
⽤程序的信息不能被另⼀应⽤程序随意访问。
虚拟化技术能够更好地利⽤物理服务器的资源，并且因为可轻松地
添加或更新应⽤程序，⽽因此可以具有更⾼的可扩缩性，以及降低
硬件成本等等的好处。通过虚拟化，你可以将⼀组物理资源呈现为
可丢弃的虚拟机集群。
每个 VM 是⼀台完整的计算机，在虚拟化硬件之上运⾏所有组件，
包括其⾃⼰的操作系统，所以由此看来，虚拟机技术占⽤资源量
⼤，在⼀台主机上最多运⾏⼗⼏台，效率不⾼。
3. 容器部署时代
容器，衍⽣于虚拟技术，但具有更宽松的隔离特性，使容器可以在
共享操作系统的同时，还能保持其轻量级的特点。⽽且每个容器都
具有⾃⼰的⽂件系统、CPU、内存、进程空间等，且具有最良好的
可移植性和平台兼容性。
**敏捷应⽤程序的创建和部署：**与使⽤ VM 镜像相⽐，提⾼了容
器镜像创建的简便性和效率
持续开发、集成和部署：通过快速简单的回滚(由于镜像不可变
性)，提供可靠且频繁的容器镜像构建和部署。**关注开发与运维的分离：**在构建、发布时创建应⽤程序容器镜
像，⽽不是在部署时，从⽽将应⽤程序与基础架构分离。
**可观察性：**不可以显示OS 级别的信息和指标，还可以显示应⽤
程序的运⾏状况和其他指标信。
**跨开发、测试和⽣产的环境⼀致性：**在笔记本计算机上也可以和
在云中运⾏⼀样的应⽤程序。
**跨云和操作系统发⾏版本的可移植性：**可在 Ubuntu、RHEL、
CoreOS、本地、GoogleKubernetes Engine 和其他任何地⽅运
⾏。
**以应⽤程序为中⼼的管理：**提⾼抽象级别，从在虚拟硬件上运⾏
OS 到使⽤逻辑资源在OS 上运⾏应⽤程序。
**松散耦合、分布式、弹性、解放的微服务：**应⽤程序被分解成较
⼩的独⽴部分， 并且可以动态部署和管理-⽽不是在⼀台⼤型单
机上整体运⾏。
**资源隔离：**可预测的应⽤程序性能。
资源利⽤：⾼效率和⾼密度。
三、Kubernetes 简介
Kubernetes 缩写：K8S，k 和 s 之间有⼋个字符，所以因此得名。
Kubernetes 由 google 的 Brog 系统作为原型，后经 Go 语⾔延⽤
Brog 的思路重写，并捐献给 CNCF 基⾦会开源。Kubernetes 是⼀个可移植的、可扩展的开源平台，⽤于管理容器化
的⼯作负载和服务，可促进声明式配置和⾃动化。
**官⽹：**https://kubernetes.io
**Github：**https://github.com/kubernetes/kubernetes
四、Kubernetes 功能
Kubernetes 的⽬标是让部署容器化的应⽤简单并且⾼效，提供了应
⽤部署，规划，更新，维护的⼀种机制。
Kubernetes 在 Docker 等容器技术的基础上，为容器化的应⽤提供
部署运⾏、资源调度、服务发现和动态伸缩等⼀系列完整功能，提
⾼了⼤规模容器集群管理的便捷性。
主要功能：
- 使⽤ Docker 等容器技术对应⽤程序包装(package)、实例化
(instantiate) 、运⾏(run)。 - 以集群的⽅式运⾏、管理跨机器的容器，解决 Docker 跨机器容
器之间的通讯问题。 - K8S 的⾃我修复机制使得容器集群总是运⾏在⽤户期望的状
态。
五、Kubernetes 特性五、Kubernetes 特性 - ⾃动装箱：Kubernetes可以根据所需资源和其他限制条件智能
地定位容器，⽽不会影响可⽤性。 - **弹性伸缩：**使⽤命令、UI 或者基于 CPU 使⽤情况⾃动快速扩容
和缩容应⽤程序实例，保证应⽤业务⾼峰并发时的⾼可⽤性；业
务低峰时回收资源，以最⼩成本运⾏服务。 - **⾃我修复：**在节点故障时重新启动失败的容器，替换和重新部
署，保证预期的副本数量；杀死健康检查失败的容器，并且在未
准备好之前不会处理客户端请求，确保线上服务不中断。 - **服务发现和负载均衡：**K8S为多个容器提供⼀个统⼀访问⼊⼝
（内部IP地址和⼀个DNS名称），并且负载均衡关联的所有容
器，使得⽤户⽆需考虑容器 IP 问题。 - **⾃动发布（默认滚动发布模式）和回滚：**K8S采⽤滚动策略更新
应⽤，⼀个更新⼀个Pod，⽽不是同时删除所有的Pod，如果更
新过程中出现问题，将回滚更改，确保升级不收影响业务。 - **集中化配置管理和密钥管理：**管理机密数据和应⽤程序配置，⽽
不需要把敏感数据暴露在镜像⾥，提⾼敏感数据安全性，并可以
将⼀些常⽤的配置存储在K8S中，⽅便应⽤程序使⽤。 - 存储编排：⽀持外挂存储并对外挂存储资源进⾏编排，挂载外部
存储系统，⽆论是来⾃本地存储，公有云（如：AWS），还是
⽹络存储（如：NFS、Glusterfs、Ceph）都作为集群资源的⼀
部分使⽤，极⼤提⾼存储使⽤灵活性。8. **任务批量处理运⾏：**提供⼀次性任务，定时任务，满⾜批量数据
处理和分析的场景。
六、K8S 解决裸跑 Docker 的痛点 - 单机使⽤，⽆法有效集群。
- 随着容器数量的上升，管理成本攀升。
- 没有有效的容灾、⾃愈机制。
- 没有预设编排模板，⽆法实现快速、⼤规模容器调度。
- 没有统⼀的配置管理中⼼⼯具。
- 没有容器⽣命周期的管理⼯具。
- 没有图形化运维管理⼯具。
七、Kubernetes 架构
K8S 属于主从设备模型（Mater-Slave架构），由 Master 节点负责
集群的调度、管理和运维（分配活的），Slave 节点是运算⼯作负载
节点（⼲活的），被称为 Worker Node 节点。
Master 需要占据⼀个独⽴服务器运⾏，因为其是整个集群的"⼤
脑"，⼀旦宕机或不可⽤，那么所有控制命令都将失效，可对主节点
进⾏⾼可⽤配置。当 Worker Node 节点宕机时，其上的⼯作负载会被 Master ⾃动转
移到其他节点上。
1. Master 节点组件
API Server
*整个集群的控制中枢，提供集群中各个模块之间的数据交换*，并将
集群状态和信息存储到分布式键-值(key-value)存储系统 Etcd 集群
中。
同时它也是集群管理、资源配额、提供完备的集群安全机制的⼊
⼝，为集群各类资源对象提供增删改查以及 watch 的 REST API 接
⼝。Controller-manager
集群状态管理器，以保证Pod 或其他资源达到期望值*。*当集群中
某个 Pod 的副本数或其他资源因故障和错误导致⽆法正常运⾏，没
有达到设定的值时，Controller Manager 会尝试⾃动修复并使其达到
期望状态。
Scheduler
*集群 Pod 的调度中⼼，主要是通过调度算法将 Pod 分配到最佳的
Node 节点*，它通过APIServer 监听所有 Pod 的状态，⼀旦发现新
的未被调度到任何 Node 节点的Pod(PodSpec.NodeName为空)，就
会根据⼀系列策略选择最佳节点进⾏调度。
Etcd
*⽤于可靠地存储集群的配置数据，是⼀种持久性、轻量型、分布式**
的键-值 (key-value) 数据存储组件*，作为Kubernetes集群的持久化
存储系统。
注：⽣产环境建议存放到单个的SSD硬盘，并且做好冗余。
2. Work Node 节点
Kubelet
*负责与 Master 通信协作，管理该节点上的 Pod，对容器进⾏健康
**检查及监控，同时负责上报节点和节点上⾯ Pod 的状态。***Kube-proxy
*运⾏在每个 node 节点上，实现 pod 的⽹络代理，维护⽹络规则和
四层负载均衡规则*，负责写⼊规则到 iptables 或 ipvs 实现服务映射
访问。
Runtime
*负责容器的管理* (新版本 K8S 使⽤的是 Containerd)。
CoreDNS
⽤于Kubernetes 集群内部Service的解析，可以让 Pod 把
Service 名称解析成 Service 的 IP 然后通过 Service 的 IP 地址进⾏
连接到对应的应⽤上。
Calico
符合 CNI 标准的⼀个⽹络插件，它*负责给每个 Pod 分配⼀个不会
重复的 IP，并且把每个节点当做⼀各"路由器"*，这样⼀个节点的
Pod 就可以通过 IP 地址访问到其他节点的 Pod。
Docker
*运⾏容器，负责本机的容器创建和管理⼯作。*
⼋、Pod 概念⼋、Pod 概念
Pod 是 Kubernetes 中的基本构建块，它代表⼀个或⼀组相互关联的
容器。Pod 是Kubernetes 的最⼩部署单元，可以包含⼀个或多个容
器，这些容器共享存储、⽹络和运⾏配置。
容器之间可以使⽤ localhost:port 相互访问，可以使⽤ volume 等实
现数据共享。根据 Docker 的构造，Pod 可被建模为⼀组具有共享命
令空间、卷、IP 地址和 Port 端⼝的容器。
Pod 的主要特点包括： - 共享存储：Pod 中的所有容器都可以访问同⼀个存储卷
（Persistent Volume），实现数据共享。 - 共享⽹络：Pod 中的所有容器都共享同⼀个⽹络命名空间，可
以相互通信。 - 共享运⾏配置：Pod 中的所有容器都共享相同的运⾏配置，例
如容器的启动参数、环境变量等。
Pause 容器：
Pod 的⽗容器，它主要负责僵⼫进程的回收管理，同时通过 Pause
容器可以使同⼀个 Pod ⾥⾯的不同容器进⾏共享存储、⽹络、
PID、IPC等。
九、Kubernetes ⼯作流程九、Kubernetes ⼯作流程 - 运维⼈员使⽤ kubectl 命令⼯具向 API Server 发送请求，API
Server 接收请求后写⼊到 Etcd 中。 - API Server 让 Controller-manager 按照预设模板去创建 Pod。
- Controller-manager 通过 API Server 读取 Etcd 中⽤户的预设信
息，再通过 API Server 找到 Scheduler，为新创建的 Pod 选择
最合适的 Node ⼯作负载节点。 - Scheduler 通过 API Server 在 Etcd 找到存储的 Node 节点元信
息、剩余资源等，⽤预选和优选策略选择最优的 Node 节点。 - Scheduler 确定 Node 节点后，通过 API Server 交给这个 Node
节点上的 Kubelet 进⾏ Pod 资源的创建。 - Kubelet 调⽤容器引擎交互创建 Pod，同时将 Pod 监控信息通
过 API Server 存储到 Etcd 中。 - 当⽤户访问时，通过 Kube-proxy 负载、转发，访问相应的
Pod。 - 注：决定创建 Pod 清单的是 Controller-manager 控制器，
Kubelet 和容器引擎只是⼲活的。⼗、K8S 创建 Pod 流程
1. 详细流程
⾸先 Kubectl 创建⼀个 Pod，在提交时转化为 json。
再经过 auth 认证(鉴权)，然后传递给 API Server 进⾏处理。
API Server 将请求信息存储到 Etcd 中。Scheduler 和 Controller-manager 会监听 API Server 的请求。
在 Scheduler 和 Controller-manager 监听到请求后，Scheduler
会提交给API Server⼀个list清单 ------ 包含的是获取node节点信
息。
当 API Server 从 Etcd 获取后端 Node 节点信息后，会同时被
Scheduler 监听到，然后 Scheduler 进⾏优选打分制，最后将评
估结果传递给 API Server。
⽽后，API Server 会提交清单给对应节点的 Kubelet（代理）。
Kubelet 代理通过 K8S 与容器的接⼝ (例如 containerd) 进⾏交
互，假设是 docker 容器，那么此时 kubelet 就会通过
dockershim 以及 runc 接⼝与 docker 的守护进程docker-server
进⾏交互，来创建对应的容器，再⽣成对应的 Pod。
Kubelet 同时会借助 Metric Server 收集本节点的所有状态信
息，然后提交给 API Server。
最后 API Server 将该节点的容器和 Pod 信息存储到 Etcd 中。
2. 简化流程
⽤户通过 kubectl 或其他 API 客户端提交 Pod Spec 给 API
Server。
API Server 尝试将 Pod 对象的相关信息存⼊ etcd 中，待写⼊操
作执⾏完成，API Server 即会返回确认信息⾄客户端。Controller 通过 API Server 的 Watch 接⼝发现新的 Pod，将
Pod 加⼊到任务队列，并启动 Pod Control 机制创建与之对应的
Pod。
所有 Controler 正常后，将结果存⼊ etcd。
Scheduler 通过 API Server 的 Watch 接⼝监测发现新的 Pod，
经过给主机打分之后，让 Pod 调度到符合要求的 Node 节点，
并将结果存⼊到 etcd 中。
Kubelet 每隔⼀段时间向 API Server 通过 Node name 获取⾃身
Node 上要运⾏的 Pod 并通过与⾃身缓存⽐较，来创建新
Pod。
Containerd 启动容器。
最后 API Server 将本节点的容器和Pod信息存储到etcd。
系统
主机名
IP 地址
⻆⾊
配置
Rocky8.7 k8s
master
192.168.15.11 master
节点
4 核 4G 内存
50G 硬盘
Rocky8.7 k8s
node01
192.168.15.22 work 节
点
4 核 4G 内存
50G 硬盘
Rocky8.7 k8s
node02
192.168.15.33 work 节
点
4 核 4G 内存
50G 硬盘
配置信息
备注
Docker 版本
24.10
Pod ⽹段
172.16.0.0/16
Service ⽹段
10.96.0.0/16
⼀、安装环境
1. 安装说明
本次以⼆进制⽅式安装⾼可⽤ k8s 1.28.0 版本，但在⽣产环境中，
建议使⽤⼩版本⼤于 5 的 Kubernetes 版本，⽐如 1.19.5 以后。
2. 系统环境
3. ⽹络及版本环境注：宿主机⽹段、Pod ⽹段、Service ⽹段不能重复，服务器 IP 地
址不能设置为 DHCP，需配置为静态 IP。
⼆、前期准备
1. 配置主机映射
2. 配置 yum 源
root@k8s-master \~# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain
localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain
localhost6 localhost6.localdomain6
192.168.15.11 k8s-master
192.168.15.11 k8s-master-lb # 若有⾼可⽤主机，这⾥为
另⼀个master的IP
192.168.15.22 k8s-node01
192.168.15.33 k8s-node02
root@k8s-master \~# cd /etc/yum.repos.d/
root@k8s-master yum.repos.d# mkdir bak
root@k8s-master yum.repos.d# mv Rocky* bak
root@k8s-master yum.repos.d# mv local.repo bakroot@k8s-master yum.repos.d# ls
aliyunbak bak
root@k8s-master yum.repos.d# vim docker-ce.repo
docker软件源
docker-ce-stable
name=Docker CE Stable - basearch
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-stable-debuginfo
name=Docker CE Stable - Debuginfo basearch
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-stable-source
name=Docker CE Stable - Sources
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/$releasever/source/stable
enabled=0gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-test
name=Docker CE Test - basearch
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-test-debuginfo
name=Docker CE Test - Debuginfo basearch
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/debug-$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-test-source
name=Docker CE Test - Sources
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/$releasever/source/test
enabled=0gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-nightly
name=Docker CE Nightly - basearch
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-nightly-debuginfo
name=Docker CE Nightly - Debuginfo basearch
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/debug-
$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
docker-ce-nightly-source
name=Docker CE Nightly - Sources
baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/$releasever/source/nightlyenabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
root@k8s-master yum.repos.d# vim Rocky
BaseOS.repo # 阿⾥云软件源
baseos
name=Rocky Linux releasever - BaseOS
#mirrorlist=https://mirrors.rockylinux.org/mirrorl
ist?arch=basearch&repo=BaseOS-releasever
baseurl=https://mirrors.aliyun.com/rockylinux/rel
easever/BaseOS/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
rockyofficial
root@k8s-master yum.repos.d# vim Rocky
AppStream.repo # 阿⾥云软件源
appstream
name=Rocky Linux releasever - AppStream
#mirrorlist=https://mirrors.rockylinux.org/mirrorl
ist?arch=basearch&repo=AppStream-releasever
baseurl=https://mirrors.aliyun.com/rockylinux/rel
easever/AppStream/$basearch/os/
gpgcheck=1enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
rockyofficial
root@k8s-master yum.repos.d# vim kubernetes.repo
K8S软件源
kubernetes
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/
repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/d
oc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm
package-key.gpg
root@k8s-master yum.repos.d# ls
aliyunbak docker-ce.repo Rocky-AppStream.repo
bak kubernetes.repo Rocky-BaseOS.repo
root@k8s-master yum.repos.d# yum clean all #
清除yum缓存
35 ⽂件已删除
root@k8s-master yum.repos.d# yum makecache #
建⽴yum元数据缓存
Rocky Linux 8 - AppStream
5.2 MB/s | 9.6 MB 00:01 3. 安装必备⼯具
Rocky Linux 8 - BaseOS
2.6 MB/s | 3.9 MB 00:01
Docker CE Stable - x86_64
54 kB/s | 52 kB 00:00
Kubernetes
193 kB/s | 182 kB 00:00
元数据缓存已建⽴。
root@k8s-master yum.repos.d# cd
root@k8s-master \~# yum install wget jq psmisc
vim net-tools telnet yum-utils device-mapper
persistent-data lvm2 git -y
......省略部分内容......
已安装:
git-2.39.3-1.el8_8.x86_64 git
core-2.39.3-1.el8_8.x86_64
git-core-doc-2.39.3-1.el8_8.noarch perl
Error-1:0.17025-2.el8.noarch
perl-Git-2.39.3-1.el8_8.noarch perl
TermReadKey-2.37-7.el8.x86_64
telnet-1:0.17-76.el8.x86_64 yum
utils-4.0.21-23.el8.noarch
完毕！4. 关闭安全及 swap 分区
5. 同步时间
root@k8s-master \~# systemctl disable --now
firewalld
root@k8s-master \~# systemctl disable --now
dnsmasq
root@k8s-master \~# setenforce 0
root@k8s-master \~# sed -i
's#SELINUX=enforcing#SELINUX=disabled#g'
/etc/sysconfig/selinux
root@k8s-master \~# sed -i
's#SELINUX=enforcing#SELINUX=disabled#g'
/etc/selinux/config
root@k8s-master \~# swapoff -a && sysctl -w
vm.swappiness=0
vm.swappiness = 0
root@k8s-master \~# sed -ri '/^\^#*swap/s@^@#@'
/etc/fstab6. 配置 limit
root@k8s-master \~# rpm -ivh
https://mirrors.wlnmp.com/rockylinux/wlnmp
release-rocky-8.noarch.rpm
获取https://mirrors.wlnmp.com/rockylinux/wlnmp
release-rocky-8.noarch.rpm
Verifying...
################################# 100%
准备中...
################################# 100%
正在升级/安装...
1:wlnmp-release-rocky-1-1
################################# 100%
root@k8s-master \~# yum -y install wntp
root@k8s-master \~# ntpdate time2.aliyun.com
19 Dec 21:02:09 ntpdate33790: adjust time server
203.107.6.88 offset -0.001396 sec
root@k8s-master \~# crontab -e
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com7. 配置免密登录
8. 安装 k8s ⾼可⽤性 Git 仓库
root@k8s-master \~# ulimit -SHn 65535 # 单个进程
可以打开的⽂件数量将被限制为 65535
root@k8s-master \~# vim /etc/security/limits.conf
末尾添加如下内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
root@k8s-master \~# ssh-keygen -t rsa
遵循默认配置，⼀路回⻋即可
root@k8s-master \~# for i in k8s-node01 k8s
node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
按照提示输⼊yes和密码9. 升级系统并重启
更新系统但不包括内核
三、配置内核模块
root@k8s-master \~# cd /root/ ; git clone
https://gitee.com/dukuan/k8s-ha-install.git
在 /root/ ⽬录下克隆⼀个名为 k8s-ha-install.git 的
Git 仓库
正克隆到 'k8s-ha-install'...
remote: Enumerating objects: 882, done.
remote: Counting objects: 100% (208/208), done.
remote: Compressing objects: 100% (130/130), done.
remote: Total 882 (delta 92), reused 145 (delta
52), pack-reused 674
接收对象中: 100% (882/882), 19.71 MiB | 2.82 MiB/s,
完成.
处理 delta 中: 100% (356/356), 完成.
root@k8s-master \~# yum update -y --
exclude=kernel* --nobest && reboot1. 配置 ipvs 模块
root@k8s-master \~# yum install ipvsadm ipset
sysstat conntrack libseccomp -y
root@k8s-master \~# modprobe -- ip_vs #
使⽤ modprobe 命令加载内核模块，核⼼ IPVS 模块。
root@k8s-master \~# modprobe -- ip_vs_rr #
IPVS 负载均衡算法 rr
root@k8s-master \~# modprobe -- ip_vs_wrr #
IPVS 负载均衡算法 wrr
root@k8s-master \~# modprobe -- ip_vs_sh #
⽤于源端负载均衡的模块
root@k8s-master \~# modprobe -- nf_conntrack #
⽤于⽹络流量过滤和跟踪的模块
root@k8s-master \~# vim /etc/modules
load.d/ipvs.conf
在系统启动时加载下列 IPVS 和相关功能所需的模块
ip_vs # 负载均衡模块
ip_vs_lc # ⽤于实现基于连接数量的负载均衡算法
ip_vs_wlc # ⽤于实现带权重的最少连接算法的模块
ip_vs_rr # 负载均衡rr算法模块
ip_vs_wrr # 负载均衡wrr算法模块
ip_vs_lblc # 负载均衡算法，它结合了最少连接（LC）算法
和基于偏置的轮询（Round Robin with Bias）算法
ip_vs_lblcr # ⽤于实现基于链路层拥塞状况的最少连接负载
调度算法的模块ip_vs_dh # ⽤于实现基于散列（Hashing）的负载均衡算
法的模块
ip_vs_sh # ⽤于源端负载均衡的模块
ip_vs_fo # ⽤于实现基于本地服务的负载均衡算法的模块
ip_vs_nq # ⽤于实现NQ算法的模块
ip_vs_sed # ⽤于实现随机早期检测（Random Early
Detection）算法的模块
ip_vs_ftp # ⽤于实现FTP服务的负载均衡模块
ip_vs_sh
nf_conntrack # ⽤于跟踪⽹络连接的状态的模块
ip_tables # ⽤于管理防护墙的机制
ip_set # ⽤于创建和管理IP集合的模块
xt_set # ⽤于处理IP数据包集合的模块，提供了与
iptables等⽹络⼯具的接⼝
ipt_set # ⽤于处理iptables规则集合的模块
ipt_rpfilter # ⽤于实现路由反向路径过滤的模块
ipt_REJECT # iptables模块之⼀，⽤于将不符合规则的数据
包拒绝，并返回特定的错误码
ipip # ⽤于实现IP隧道功能的模块，使得数据可以在
两个⽹络之间进⾏传输
root@k8s-master \~# systemctl enable --now
systemd-modules-load.service # 开机⾃启systemd默
认提供的⽹络管理服务
The unit files have no installation config
(WantedBy, RequiredBy, Also, Alias
settings in the Install section, and
DefaultInstance for template units).This means they are not meant to be enabled using
systemctl.
Possible reasons for having this kind of units
are:
- A unit may be statically enabled by being
symlinked from another unit's
.wants/ or .requires/ directory. - A unit's purpose may be to act as a helper for
some other unit which has
a requirement dependency on it. - A unit may be started when needed via
activation (socket, path, timer,
D-Bus, udev, scripted systemctl call, ...). - In case of template units, the unit is meant to
be enabled with some
instance name specified.
root@k8s-master \~# lsmod | grep -e ip_vs -e
nf_conntrack # 查看已写⼊加载的模块
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 172032 6
ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 172032 4
xt_conntrack,nf_nat,ipt_MASQUERADE,ip_vs
nf_defrag_ipv6 20480 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack2. 配置 k8s 内核
libcrc32c 16384 5
nf_conntrack,nf_nat,nf_tables,xfs,ip_vs
root@k8s-master \~# vim /etc/sysctl.d/k8s.conf
写⼊k8s所需内核模块
net.bridge.bridge-nf-call-iptables = 1 # 控制⽹络
桥接与iptables之间的⽹络转发⾏为
net.bridge.bridge-nf-call-ip6tables = 1 # ⽤于控制
⽹络桥接（bridge）的IP6tables过滤规则。当该参数设置为1
时，表示启⽤对⽹络桥接的IP6tables过滤规则
fs.may_detach_mounts = 1 # ⽤于控制⽂件系统是否允许
分离挂载，1表示允许
net.ipv4.conf.all.route_localnet = 1 # 允许本地⽹
络上的路由。设置为1表示允许，设置为0表示禁⽌。
vm.overcommit_memory=1 # 控制内存分配策略。设置为1表
示允许内存过量分配，设置为0表示不允许。
vm.panic_on_oom=0 # 决定当系统遇到内存不⾜（OOM）时是
否产⽣panic。设置为0表示不产⽣panic，设置为1表示产⽣
panic。
fs.inotify.max_user_watches=89100 # inotify可以监
视的⽂件和⽬录的最⼤数量。
fs.file-max=52706963 # 系统级别的⽂件描述符的最⼤数
量。fs.nr_open=52706963 # 单个进程可以打开的⽂件描述符
的最⼤数量。
net.netfilter.nf_conntrack_max=2310720 # ⽹络连接
跟踪表的最⼤⼤⼩。
net.ipv4.tcp_keepalive_time = 600 # TCP保活机制发
送探测包的间隔时间（秒）。
net.ipv4.tcp_keepalive_probes = 3 # TCP保活机制发
送探测包的最⼤次数。
net.ipv4.tcp_keepalive_intvl =15 # TCP保活机制在
发送下⼀个探测包之前等待响应的时间（秒）。
net.ipv4.tcp_max_tw_buckets = 36000 # TCP
TIME_WAIT状态的bucket数量。
net.ipv4.tcp_tw_reuse = 1 # 允许重⽤TIME_WAIT套接
字。设置为1表示允许，设置为0表示不允许。
net.ipv4.tcp_max_orphans = 327680 # 系统中最⼤的孤
套接字数量。
net.ipv4.tcp_orphan_retries = 3 # 系统尝试重新分
配孤套接字的次数。
net.ipv4.tcp_syncookies = 1 # ⽤于防⽌SYN洪⽔攻击。
设置为1表示启⽤SYN cookies，设置为0表示禁⽤。
net.ipv4.tcp_max_syn_backlog = 16384 # SYN连接请
求队列的最⼤⻓度。
net.ipv4.ip_conntrack_max = 65536 # IP连接跟踪表的
最⼤⼤⼩。
net.ipv4.tcp_max_syn_backlog = 16384 # 系统中最⼤
的监听队列的⻓度。net.ipv4.tcp_timestamps = 0 # ⽤于关闭TCP时间戳选
项。
net.core.somaxconn = 16384 # ⽤于设置系统中最⼤的监
听队列的⻓度
root@k8s-master \~# reboot
保存后，所有节点重启，保证重启后内核依然加载
root@k8s-master \~# lsmod | grep --color=auto -e
ip_vs -e nf_conntrack
ip_vs_ftp 16384 0
nf_nat 45056 3
ipt_MASQUERADE,nft_chain_nat,ip_vs_ftp
ip_vs_sed 16384 0
ip_vs_nq 16384 0
ip_vs_fo 16384 0
ip_vs_sh 16384 0
ip_vs_dh 16384 0
ip_vs_lblcr 16384 0
ip_vs_lblc 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs_wlc 16384 0
ip_vs_lc 16384 0
ip_vs 172032 25
ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,i
p_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_
vs_sed,ip_vs_ftp四、基本组件安装
1. 安装 Containerd
（1）安装 Docker
（2）配置 Containerd 所需模块
nf_conntrack 172032 4
xt_conntrack,nf_nat,ipt_MASQUERADE,ip_vs
nf_defrag_ipv6 20480 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 5
nf_conntrack,nf_nat,nf_tables,xfs,ip_vs
root@k8s-master \~# yum remove -y podman runc
containerd # 卸载之前的containerd
root@k8s-master \~# yum install docker-ce docker
ce-cli containerd.io -y # 安装Docker和containerd（3）配置 Containerd 所需内核
root@k8s-master \~# cat <<EOF | sudo tee
/etc/modules-load.d/containerd.conf
> overlay # ⽤于⽀持Overlay⽹络⽂件系统的模块，它可以
在现有的⽂件系统之上创建叠加层，以实现虚拟化、隔离和管理等功
能。
> br_netfilter # ⽤于containerd的⽹络过滤模块，它可
以对进出容器的⽹络流量进⾏过滤和管理。
> EOF
overlay
br_netfilter
root@k8s-master \~# modprobe -- overlay
root@k8s-master \~# modprobe -- br_netfilter（4）Containerd 配置⽂件
root@k8s-master \~# cat <<EOF | sudo tee
/etc/sysctl.d/99-kubernetes-cri.conf # tee：读取
的数据写⼊到⼀个或多个⽂件中，同时还将其复制到标准输出
> net.bridge.bridge-nf-call-iptables = 1 # ⽤于
控制⽹络桥接是否调⽤iptables进⾏包过滤和转发。
> net.ipv4.ip_forward = 1 # 路由
转发，1为开启
> net.bridge.bridge-nf-call-ip6tables = 1 # 控制
是否在桥接接⼝上调⽤IPv6的iptables进⾏数据包过滤和转发。
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
root@k8s-master \~# sysctl --system
root@k8s-master \~# mkdir -p /etc/containerd
root@k8s-master \~# containerd config default |
tee /etc/containerd/config.toml # 读取containerd
的配置并保存到/etc/containerd/config.toml
root@k8s-master \~# vim
/etc/containerd/config.toml
找到containerd.runtimes.runc.options模块，添加
SystemdCgroup = true，如果已经存在则直接修改
plugins."io.containerd.grpc.v1.cri".containerd.ru
ntimes.runc.options
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = false # 没有就添加，有
的话就修改
找到sandbox_image修改为如下参数
sandbox_image = "registry.cn
hangzhou.aliyuncs.com/google_containers/pause:3.9"
root@k8s-master \~# systemctl daemon-reload #
加载systemctl控制脚本
root@k8s-master \~# systemctl enable --now
containerd # 启动containerd并设置开机启动
Created symlink /etc/systemd/system/multi
user.target.wants/containerd.service "
/usr/lib/systemd/system/containerd.service.（5）配置 crictl 客户端连接的运⾏位置
2. 安装 Kubernetes 组件
安装 Kubeadm、Kubelet 和 Kubectl
root@k8s-master \~# cat > /etc/crictl.yaml <<EOF
配置容器运⾏环境的crictl.yml⽂件
> runtime-endpoint:
unix:///run/containerd/containerd.sock # 指定了容
器运⾏时的地址为：unix://...
> image-endpoint:
unix:///run/containerd/containerd.sock # 指定
了镜像运⾏时的地址为：unix://...
> timeout: 10 # 设置了超时时间为10秒
> debug: false # 关闭调试模式
> EOF3. Kubernetes 集群初始化
root@k8s-master \~# yum list kubeadm.x86_64 --
showduplicates | sort -r
查询最新的Kubernetes版本号
root@k8s-master \~# yum install kubeadm-1.28*
kubelet-1.28* kubectl-1.28* -y
安装1.28最新版本kubeadm、kubelet和kubectl
root@k8s-master \~# systemctl daemon-reload
root@k8s-master \~# systemctl enable --now
kubelet # 允许开机⾃启kubelet
Created symlink /etc/systemd/system/multi
user.target.wants/kubelet.service "
/usr/lib/systemd/system/kubelet.service.
root@k8s-master \~# kubeadm version # 查看当前安
装的kubeadm版本号
kubeadm version: &version.Info{Major:"1",
Minor:"28", GitVersion:"v1.28.2",
GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2
f", GitTreeState:"clean", BuildDate:"2023-09-
13T09:34:32Z", GoVersion:"go1.20.8",
Compiler:"gc", Platform:"linux/amd64"}（1）Kubeadm 配置⽂件
root@k8s-master \~# vim kubeadm-config.yaml #
修改kubeadm配置⽂件
apiVersion: kubeadm.k8s.io/v1beta3 # 指定
Kubernetes配置⽂件的版本，使⽤的是kubeadm API的v1beta3
版本
bootstrapTokens: # 定义bootstrap tokens的信息。这
些tokens⽤于在Kubernetes集群初始化过程中进⾏身份验证
- groups: # 定义了与此token关联的组
- system:bootstrappers:kubeadm:default-node
token
token: 7t2weq.bjbawausm0jaxury # bootstrap
token的值
ttl: 24h0m0s # token的⽣存时间，这⾥设置为24⼩时
usages: # 定义token的⽤途 - signing # 数字签名
- authentication # 身份验证
kind: InitConfiguration # 指定配置对象的类型，
InitConfiguration：表示这是⼀个初始化配置
localAPIEndpoint: # 定义本地API端点的地址和端⼝
advertiseAddress: 192.168.15.11
bindPort: 6443
nodeRegistration: # 定义节点注册时的配置
criSocket:
unix:///var/run/containerd/containerd.sock # 容器
运⾏时(CRI)的套接字路径 name: k8s-master # 节点的名称
taints: # 标记 - effect: NoSchedule # 免调度节点
key: node-role.kubernetes.io/control-plane
该节点为控制节点
apiServer: # 定义了API服务器的配置
certSANs: # 为API服务器指定了附加的证书主体名称
(SAN)，指定IP即可
- 192.168.15.11
timeoutForControlPlane: 4m0s # 控制平⾯的超时时
间，这⾥设置为4分钟
apiVersion: kubeadm.k8s.io/v1beta3 # 指定API
Server版本
certificatesDir: /etc/kubernetes/pki # 指定了证书的
存储⽬录
clusterName: kubernetes # 定义了集群的名称
为"kubernetes"
controlPlaneEndpoint: 192.168.15.11:6443 # 定义
了控制节点的地址和端⼝
controllerManager: {} # 控制器管理器的配置，为空表示
使⽤默认配置
etcd: # 定义了etcd的配置
local: # 本地etcd实例
dataDir: /var/lib/etcd # 数据⽬录（2）下载组件镜像
imageRepository: registry.cn
hangzhou.aliyuncs.com/google_containers # 指定了
Kubernetes使⽤的镜像仓库的地址，阿⾥云的镜像仓库。
kind: ClusterConfiguration # 指定了配置对象的类型，
ClusterConfiguration：表示这是⼀个集群配置
kubernetesVersion: v1.28.2 # 指定了kubernetes的版
本
networking: # 定义了kubernetes集群⽹络设置
dnsDomain: cluster.local # 定义了集群的DNS域为：
cluster.local
podSubnet: 172.16.0.0/16 # 定义了Pod的⼦⽹
serviceSubnet: 10.96.0.0/16 # 定义了服务的⼦⽹
scheduler: {} # 使⽤默认的调度器⾏为
root@k8s-master \~# kubeadm config migrate --old
config kubeadm-config.yaml --new-config new.yaml
将旧的kubeadm配置⽂件转换为新的格式root@k8s-master \~# kubeadm config images pull --
config /root/new.yaml # 通过新的配置⽂件new.yaml从指
定的阿⾥云仓库拉取kubernetes组件镜像
config/images Pulled registry.cn
hangzhou.aliyuncs.com/google_containers/kube
apiserver:v1.28.2
config/images Pulled registry.cn
hangzhou.aliyuncs.com/google_containers/kube
controller-manager:v1.28.2
config/images Pulled registry.cn
hangzhou.aliyuncs.com/google_containers/kube
scheduler:v1.28.2
config/images Pulled registry.cn
hangzhou.aliyuncs.com/google_containers/kube
proxy:v1.28.2
config/images Pulled registry.cn
hangzhou.aliyuncs.com/google_containers/pause:3.9
config/images Pulled registry.cn
hangzhou.aliyuncs.com/google_containers/etcd:3.5.9
-0
config/images Pulled registry.cn
hangzhou.aliyuncs.com/google_containers/coredns:v1
.10.1（3）集群初始化
root@k8s-master \~# kubeadm init --config
/root/new.yaml --upload-certs
You can now join any number of the control-plane
node running the following command on each as
root:
kubeadm join 192.168.15.11:6443 --token
7t2weq.bjbawausm0jaxury \
--discovery-token-ca-cert-hash
sha256:73dc6f8d973fc70818e309386c1bfc5d330c19d52b4
94c6f88f634a6b1250a2f \
--control-plane --certificate-key
80fcc505867ccbc6550c18ed11f40e64ecf486d626403823f5
48dda65c19953d
等待初始化后保存这些命令
root@k8s-master \~# vim token.txt
kubeadm join 192.168.15.11:6443 --token
7t2weq.bjbawausm0jaxury \
--discovery-token-ca-cert-hash
sha256:73dc6f8d973fc70818e309386c1bfc5d330c19d52b4
94c6f88f634a6b1250a2f \ # 当需要加⼊新node节点时，
只复制到这即可（4）加载环境变量
（5）查看组件容器状态
之前采⽤初始化安装⽅式，所有的系统组件均以容器的⽅式运⾏
并且在 kube-system 命名空间内，此时可以查看 Pod（容器
组）状态
--control-plane --certificate-key
80fcc505867ccbc6550c18ed11f40e64ecf486d626403823f5
48dda65c19953d # 当需要⾼可⽤master集群时，将整个
token复制下来
root@k8s-master \~# cat <<EOF >> /root/.bashrc
> export KUBECONFIG=/etc/kubernetes/admin.conf
> EOF
root@k8s-master \~# source /root/.bashrc
root@k8s-master \~# kubectl get po -n kube-system
NAME READY
STATUS RESTARTS AGE
coredns-6554b8b87f-77brw 0/1
Pending 0 6m1s
coredns-6554b8b87f-8hwr4 0/1
Pending 0 6m1s
etcd-k8s-master 1/1
Running 0 6m16s（6）初始化失败排查
Ⅰ. 初始化重置
如果初始化失败，重置后再次初始化，命令如下**（没有失败不要执**
⾏！！！）
Ⅱ. 多次初始化失败
⾸先排查系统⽇志
kube-apiserver-k8s-master 1/1
Running 0 6m16s
kube-controller-manager-k8s-master 1/1
Running 0 6m16s
kube-proxy-j778p 1/1
Running 0 6m2s
kube-scheduler-k8s-master 1/1
Running 0 6m16s
kubectl：k8s控制命令
get：获取参数
po：pod缩写
-n：指定命名空间
kube-system：命名空间
kubeadm reset -f ; ipvsadm --clear ; rm -rf
~/.kubeCentOS⽇志路径：/var/log/messages
Ubuntu⽇志路径:/var/log/syslog
通过⽇志找到错误原因
最后再检查之前所有的配置⽂件是否有编写错误，有的配置⽂件
在修改后需要重新载⼊，可以根据刚才的步骤进⾏修改及载⼊，
最终确认⽆误后输⼊重置命令，再进⾏初始化。
经常出错的原因：
Containerd 的配置⽂件修改的不对，⾃⾏参考《安装
containerd》⼩节核对。
new.yaml 配置问题，⽐如⾮⾼可⽤集群忘记修改 6443 端
⼝为 6443。
new.yaml 配置问题，三个⽹段有交叉，出现 IP 地址冲突。
VIP 不通导致⽆法初始化成功，此时 messages ⽇志会有
VIP 超时的报错。
Ⅲ.连接 API 服务器超时
当获取集群状态出现以下信息时：
tail -f /var/log/messages | grep -v "not found"
root@master \~\]# kubectl get poE1221 14:39:38.091729 2782 memcache.go:265
couldn't get current server API group list: Get
"https://192.168.15.11:6443/api?timeout=32s": dial
tcp 192.168.15.11:6443: connect: connection
refused
E1221 14:39:38.092239 2782 memcache.go:265]
couldn't get current server API group list: Get
"https://192.168.15.11:6443/api?timeout=32s": dial
tcp 192.168.15.11:6443: connect: connection
refused
E1221 14:39:38.094041 2782 memcache.go:265]
couldn't get current server API group list: Get
"https://192.168.15.11:6443/api?timeout=32s": dial
tcp 192.168.15.11:6443: connect: connection
refused
E1221 14:39:38.095440 2782 memcache.go:265]
couldn't get current server API group list: Get
"https://192.168.15.11:6443/api?timeout=32s": dial
tcp 192.168.15.11:6443: connect: connection
refused
E1221 14:39:38.097007 2782 memcache.go:265]
couldn't get current server API group list: Get
"https://192.168.15.11:6443/api?timeout=32s": dial
tcp 192.168.15.11:6443: connect: connection
refusedThe connection to the server 192.168.15.11:6443
was refused - did you specify the right host or
port?
此时可以修改系统环境变量
临时修改：
export KUBECONFIG=/etc/kubernetes/admin.conf
⻓期修改：
mkdir ~/.kube
cp /etc/kubernetes/admin.conf ~/.kube/config
如果修改环境变量后也不⾏时，需要重新进⾏初始化（依次执⾏下
⾯两条命令）
kubeadm reset -f ; ipvsadm --clear ; rm -rf
~/.kube
kubeadm init --config /root/new.yaml --upload
certs4. Master ⾼可⽤
其他 master 加⼊集群时，输⼊如下命令
如：需要⾼可⽤时，⼜克隆了 master02、03...等，那么这些节
点都执⾏下⾯的命令
**注意：**每个主机的 token 值是不⼀样的，下⾯是我
192.168.15.11 (master)主机的 token 值，这是集群初始化⽣成
的代码，需要在当时记录下来。
5. Token 过期处理
注意：**以下步骤是上述初始化命令产⽣的 Token 过期了才需要执
⾏以下步骤，如果没有过期不需要执⾏，直接 join 即可。**
Token 过期后⽣成新的 token
kubeadm join 192.168.15.11:6443 --token
7t2weq.bjbawausm0jaxury \
--discovery-token-ca-cert-hash \
sha256:73dc6f8d973fc70818e309386c1bfc5d330c19d52b4
94c6f88f634a6b1250a2f \
--control-plane --certificate-key \
80fcc505867ccbc6550c18ed11f40e64ecf486d626403823f5
48dda65c19953dMaster 需要⽣成 --certificate-key：
6. Node 节点配置
Node 节点上主要部署公司的⼀些业务应⽤，⽣产环境中不建议
Master 节点部署系统组件之外的其他 Pod，测试环境可以允许
Master 节点部署 Pod 以节省系统资源。
（1）node 加⼊集群
kubeadm token create --print-join-command
kubeadm init phase upload-certs --upload-certs
root@k8s-node01 \~# kubeadm join
192.168.15.11:6443 --token 7t2weq.bjbawausm0jaxury
\ # node01通过复制master初始化⽣成的token来加⼊集群
> --discovery-token-ca-cert-hash \
>
sha256:73dc6f8d973fc70818e309386c1bfc5d330c19d52b4
94c6f88f634a6b1250a2f
preflight Running pre-flight checks
preflight Reading configuration from the
cluster...
preflight FYI: You can look at this config file
with 'kubectl -n kube-system get cm kubeadm-config
-o yaml'（2）查看集群状态
master 上查看集群状态（NotReady 不影响）
kubelet-start Writing kubelet configuration to
file "/var/lib/kubelet/config.yaml"
kubelet-start Writing kubelet environment file
with flags to file "/var/lib/kubelet/kubeadm
flags.env"
kubelet-start Starting the kubelet
kubelet-start Waiting for the kubelet to perform
the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to
apiserver and a response was received.
* The Kubelet was informed of the new secure
connection details.
Run 'kubectl get nodes' on the control-plane to
see this node join the cluster.
正确加⼊集群后的输出信息到此建议打快照
7. Calico 组件安装
（1）切换 git 分⽀
root@k8s-master \~# kubectl get node # 获取所有节
点信息
NAME STATUS ROLES AGE
VERSION
k8s-master NotReady control-plane 35m
v1.28.2
k8s-node01 NotReady <none> 6m39s
v1.28.2
k8s-node02 NotReady <none> 7m27s
v1.28.2
root@k8s-master \~# cd /root/k8s-ha-install &&
git checkout manual-installation-v1.28.x && cd
calico/
分⽀ 'manual-installation-v1.28.x' 设置为跟踪
'origin/manual-installation-v1.28.x'。
切换到⼀个新分⽀ 'manual-installation-v1.28.x'（2）修改 Pod ⽹段
（3）查看容器和节点状态
root@k8s-master calico# POD_SUBNET=`cat
/etc/kubernetes/manifests/kube-controller
manager.yaml | grep cluster-cidr= | awk -F=
'{print NF}'\` # 获取已定义的Pod⽹段
\[root@k8s-master calico\]# sed -i
"s#POD_CIDR#{POD_SUBNET}#g" calico.yaml # 修改
calico.yml⽂件中的pod⽹段
root@k8s-master calico# kubectl apply -f
calico.yaml # 创建calico的pod
root@k8s-master calico# kubectl get po -n kube
system
NAME READY
STATUS RESTARTS AGE
calico-kube-controllers-6d48795585-wj8g5 1/1
Running 0 130m
calico-node-bk4p5 1/1
Running 0 130m
calico-node-kmsh7 1/1
Running 0 130m
calico-node-qthgh 1/1
Running 0 130mcoredns-6554b8b87f-jdc2b 1/1
Running 0 133m
coredns-6554b8b87f-thftb 1/1
Running 0 133m
etcd-master 1/1
Running 0 133m
kube-apiserver-master 1/1
Running 0 133m
kube-controller-manager-master 1/1
Running 0 133m
kube-proxy-46j4z 1/1
Running 0 131m
kube-proxy-8g887 1/1
Running 0 133m
kube-proxy-vwp27 1/1
Running 0 131m
kube-scheduler-master 1/1
Running 0 133m
root@k8s-master calico# kubectl get node # 此
时节点全部准备完成
NAME STATUS ROLES AGE
VERSION
k8s-master Ready control-plane 40m
v1.28.2
k8s-node01 Ready <none> 12m
v1.28.28. Metrics 部署
在新版的 Kubernetes 中系统资源的采集均使⽤ Metrics-server，可
以通过 Metrics 采集节点和 Pod 的内存、磁盘、CPU和⽹络的使⽤
率。
（1）复制证书到所有 node 节点
将 master 节点的 front-proxy-ca.crt 复制到所有 Node 节点，每有
⼀个节点执⾏⼀次，仅需修改命令内的 node 节点主机名即可。
k8s-node02 Ready <none> 12m
v1.28.2（2）安装 metrics server
root@k8s-master calico# scp
/etc/kubernetes/pki/front-proxy-ca.crt k8s
node01:/etc/kubernetes/pki/front-proxy-ca.crt #
向node01节点发送代理证书
front-proxy-ca.crt
100% 1123 937.0KB/s 00:00
root@k8s-master calico# scp
/etc/kubernetes/pki/front-proxy-ca.crt k8s
node02:/etc/kubernetes/pki/front-proxy-ca.crt #
向node02节点发送代理证书
front-proxy-ca.crt
100% 1123 957.4KB/s 00:00
若有其他node节点，按照格式执⾏下⾯命令，这⾥不⽤执⾏，因
为node只有两台主机
root@k8s-master calico# scp
/etc/kubernetes/pki/front-proxy-ca.crt k8s
node03:/etc/kubernetes/pki/front-proxy-ca.crt（3）查看 metrics server 状态
root@k8s-master calico# cd /root/k8s-ha
install/kubeadm-metrics-server
root@k8s-master kubeadm-metrics-server# kubectl
create -f comp.yaml # 添加metric server的pod资源
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggre
gated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metri
cs-server created
rolebinding.rbac.authorization.k8s.io/metrics
server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metri
cs-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/syste
m:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.
k8s.io created
root@master kubeadm-metrics-server# kubectl get
po -n kube-system -l k8s-app=metrics-server # 在
kube-system命名空间下查看metrics server的pod运⾏状态NAME READY STATUS
RESTARTS AGE
metrics-server-8df99c47f-mkbfd 1/1 Running
0 34s
root@master kubeadm-metrics-server# kubectl top
node # 查看node节点的系统资源使⽤情况
NAME CPU(cores) CPU% MEMORY(bytes)
MEMORY%
k8s-node01 51m 1% 831Mi
23%
k8s-node02 55m 1% 931Mi
25%
master 107m 2% 1412Mi
39%
root@master kubeadm-metrics-server# kubectl top
po -A
NAMESPACE NAME
CPU(cores) MEMORY(bytes)
kube-system calico-kube-controllers-6d48795585-
wj8g5 2m 25Mi
kube-system calico-node-bk4p5
20m 155Mi
kube-system calico-node-kmsh7
25m 152Mi
kube-system calico-node-qthgh
24m 145Mi 9. Dashboard部署
Dashboard ⽤于展示集群中的各类资源，同时也可以通过
Dashboard 实时查看 Pod 的⽇志和在容器中执⾏⼀些命令等。
kube-system coredns-6554b8b87f-jdc2b
1m 22Mi
kube-system coredns-6554b8b87f-thftb
1m 20Mi
kube-system etcd-master
14m 66Mi
kube-system kube-apiserver-master
29m 301Mi
kube-system kube-controller-manager-master
10m 56Mi
kube-system kube-proxy-46j4z
1m 22Mi
kube-system kube-proxy-8g887
1m 24Mi
kube-system kube-proxy-vwp27
1m 22Mi
kube-system kube-scheduler-master
2m 26Mi
kube-system metrics-server-8df99c47f-mkbfd
3m 29Mi （1）安装组件
root@master kubeadm-metrics-server# cd
/root/k8s-ha-install/dashboard/
root@master dashboard# kubectl create -f . #
建⽴dashboard的pod资源
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin
-user created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes
dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes
dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes
dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kuber
netes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created（2）登录 dashboard
如果是⾕歌浏览器，需要在启动⽂件中加⼊下⾯的启动参数，⽤
于解决⽆法访问 Dashboard 的问题
--test-type --ignore-certificate-errors
（3）更改 svc 模式root@master dashboard# kubectl edit svc
kubernetes-dashboard -n kubernetes-dashboard
edit：进⼊kubernetes的⽂本编辑器
svc：指定某个服务项，这⾥指定的是kubernetes-dashboard
-n：指定命名空间，kubernetes-dashboard
命令执⾏后相当于进⼊vim⽂本编辑器，不要⽤⿏标滚轮，会输出
乱码的！可以使⽤"/"搜索，输⼊"/type"找到⽬标，如果已经为
NodePort忽略此步骤
......省略部分内容......
selector:
k8s-app: kubernetes-dashboard
sessionAffinity: None
type: NodePort
图示：（4）查看访问端⼝号
找到端⼝号后，通过 master 的 IP+端⼝即可访问 dashboard（端⼝
为终端查询到的端⼝，要⽤ https 协议访问）
root@master dashboard# kubectl get svc
kubernetes-dashboard -n kubernetes-dashboard # 获
取kubernetes-dashboard状态信息，包含端⼝，服务IP等
NAME TYPE CLUSTER-IP
EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.96.137.94
<none> 443:30582/TCP 8m50s（5）创建登录 token
在"输⼊ token *"内输⼊终端⽣成的 token
root@master dashboard# kubectl create token
admin-user -n kube-system
eyJhbGciOiJSUzI1NiIsImtpZCI6Inlvc2g1cWhWcjduaXI1ZU
FpQWNwRFJYYW1saXVFM3lrdlJnaHlUSmY0RTAifQ.eyJhdWQiO
lsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN
0ZXIubG9jYWwiXSwiZXhwIjoxNzAzMDU2Nzg4LCJpYXQiOjE3M
DMwNTMxODgsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZ
hdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pb
yI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2V
hY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiN
zE0YWU1N2UtNjRiNC00NTU0LTk5OTctYjE2NmEwZTQyNzhjIn1
9LCJuYmYiOjE3MDMwNTMxODgsInN1YiI6InN5c3RlbTpzZXJ2a
WNlYWNjb3VudDprdWJlLXN5c3RlbTphZG1pbi11c2VyIn0.l6z
gXfNhppndKAqFJrR_vCi7w0_esGp7yQGNYdrQGlE5latyFKRXN
Jub8dvDe-ZyquW1H-KTvZntAluwOXv79W
KY8Z8d31FePN9LHzCXPDordzyg8rE7qvgAPNeU8Fg
VnYtr_ujpB
muBinjnzT7LjysJiBi6fsndiD5RUYcYr6bsLg91bcLgAdW3bn_
9W5587z_q-910wpxl9AwUL9xVzy
vsVDDdXe1VthkoGYxyaznRf5omkmpwabQ3JQ0L8U_8Oop6HaZs
g5cEBCqBHrgyjBsYRALjzRlFlC9CB4hrYY4P_zRSdoI0lyiG4Z
eh0ber6awoeeKSMbJMTqwMlw10. Kube-proxy
（1）改为 ipvs模式
（2）更新 Kube-Proxy 的 Pod
root@master \~# kubectl edit cm kube-proxy -n
kube-system
使⽤"/"找到"mode"，按照如下修改
mode: ipvs五、集群可⽤性验证
1. 验证节点
root@master \~# kubectl patch daemonset kube
proxy -p "{\"spec\":{\"template\":{\"metadata\":
{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
-n kube-system
daemonset.apps/kube-proxy patched
root@master \~# curl 127.0.0.1:10249/proxyMode
ipvs
root@master \~# kubectl get node # 全部为Ready，
是正常
NAME STATUS ROLES AGE
VERSION
k8s-node01 Ready <none> 156m
v1.28.2
k8s-node02 Ready <none> 155m
v1.28.2
master Ready control-plane 157m
v1.28.22. 验证 Pod
root@master \~# kubectl get po -A # 全部为
running，表示正常
NAMESPACE NAME
READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-
6d48795585-wj8g5 1/1 Running 0
156m
kube-system calico-node-bk4p5
1/1 Running 0 156m
kube-system calico-node-kmsh7
1/1 Running 0 156m
kube-system calico-node-qthgh
1/1 Running 0 156m
kube-system coredns-6554b8b87f-jdc2b
1/1 Running 0
159m
kube-system coredns-6554b8b87f-thftb
1/1 Running 0
159m
kube-system etcd-master
1/1 Running 0 159m
kube-system kube-apiserver-master
1/1 Running 0 159mkube-system kube-controller-manager
master 1/1 Running 0
159m
kube-system kube-proxy-9sxt9
1/1 Running 0
5m6s
kube-system kube-proxy-g79z5
1/1 Running 0
5m7s
kube-system kube-proxy-scwgn
1/1 Running 0
5m9s
kube-system kube-scheduler-master
1/1 Running 0 159m
kube-system metrics-server-8df99c47f
mkbfd 1/1 Running 0
154m
kubernetes-dashboard dashboard-metrics-scraper-
7b554c884f-92jwb 1/1 Running 0
24m
kubernetes-dashboard kubernetes-dashboard-
54b699784c-f7trp 1/1 Running 0
24m3. 验证集群⽹段是否冲突
三⽅⽹段均不冲突（service、Pod、宿主机）
root@master \~# kubectl get svc # 查看服务的⽹段
NAME TYPE CLUSTER-IP EXTERNAL-IP
PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none>
443/TCP 160m
root@master \~# kubectl get po -A -owide # 查看
所有命名空间下的所有⽹段，再与服务的⽹段进⾏⽐较
NAMESPACE NAME
READY STATUS RESTARTS AGE
IP NODE NOMINATED NODE
READINESS GATES
kube-system calico-kube-controllers-
6d48795585-wj8g5 1/1 Running 0
158m 172.16.58.194 k8s-node02 <none>
<none>
kube-system calico-node-bk4p5
1/1 Running 0 158m
192.168.15.22 k8s-node01 <none>
<none>
kube-system calico-node-kmsh7
1/1 Running 0 158m
192.168.15.33 k8s-node02 <none>
<none>kube-system calico-node-qthgh
1/1 Running 0 158m
192.168.15.11 master <none>
<none>
kube-system coredns-6554b8b87f-jdc2b
1/1 Running 0
160m 172.16.58.195 k8s-node02 <none>
<none>
kube-system coredns-6554b8b87f-thftb
1/1 Running 0
160m 172.16.58.193 k8s-node02 <none>
<none>
kube-system etcd-master
1/1 Running 0 160m
192.168.15.11 master <none>
<none>
kube-system kube-apiserver-master
1/1 Running 0 160m
192.168.15.11 master <none>
<none>
kube-system kube-controller-manager
master 1/1 Running 0
160m 192.168.15.11 master <none>
<none>kube-system kube-proxy-9sxt9
1/1 Running 0
6m29s 192.168.15.11 master <none>
<none>
kube-system kube-proxy-g79z5
1/1 Running 0
6m30s 192.168.15.33 k8s-node02 <none>
<none>
kube-system kube-proxy-scwgn
1/1 Running 0
6m32s 192.168.15.22 k8s-node01 <none>
<none>
kube-system kube-scheduler-master
1/1 Running 0 160m
192.168.15.11 master <none>
<none>
kube-system metrics-server-8df99c47f
mkbfd 1/1 Running 0
155m 172.16.85.193 k8s-node01 <none>
<none>
kubernetes-dashboard dashboard-metrics-scraper-
7b554c884f-92jwb 1/1 Running 0
25m 172.16.85.195 k8s-node01 <none>
<none>4. 验证是否可正常创建参数
kubernetes-dashboard kubernetes-dashboard-
54b699784c-f7trp 1/1 Running 0
25m 172.16.85.194 k8s-node01 <none>
<none>
root@master \~# kubectl create deploy cluster
test --image=registry.cn
beijing.aliyuncs.com/dotbalo/debug-tools -- sleep
3600
deployment.apps/cluster-test created # 已创建，表
示正常
root@master \~# kubectl get po
NAME READY STATUS
RESTARTS AGE
cluster-test-66bb44bd88-sq8fx 1/1 Running
0 41s
root@master \~# kubectl get po -owide
NAME READY STATUS
RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
cluster-test-66bb44bd88-sq8fx 1/1 Running
0 48s 172.16.58.196 k8s-node02
<none> <none>5. Pod 必须能够解析 Service
同 namespace 和跨 namespace
（1）nslookup kubernetes
（2）nslookup kube-dns.kube-system
root@master \~# kubectl exec -it cluster-test-
66bb44bd88-sq8fx -- bash # 进⼊pod下的某个容器
(06:36 cluster-test-66bb44bd88-sq8fx:/) nslookup
kubernetes
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
可以解析到server的IP地址说明同namespace可以解析(06:36 cluster-test-66bb44bd88-sq8fx:/) nslookup
kube-dns.kube-system
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kube-dns.kube-system.svc.cluster.local
Address: 10.96.0.10
可以解析到server的第⼗个ip，说明可以解析到kube-dns，说
明跨namespace也可解析
6. 确认是否可访问 Kubernetes 的 443 和 kube-dns 的
53
每个节点都必须能访问 Kubernetes 的 kubernetes svc 443 和
kube-dns 的 service 537. 确认各 Pod 之间是否可正常通信
同 namespace 和跨 namespace
root@master \~# curl https://10.96.0.1:443
curl: (60) SSL certificate problem: unable to get
local issuer certificate
More details here:
https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server
and therefore could not
establish a secure connection to it. To learn more
about this situation and
how to fix it, please visit the web page mentioned
above.
root@master \~# curl 10.96.0.10:53
curl: (52) Empty reply from server
root@master \~# kubectl get po -nkube-system -
owide
NAME READY
STATUS RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATEScalico-kube-controllers-6d48795585-wj8g5 1/1
Running 0 170m 172.16.58.194 k8s
node02 <none> <none>
calico-node-bk4p5 1/1
Running 0 170m 192.168.15.22 k8s
node01 <none> <none>
calico-node-kmsh7 1/1
Running 0 170m 192.168.15.33 k8s
node02 <none> <none>
calico-node-qthgh 1/1
Running 0 170m 192.168.15.11 master
<none> <none>
coredns-6554b8b87f-jdc2b 1/1
Running 0 173m 172.16.58.195 k8s
node02 <none> <none>
coredns-6554b8b87f-thftb 1/1
Running 0 173m 172.16.58.193 k8s
node02 <none> <none>
etcd-master 1/1
Running 0 173m 192.168.15.11 master
<none> <none>
kube-apiserver-master 1/1
Running 0 173m 192.168.15.11 master
<none> <none>
kube-controller-manager-master 1/1
Running 0 173m 192.168.15.11 master
<none> <none>kube-proxy-9sxt9 1/1
Running 0 19m 192.168.15.11 master
<none> <none>
kube-proxy-g79z5 1/1
Running 0 19m 192.168.15.33 k8s
node02 <none> <none>
kube-proxy-scwgn 1/1
Running 0 19m 192.168.15.22 k8s
node01 <none> <none>
kube-scheduler-master 1/1
Running 0 173m 192.168.15.11 master
<none> <none>
metrics-server-8df99c47f-mkbfd 1/1
Running 0 168m 172.16.85.193 k8s
node01 <none> <none>
root@master \~# kubectl get po -owide
NAME READY STATUS
RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
cluster-test-66bb44bd88-sq8fx 1/1 Running
0 12m 172.16.58.196 k8s-node02
<none> <none>
root@master \~# kubectl exec -it cluster-test-
66bb44bd88-sq8fx -- bash
(06:46 cluster-test-66bb44bd88-sq8fx:/) ping
172.16.58.195 -c 3PING 172.16.58.195 (172.16.58.195) 56(84) bytes of
data.
64 bytes from 172.16.58.195: icmp_seq=1 ttl=63
time=0.455 ms
64 bytes from 172.16.58.195: icmp_seq=2 ttl=63
time=0.082 ms
64 bytes from 172.16.58.195: icmp_seq=3 ttl=63
time=0.082 ms
--- 172.16.58.195 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss,
time 2083ms
rtt min/avg/max/mdev = 0.082/0.206/0.455/0.176 ms
同机器和跨机器六、注意事项
root@master \~# kubectl get po -owide
NAME READY STATUS
RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
cluster-test-66bb44bd88-sq8fx 1/1 Running
0 13m 172.16.58.196 k8s-node02
<none> <none>
root@master \~# ping 172.16.58.196 -c 3
PING 172.16.58.196 (172.16.58.196) 56(84) bytes of
data.
64 bytes from 172.16.58.196: icmp_seq=1 ttl=63
time=0.676 ms
64 bytes from 172.16.58.196: icmp_seq=2 ttl=63
time=0.303 ms
64 bytes from 172.16.58.196: icmp_seq=3 ttl=63
time=0.284 ms
--- 172.16.58.196 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss,
time 2043ms
rtt min/avg/max/mdev = 0.284/0.421/0.676/0.180 ms注意：kubeadm 安装的集群，证书有效期默认是⼀年。master 节点
的 kube-apiserver、kube-scheduler、kube-controller-manager、
etcd 都是以容器运⾏的。可以通过 kubectl get po -n kube-system
查看。
启动和⼆进制不同的是，
kubelet 的配置⽂件在 /etc/sysconfig/kubelet
和/var/lib/kubelet/config.yaml，修改后需要重启 kubelet 进程。
其他组件的配置⽂件在 /etc/kubernetes/manifests ⽬录下，⽐如
kube-apiserver.yaml，该 yaml ⽂件更改后，kubelet 会⾃动刷新配
置，也就是会重启 pod。不能再次创建该⽂件
kube-proxy 的配置在 kube-system 命名空间下的 configmap 中，
可以通过：
kubectl edit cm kube-proxy -n kube-system
进⾏更改，更改完成后，可以通过 patch 重启 kube-proxy：
kubectl patch daemonset kube-proxy -p "{\"spec\":
{\"template\":{\"metadata\":{\"annotations\":
{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system
Kubeadm 安装后，master 节点默认不允许部署 Pod，可以通过以
下⽅式删除 Taint，即可部署 Pod：kubectl taint node -l node
role.kubernetes.io/control-plane node
role.kubernetes.io/control-plane:NoSchedule-
主机名
IP地址
说明
k8s-master01 ~
03
192.168.15.101~103
master节点 * 3
k8s-node01 ~ 02 192.168.15.22 |
192.168.15.33
worker节点 * 2
k8s-master-lb
192.168.15.88
keepalived虚拟
IP
⼀、安装说明
本⽂章将演示Rocky 8 ⼆进制⽅式安装⾼可⽤k8s 1.28.0版本。
⽣产环境中，建议使⽤⼩版本⼤于5的Kubernetes版本，⽐如1.19.5
以后的才可⽤于⽣产环境。
⼆、集群安装
2.1 基本环境配置配置信息
备注
系统版本
Rocky linux 8.7
Docker版本
24.10
Pod⽹段
172.16.0.0/16
Service⽹段
10.96.0.0/16
请统⼀替换这些⽹段，Pod⽹段和service和宿主机⽹段不要重
复！！！
注意：宿主机⽹段、K8s Service⽹段、Pod⽹段不能重复
主机信息，服务器IP地址不能设置成dhcp，要配置成静态IP。
系统环境：
各节点配置：
root@k8s-master01 \~# cat /etc/redhat-release
Rocky Linux release 8.7 (Green Obsidian)
root@k8s-master01 \~#192.168.100.61 k8s-master01 # 2C2G 20G
192.168.100.62 k8s-master02 # 2C2G 20G
192.168.100.63 k8s-master03 # 2C2G 20G
192.168.100.69 k8s-master-lb # VIP 虚拟IP不占⽤机器
资源 # 如果不是⾼可⽤集群，该IP为Master01的IP
192.168.100.64 k8s-node01 # 2C2G 20G
192.168.100.65 k8s-node02 # 2C2G 20G
配置**所有节点hosts⽂件：**
root@k8s-master01 \~# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain
localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain
localhost6 localhost6.localdomain6
192.168.100.61 k8s-master01
192.168.100.62 k8s-master02
192.168.100.63 k8s-master03
192.168.100.69 k8s-master-lb # 如果不是⾼可⽤集群，
该IP为Master01>的IP
192.168.100.64 k8s-node01
192.168.100.65 k8s-node02Rocky8**所有节点配置yum源：**
root@k8s-master01 \~# cd /etc/yum.repos.d/
root@k8s-master01 yum.repos.d# ls
bak
root@k8s-master01 yum.repos.d# cat
>>/etc/yum.repos.d/docker-ce.repo<<'EOF'
> docker-ce-stable
> name=Docker CE Stable - basearch
\> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/basearch/stable
\> enabled=1
\> gpgcheck=1
\> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
\>
\> \[docker-ce-stable-debuginfo\]
\> name=Docker CE Stable - Debuginfo basearch
> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/debug-basearch/stable
> enabled=0
> gpgcheck=1
> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
>
> docker-ce-stable-source
> name=Docker CE Stable - Sources> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/source/stable
\> enabled=0
\> gpgcheck=1
\> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
\>
\> \[docker-ce-test\]
\> name=Docker CE Test - basearch
> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/basearch/test
> enabled=0
> gpgcheck=1
> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
>
> docker-ce-test-debuginfo
> name=Docker CE Test - Debuginfo basearch
\> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/debug-basearch/test
\> enabled=0
\> gpgcheck=1
\> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
\>
\> \[docker-ce-test-source\]
\> name=Docker CE Test - Sources\> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/source/test
> enabled=0
> gpgcheck=1
> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
>
> docker-ce-nightly
> name=Docker CE Nightly - basearch
\> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/basearch/nightly
\> enabled=0
\> gpgcheck=1
\> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
\>
\> \[docker-ce-nightly-debuginfo\]
\> name=Docker CE Nightly - Debuginfo basearch
> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/debug-basearch/nightly
> enabled=0
> gpgcheck=1
> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
>
> docker-ce-nightly-source
> name=Docker CE Nightly - Sources> baseurl=https://mirrors.aliyun.com/docker
ce/linux/centos/releasever/source/nightly
\> enabled=0
\> gpgcheck=1
\> gpgkey=https://mirrors.aliyun.com/docker
ce/linux/centos/gpg
\> EOF
\[root@k8s-master01 yum.repos.d\]#
\[root@k8s-master01 yum.repos.d\]# cat
\>\>/etc/yum.repos.d/Rocky-BaseOS.repo\<\<'EOF'
\> # Rocky-BaseOS.repo
\> #
\> # The mirrorlist system uses the connecting IP
address of the client and the
\> # update status of each mirror to pick current
mirrors that are geographically
\> # close to the client. You should use this for
Rocky updates unless you are
\> # manually picking other mirrors.
\> #
\> # If the mirrorlist does not work for you, you
can try the commented out
\> # baseurl line instead.
\>
\> \[baseos\]
\> name=Rocky Linux releasever - BaseOS>
#mirrorlist=https://mirrors.rockylinux.org/mirrorli
st?arch=basearch\&repo=BaseOS-releasever
>
baseurl=https://mirrors.aliyun.com/rockylinux/$rele
asever/BaseOS/basearch/os/
\> gpgcheck=1
\> enabled=1
\> gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
rockyofficial
\> EOF
\[root@k8s-master01 yum.repos.d\]#
\[root@k8s-master01 yum.repos.d\]# cat \>\>Rocky
AppStream.repo\<\<'EOF'
\> # Rocky-AppStream.repo
\> #
\> # The mirrorlist system uses the connecting IP
address of the client and the
\> # update status of each mirror to pick current
mirrors that are geographically
\> # close to the client. You should use this for
Rocky updates unless you are
\> # manually picking other mirrors.
\> #
\> # If the mirrorlist does not work for you, you
can try the commented out
\> # baseurl line instead.\>
\> \[appstream\]
\> name=Rocky Linux releasever - AppStream
>
#mirrorlist=https://mirrors.rockylinux.org/mirrorli
st?arch=basearch\&repo=AppStream-releasever
>
baseurl=https://mirrors.aliyun.com/rockylinux/$rele
asever/AppStream/$basearch/os/
> gpgcheck=1
> enabled=1
> gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
rockyofficial
> EOF
root@k8s-master01 yum.repos.d# ls
bak docker-ce.repo Rocky-AppStream.repo Rocky
BaseOS.repo
root@k8s-master01 yum.repos.d# yum clean all
38 个⽂件已删除
root@k8s-master01 yum.repos.d# yum makecache
Rocky Linux 8 - AppStream 3.6 MB/s | 11
MB 00:02
Rocky Linux 8 - BaseOS 1.6 MB/s | 6.0
MB 00:03
Docker CE Stable - x86_64 75 kB/s | 49
kB 00:00
元数据缓存已建⽴。root@k8s-master01 yum.repos.d#
所有节点**必备⼯具安装**
root@k8s-master01 yum.repos.d# cd
root@k8s-master01 \~# yum install wget jq psmisc
vim net-tools telnet yum-utils device-mapper
persistent-data lvm2 git -y
所有节点**关闭firewalld 、dnsmasq、selinux**
root@k8s-master01 \~# systemctl disable --now
firewalld
root@k8s-master01 \~# systemctl disable --now
dnsmasq
root@k8s-master01 \~# setenforce 0
root@k8s-master01 \~# sed -i
's#SELINUX=enforcing#SELINUX=disabled#g'
/etc/sysconfig/selinux
root@k8s-master01 \~# sed -i
's#SELINUX=enforcing#SELINUX=disabled#g'
/etc/selinux/config
root@k8s-master01 \~#所有节点**关闭swap分区，fstab注释swap**
root@k8s-master01 \~# swapoff -a && sysctl -w
vm.swappiness=0
vm.swappiness = 0
root@k8s-master01 \~# sed -ri '/^\^#*swap/s@^@#@'
/etc/fstab
root@k8s-master01 \~#
所有节点**同步时间**root@k8s-master01 \~# rpm -ivh
https://mirrors.wlnmp.com/rockylinux/wlnmp-release
rocky-8.noarch.rpm
获取https://mirrors.wlnmp.com/rockylinux/wlnmp
release-rocky-8.noarch.rpm
Verifying...
################################# 100%
准备中...
################################# 100%
正在升级/安装...
1:wlnmp-release-rocky-1-1
################################# 100%
root@k8s-master01 \~# yum -y install wntp
root@k8s-master01 \~# ntpdate time2.aliyun.com
31 Aug 22:43:34 ntpdate3549: adjust time server
203.107.6.88 offset +0.000713 sec
root@k8s-master01 \~# crontab -e
no crontab for root - using an empty one
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com
所有节点**配置limit**root@k8s-node02 \~# ulimit -SHn 65535
root@k8s-node02 \~# vim /etc/security/limits.conf
末尾添加如下内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
Master01**节点免密钥登录其他节点，安装过程中⽣成配置⽂件和
证书均在Master01上操作，集群管理也在Master01上操作。密钥配
置如下：**
root@k8s-master01 \~# ssh-keygen -t rsa
Master01**配置免密码登录其他节点**
root@k8s-master01 \~# for i in k8s-master01 k8s
master02 k8s-master03 k8s-node01 k8s-node02;do ssh
copy-id -i .ssh/id_rsa.pub $i;doneMaster01**下载安装⽂件**
root@k8s-master01 \~# cd /root/ ; git clone
https://gitee.com/dukuan/k8s-ha-install.git
正克隆到 'k8s-ha-install'...
remote: Enumerating objects: 879, done.
remote: Counting objects: 100% (205/205), done.
remote: Compressing objects: 100% (127/127), done.
remote: Total 879 (delta 90), reused 145 (delta
52), pack-reused 674
接收对象中: 100% (879/879), 19.70 MiB | 2.37 MiB/s,
完成.
处理 delta 中: 100% (354/354), 完成.
root@k8s-master01 \~# ls
点名.txt 视频 下载 anaconda-ks.cfg k8s-ha
install
公共 图⽚ ⾳乐 dianming.sh yum.sh
模板 ⽂档 桌⾯ initial-setup-ks.cfg
root@k8s-master01 \~#
所有节点* *升级系统并重启，此处升级没有升级内核**2.2 安装ipvsadm
所有节点**安装 ipvsadm 及检测系统的⼯具**
所有节点配置ipvs模块
root@k8s-master01 \~# yum update -y --
exclude=kernel* --nobest
root@k8s-master01 \~# reboot
root@k8s-master01 \~# yum install ipvsadm ipset
sysstat conntrack libseccomp -y
root@k8s-master01 \~# modprobe -- ip_vs
root@k8s-master01 \~# modprobe -- ip_vs_rr
root@k8s-master01 \~# modprobe -- ip_vs_wrr
root@k8s-master01 \~# modprobe -- ip_vs_sh
root@k8s-master01 \~# modprobe -- nf_conntrack
root@k8s-master01 \~# vim /etc/modules
load.d/ipvs.conf
加⼊以下内容
ip_vsip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
检查是否加载：
root@k8s-master01 \~# lsmod | grep -e ip_vs -e
nf_conntrack
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0ip_vs 172032 6
ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 172032 4
xt_conntrack,nf_nat,ipt_MASQUERADEip_vs
nf_defrag_ipv6 20480 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 5
nf_conntrack,nf_nat,nf_tables,xfs,ip_vs
开启⼀些k8s集群中必须的内核参数，**所有节点配置k8s内核：**
root@k8s-master01 \~# cat <<EOF >
/etc/sysctl.d/k8s.conf
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> fs.may_detach_mounts = 1
> vm.overcommit_memory=1
> net.ipv4.conf.all.route_localnet = 1
>
> vm.panic_on_oom=0
> fs.inotify.max_user_watches=89100
> fs.file-max=52706963
> fs.nr_open=52706963
> net.netfilter.nf_conntrack_max=2310720>
> net.ipv4.tcp_keepalive_time = 600
> net.ipv4.tcp_keepalive_probes = 3
> net.ipv4.tcp_keepalive_intvl =15
> net.ipv4.tcp_max_tw_buckets = 36000
> net.ipv4.tcp_tw_reuse = 1
> net.ipv4.tcp_max_orphans = 327680
> net.ipv4.tcp_orphan_retries = 3
> net.ipv4.tcp_syncookies = 1
> net.ipv4.tcp_max_syn_backlog = 16384
> net.ipv4.ip_conntrack_max = 65536
> net.ipv4.tcp_max_syn_backlog = 16384
> net.ipv4.tcp_timestamps = 0
> net.core.somaxconn = 16384
> EOF
root@k8s-master01 \~# sysctl --system
所有节点**配置完内核后，重启服务器，保证重启后内核依旧加载**
root@k8s-master01 \~# reboot
root@k8s-master01 \~# lsmod | grep --color=auto -e
ip_vs -e nf_conntrack
ip_vs_ftp 16384 0
nf_nat 45056 3
ipt_MASQUERADE,nft_chain_nat,ip_vs_ftpip_vs_sed 16384 0
ip_vs_nq 16384 0
ip_vs_fo 16384 0
ip_vs_sh 16384 0
ip_vs_dh 16384 0
ip_vs_lblcr 16384 0
ip_vs_lblc 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs_wlc 16384 0
ip_vs_lc 16384 0
ip_vs 172032 24
ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip
_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs
_sed,ip_vs_ftp
nf_conntrack 172032 4
xt_conntrack,nf_nat,ipt_MASQUERADEip_vs
nf_defrag_ipv6 20480 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 5
nf_conntrack,nf_nat,nf_tables,xfs,ip_vs
root@k8s-master01 \~#
第三章 基本组件安装第三章 基本组件安装
本节主要安装的是集群中⽤到的各种组件，⽐如Docker-ce、
Kubernetes各组件等
此处建议打个快照
3.1 Containerd作为Runtime
所有节点**安装docker-ce-24.0（如果已经有安装，也需要执⾏安装
升级到最新版）**
配置Containerd所需的模块（**所有节点）：**
root@k8s-master01 \~# yum remove -y podman runc
containerd
root@k8s-master01 \~# yum install docker-ce
docker-ce-cli containerd.io -yroot@k8s-master01 \~# cat <<EOF | sudo tee
/etc/modules-load.d/containerd.conf
> overlay
> br_netfilter
> EOF
overlay
br_netfilter
root@k8s-master01 \~#
所有节点**加载模块：**
root@k8s-master01 \~# modprobe -- overlay
root@k8s-master01 \~# modprobe -- br_netfilter
root@k8s-master01 \~#
所有节点**，配置Containerd所需的内核：**root@k8s-master01 \~# cat <<EOF | sudo tee
/etc/sysctl.d/99-kubernetes-cri.conf
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
root@k8s-master01 \~# sysctl --system
所有节点**配置Containerd的配置⽂件：**
（使⽤ containerd 默认配置⽣成⼀个 config.toml 配置⽂件，并将其
内容输出到 /etc/containerd/config.toml ⽂件中）
root@k8s-master01 \~# mkdir -p /etc/containerd
root@k8s-master01 \~# containerd config default |
tee /etc/containerd/config.toml
所有节点将Containerd的Cgroup改为Systemd：root@k8s-master01 \~# vim
/etc/containerd/config.toml
找到 containerd.runtimes.runc.options，添加 SystemdCgroup =
true（如果已存在直接修改，否则会报错），如下图所示：
所有节点**将 sandbox_image 的 Pause 镜像改成符合⾃⼰版本的地
址 registry.cn
hangzhou.aliyuncs.com/google_containers/pause:3.9**所有节点**启动Containerd，并配置开机⾃启动：**
root@k8s-master01 \~# systemctl daemon-reload
root@k8s-master01 \~# systemctl enable --now
containerd
Created symlink /etc/systemd/system/multi
user.target.wants/containerd.service "
/usr/lib/systemd/system/containerd.service.
root@k8s-master01 \~#
所有节点* *配置crictl客户端连接的运⾏时位置：**4.2 K8s及etcd安装
Master01下载kubernetes安装包
master01**下载etcd安装包**
root@k8s-master01 \~# cat > /etc/crictl.yaml <<EOF
> runtime-endpoint:
unix:///run/containerd/containerd.sock
> image-endpoint:
unix:///run/containerd/containerd.sock
> timeout: 10
> debug: false
> EOF
root@k8s-master01 \~#
root@k8s-master01 \~# wget
https://dl.k8s.io/v1.28.0/kubernetes-server-linux
amd64.tar.gzroot@k8s-master01 \~# wget
https://github.com/etcd
io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux
amd64.tar.gz
master01**解压kubernetes安装⽂件**
root@k8s-master01 \~# tar -xf kubernetes-server
linux-amd64.tar.gz --strip-components=3 -C
/usr/local/bin kubernetes/server/bin/kube{let,ctl,-
apiserver,-controller-manager,-scheduler,-proxy}
master01**解压etcd安装⽂件**root@k8s-master01 \~# tar -zxvf etcd-v3.5.9-linux
amd64.tar.gz --strip-components=1 -C /usr/local/bin
etcd-v3.5.9-linux-amd64/etcd{,ctl}
etcd-v3.5.9-linux-amd64/etcdctl
etcd-v3.5.9-linux-amd64/etcd
root@k8s-master01 \~# kubelet --version
Kubernetes v1.28.0
root@k8s-master01 \~# etcdctl version
etcdctl version: 3.5.9
API version: 3.5
root@k8s-master01 \~#
master01**将组件发送到其他节点**
root@k8s-master01 \~# MasterNodes='k8s-master02
k8s-master03'
root@k8s-master01 \~# WorkNodes='k8s-node01 k8s
node02'
root@k8s-master01 \~# for NODE in MasterNodes; do
echo NODE; scp /usr/local/bin/kube{let,ctl,-
apiserver,-controller-manager,-scheduler,-proxy}
NODE:/usr/local/bin/; scp /usr/local/bin/etcd\*
NODE:/usr/local/bin/; done
k8s-master02kubelet 100% 106MB
281.6MB/s 00:00
kubectl 100% 48MB
302.1MB/s 00:00
kube-apiserver 100% 116MB
146.9MB/s 00:00
kube-controller-manager 100% 112MB
129.0MB/s 00:00
kube-scheduler 100% 53MB
144.0MB/s 00:00
kube-proxy 100% 52MB
145.7MB/s 00:00
etcd 100% 21MB
105.1MB/s 00:00
etcdctl 100% 16MB
113.7MB/s 00:00
k8s-master03
kubelet 100% 106MB
269.0MB/s 00:00
kubectl 100% 48MB
212.2MB/s 00:00
kube-apiserver 100% 116MB
150.4MB/s 00:00
kube-controller-manager 100% 112MB
109.9MB/s 00:01
kube-scheduler 100% 53MB
154.3MB/s 00:00 kube-proxy 100% 52MB
137.2MB/s 00:00
etcd 100% 21MB
126.1MB/s 00:00
etcdctl 100% 16MB
105.4MB/s 00:00
root@k8s-master01 \~# for NODE in WorkNodes; do
scp /usr/local/bin/kube{let,-proxy}
NODE:/usr/local/bin/ ; done
kubelet 100% 106MB
239.4MB/s 00:00
kube-proxy 100% 52MB
135.6MB/s 00:00
kubelet 100% 106MB
306.6MB/s 00:00
kube-proxy 100% 52MB
295.4MB/s 00:00
root@k8s-master01 \~#
Master01节点**切换到1.28.x分⽀**
（其他版本可以切换到其他分⽀，.x即可，不需要更改为具体的⼩版
本）第四章 ⽣成证书
⼆进制安装最关键步骤，⼀步错误全盘皆输，⼀定要注意每个步骤都
要是正确的。
所以此处建议打快照！
Master01**下载⽣成证书⼯具**
root@k8s-master01 \~# cd /root/k8s-ha-install &&
git checkout manual-installation-v1.28.x
分⽀ 'manual-installation-v1.28.x' 设置为跟踪
'origin/manual-installation-v1.28.x'。
切换到⼀个新分⽀ 'manual-installation-v1.28.x'
root@k8s-master01 k8s-ha-install#4.1 Etcd证书
所有Master节点**创建etcd证书⽬录**
所有节点**创建kubernetes相关⽬录**
Master01节点**⽣成etcd证书**
root@k8s-master01 k8s-ha-install# wget
"https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O
/usr/local/bin/cfssl
root@k8s-master01 k8s-ha-install# wget
"https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64"
-O /usr/local/bin/cfssljson
root@k8s-master01 k8s-ha-install# chmod +x
/usr/local/bin/cfssl /usr/local/bin/cfssljson
root@k8s-master01 k8s-ha-install#
root@k8s-master01 \~# mkdir /etc/etcd/ssl -p
root@k8s-master01 \~#
root@k8s-master01 \~# mkdir -p /etc/kubernetes/pki
root@k8s-master01 \~#⽣成证书的CSR⽂件：证书签名请求⽂件，配置了⼀些域名、公司、
单位
root@k8s-master01 \~# cd /root/k8s-ha-install/pki
root@k8s-master01 pki# cfssl gencert -initca
etcd-ca-csr.json | cfssljson -bare
/etc/etcd/ssl/etcd-ca ## ⽣成etcd CA证书和CA证书的
key
2023/09/01 01:03:37 INFO generating a new CA key
and certificate from CSR
2023/09/01 01:03:37 INFO generate received
request
2023/09/01 01:03:37 INFO received CSR
2023/09/01 01:03:37 INFO generating key: rsa-2048
2023/09/01 01:03:37 INFO encoded CSR
2023/09/01 01:03:37 INFO signed certificate with
serial number
344130823394962800880998772770250831901274089808
root@k8s-master01 pki# cfssl gencert \
> -ca=/etc/etcd/ssl/etcd-ca.pem \
> -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
> -config=ca-config.json \
> -hostname=127.0.0.1,k8s-master01,k8s
master02,k8s
master03,192.168.100.61,192.168.100.62,192.168.100.
63 \ # 注意主节点IP地址
> -profile=kubernetes \> etcd-csr.json | cfssljson -bare
/etc/etcd/ssl/etcd
2023/09/01 01:04:39 INFO generate received
request
2023/09/01 01:04:39 INFO received CSR
2023/09/01 01:04:39 INFO generating key: rsa-2048
2023/09/01 01:04:39 INFO encoded CSR
2023/09/01 01:04:39 INFO signed certificate with
serial number
467674155326432618988047679680337371584557335423
root@k8s-master01 pki#
master01**将证书复制到其他节点**
root@k8s-master01 pki# MasterNodes='k8s-master02
k8s-master03'
root@k8s-master01 pki# WorkNodes='k8s-node01 k8s
node02'
root@k8s-master01 pki# for NODE in MasterNodes;
do
\> ssh NODE "mkdir -p /etc/etcd/ssl"
> for FILE in etcd-ca-key.pem etcd-ca.pem
etcd-key.pem etcd.pem; do
> scp /etc/etcd/ssl/{FILE}
NODE:/etc/etcd/ssl/${FILE}4.2 k8s组件证书
Master01**⽣成kubernetes证书**
> done
> done
etcd-ca-key.pem 100% 1675
1.0MB/s 00:00
etcd-ca.pem 100% 1367
1.7MB/s 00:00
etcd-key.pem 100% 1675
1.1MB/s 00:00
etcd.pem 100% 1509
1.9MB/s 00:00
etcd-ca-key.pem 100% 1675
1.2MB/s 00:00
etcd-ca.pem 100% 1367
592.0KB/s 00:00
etcd-key.pem 100% 1675
1.5MB/s 00:00
etcd.pem 100% 1509
1.3MB/s 00:00
root@k8s-master01 pki#root@k8s-master01 pki# cd /root/k8s-ha
install/pki
root@k8s-master01 pki# cfssl gencert -initca ca
csr.json | cfssljson -bare /etc/kubernetes/pki/ca
2023/09/01 01:06:52 INFO generating a new CA key
and certificate from CSR
2023/09/01 01:06:52 INFO generate received
request
2023/09/01 01:06:52 INFO received CSR
2023/09/01 01:06:52 INFO generating key: rsa-2048
2023/09/01 01:06:53 INFO encoded CSR
2023/09/01 01:06:53 INFO signed certificate with
serial number
700660882784581217234688273261359631764300667154
root@k8s-master01 pki#
10.96.0.0是k8s service的⽹段，如果说需要更改k8s service⽹段，
那就需要更改10.96.0.1，
# 如果不是⾼可⽤集群，192.168.15.88 为 Master01 的 IP
Master01 ⽣成 ca 证书root@k8s-master01 pki# cfssl gencert -
ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -
hostname=10.96.0.1,192.168.15.88,127.0.0.1,kubernet
es,kubernetes.default,kubernetes.default.svc,kubern
etes.default.svc.cluster,kubernetes.default.svc.clu
ster.local,192.168.15.101,192.168.15.102,192.168.15
.103 \
> -profile=kubernetes \
> apiserver-csr.json | cfssljson -bare
/etc/kubernetes/pki/apiserver
2023/09/01 01:09:03 INFO generate received
request
2023/09/01 01:09:03 INFO received CSR
2023/09/01 01:09:03 INFO generating key: rsa-2048
2023/09/01 01:09:03 INFO encoded CSR
2023/09/01 01:09:03 INFO signed certificate with
serial number
700805877078105797489323146582505553062943677004
root@k8s-master01 pki#
Master01**⽣成apiserver的聚合证书**Requestheader-client-xxx requestheader-allowwd-xxx:aggerator
root@k8s-master01 pki# cfssl gencert -initca
front-proxy-ca-csr.json | cfssljson -bare
/etc/kubernetes/pki/front-proxy-ca
2023/09/01 01:09:49 INFO generating a new CA key
and certificate from CSR
2023/09/01 01:09:49 INFO generate received
request
2023/09/01 01:09:49 INFO received CSR
2023/09/01 01:09:49 INFO generating key: rsa-2048
2023/09/01 01:09:50 INFO encoded CSR
2023/09/01 01:09:50 INFO signed certificate with
serial number
80744563078517085527684184401947310203460343599
root@k8s-master01 pki# cfssl gencert -
ca=/etc/kubernetes/pki/front-proxy-ca.pem \
> -ca-key=/etc/kubernetes/pki/front-proxy-ca
key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> front-proxy-client-csr.json | cfssljson -bare
/etc/kubernetes/pki/front-proxy-client
返回结果（忽略警告）
2023/09/01 01:09:56 INFO generate received
request
2023/09/01 01:09:56 INFO received CSR2023/09/01 01:09:56 INFO generating key: rsa-2048
2023/09/01 01:09:56 INFO encoded CSR
2023/09/01 01:09:56 INFO signed certificate with
serial number
269856273999475183076691732446396898422130380732
2023/09/01 01:09:56 WARNING This certificate
lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline
Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the
CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information
Requirements").
root@k8s-master01 pki#
Master01**⽣成controller-manage的证书**
root@k8s-master01 pki# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> manager-csr.json | cfssljson -bare
/etc/kubernetes/pki/controller-manager2023/09/01 01:11:24 INFO generate received
request
2023/09/01 01:11:24 INFO received CSR
2023/09/01 01:11:24 INFO generating key: rsa-2048
2023/09/01 01:11:24 INFO encoded CSR
2023/09/01 01:11:24 INFO signed certificate with
serial number
59801199172235379177295009346141259163675889793
2023/09/01 01:11:24 WARNING This certificate
lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline
Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the
CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information
Requirements").
root@k8s-master01 pki#
注意，如果不是⾼可⽤集群，192.168.15.88:8443改为master01的
地址，8443改为apiserver的端⼝，默认是6443
set-cluster：**Master01设置⼀个集群项**root@k8s-master01 pki# kubectl config set-cluster
kubernetes \
> --certificate
authority=/etc/kubernetes/pki/ca.pem \
> --embed-certs=true \
> --server=https://192.168.15.88:8443 \
> --kubeconfig=/etc/kubernetes/controller
manager.kubeconfig
Cluster "kubernetes" set.
root@k8s-master01 pki#
Master01**设置⼀个环境项**
root@k8s-master01 pki# kubectl config set-context
system:kube-controller-manager@kubernetes \
> --cluster=kubernetes \
> --user=system:kube-controller-manager \
> --kubeconfig=/etc/kubernetes/controller
manager.kubeconfig
Context "system:kube-controller-manager@kubernetes"
created.
root@k8s-master01 pki#set-credentials：**Master01设置⼀个⽤户项**
root@k8s-master01 pki# kubectl config set
credentials system:kube-controller-manager \
> --client
certificate=/etc/kubernetes/pki/controller
manager.pem \
> --client-key=/etc/kubernetes/pki/controller
manager-key.pem \
> --embed-certs=true \
> --kubeconfig=/etc/kubernetes/controller
manager.kubeconfig
User "system:kube-controller-manager" set.
root@k8s-master01 pki#
Master01**使⽤某个环境当做默认环境**
root@k8s-master01 pki# kubectl config use-context
system:kube-controller-manager@kubernetes \
> --kubeconfig=/etc/kubernetes/controller
manager.kubeconfig
Switched to context "system:kube-controller
manager@kubernetes".
root@k8s-master01 pki# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> scheduler-csr.json | cfssljson -bare
/etc/kubernetes/pki/scheduler
2023/09/01 01:14:36 INFO generate received
request
2023/09/01 01:14:36 INFO received CSR
2023/09/01 01:14:36 INFO generating key: rsa-2048
2023/09/01 01:14:36 INFO encoded CSR
2023/09/01 01:14:36 INFO signed certificate with
serial number
88018517488547413954267519985609730397109488269
2023/09/01 01:14:36 WARNING This certificate
lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline
Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the
CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information
Requirements").
root@k8s-master01 pki#
注意，如果不是⾼可⽤集群，192.168.15.88:8443改为master01的地
址，8443改为apiserver的端⼝，默认是6443Master01 设置
root@k8s-master01 pki# kubectl config set-cluster
kubernetes \
> --certificate
authority=/etc/kubernetes/pki/ca.pem \
> --embed-certs=true \
> --server=https://192.168.15.88:8443 \
> --
kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Cluster "kubernetes" set.
root@k8s-master01 pki# kubectl config set
credentials system:kube-scheduler \
> --client
certificate=/etc/kubernetes/pki/scheduler.pem \
> --client-key=/etc/kubernetes/pki/scheduler
key.pem \
> --embed-certs=true \
> --
kubeconfig=/etc/kubernetes/scheduler.kubeconfig
User "system:kube-scheduler" set.
root@k8s-master01 pki# kubectl config set-context
system:kube-scheduler@kubernetes \
> --cluster=kubernetes \
> --user=system:kube-scheduler \
> --
kubeconfig=/etc/kubernetes/scheduler.kubeconfigContext "system:kube-scheduler@kubernetes" created.
root@k8s-master01 pki# kubectl config use-context
system:kube-scheduler@kubernetes \
> --
kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Switched to context "system:kube
scheduler@kubernetes".
root@k8s-master01 pki# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> admin-csr.json | cfssljson -bare
/etc/kubernetes/pki/admin
2023/09/01 01:16:13 INFO generate received
request
2023/09/01 01:16:13 INFO received CSR
2023/09/01 01:16:13 INFO generating key: rsa-2048
2023/09/01 01:16:13 INFO encoded CSR
2023/09/01 01:16:13 INFO signed certificate with
serial number
660421027543221851817737469949130636763120428998
2023/09/01 01:16:13 WARNING This certificate
lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline
Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the
CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information
Requirements").
root@k8s-master01 pki#
注意，如果不是⾼可⽤集群，192.168.15.88：8443改为master01的
地址，8443改为apiserver的端⼝，默认是6443
Master01 配置
root@k8s-master01 pki# kubectl config set-cluster
kubernetes \
> --certificate
authority=/etc/kubernetes/pki/ca.pem \
> --embed-certs=true \
> --server=https://192.168.15.88:8443 \
> --kubeconfig=/etc/kubernetes/admin.kubeconfig
Cluster "kubernetes" set.
root@k8s-master01 pki# kubectl config set
credentials kubernetes-admin \
> --client
certificate=/etc/kubernetes/pki/admin.pem \
> --client-key=/etc/kubernetes/pki/admin-key.pem
\> --embed-certs=true \
> --kubeconfig=/etc/kubernetes/admin.kubeconfig
User "kubernetes-admin" set.
root@k8s-master01 pki# kubectl config set-context
kubernetes-admin@kubernetes --
cluster=kubernetes --user=kubernetes-admin
--kubeconfig=/etc/kubernetes/admin.kubeconfig
Context "kubernetes-admin@kubernetes" created.
root@k8s-master01 pki# kubectl config use-context
kubernetes-admin@kubernetes --
kubeconfig=/etc/kubernetes/admin.kubeconfig
Switched to context "kubernetes-admin@kubernetes".
root@k8s-master01 pki#
Master01**创建ServiceAccount Key =》secret**root@k8s-master01 pki# openssl genrsa -out
/etc/kubernetes/pki/sa.key 2048
Generating RSA private key, 2048 bit long modulus
(2 primes)
............+++++
...................................................
..........................+++++
e is 65537 (0x010001)
root@k8s-master01 pki# openssl rsa -in
/etc/kubernetes/pki/sa.key -pubout -out
/etc/kubernetes/pki/sa.pub
writing RSA key
root@k8s-master01 pki#
Master01**发送证书⾄其他节点**root@k8s-master01 pki# for NODE in k8s-master02
k8s-master03; do > for FILE in (ls
/etc/kubernetes/pki \| grep -v etcd); do
\> scp /etc/kubernetes/pki/{FILE}
NODE:/etc/kubernetes/pki/{FILE};
> done;
> for FILE in admin.kubeconfig controller
manager.kubeconfig scheduler.kubeconfig; do
> scp /etc/kubernetes/{FILE}
NODE:/etc/kubernetes/{FILE};
\> done;
\> done
**查看证书⽂件**
\[root@k8s-master01 pki\]# ls /etc/kubernetes/pki/
admin.csr front-proxy-ca.csr
admin-key.pem front-proxy-ca-key.pem
admin.pem front-proxy-ca.pem
apiserver.csr front-proxy-client.csr
apiserver-key.pem front-proxy-client
key.pem
apiserver.pem front-proxy-client.pem
ca.csr sa.key
ca-key.pem sa.pub此处建议打快照！
第五章 ⾼可⽤配置
⾼可⽤配置（注意：如果不是⾼可⽤集群，haproxy和keepalived⽆
需安装）
如果在云上安装也⽆需执⾏此章节的步骤，可以直接使⽤云上的lb，
⽐如阿⾥云slb，腾讯云elb等
公有云要⽤公有云⾃带的负载均衡，⽐如阿⾥云的SLB，腾讯云的
ELB，⽤来替代haproxy和keepalived，因为公有云⼤部分都是不⽀
持keepalived的。
ca.pem scheduler.csr
controller-manager.csr scheduler-key.pem
controller-manager-key.pem scheduler.pem
controller-manager.pem
\[root@k8s-master01 pki\]# ls /etc/kubernetes/pki/
\|wc -l
23
\[root@k8s-master01 pki\]#**所有Master节点\***\*安装keepalived和haproxy\*\*
**5.1 HAProxy 配置**
**所有Master\***\*配置HAProxy，配置⼀样\*\*
**（注意修改 IP）**
\[root@k8s-master01 \~\]# yum install keepalived
haproxy -y
\[root@k8s-master01 \~\]# vim /etc/haproxy/haproxy.cfg
global
maxconn 2000
ulimit-n 16384
log 127.0.0.1 local0 err
stats timeout 30s
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000**5.1 Keepalived 配置**
**所有Master节点配置KeepAlived，配置不⼀样，注意区分**
timeout http-request 15s
timeout http-keep-alive 15s
frontend k8s-master
bind 0.0.0.0:8443
bind 127.0.0.1:8443
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master
backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall
2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server k8s-master01 192.168.15.101:6443 check
server k8s-master02 192.168.15.102:6443 check
server k8s-master03 192.168.15.103:6443 check【注意每个节点的IP、VIP和⽹卡（interface参数），在此只列出⼀
台 master01 的配置】
\[root@k8s-master01 \~\]# vim
/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface ens32
mcast_src_ip 192.168.15.101
virtual_router_id 51
priority 101
nopreempt
advert_int 2
authentication {
auth_type PASS**5.2 健康检查配置**
**所有master节点**
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.15.88
}
track_script {
chk_apiserver
} }
\[root@k8s-master01 \~\]# vim
/etc/keepalived/check_apiserver.sh
#!/bin/bash
err=0
for k in (seq 1 3)
do
check_code=(pgrep haproxy)
if \[\[ check_code == "" ]]; then
err=(expr err + 1)
sleep 1
continue
elseerr=0
break
fi
done
if \[ $err != "0" ]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
root@k8s-master01 \~# chmod +x
/etc/keepalived/check_apiserver.sh
root@k8s-master01 \~#
所有master节点**启动haproxy和keepalived**root@k8s-master01 \~# systemctl daemon-reload
root@k8s-master01 \~# systemctl enable --now
haproxy
Created symlink /etc/systemd/system/multi
user.target.wants/haproxy.service "
/usr/lib/systemd/system/haproxy.service.
root@k8s-master01 \~# systemctl enable --now
keepalived
Created symlink /etc/systemd/system/multi
user.target.wants/keepalived.service "
/usr/lib/systemd/system/keepalived.service.
root@k8s-master01 \~#
VIP测试root@k8s-master01 \~# ping 192.168.15.88 -c 3
PING 192.168.100.69 (192.168.100.69) 56(84) bytes
of data.
64 bytes from 192.168.15.88: icmp_seq=1 ttl=64
time=0.052 ms
64 bytes from 192.168.15.88: icmp_seq=2 ttl=64
time=0.046 ms
64 bytes from 192.168.15.88: icmp_seq=3 ttl=64
time=0.114 ms
--- 192.168.100.69 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss,
time 2044ms
rtt min/avg/max/mdev = 0.046/0.070/0.114/0.032 ms
root@k8s-master01 \~#
**重要：**如果安装了keepalived和haproxy，需要测试keepalived是否
是正常的！root@k8s-master01 \~# telnet 192.168.15.88 8443
Trying 192.168.15.88...
Connected to 192.168.15.88.
Escape character is '^]'.
Connection closed by foreign host.
root@k8s-master01 \~#
如果ping不通且telnet没有出现 '^]'，则认为VIP不可以，不可在继续
往下执⾏，需要排查keepalived的问题。⽐如防⽕墙和 selinux，
haproxy和keepalived 的状态，监听端⼝等
所有节点**查看防⽕墙状态必须为disable和inactive：systemctl
status firewalld**
所有节点**查看selinux状态，必须为disable：getenforce**
master节点**查看haproxy和keepalived状态：systemctl status
keepalived haproxy**
master节点**查看监听端⼝：netstat -lntp**
第六章 Kubernetes组件配第六章 Kubernetes组件配
置
6.1 etcd配置
etcd配置⼤致相同，注意修改每个Master节点的etcd配置的主机名
和IP地址
6.1.1 master01
当前只列出⼀台 master01 的配置，02、03 修改 IP 即可
root@k8s-master01 \~# vim
/etc/etcd/etcd.config.yml
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.15.101:2380'
listen-client-urls:
'https://192.168.15.101:2379,http://127.0.0.1:2379'
max-snapshots: 3max-wals: 5
cors:
initial-advertise-peer-urls:
'https://192.168.15.101:2380'
advertise-client-urls:
'https://192.168.15.101:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s
master01=https://192.168.15.101:2380,k8s
master02=https://192.168.15.102:2380,k8s
master03=https://192.168.15.103:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'6.1.2 创建service
所有Master节点**创建etcd service并启动**
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd
ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd
ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: default
force-new-cluster: false
root@k8s-master01 \~# vim
/usr/lib/systemd/system/etcd.service
Unit
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/After=network.target
Service
Type=notify
ExecStart=/usr/local/bin/etcd --config
file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
Install
WantedBy=multi-user.target
Alias=etcd3.service
所有Master节点**创建etcd的证书⽬录**root@k8s-master01 \~# mkdir
/etc/kubernetes/pki/etcd
root@k8s-master01 \~# ln -s /etc/etcd/ssl/*
/etc/kubernetes/pki/etcd/
root@k8s-master01 \~# systemctl daemon-reload
root@k8s-master01 \~# systemctl enable --now etcd
Created symlink /etc/systemd/system/etcd3.service "
/usr/lib/systemd/system/etcd.service.
Created symlink /etc/systemd/system/multi
user.target.wants/etcd.service "
/usr/lib/systemd/system/etcd.service.
root@k8s-master01 \~#
查看etcd状态
root@k8s-master01 \~# export ETCDCTL_API=3
root@k8s-master01 \~# etcdctl --
endpoints="192.168.15.103:2379,192.168.15.102:2379,
192.168.15.101:2379" \
> --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem \
> --cert=/etc/kubernetes/pki/etcd/etcd.pem \
> --key=/etc/kubernetes/pki/etcd/etcd-key.pem \
> endpoint status --write-out=table+---------------------+------------------+---------
+---------+-----------+------------+-----------+---
---------+--------------------+--------+
| ENDPOINT | ID | VERSION
| DB SIZE | IS LEADER | IS LEARNER | RAFT TERM |
RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------
+---------+-----------+------------+-----------+---
---------+--------------------+--------+
| 192.168.100.63:2379 | a1dc876ddb79d945 | 3.5.9
| 20 kB | false | false | 2 |
8 | 8 | |
| 192.168.100.62:2379 | 94fc7c634260ffca | 3.5.9
| 20 kB | false | false | 2 |
8 | 8 | |
| 192.168.100.61:2379 | 578060df8daece21 | 3.5.9
| 20 kB | true | false | 2 |
8 | 8 | |
+---------------------+------------------+---------
+---------+-----------+------------+-----------+---
---------+--------------------+--------+
root@k8s-master01 \~#
6.2 API Server6.2 API Server
所有Master节点**创建kube-apiserver service**
注意，如果不是⾼可⽤集群，192.168.15.88 改为master01的地址
master01**配置 API Server 程序控制脚本**
注意使⽤的k8s service⽹段为10.96.0.0/16，该⽹段不能和宿主机的
⽹段、Pod⽹段的重复，请按需修改
在此只列出 master01 配置，其他 master 节点按照该配置修改 IP 即
可
root@k8s-master01 \~# vim
/usr/lib/systemd/system/kube-apiserver.service
Unit
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kuberne
tes
After=network.target
Service
ExecStart=/usr/local/bin/kube-apiserver \
--v=2 \--allow-privileged=true \
--bind-address=0.0.0.0 \
--secure-port=6443 \
--advertise-address=192.168.15.101 \
--service-cluster-ip-range=10.96.0.0/16 \
--service-node-port-range=30000-32767 \
--etcd
servers=https://192.168.15.101:2379,https://192.168
.15.102:2379,https://192.168.15.103:2379 \
--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--client-ca-file=/etc/kubernetes/pki/ca.pem
\
--tls-cert
file=/etc/kubernetes/pki/apiserver.pem \
--tls-private-key
file=/etc/kubernetes/pki/apiserver-key.pem \
--kubelet-client
certificate=/etc/kubernetes/pki/apiserver.pem \
--kubelet-client
key=/etc/kubernetes/pki/apiserver-key.pem \
--service-account-key
file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key
file=/etc/kubernetes/pki/sa.key \--service-account
issuer=https://kubernetes.default.svc.cluster.local
\
--kubelet-preferred-address
types=InternalIP,ExternalIP,Hostname \
--enable-admission
plugins=NamespaceLifecycle,LimitRanger,ServiceAccou
nt,DefaultStorageClass,DefaultTolerationSeconds,Nod
eRestriction,ResourceQuota \
--authorization-mode=Node,RBAC \
--enable-bootstrap-token-auth=true \
--requestheader-client-ca
file=/etc/kubernetes/pki/front-proxy-ca.pem \
--proxy-client-cert
file=/etc/kubernetes/pki/front-proxy-client.pem \
--proxy-client-key
file=/etc/kubernetes/pki/front-proxy-client-key.pem
\
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group
\
--requestheader-extra-headers-prefix=X
Remote-Extra- \
--requestheader-username-headers=X-Remote
User
--token-auth-file=/etc/kubernetes/token.csvRestart=on-failure
RestartSec=10s
LimitNOFILE=65535
Install
WantedBy=multi-user.target
root@k8s-master01 \~# systemctl daemon-reload
root@k8s-master01 \~# systemctl enable --now kube
apiserver
Created symlink /etc/systemd/system/multi
user.target.wants/kube-apiserver.service "
/usr/lib/systemd/system/kube-apiserver.service.
root@k8s-master01 \~#
检测kube-server状态
root@k8s-master01 \~# systemctl status kube
apiserver
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube
apiserver.service;>
Active: active (running) since Fri 2023-09-01
13:54:46 CST; 38s>
Docs: https://github.com/kubernetes/kubernetes
Main PID: 3164 (kube-apiserver) Tasks: 7 (limit: 11057)
Memory: 296.3M
CGroup: /system.slice/kube-apiserver.service
#$3164 /usr/local/bin/kube-apiserver --
v=2 --allow-priv>
9⽉ 01 13:54:53 k8s-master01 kube-apiserver3164:
I0901 13:54:53.>
9⽉ 01 13:54:53 k8s-master01 kube-apiserver3164:
I0901 13:54:53.>
9⽉ 01 13:54:53 k8s-master01 kube-apiserver3164:
I0901 13:54:53.>
9⽉ 01 13:54:53 k8s-master01 kube-apiserver3164:
I0901 13:54:53.>
9⽉ 01 13:54:53 k8s-master01 kube-apiserver3164:
-poststarthoo>
9⽉ 01 13:54:54 k8s-master01 kube-apiserver3164:
E0901 13:54:54.>
9⽉ 01 13:55:04 k8s-master01 kube-apiserver3164:
W0901 13:55:04.>
9⽉ 01 13:55:04 k8s-master01 kube-apiserver3164:
I0901 13:55:04.>
9⽉ 01 13:55:04 k8s-master01 kube-apiserver3164:
I0901 13:55:04.>
9⽉ 01 13:55:20 k8s-master01 kube-apiserver3164:
I0901 13:55:20.>
root@k8s-master01 \~#6.3 Controller Manager
所有Master节点**配置kube-controller-manager service（所有
master节点配置⼀样）**
注意使⽤的k8s Pod⽹段为172.16.0.0/16，该⽹段不能和宿主机的⽹
段、k8s Service⽹段的重复，请按需修改
root@k8s-master01 \~# vim
/usr/lib/systemd/system/kube-controller
manager.service
Unit
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kuberne
tes
After=network.target
Service
ExecStart=/usr/local/bin/kube-controller-manager \
--v=2 \
--root-ca-file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-cert
file=/etc/kubernetes/pki/ca.pem \
--cluster-signing-key
file=/etc/kubernetes/pki/ca-key.pem \--service-account-private-key
file=/etc/kubernetes/pki/sa.key \
--kubeconfig=/etc/kubernetes/controller
manager.kubeconfig \
--authentication
kubeconfig=/etc/kubernetes/controller
manager.kubeconfig \
--authorization
kubeconfig=/etc/kubernetes/controller
manager.kubeconfig \
--leader-elect=true \
--use-service-account-credentials=true \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--controllers=*,bootstrapsigner,tokencleaner
\
--allocate-node-cidrs=true \
--cluster-cidr=172.16.0.0/16 \
--requestheader-client-ca
file=/etc/kubernetes/pki/front-proxy-ca.pem \
--node-cidr-mask-size=24
Restart=always
RestartSec=10s
Install
WantedBy=multi-user.target所有Master节点**启动kube-controller-manager**
root@k8s-master01 \~# systemctl daemon-reload
root@k8s-master01 \~# systemctl enable --now kube
controller-manager
Created symlink /etc/systemd/system/multi
user.target.wants/kube-controller-manager.service "
/usr/lib/systemd/system/kube-controller
manager.service.
root@k8s-master01 \~# systemctl status kube
controller-manager
● kube-controller-manager.service - Kubernetes
Controller Manager
Loaded: loaded (/usr/lib/systemd/system/kube
controller-manager>
Active: active (running) since Fri 2023-09-01
14:01:21 CST; 52s>
Docs: https://github.com/kubernetes/kubernetes
Main PID: 3649 (kube-controller)
Tasks: 5 (limit: 11057)
Memory: 132.2M
CGroup: /system.slice/kube-controller
manager.service
#$3649 /usr/local/bin/kube-controller
manager --v=2 --r>6.4 Scheduler
所有Master节点**配置kube-scheduler service（所有master节点配
置⼀样）**
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
9⽉ 01 14:01:22 k8s-master01 kube-controller
manager3649: I0901 >
lines 1-20/20 (END)root@k8s-master01 \~# cat
/usr/lib/systemd/system/kube-scheduler.service
Unit
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kuberne
tes
After=network.target
Service
ExecStart=/usr/local/bin/kube-scheduler \
--v=2 \
--leader-elect=true \
--authentication
kubeconfig=/etc/kubernetes/scheduler.kubeconfig \
--authorization
kubeconfig=/etc/kubernetes/scheduler.kubeconfig \
kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Restart=always
RestartSec=10s
Install
WantedBy=multi-user.target
root@k8s-master01 \~# systemctl daemon-reload
root@k8s-master01 \~# systemctl enable --now kube
schedulerCreated symlink /etc/systemd/system/multi
user.target.wants/kube-scheduler.service "
/usr/lib/systemd/system/kube-scheduler.service.
root@k8s-master01 \~# systemctl status kube
scheduler
● kube-scheduler.service - Kubernetes Scheduler
Loaded: loaded (/usr/lib/systemd/system/kube
scheduler.service;>
Active: active (running) since Fri 2023-09-01
14:04:15 CST; 5s >
Docs: https://github.com/kubernetes/kubernetes
Main PID: 3857 (kube-scheduler)
Tasks: 7 (limit: 11057)
Memory: 71.2M
CGroup: /system.slice/kube-scheduler.service
#3857 /usr/local/bin/kube-scheduler --
v=2 --leader-ele\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
9⽉ 01 14:04:17 k8s-master01 kube-scheduler\[3857\]:
I0901 14:04:17.\>
\[root@k8s-master01 \~\]#
第七章 TLS Bootstrapping
配置
TLS Bootstrapping（TLS 引导）是⼀种机制，⽤于在 Kubernetes
集群中⾃动为节点（Node）颁发和配置安全传输所需的 TLS 证书。
注意，如果不是⾼可⽤集群，192.168.15.88:8443改为master01的地
址，8443改为apiserver的端⼝，默认是6443**Master01\***\*创建bootstrap\*\*
\[root@k8s-master01 \~\]# cd /root/k8s-ha
install/bootstrap
\[root@k8s-master01 bootstrap\]# kubectl config set
cluster kubernetes \\
\> --certificate
authority=/etc/kubernetes/pki/ca.pem \\
\> --embed-certs=true \\
\> --server=https://192.168.15.88:8443 \\
\> --kubeconfig=/etc/kubernetes/bootstrap
kubelet.kubeconfig
Cluster "kubernetes" set.
\[root@k8s-master01 bootstrap\]# kubectl config set
credentials tls-bootstrap-token-user \\
\> --token=c8ad9c.2e4d610cf3e7426e \\
\> --kubeconfig=/etc/kubernetes/bootstrap
kubelet.kubeconfig
User "tls-bootstrap-token-user" set.
\[root@k8s-master01 bootstrap\]# kubectl config set
context tls-bootstrap-token-user@kubernetes \\
\> --cluster=kubernetes \\
\> --user=tls-bootstrap-token-user \\
\> --kubeconfig=/etc/kubernetes/bootstrap
kubelet.kubeconfigContext "tls-bootstrap-token-user@kubernetes"
created.
\[root@k8s-master01 bootstrap\]# kubectl config use
context tls-bootstrap-token-user@kubernetes --
kubeconfig=/etc/kubernetes/bootstrap
kubelet.kubeconfig
Switched to context "tls-bootstrap-token
user@kubernetes".
\[root@k8s-master01 bootstrap\]# mkdir -p /root/.kube
; cp /etc/kubernetes/admin.kubeconfig
/root/.kube/config
\[root@k8s-master01 bootstrap\]#
**可以正常查询集群状态，才可以继续往下，否则不⾏，需要排查k8s**
**组件是否有故障**
\[root@k8s-master01 bootstrap\]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy ok
\[root@k8s-master01 bootstrap\]# kubectl create -f
bootstrap.secret.yaml
secret/bootstrap-token-c8ad9c created第⼋章 Node节点配置
**8.1 复制证书**
**Master01节点\***\*复制证书⾄Node节点\*\*
clusterrolebinding.rbac.authorization.k8s.io/kubele
t-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node
autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node
autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube
apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system
:kube-apiserver created
\[root@k8s-master01 bootstrap\]#
\[root@k8s-master01 \~\]# cd /etc/kubernetes/
\[root@k8s-master01 kubernetes\]# for NODE in k8s
master02 k8s-master03 k8s-node01 k8s-node02; do
\> ssh NODE mkdir -p /etc/kubernetes/pki> for FILE in pki/ca.pem pki/ca-key.pem
pki/front-proxy-ca.pem bootstrap
kubelet.kubeconfig; do
> scp /etc/kubernetes/FILE
NODE:/etc/kubernetes/${FILE}
> done
> done
ca.pem 100%
1411 611.5KB/s 00:00
ca-key.pem 100%
1675 560.3KB/s 00:00
front-proxy-ca.pem 100%
1143 738.2KB/s 00:00
bootstrap-kubelet.kubeconfig 100%
2301 598.7KB/s 00:00
ca.pem 100%
1411 529.5KB/s 00:00
ca-key.pem 100%
1675 417.6KB/s 00:00
front-proxy-ca.pem 100%
1143 307.8KB/s 00:00
bootstrap-kubelet.kubeconfig 100%
2301 603.5KB/s 00:00
ca.pem 100%
1411 404.2KB/s 00:00
ca-key.pem 100%
1675 515.8KB/s 00:00 8.2 Kubelet配置
所有节点**创建相关⽬录**
所有节点**配置kubelet service**
front-proxy-ca.pem 100%
1143 380.2KB/s 00:00
bootstrap-kubelet.kubeconfig 100%
2301 550.5KB/s 00:00
ca.pem 100%
1411 645.2KB/s 00:00
ca-key.pem 100%
1675 1.4MB/s 00:00
front-proxy-ca.pem 100%
1143 1.2MB/s 00:00
bootstrap-kubelet.kubeconfig 100%
2301 760.2KB/s 00:00
root@k8s-master01 kubernetes#
root@k8s-master01 \~# mkdir -p /var/lib/kubelet
/var/log/kubernetes
/etc/systemd/system/kubelet.service.d
/etc/kubernetes/manifests/
root@k8s-master01 \~#root@k8s-master01 \~# vim
/usr/lib/systemd/system/kubelet.service
Unit
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kuberne
tes
Service
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
Install
WantedBy=multi-user.target
所有节点**配置kubelet service的配置⽂件**
（也可以写到kubelet.service）root@k8s-master01 \~# vim
/etc/systemd/system/kubelet.service.d/10-
kubelet.conf
Service
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap
kubeconfig=/etc/kubernetes/bootstrap
kubelet.kubeconfig --
kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--container
runtime
endpoint=unix:///run/containerd/containerd.sock"
Environment="KUBELET_CONFIG_ARGS=--
config=/etc/kubernetes/kubelet-conf.yml"
Environment="KUBELET_EXTRA_ARGS=--node
labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet
KUBELET_KUBECONFIG_ARGS KUBELET_CONFIG_ARGS
KUBELET_SYSTEM_ARGS KUBELET_EXTRA_ARGS
所有节点**创建kubelet的配置⽂件**注意：如果更改了k8s的service⽹段，需要更改kubelet-conf.yml 的
clusterDNS:配置，改成k8s Service⽹段的第⼗个地址，⽐如
10.96.0.10
root@k8s-master01 \~# vim /etc/kubernetes/kubelet
conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
root@k8s-master01 \~#
启动**所有节点kubelet**root@k8s-master01 \~# systemctl daemon-reload
root@k8s-master01 \~# systemctl enable --now
kubelet
Created symlink /etc/systemd/system/multi
user.target.wants/kubelet.service "
/usr/lib/systemd/system/kubelet.service.
root@k8s-master01 \~# systemctl status kubelet
此时系统⽇志/var/log/messages显示只有如下两种信息为正常，安
装calico后即可恢复
Unable to update cni config: no networks found in /etc/cni/net.d
如果有很多报错⽇志，或者有⼤量看不懂的报错，说明kubelet的配
置有误，需要检查kubelet配置
查看集群状态(Ready 或 NotReady 都正常)8.3 kube-proxy配置
注意，如果不是⾼可⽤集群，192.168.15.88:8443改为master01的地
址，8443改为apiserver的端⼝，默认是6443
Master01**执⾏**
root@k8s-master01 \~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady <none> 2m31s v1.28.0
k8s-master02 NotReady <none> 2m31s v1.28.0
k8s-master03 NotReady <none> 2m31s v1.28.0
k8s-node01 NotReady <none> 2m31s v1.28.0
k8s-node02 NotReady <none> 2m31s v1.28.0
root@k8s-master01 \~#
root@k8s-master01 \~# cd /root/k8s-ha-install/pki
root@k8s-master01 pki# cfssl gencert \
> -ca=/etc/kubernetes/pki/ca.pem \
> -ca-key=/etc/kubernetes/pki/ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> kube-proxy-csr.json | cfssljson -bare
/etc/kubernetes/pki/kube-proxy2023/09/01 20:58:25 INFO generate received
request
2023/09/01 20:58:25 INFO received CSR
2023/09/01 20:58:25 INFO generating key: rsa-2048
2023/09/01 20:58:25 INFO encoded CSR
2023/09/01 20:58:25 INFO signed certificate with
serial number
409810819523538273888266911652028641582986884038
2023/09/01 20:58:25 WARNING This certificate
lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline
Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the
CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information
Requirements").
root@k8s-master01 pki#
注意，如果不是⾼可⽤集群，192.168.100.69:8443改为master01的
地址，8443改为apiserver的端⼝，默认是6443
Master01**执⾏**
root@k8s-master01 pki# kubectl config set-cluster
kubernetes \> --certificate
authority=/etc/kubernetes/pki/ca.pem \
> --embed-certs=true \
> --server=https://192.168.15.88:8443 \
> --kubeconfig=/etc/kubernetes/kube
proxy.kubeconfig
Cluster "kubernetes" set.
root@k8s-master01 pki# kubectl config set
credentials system:kube-proxy \
> --client
certificate=/etc/kubernetes/pki/kube-proxy.pem \
> --client-key=/etc/kubernetes/pki/kube-proxy
key.pem \
> --embed-certs=true \
> --kubeconfig=/etc/kubernetes/kube
proxy.kubeconfig
User "system:kube-proxy" set.
root@k8s-master01 pki# kubectl config set-context
system:kube-proxy@kubernetes \
> --cluster=kubernetes \
> --user=system:kube-proxy \
> --kubeconfig=/etc/kubernetes/kube
proxy.kubeconfig
Context "system:kube-proxy@kubernetes" created.
root@k8s-master01 pki# kubectl config use-context
system:kube-proxy@kubernetes \> --kubeconfig=/etc/kubernetes/kube
proxy.kubeconfig
Switched to context "system:kube-proxy@kubernetes".
root@k8s-master01 pki#
Master01**将kubeconfig发送⾄其他节点**root@k8s-master01 pki# for NODE in k8s-master02
k8s-master03; do
> scp /etc/kubernetes/kube-proxy.kubeconfig
NODE:/etc/kubernetes/kube-proxy.kubeconfig \> done kube-proxy.kubeconfig 100% 6482 1.7MB/s 00:00 kube-proxy.kubeconfig 100% 6482 218.3KB/s 00:00 \[root@k8s-master01 pki\]# for NODE in k8s-node01 k8s-node02; do \> scp /etc/kubernetes/kube-proxy.kubeconfig NODE:/etc/kubernetes/kube-proxy.kubeconfig
> done
kube-proxy.kubeconfig 100%
6482 1.3MB/s 00:00
kube-proxy.kubeconfig 100%
6482 2.2MB/s 00:00
root@k8s-master01 pki#
所有节点**添加kube-proxy的配置和service⽂件：**
root@k8s-master01 pki# vim
/usr/lib/systemd/system/kube-proxy.serviceUnit
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kuberne
tes
After=network.target
Service
ExecStart=/usr/local/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy.yaml \
--v=2
Restart=always
RestartSec=10s
Install
WantedBy=multi-user.target
如果更改了集群Pod的⽹段，需要**更改kube-proxy.yaml的
clusterCIDR为更改后的Pod⽹段：**
root@k8s-master01 pki# vim /etc/kubernetes/kube
proxy.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection: acceptContentTypes: ""
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
qps: 5
clusterCIDR: 172.16.0.0/16
configSyncPeriod: 15m0s
conntrack:
max: null
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
syncPeriod: 30s
ipvs:
masqueradeAll: true
minSyncPeriod: 5s
scheduler: "rr"
syncPeriod: 30s
kind: KubeProxyConfigurationmetricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
root@k8s-master01 pki#
所有节点**启动kube-proxy**
root@k8s-master01 pki# systemctl daemon-reload
root@k8s-master01 pki# systemctl enable --now
kube-proxy
Created symlink /etc/systemd/system/multi
user.target.wants/kube-proxy.service "
/usr/lib/systemd/system/kube-proxy.service.
root@k8s-master01 pki#
此时系统⽇志/var/log/messages显示只有如下两种信息为正常，安
装calico后即可恢复
Unable to update cni config: no networks found in /etc/cni/net.d第九章 安装Calico
9.1 安装官⽅推荐版本
master01**执⾏**
master01**更改calico的⽹段，主要需要将红⾊部分的⽹段，改为⾃
⼰的Pod⽹段**
确保这个⽹段是⾃⼰的Pod⽹段
root@k8s-master01 pki# cd /root/k8s-ha
install/calico/
root@k8s-master01 calico# ls
calico.yaml
root@k8s-master01 calico# sed -i
"s#POD_CIDR#172.16.0.0/16#g" calico.yaml
root@k8s-master01 calico# grep "IPV4POOL_CIDR"
calico.yaml -A 1
- name: CALICO_IPV4POOL_CIDR
value: "172.16.0.0/16"
root@k8s-master01 calico# kubectl apply -f
calico.yamlmaster01**查看容器状态**
root@k8s-master01 calico# kubectl get po -n kube
system
NAME READY
STATUS RESTARTS AGE
calico-kube-controllers-6d48795585-7bpnl 1/1
Running 0 95s
calico-node-k27fv 1/1
Running 0 95s
calico-node-nzgms 1/1
Running 0 95s
calico-node-pfw59 1/1
Running 0 95s
calico-node-qrg2q 1/1
Running 0 95s
calico-node-wts56 1/1
Running 0 95s
root@k8s-master01 calico#
如果容器状态异常可以使⽤kubectl describe 或者kubectl logs查看
容器的⽇志第⼗章 安装CoreDNS
master01**安装官⽅推荐版本**
如果更改了k8s service的⽹段需要将coredns的serviceIP改成k8s
service⽹段的第⼗个IP（**master01 执⾏）**
Kubectl logs -f POD_NAME -n kube-system
Kubectl logs -f POD_NAME -c upgrade-ipam -n kube
system
root@k8s-master01 \~# cd /root/k8s-ha-install/
root@k8s-master01 k8s-ha-install#
root@k8s-master01 k8s-ha-install#
COREDNS_SERVICE_IP=`kubectl get svc | grep
kubernetes | awk '{print 3}'\`0 \[root@k8s-master01 k8s-ha-install\]# sed -i "s#KUBEDNS_SERVICE_IP#{COREDNS_SERVICE_IP}#g"
CoreDNS/coredns.yaml
root@k8s-master01 k8s-ha-install#master01**安装coredns**
第⼗⼀章 安装Metrics
Server
在新版的Kubernetes中系统资源的采集均使⽤Metrics-server，可以
通过Metrics采集节点和Pod的内存、磁盘、CPU和⽹络的使⽤率。
root@k8s-master01 k8s-ha-install# kubectl create
-f CoreDNS/coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredn
s created
clusterrolebinding.rbac.authorization.k8s.io/system
:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
root@k8s-master01 k8s-ha-install#master01**安装metrics server**
root@k8s-master01 k8s-ha-install# cd /root/k8s
ha-install/metrics-server
root@k8s-master01 metrics-server# kubectl create
-f .
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggreg
ated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metric
s-server created
rolebinding.rbac.authorization.k8s.io/metrics
server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metric
s-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system
:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k
8s.io created
root@k8s-master01 metrics-server#等待metrics server启动然后查看状态
root@k8s-master01 metrics-server# kubectl top
node
NAME CPU(cores) CPU% MEMORY(bytes)
MEMORY%
k8s-master01 436m 21% 936Mi
55%
k8s-master02 434m 21% 992Mi
58%
k8s-master03 160m 8% 929Mi
55%
k8s-node01 113m 5% 687Mi
40%
k8s-node02 108m 5% 770Mi
45%
root@k8s-master01 metrics-server#
如果有如下报错，可以等待10分钟后，再次查看：
Error from server (ServiceUnavailable): the server is currently
unable to handle the request (get nodes.metrics.k8s.io)
第⼗⼆章 安装dashboard第⼗⼆章 安装dashboard
12.1 Dashboard部署
Kubernetes Dashboard（仪表盘）是⼀个⽤于可视化管理和监控
Kubernetes 集群的 web UI ⼯具。它提供了⼀个⽤户友好的界⾯，⽤
于查看和管理集群中的应⽤程序、服务、节点等重要资源。
Dashboard⽤于展示集群中的各类资源，同时也可以通过Dashboard
实时查看Pod的⽇志和在容器中执⾏⼀些命令等。
12.1.1 安装指定版本dashboard
Master01 执⾏
root@k8s-master01 metrics-server# cd /root/k8s
ha-install/dashboard/
root@k8s-master01 dashboard# kubectl create -f .
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin
user created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created12.1.2 登录dashboard
Master01**更改dashboard的svc为NodePort**
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard
created
clusterrole.rbac.authorization.k8s.io/kubernetes
dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes
dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubern
etes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
root@k8s-master01 dashboard#
root@k8s-master01 dashboard# kubectl edit svc
kubernetes-dashboard -n kubernetes-dashboard将ClusterIP更改为NodePort（如果已经为NodePort忽略此步骤）
Master01**查看端⼝号：**
root@k8s-master01 dashboard# kubectl get svc
kubernetes-dashboard -n kubernetes-dashboard
NAME TYPE CLUSTER-IP
EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.96.120.157
<none> 443:30237/TCP 4m55s
root@k8s-master01 dashboard#
根据⾃⼰的实例端⼝号，通过任意安装了kube-proxy的宿主机的
IP+端⼝ 即可访问到dashboard：访问Dashboard：https://192.168.15.88:30237（请更改30237为⾃
⼰的端⼝），选择登录⽅式为令牌（即token⽅式）
Master01**创建登录Token：**
root@k8s-master01 dashboard# kubectl create token
admin-user -n kube-systemeyJhbGciOiJSUzI1NiIsImtpZCI6Im9LMEFwQS11Rzc1YU40Zkp
5QnVPY0p6akxiRktfZ0hNbWtZd1ZrMlh2UkUifQ.eyJhdWQiOls
iaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZX
IubG9jYWwiXSwiZXhwIjoxNjkzNTc4Nzg2LCJpYXQiOjE2OTM1N
zUxODYsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0
LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJ
uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW
50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiMzdiNWY2M
jEtOTEzNC00NGYwLWIyY2UtZWViZWFlZmZlYjlhIn19LCJuYmYi
OjE2OTM1NzUxODYsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3V
udDprdWJlLXN5c3RlbTphZG1pbi11c2VyIn0.TFzNWCVfudxyNl
mkbWrkc4d21bx4VqYmArvr3eUvNXl1DrKFUyDLUWMLyv7R
x1rHPb3avlJoA3Zn40LwiOHltV9blQypuHr52-
bbzWbv5uy7Pa_vPw9LMELwZ4Vf7E807LLkhu0tTPCaL_iLbSFrL
tZRkiJUGkVmfMZLeHAh4BzKi1DOxZhBCIgsbK
GTfQWSwn5bApwvD7fqMZTL4ou9oIq2CGELpp9eVEdwtqq80OKgi
LfBA9KKYtmAlhbXAib_G1uNV4tN3XfdwkQwYx2ZDEazQ06y5tGm
XlfcBIq4hH7rCN7Kl7Pvo3C0OEqucuJGHJ820uJyQ8yzqvPXzcr
A
root@k8s-master01 dashboard#
将token值输⼊到令牌后，单击登录即可访问Dashboard：第⼗三章 集群可⽤性验证
13.1 节点均正常13.2 Pod均正常
root@k8s-master01 \~# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready <none> 47m v1.28.0
k8s-master02 Ready <none> 47m v1.28.0
k8s-master03 Ready <none> 47m v1.28.0
k8s-node01 Ready <none> 47m v1.28.0
k8s-node02 Ready <none> 47m v1.28.0
root@k8s-master01 \~#
root@k8s-master01 \~# kubectl get po -A
NAMESPACE NAME
READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-
6d48795585-7bpnl 1/1 Running 1 (23m ago)
28m
kube-system calico-node-k27fv
1/1 Running 1 (23m ago)
28m
kube-system calico-node-nzgms
1/1 Running 1 (22m ago)
28m
kube-system calico-node-pfw59
1/1 Running 1 (22m ago)
28m13.3 集群⽹段⽆任何冲突
kube-system calico-node-qrg2q
1/1 Running 1 (23m ago)
28m
kube-system calico-node-wts56
1/1 Running 1 (22m ago)
28m
kube-system coredns-788958459b-gzrj7
1/1 Running 0 18m
kube-system metrics-server-8f77d49f6-
fhcbj 1/1 Running 0
17m
kubernetes-dashboard dashboard-metrics-scraper-
7b554c884f-6wlcz 1/1 Running 0
13m
kubernetes-dashboard kubernetes-dashboard-
54b699784c-8s77s 1/1 Running 0
13m
root@k8s-master01 \~#
root@k8s-master01 \~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP
PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none>
443/TCP 7h45m
root@k8s-master01 \~# kubectl get po -A -owideNAMESPACE NAME
READY STATUS RESTARTS AGE
IP NODE NOMINATED NODE
READINESS GATES
kube-system calico-kube-controllers-
6d48795585-7bpnl 1/1 Running 1 (23m ago)
29m 172.16.58.194 k8s-node02 <none>
<none>
kube-system calico-node-k27fv
1/1 Running 1 (23m ago)
29m 192.168.15.105 k8s-node02 <none>
<none>
kube-system calico-node-nzgms
1/1 Running 1 (23m ago)
29m 192.168.15.101 k8s-master01 <none>
<none>
kube-system calico-node-pfw59
1/1 Running 1 (23m ago)
29m 192.168.15.103 k8s-master03 <none>
<none>
kube-system calico-node-qrg2q
1/1 Running 1 (23m ago)
29m 192.168.15.104 k8s-node01 <none>
<none>kube-system calico-node-wts56
1/1 Running 1 (23m ago)
29m 192.168.15.102 k8s-master02 <none>
<none>
kube-system coredns-788958459b-gzrj7
1/1 Running 0 19m
172.16.32.129 k8s-master01 <none>
<none>
kube-system metrics-server-8f77d49f6-
fhcbj 1/1 Running 0
17m 172.16.85.193 k8s-node01 <none>
<none>
kubernetes-dashboard dashboard-metrics-scraper-
7b554c884f-6wlcz 1/1 Running 0
14m 172.16.195.1 k8s-master03 <none>
<none>
kubernetes-dashboard kubernetes-dashboard-
54b699784c-8s77s 1/1 Running 0
14m 172.16.122.129 k8s-master02 <none>
<none>
root@k8s-master01 \~#
13.4 能够正常创建资源13.4 能够正常创建资源
13.5 Pod必须能够解析Service（同
root@k8s-master01 \~# kubectl create deploy
cluster-test --image=registry.cn
beijing.aliyuncs.com/dotbalo/debug-tools -- sleep
3600
deployment.apps/cluster-test created
root@k8s-master01 \~# kubectl get po
NAME READY STATUS
RESTARTS AGE
cluster-test-66bb44bd88-gkq8m 1/1 Running 0
100s
root@k8s-master01 \~# kubectl get po -owide
NAME READY STATUS
RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
cluster-test-66bb44bd88-gkq8m 1/1 Running 0
103s 172.16.85.194 k8s-node01 <none>
<none>
root@k8s-master01 \~#13.5 Pod必须能够解析Service（同
namespace和跨namespace）
- nslookup kubernetes
可以解析到server的IP地址说明同namespace可以解析 - nslookup kube-dns.kube-system
root@k8s-master01 \~# kubectl exec -it cluster
test-66bb44bd88-gkq8m -- bash
(13:58 cluster-test-66bb44bd88-gkq8m:/) nslookup
kubernetes
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1 # 可以解析到server的IP地址说明同
namespace可以解析
(13:58 cluster-test-66bb44bd88-gkq8m:/)(13:58 cluster-test-66bb44bd88-gkq8m:/) nslookup
kube-dns.kube-system
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kube-dns.kube-system.svc.cluster.local
Address: 10.96.0.10 # 可以解析到server的第⼗个ip，说
明可以解析到kube-dns
(14:00 cluster-test-66bb44bd88-gkq8m:/)
可以解析到server的第⼗个ip，说明可以解析到kube-dns，说明跨
namespace也可解析
13.6 每个节点都必须能访问Kubernetes
的kubernetes svc 443 和kube-dns的
service 53root@k8s-master01 \~# curl https://10.96.0.1:443
curl: (60) SSL certificate problem: unable to get
local issuer certificate
More details here:
https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server
and therefore could not
establish a secure connection to it. To learn more
about this situation and
how to fix it, please visit the web page mentioned
above.
root@k8s-master01 \~#
root@k8s-master01 \~# curl 10.96.0.10:53
curl: (52) Empty reply from server
root@k8s-master01 \~#
13.7 Pod和Pod之间要能够正常通讯（同
namespace和跨namespace）
root@k8s-master01 \~# kubectl get po -nkube-system
-owide
NAME READY
STATUS RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATEScalico-kube-controllers-6d48795585-7bpnl 1/1
Running 1 (61m ago) 66m 172.16.58.194 k8s
node02 <none> <none>
calico-node-k27fv 1/1
Running 1 (61m ago) 66m 192.168.15.105 k8s
node02 <none> <none>
calico-node-nzgms 1/1
Running 1 (61m ago) 66m 192.168.15.101 k8s
master01 <none> <none>
calico-node-pfw59 1/1
Running 1 (61m ago) 66m 192.168.15.103 k8s
master03 <none> <none>
calico-node-qrg2q 1/1
Running 1 (61m ago) 66m 192.168.15.104 k8s
node01 <none> <none>
calico-node-wts56 1/1
Running 1 (61m ago) 66m 192.168.15.102 k8s
master02 <none> <none>
coredns-788958459b-gzrj7 1/1
Running 0 56m 172.16.32.129 k8s
master01 <none> <none>
metrics-server-8f77d49f6-fhcbj 1/1
Running 0 55m 172.16.85.193 k8s
node01 <none> <none>
root@k8s-master01 \~# kubectl get po -owideNAME READY STATUS
RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
cluster-test-66bb44bd88-gkq8m 1/1 Running 0
27m 172.16.85.194 k8s-node01 <none>
<none>
root@k8s-master01 \~# kubectl exec -it cluster
test-66bb44bd88-gkq8m -- bash
(14:18 cluster-test-66bb44bd88-gkq8m:/) ping
172.16.58.194 -c 3
PING 172.16.58.194 (172.16.58.194) 56(84) bytes of
data.
64 bytes from 172.16.58.194: icmp_seq=1 ttl=62
time=0.643 ms
64 bytes from 172.16.58.194: icmp_seq=2 ttl=62
time=1.59 ms
64 bytes from 172.16.58.194: icmp_seq=3 ttl=62
time=1.44 ms
--- 172.16.58.194 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss,
time 2024ms
rtt min/avg/max/mdev = 0.643/1.228/1.599/0.420 ms
(14:19 cluster-test-66bb44bd88-gkq8m:/)
进⼊node1节点的pod，ping测node2节点的pod可ping通
13.8 Pod和Pod之间要能够正常通讯（同13.8 Pod和Pod之间要能够正常通讯（同
机器和跨机器）
所有节点root@k8s-master01 \~# kubectl get po -owide
NAME READY STATUS
RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
cluster-test-66bb44bd88-gkq8m 1/1 Running 0
19m 172.16.85.194 k8s-node01 <none>
<none>
root@k8s-master01 \~# ping 172.16.85.194 -c 3 #
所有节点都要ping
PING 172.16.85.194 (172.16.85.194) 56(84) bytes of
data.
64 bytes from 172.16.85.194: icmp_seq=1 ttl=63
time=0.574 ms
64 bytes from 172.16.85.194: icmp_seq=2 ttl=63
time=0.262 ms
64 bytes from 172.16.85.194: icmp_seq=3 ttl=63
time=0.479 ms
--- 172.16.85.194 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss,
time 2041ms
rtt min/avg/max/mdev = 0.262/0.438/0.574/0.131 ms
root@k8s-master01 \~#