最新！银河麒麟v11 kubeadm部署k8s v1.35.0高可用集群

银河麒麟v11 kubeadm部署k8s v1.35.0高可用集群

系统环境

# nkvers 
############## Kylin Linux Version #################
Release:
Kylin Linux Advanced Server release V11 (Swan25)

Kernel:
6.6.0-32.7.v2505.ky11.x86_64

Build:
Kylin Linux Advanced Server
release V11 2503/(Swan25)-x86_64-Build20/20250715
#################################################

主机名	ip地址	配置	服务
demo-master-01	192.168.122.171	2c4g	k8s master、etcd、keepalived、haproxy、containerd
demo-master-02	192.168.122.172	2c4g	k8s master、etcd、keepalived、haproxy、containerd
demo-worker-01	192.168.122.173	2c4g	k8s worker、etcd

服务版本介绍

服务名称	版本
kubernetes	1.35.0
containerd	2.2.1
etcd	3.6.7
cni-plugins	1.3.0
runc	1.4.0
keepalived	2.3.4
haproxy	3.2.10
cilium	1.18.6

前言

此文档主要用于演示使用二进制kubeadm部署Kubernetes高可用集群。

此文档仅进行流程演示，后续用户可以使用ansible或编写Shell、Go脚本进行自动化配置，提高部署效率。

v1.35.0版本较之前版本还是有些差异和特性的，再加上新发布的国产服务器操作系统银河麒麟v11，相信看完会有新的收获！

环境准备

此步骤主要将一些部署k8s前的前置条件配置好，每个主机都要进行操作。

配置主机名，配置hosts解析

# 主机名
hostnamectl set-hostname demo-master-01
hostnamectl set-hostname demo-master-02
hostnamectl set-hostname demo-worker-01

# /etc/hosts
192.168.122.171 demo-master-01
192.168.122.172 demo-master-02
192.168.122.173 demo-worker-01

关闭防火墙，SElinux，swap分区

# 防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service

# SElinux，默认就是关闭状态，若是开启状态可执行以下命令
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

# swap分区
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab

配置时钟同步

# 配置时区
timedatectl set-timezone Asia/Shanghai

# 配置ntp服务器
vim /etc/chrony.conf
...
server ntp1.aliyun.com iburst
...

# 启动服务，并开启自启
systemctl enable --now chronyd.service

# 检查同步服务、同步状态
systemctl status chronyd.service
timedatectl 
chronyc sources -v 

内核配置

由于银河麒麟V11自带内核已为6.6版本，故不需再进行大版本的升级，仅需进行某些内核参数配置即可。

因为CNI插件准备采用cilium，直接替代kube-proxy，故不需加载ipvs模块，但由于k8s节点与keepalived服务在同一节点，故需加载ipvs模块。

另外银河麒麟的sysctl内核参数加载顺序，可参考前文：银河麒麟v10 sysctl内核参数加载顺序思考。

1. 配置CGroup版本为v2

由于k8s 1.35版本默认的cgroup版本就是v2，同时k8s官方也强烈建议使用cgroup v2，使用cgroup v2有以下几点要求：

• 操作系统发行版启用 CGroup v2
• Linux 内核为 5.8 或更高版本
• 容器运行时支持 CGroup v2。例如：containerd v1.4 和更高版本、cri-o v1.20 和更高版本
• kubelet 和容器运行时被配置为使用 systemd CGroup 驱动

查看当前默认cgroup版本：

stat -fc %T /sys/fs/cgroup/
tmpfs

tmpfs表示CGroup v1、cgroup2fs表示CGroup v2

修改内核cmdline参数，重启，使其默认支持CGroup v2：

vim /etc/default/grub

# 最后添加systemd.unified_cgroup_hierarchy=1参数
GRUB_CMDLINE_LINUX="... systemd.unified_cgroup_hierarchy=1"

重新生成GRUB配置文件，并重启服务器：

grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

再次验证cgroup版本：

stat -fc %T /sys/fs/cgroup/
cgroup2fs

2. 加载内核模块

# 配置模块
cat >/etc/modules-load.d/k8s.conf << EOF
overlay
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

# 将管理服务开机自启，并重新加载生效
systemctl enable --now systemd-modules-load
systemctl restart systemd-modules-load

# 临时加载某模块
modprobe overlay 
modprobe br_netfilter

3. 优化内核参数

将现有的/etc/sysctl.conf和/etc/sysctl.d/99-sysctl.conf文件中的net.ipv4.ip_forward配置都删除掉，统一配置到kubernetes的文件中：

# 删除原配置
sed -i '/net.ipv4.ip_forward/d' /etc/sysctl.conf /etc/sysctl.d/99-sysctl.conf

# 添加新配置
cat > /etc/sysctl.d/99-kubernetes-cri.conf << EOF
# ========== 网络相关（K8S 网络插件/容器通信核心） ==========
# 让桥接流量经过 iptables 规则（Calico/Flannel/CNI 插件必需）
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
# 开启 IP 转发（Pod 跨节点通信、Service 转发必需）
net.ipv4.ip_forward = 1
# 复用 TIME_WAIT 套接字，提升高并发网络性能
net.ipv4.tcp_tw_reuse = 1
# 增大套接字监听队列上限（避免高并发下连接被拒绝）
net.core.somaxconn = 65535
# 增大网络设备接收队列（提升网络吞吐）
net.core.netdev_max_backlog = 65535
# 增大 TCP SYN 队列（抵御 SYN 洪水，提升连接建立效率）
net.ipv4.tcp_max_syn_backlog = 65535
# 缩短 FIN_WAIT2 超时时间（快速释放资源）
net.ipv4.tcp_fin_timeout = 30
# 优化 ARP 缓存（大规模集群减少缓存回收频率）
net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 32768
net.ipv4.neigh.default.gc_thresh3 = 65536

# ========== Containerd/容器运行时相关 ==========
# 增大 inotify 实例数/监控文件数（避免 containerd 监控容器文件时达到上限）
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
# 增大系统最大文件句柄数（支持大规模容器，每个容器占用多个文件句柄）
fs.file-max = 52706963
# 增大单个进程最大打开文件数（配合 file-max）
fs.nr_open = 52706963
# 增大虚拟内存映射区数量（满足 containerd/容器内应用如 ES 的需求）
vm.max_map_count = 262144
# 禁用交换分区（K8S 要求，避免容器内存使用 swap 导致性能下降）
vm.swappiness = 0
# 增大最大 PID 数（支持大规模 Pod，每个 Pod 包含多个进程）
kernel.pid_max = 4194304

# ========== K8S 核心稳定性相关 ==========
# 内核 panic 后 10 秒重启（提升集群自愈能力）
kernel.panic = 10
# 内核出错时触发 panic（避免节点僵死）
kernel.panic_on_oops = 1
# 检测到软锁死时触发 panic（避免节点无响应）
kernel.softlockup_panic = 1
EOF

# 加载新配置
sysctl --system

# 检查配置是否生效
sysctl -a | grep -E 'ip_forward|nr_open'

4. 其他：内核升级（v11跳过）

另外，若是KylinOS V10 SP3的版本，可先进行内核升级，再进行上述参数的修改（跳过设置cgroup v2的步骤）。

KylinOS V10 SP3 2403默认的内核版本是4.19.90-89.15，目前此系统版本适配的内核最高版本是4.19.90-89.18，如有需求，可以进行升级，步骤如下。

注：由于KylinOS的内核具有高度定制性，故不建议使用Linux Kernel官方提供高版本内核的rpm包或源码编译包直接安装到KylinOS中，以免导致一些非必要的问题。

下载内核包

# mkdir kernel_rpm && cd kernel_rpm
# for i in kernel kernel-core kernel-devel kernel-headers kernel-modules kernel-modules-extra kernel-modules-internal; do wget https://update.cs2c.com.cn/NS/V10/V10SP3-2403/os/adv/lic/updates/x86_64/Packages/${i}-4.19.90-89.18.v2401.ky10.x86_64.rpm ;done

安装内核

# rpm -ivh kernel-4.19.90-89.18.v2401.ky10.x86_64.rpm kernel-core-4.19.90-89.18.v2401.ky10.x86_64.rpm  kernel-modules-4.19.90-89.18.v2401.ky10.x86_64.rpm kernel-modules-extra-4.19.90-89.18.v2401.ky10.x86_64.rpm kernel-modules-internal-4.19.90-89.18.v2401.ky10.x86_64.rpm
# rpm -Uvh kernel-devel-4.19.90-89.18.v2401.ky10.x86_64.rpm
# rpm -Uvh kernel-headers-4.19.90-89.18.v2401.ky10.x86_64.rpm

设定默认启动

# grub2-set-default kernel-4.19.90-89.18.v2401.ky10.x86_64.rpm

验证升级

# grub2-editenv list
# reboot
# nkvers

开始部署k8s集群

containerd安装

所有节点均需要操作安装。

cni插件安装

若服务器可代理上网，可直接执行以下脚本：

CNI_PLUGINS_VERSION="v1.3.0"
ARCH="amd64"
DEST="/opt/cni/bin"
sudo mkdir -p "$DEST"
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-${ARCH}-${CNI_PLUGINS_VERSION}.tgz" | sudo tar -C "$DEST" -xz

若不可代理上网，本文所需所有包及镜像均已上传至网盘，可到文末获取。

将对应包上传至服务器中：cni-plugins-linux-amd64-v1.3.0.tgz，执行：

mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

runc安装

wget https://github.com/opencontainers/runc/releases/download/v1.4.0/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc

containerd运行时安装

wget https://github.com/containerd/containerd/releases/download/v2.2.1/containerd-2.2.1-linux-amd64.tar.gz
tar Cxzvf /usr/local containerd-2.2.1-linux-amd64.tar.gz

配置systemd管理：vim /usr/lib/systemd/system/containerd.service

# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target dbus.service

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

加载systemd，并启动containerd

systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd.service

安装crictl客户端工具，配合管理containerd、k8s集群，可代理上网时，直接执行以下脚本：

DOWNLOAD_DIR="/usr/local/bin"
CRICTL_VERSION="v1.31.0"
ARCH="amd64"
curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz

否则，上传crictl-v1.31.0-linux-amd64.tar.gz，执行：

tar Cxzvf /usr/local/bin crictl-v1.31.0-linux-amd64.tar.gz

配置crictl：

cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true
pull-image-on-create: false
EOF

安装nerdctl工具，用于containerd本身的容器管理更方便：

# 下载
wget https://github.com/containerd/nerdctl/releases/download/v2.2.1/nerdctl-2.2.1-linux-amd64.tar.gz
# 解压
tar xf nerdctl-2.2.1-linux-amd64.tar.gz -C /usr/local/bin/
# 运行查看
nerdctl images

配置/etc/containerd/config.toml：

containerd config default > /etc/containerd/config.toml

vim /etc/containerd/config.toml
# 配置 systemd cgroup 驱动
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  ...
  [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
    SystemdCgroup = true

# 修改 sandbox 容器镜像仓库
    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = 'registry.aliyuncs.com/google_containers/pause:3.10.1'

配置完成，重启containerd服务：systemctl restart containerd.service

kubeadm/kubelet/kubectl安装

所有节点均需要操作安装。

kubeadm/kubelet install

可代理上网，直接执行以下脚本：

DOWNLOAD_DIR="/usr/local/bin"
RELEASE="v1.35.0"
ARCH="amd64"
cd $DOWNLOAD_DIR
sudo curl -L --remote-name-all https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet}
sudo chmod +x {kubeadm,kubelet}

RELEASE_VERSION="v0.16.2"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubelet/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service
sudo mkdir -p /usr/lib/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

不可代理上网，上传kubeadm、kubelet文件至服务器，随后执行：

chmod +x kubeadm kubelet 
mv kubeadm kubelet /usr/local/bin/

配置systemd：vim /usr/lib/systemd/system/kubelet.service

[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

编辑kubelet管理配置文件：

mkdir -p /usr/lib/systemd/system/kubelet.service.d

vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

启动kubelet，查看状态：

systemctl enable --now kubelet
systemctl status kubelet

kubectl install

上传kubectl至服务器，并执行：

install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client

配置命令补全

yum install bash-completion
source /usr/share/bash-completion/bash_completion

# kubectl补全
source <(kubectl completion bash)
kubectl completion bash >/etc/bash_completion.d/kubectl

# crictl补全
source <(crictl completion bash)
crictl completion bash >/etc/bash_completion.d/crictl

# nerctl补全
source <(nerdctl completion bash)
nerdctl completion bash >/etc/bash_completion.d/nerdctl

高可用etcd安装

本节安装3节点高可用集群

1. 运行以下脚本，为每个etcd主机创建kubeadm配置文件

# 使用你的主机 IP 更新 HOST0、HOST1 和 HOST2 的 IP 地址
export HOST0=192.168.122.171
export HOST1=192.168.122.172
export HOST2=192.168.122.173

# 使用你的主机名更新 NAME0、NAME1 和 NAME2
export NAME0="demo-master-01"
export NAME1="demo-master-02"
export NAME2="demo-worker-01"

# 创建临时目录来存储将被分发到其它主机上的文件
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

HOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=(${NAME0} ${NAME1} ${NAME2})

for i in "${!HOSTS[@]}"; do
HOST=${HOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
---
apiVersion: "kubeadm.k8s.io/v1beta4"
kind: InitConfiguration
nodeRegistration:
    name: ${NAME}
localAPIEndpoint:
    advertiseAddress: ${HOST}
---
apiVersion: "kubeadm.k8s.io/v1beta4"
kind: ClusterConfiguration
etcd:
    local:
        serverCertSANs:
        - "${HOST}"
        peerCertSANs:
        - "${HOST}"
        extraArgs:
        - name: initial-cluster
          value: ${NAMES[0]}=https://${HOSTS[0]}:2380,${NAMES[1]}=https://${HOSTS[1]}:2380,${NAMES[2]}=https://${HOSTS[2]}:2380
        - name: initial-cluster-state
          value: new
        - name: name
          value: ${NAME}
        - name: listen-peer-urls
          value: https://${HOST}:2380
        - name: listen-client-urls
          value: https://${HOST}:2379
        - name: advertise-client-urls
          value: https://${HOST}:2379
        - name: initial-advertise-peer-urls
          value: https://${HOST}:2380
EOF
done

2. 生成CA证书

kubeadm init phase certs etcd-ca

# 会生成这两个文件
ls /etc/kubernetes/pki/etcd
ca.crt  ca.key

3. 为每个etcd主机创建证书

# 使用你的主机 IP 更新 HOST0、HOST1 和 HOST2 的 IP 地址
export HOST0=192.168.122.171
export HOST1=192.168.122.172
export HOST2=192.168.122.173

kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
# 清理不可重复使用的证书
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# 不需要移动 certs 因为它们是给 HOST0 使用的

# 清理不应从此主机复制的证书
find /tmp/${HOST2} -name ca.key -type f -delete
find /tmp/${HOST1} -name ca.key -type f -delete

4. 复制证书和 kubeadm 配置到目标主机

USER=root

# demo-master-02
scp -r /tmp/${HOST1}/* ${USER}@${HOST1}:
ssh ${USER}@${HOST1}
root@HOST $ chown -R root:root pki
root@HOST $ mkdir /etc/kubernetes
root@HOST $ mv pki /etc/kubernetes/

# demo-worker-01
scp -r /tmp/${HOST2}/* ${USER}@${HOST2}:
ssh ${USER}@${HOST2}
root@HOST $ chown -R root:root pki
root@HOST $ mkdir /etc/kubernetes
root@HOST $ mv pki /etc/kubernetes/

5. 检查目标主机证书

# demo-master-01
tree /tmp/192.168.122.171/ /etc/kubernetes/pki
/tmp/192.168.122.171/
└── kubeadmcfg.yaml
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
    ├── ca.crt
    ├── ca.key
    ├── healthcheck-client.crt
    ├── healthcheck-client.key
    ├── peer.crt
    ├── peer.key
    ├── server.crt
    └── server.key

# demo-master-02
tree /root/ /etc/kubernetes/pki
/root/
└── kubeadmcfg.yaml
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
    ├── ca.crt
    ├── healthcheck-client.crt
    ├── healthcheck-client.key
    ├── peer.crt
    ├── peer.key
    ├── server.crt
    └── server.key

# demo-worker-01
tree /root/ /etc/kubernetes/pki
/root/
└── kubeadmcfg.yaml
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
    ├── ca.crt
    ├── healthcheck-client.crt
    ├── healthcheck-client.key
    ├── peer.crt
    ├── peer.key
    ├── server.crt
    └── server.key

6. 安装及配置etcd服务

注：所有etcd主机均需操作，yum源自带etcd服务版本为3.4.14，而k8s1.35要求etcd不低于3.5.24-0。

# 下载
wget https://github.com/etcd-io/etcd/releases/download/v3.6.7/etcd-v3.6.7-linux-amd64.tar.gz
# 解压缩
tar xf etcd-v3.6.7-linux-amd64.tar.gz && cd etcd-v3.6.7-linux-amd64
# 编辑运行环境
cp etcdctl etcdutl etcd /usr/bin/
mkdir /var/lib/etcd /etc/etcd && chown -R etcd:etcd /var/lib/etcd && chmod 700 /var/lib/etcd/ && chown -R etcd /etc/kubernetes/pki/etcd

vim /etc/etcd/etcd.conf

# etcd节点名称，要和ETCD_INITIAL_CLUSTER中的名称一致
ETCD_NAME=demo-master-01

#数据存储目录
ETCD_DATA_DIR="/var/lib/etcd"

# 服务的监听和通信地址与端口
ETCD_LISTEN_PEER_URLS="https://192.168.122.171:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.122.171:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.122.171:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.122.171:2379"

# 集群所有的成员
ETCD_INITIAL_CLUSTER="demo-master-01=https://192.168.122.171:2380,demo-master-02=https://192.168.122.172:2380,demo-worker-01=https://192.168.122.173:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

# 证书配置
ETCD_CERT_FILE="/etc/kubernetes/pki/etcd/server.crt"
ETCD_KEY_FILE="/etc/kubernetes/pki/etcd/server.key"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/pki/etcd/ca.crt"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/kubernetes/pki/etcd/peer.crt"
ETCD_PEER_KEY_FILE="/etc/kubernetes/pki/etcd/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/pki/etcd/ca.crt"
ETCD_PEER_AUTO_TLS="true"

3台etcd的配置文件主体内容相同，仅ETCD_NAME及监听的地址需要根据实际情况修改，例如demo-master-02主机需更改为ETCD_NAME=demo-master-02，并将所有ip地址更改为192.168.122.172。

vim /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

7. 启动服务，并检查etcd集群状态

systemctl daemon-reload
# 所有节点，尽量同时间启动
systemctl enable --now etcd

检查集群状态：

export HOST0=192.168.122.171

ETCDCTL_API=3 etcdctl \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://${HOST0}:2379 endpoint health

配置高可用

配置keepalived and haproxy

第1-4步在两台机器的操作是相同的，第5步需差异配置。

1. 下载源码到本地

wget https://www.keepalived.org/software/keepalived-2.3.4.tar.gz
wget https://www.haproxy.org/download/3.2/src/haproxy-3.2.10.tar.gz

2. 编译安装keepalived

tar xf keepalived-2.3.4.tar.gz && cd keepalived-2.3.4
yum -y install openssl-devel gcc 
./configure --prefix=/usr/local/keepalived-2.3.4
make && make install

# 配置软连接
ln -sv /usr/local/keepalived-2.3.4/sbin/keepalived /usr/sbin/
ln -sv /usr/local/keepalived-2.3.4/etc/keepalived /etc/keepalived

2. 编译安装haproxy

tar xf haproxy-3.2.10.tar.gz && cd haproxy-3.2.10
yum -y install zlib-devel
make -j 2 TARGET=generic USE_ZLIB=1 USE_OPENSSL=1
make install PREFIX=/usr/local/haproxy-3.2.10

# 配置软连接
ln -sv /usr/local/haproxy-3.2.10/sbin/haproxy /usr/sbin/
mkdir /usr/local/haproxy-3.2.10/conf && ln -sv /usr/local/haproxy-3.2.10/conf /etc/haproxy && mkdir /etc/haproxy/conf.d

3. 配置systemd管理单元

vim /usr/lib/systemd/system/haproxy.service

[Unit]
Description=HAProxy Load Balancer
After=network-online.target            
Wants=network-online.target

[Service]
EnvironmentFile=-/etc/sysconfig/haproxy
Environment="HAPROXY_CONF=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" "CFGDIR=/etc/haproxy/conf.d"
ExecStartPre=/usr/sbin/haproxy -f $HAPROXY_CONF -f $CFGDIR -c
ExecStart=/usr/sbin/haproxy -Ws -f $HAPROXY_CONF -f $CFGDIR -p $PIDFILE
ExecReload=/usr/sbin/haproxy -f $HAPROXY_CONF -f $CFGDIR -c
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
SuccessExitStatus=143
Type=notify

[Install]
WantedBy=multi-user.target

vim /usr/lib/systemd/system/keepalived.service

[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/usr/local/keepalived-2.3.4/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived-2.3.4/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
StartLimitBurst=1
StartLimitInterval=1s

[Install]
WantedBy=multi-user.target

加载管理单元配置文件：

systemctl daemon-reload

4. 配置haproxy的kube-apiserver负载均衡

vim /etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log stdout format raw local0
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          35s
    timeout server          35s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:16443
    mode tcp
    option tcplog
    default_backend apiserverbackend

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserverbackend
    option httpchk

    http-check connect ssl
    http-check send meth GET uri /healthz
    http-check expect status 200

    mode tcp
    balance     roundrobin
    
    server apiserver1 192.168.122.171:6443 check verify none
    server apiserver2 192.168.122.172:6443 check verify none
    # [...]

5. 配置keepalived的kube-apiserver高可用

先在两台keepalived服务器将健康检测脚本准备好：vim /etc/keepalived/check_apiserver.sh

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl -sfk --max-time 2 https://localhost:16443/healthz -o /dev/null || errorExit "Error GET https://localhost:16443/healthz"

https://localhost:16443/healthz地址端口改为haproxy ip和监听的apiserver端口

由于此时kube-apiserver还未进行部署，6443端口还未被监听，所以现在仅用pgrep haproxy检查进程存活，不检测 HAProxy 后端状态：

demo-master-01下：vim /etc/keepalived/keepalived.conf

global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}
vrrp_script check_haproxy {
    script "/usr/bin/pgrep haproxy"  # 仅检查 haproxy 进程是否存在
    interval 2                       # 每 2 秒检查一次
    weight 2                         # 检查成功时权重+2（非必需，仅用于优先级调整）
    fall 1                           # 1 次检查失败即判定进程异常
    rise 1                           # 1 次检查成功即恢复
}

vrrp_instance VI_1 {
    state MASTER
    interface enp1s0
    virtual_router_id 51
    priority 101
    authentication {
        auth_type PASS
        auth_pass 42
    }
    virtual_ipaddress {
        192.168.122.170
    }
    track_script {
        check_haproxy
    }
}

demo-master-02下：vim /etc/keepalived/keepalived.conf

global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}
vrrp_script check_haproxy {
    script "/usr/bin/pgrep haproxy"  # 仅检查 haproxy 进程是否存在
    interval 2                       # 每 2 秒检查一次
    weight 2                         # 检查成功时权重+2（非必需，仅用于优先级调整）
    fall 1                           # 1 次检查失败即判定进程异常
    rise 1                           # 1 次检查成功即恢复
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp1s0
    virtual_router_id 51
    priority 100
    authentication {
        auth_type PASS
        auth_pass 42
    }
    virtual_ipaddress {
        192.168.122.170
    }
    track_script {
        check_haproxy
    }
}

6. 至此haproxy+keepalived的部署和配置完毕，可以暂时先不启动服务，待K8S控制节点部署完成后再进行启动。
7. 启动haproxy和keepalived，验证vip是否监听

systemctl enable --now haproxy.service
systemctl enable --now keepalived.service

ip a
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:20:d9:60 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.171/24 brd 192.168.122.255 scope global noprefixroute enp1s0
       valid_lft forever preferred_lft forever
    inet 192.168.122.170/32 scope global proto keepalived enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe20:d960/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

此时可在主节点看到VIP

部署集群

拉取镜像

kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers

# crictl images 
IMAGE                                                             TAG                 IMAGE ID            SIZE
registry.aliyuncs.com/google_containers/coredns                   v1.13.1             aa5e3ebc0dfed       23.6MB
registry.aliyuncs.com/google_containers/etcd                      3.6.6-0             0a108f7189562       23.6MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.35.0             5c6acd67e9cd1       27.7MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.35.0             2c9a4b058bd7e       23.1MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.35.0             32652ff1bbe6b       25.8MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.35.0             550794e3b12ac       17.2MB
registry.aliyuncs.com/google_containers/pause                     3.10.1              cd073f4c5f6a8       320kB

初始化kubeadm配置

kubeadm config print init-defaults > kubeadm-init.yaml

有些需要自定义更改的参数与值：

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
# apiserver的地址与端口
localAPIEndpoint:
  advertiseAddress: 192.168.122.171
  bindPort: 6443
# 节点名称
  name: demo-master-01
  taints: null
---
apiVersion: kubeadm.k8s.io/v1beta4
# 证书有效期，有需求可以改为10年
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
# 外部etcd地址配置，若此处不指定，则k8s会自己创建单节点etcd
etcd:
  external:
    endpoints:
    - https://192.168.122.171:2379
    - https://192.168.122.172:2379
    - https://192.168.122.173:2379
    caFile: "/etc/kubernetes/pki/etcd/ca.crt"
    certFile: "/etc/kubernetes/pki/apiserver-etcd-client.crt"
    keyFile: "/etc/kubernetes/pki/apiserver-etcd-client.key"
# 镜像仓库地址更改
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.35.0
# svc的网段与pod的网段指定，与CNI网络插件挂钩
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/16
  podSubnet: 10.98.0.0/16
# 连接控制平面的地址，指定为vip地址和haproxy的监听端口
controlPlaneEndpoint: "192.168.122.170:16443"

仅保留了需要注意且更改的字段。

kubeadm创建集群

1. 初始化集群

kubeadm init --config=./kubeadm-init.yaml --upload-certs

kubeadm init --skip-phases=addon/kube-proxy，跳过kube-proxy组件的安装

2. 根据提示，执行kubeconfig配置

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. 根据提示，加入新的控制节点，在demo-master-02执行

kubeadm join 192.168.122.170:16443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:6d07b2c99b32777a816247f22bd36c52a9bba7a2547c9025f7be24073fdb6023 \
        --control-plane --certificate-key 0bbadf90b312676417c686ac4437800ca251a2e216f2aa9da02810bc66cf8325

4. 根据提示，加入新的worker节点，在demo-worker-01执行

kubeadm join 192.168.122.170:16443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:6d07b2c99b32777a816247f22bd36c52a9bba7a2547c9025f7be24073fdb6023

5. 查看集群状态

# kubectl get nodes 
NAME             STATUS     ROLES           AGE     VERSION
demo-master-01   NotReady   control-plane   4m30s   v1.35.0
demo-master-02   NotReady   control-plane   103s    v1.35.0
demo-worker-01   NotReady   <none>          30s     v1.35.0

# kubectl get pod -n kube-system 
NAME                                     READY   STATUS    RESTARTS   AGE
coredns-bbdc5fdf6-8qmkt                  0/1     Pending   0          21m
coredns-bbdc5fdf6-k8qhn                  0/1     Pending   0          21m
kube-apiserver-demo-master-01            1/1     Running   0          21m
kube-apiserver-demo-master-02            1/1     Running   0          19m
kube-controller-manager-demo-master-01   1/1     Running   0          21m
kube-controller-manager-demo-master-02   1/1     Running   0          19m
kube-proxy-f78gj                         1/1     Running   0          21m
kube-proxy-hwf9q                         1/1     Running   0          19m
kube-proxy-wxn5r                         1/1     Running   0          18m
kube-scheduler-demo-master-01            1/1     Running   0          21m
kube-scheduler-demo-master-02            1/1     Running   0          19m

未安装CNI插件，集群状态为NotReady

部署CNI插件cilium

cilium是目前最流行的CNI插件之一，其基于eBPF，性能高，且支持网络负载、策略配置、可观测一体化，低耗高效，可替代kube-proxy组件。

部署helm工具

wget https://get.helm.sh/helm-v4.1.0-linux-amd64.tar.gz
mv linux-amd64/helm /usr/bin/

# helm命令补全
source <(helm completion bash)
helm completion bash >/etc/bash_completion.d/helm

部署cilium chart

1. 添加cilium仓库

# 添加repo
helm repo add cilium https://helm.cilium.io/

# 查看版本
helm search repo cilium/cilium --versions | head

2. 获取并编辑chart

helm pull cilium/cilium --untar --version 1.18.6

修改几个关键变量值：

vim cilium/values.yaml

# 替换kube-Proxy组件
kubeProxyReplacement: "true"
# 指定apiserver地址，此处写VIP
k8sServiceHost: "192.168.122.170"
# 指定apiserver端口，此处写haproxy的负载端口
k8sServicePort: "16443"
# pod的ipam范围
clusterPoolIPv4PodCIDRList: ["10.98.0.0/16"]
# 每个node节点下pod网段的掩码位
clusterPoolIPv4MaskSize: 24

3. 提前拉取或准备所必须镜像

quay.io/cilium/cilium:v1.18.6
quay.io/cilium/cilium-envoy:v1.35.9-1767794330-db497dd19e346b39d81d7b5c0dedf6c812bcc5c9
quay.io/cilium/operator-generic:v1.18.6

4. 删除kube-proxy组件

kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy
# Run on each node with root permissions:
iptables-save | grep -v KUBE | iptables-restore

kubeadm init --skip-phases=addon/kube-proxy可以在init阶段直接跳过安装kube-proxy。

5. 部署cilium chart

helm install cilium cilium -f cilium/values.yaml --namespace kube-system

6. 查看pod状态

# kubectl get pod -n kube-system 
NAME                                     READY   STATUS    RESTARTS       AGE
cilium-8lqwz                             2/2     Running   0              52s
cilium-envoy-78rx5                       1/1     Running   0              52s
cilium-envoy-8ssc8                       1/1     Running   0              52s
cilium-envoy-sc92g                       1/1     Running   0              52s
cilium-nnl6k                             2/2     Running   0              52s
cilium-operator-698cfb64f9-d2p2n         1/1     Running   0              52s
cilium-operator-698cfb64f9-smdvt         1/1     Running   0              52s
cilium-vlx9m                             2/2     Running   0              52s

安装cilium cli工具

wget https://github.com/cilium/cilium-cli/releases/download/v0.19.0/cilium-linux-amd64.tar.gz
tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

查看cilium集群状态：

# cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet              cilium-envoy             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator          Desired: 2, Ready: 2/2, Available: 2/2
Containers:            cilium                   Running: 3
                       cilium-envoy             Running: 3
                       cilium-operator          Running: 2
                       clustermesh-apiserver    
                       hubble-relay             
Cluster Pods:          2/2 managed by Cilium

开启和管理hubble

1. 开启hubble进行网络观测：

# 开启hubble
cilium hubble enable
# 开启hubble ui
cilium hubble enable --ui

也可以在chart value.yaml中修改：hubble.relay.enabled=true、hubble.ui.enabled=true

2. pod启动完毕，修改svc暴露端口给外部访问

# kubectl get pod -n kube-system | grep hubble
hubble-relay-66495f87cb-78gpg            1/1     Running   0   16h
hubble-ui-7bcb645fcd-g99rm               2/2     Running   0   16h

# kubectl get svc -n kube-system | grep ui
hubble-ui      NodePort    10.96.75.161   <none>        80:32000/TCP             16h

3. 浏览器访问http://192.168.122.171:32000/就可以查看流量转发详情了！

检查k8s集群状态

# kubectl get nodes 
NAME             STATUS   ROLES           AGE     VERSION
demo-master-01   Ready    control-plane   4d18h   v1.35.0
demo-master-02   Ready    control-plane   4d18h   v1.35.0
demo-worker-01   Ready    <none>          4d18h   v1.35.0

到此，k8s高可用集群已搭建完毕，实际生产环境中，etcd、keepalived、haproxy应尽可能的选择单独的服务器进行部署，且在集群规模较大时，根据实际情况进行优化。

结语

本文以k8s官方的kubeadm工具构建高可用集群，且基本所有服务都是二进制安装的方式，此种方式通用性较强，若您是arm架构机器则需下载对应服务的arm版本的二进制文件即可，步骤几乎相同。目前市面上大多k8s集群自动化部署工具都是采用此种方式安装的。

本文的目的就是带领大家了解在国产的银河麒麟v11下部署高可用k8s的全过程，顺便理解自动化部署k8s工具的底层逻辑，当然，真正的生产环境大家肯定是优先选择自动化部署，银河麒麟v11官方也推出了自动化部署工具，大家可以尝试着使用一下~

后续会针对网络插件cilium进行一波网络走向抓包分析，来深入了解一下其原理，欢迎感兴趣的朋友关注。

最后放一下本次部署所使用到的所有安装包和镜像，点击这里获取。

参考链接：

关于cgroup v2使用与介绍

containerd官方部署手册

kubeadm、kubelet、kubectl官方部署手册

kubeadm集群安装官方引导手册

容器运行时cgroup驱动官方指导手册

cilium官方指导手册