Ubuntu与RedHat多操作系统安装Kubernetes

前置

说明

1. Kubernetes版本选择: 截止目前, Kubernetes最新版本为v1.28.2, 使用最新的版本进行安装
2. 容器运行时: 使用containerd作为CRI截止目前, containerd最新版本为v1.7.7
3. 网络插件: 使用kube-fannel

环境

使用最基本的1+2的机器组成集群, 以下是推荐的配置

Ubuntu和RedHat系列(Centos/RockyLinux)操作系统是最受欢迎的Linux发行版之一, 此示例加入不同的操作系统作为工作节点, 方便使用者参考

节点类型	IP	操作系统	内存	包管理器
master1	192.168.0.151	Ubuntu22.04	4Gi	apt
master2	192.168.0.152	Ubuntu22.04	4Gi	apt
worknode1	192.168.0.155	Ubuntu22.04	12Gi	apt
worknode2	192.168.0.156	Ubuntu22.04	12Gi	yum/dnf
worknode3	192.168.0.157	Centos7	12Gi	yum/dnf
worknode4	192.168.0.158	Centos7	12Gi	yum/dnf
worknode5	192.168.0.159	Rockylinux9	24Gi	yum/dnf
worknode6	192.168.0.160	Rockylinux9	24Gi	yum/dnf
worknode7	192.168.0.161	Rockylinux9	24Gi	yum/dnf

配置

修改hostname

方便区分node master节点命名为master1,以此类推

hostnamectl set-hostname master1

worknode命名为node1,以此类推

hostnamectl set-hostname node1

修改 hosts(推荐, 非必须)

所有节点都修改 hosts

echo "192.168.58.131 master" >> /etc/hosts
echo "192.168.58.135 node1" >> /etc/hosts
echo "192.168.58.136 node2" >> /etc/hosts

关闭 SELinux

所有节点关闭 SELinux

RedHat

setenforce 0 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

Ubuntu

sudo setenforce 0 
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/'    /etc/selinux/config

禁用交换分区

sed -i 's/^\(.*swap.*\)$/#\1/g' /etc/fstab

数据包转发

tee /etc/sysctl.d/kubernetes.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sysctl --system

配置IPVS(仅适用于RedHat)

yum install ipvsadm ipset sysstat conntrack libseccomp -y 
cat >> /etc/modules-load.d/ipvs.conf <<EOF 
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip 
EOF

systemctl restart systemd-modules-load.service

lsmod | grep -e ip_vs -e nf_conntrack

内核参数(仅适用于RedHat)

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384

net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
EOF

sysctl --system

安装容器运行时

所以节点均需安装与配置

下载

使用二进制containerd, 也可以使用其他方式安装, 安装配置即可

1. 下载, 如果机器无法访问github, 请自行下载并上传

wget https://github.com/containerd/containerd/releases/download/v1.7.7/containerd-1.7.7-linux-amd64.tar.gz

创建配置文件

mkdir -p /etc/containerd/
containerd config default | tee /etc/containerd/config.toml

修改配置文件

配置文件默认在/etc/containerd/config.toml 这里仅修改两处配置, 读者可以修改自己想要的配置 1. 修改registry.k8s.io/pause:3.8为registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 2. 修改SystemdCgroup = false为SystemdCgroup = true

! 必须注意, 截止2023.10.21, registry.k8s.io/pause的版本是3.8 请将shell命令修改你自己的版本所对应的版本 如果你跟着本教程的版本, 直接执行以下命令, 否则你需要手动修改配置文件

sed -i 's#registry.k8s.io/pause:3.8#registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9#g' /etc/containerd/config.toml 

sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml

systemctl restart containerd

修改crictl配置文件，获得containerd的sock信息

cat << EOF > /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false 
EOF

网络内核模块

tee /etc/modules-load.d/containerd.conf << EOF 
overlay 
br_netfilter 
EOF 

modprobe overlay 
modprobe br_netfilter

配置Kubernetes

安装

RedHat

1. 添加Kubernetes源(适用于国内服务器)
2. 安装Kubernetes
3. 把kubelet加入到自启列表并启动

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum install -y kubelet kubeadm kubectl

systemctl enable kubelet && systemctl start kubelet

Ubuntu

1. 添加Kubernetes源
2. 安装Kubernetes
3. 把kubelet加入到自启列表并启动

apt-get update && apt-get install -y apt-transport-https 
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list 
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main 
EOF

apt-get update

apt-get install -y kubelet kubeadm kubectl

sudo systemctl enable --now kubelet && systemctl start kubelet

初始化(Master节点)

apiserver-advertise-address需要使用本机上网卡的ip，否则的话会导致etcd绑定ip失败启动不了，从而apiserver也启动不了, 选择一个初始化方式执行即可

Shell方式

1. 把192.168.0.152替换为masterIP
2. 填写--kubernetes-version为你的Kubernetes版本号, 此示例版本为1.28.2

kubeadm init --kubernetes-version=1.28.2 --apiserver-advertise-address=192.168.0.152 --apiserver-bind-port=6443 --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///run/containerd/containerd.sock 

文件配置方式

1. 生成

kubeadm config print init-defaults > kube-config.yaml

2. 修改

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.16.21.135 # 改为你的Master IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.27.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16 #Pod的网段地址配置
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

3. 执行初始化

kubeadm init --config kube-config.yaml

初始化成功一般有如下输出:

[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.152:6443 --token mv274d.pnyhrlx36y6dle1b \
        --discovery-token-ca-cert-hash sha256:59b2b7da05ffe85d0686595a1a3e388f1bd403e045e85712c5884faf6cdf0ea7

复制kubeadm join ...整段, 保存到方便记忆的地方, 下文需要使用

创建配置文件

mkdir -p $HOME/.kube 
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装网络插件flannel

访问https://kubernetes.io/docs/concepts/cluster-administration/addons/ 获取flannel配置文件

如果访问不了github, 手动下载yaml配置文件并上传再使用 kubectl apply -f <filename> 命令

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

工作节点加入集群

将复制kubeadm join ...整段代码到工作节点上执行:

kubeadm join ...

检查集群状态

master执行

kubectl get no -owide

常见问题

1. 节点NotReady 检查所有Pod是否成功启动

kubectl get po -A

使用kubectl describe po/<pod-name> 查看Pod的具体问题, 具体问题具体处理

2. Pod出现异常: Failed to create pod sandbox: open /run/systemd/resolve/resolv.conf: no such file or directory 多为RedHat操作系统出现问题, 解决方案: 参考Failed to create pod sandbox与安装systemd-resolved Ubuntu与RedHat多操作系统安装Kubernetes