kubeadm升级k8s之1.23.17->1.24.17

查看当前版本

[root@k8s-master31 ~]# kubectl get nodes -o wide
NAME           STATUS   ROLES                  AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION                        CONTAINER-RUNTIME
k8s-master31   Ready    control-plane,master   47h   v1.23.17   10.0.0.31     <none>        openEuler 22.03 (LTS-SP1)   5.10.0-136.12.0.86.oe2203sp1.x86_64   docker://26.1.4
k8s-node34     Ready    <none>                 47h   v1.23.17   10.0.0.34     <none>        openEuler 22.03 (LTS-SP1)   5.10.0-136.12.0.86.oe2203sp1.x86_64   docker://26.1.4
k8s-node35     Ready    <none>                 47h   v1.23.17   10.0.0.35     <none>        openEuler 22.03 (LTS-SP1)   5.10.0-136.12.0.86.oe2203sp1.x86_64   docker://26.1.4

Master 节点操作

升级 kubeadm

[root@k8s-master31 ~]# yum install -y kubeadm-1.24.17-0 --disableexcludes=kubernetes
[root@k8s-master31 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.17", GitCommit:"22a9682c8fe855c321be75c5faacde343f909b04", GitTreeState:"clean", BuildDate:"2023-08-23T23:43:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master31 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0822 10:35:30.450616   51429 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/dockershim.sock". Please update your configuration!
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.23.17
[upgrade/versions] kubeadm version: v1.24.17
I0822 10:35:38.878235   51429 version.go:256] remote version is much newer: v1.31.0; falling back to: stable-1.24
[upgrade/versions] Target version: v1.24.17
[upgrade/versions] Latest version in the v1.23 series: v1.23.17
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT        TARGET
kubelet     3 x v1.23.17   v1.24.17
Upgrade to the latest stable version:
COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.23.17   v1.24.17
kube-controller-manager   v1.23.17   v1.24.17
kube-scheduler            v1.23.17   v1.24.17
kube-proxy                v1.23.17   v1.24.17
CoreDNS                   v1.8.6     v1.8.6
etcd                      3.5.6-0    3.5.6-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.24.17

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

修改节点 runtime

PS：安裝了docker 24版本，默認安裝了containerd
[root@k8s-master-01 ~]# kubectl edit nodes k8s-master31
apiVersion: v1
kind: Node
metadata:
  annotations:
    csi.volume.kubernetes.io/nodeid: '{"csi.tigera.io":"k8s-master31"}'
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
# K8s 1.23版本之前使用的runtime是/var/run/dockershim.sock
# 修改为/var/run/containerd/containerd.sock

配置containerd修改默认Cgroup驱动

[root@k8s-master31 ~]# containerd config default > /etc/containerd/config.toml
[root@k8s-master31 ~]# sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml

[root@k8s-master31 ~]# vim /var/lib/kubelet/kubeadm-flags.env
# 移除--network-plugin=cni 
# 添加--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.6 --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"

systemctl daemon-reload
#重启containerd
systemctl restart containerd
#重启kubelet
systemctl restart kubelet

在 Kubernetes 的较早版本中，--network-plugin 选项用于指定 Kubelet 应该使用的网络插件，例如 cni、kubenet 等。然而，从 Kubernetes v1.24 版本开始，dockershim（包括 kubenet）已被完全移除，同时许多与 dockershim 相关的标志也不再被支持。这可能是你遇到这个问题的原因。

定义 crictl 如何连接到容器运行时。

[root@k8s-master31 ~]# cat >/etc/crictl.yaml<<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

# 加载
systemctl daemon-reload
systemctl restart containerd

执行升级

[root@k8s-master31 ~]# kubeadm upgrade apply v1.24.17
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0822 10:54:49.815166   15201 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/containerd/containerd.sock". Please update your configuration!
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.24.17"
[upgrade/versions] Cluster version: v1.23.17
[upgrade/versions] kubeadm version: v1.24.17
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.24.17" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3701995785"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-08-22-10-55-29/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-08-22-10-55-29/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-08-22-10-55-29/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Removing the deprecated label node-role.kubernetes.io/master='' from all control plane Nodes. After this step only the label node-role.kubernetes.io/control-plane='' will be present on control plane Nodes.
[upgrade/postupgrade] Adding the new taint &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} to all control plane Nodes. After this step both taints &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} and &Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,} should be present on control plane Nodes.
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.24.17". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

升级Calico

# 查看当前集群配置
[root@k8s-master31 ~]# kubectl -n kube-system get cm kubeadm-config -o yaml

# 下载 Tigera Calico 操作器和自定义资源定义。
# 通过创建必要的自定义资源来安装 Calico。有关此清单中可用的配置选项的更多信息，请参阅安装参考。
wget https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/tigera-operator.yaml
wget https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/custom-resources.yaml
参考链接: https://docs.tigera.io/archive/v3.24/getting-started/kubernetes/quickstart
修改custom-resources.yaml里的IP段为上面集群的配置

Node 节点升级

在master上腾空worker 节点

[root@k8s-master31 ~]# kubectl drain k8s-node34 --ignore-daemonsets --delete-emptydir-data
node/k8s-node34 cordoned
WARNING: ignoring DaemonSet-managed Pods: calico-system/calico-node-5v6tj, calico-system/csi-node-driver-jhkkt, kube-system/kube-proxy-r5fz4
evicting pod tigera-operator/tigera-operator-6c49dc8ddf-99nr7
evicting pod calico-system/calico-kube-controllers-68995875fb-bpgfz
evicting pod calico-apiserver/calico-apiserver-86c46fdb85-hmglw
evicting pod calico-system/calico-typha-88d5b6455-4d9df
evicting pod kube-system/upgrade-health-check-w2jkr
pod/calico-typha-88d5b6455-4d9df evicted
pod/calico-apiserver-86c46fdb85-hmglw evicted
pod/tigera-operator-6c49dc8ddf-99nr7 evicted
pod/calico-kube-controllers-68995875fb-bpgfz evicted
pod/upgrade-health-check-w2jkr evicted

在 node 节点操作

[root@k8s-node34 ~]# yum install -y kubeadm-1.24.17-0 --disableexcludes=kubernetes
[root@k8s-node34 ~]# vim /var/lib/kubelet/kubeadm-flags.env
[root@k8s-node34 ~]# containerd config default > /etc/containerd/config.toml
[root@k8s-node34 ~]# sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
[root@k8s-node34 ~]# vim /etc/containerd/config.toml
[root@k8s-node34 ~]# cat >/etc/crictl.yaml<<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
[root@k8s-node34 ~]# systemctl daemon-reload
#重启containerd
systemctl restart containerd
#重启kubelet
systemctl restart kubelet
[root@k8s-node34 ~]# yum install -y kubelet-1.24.17-0 kubectl-1.24.17-0 --disableexcludes=kubernetes
Last metadata expiration check: 0:24:58 ago on 2024年08月22日 星期四 14时02分20秒.
Dependencies resolved.
=========================================================================================================================================
 Package                          Architecture                    Version                             Repository                    Size
=========================================================================================================================================
Upgrading:
 kubectl                          x86_64                          1.24.17-0                           k8s                           10 M
 kubelet                          x86_64                          1.24.17-0                           k8s                           21 M

Transaction Summary
=========================================================================================================================================
Upgrade  2 Packages

Total download size: 31 M
Downloading Packages:
(1/2): c3dc5ffa817d2c69bdd77494b5b9240568c4eb0d06b7b1bf3546bdab971741f5-kubectl-1.24.17-0.x86_64.rpm     405 kB/s |  10 MB     00:25
(2/2): f46e0356e279308a525195d1ae939268faaea772a119cb752480be2b998bec54-kubelet-1.24.17-0.x86_64.rpm     401 kB/s |  21 MB     00:53
-----------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                    595 kB/s |  31 MB     00:53
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                 1/1
  Running scriptlet: kubelet-1.24.17-0.x86_64                                                                                        1/1
  Upgrading        : kubelet-1.24.17-0.x86_64                                                                                        1/4
  Upgrading        : kubectl-1.24.17-0.x86_64                                                                                        2/4
  Cleanup          : kubectl-1.23.17-0.x86_64                                                                                        3/4
  Cleanup          : kubelet-1.23.17-0.x86_64                                                                                        4/4
  Running scriptlet: kubelet-1.23.17-0.x86_64                                                                                        4/4
  Verifying        : kubectl-1.24.17-0.x86_64                                                                                        1/4
  Verifying        : kubectl-1.23.17-0.x86_64                                                                                        2/4
  Verifying        : kubelet-1.24.17-0.x86_64                                                                                        3/4
  Verifying        : kubelet-1.23.17-0.x86_64                                                                                        4/4

Upgraded:
  kubectl-1.24.17-0.x86_64                                            kubelet-1.24.17-0.x86_64

Complete!
[root@k8s-node34 ~]# echo "KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroup-driver=systemd" >/etc/sysconfig/kubelet
[root@k8s-node34 ~]# systemctl restart kubelet

在 master 节点操作

[root@k8s-master31 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.24.17"...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2793172026"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Current and new manifests of kube-apiserver are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Current and new manifests of kube-controller-manager are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Current and new manifests of kube-scheduler are equal, skipping upgrade
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

查看集群状态

[root@k8s-master31 ~]# kubectl get nodes -o wide
NAME           STATUS                     ROLES           AGE    VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION                        CONTAINER-RUNTIME
k8s-master31   Ready                      control-plane   2d3h   v1.24.17   10.0.0.31     <none>        openEuler 22.03 (LTS-SP1)   5.10.0-136.12.0.86.oe2203sp1.x86_64   containerd://1.6.33
k8s-node34     Ready,SchedulingDisabled   <none>          2d3h   v1.24.17   10.0.0.34     <none>        openEuler 22.03 (LTS-SP1)   5.10.0-136.12.0.86.oe2203sp1.x86_64   containerd://1.6.33
k8s-node35     Ready,SchedulingDisabled   <none>          2d3h   v1.24.17   10.0.0.35     <none>        openEuler 22.03 (LTS-SP1)   5.10.0-136.12.0.86.oe2203sp1.x86_64   containerd://1.6.33

测试

# 创建一个临时的 Nginx Pod
[root@k8s-master31 ~]# kubectl run temp-nginx --image=nginx --restart=Never
pod/temp-nginx created

# 创建一个 Service 来暴露 Nginx Pod
[root@k8s-master31 ~]# kubectl expose pod temp-nginx --port=80 --target-port=80 --name=temp-nginx-svc
service/temp-nginx-svc exposed

[root@k8s-master31 ~]# kubectl get svc,pod -o wide
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE    SELECTOR
service/kubernetes       ClusterIP   172.96.0.1      <none>        443/TCP   2d3h   <none>
service/temp-nginx-svc   ClusterIP   172.101.81.80   <none>        80/TCP    31s    run=temp-nginx

NAME             READY   STATUS    RESTARTS   AGE   IP                NODE           NOMINATED NODE   READINESS GATES
pod/temp-nginx   1/1     Running   0          34s   172.244.221.136   k8s-master31   <none>           <none>

[root@k8s-master31 ~]# curl 172.244.221.136
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8s-node34 ~]# curl 172.101.81.80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

#  删除 Pod 和 Service
[root@k8s-master31 ~]# kubectl delete pod temp-nginx
pod "temp-nginx" deleted
[root@k8s-master31 ~]# kubectl delete svc temp-nginx-svc
service "temp-nginx-svc" deleted