背景
机房有套老k8s集群，上面的比较重要的服务迁移走了。使用的是ubuntu18的系统，升级至ubuntu20再到ubuntu22，然后升级上面的k8s集群，方便后期维护
思路
注意的点：升级node节点的系统时，注意先将node上的pod及其他服务cordon&驱逐后再进行升级
步骤
- 配置机器路由&dns使得机器可以访问真正的互联网(你懂得)
- 移除第三方源
- 禁止GPU节点nvidia相关包的升级
- 执行升级
- 卸载gnome环境&也可以在升级前先卸载gnome环境
第一步骤配置路由略，配置私有dns
bash
grep nameserver /etc/resolv.conf
nameserver 192.168.0.220
nameserver 127.0.0.53移除第三方软件源&将软件源换为ubuntu官方源，不然升级到最后有可能重启的时候会出现kernel panic等诸多问题
bash
mkdir /tmp/apt
mv /etc/apt/sources.list.d/* /tmp/apt
cat /etc/sources.list
# deb cdrom:[Ubuntu 18.04.3 LTS _Bionic Beaver_ - Release amd64 (20190805)]/ bionic main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://us.archive.ubuntu.com/ubuntu/ bionic main restricted
# deb-src http://us.archive.ubuntu.com/ubuntu/ bionic main restricted
## Major bug fix updates produced after the final release of the
## distribution.
deb http://us.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
# deb-src http://us.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://us.archive.ubuntu.com/ubuntu/ bionic universe
# deb-src http://us.archive.ubuntu.com/ubuntu/ bionic universe
deb http://us.archive.ubuntu.com/ubuntu/ bionic-updates universe
# deb-src http://us.archive.ubuntu.com/ubuntu/ bionic-updates universe
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://us.archive.ubuntu.com/ubuntu/ bionic multiverse
# deb-src http://us.archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://us.archive.ubuntu.com/ubuntu/ bionic-updates multiverse
# deb-src http://us.archive.ubuntu.com/ubuntu/ bionic-updates multiverse
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://us.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src http://us.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu bionic partner
# deb-src http://archive.canonical.com/ubuntu bionic partner
deb http://security.ubuntu.com/ubuntu bionic-security main restricted
# deb-src http://security.ubuntu.com/ubuntu bionic-security main restricted
deb http://security.ubuntu.com/ubuntu bionic-security universe
# deb-src http://security.ubuntu.com/ubuntu bionic-security universe
deb http://security.ubuntu.com/ubuntu bionic-security multiverse
# deb-src http://security.ubuntu.com/ubuntu bionic-security multiverse更新源
bash
apt update
apt upgrade
# 配置允许升级到的版本
grep Prompt /etc/update-manager/release-upgrades
Prompt=lts
# 需要完全升级的包
apt full-upgrade
# 卸载掉不需要的包
apt autoremove -y
# 检查可以更新的版本
do-release-upgrade -c
# -d 表示更新到lts版本，这里无
# 开始升级 会使用一个终端与回话分离工具分开
do-release-upgrade
Can not upgrade
Your python3 install is corrupted. Please fix the '/usr/bin/python3'
symlink.
# 之前使用alter管理python版本 软链接定向到/etc/alter下了 重新更改回来即可
ln -sf /usr/bin/python3.6 /usr/bin/python3在使用了gnome环境的桌面中，大概需要下载1500个包
过程中出现的一些提示
bash
# 选项1
Do you want to start the upgrade?
8 installed packages are no longer supported by Canonical. You can
still get support from the community.
26 packages are going to be removed. 222 new packages are going to be
installed. 1344 packages are going to be upgraded.
You have to download a total of 1,223 M. This download will take
about 1 minute with your connection.
Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.
Continue [yN] Details [d]# 输入y
# 选项2
系统正在询问你是否允许在升级过程中自动重启某些服务（例如 libpam、libc 和 libssl 等关键库相关的服务
Configuring libc6
There are services installed on your system which need to be restarted when certain libraries, such as libpam, libc, and libssl, are upgraded. Since these restarts may cause interruptions of service for the system, you will normally be prompted on each upgrade for the list of services you wish to restart. You │
│ can choose this option to avoid being prompted; instead, all necessary restarts will be done for you automatically so you can avoid being asked questions on each library upgrade. │
│ │
│ Restart services during package upgrades without asking? # 选择yes
# 选项3
A new version (/tmp/fileG81NRI) of configuration file /etc/ssh/sshd_config is available, but the version installed currently has been locally modified. │
│ │
│ What do you want to do about modified configuration file sshd_config? │
│ │
│ install the package maintainer's version │
│ keep the local version currently installed │
│ show the differences between the versions │
│ show a side-by-side difference between the versions │
│ show a 3-way difference between available versions │
│ do a 3-way merge between available versions │
│ start a new shell to examine the situation
# 保持当前版本即可，选择keep the local version currently installed
# 选项4
Configuration file '/etc/security/limits.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** limits.conf (Y/I/N/O/D/Z) [default=N] ? # 输入N
# 选项4 lxd版本选择
The LXD project puts out monthly feature releases which while backward compatible at an API and CLI level, will contain some behavior change and potentially require manual intervention during │
│ an upgrade. │
│ │
│ In addition to those, every 2 years a LTS release is made which comes with 5 years of support through frequent bugfix-only releases. │
│ │
│ The LXD team recommends you pick "4.0" for production environments and use "latest" if you're interested in getting the latest LXD features. │
│ │
│ LXD snap track │
│ │
│ 3.0 │
│ 4.0
选择4.0 长期支持版本升级过程中出现的提示视情况选择即可
升级完毕，提示是否执行重启
bash
8 installed packages are no longer supported by Canonical. You can
still get support from the community.
26 packages are going to be removed. 222 new packages are going to be
installed. 1344 packages are going to be upgraded.
You have to download a total of 1,223 M. This download will take
about 1 minute with your connection.
Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.
Continue [yN] Details [d]
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN] y卸载gnome桌面环境
bash
apt remove --purge ubuntu-desktop gnome-shell gnome-session gdm3
apt remove --purge gnome* gnome-shell* gnome-session* gdm3* nautilus* ubuntu-gnome-desktop
apt autoremove --purge
apt autoclean
# 手动卸载未清理的gnome包
apt remove --purge 包名
# 卸载snap包管理工具安装的gnome工具
snap list
Name Version Rev Tracking Publisher Notes
bare 1.0 5 latest/stable canonical✓ base
core18 20240920 2846 latest/stable canonical✓ base
core20 20240911 2434 latest/stable canonical✓ base
core22 20241119 1722 latest/stable canonical✓ base
gnome-3-34-1804 0+git.3556cb3 93 latest/stable/... canonical✓ -
gnome-3-38-2004 0+git.efb213a 143 latest/stable/... canonical✓ -
gnome-42-2204 0+git.510a601 176 latest/stable canonical✓ -
gnome-calculator 45.0.2 955 latest/stable/... canonical✓ -
gnome-characters 46.0 797 latest/stable/... canonical✓ -
gnome-logs 45.0 123 latest/stable/... canonical✓ -
gnome-system-monitor 45.0 186 latest/stable/... canonical✓ -
gtk-common-themes 0.1-81-g442e511 1535 latest/stable/... canonical✓ -
snapd 2.66.1 23258 latest/stable canonical✓ snapd
# 卸载
snap remove gnome-3-34-1804
snap remove gnome-3-38-2004
snap remove gnome-42-2204
snap remove gnome-calculator
snap remove gnome-characters
snap remove gnome-logs
snap remove gnome-system-monitor取消grub参数
bash
grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT=""
update-grub
rebootGPU节点升级日志
bash
# 禁止升级和nvidia相关的包
root@k8s-gpu-node3:/etc/apt# apt update
Hit:1 http://cn.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://cn.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://cn.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
61 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@k8s-gpu-node3:/etc/apt# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
libnvidia-compute-470 : Breaks: libnvidia-compute-470:i386 (!= 470.129.06-0ubuntu0.18.04.1) but 470.182.03-0ubuntu0.18.04.1 is to be installed
libnvidia-compute-470:i386 : Breaks: libnvidia-compute-470 (!= 470.182.03-0ubuntu0.18.04.1) but 470.129.06-0ubuntu0.18.04.1 is to be installed
libnvidia-decode-470 : Breaks: libnvidia-decode-470:i386 (!= 470.129.06-0ubuntu0.18.04.1) but 470.182.03-0ubuntu0.18.04.1 is to be installed
libnvidia-decode-470:i386 : Breaks: libnvidia-decode-470 (!= 470.182.03-0ubuntu0.18.04.1) but 470.129.06-0ubuntu0.18.04.1 is to be installed
libnvidia-encode-470 : Breaks: libnvidia-encode-470:i386 (!= 470.129.06-0ubuntu0.18.04.1) but 470.182.03-0ubuntu0.18.04.1 is to be installed
libnvidia-encode-470:i386 : Breaks: libnvidia-encode-470 (!= 470.182.03-0ubuntu0.18.04.1) but 470.129.06-0ubuntu0.18.04.1 is to be installed
libnvidia-fbc1-470 : Breaks: libnvidia-fbc1-470:i386 (!= 470.129.06-0ubuntu0.18.04.1) but 470.182.03-0ubuntu0.18.04.1 is to be installed
libnvidia-fbc1-470:i386 : Breaks: libnvidia-fbc1-470 (!= 470.182.03-0ubuntu0.18.04.1) but 470.129.06-0ubuntu0.18.04.1 is to be installed
libnvidia-gl-470 : Breaks: libnvidia-gl-470:i386 (!= 470.129.06-0ubuntu0.18.04.1) but 470.182.03-0ubuntu0.18.04.1 is to be installed
libnvidia-gl-470:i386 : Breaks: libnvidia-gl-470 (!= 470.182.03-0ubuntu0.18.04.1) but 470.129.06-0ubuntu0.18.04.1 is to be installed
libnvidia-ifr1-470 : Breaks: libnvidia-ifr1-470:i386 (!= 470.129.06-0ubuntu0.18.04.1) but 470.182.03-0ubuntu0.18.04.1 is to be installed
libnvidia-ifr1-470:i386 : Breaks: libnvidia-ifr1-470 (!= 470.182.03-0ubuntu0.18.04.1) but 470.129.06-0ubuntu0.18.04.1 is to be installed
E: Broken packages
root@k8s-gpu-node3:/etc/apt# apt-mark hold libnvidia-compute-470 libnvidia-decode-470 libnvidia-encode-470 libnvidia-fbc1-470 libnvidia-gl-470 libnvidia-ifr1-470
root@k8s-gpu-node3:/etc/apt# apt-mark hold libnvidia-compute-470:i386 libnvidia-decode-470:i386 libnvidia-encode-470:i386 libnvidia-fbc1-470:i386 libnvidia-gl-470:i386 libnvidia-ifr1-470:i386
root@k8s-gpu-node3:/etc/apt# apt-mark showhold
apt upgrade -y后面发现好像还是不能mark hold，还是需要将包升级完成才行
建议先卸载gnome桌面环境后再执行升级
reference
https://ubuntu.com/blog/how-to-upgrade-from-ubuntu-18-04-lts-to-20-04-lts-today