Proxmox 7.4 使用vgpu_unlock,为GTX1060开启vGPU支持

本文在 2021年发布的博客《Proxmox 5.4使用vgpu_unlock,为GTX1060开启vGPU支持》,介绍了 Proxmox VE 5.4 上部署vGPU unlock 的操作步骤。

后续有发布了在 Proxmox VE 7.x 上支持 vGPU的博客《Proxmox 7.2 部署 DoraCloud桌面云,支持vGPU》, 实现了通过 3个脚本完成 vGPU的配置。

本文在上述两个文章的基础上,实现通过4个脚本,实现利用消费级显卡实现vGPU。包含了对 Proxmox VE的更新,启用iommu,安装 nvidia grid 驱动,以及加载 vgpu_unlock。

准备工作:

硬件: 一台服务器,配置有ssd硬盘, 安装有GTX 1060 显卡。

软件: 安装了 Proxmox VE 7.4 。

网络: 网络中可以访问互联网。

步骤1:换源,更新

复制代码
curl -o- http://www1.deskpool.com:9000/software/gpu01.sh |bash

上面的脚本的内容,为了方便阅读,贴在这里

复制代码
#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# DoraCloud for Proxmox Enable vGPU
# Phase 1: update source to mirrors.ustc.edu.cn

cp /etc/apt/sources.list /etc/apt/sources.list.backup
sed -i 's|^deb http://ftp.debian.org|deb https://mirrors.ustc.edu.cn|g' /etc/apt/sources.list
sed -i 's|^deb http://security.debian.org|deb https://mirrors.ustc.edu.cn/debian-security|g' /etc/apt/sources.list

mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.bak
CODENAME=`cat /etc/os-release |grep PRETTY_NAME |cut -f 2 -d "(" |cut -f 1 -d ")"`

#echo "deb https://mirrors.nju.edu.cn/proxmox/debian/pve $CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
#echo "deb https://mirrors.ustc.edu.cn/proxmox/debian $CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list

wget https://mirrors.ustc.edu.cn/proxmox/debian/proxmox-release-$CODENAME.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-$CODENAME.gpg

#更新
apt update && apt install  pve-kernel-5.15 pve-headers-5.15 dkms mdevctl -y

步骤2: 启用iommu

复制代码
curl -o- http://www1.deskpool.com:9000/software/gpu02.sh |bash

上面的脚本的内容,为了方便阅读,贴在下方

复制代码
#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# Enable IO-MMU on PVE Server


# 复制如下脚本,启用IO-MMU
echo ""
echo "********************************************"
echo "***  Enable IO-MMU on proxmox host       ***"
echo "********************************************"
# /etc/default/grub 的GRUB_CMDLINE_LINUX_DEFAULT,增加 intel_iommu=on iommu=pt
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"/g' /etc/default/grub

echo ""
echo "    Update grub .... "
update-grub

# 加载 vfio vfio_iommu_type1 vfio_pci vfio_virqfd 4个Modules
echo vfio >> /etc/modules
echo vfio_iommu_type1 >> /etc/modules
echo vfio_pci >> /etc/modules
echo vfio_virqfd >> /etc/modules


echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
echo "blacklist nouveau" >>/etc/modprobe.d/disable-nouveau.conf
echo "options nouveau modeset=0" >>/etc/modprobe.d/disable-nouveau.conf

update-initramfs -u
echo ""
echo "    Proxmox host reboot ............."
reboot

步骤3:安装 NVIDIA GRID 13.6的驱动

复制代码
curl -o- http://www1.deskpool.com:9000/software/gpu03e.sh |bash

上面的脚本的内容,请自己阅读。 这里选择相对稳定的 13.6 版本。

复制代码
#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# Install NVIDIA Linux vGPU Driver 470.161.02
echo ""
echo "********************************************************"
echo "*** Install NVIDIA Grid vGPU Driver 13.6 V470.161.02 ***"
echo "********************************************************"

echo "    Downloading NVIDIA driver "
wget http://www1.deskpool.com:9000/software/NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run
chmod +x NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run

echo ""
echo "    Extracting driver .... "
./NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run -x

cd NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm/

echo ""
echo "    Patching NVIDIA driver "
sed -i 's|.open             = nv_vgpu_vfio_open|.open_device = nv_vgpu_vfio_open|g'    kernel/nvidia-vgpu-vfio/nvidia-vgpu-vfio.c
sed -i 's|.release          = nv_vgpu_vfio_close|.close_device = nv_vgpu_vfio_close|g'    kernel/nvidia-vgpu-vfio/nvidia-vgpu-vfio.c

echo ""
echo "    Installing driver .... "
./nvidia-installer -dkms -s

echo ""
echo "    deamon-reload && rebooting ...."
systemctl daemon-reload
reboot

步骤4:运行 vgpu_unlock 和 vgpu_unlock-rs

复制代码
curl -o- http://www1.deskpool.com:9000/software/gpu03u.sh |bash

上面的脚本的内容,请自己阅读。

复制代码
#!/bin/sh
# Author: DoraCloud Technology Ltd.co
#
# Date: 2022/05/07
#
# Unlock NVIDIA GTX 1060

echo ""
echo "********************************************************"
echo "***  Unlock NVIDIA GTX1060                           ***"
echo "********************************************************"


echo ""
echo ""
echo "    Download vgpu_lock "
echo ""
apt install git-core -y

git clone https://gitee.com/deskpool/vgpu_unlock.git
chmod -R +x vgpu_unlock


echo ""
echo ""
echo "********************************************************"
echo "    Apply vgpu_unlock to 470.161.02"
echo ""
sed -i 's/#include "nv-time.h"/#include "nv-time.h"\n\n#include "\/root\/vgpu_unlock\/vgpu_unlock_hooks.c"/g' /usr/src/nvidia-470.161.02/nvidia/os-interface.c
echo "ldflags-y += -T /root/vgpu_unlock/kern.ld" >> /usr/src/nvidia-470.161.02/nvidia/nvidia.Kbuild
dkms remove -m nvidia -v 470.161.02
dkms install -m nvidia -v 470.161.02




echo ""
echo ""
echo "********************************************************"
echo "    Install RUST "
echo ""
#设置rustup的source为 USTC
export RUSTUP_DIST_SERVER=https://mirrors.ustc.edu.cn/rust-static
export RUSTUP_UPDATE_ROOT=https://mirrors.ustc.edu.cn/rust-static/rustup
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
#wget -qO- https://cdn.jsdelivr.net/gh/rust-lang-nursery/rustup.rs/rustup-init.sh |sh  -s -- -y

echo ""
echo ""
echo "********************************************************"
echo "    Config cargo "
echo ""
#加载cargo的环境变量
source ~/.cargo/env

#设置cargo的源镜像为 USTC
cat >~/.cargo/config <<EOF
[source.crates-io]
replace-with = 'ustc'

[source.ustc]
registry = "git://mirrors.ustc.edu.cn/crates.io-index"
EOF


echo ""
echo ""
echo "********************************************************"
echo "   Building vgpu_unlock-rs  "
echo ""

#下载 vgpu_unlock-rs 项目
git clone https://gitee.com/deskpool/vgpu_unlock-rs
cd vgpu_unlock-rs/
cargo build --release

echo ""
echo ""
echo "********************************************************"
echo "   Create vgpu.service && vgpu-mgr.service "
echo ""

#nvidia-vgpud 的unlock 服务
mkdir /etc/systemd/system/nvidia-vgpud.service.d
cat >/etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf <<EOF
[Service]
Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so
EOF

#nvidia-vgpu-mgr 的unlock服务
mkdir /etc/systemd/system/nvidia-vgpu-mgr.service.d
cat >/etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf <<EOF
[Service]
Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so
EOF

#描述文件,控制GPU的显示配置,以及cuda
mkdir /etc/vgpu_unlock
cat >/etc/vgpu_unlock/profile_override.toml <<EOF
[profile.nvidia-55]
num_displays = 1
display_width = 1920
display_height = 1080
max_pixels = 2073600
cuda_enabled = 1
frl_enabled = 0
EOF

echo ""
echo ""
echo "********************************************************"
echo "    Reboot Proxmox VE .....  "
echo "    run  mdevctl types to check vGPU is installed.     "
echo ""

sync
reboot

里面的内容包括了安装 vgpu_unlock ,安装 rust 环境, 下载和编译 vgpu_unlock-rs, 配置 nvidia-vgpu 和nvidia-vgpu-mgr的服务,

生成vgpu_unlock 显卡配置文件 profile_override.toml 等内容。

全部执行完毕后,系统重启之后,运行 mdevctl types ,可以检查到 vGPU的类型已经生成。

复制代码
root@pve03:~# mdevctl types
0000:01:00.0
  nvidia-156
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-215
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B4
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-241
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B4
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-283
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4C
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2160, max_instance=6
  nvidia-284
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6C
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2160, max_instance=4
  nvidia-285
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8C
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2160, max_instance=3
  nvidia-286
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12C
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2
  nvidia-287
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24C
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2160, max_instance=1
  nvidia-46
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-47
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
  nvidia-48
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3Q
    Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
  nvidia-49
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
  nvidia-50
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6Q
    Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
  nvidia-51
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
  nvidia-52
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12Q
    Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
  nvidia-53
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24Q
    Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
  nvidia-54
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
  nvidia-55
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
  nvidia-56
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3A
    Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
  nvidia-57
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
  nvidia-58
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6A
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
  nvidia-59
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
  nvidia-60
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12A
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
  nvidia-61
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24A
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
  nvidia-62
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24

剩下的工作是部署 GuestOS,并安装 NVIDIA GRID 驱动。 这里推荐部署 DoraCloud,下载 win10LTSC2021V3 模板。 内置了GRID 13.6的驱动。

部署的方法,参考 《Proxmox 7.2 部署 DoraCloud桌面云,支持vGPU》 ,以及 《ShareStation工作站虚拟化部署 (doracloud.cn)

最后分享以下, GTX 1060 虚拟化之后,vGPU 运行《原神》游戏的效果。

相关推荐
云达闲人12 天前
Proxmox VE 用户与权限管理命令大全
运维·网络·云原生·容器·proxmox·用户权限管理·命令大全
陌客依天涯4 个月前
Virtualbox7.1.4安装Proxmox
pve·virtualbox·proxmox
4/5$全真龙门10 个月前
Proxmox VE 8虚拟机直通USB磁盘
linux·运维·服务器·proxmox
USTC-lup1 年前
proxmox宿主机安装桌面
linux·运维·服务器·proxmox
Jockey2 年前
AMD CPU 虚拟机安装 macos 系统的各虚拟机系统对比
macos·vmware·虚拟机·kvm·proxmox