基于open-gpu-kernel-modules的p2p vram映射bar1提高通信效率

知本知至2024-09-23 21:05

背景

bar1 Base Address Register 1 用于内存映射的寄存器，定义了设备的内存映射区域，BAR1专门分配给gpu的一部分内存区域，允许cpu通过pcie总线直接访问显存VRAM中的数据。但bar1的大小是有限的，在常规的4090上，bar1只有256M，基于nvidia开源的open-gpu-kernel-modules模块通过将bar1的寄存器地址增大至32G来提高计算效率

系统版本

bash 复制代码 
root@exai-165:~# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@exai-165:~# uname -a 
Linux exai-165 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 14:36:16 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

实施

  1. 编译开源的nvidia驱动模块
  2. 编译p2p模块

破解前bar1大小

bash 复制代码 
root@exai-165:/opt# lspci -s 0000:81:00.0 -v
81:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device 167c
	Flags: bus master, fast devsel, latency 0, IRQ 164, IOMMU group 27
	Memory at b8000000 (32-bit, non-prefetchable) [size=16M]
	Memory at 20030000000 (64-bit, prefetchable) [size=256M]  # 这里
	Memory at 20040000000 (64-bit, prefetchable) [size=32M]
	I/O ports at 6000 [size=128]
	Expansion ROM at b9000000 [virtual] [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Legacy Endpoint, MSI 00
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [258] L1 PM Substates
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [420] Advanced Error Reporting
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] Secondary PCI Express
	Capabilities: [bb0] Physical Resizable BAR
	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
	Capabilities: [d00] Lane Margining at the Receiver <?>
	Capabilities: [e00] Data Link Feature <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

nvidia驱动模块

卸载机器上原本的驱动

bash 复制代码 
./NVIDIA-Linux-x86_64-535.183.01.run --uninstall

克隆开源的驱动

自行配置git使用代理

bash 复制代码 
git clone --branch 550.54.15 --single-branch https://github.com/NVIDIA/open-gpu-kernel-modules.git
git branch
git checkout -b 550.54.15

因为机器上的CC和编译内核使用的gcc不是同一个版本，所以这里手工指定make使用哪个gcc

bash 复制代码 
make CC=x86_64-linux-gnu-gcc-12 modules -j$(nproc)
make modules_install CC=x86_64-linux-gnu-gcc-12 modules -j$(nproc)

备注：通过机器上的多版本管理工具来实现cc版本管理不生效

验证

bash 复制代码 
root@exai-165:~# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  550.54.15  Release Build  (root@exai-165)  2024年 09月 06日 星期五 10:49:38 CST
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04)

p2p

https://github.com/tinygrad/open-gpu-kernel-modules

克隆，编译,按照readme里面的来没啥问题

bash 复制代码 
root@exai-165:/opt/nvidia-p2p/open-gpu-kernel-modules# ./install.sh 
make -C src/nvidia
make -C src/nvidia-modeset
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia'
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia-modeset'
make[1]: Nothing to be done for 'default'.
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia-modeset'
cd kernel-open/nvidia-modeset/ && ln -sf ../../src/nvidia-modeset/_out/Linux_x86_64/nv-modeset-kernel.o nv-modeset-kernel.o_binary
make[1]: Nothing to be done for 'default'.
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia'
cd kernel-open/nvidia/ && ln -sf ../../src/nvidia/_out/Linux_x86_64/nv-kernel.o nv-kernel.o_binary
make -C kernel-open modules
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
make[2]: Entering directory '/usr/src/linux-headers-6.5.0-44-generic'
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
  You are using:           cc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
make[2]: Leaving directory '/usr/src/linux-headers-6.5.0-44-generic'
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
make -C kernel-open modules_install
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
make[2]: Entering directory '/usr/src/linux-headers-6.5.0-44-generic'
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-uvm.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-modeset.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-drm.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-peermem.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-peermem.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-modeset.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-drm.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-uvm.ko
  DEPMOD  /lib/modules/6.5.0-44-generic
Warning: modules_install: missing 'System.map' file. Skipping depmod.
make[2]: Leaving directory '/usr/src/linux-headers-6.5.0-44-generic'
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
Fri Sep  6 15:24:49 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
| 30%   36C    P0             53W /  450W |       0MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:81:00.0 Off |                  Off |
| 31%   44C    P0             69W /  450W |       0MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 4090        Off |   00000000:C1:00.0 Off |                  Off |
| 31%   39C    P0             55W /  450W |       0MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 4090        Off |   00000000:C2:00.0 Off |                  Off |
| 31%   42C    P0             64W /  450W |       0MiB /  24564MiB |      3%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

验证

bash 复制代码 
root@exai-165:/opt/nvidia-p2p/open-gpu-kernel-modules# lspci -s 0000:81:00.0 -v
81:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device 167c
	Flags: bus master, fast devsel, latency 0, IRQ 164, IOMMU group 27
	Memory at b8000000 (32-bit, non-prefetchable) [size=16M]
	Memory at 18800000000 (64-bit, prefetchable) [size=32G]  # 这里
	Memory at 18400000000 (64-bit, prefetchable) [size=32M]
	I/O ports at 6000 [size=128]
	Expansion ROM at b9000000 [virtual] [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Legacy Endpoint, MSI 00
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [258] L1 PM Substates
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [420] Advanced Error Reporting
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] Secondary PCI Express
	Capabilities: [bb0] Physical Resizable BAR
	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
	Capabilities: [d00] Lane Margining at the Receiver <?>
	Capabilities: [e00] Data Link Feature <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

/var/log/kernel.log中有读取registry address错误的信息，syslog中有不断向内核中注册bar1的信息，判断应该是p2p的版本不兼容4090卡，具体的原因由于其他事情未继续进行，等后面看看

Sep 19 16:33:03 exai-165 kernel: [436359.365867] NVRM: gpuHandleSanityCheckRegReadError_GM107: Possible bad register read: addr: 0x110100, regvalue: 0xbadf5620, error code: Unknown SYS_PRI_ERROR_CODE

回退

即卸载通过open-gpu-kernel-modules编译安装的550.54.15版本，然后重新安装原来的535版本驱动

  1. 卸载内核模块
  2. 卸载550版本驱动
  3. 安装535版本驱动
  4. 如果nvidia-smi无法显示，手工删除550内核模块使用dkms重新编译到内核中

装完535版本驱动报错

nvidia-smi

Failed to initialize NVML: Driver/library version mismatch

NVML library version: 535.183

查看内核中注册的驱动版本

bash 复制代码 
dkms status
nvidia/535.183.01, 6.5.0-44-generic, x86_64: installed

查看驱动内核信息

bash 复制代码 
cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  550.54.15  Release Build  (root@exai-165)  2024年 09月 06日 星期五 10:49:38 CST
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04)

查看内核模块

bash 复制代码 
lsmod |grep nvidia
nvidia_drm            122880  0
nvidia_modeset       1490944  1 nvidia_drm
nvidia               8675328  1 nvidia_modeset
video                  73728  1 nvidia_modeset
ecc                    45056  1 nvidia
drm_kms_helper        274432  4 ast,nvidia_drm
drm                   765952  6 drm_kms_helper,ast,drm_shmem_helper,nvidia,nvidia_drm
# 找到内核模块的路径
modinfo nvidia
filename:       /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia.ko
import_ns:      DMA_BUF
alias:          char-major-195-*
version:        550.54.15
supported:      external
license:        Dual MIT/GPL
firmware:       nvidia/550.54.15/gsp_tu10x.bin
firmware:       nvidia/550.54.15/gsp_ga10x.bin

卸载内核模块后，手动删除

bash 复制代码 
mkdir /tmp/nvidia-module
mv /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia* /tmp/nvidia-module/

此时nvidia-smi显示

nvidia-smi

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

先卸载再安装

bash 复制代码 
dkms remove -m nvidia -v 535.183.01 --all
dkms install -m nvidia -v 535.183.01

ok

bash 复制代码 
nvidia-smi 
Fri Sep 20 10:14:32 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:81:00.0 Off |                  Off |
| 30%   34C    P0              64W / 450W |      0MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:C1:00.0 Off |                  Off |
| 31%   32C    P0              50W / 450W |      0MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

reference:

https://github.com/NVIDIA/open-gpu-kernel-modules

https://github.com/tinygrad/open-gpu-kernel-modules

相关推荐
幽兰的天空1 小时前
介绍 HTTP 请求如何实现跨域
网络·网络协议·http
lisenustc1 小时前
HTTP post请求工具类
网络·网络协议·http
心平气和️1 小时前
HTTP 配置与应用（不同网段）
网络·网络协议·计算机网络·http
心平气和️1 小时前
HTTP 配置与应用（局域网）
网络·计算机网络·http·智能路由器
Gworg2 小时前
网站HTTP改成HTTPS
网络协议·http·https
Mbblovey2 小时前
Picsart美易照片编辑器和视频编辑器
网络·windows·软件构建·需求分析·软件需求
北顾南栀倾寒3 小时前
[Qt]系统相关-网络编程-TCP、UDP、HTTP协议
开发语言·网络·c++·qt·tcp/ip·http·udp
GZ_TOGOGO3 小时前
PIM原理与配置
网络·华为·智能路由器
7ACE4 小时前
Wireshark TS | 虚假的 TCP Spurious Retransmission
网络·网络协议·tcp/ip·wireshark·tcpdump
大丈夫立于天地间4 小时前
ISIS基础知识
网络·网络协议·学习·智能路由器·信息与通信
热门推荐
01centos7 init.d 和system.d02Dell服务器升级ubuntu 22.04失败解决03半导体应用系统一些小知识收集(strip&wafer mapping,EAP&scada)04xgboost: Why not implement distributed XGBoost on top of spark05(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明06密码学原理技术-第六章-introduction to pulibc-key cryptography07PyTorch AMP 混合精度中grad_scaler.py的scale函数解析08Windows10安装PCL1.14.0及点云配准09优化手机性能，解决卡顿问题：关闭这3个微信开关，释放内存空间10如何实现抖音直播弹幕抓取（PHP版本）