基于open-gpu-kernel-modules的p2p vram映射bar1提高通信效率

知本知至2024-09-23 21:05

背景

bar1 Base Address Register 1 用于内存映射的寄存器，定义了设备的内存映射区域，BAR1专门分配给gpu的一部分内存区域，允许cpu通过pcie总线直接访问显存VRAM中的数据。但bar1的大小是有限的，在常规的4090上，bar1只有256M，基于nvidia开源的open-gpu-kernel-modules模块通过将bar1的寄存器地址增大至32G来提高计算效率

系统版本

bash 复制代码 
root@exai-165:~# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@exai-165:~# uname -a 
Linux exai-165 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 14:36:16 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

实施

  1. 编译开源的nvidia驱动模块
  2. 编译p2p模块

破解前bar1大小

bash 复制代码 
root@exai-165:/opt# lspci -s 0000:81:00.0 -v
81:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device 167c
	Flags: bus master, fast devsel, latency 0, IRQ 164, IOMMU group 27
	Memory at b8000000 (32-bit, non-prefetchable) [size=16M]
	Memory at 20030000000 (64-bit, prefetchable) [size=256M]  # 这里
	Memory at 20040000000 (64-bit, prefetchable) [size=32M]
	I/O ports at 6000 [size=128]
	Expansion ROM at b9000000 [virtual] [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Legacy Endpoint, MSI 00
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [258] L1 PM Substates
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [420] Advanced Error Reporting
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] Secondary PCI Express
	Capabilities: [bb0] Physical Resizable BAR
	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
	Capabilities: [d00] Lane Margining at the Receiver <?>
	Capabilities: [e00] Data Link Feature <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

nvidia驱动模块

卸载机器上原本的驱动

bash 复制代码 
./NVIDIA-Linux-x86_64-535.183.01.run --uninstall

克隆开源的驱动

自行配置git使用代理

bash 复制代码 
git clone --branch 550.54.15 --single-branch https://github.com/NVIDIA/open-gpu-kernel-modules.git
git branch
git checkout -b 550.54.15

因为机器上的CC和编译内核使用的gcc不是同一个版本，所以这里手工指定make使用哪个gcc

bash 复制代码 
make CC=x86_64-linux-gnu-gcc-12 modules -j$(nproc)
make modules_install CC=x86_64-linux-gnu-gcc-12 modules -j$(nproc)

备注：通过机器上的多版本管理工具来实现cc版本管理不生效

验证

bash 复制代码 
root@exai-165:~# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  550.54.15  Release Build  (root@exai-165)  2024年 09月 06日 星期五 10:49:38 CST
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04)

p2p

https://github.com/tinygrad/open-gpu-kernel-modules

克隆，编译,按照readme里面的来没啥问题

bash 复制代码 
root@exai-165:/opt/nvidia-p2p/open-gpu-kernel-modules# ./install.sh 
make -C src/nvidia
make -C src/nvidia-modeset
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia'
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia-modeset'
make[1]: Nothing to be done for 'default'.
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia-modeset'
cd kernel-open/nvidia-modeset/ && ln -sf ../../src/nvidia-modeset/_out/Linux_x86_64/nv-modeset-kernel.o nv-modeset-kernel.o_binary
make[1]: Nothing to be done for 'default'.
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/src/nvidia'
cd kernel-open/nvidia/ && ln -sf ../../src/nvidia/_out/Linux_x86_64/nv-kernel.o nv-kernel.o_binary
make -C kernel-open modules
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
make[2]: Entering directory '/usr/src/linux-headers-6.5.0-44-generic'
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
  You are using:           cc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
make[2]: Leaving directory '/usr/src/linux-headers-6.5.0-44-generic'
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
make -C kernel-open modules_install
make[1]: Entering directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
make[2]: Entering directory '/usr/src/linux-headers-6.5.0-44-generic'
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-uvm.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-modeset.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-drm.ko
  INSTALL /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-peermem.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-peermem.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-modeset.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-drm.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia.ko
  SIGN    /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia-uvm.ko
  DEPMOD  /lib/modules/6.5.0-44-generic
Warning: modules_install: missing 'System.map' file. Skipping depmod.
make[2]: Leaving directory '/usr/src/linux-headers-6.5.0-44-generic'
make[1]: Leaving directory '/opt/nvidia-p2p/open-gpu-kernel-modules/kernel-open'
Fri Sep  6 15:24:49 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
| 30%   36C    P0             53W /  450W |       0MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:81:00.0 Off |                  Off |
| 31%   44C    P0             69W /  450W |       0MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 4090        Off |   00000000:C1:00.0 Off |                  Off |
| 31%   39C    P0             55W /  450W |       0MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 4090        Off |   00000000:C2:00.0 Off |                  Off |
| 31%   42C    P0             64W /  450W |       0MiB /  24564MiB |      3%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

验证

bash 复制代码 
root@exai-165:/opt/nvidia-p2p/open-gpu-kernel-modules# lspci -s 0000:81:00.0 -v
81:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device 167c
	Flags: bus master, fast devsel, latency 0, IRQ 164, IOMMU group 27
	Memory at b8000000 (32-bit, non-prefetchable) [size=16M]
	Memory at 18800000000 (64-bit, prefetchable) [size=32G]  # 这里
	Memory at 18400000000 (64-bit, prefetchable) [size=32M]
	I/O ports at 6000 [size=128]
	Expansion ROM at b9000000 [virtual] [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Legacy Endpoint, MSI 00
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [258] L1 PM Substates
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [420] Advanced Error Reporting
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] Secondary PCI Express
	Capabilities: [bb0] Physical Resizable BAR
	Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
	Capabilities: [d00] Lane Margining at the Receiver <?>
	Capabilities: [e00] Data Link Feature <?>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

/var/log/kernel.log中有读取registry address错误的信息，syslog中有不断向内核中注册bar1的信息，判断应该是p2p的版本不兼容4090卡，具体的原因由于其他事情未继续进行，等后面看看

Sep 19 16:33:03 exai-165 kernel: [436359.365867] NVRM: gpuHandleSanityCheckRegReadError_GM107: Possible bad register read: addr: 0x110100, regvalue: 0xbadf5620, error code: Unknown SYS_PRI_ERROR_CODE

回退

即卸载通过open-gpu-kernel-modules编译安装的550.54.15版本，然后重新安装原来的535版本驱动

  1. 卸载内核模块
  2. 卸载550版本驱动
  3. 安装535版本驱动
  4. 如果nvidia-smi无法显示，手工删除550内核模块使用dkms重新编译到内核中

装完535版本驱动报错

nvidia-smi

Failed to initialize NVML: Driver/library version mismatch

NVML library version: 535.183

查看内核中注册的驱动版本

bash 复制代码 
dkms status
nvidia/535.183.01, 6.5.0-44-generic, x86_64: installed

查看驱动内核信息

bash 复制代码 
cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  550.54.15  Release Build  (root@exai-165)  2024年 09月 06日 星期五 10:49:38 CST
GCC version:  gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04)

查看内核模块

bash 复制代码 
lsmod |grep nvidia
nvidia_drm            122880  0
nvidia_modeset       1490944  1 nvidia_drm
nvidia               8675328  1 nvidia_modeset
video                  73728  1 nvidia_modeset
ecc                    45056  1 nvidia
drm_kms_helper        274432  4 ast,nvidia_drm
drm                   765952  6 drm_kms_helper,ast,drm_shmem_helper,nvidia,nvidia_drm
# 找到内核模块的路径
modinfo nvidia
filename:       /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia.ko
import_ns:      DMA_BUF
alias:          char-major-195-*
version:        550.54.15
supported:      external
license:        Dual MIT/GPL
firmware:       nvidia/550.54.15/gsp_tu10x.bin
firmware:       nvidia/550.54.15/gsp_ga10x.bin

卸载内核模块后，手动删除

bash 复制代码 
mkdir /tmp/nvidia-module
mv /lib/modules/6.5.0-44-generic/kernel/drivers/video/nvidia* /tmp/nvidia-module/

此时nvidia-smi显示

nvidia-smi

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

先卸载再安装

bash 复制代码 
dkms remove -m nvidia -v 535.183.01 --all
dkms install -m nvidia -v 535.183.01

ok

bash 复制代码 
nvidia-smi 
Fri Sep 20 10:14:32 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:81:00.0 Off |                  Off |
| 30%   34C    P0              64W / 450W |      0MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:C1:00.0 Off |                  Off |
| 31%   32C    P0              50W / 450W |      0MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

reference:

https://github.com/NVIDIA/open-gpu-kernel-modules

https://github.com/tinygrad/open-gpu-kernel-modules

相关推荐
hakesashou1 小时前
Python中常用的函数介绍
java·网络·python
C++忠实粉丝2 小时前
计算机网络socket编程(4)_TCP socket API 详解
网络·数据结构·c++·网络协议·tcp/ip·计算机网络·算法
九州ip动态2 小时前
做网络推广及游戏注册为什么要换IP
网络·tcp/ip·游戏
Estar.Lee2 小时前
时间操作[取当前北京时间]免费API接口教程
android·网络·后端·网络协议·tcp/ip
蝶开三月2 小时前
php：使用socket函数创建WebSocket服务
网络·websocket·网络协议·php·socket
G丶AEOM2 小时前
SSL/TLS，SSL，TLS分别是什么
网络·网络协议·网络安全
儒道易行2 小时前
【DVWA】RCE远程命令执行实战
网络·安全·网络安全
Koi慢热3 小时前
路由基础（全）
linux·网络·网络协议·安全
hzyyyyyyyu5 小时前
内网安全隧道搭建-ngrok-frp-nps-sapp
服务器·网络·安全
热门推荐
01(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明02PyTorch机器学习实现液态神经网络03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04玄机平台应急响应—webshell查杀05Coze扣子平台完整体验和实践（附国内和国际版对比）06Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO07【目标跟踪】相机运动补偿08Unity中PICO实现 隔空取物 和 接触抓取物体09RAG 实践- Ollama+RagFlow 部署本地知识库10怎样让音频速度变慢？请跟随以下方法进行操作