nvme over rdma 环境配置
本文主要介绍NVMe over RDMA的安装和配置。关于什么是NVMe over Fabrics,什么是NVMe over RDMA，本文就不做介绍了，网上资料一大堆。
RDMA（全称：Remote Direct Memory Access）是一种远程直接内存访问技术，通过在硬件中实现传输层协议，将内存/消息原语接口暴露至用户空间，通过绕过CPU和内核网络协议栈来实现高吞吐和低延迟的网络。RoCE（RDMA over Converged Ethernet）是一种允许通过以太网使用远程直接内存访问（RDMA）的网络协议。
前面一文已经提到过centos 7.8 x86_64上面部署rdma环境，下面主要测试一下基于内核态的 nvme of rdma
#show_gids //看看网卡支持的roce版本
DEV PORT INDEX GID IPv4 VER DEV
--- ---- ----- --- ------------ --- ---
mlx5_0 1 0 fe80:0000:0000:0000:1270:fdff:fe27:f982 v1 eth2
mlx5_0 1 1 fe80:0000:0000:0000:1270:fdff:fe27:f982 v2 eth2
mlx5_0 1 2 0000:0000:0000:0000:0000:ffff:0ac0:3398 10.192.51.152 v1 eth2
mlx5_0 1 3 0000:0000:0000:0000:0000:ffff:0ac0:3398 10.192.51.152 v2 eth2
mlx5_1 1 0 fe80:0000:0000:0000:1270:fdff:fe27:f983 v1 eth3
mlx5_1 1 1 fe80:0000:0000:0000:1270:fdff:fe27:f983 v2 eth3
n_gids_found=6
NVMe target配置
#modprobe nvmet
#modprobe nvmet-rdma
#modprobe nvme-rdma
参考前面 nvme of tcp 的步骤
# ls /sys/kernel/config/nvmet/
hosts ports subsystems
#cd /sys/kernel/config/nvmet/subsystems
#mkdir nvme-test-target
#tree
.
└── nvme-test-target
├── allowed_hosts
├── attr_allow_any_host
├── attr_serial
├── attr_version
└── namespaces
3 directories, 3 files
#cd nvme-test-target/
#允许所有主机访问
#echo 1 > attr_allow_any_host
#nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 BTAC210600LH3P8AGN INTEL SSDPF2KX038TZ 1 3.84 TB / 3.84 TB 512 B + 0 B JCV10200
/dev/nvme1n1 BTAC210006RS3P8AGN INTEL SSDPF2KX038TZ 1 3.84 TB / 3.84 TB 512 B + 0 B JCV10200
/dev/nvme2n1 BTAC209007EB3P8AGN INTEL SSDPF2KX038TZ 1 3.84 TB / 3.84 TB 512 B + 0 B JCV10200
/dev/nvme3n1 BTAC210007A73P8AGN INTEL SSDPF2KX038TZ 1 3.84 TB / 3.84 TB 512 B + 0 B JCV10200
#可以看出/dev/nvme0n1 是namespace 1
#mkdir namespaces/1
#ll namespaces/1
total 0
-rw-r--r-- 1 root root 4096 Sep 3 20:17 device_nguid
-rw-r--r-- 1 root root 4096 Sep 3 20:17 device_path
-rw-r--r-- 1 root root 4096 Sep 3 20:17 device_uuid
-rw-r--r-- 1 root root 4096 Sep 3 20:17 enable
指定namespace 1的device并启用
#echo '/dev/nvme0n1' > namespaces/1/device_path
#echo 1 > namespaces/1/enable
创建端口号配置IP
#mkdir /sys/kernel/config/nvmet/ports/1
#cd /sys/kernel/config/nvmet/ports/1
#ll
total 0
-rw-r--r-- 1 root root 4096 Sep 3 20:21 addr_adrfam
-rw-r--r-- 1 root root 4096 Sep 3 20:21 addr_traddr
-rw-r--r-- 1 root root 4096 Sep 3 20:21 addr_treq
-rw-r--r-- 1 root root 4096 Sep 3 20:21 addr_trsvcid
-rw-r--r-- 1 root root 4096 Sep 3 20:21 addr_trtype
-rw-r--r-- 1 root root 4096 Sep 3 20:21 param_inline_data_size
drwxr-xr-x 2 root root 0 Sep 3 20:21 referrals
drwxr-xr-x 2 root root 0 Sep 3 20:21 subsystems
#echo '10.192.51.152' > addr_traddr
#低版本内核 4.19* 下面报过错误，升级5.*后正常
#echo rdma > addr_trtype
#echo 4420 > addr_trsvcid
#echo ipv4 > addr_adrfam
端口子接口绑定
#ln -s /sys/kernel/config/nvmet/subsystems/nvme-test-target/ /sys/kernel/config/nvmet/ports/1/subsystems/nvme-test-target
#dmesg -T| grep "enabling port"
[Fri Sep 13 18:00:05 2024] nvmet_rdma: enabling port 1 (10.192.51.152:4420)
NVMe initiator配置
#yum -y install nvme-cli
#modprobe nvmet
#modprobe nvmet-rdma
#modprobe nvme-rdma
# nvme discover -t rdma -q nvme-test-target -a 10.192.51.152 -s 4420
=====Discovery Log Entry 0======
trtype: rdma
adrfam: ipv4
subtype: unrecognized
treq: not specified, sq flow control disable supported
portid: 1
trsvcid: 4420
subnqn: nqn.2014-08.org.nvmexpress.discovery
traddr: 10.192.51.152
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms: rdma-cm
rdma_pkey: 0x0000
=====Discovery Log Entry 1======
trtype: rdma
adrfam: ipv4
subtype: nvme subsystem
treq: not specified, sq flow control disable supported
portid: 1
trsvcid: 4420
subnqn: nvme-test-target
traddr: 10.192.51.152
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms: rdma-cm
rdma_pkey: 0x0000
# nvme connect -t rdma -q nvme-test-target -n nvme-test-target -a 10.192.51.152 -s 4420
# nvme list #多出来了nvme4n1磁盘
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 PHAX333201XZ3P8CGN INTEL SSDPF2KX038T1 1 3.84 TB / 3.84 TB 512 B + 0 B 9CV10200
/dev/nvme1n1 BTAC209500853P8AGN INTEL SSDPF2KX038TZ 1 3.84 TB / 3.84 TB 512 B + 0 B JCV10200
/dev/nvme2n1 BTAC2095012Z3P8AGN INTEL SSDPF2KX038TZ 1 3.84 TB / 3.84 TB 512 B + 0 B JCV10200
/dev/nvme3n1 BTAC210601LW3P8AGN INTEL SSDPF2KX038TZ 1 3.84 TB / 3.84 TB 512 B + 0 B JCV10200
/dev/nvme4n1 2fb38638ea25c2ad3ee4 Linux 1 3.84 TB / 3.84 TB 512 B + 0 B 3.10.0-1