SPDK 部署NVMe over TCP

SPDK NVMe over Fabrics Target是一个用户空间应用程序,通过以太网,Infiniband或光纤通道等结构呈现块设备,SPDK目前支持RDMA和TCP传输。默认TCP transport就支持了,如果要走RDMA需要spdk编译时加相关参数开启。

前文讲到基于 Anolis OS release 8.6 kernel 5.10.134-13.an8.x86_64 安装了spdk 22.0

target端配置
接管nvme磁盘
复制代码
 #cd spdk
 #scripts/setup.sh status
 Hugepages
node     hugesize     free /  total
node0   1048576kB        0 /      0
node0      2048kB        0 /      0
node1   1048576kB        0 /      0
node1      2048kB        0 /      0

Type     BDF             Vendor Device NUMA    Driver           Device     Block devices
I/OAT    0000:00:04.0    8086   0e20   0       ioatdma          -          -
I/OAT    0000:00:04.1    8086   0e21   0       ioatdma          -          -
I/OAT    0000:00:04.2    8086   0e22   0       ioatdma          -          -
I/OAT    0000:00:04.3    8086   0e23   0       ioatdma          -          -
I/OAT    0000:00:04.4    8086   0e24   0       ioatdma          -          -
I/OAT    0000:00:04.5    8086   0e25   0       ioatdma          -          -
I/OAT    0000:00:04.6    8086   0e26   0       ioatdma          -          -
I/OAT    0000:00:04.7    8086   0e27   0       ioatdma          -          -
I/OAT    0000:80:04.0    8086   0e20   1       ioatdma          -          -
I/OAT    0000:80:04.1    8086   0e21   1       ioatdma          -          -
I/OAT    0000:80:04.2    8086   0e22   1       ioatdma          -          -
I/OAT    0000:80:04.3    8086   0e23   1       ioatdma          -          -
I/OAT    0000:80:04.4    8086   0e24   1       ioatdma          -          -
I/OAT    0000:80:04.5    8086   0e25   1       ioatdma          -          -
I/OAT    0000:80:04.6    8086   0e26   1       ioatdma          -          -
I/OAT    0000:80:04.7    8086   0e27   1       ioatdma          -          -
NVMe     0000:84:00.0    1c5f   0540   1       nvme             nvme0      nvme0n1

可以看到nvme driver是nvme,需要将设备控制权交给SPDK

复制代码
#scripts/setup.sh
0000:84:00.0 (1c5f 0540): Active devices: data@nvme0n1, so not binding PCI dev
0000:80:04.2 (8086 0e22): ioatdma -> uio_pci_generic
0000:80:04.3 (8086 0e23): ioatdma -> uio_pci_generic
0000:80:04.0 (8086 0e20): ioatdma -> uio_pci_generic
0000:80:04.1 (8086 0e21): ioatdma -> uio_pci_generic
0000:80:04.6 (8086 0e26): ioatdma -> uio_pci_generic
0000:80:04.7 (8086 0e27): ioatdma -> uio_pci_generic
0000:80:04.4 (8086 0e24): ioatdma -> uio_pci_generic
0000:80:04.5 (8086 0e25): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 0e22): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 0e23): ioatdma -> uio_pci_generic
0000:00:04.0 (8086 0e20): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 0e21): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 0e26): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 0e27): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 0e24): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 0e25): ioatdma -> uio_pci_generic
显然nvme 磁盘接管失败,原因是曾经被我格式化过了
# blkid /dev/nvme0n1
/dev/nvme0n1: UUID="3ea74d7b-1ee4-45c1-9a9e-6eee7c424270" BLOCK_SIZE="512" TYPE="xfs"
这里通过dd 擦除后重新接管即可

spdk 接管完成nvme磁盘后,内核态已经看不到原nvme磁盘

复制代码
# scripts/setup.sh status
Hugepages
node     hugesize     free /  total
node0   1048576kB        0 /      0
node0      2048kB     1024 /   1024
node1   1048576kB        0 /      0
node1      2048kB        0 /      0

Type     BDF             Vendor Device NUMA    Driver           Device     Block devices
I/OAT    0000:00:04.0    8086   0e20   0       uio_pci_generic  -          -
I/OAT    0000:00:04.1    8086   0e21   0       uio_pci_generic  -          -
I/OAT    0000:00:04.2    8086   0e22   0       uio_pci_generic  -          -
I/OAT    0000:00:04.3    8086   0e23   0       uio_pci_generic  -          -
I/OAT    0000:00:04.4    8086   0e24   0       uio_pci_generic  -          -
I/OAT    0000:00:04.5    8086   0e25   0       uio_pci_generic  -          -
I/OAT    0000:00:04.6    8086   0e26   0       uio_pci_generic  -          -
I/OAT    0000:00:04.7    8086   0e27   0       uio_pci_generic  -          -
I/OAT    0000:80:04.0    8086   0e20   1       uio_pci_generic  -          -
I/OAT    0000:80:04.1    8086   0e21   1       uio_pci_generic  -          -
I/OAT    0000:80:04.2    8086   0e22   1       uio_pci_generic  -          -
I/OAT    0000:80:04.3    8086   0e23   1       uio_pci_generic  -          -
I/OAT    0000:80:04.4    8086   0e24   1       uio_pci_generic  -          -
I/OAT    0000:80:04.5    8086   0e25   1       uio_pci_generic  -          -
I/OAT    0000:80:04.6    8086   0e26   1       uio_pci_generic  -          -
I/OAT    0000:80:04.7    8086   0e27   1       uio_pci_generic  -          -
NVMe     0000:84:00.0    1c5f   0540   1       uio_pci_generic  -          -

#lsblk |grep nvme
启动 nvmf-tgf
复制代码
# build/bin/nvmf_tgt
[2024-09-10 11:24:44.345225] Starting SPDK v22.09.1-pre git sha1 5668dcd58 / DPDK 22.07.0 initialization...
[2024-09-10 11:24:44.345344] [ DPDK EAL parameters: [2024-09-10 11:24:44.345370] nvmf [2024-09-10 11:24:44.345385] --no-shconf [2024-09-10 11:24:44.345397] -c 0x1 [2024-09-10 11:24:44.345424] --huge-unlink [2024-09-10 11:24:44.345440] --log-level=lib.eal:6 [2024-09-10 11:24:44.345452] --log-level=lib.cryptodev:5 [2024-09-10 11:24:44.345464] --log-level=user1:6 [2024-09-10 11:24:44.345476] --iova-mode=pa [2024-09-10 11:24:44.345488] --base-virtaddr=0x200000000000 [2024-09-10 11:24:44.345499] --match-allocations [2024-09-10 11:24:44.345511] --file-prefix=spdk_pid709777 [2024-09-10 11:24:44.345523] ]
EAL: No free 2048 kB hugepages reported on node 1
TELEMETRY: No legacy callbacks, legacy socket not created
[2024-09-10 11:24:44.392733] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2024-09-10 11:24:44.488248] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2024-09-10 11:24:44.488341] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
创建tcp传输

The TCP transport is configured with an I/O unit size of 16384 bytes, 8 max qpairs per controller, and an in capsule data size of 8192 bytes.

复制代码
#scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192
创建nvme块设备
复制代码
#scripts/rpc.py bdev_nvme_attach_controller -b NVMe1 -t PCIe -a 0000:84:00.0
NVMe1n1
将NVMe块设备分配给子系统
复制代码
# 创建子系统
#scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

# 将NVMe块设备分配给刚刚创建的子系统
#scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 NVMe1n1

# 为子系统添加TCP监听器
#scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 10.203.173.30 -s 4420
Host端配置

发现磁盘

复制代码
#nvme discover -t tcp -a 10.203.173.30 -s 4420

Discovery Log Number of Records 1, Generation counter 1
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  0
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode1
traddr:  10.203.173.30
sectype: none

挂载磁盘

复制代码
# nvme connect -t tcp -n nqn.2016-06.io.spdk:cnode1 -a 10.203.173.30 -s 4420   
#lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda       8:0    0   50G  0 disk 
└─sda1    8:1    0   50G  0 part /
nvme0n1 259:3    0  1.5T  0 disk
相关推荐
DoraBigHead11 分钟前
《电磁波的浪漫,铜线上的灵魂》——计算机网络·物理层全解版
网络协议
ssswywywht1 小时前
OSPF实验
网络
FCM661 小时前
HCIA第一次实验报告:静态路由综合实验
网络·tcp/ip·信息与通信
apihz1 小时前
VM虚拟机全版本网盘+免费本地网络穿透端口映射实时同步动态家庭IP教程
android·服务器·开发语言·网络·数据库·网络协议·tcp/ip
dog2502 小时前
TCP 传输时 sk_buff 的 clone 和 unclone
网络·网络协议·tcp/ip
学习溢出3 小时前
【网络安全】理解安全事件的“三分法”流程:应对警报的第一道防线
网络·安全·web安全·网络安全·ids
智慧化智能化数字化方案4 小时前
华为IPD(集成产品开发)流程是其研发管理的核心体系
网络·华为ipd流程·ipd流程体系·ipd产品研发
kfepiza4 小时前
Linux的NetworkManager的nmcli配置网桥(bridge) 笔记250712
linux·运维·网络·笔记·tcp/ip·ip·tcp
cui_win5 小时前
【网络】Linux 内核优化实战 - net.netfilter.nf_conntrack_buckets
linux·网络·.net
cui_win5 小时前
【网络】Linux 内核优化实战 - net.netfilter.nf_conntrack_tcp_timeout_established
linux·网络·.net