SPDK 部署NVMe over TCP
SPDK NVMe over Fabrics Target是一个用户空间应用程序，通过以太网，Infiniband或光纤通道等结构呈现块设备,SPDK目前支持RDMA和TCP传输。默认TCP transport就支持了，如果要走RDMA需要spdk编译时加相关参数开启。
前文讲到基于 Anolis OS release 8.6 kernel 5.10.134-13.an8.x86_64 安装了spdk 22.0
target端配置
接管nvme磁盘
#cd spdk#scripts/setup.sh statusHugepages
node hugesize free / total
node0 1048576kB 0 / 0
node0 2048kB 0 / 0
node1 1048576kB 0 / 0
node1 2048kB 0 / 0Type BDF Vendor Device NUMA Driver Device Block devices
I/OAT 0000:00:04.0 8086 0e20 0 ioatdma - -
I/OAT 0000:00:04.1 8086 0e21 0 ioatdma - -
I/OAT 0000:00:04.2 8086 0e22 0 ioatdma - -
I/OAT 0000:00:04.3 8086 0e23 0 ioatdma - -
I/OAT 0000:00:04.4 8086 0e24 0 ioatdma - -
I/OAT 0000:00:04.5 8086 0e25 0 ioatdma - -
I/OAT 0000:00:04.6 8086 0e26 0 ioatdma - -
I/OAT 0000:00:04.7 8086 0e27 0 ioatdma - -
I/OAT 0000:80:04.0 8086 0e20 1 ioatdma - -
I/OAT 0000:80:04.1 8086 0e21 1 ioatdma - -
I/OAT 0000:80:04.2 8086 0e22 1 ioatdma - -
I/OAT 0000:80:04.3 8086 0e23 1 ioatdma - -
I/OAT 0000:80:04.4 8086 0e24 1 ioatdma - -
I/OAT 0000:80:04.5 8086 0e25 1 ioatdma - -
I/OAT 0000:80:04.6 8086 0e26 1 ioatdma - -
I/OAT 0000:80:04.7 8086 0e27 1 ioatdma - -
NVMe 0000:84:00.0 1c5f 0540 1 nvme nvme0 nvme0n1
可以看到nvme driver是nvme，需要将设备控制权交给SPDK
#scripts/setup.sh
0000:84:00.0 (1c5f 0540): Active devices: data@nvme0n1, so not binding PCI dev
0000:80:04.2 (8086 0e22): ioatdma -> uio_pci_generic
0000:80:04.3 (8086 0e23): ioatdma -> uio_pci_generic
0000:80:04.0 (8086 0e20): ioatdma -> uio_pci_generic
0000:80:04.1 (8086 0e21): ioatdma -> uio_pci_generic
0000:80:04.6 (8086 0e26): ioatdma -> uio_pci_generic
0000:80:04.7 (8086 0e27): ioatdma -> uio_pci_generic
0000:80:04.4 (8086 0e24): ioatdma -> uio_pci_generic
0000:80:04.5 (8086 0e25): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 0e22): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 0e23): ioatdma -> uio_pci_generic
0000:00:04.0 (8086 0e20): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 0e21): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 0e26): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 0e27): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 0e24): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 0e25): ioatdma -> uio_pci_generic
显然nvme 磁盘接管失败，原因是曾经被我格式化过了
# blkid /dev/nvme0n1
/dev/nvme0n1: UUID="3ea74d7b-1ee4-45c1-9a9e-6eee7c424270" BLOCK_SIZE="512" TYPE="xfs"
这里通过dd 擦除后重新接管即可
spdk 接管完成nvme磁盘后，内核态已经看不到原nvme磁盘
# scripts/setup.sh status
Hugepages
node hugesize free / total
node0 1048576kB 0 / 0
node0 2048kB 1024 / 1024
node1 1048576kB 0 / 0
node1 2048kB 0 / 0Type BDF Vendor Device NUMA Driver Device Block devices
I/OAT 0000:00:04.0 8086 0e20 0 uio_pci_generic - -
I/OAT 0000:00:04.1 8086 0e21 0 uio_pci_generic - -
I/OAT 0000:00:04.2 8086 0e22 0 uio_pci_generic - -
I/OAT 0000:00:04.3 8086 0e23 0 uio_pci_generic - -
I/OAT 0000:00:04.4 8086 0e24 0 uio_pci_generic - -
I/OAT 0000:00:04.5 8086 0e25 0 uio_pci_generic - -
I/OAT 0000:00:04.6 8086 0e26 0 uio_pci_generic - -
I/OAT 0000:00:04.7 8086 0e27 0 uio_pci_generic - -
I/OAT 0000:80:04.0 8086 0e20 1 uio_pci_generic - -
I/OAT 0000:80:04.1 8086 0e21 1 uio_pci_generic - -
I/OAT 0000:80:04.2 8086 0e22 1 uio_pci_generic - -
I/OAT 0000:80:04.3 8086 0e23 1 uio_pci_generic - -
I/OAT 0000:80:04.4 8086 0e24 1 uio_pci_generic - -
I/OAT 0000:80:04.5 8086 0e25 1 uio_pci_generic - -
I/OAT 0000:80:04.6 8086 0e26 1 uio_pci_generic - -
I/OAT 0000:80:04.7 8086 0e27 1 uio_pci_generic - -
NVMe 0000:84:00.0 1c5f 0540 1 uio_pci_generic - -#lsblk |grep nvme
启动 nvmf-tgf
# build/bin/nvmf_tgt
[2024-09-10 11:24:44.345225] Starting SPDK v22.09.1-pre git sha1 5668dcd58 / DPDK 22.07.0 initialization...
[2024-09-10 11:24:44.345344] [ DPDK EAL parameters: [2024-09-10 11:24:44.345370] nvmf [2024-09-10 11:24:44.345385] --no-shconf [2024-09-10 11:24:44.345397] -c 0x1 [2024-09-10 11:24:44.345424] --huge-unlink [2024-09-10 11:24:44.345440] --log-level=lib.eal:6 [2024-09-10 11:24:44.345452] --log-level=lib.cryptodev:5 [2024-09-10 11:24:44.345464] --log-level=user1:6 [2024-09-10 11:24:44.345476] --iova-mode=pa [2024-09-10 11:24:44.345488] --base-virtaddr=0x200000000000 [2024-09-10 11:24:44.345499] --match-allocations [2024-09-10 11:24:44.345511] --file-prefix=spdk_pid709777 [2024-09-10 11:24:44.345523] ]
EAL: No free 2048 kB hugepages reported on node 1
TELEMETRY: No legacy callbacks, legacy socket not created
[2024-09-10 11:24:44.392733] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2024-09-10 11:24:44.488248] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2024-09-10 11:24:44.488341] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
创建tcp传输
The TCP transport is configured with an I/O unit size of 16384 bytes, 8 max qpairs per controller, and an in capsule data size of 8192 bytes.
#scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192创建nvme块设备
#scripts/rpc.py bdev_nvme_attach_controller -b NVMe1 -t PCIe -a 0000:84:00.0
NVMe1n1
将NVMe块设备分配给子系统
# 创建子系统
#scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1# 将NVMe块设备分配给刚刚创建的子系统
#scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 NVMe1n1# 为子系统添加TCP监听器
#scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 10.203.173.30 -s 4420Host端配置
发现磁盘
#nvme discover -t tcp -a 10.203.173.30 -s 4420Discovery Log Number of Records 1, Generation counter 1
=====Discovery Log Entry 0======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not required
portid: 0
trsvcid: 4420
subnqn: nqn.2016-06.io.spdk:cnode1
traddr: 10.203.173.30
sectype: none
挂载磁盘
# nvme connect -t tcp -n nqn.2016-06.io.spdk:cnode1 -a 10.203.173.30 -s 4420
#lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
└─sda1 8:1 0 50G 0 part /
nvme0n1 259:3 0 1.5T 0 disk