在虚拟机上使用SoftRoCE部署SPDK NVMe-oF
背景介绍
为什么我们需要在虚拟机上部署SPDK NVMe-oF呢？原因很明显：因为随着主机拥有越来越强大的内核，我们能更有效地利用内核资源；而对于NVMe-oF功能测试用例，不需要太多计算和内存资源，那么，我们可以充分利用虚拟机来进行NVMe-oF功能测试。 接下来，我们将讨论如何使用虚拟机和SoftRoCE来实现此目的。
设置SoftRoCE环境
设置两个Fedora26内核为4.15.0-041500-generic的虚拟机。两个虚拟机使用Vagrantfile连接到主机上名为vboxnet0的的网桥。 拓扑结构如下图：
图1 使用SoftRoCE在两台虚拟机上部署NVMe-oF的拓扑结构图
1. 在主机中，您可以使用如下vagrant命令列出有效的虚拟机：
# vagrant global-status --prune
id name provider state directory
--------------------------------------------------------------------------
440cdef sss virtualbox running /home/yidong/spdk_init/scripts/vagrant
c5e4e5a www virtualbox running /home/yidong/spdk/scripts/vagrant 以上显示了有关此计算机上所有已知Vagrant环境的信息。 这是缓存数据，可能不是最新的信息（使用“vagrant global-status --prune”来修剪无效条目）。 要与任何计算机进行交互，您可以转到该目录并运行Vagrant，也可以直接使用任何目录中的Vagrant命令使用该ID。 例如：
"vagrant destroy 1a2b3c4d"2. 然后，您可以使用以下命令登录虚拟机：
# vagrant ssh 440cdef
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.15.0-041500-generic x86_64)
* Documentation: Official Ubuntu Documentation
* Management:
* Support:
Get cloud support with Ubuntu Advantage Cloud Guest:
43 packages can be updated.
0 updates are security updates.
Last login: Tue Aug 7 01:48:33 2018 from 10.0.2.23. 使用场景- NVMe-oF
注意：
1. 以NVMe-oF为例
2. 此处我们使用rdma模式或本地映射模式运行fio、perf
使用fio运行NVMe-oF测试的步骤
NVMe-oF target(虚拟机1)
1. 克隆最新的SPDK
git clone GitHub - spdk/spdk: Storage Performance Development Kit2. 使用with rdma编译SPDK
./configure --with-rdma
make -j 643. 卸载mlx_ib驱动程序并加载NVMe-oF RDMA驱动程序
modprobe –rv mlx4_ib
modprobe nvme_rdma
modprobe nvme_fabrics4. 为非rdma NIC配置SoftRoCE
rxe_cfg start
rxe_cfg add enp0s8 (这里我们使用e1000网卡)
ifconfig enp0s8 172.28.128.125. 使用以下命令运行nvmf_tgt应用程序
cd spdk
./app/nvmf_tgt/nvmf_tgt -c app/nvmf_tgt/nvmf.conf.in
[Global]
[Nvmf]
MaxQueuesPerSession 4
AcceptorPollRate 10000
[Nvme]
TransportId "trtype:PCIe traddr:0000:0e:00.0" Nvme0
RetryCount 4
Timeout 0
ActionOnTimeout None
AdminPollRate 100000
HotplugEnable No
[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode1
Listen RDMA 172.28.128.12:4420
AllowAnyHost yes
Host nqn.2016-06.io.spdk:init
SN SPDK00000000000001
Namespace Nvme0n1 6. 当nvmf_tgt应用程序成功运行时，将看到以下内容
# ./app/nvmf_tgt/nvmf_tgt -c ../nvmf.conf
Starting SPDK v18.07 / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: nvmf -c 0x1 --legacy-mem --file-prefix=spdk_pid5808 ]
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/spdk_pid5808/mp_socket
EAL: Probing VFIO support...
app.c: 530:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 718:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
reactor.c: 492:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
bdev_nvme.c:1268:bdev_nvme_library_init: *WARNING*: Timeout (in seconds) was renamed to TimeoutUsec (in microseconds)
bdev_nvme.c:1269:bdev_nvme_library_init: *WARNING*: Please update your configuration file
EAL: PCI device 0000:00:0e.0 on NUMA socket 0
EAL: probe driver: 80ee:4e56 spdk_nvmeNVMe-oF Initiator (虚拟机 2)
1. 卸载mlx_ib驱动程序并加载NVMe-oF RDMA主机驱动程序
modprobe –rv mlx4_ib
modprobe nvme_rdma
modprobe nvme_fabrics2. 为非rdma NIC配置SoftRoCE
rxe_cfg start
rxe_cfg add enp0s8
(在这里我们使用e1000网卡）
ifconfig enp0s8 172.28.128.43. 连接子系统
nvme connect -t rdma -n "nqn.2016-06.io.spdk:cnode1" -a 172.28.128.12 -s 44204. 执行fio任务
fio fio_softroce.jobFio配置文件fio_softroce.job如下:
[global]
invalidate=1
norandommap=1
thread=1
rw=randrw
runtime=10
ioengine=libaio
direct=1
bs=4096
size=1G
iodepth=128
group_reporting
time_based=1
[job0]
filename=/dev/nvme0n15. 当fio成功完成时，将看到以下内容
# fio fio_softroce.job
job0: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.2.10
Starting 1 thread
open path: No such file or directory
Error getting slave device numbers.: No such file or directory
Jobs: 1 (f=1): [m(1)] [1.1% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 18m:59s]
job0: (groupid=0, jobs=1): err= 0: pid=6477: Tue Aug 7 01:26:12 2018
read : io=6388.0KB, bw=534202B/s, iops=130, runt= 12245msec
slat (usec): min=3, max=25613, avg=568.48, stdev=2447.25
clat (usec): min=600, max=2540.4K, avg=40228.82, stdev=227386.16
lat (usec): min=710, max=2557.8K, avg=40797.85, stdev=227690.07
clat percentiles (usec):
| 1.00th=[ 964], 5.00th=[ 1336], 10.00th=[ 1656], 20.00th=[ 2160],
| 30.00th=[ 2608], 40.00th=[ 5856], 50.00th=[ 9280], 60.00th=[12736],
| 70.00th=[15424], 80.00th=[18048], 90.00th=[23936], 95.00th=[30592],
| 99.00th=[995328], 99.50th=[2277376], 99.90th=[2473984], 99.95th=[2539520],
| 99.99th=[2539520]
bw (KB /s): min= 64, max= 3696, per=100.00%, avg=1770.71, stdev=1051.85
write: io=5944.0KB, bw=497072B/s, iops=121, runt= 12245msec
slat (usec): min=4, max=23449, avg=656.70, stdev=2460.03
clat (msec): min=3, max=2988, avg=416.84, stdev=605.87
lat (msec): min=3, max=2988, avg=417.50, stdev=605.89
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 21], 10.00th=[ 85], 20.00th=[ 119],
| 30.00th=[ 151], 40.00th=[ 182], 50.00th=[ 217], 60.00th=[ 265],
| 70.00th=[ 334], 80.00th=[ 465], 90.00th=[ 709], 95.00th=[ 2376],
| 99.00th=[ 2573], 99.50th=[ 2638], 99.90th=[ 2704], 99.95th=[ 2999],
| 99.99th=[ 2999]
bw (KB /s): min= 0, max= 2416, per=100.00%, avg=1335.50, stdev=846.13
lat (usec) : 750=0.10%, 1000=0.55%
lat (msec) : 2=8.27%, 4=10.12%, 10=8.08%, 20=18.52%, 50=7.82%
lat (msec) : 100=3.15%, 250=21.51%, 500=12.81%, 750=3.99%, 1000=0.94%
lat (msec) : 2000=0.03%, >=2000=4.12%
cpu : usr=0.00%, sys=1.80%, ctx=1999, majf=0, minf=1
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=1.0%, >=64=98.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1597/w=1486/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: io=6388KB, aggrb=521KB/s, minb=521KB/s, maxb=521KB/s, mint=12245msec, maxt=12245msec
WRITE: io=5944KB, aggrb=485KB/s, minb=485KB/s, maxb=485KB/s, mint=12245msec, maxt=12245msec
Disk stats (read/write):
nvme0n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%6. 断开子系统
nvme disconnect -d /dev/nvme0n1在rdma模式下使用perf在NVMe-oF测试上运行的步骤
NVMe-oF Target (虚拟机 1)
步骤和fio完全一样。
NVMe-oF Target (虚拟机 2)
1. 断开现有的nvme设备
nvme disconnect –d /dev/nvme0n12. 使用 setup.sh运行安装程序
./script/setup.sh3. 使用rdma模式运行perf测试
./examples/nvme/perf/perf –q 1 –w randread –s 4096 –t 10 –r ‘trtype:RDMA adrfam:IPv4 traddr:172.28.128.12 trsvcid:4420’4. 当perf成功完成运行后，您将看到打印的内容如下
# ./examples/nvme/perf/perf -q 1 -w randread -s 4096 -t 10 -r 'trtype:RDMA adrfam:IPv4 traddr:172.28.128.12 trsvcid:4420'
Starting SPDK v18.10-pre / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --no-pci --legacy-mem --file-prefix=spdk_pid5138 ]
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/spdk_pid5138/mp_socket
EAL: Probing VFIO support...
Initializing NVMe Controllers
Attaching to NVMe over Fabrics controller at 172.28.128.12:4420: nqn.2016-06.io.spdk:cnode1
Attached to NVMe over Fabrics controller at 172.28.128.12:4420: nqn.2016-06.io.spdk:cnode1
Associating SPDK bdev Controller (SPDK00000000000001 ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
SPDK bdev Controller (SPDK00000000000001 ) from core 0: 3389.30 13.24 294.97 143.31 17376.58
========================================================
Total : 3389.30 13.24 294.97 143.31 17376.58在NVMe-oF测试中使用perf在本地运行的步骤
NVMe-oF Target (虚拟机 1)
步骤和fio完全一样。
NVMe-oF Initiator (虚拟机 2)
1. 连接子系统
nvme connect –t rdma –n “nqn.2016-06.io.spdk:cnode1” –a 172.28.128.12 –s 44202. 列出连接的nvme设备
nvme list3. 在本地设备上运行perf测试
./examples/nvme/perf/perf /dev/nvme0n1 –q 1 –w randread –s 4096 –t 104. 当perf成功完成运行后，您将看到打印的消息如下
# ./examples/nvme/perf/perf /dev/nvme0n1 -q 1 -w randread -s 4096 -t 10
Starting SPDK v18.10-pre / DPDK 18.05.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --legacy-mem --file-prefix=spdk_pid5122 ]
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/spdk_pid5122/mp_socket
EAL: Probing VFIO support...
Initializing NVMe Controllers
EAL: PCI device 0000:00:0e.0 on NUMA socket 0
EAL: probe driver: 80ee:4e56 spdk_nvme
Attaching to NVMe Controller at 0000:00:0e.0
Attached to NVMe Controller at 0000:00:0e.0 [80ee:4e56]
Associating ORCL-VBOX-NVME-VER12 (VB1234-56789 ) with lcore 0
Associating /dev/nvme0n1 with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS MB/s Average min max
/dev/nvme0n1 from core 0: 0.50 0.00 2286420.25 2223980.88 2307051.46
ORCL-VBOX-NVME-VER12 (VB1234-56789 ) from core 0: 11283.60 44.08 101.63 33.21 1474178.36
========================================================
Total : 11284.10 44.08 202.94 33.21 2307051.465. 断开子系统
nvme disconnect -d /dev/nvme0n1Q & A
1. 在NVMe-oF initiator连接到NVMe-oFtarget后，如果使用lsblk命令检查连接的命名空间，将看到以下内容。也可以使用nvme list检查命名空间。
# lsblk
lsblk: nvme0c1n1: unknown device name
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 10M 0 disk
sda 8:0 0 40G 0 disk
└─sda1 8:1 0 40G 0 part /2. 在NVMe-oF initiator上，如果在使用perf命令“./examples/nvme/perf/perf / dev / nvme0n1 -q 1 -s 4096 -w randread -t 10”运行时遇到以下问题，可以重新启动虚拟机，此问题将得到解决。
# ./examples/nvme/perf/perf /dev/nvme0n1 -q 1 -s 4096 -w randread -t 10
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --file-prefix=spdk_pid8197 ]
EAL: Detected 2 lcore(s)
EAL: No free hugepages reported in hugepages-2048kB
EAL: FATAL: Cannot get hugepage information.
EAL: Cannot get hugepage information.
Failed to initialize DPDK
Unable to initialize SPDK env
./examples/nvme/perf/perf: errors occured原文链接：https://blog.csdn.net/weixin_37097605/article/details/101514499
学习地址：http://ke.qq.com/course/5066203?flowToken=1043717
更多DPDK学习资料有需要的可以自行添加进入学习交流君 羊 793599096 免费获取，或自行报名学习，免费订阅，永久学习，关注我持续更新哦！！！