KVM Network Configuration

qemu-kvm guest network

KVM主机上一般网络配置有5种类型：

• Using QEMU default network 10.0.2.15

• Using a Linux bridge with NAT for KVM guests

• Using a Linux bridge (without NAT) for KVM guests

• Using an Open vSwitch bridge with KVM guests

• Using the MacVTap driver with KVM guests

QEMU default

qemu启动虚机，如果不做网卡的特定配置，默认起来后网卡是10.0.2.15地址。

    -device e1000,netdev=net0,mac=00:16:3e:0c:21:28
    -netdev user,id=net0,hostfwd=tcp::19888-:22

这里加了个hostfwd进行端口转发，hostfwd=tcp::19888-:22的含义是将host主机的19888端口的tcp报文转发给guest虚机的22端口（ssh登陆）。

NAT-based Net

依赖支持：libvirt，tap.

这个是最常用的配置了，通过Linux bridge以及NAT的结合使用，使得guest OS可以通过任何网络形式（wired，wireless，dial-up等）连接到主机之外。

所有的guest虚机都可以通过该bridge互通，即使该bridge并没有连接到host主机的实际网卡上或者host根本就没有physical网卡，也不影响虚机之间通过该bridge互相通信。

该模式下，有3中主要特性（主要是NAT带来的特性）：

• 该NAT-based的bridge使用的是私网地址，192.168.x.x网段的地址；
• 因为guest躲在NAT之后，所以不同的host之间是无法感知到某个host内部如何分配ip给guest的，也即外部不能直通guest；
• 使用NAT和software bridge会带来额外的CPU和memory开销，网络throughput以及latency性能会打折扣；

通常，该模式下默认的NAT bridge为virbr0，默认的network为default。

[root@pc:ubuntu#] virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes

[root@pc:ubuntu#] brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.5254003203df       no              tap0
                                                        virbr0-nic

Tutorial

1. 创建配置文件

创建配置文件：vi ~/jrgnet.xml

<network>
    <name>newnatnetwork</name>
    <forward mode=’nat’>
        <nat>
            <port start=’1024’ end=’65535’/>
        </nat>
    </forward>
    <bridge name=’my-bridge-name’ stp=’on’ delay=’0’/>
    <ip address=’192.168.X.1’ netmask=’255.255.255.0’>
        <dhcp>
            <range start=’192.168.X.2’ end=’192.168.X.254’/>
        </dhcp>
    </ip>
</network>

其中需要修改<name>, <bridge name>, <ip address>。

• name字段需要修改，比如jrgnet
• bridge name需要修改，比如jrgbr
• 192.168.X.1地址需要指明网段，比如192.168.128.1

我这里的jrgnet.xml内容如下

<network>
    <name>jrgnet</name>
    <forward mode='nat'>
        <nat>
            <port start='1024' end='65535'/>
        </nat>
    </forward>
    <bridge name='jrgbr0' stp='on' delay='0'/>
    <ip address='192.168.128.1' netmask='255.255.255.0'>
        <dhcp>
            <range start='192.168.128.2' end='192.168.128.254'/>
        </dhcp>
    </ip>
</network>

2. 创建新的network

virsh net-create ~/jrgnet.xml

该步骤中会自动创建配置文件中的bridge，比如jrgbr0，通过brctl show可以查看；virsh net-list可以查看创建的network比如jrgnet；

[root@pc:rg#] virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 jrgnet               active     no            no

[root@pc:rg#] brctl show
bridge name     bridge id               STP enabled     interfaces
jrgbr0          8000.525400012bad       yes             jrgbr0-nic
virbr0          8000.5254003203df       no              tap0
                                                        virbr0-nic

3. 启动新的network

virsh net-autostart <network-name-from-xml>，比如virsh net-autostart jrgnet。

不过这里可能会出问题

[root@pc:rg#] virsh net-autostart jrgnet
error: failed to mark network jrgnet as autostarted
error: Requested operation is not valid: cannot set autostart for transient network

解决办法：

应该是我们的配置文件jrgnet.xml没有通过virsh命令去配置

virsh net-edit jrgnet

并实际修改并保存再退出，然后运行virsh net-autostart jrgnet，这样就会自动拷贝jrgnet.xml 至*/etc/libvirtd/qemu/networks/autostart/*目录下面了，就可以了。

[root@pc:rg#] virsh net-autostart jrgnet
Network jrgnet marked as autostarted

[root@pc:rg#] virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 jrgnet               active     yes           yes

4. 配置guest使用该network或bridge

我这里需要做两个操作。

• 修改qemu启动命令中，br=virbr0改为br=jrgbr0
• 修改 /usr/local/etc/qemu/bridge.conf中添加allow jrgbr0

然后重启guest虚机，就可以看到内部的IP地址变为我们指定的subnet网段了，这里是192.168.128.96。

qemu命令中需要添加的网卡配置，其中mac地址可以随便改，只要br=jrgbr0对应到指定的bridge即可（注意需要qemu-bridge-helper的支持）。

    -device e1000,netdev=nic0,mac=00:16:3e:0c:12:27 \
    -netdev tap,id=nic0,br=jrgbr0,helper=/home/rg/project/qemu-5.1.0/build/qemu-bridge-helper,vhost=on \

/usr/local/etc/qemu/bridge.conf中的配置如下，可能第一次是没有该bridge.conf文件的，直接创建。

[root@pc:ubuntu#] cat /usr/local/etc/qemu/bridge.conf
allow virbr0
allow jrgbr0

Linux-bridge

使用原生的Linux网桥。

A network bridge is a Link Layer device which forwards traffic between networks based on MAC addresses and is therefore also referred to as a Layer 2 device. It makes forwarding decisions based on tables of MAC addresses which it builds by learning what hosts are connected to each network. A software bridge can be used within a Linux host in order to emulate a hardware bridge, for example in virtualization applications for sharing a NIC with one or more virtual NICs.

Tutorial

1. 创建bridge

首先我们需要创建一个bridge，其slave是我们想和外部网络连接的interface。可以参考这篇文章，这样虚拟机可以基于该bridge和外部网络进行直连。

可以使用nmtui界面化工具配置网桥，配置完成后systemctl restart network即可。

配置完bridge，比如我这里配置jrgbr1，其效果如下

[root@pc:ubuntu#] brctl show
bridge name     bridge id               STP enabled     interfaces
jrgbr0          8000.525400012bad       no              jrgbr0-nic
jrgbr1          8000.008cfac79928       no              ens1f0
virbr0          8000.5254003203df       no              virbr0-nic

[root@pc:ubuntu#] nmcli connection show --active
NAME       UUID                                  TYPE      DEVICE
br1        3d2c04c9-6016-483c-9a4f-d2e31c642c79  bridge    jrgbr1
jrgbr0     2701003f-d845-4a41-b6cc-50ee7d027ebd  bridge    jrgbr0
virbr0     c0627583-68bc-4660-9a63-7957362c0871  bridge    virbr0
br1-slave  0646fd10-1580-4229-890d-dd6c0f46f716  ethernet  ens1f0

[root@pc:ubuntu#] ip address show master jrgbr1
2: ens1f0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master jrgbr1 state UP group default qlen 1000
    link/ether 00:8c:fa:c7:99:28 brd ff:ff:ff:ff:ff:ff

[root@pc:ubuntu#] ip a show jrgbr1
17: jrgbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:8c:fa:c7:99:28 brd ff:ff:ff:ff:ff:ff
    inet 172.17.81.244/24 brd 172.17.81.255 scope global noprefixroute jrgbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::28c:faff:fec7:9928/64 scope link
       valid_lft forever preferred_lft forever

2. qemu命令支持

qemu的启动命令其实不用怎么修改，只要修改br=jrgbr1即可，比如下面这个我启动了两张网卡，net0网卡启动后是10.0.2.15地址，nic0网卡启动后是配置static ip或dhcp获取ens1f0上级dhcpserver分配的地址。

    -device e1000,netdev=net0,mac=00:16:3e:0c:21:28 \
    -netdev user,id=net0,hostfwd=tcp::19888-:22 \
    -device e1000,netdev=nic0,mac=00:16:3e:0c:12:27 \
    -netdev tap,id=nic0,br=jrgbr1,helper=/home/rg/project/qemu-5.1.0/build/qemu-bridge-helper,vhost=on \

启动虚机后，可以看到我找了建立了两个虚拟网卡（不是必须两个网卡，一个网卡即可）。其中，mac地址00:16:3e:0c:21:28的网卡地址是10.0.2.15（这是qemu创建虚机默认配置的ip地址），另一张mac地址00:16:3e:0c:12:27的网卡地址这里还没有配置（这里我host主机的ens1f0物理网卡设置的是static ip，没有设置dhcp）。

设置ens4的ip地址，可以发现其已经能和host主机网络中的其他主机ping通。

ip addr add 172.17.81.27/24 dev ens4

这里再看一下，如果ping host主机网络中的非同一个网段的，直接ping貌似ping不通，这是因为我这里建立了两张网卡，ens3的ip地址为10.0.2.15，如果ping非同一网段的，按照默认路由走就走10.0.2.15网卡，走到qemu的虚拟网络协议栈，qemu默认网卡是不支持ping功能的，所以ping不通，如果ping的时候指定网卡ens4，则可以。

在host网络中其他主机内ping我们的guest OS虚机172.17.81.27，也是可以通的，bridge桥接模式创建虚机网卡设置成功。

OpenvSwitch

需要安装openvswitch

其主要步骤也即：

• 创建网桥，其slave是host主机的interface网卡
• 指定qemu使用该bridge

MacVTap

Macvtap is a new device driver meant to simplify virtualized bridged networking. It replaces the combination of the tun/tap and bridge drivers with a single module based on the macvlan device driver. A macvtap endpoint is a character device that largely follows the tun/tap ioctl interface and can be used directly by kvm/qemu and other hypervisors that support the tun/tap interface.

MacVTap可以配置3中模式：

1. 默认模式是VEPA(Virtual Ethernet Port Aggregator，虚拟网络端口聚合)，数据从某个endpoint传输给外部switch，如果switch支持haripin功能，则数据还能回流给source设备然后传给其他endpoint。

   Virtual Ethernet Port Aggregator (VEPA) is the default mode. Data flows from one endpoint down through the source device in the KVM host out to the external switch. If the switch supports hairpin mode, the data is sent back to the source device in the KVM host and from there sent to the destination endpoint.
   Most switches today do not support hairpin mode, so the two endpoints are not able to exchange ethernet frames, although they might still be able to communicate using an tcp/ip router. A linux host used as the adjacent bridge can be put into hairpin mode by writing to /sys/class/net/dev/brif/port/ hairpin_mode. This mode is particularly interesting for data-centers and cloud provides where the ability to manage virtual machine networking at the switch level is desirable. Switches aware of the VEPA guests can enforce filtering and bandwidth limits per MAC address without the Linux host knowing about it.

2. 网桥模式，这是最常用的模式。两个终端之间直接传输帧数据。

   Bridge, connecting all endpoints directly to each other. Two endpoints that are both in bridge mode can exchange frames directly, without the round trip through the external bridge. This is the most useful mode for setups with classic switches, and when inter-guest communication is performance critical.

   原理图如下所示：
   

3. 私有模式，终端之间不能互通。

   For completeness, a private mode exists that behaves like a VEPA mode endpoint in the absence of a hairpin aware switch. Even when the switch is in hairpin mode, a private endpoint can never communicate to any other endpoint on the same lowerdev.

实例

附上一个完整的qemu启动命令以及host主机网桥配置。

qemu启动命令：

#!/bin/sh

/home/rg/project/qemu-5.1.0/build/x86_64-softmmu/qemu-system-x86_64 \
    -enable-kvm \
    -m ${1:-8}G \
    -cpu kvm64,+vmx \
    -vga virtio \
    -display default \
    -smp ${2:-4} \
    -device e1000,netdev=net0,mac=00:16:3e:0c:21:28 \
    -netdev user,id=net0,hostfwd=tcp::19888-:22 \
    -device e1000,netdev=nic0,mac=00:16:3e:0c:12:27 \
    -netdev tap,id=nic0,br=jrgbr1,helper=/home/rg/project/qemu-5.1.0/build/qemu-bridge-helper,vhost=on \
    -device virtio-serial \
    -chardev spicevmc,id=vdagent,debug=0,name=vdagent \
    -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \
    -drive aio=threads,file=/home/rg/ubuntu/ubuntu-20.04.1.qcow2,if=virtio \
    -name jrgtest,process=testjrg,debug-threads=on \
    -spice port=8000,addr=172.17.81.244,disable-ticketing

#    -cdrom ./ubuntu-20.04.1-desktop-amd64.iso \	#用于初次安装ubuntu系统

#    -device e1000,netdev=net0,mac=00:16:3e:0c:21:28 \		#用于默认的qemu网卡配置，并转发ssh登陆
#    -netdev user,id=net0,hostfwd=tcp::19888-:22 \

#    -device e1000,netdev=net0,mac=00:16:3e:0c:22:19 \		#用于默认的qemu网卡配置
#    -netdev user,id=net0 \

#    -device e1000,netdev=nic0,mac=00:16:3e:0c:12:27 \		#用于自创建的网桥NAT模式
#    -netdev tap,id=nic0,br=jrgbr0,helper=/home/rg/project/qemu-5.1.0/build/qemu-bridge-helper,vhost=on \

#    -device virtio-net-pci,netdev=nic0,mac=00:16:3e:0c:12:29 \		#用于默认default网络的网桥NAT模式
#    -netdev tap,id=nic0,br=virbr0,helper=/home/rg/project/qemu-5.1.0/build/qemu-bridge-helper,vhost=on \

# jrgbr1 whose slave dev is ens1f0, guest set ip 172.17.81.x/24		#用于桥接到host主机网卡模式
#    -device e1000,netdev=nic0,mac=00:16:3e:0c:12:27 \
#    -netdev tap,id=nic0,br=jrgbr1,helper=/home/rg/project/qemu-5.1.0/build/qemu-bridge-helper,vhost=on 

host主机网桥配置：tap0是启动qemu虚机其自动创建的，并不是我预先创建的

[root@pc:rg#] nmcli connection show --active
NAME       UUID                                  TYPE      DEVICE
br1        3d2c04c9-6016-483c-9a4f-d2e31c642c79  bridge    jrgbr1
jrgbr0     2701003f-d845-4a41-b6cc-50ee7d027ebd  bridge    jrgbr0
virbr0     c0627583-68bc-4660-9a63-7957362c0871  bridge    virbr0
br1-slave  0646fd10-1580-4229-890d-dd6c0f46f716  ethernet  ens1f0
tap0       efc78717-334e-4b8b-a86b-043c61dfb063  tun       tap0

[root@pc:ubuntu#] brctl show
bridge name     bridge id               STP enabled     interfaces
jrgbr0          8000.525400012bad       no              jrgbr0-nic
jrgbr1          8000.008cfac79928       no              ens1f0
                                                        tap0
virbr0          8000.5254003203df       no              virbr0-nic

[root@pc:ubuntu#] virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 jrgnet               active     yes           yes

default.xml配置文件：

[root@pc:ubuntu#] cat /etc/libvirt/qemu/networks/default.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh net-edit default
or other application using the libvirt API.
-->

<network>
  <name>default</name>
  <uuid>f6dd973b-dceb-493e-88ac-6a5d830c23e9</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='off' delay='0'/>
  <mac address='52:54:00:32:03:df'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

jrgnet.xml配置：

  virsh net-eidt jrgnet    
      1 <network>
      2   <name>jrgnet</name>
      3   <uuid>d6d5dede-1ecd-4d1f-8ef2-86ee57b88535</uuid>
      4   <forward mode='nat'>
      5     <nat>
      6       <port start='1024' end='65535'/>
      7     </nat>
      8   </forward>
      9   <bridge name='jrgbr0' stp='off' delay='0'/>
     10   <mac address='52:54:00:01:2b:ad'/>
     11   <ip address='192.168.128.1' netmask='255.255.255.0'>
     12     <dhcp>
     13       <range start='192.168.128.2' end='192.168.128.254'/>
     14     </dhcp>
     15   </ip>
     16 </network>\

bridge.conf配置：

[root@pc:ubuntu#] cat /usr/local/etc/qemu/bridge.conf
allow virbr0
allow jrgbr0
allow jrgbr1

Reference

[KVM Network Performance -Best Practices and Tuning Recommendations]

配置QEMU虚机网络