Testing XDP with virtio-net

Jump to: navigation, search

The XDP support was added into virtio-net since kernel 4.10. It's convenient to test the XDP programs without any high-end network card. However, it's not so trivial to setup the VM guest.

To make XDP work on the virtio-net device through a TAP interface, two things are necessary: enabling multiqueue virtio-net and disabling LRO support in the virtio-net device.

Enable multiqueue virtio-net

  • To enable multiqueue virtio-net, the TAP interface also has to support multiqueue. However, the "tunctl" tool doesn't support the creation of the multiqueue tap, and we need "ip tuntap". ex:
 # ip tuntap add mode tap multi_queue user <user_name> name tap0 
This will create tap0 with multi_queue support for the specific user.
Note: To delete a multiqueue tap, "multi_queue" is necessary, or "ip tuntap" will get "TUNSETIFF: Invalid argument" from the kernel.
 # ip tuntap del mode tap multi_queue name tap0
This will delete tap0.
  • Besides the multiqueue TAP, we also need some special parameters for qemu.
1. enable multiqueue for the tap device:
 -netdev tap,vhost=on,queues=N
where N stands for the number of queue pairs.
Note: If qemu complains the permisssion issue, change the access right of /dev/vhost-net and try to start the guest again.
 # chmod 666 /dev/vhost-net
2. enable multiqueue and specify MSI-X (Message Signaled Interrupt) vectors for the virtio-net-pci device:
 -device virtio-net-pci,mq=on,vectors=2*N+2
where the formula for the number of MSI-X vectors results from: N vectors for TX (transmit) queues, N for RX (receive) queues, one for configuration purposes, and one for possible VQ (vector quantization) control.
By default, virtio-net would allocate at most (# of CPU) queues for RX/TX. When attaching an eBPF program to XDP, it would request extra (# of CPU) queues, so N is recommended to be at least 2*(# of CPU).
Ref: https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/cha.qemu.host.html#kvm.qemu.multiqueue
Note: Don't execute the command "ethtool -L eth0 combined 2*N". Check the maximum number of queues (M) with "ethtool -l eth0" and leave at least (# of CPU) queues for XDP.

Disable LRO support

  • In the virtio-net driver, the LRO support means the following bits exist in the virtio device features:
 VIRTIO_NET_F_GUEST_TSO4
 VIRTIO_NET_F_GUEST_TSO6
 VIRTIO_NET_F_GUEST_ECN
 VIRTIO_NET_F_GUEST_UFO
  • Using "ethtool -K eth0 tso off" doesn't work since the driver checks the feature bits from the virtio device directly.
  • Those feature bits actually rely on a special flag in the tun/tap device: IFF_VNET_HDR. If the flag is enabled, virtio-net assumes LRO is supported.
  • IFF_VNET_HDR can be disabled with the qemu parameter:
 -netdev tap,vnet_hdr=off
  • Those features also can be disabled individually.
 -device guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off

Start&Test

So, in the end, this is how I create the VM guest with 1 CPU to test XDP programs.

# ip tuntap add mode tap multi_queue user gary name tap0 
$ qemu-system-x86_64 [...] \
        -smp 1 \
        -netdev tap,id=hn1,ifname=tap0,script=no,downscript=no,vnet_hdr=off,vhost=on,queues=2 \
        -device virtio-net-pci,netdev=hn1,mq=on,vectors=6

or

$ qemu-system-x86_64 [...] \
        -smp 1 \
        -netdev tap,id=hn1,ifname=tap0,script=no,downscript=no,vhost=on,queues=2 \
        -device virtio-net-pci,netdev=hn1,mq=on,vectors=6,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off


  • The XDP test program is available in the package, bcc-examples. First, Set the IPs for the host and the VM guest. Assume the TAP interface in the host is tap0, the corresponding network interface in the guest is enp0s5, and the MAC address of enp0s5 is 52:54:00:12:34:56.
Host:
 # ifconfig tap0 192.168.100.1
Guest:
 # ifconfig enp0s5 192.168.100.2
Install python-bcc and bcc-examples in the VM guest, and run
 # python /usr/share/bcc/examples/networking/xdp/xdp_drop_count.py enp0s5
Then run the pktgen script from the kernel (samples/pktgen/):
 # /pktgen_sample03_burst_single_flow.sh -vx -i tap0 -d 192.168.100.2 -m 52:54:00:12:34:56 -t 1 -c 0 -b 0
If everything goes well, xdp_drop_count.py will show the packet drop rate.

Trouble-shootings

  • If the XDP program fails to be loaded, "dmesg" could provide some hints.
  • If you get the following message in dmesg or "operation not supported" from a bcc script, make sure vnet_hdr is off.
 "can't set XDP while host is implementing LRO, disable LRO first"
  • If you get the following message in dmesg or "Cannot allocate memory" from a bcc script, make sure multiqueue is enabled and the number of queues is sufficient.
 "request 3 queues but max is 1"