Xen3: Using multiple network cards

From openSUSE

Credits

This is a direct adaptation from a great article [1]. All credits go to original author. There are some OpenSuSE specific remarks.

Introduction

Xen is great. But installing more than one network card became a pain. There are some documents describing the principle but there seem no to be any real life examples.

Using a bridge for a Dom is generally a good idea but then all packets traversing the bridge can be intercepted by any Dom that is using the same bridge. Having a single network card in a Xen landscape also means that theoretically each Dom would be able to sniff all packets traversing this single network card including packets to and from other Doms. A solution is to have more than one network card attached to Xen using a single network card for a single dom.

Topology

The scenario described here has a server with 3 network cards installed and running Dom0 and at least two guest DomU. The first card should be used to access Dom0 and some other DomNs while the second and third network card should be used to purely access Dom1 rsp. Dom2. The Dom configuration file just needs to select the appropriate bridge for each dom.

eth0 - xenbr0 - Dom0, DomN
eth1 - xenbr1 - Dom1 (cannot be sniffed by Dom0, DomN or Dom2)
eth2 - xenbr2 - Dom2 (cannot be sniffed by Dom0, Dom1 or Domn)

Configuration

1. Change /etc/xen/xend-config.sxp to override default bridging script:
   #(network-script network-bridge)
   (network-script my-network-script)
2. Create /etc/xen/scripts/my-network-script to contain:
   #!/bin/sh
   
   dir=$(dirname "$0")
   "$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0
   "$dir/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=xenbr1
   "$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=xenbr2
3. In /etc/xen/vm/
   1. For Dom1:
      # use eth1 for Dom1
      vif = [ 'ip=xxx.yyy.zzz.ip,mac=xx:xx:xx:xx:xx:xx,bridge=xenbr1' ]
   2. For Dom2:
      # use eth2 for Dom2
      vif = [ 'ip=xxx.yyy.zzz.ip,mac=xx:xx:xx:xx:xx:xx,bridge=xenbr2' ]
   3. For any DomU:
      # use eth0 for all other DomU
      vif = [ 'ip=xxx.yyy.zzz.ip,mac=xx:xx:xx:xx:xx:xx,bridge=xenbr0' ]
4. You're good! The changes require restart to take effect. As said, there is no additional routing required in Dom0 or in DomU besides just normal routing as you would do with a single network card attached to Xen.

From DomU perspective nothing changes. Each DomU will automatically use the bridge defined in the configuration file. The only change in behavior you will notice is that the LEDs of the second and third NIC start blinking as soon as Dom1 rsp. Dom2 send or receive packets. You can even pull out the cable from the first NIC (eth0) while Dom1 (eth1) and Dom2 (eth2) continue working normally.

Configuration for SLES10 SP2 (xen-3.2.0)

1. Change /etc/xen/xend-config.sxp to override default bridging script:
   #(network-script network-bridge)
   (network-script my-network-script)
2. Create /etc/xen/scripts/my-network-script to contain:
   #!/bin/sh
   
   dir=$(dirname "$0")
   "$dir/network-bridge" "$@" netdev=eth0 bridge=eth0
   "$dir/network-bridge" "$@" netdev=eth1 bridge=eth1
   "$dir/network-bridge" "$@" netdev=eth2 bridge=eth2
3. In /etc/xen/vm/
   1. For Dom1:
      # use eth1 for Dom1
      vif = [ 'ip=xxx.yyy.zzz.ip,mac=xx:xx:xx:xx:xx:xx,bridge=xenbr1' ]
   2. For Dom2:
      # use eth2 for Dom2
      vif = [ 'ip=xxx.yyy.zzz.ip,mac=xx:xx:xx:xx:xx:xx,bridge=xenbr2' ]
   3. For any DomU:
      # use eth0 for all other DomU
      vif = [ 'ip=xxx.yyy.zzz.ip,mac=xx:xx:xx:xx:xx:xx,bridge=xenbr0' ]

Links

• http://www.debian-administration.org/articles/470 (The original Debian article)
• http://wiki.xensource.com/xenwiki/XenNetworking
• http://wiki.xensource.com/xenwiki/XenNetworkingSuse
• http://www.fridu.org/content/view/50/80/ (Script to automate complex XEN network setup)