Using Linux virtual bridging, User-Mode-Linux and Zebra for IP routing exercises
Christian Hammers <firstname.lastname@example.org>
v1.0 - last modified at 2003-06-21
- Description of my test setup
- Preperation of UML
- Basic Networking
- Setting up Zebra
- Benchmarking UML
- Sample config files
- References and further literature
Shameless self advertising:
by me was printed in the 01/2004 issue of the German "Linux-Magazin". Go and read it, too! :-)
This tutorial shows how to create several, via OSPF interconnected, IP over
Ethernet networks consisting of up to approx. ten hosts (depending on the
available hardware) without additional hardware by using simulated hosts and
Only open and free software is used: GNU/Linux for the host system, Linux
Bridging to for the virtual network multiport hub/switch, TUN/TAP for the
virtual network interfaces, User-Mode-Linux to host the virtual systems and
Zebra as OSPF and optionally BGP routing software.
Other projects/howtos that aim at similar goals:
- "UML-based pseudo-dedicated hosting service" 
- Describes how to set up commercial hosting services with uml as compromise between dedicated hosting and virtual servers.
- Sharing a hosted server at 
- An article about using UML to share an ISP-hosted server between several people to save money.
User-Mode-Linux (short "UML")  is a project that enhances the
standard linux kernel  in a way that allowes it to run as an
executable program inside a normal linux environment. Unlike chroot which only
jails one into another file system hierarchy, uml also has a completely
seperated process list, devices, process sheduling memory and what is important
for this scenario its own IP stack.
More resources can also be found at the UML community site  and
at yet another UML fan site .
About Linux Bridging and TUN/TAP
The Linux Bridging  Code simulates the behaviour of a
multiport Ethernet switch or bridge. The main part is already included into the
Linux kernel code and has to be enabled there.
The TUN/TAP software  provides ethernet network interfaces
that can be configured for an unprivileged user and attached to a virtual bridge.
The Zebra  routing suite is a modular application for RIP, OSPF and BGP routing.
Zebra has a Cisco ®  like configuration style which makes it especially
usefull for practicing. It is stable and capable enough to run in production at various ISPs.
More info about Zebra can be found at the "Unofficial GNU Zebra FAQ"  and
at zebra-pj  which is the unofficial successor to the somewhat orphaned Zebra.
About OSPF and BGP
OSPF is a link-state intranet routing protocol for IP networks. If you read
this, you should already know this. Some good information can be found in the
references ,  and . OSPF
is used as example for this scenario because it is the most commonly used IP
over Ethernet routing protocol and uses Ethernet specific features like
multicast which are especially interesting to test the capabilities of the
BGP  was also tried successfully but is not further discussed
in this text.
Another ospf testing project can be found at ospf.org .
Description of my test setup
The setup to which this document references consists of one real host called "app109"
and a virtual network with a triangular shape that consists of three virtual UML hosts
called "uml1", "uml2" and "uml3" which are connected to each other by three virtual
Click on the picture below to enlarge it and
here for the Dia  source.
The "networks" are in fact virtual bridges. They are called "brx" and "bra" to "brc".
The "brx" bridge connects all of the three virtual hosts and the real host and is used
for maintenance like FTP updates and DNS service which is provided by the real host.
The three hosts all talk OSPF and are all connected via eth0 to the real system (for
maintenance like FTP updates and DNS service) and with eth1 and eth2 to the other virtual
hosts. The three interface have their counterparts on the real host where they are called
"umlx1", "umlc3" etc. where the "umlc3" means host "uml3" on bridge "brc".
As bridges need different MAC addresses for each interface to work, the virtual
ones get theirs, too. They are created so that "00:00:00:00:03:01" means host "uml1"
on the third bridge ("brc"). This eases debugging.
The relation between the interface names as well as the invented MAC addresses
for the virtual interfaces is defined in the uml start scripts.
Preperation of UML
Creating the simulation is basically done in the following steps:
The host system can be any common GNU/Linux (maybe others, too) system. This
tutorial uses the Debian  distribution as it is freely available
and commonly used at ISPs.
Hardware requirements are depending on the number of hosts that should be
simulated. An AMD Athlon 750MHz with 256MB RAM running a memory hungry X11
Gnome/KDE desktop in the background is sufficient for at least three hosts.
- Configuring Linux virtual Bridging and Interfaces
Setting up the virtual network is probably the hardest part as the software is
not documented very well. A sample configuration file is given in the appendix.
Basically the following steps were made:
Several dummy interfaces have to be created. One for each network. Note that you
have to use the "-o" option to enable more than three interfaces!
The bridge is created with brctl and set to state up with the "ip" command.
The dummy interface, which is the connection between the host system and the
virtual network is set to state "up", too and then attached to the bridge.
All other interfaces on the virtual network are TAP devices which get set up by
"tunctl" and "ip" before adding them to the bridge.
Deconfiguring works the other way around. Be warned that this code is not
very tolerant to senseless commands. Killing a bridge which still has devices
on it or similar stupid things will lock up the host system!
The networking script has of course to be started before any of the UML hosts!
More information can be found on the TUN/TAP  and UML 
mailing lists and in the UML network docs .
- Preparing a chroot environment
Every UML host needs its own root file system to make it as autarchic as
possible. The file system is created inside a normal file using the loopback
filesystem. After that, the debootstrap utillity is used to create a plain
Debian GNU/Linux installation on this new file system.
- We create a file large enough to hold all the file system data.
- dd if=/dev/zero of=uml1.loop bs=1M count=250
- losetup makes the file appear like a normal block device.
- losetup /dev/loop0 uml1.loop
- We create a file system there,
- mkfs.ext3 -j /dev/loop0
- mount it,
- mount -t ext3 /dev/loop0 /mnt/uml1
- and finally the Debian GNU/Linux system is created inside the loop file.
You can add a mirror of your choice as last argument.
- debootstrap woody /mnt/uml1 http://ftp.de.debian.org/debian
- Further configuration to the system can be made by chrooting into it.
They can of course also be made later when starting the chroot with UML.
- chroot /mnt/uml1
- Before using the file system with UML it has to be unmountet.
- umount /mnt/uml1
- Reproducing the file system images
- At this point, after the final configuration when setting up
additional hosts and generally every now and then a backup of the
UML file systems should be made. This is actually quite easy the
systems are plain files which can be replicated the fastest by
using rsync which only copies the differences between two files.
- fsck.ext3 -v -f uml1.loop
rsync --archive --verbose --progress uml1.loop uml1.2003-05-25.loop
- Installing UML
User-Mode-Linux is available as Debian package and should be installed as such.
For basic operation no kernel patch is needed. Only compiling an UML kernel
requires patching. The kernel can be tested by simply typing "linux" which
should output the well known Linux boot up text and stop with "VFS: Unable to
mount root fs" which is where our chroot device comes into play.
- Configuring UML
An UML instance is started at the command line as followes:
con0=fd:0,fd:1 con=xterm \
The first parameter is the unique id of this host. It has no relation to the
hostname which can be freely configurable for every system.
ubd0 is the first user mode block device, the one where the root file system is
searched on, i.e. the just generated loop file.
How much memory a router depends depends mainly on the number of routes it has
to deal with. In this test scenario 48MB would probably be sufficient, too.
The last option opens a new xterm which is the tty0 console of the booted
system where the dmesg text and the login prompt will appear.
As User-mode-linux is a normal application it can be started by any arbitrary
unprivileged user and can only do things on the hosting system that this use
could do, too, of course. Starting it as root is discouraged for security
The network interface are only usable after configuring the virtual network, of
course. The second argument in each line is the interface after which followes
an arbitrary choosen but unique MAC address and the IP address you want to configure
inside the uml system later.
Interestingly configuring the same IP address that was entered on the command
line results in a warning:
The tap IP address and the UML eth IP address must be different
It is debatable if my setup is wrong or not, at least it is the only working one
I am aware of.
FIXME: If you know a better solutions, please mail me!
While starting up the following line should apper and be checked:
Netdevice 0 (00:00:00:00:00:01) : TUN/TAP backend - IP = 172.16.42.1
You can do this later with the "dmesg(8)" command inside the uml host.
After starting up the uml linux should appear in the host's process list like
ch 1642 0.0 0.9 6696 2384 ? S 17:38 0:00 xterm
ch 1643 0.0 0.6 4420 1584 pts/2 S 17:38 0:00 \_ bash
ch 1665 0.0 0.4 4244 1172 pts/2 S 17:38 0:00 \_ /bin/sh ./uml2.sh
ch 1666 1.3 6.1 25696 15904 pts/2 S 17:38 0:04 \_ linux (uml2) [(Unknown)]
ch 1668 0.3 0.3 1748 928 pts/2 T 17:38 0:00 \_ [linux]
ch 1673 0.0 6.1 25696 15904 pts/2 S 17:38 0:00 \_ linux (uml2) [(Unknown)]
ch 1674 0.0 6.1 25696 15904 pts/2 S 17:38 0:00 \_ linux (uml2) [(Unknown)]
ch 1695 0.0 0.9 6692 2364 pts/2 S 17:38 0:00 \_ x-terminal-emulator -T Virtual Console #1 (uml2) -e /usr/lib/uml/port-he
ch 1696 0.0 0.1 1180 296 ? S 17:38 0:00 | \_ /usr/lib/uml/port-helper -uml-socket /tmp/xterm-pipeHqldhC
ch 1697 0.0 2.3 25696 6004 pts/4 S 17:38 0:00 \_ linux (uml2) [(Unknown)]
Notice that this snapshot was made with mem=25M and that the process always uses exactly this amount of memory
on the host.
Inside the uml host the network has also to be configured via /etc/network/interfaces.
The file should contain a stanca like this:
iface eth0 inet static
The interface is then controlable with ifdown eth0 and ifup eth0
or the common ifconfig(8) or better the ip(8) command.
- Installing Zebra
Inside the chroot the Zebra software can be installed just like any other
Debian packet with apt-get -u install zebra/ or downloaded as source
and build in /usr/local/. Important: The current Debian stable release
3.0 "woody" ships the 0.92a version of Zebra as it has a bit more changes as
written on the Zebra web page. An upgrade to 0.93b which is e.g. in unstable is
When installed as Debian package the servers can be configured by editing
/etc/default/zebra and the files in /etc/zebra/. There is one
file for every protocol daemon, zebra.conf for the main zebra daemon who only
gathers the routes from the different protocols and mixes them with the kernel
routing table and finally the daemons where protocol daemons can be
en- or disabled.
After the initial configuration the servers can be restarted by typing
/etc/init.d/zebra restart and then configured like a normal router
on the command line interface (cli). To get access to the cli telnet to the
daemons special port number (/etc/services knows them): telnet localhost
ospfd. The typical copy running-config startup-config works in Zebra and
saves the config to /etc/zebra/ (Debian) or /usr/local/etc/ (self compiled)
The ip adresses and interfaces whould be configured in the zebra.conf as well as
in the standard /etc/network/interfaces file which is also the right location to
setup MTU sizes and similar.
Once the uml host was correctly configured and has booted the network should
be immediately be usable. This can be checked by pinging the real host (app109)
or the neighboring hosts, if they are already up. After pinging their MAC addresses
should be visible in the arp table which can be displayed by "arp -n".
Hint: If nothing works ensure that all firewall rules are turned of and the
bridge initializing script was started before booting the first virtual host.
I suggest to install bind and apt-proxy on the real host and make it accessable
by the virtual UML hosts. This way you can use easy to remember hostname and
install Debian packages without connecting the test networks to the real network.
Sample BIND configuration files can be found in the config section below.
Setting up Zebra
The Zebra daemon itself needs no real configuration. It just collects the routes
from the various routing protocol daemons like ospfd and bgpd and merges them with
the kernel routing table.
As telnetting to the Zebra daemon with "telnet localhost zebra" is usefull
to debug, at least the passwords should be set although. Interface descriptions can
be usefull, too.
A typical Zebra debug command is:
uml1-zebra# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
B - BGP, > - selected route, * - FIB route
O 10.0.1.0/24 [110/10] is directly connected, eth1, 00:07:11
C>* 10.0.1.0/24 is directly connected, eth1
O>* 10.0.2.0/24 [110/20] via 10.0.1.2, eth1, 00:07:01
O 10.0.3.0/24 [110/10] is directly connected, eth2, 00:07:11
C>* 10.0.3.0/24 is directly connected, eth2
C>* 127.0.0.0/8 is directly connected, lo
O 172.16.42.0/24 [110/20] via 10.0.3.3, eth2, 00:07:00
C>* 172.16.42.0/24 is directly connected, eth0
Warning: If one of the Zebra
daemons crashes, all daemons should be killed, "ip route flush proto zebra"
be issued on the command line and Zebra be restarted. Although restarting only
the crashed daemon seems to work, those kernel routing table entries that this daemon
caused will eventually never be deleted! This can be very hard to debug :-(
My Zebra setup for this environment on host uml1 is below. The other configs are
in the config file section further down this text.
! Zebra configuration saved from vty
! 2003/02/02 23:39:57
enable password q
service terminal-length 0
ip address 192.168.1.1/30
smux peer 18.104.22.168.4.1.3322.214.171.124 zebra
The OSPF config can be as complicated as you like :-) For this environment I
configured the following:
- The Router-Id was set to the ip address of the maintenance network to
help identifying the router quickly.
- All connected and static routes are distributed via OSPF so that I can
add dummy addresses to some interface and "see" this address appear in
another hosts routing table.
- The two neighboring networks are activated for OSPF.
The ospf daemon can be configured either by editing the config file and restarting
it or by simply "telnet localhost ospfd".
The final config for host uml1 is below. The others in the config file section of
! Zebra configuration saved from vty
! 2003/02/02 23:12:23
enable password q
service terminal-length 0
ospf router-id 172.16.42.1
network 10.0.1.0/24 area 0
network 10.0.3.0/24 area 3
smux peer 126.96.36.199.4.1.33188.8.131.52 zebra_ospfd
A typical OSPF setup looks like this:
uml1-ospfd# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
172.16.42.2 1 Full/DR 00:00:32 10.0.1.2 eth1:10.0.1.1 0 0 0
172.16.42.3 1 Full/DR 00:00:32 10.0.3.3 eth2:10.0.3.1 0 0 0
And finally some pretty pictures, made with Ethereal , from
lonely HELO packets flying across the ether trying to mate...
Host1 sending initial HELO
Host2 sending initial HELO
Host1 saw Host2
After OSPF sessions are established you should be able to ping every ip address. A typical IP configuration
then looks like this:
uml1:~# ip address
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
inet 172.16.42.1/24 brd 172.16.42.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:00:00:01:01 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.1/24 brd 10.0.1.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:00:00:03:01 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global eth2
uml1:~# ip neighbor
172.16.42.2 dev eth0 lladdr 00:00:00:00:00:02 nud reachable
172.16.42.3 dev eth0 lladdr 00:00:00:00:00:03 nud reachable
172.16.42.254 dev eth0 lladdr 00:00:00:00:00:00 nud stale
10.0.1.2 dev eth1 lladdr 00:00:00:00:01:02 nud stale
10.0.3.3 dev eth2 lladdr 00:00:00:00:03:03 nud stale
uml1:~# ip route
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.1
10.0.2.0/24 via 10.0.1.2 dev eth1 proto zebra metric 20
10.0.3.0/24 dev eth2 proto kernel scope link src 10.0.3.1
172.16.42.0/24 dev eth0 proto kernel scope link src 172.16.42.1
The following benchmarks were made:
- kernel compile and "time tar xjf linux-2.4.20.tar.bz2"
- It's my standard stability test for new systems and usefull to get
some comparable data of a real world application. Memory, CPU and I/O
are heavily used.
- "openssl speed" and "john" cracking the DES encrypted string "626"
- These tests are useful to measure pure CPU performance.
- My own benchmark programm
- Is written in C, precalculates 100MB of random data and then writes and reads
it from disk so that I'm sure that nothing else but fread/fwrite is involved.
It can be found here.
The results can be found here. My
preliminary conclusion is that the CPU is not much affected by the
User-Mode-Linux but I/O performance is quite heavily decreasing (writing more than
reading). This test did not measure the performance in relation to the uml or host "load".
Sample Config Files
The following config files can be used to reproduce the setup described in this text:
- This script openes my personal firewall for virtual bridging.
You probably don't need this.
- This is where the black magic starts. At least it works.
Get it here.
- The start scripts for the individual uml hosts.
- And a sample /etc/network/interfaces file, here from uml1:
- BIND configuration files
- apt-proxy configuration files
- app109:/etc/apt-proxy/apt-proxy.conf (excerpt)
- app109:/etc/inetd.conf (excerpt)
- Zebra configuration files
- for host uml1: zebra.conf and ospfd.conf
- for host uml2: zebra.conf and ospfd.conf
- for host uml3: zebra.conf and ospfd.conf
References and further literature
-  The permanent location of this text is http://www.lathspell.de/linux/uml/
-  The User-Mode-Linux homepage at http://user-mode-linux.sourceforge.net/
-  The Linux Kernel homepage at http://www.kernel.org/
-  OSPF, Anatomy of an Internet Routing Protocol, John T. Moy, 1998, Addison Wesley Longman, ISBN: 0201634724
-  Cisco's "OSPF Design Guide" at http://www.cisco.com/warp/public/104/1.html
-  The OSPFv2 RFC at http://www.ietf.org/rfc/rfc2328.txt
-  The BGP4 RFC at http://www.ietf.org/rfc/rfc1771.txt (and others)
-  The Debian Project at http://www.debian.org/
-  The Zebra routing suite at http://www.zebra.org/
-  Cisco Systems at http://www.cisco.com/
-  Linux Bridging at http://bridge.sourceforge.net/
-  Linux TUN/TAP driver at http://vtun.sourceforge.net/
-  Linux TUN/TAP mailing list at http://vtun.sourceforge.net/mlist/
-  User-Mode-Linux mailing list at http://lists.sourceforge.net/lists/listinfo/user-mode-linux-user/
-  User-Mode-Linux network docs at http://user-mode-linux.sourceforge.net/networking.html
-  Gnome Dia at http://www.gnome.org/gnome-office/dia.shtml
-  Ethereal at http://www.ethereal.com/
-  User-Mode-Linux Community Site at http://usermodelinux.org/
-  Another UML Fan site at http://usermodelinux.co.uk/
-  The unofficial GNU Zebra FAQ at http://pilot.org.ua/zebra/
-  Zebra Patched (zebra-pj) at http://zebra.dishone.st/
-  OSPFD Routing Software Resources at http://ospf.org/
-  UML for hosting services at http://uml.openconsultancy.com/
-  UML for shared hosting at http://www.stearns.org/slartibartfast/uml-coop.current.html
-  Schein-Netz - Simulierte Netze mit User Mode Linux als Basis für Firewall-Tests at http://www.linux-magazin.de/Artikel/ausgabe/2004/01/060_uml_netzwerk/uml_netzwerk.html