User:Andy/Xen

From Strugglers
Jump to: navigation, search

The Xen virtual machine monitor is a set of tools and a patch to the x86 Linux kernel to enable it to host multiple virtual machines with close to native performance. Any x86 operating system can be ported to run as a Xen guest, and ports already exist for Linux, FreeBSD and NetBSD.

Why?

I've started playing with Xen for my own interests and because it has the potential to help with hosting issues of Linux User Groups UK. This page is for notes of my experiences with Xen.

Installation

We've already set Xen up once for lug.org.uk, on a Fedora Core host, but it seems rather buggy. This could be due to Xen, or the kernel used. I've recently installed Xen on my own Debian Sarge machine and this seems to be working much better, with a total so far of 6 unprivileged domains. Here's how I did that.

Xen kernel patch

Downloaded a snapshot of xen-testing from http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0-testing-src.tgz and unpacked it into /opt/xen. This archive contains (amongst other things) a number of trees of kernel source where only files changed from the stock kernels are present. These are the so-called "sparse" trees. I deleted all spares trees that I was not interested in:

$ rm -fr freebsd-5.3-xen-sparse linux-2.4.30-xen-sparse netbsd-2.0-xen-sparse

and then generated a patch against stock 2.6.11 kernel:

[andy@curacao xen-2.0-testing]$ make mkpatches
for i in linux-2.6.11 ; do make $i-xen.patch; done
make[1]: Entering directory `/opt/xen/xen-2.0-testing'
Cannot find linux-2.6.11.tar.bz2 in path .:..
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2 -O./linux-2.6.11.tar.bz2
--14:09:14--  http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
           => `./linux-2.6.11.tar.bz2'
Resolving www.kernel.org... 204.152.191.37, 204.152.191.5
Connecting to www.kernel.org[204.152.191.37]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37,075,679 [application/x-bzip2]

100%[====================================>] 37,075,679   143.88K/s    ETA 00:00

14:13:06 (156.07 KB/s) - `./linux-2.6.11.tar.bz2' saved [37075679/37075679]

rm -rf tmp-pristine-linux-2.6.11 pristine-linux-2.6.11
mkdir -p tmp-pristine-linux-2.6.11
tar -C tmp-pristine-linux-2.6.11 -jxf linux-2.6.11.tar.bz2
mv tmp-pristine-linux-2.6.11/* pristine-linux-2.6.11
touch pristine-linux-2.6.11/.valid-pristine # update timestamp to avoid rebuild
rm -rf ref-linux-2.6.11
cp -al pristine-linux-2.6.11 ref-linux-2.6.11
([ -d patches/linux-2.6.11 ] && \
  for i in patches/linux-2.6.11/*.patch ; \
do ( cd ref-linux-2.6.11 ; patch -p1 <../$i || exit 1 ) ; done) || true
patching file drivers/char/agp/agp.h
patching file drivers/char/agp/ali-agp.c
patching file drivers/char/agp/amd-k7-agp.c
patching file drivers/char/agp/amd64-agp.c
patching file drivers/char/agp/ati-agp.c
patching file drivers/char/agp/backend.c
patching file drivers/char/agp/efficeon-agp.c
patching file drivers/char/agp/generic.c
patching file drivers/char/agp/hp-agp.c
patching file drivers/char/agp/i460-agp.c
patching file drivers/char/agp/intel-agp.c
patching file drivers/char/agp/intel-mch-agp.c
patching file drivers/char/agp/sworks-agp.c
patching file drivers/char/agp/uninorth-agp.c
patching file include/asm-alpha/agp.h
patching file include/asm-i386/agp.h
patching file include/asm-ia64/agp.h
patching file include/asm-ppc/agp.h
patching file include/asm-sparc64/agp.h
patching file include/asm-x86_64/agp.h
patching file drivers/char/agp/frontend.c
patching file drivers/char/drm/drm_vm.c
patching file drivers/char/drm/i810_dma.c
patching file drivers/char/drm/i830_dma.c
patching file drivers/char/hpet.c
patching file drivers/sbus/char/flash.c
patching file include/linux/mm.h
patching file Documentation/SecurityBugs
patching file MAINTAINERS
patching file Makefile
patching file REPORTING-BUGS
patching file arch/ia64/kernel/fsys.S
patching file arch/ia64/kernel/signal.c
patching file arch/ppc/oprofile/op_model_fsl_booke.c
patching file arch/ppc/platforms/4xx/ebony.h
patching file arch/ppc/platforms/4xx/luan.h
patching file arch/ppc/platforms/4xx/ocotea.h
patching file arch/ppc64/kernel/pSeries_iommu.c
patching file arch/sparc/kernel/ptrace.c
patching file arch/sparc64/kernel/ptrace.c
patching file arch/sparc64/kernel/signal32.c
patching file arch/sparc64/kernel/systbls.S
patching file arch/um/include/sysdep-i386/syscalls.h
patching file arch/um/include/sysdep-x86_64/syscalls.h
patching file arch/um/kernel/skas/uaccess.c
patching file arch/um/kernel/sys_call_table.c
patching file arch/x86_64/kernel/apic.c
patching file arch/x86_64/kernel/ptrace.c
patching file arch/x86_64/kernel/smpboot.c
patching file arch/x86_64/mm/fault.c
patching file arch/x86_64/mm/ioremap.c
patching file drivers/block/ioctl.c
patching file drivers/block/pktcdvd.c
patching file drivers/char/drm/drm_ioctl.c
patching file drivers/char/raw.c
patching file drivers/i2c/chips/eeprom.c
patching file drivers/i2c/chips/it87.c
patching file drivers/i2c/chips/via686a.c
patching file drivers/ide/ide-disk.c
patching file drivers/input/serio/i8042-x86ia64io.h
patching file drivers/md/raid6altivec.uc
patching file drivers/media/video/adv7170.c
patching file drivers/media/video/adv7175.c
patching file drivers/media/video/bt819.c
patching file drivers/media/video/bttv-cards.c
patching file drivers/media/video/saa7110.c
patching file drivers/media/video/saa7114.c
patching file drivers/media/video/saa7185.c
patching file drivers/net/3c59x.c
patching file drivers/net/amd8111e.c
patching file drivers/net/ppp_async.c
patching file drivers/net/r8169.c
patching file drivers/net/sis900.c
patching file drivers/net/tun.c
patching file drivers/net/via-rhine.c
patching file drivers/net/wan/hd6457x.c
patching file drivers/pci/hotplug/pciehp_ctrl.c
patching file drivers/usb/serial/visor.c
patching file drivers/video/matrox/matroxfb_accel.c
patching file drivers/video/matrox/matroxfb_base.h
patching file fs/binfmt_elf.c
patching file fs/cramfs/inode.c
patching file fs/eventpoll.c
patching file fs/exec.c
patching file fs/ext2/dir.c
patching file fs/ext3/balloc.c
patching file fs/hfs/mdb.c
patching file fs/hfs/super.c
patching file fs/hfsplus/super.c
patching file fs/isofs/inode.c
patching file fs/isofs/rock.c
patching file fs/jbd/checkpoint.c
patching file fs/jbd/transaction.c
patching file include/asm-x86_64/processor.h
patching file include/linux/err.h
patching file kernel/exit.c
patching file kernel/signal.c
patching file lib/rwsem-spinlock.c
patching file lib/rwsem.c
patching file mm/mmap.c
patching file mm/rmap.c
patching file net/bluetooth/af_bluetooth.c
patching file net/bridge/br_input.c
patching file net/bridge/br_stp_bpdu.c
patching file net/bridge/netfilter/ebtables.c
patching file net/ipv4/fib_hash.c
patching file net/ipv4/netfilter/ip_queue.c
patching file net/ipv4/tcp_input.c
patching file net/ipv4/tcp_timer.c
patching file net/ipv4/xfrm4_output.c
patching file net/ipv6/xfrm6_output.c
patching file net/netrom/nr_in.c
patching file net/rose/rose_route.c
patching file net/sched/sch_netem.c
patching file net/xfrm/xfrm_state.c
patching file security/keys/key.c
patching file sound/core/timer.c
patching file sound/pci/ac97/ac97_codec.c
patching file sound/usb/usbaudio.c
patching file sound/usb/usx2y/usbusx2y.c
patching file drivers/mtd/maps/nettel.c
patching file kernel/rcupdate.c
patching file net/ipv4/udp.c
Hunk #1 succeeded at 737 (offset -1 lines).
Hunk #2 succeeded at 747 (offset -1 lines).
Hunk #3 succeeded at 847 (offset -1 lines).
Hunk #4 succeeded at 1331 (offset -3 lines).
Hunk #5 succeeded at 1342 (offset -3 lines).
touch ref-linux-2.6.11/.valid-ref # update timestamp to avoid rebuild
rm -rf tmp-linux-2.6.11-xen.patch
cp -al ref-linux-2.6.11 tmp-linux-2.6.11-xen.patch
( cd linux-2.6.11-xen-sparse && ./mkbuildtree ../tmp-linux-2.6.11-xen.patch )
diff -Nurp ref-linux-2.6.11 tmp-linux-2.6.11-xen.patch > linux-2.6.11-xen.patch || true
rm -rf tmp-linux-2.6.11-xen.patch
make[1]: Leaving directory `/opt/xen/xen-2.0-testing'
[andy@curacao xen-2.0-testing]$

That left me with a pristine 2.6.11 kernel archive in linux-2.6.11.tar.bz2 and Xen's patch to that in linux-2.6.11-xen.patch.

I then unpacked the kernel to /usr/src and applied the patch:

[andy@curacao xen-2.0-testing]$ cd /usr/src
[andy@curacao src]$ sudo tar jxf /opt/xen/xen-2.0-testing/linux-2.6.11.tar.bz2
[andy@curacao src]$ sudo mv linux-2.6.11 linux-2.6.11-xen
[andy@curacao src]$ cd linux-2.6.11-xen
[andy@curacao linux-2.6.11-xen]$ sudo patch -p1 < /opt/xen/xen-2.0-testing/linux-2.6.11-xen.patch
patching file arch/xen/boot/Makefile
patching file arch/xen/configs/xen0_defconfig
patching file arch/xen/configs/xenU_defconfig
patching file arch/xen/i386/Kconfig
patching file arch/xen/i386/kernel/cpu/common.c
patching file arch/xen/i386/kernel/cpu/Makefile
patching file arch/xen/i386/kernel/cpu/mtrr/main.c
patching file arch/xen/i386/kernel/cpu/mtrr/Makefile
patching file arch/xen/i386/kernel/entry.S
patching file arch/xen/i386/kernel/head.S
patching file arch/xen/i386/kernel/i386_ksyms.c
patching file arch/xen/i386/kernel/ioport.c
patching file arch/xen/i386/kernel/ldt.c
patching file arch/xen/i386/kernel/Makefile
patching file arch/xen/i386/kernel/microcode.c
patching file arch/xen/i386/kernel/pci-dma.c
patching file arch/xen/i386/kernel/process.c
patching file arch/xen/i386/kernel/setup.c
patching file arch/xen/i386/kernel/signal.c
patching file arch/xen/i386/kernel/time.c
patching file arch/xen/i386/kernel/timers/Makefile
patching file arch/xen/i386/kernel/timers/timer_tsc.c
patching file arch/xen/i386/kernel/traps.c
patching file arch/xen/i386/kernel/vsyscall.S
patching file arch/xen/i386/Makefile
patching file arch/xen/i386/mm/fault.c
patching file arch/xen/i386/mm/highmem.c
patching file arch/xen/i386/mm/hypervisor.c
patching file arch/xen/i386/mm/init.c
patching file arch/xen/i386/mm/ioremap.c
patching file arch/xen/i386/mm/Makefile
patching file arch/xen/i386/mm/pageattr.c
patching file arch/xen/i386/mm/pgtable.c
patching file arch/xen/i386/pci/direct.c
patching file arch/xen/i386/pci/irq.c
patching file arch/xen/i386/pci/Makefile
patching file arch/xen/Kconfig
patching file arch/xen/Kconfig.drivers
patching file arch/xen/kernel/ctrl_if.c
patching file arch/xen/kernel/devmem.c
patching file arch/xen/kernel/evtchn.c
patching file arch/xen/kernel/fixup.c
patching file arch/xen/kernel/Makefile
patching file arch/xen/kernel/reboot.c
patching file arch/xen/kernel/skbuff.c
patching file arch/xen/kernel/xen_proc.c
patching file arch/xen/Makefile
patching file arch/xen/x86_64/kernel/early_printk.c
patching file drivers/char/mem.c
patching file drivers/char/tty_io.c
patching file drivers/Makefile
patching file drivers/xen/balloon/balloon.c
patching file drivers/xen/balloon/Makefile
patching file drivers/xen/blkback/blkback.c
patching file drivers/xen/blkback/common.h
patching file drivers/xen/blkback/control.c
patching file drivers/xen/blkback/interface.c
patching file drivers/xen/blkback/Makefile
patching file drivers/xen/blkback/vbd.c
patching file drivers/xen/blkfront/blkfront.c
patching file drivers/xen/blkfront/block.h
patching file drivers/xen/blkfront/Kconfig
patching file drivers/xen/blkfront/Makefile
patching file drivers/xen/blkfront/vbd.c
patching file drivers/xen/console/console.c
patching file drivers/xen/console/Makefile
patching file drivers/xen/evtchn/evtchn.c
patching file drivers/xen/evtchn/Makefile
patching file drivers/xen/Makefile
patching file drivers/xen/netback/common.h
patching file drivers/xen/netback/control.c
patching file drivers/xen/netback/interface.c
patching file drivers/xen/netback/loopback.c
patching file drivers/xen/netback/Makefile
patching file drivers/xen/netback/netback.c
patching file drivers/xen/netfront/Kconfig
patching file drivers/xen/netfront/Makefile
patching file drivers/xen/netfront/netfront.c
patching file drivers/xen/privcmd/Makefile
patching file drivers/xen/privcmd/privcmd.c
patching file include/asm-generic/pgtable.h
patching file include/asm-xen/asm-i386/agp.h
patching file include/asm-xen/asm-i386/bugs.h
patching file include/asm-xen/asm-i386/desc.h
patching file include/asm-xen/asm-i386/dma-mapping.h
patching file include/asm-xen/asm-i386/fixmap.h
patching file include/asm-xen/asm-i386/floppy.h
patching file include/asm-xen/asm-i386/highmem.h
patching file include/asm-xen/asm-i386/io.h
patching file include/asm-xen/asm-i386/mach-xen/irq_vectors.h
patching file include/asm-xen/asm-i386/mach-xen/setup_arch_post.h
patching file include/asm-xen/asm-i386/mach-xen/setup_arch_pre.h
patching file include/asm-xen/asm-i386/mmu_context.h
patching file include/asm-xen/asm-i386/msr.h
patching file include/asm-xen/asm-i386/page.h
patching file include/asm-xen/asm-i386/param.h
patching file include/asm-xen/asm-i386/pci.h
patching file include/asm-xen/asm-i386/pgalloc.h
patching file include/asm-xen/asm-i386/pgtable-2level-defs.h
patching file include/asm-xen/asm-i386/pgtable-2level.h
patching file include/asm-xen/asm-i386/pgtable.h
patching file include/asm-xen/asm-i386/processor.h
patching file include/asm-xen/asm-i386/ptrace.h
patching file include/asm-xen/asm-i386/segment.h
patching file include/asm-xen/asm-i386/setup.h
patching file include/asm-xen/asm-i386/synch_bitops.h
patching file include/asm-xen/asm-i386/system.h
patching file include/asm-xen/asm-i386/tlbflush.h
patching file include/asm-xen/asm-i386/vga.h
patching file include/asm-xen/asm-i386/xor.h
patching file include/asm-xen/balloon.h
patching file include/asm-xen/ctrl_if.h
patching file include/asm-xen/evtchn.h
patching file include/asm-xen/foreign_page.h
patching file include/asm-xen/hypervisor.h
patching file include/asm-xen/linux-public/privcmd.h
patching file include/asm-xen/linux-public/suspend.h
patching file include/asm-xen/multicall.h
patching file include/asm-xen/queues.h
patching file include/asm-xen/xen_proc.h
patching file include/asm-xen/xen-public/arch-x86_32.h
patching file include/asm-xen/xen-public/arch-x86_64.h
patching file include/asm-xen/xen-public/COPYING
patching file include/asm-xen/xen-public/dom0_ops.h
patching file include/asm-xen/xen-public/event_channel.h
patching file include/asm-xen/xen-public/grant_table.h
patching file include/asm-xen/xen-public/io/blkif.h
patching file include/asm-xen/xen-public/io/domain_controller.h
patching file include/asm-xen/xen-public/io/netif.h
patching file include/asm-xen/xen-public/physdev.h
patching file include/asm-xen/xen-public/sched_ctl.h
patching file include/asm-xen/xen-public/trace.h
patching file include/asm-xen/xen-public/xen.h
patching file include/linux/gfp.h
patching file include/linux/highmem.h
patching file include/linux/irq.h
patching file kernel/irq/manage.c
patching file mm/highmem.c
patching file mm/memory.c
patching file mm/page_alloc.c
[andy@curacao linux-2.6.11-xen]$

At this point I had a 2.6.11 kernel with Xen patches in usr/src/linux-2.6.11-xen.

Domain 0 kernel

Then it was time to build a dom0 kernel.

I copied my old kernel config file from /boot as /usr/src/linux-2.6.11-xen/.config

I needed to add the following at the top of my .config otherwise menuconfig wouldn't work properly:

CONFIG_XEN=y
CONFIG_ARCH_XEN=y
CONFIG_NO_IDLE_HZ=y

#
# XEN
#

CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PHYSDEV_ACCESS=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y

# CONFIG_XEN_BLKDEV_FRONTEND is not set
# CONFIG_XEN_NETDEV_FRONTEND is not set

CONFIG_XEN_WRITABLE_PAGETABLES=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_X86=y

# CONFIG_X86_64 is not set
CONFIG_HAVE_ARCH_DEV_ALLOC_SKB=y 

I configured and compiled a new kernel:

[andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \
--arch=xen --revision=1 --append-to-version=curacaoxen0 kernel_image

This brought up a menuconfig as normal, based on my normal kernel's config, but with some extra Xen options. I made sure to have the following settings:

  • XEN
    • Privileged Guest
  • X86 Processor Configuration
    • Kernel hacking
      • Magic SysRq key
  • Device Drivers
    • Multi-device support (RAID and LVM)
      • Device mapper support
      • Snapshot target
    • Networking support
      • Networking options
        • Network packet filtering

I later found it was very important to also disable anything related to AGP.

After this had finished compiling I was left with /usr/src/kernel-xen0-2.6.11curacaoxen0_1_i386.deb

Finally I copied .config to /usr/src/config-2.6.11-xen0 for safe keeping.

Unprivileged domain (domU) kernel

Cleaned out old compile and started another:

[andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \
--arch=xen --revision=1 --append-to-version=curacaotestxenu clean
[andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \
--arch=xen --revision=1 --append-to-version=curacaotestxenu kernel_image

The unprivileged kernel can be very stripped down, with no support for any physical devices. It shouldn't have loadable module support. My domUs would not have loadable module support, although that is possible. Aside from general stripping-down, the following options in menuconfig definitely needed to be changed:

  • XEN
    • (DISABLE) Privileged Guest
    • Network-device frontend driver
    • Block-device frontend driver
  • (DISABLE) Loadable module support
  • File systems
    • Pseudo filesystems
      • (DISABLE) /dev filesystem support

make-kpkg does not like the loadable module support being toggled while it is running, so the first build will fail. This is documented in the man page. I just did the clean and kernel_image steps again and it was fine. I was left with /usr/src/kernel-xen0-2.6.11curacaoxenu_1_i386.deb.

Finally I copied .config to /usr/src/config-2.6.11-xenu for safe keeping.

Xen packages and dependencies

I installed required dependencies and dom0 kernel:

[andy@curacao andy]$ sudo apt-get install iproute libatm1 \
python2.3-twisted python2.3-twisted-bin libcurl3 bridge-utils
[andy@curacao andy]$ sudo dpkg -i /usr/src/kernel-xen0-2.6.11curacaoxen0_1_i386.deb

Disabled thread-local storage as recommended in Xen docs:

[andy@curacao andy]$ sudo mv /lib/tls /lib/tls.disabled
[andy@curacao andy]$ sudo touch /lib/tls
[andy@curacao andy]$ sudo chmod 0 /lib/tls
[andy@curacao andy]$ sudo chattr +i /lib/tls

Installed the xen kernel itself, and xen tools:

[andy@curacao andy]$ cd /opt/xen/xen-2.0-testing
[andy@curacao xen-2.0-testing]$ sudo make xen tools

That installed xen.gz into /boot, and various other things around the system including xend and libxen.

Grub

Added a static stanza to my /boot/grub/menu.lst:

title Debian GNU/Linux, Xen 2.6.11xen0, testing 25/5/2005
kernel /xen.gz dom0_mem=1966080
root (hd0,0)
module /xen-linux-2.6.11curacaoxen0 root=/dev/sda2 ro console=tty0 console=ttyS0

Pray and reboot into the Xen kernel

First time around my praying wasn't enough as I forgot to remove the AGP stuff. This caused a nice kernel oops which locked up the boot process, and I needed to go and power cycle the machine. After building a kernel without AGP support it seemed to work fine.

xend

xend is the Xen control daemon. It should have already been started by sysv init. Commands are issued to it with the xm command:

# xm list
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0     1915    0  r----   1785.2

Filesystems for first unprivileged domain

I already had one LVM volume group with free space (mainvg), so I just created two new logical volumes within this; one for root and one for swap, then initialised them:

# lvcreate -L1024M -n strugglersroot mainvg
# lvcreate -L256M -n strugglersswap mainvg
# mke2fs -j /dev/mainvg/strugglersroot
# mkswap /dev/mainvg/strugglersswap

debootstrap

For my first domain I decided to just install a minimal Debian Sarge. This was easily done with debootstrap:

# apt-get install debootstrap
# mkdir /mnt/xen
# mount /dev/mainvg/strugglersroot /mnt/xen
# debootstrap --arch i386 sarge /mnt/xen http://www.uk.debian.org/debian/

Domain config

Before doing anything I took an archive of the Sarge install to keep for any other domains I might want to make:

# mkdir /data/xen-images
# cd /mnt/xen
# tar jpcf /data/xen-images/debian-sarge-root.tar.bz2
# ls -sh /data/xen-images/debian-sarge-root.tar.bz2
65M /data/xen-images/debian-sarge-root.tar.bz2

Before the files in /mnt/xen can be turned into a bootable Linux install there are a few things that need to be configured. Note that all filenames are relative to /mnt/xen:

etc/fstab

We will be exporting /dev/mainvg/strugglersroot as /dev/sda1 and /dev/mainvg/strugglersswap as /dev/sda2:

/dev/sda1       /       ext3    defaults        0       1
/dev/sda2       swap    swap    defaults        0       0
proc            /proc   proc    defaults        0       0

etc/hostname

I haven't quite decided on a naming scheme yet but since this domain will be for strugglers stuff I decided to call it strugglers, and its hostname will be strugglers too. Its domain name will probably be something like domu.curacao.strugglers.net.

strugglers

etc/hosts

Should contain at least:

127.0.0.1    localhost

etc/network/interfaces

I decided to give the IP 212.13.198.70 to this domain's eth0, which will be bridged to a virtual interface in domain 0:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 212.13.198.70
        netmask 255.255.255.224
        gateway 212.13.198.65
        dns-search strugglers.net
        dns-nameservers 212.13.198.69

etc/apt/sources.list

Eventually I will probably run my own apt mirror, but for now this is just a copy of what's in domain 0's file:

deb http://the.earth.li/debian/ testing main
deb-src http://the.earth.li/debian/ testing main
deb http://ftp.uk.debian.org/debian/ testing main
deb-src http://ftp.uk.debian.org/debian/ testing main
deb http://security.debian.org/ testing/updates main

lib/tls

This needs to be renamed to lib/tls.disabled like in domain 0.

/mnt/xen should now be unmounted.

Xen domain configuration file

I wanted to:

  • create a domain called "strugglers"
  • with 128M RAM
  • 1 network interface
  • the two LVM logical volumes available as /dev/sda1 and /dev/sda2

The following file saved as /etc/xen/strugglers.conf achieves this:

name="strugglers"
memory=128
kernel="/boot/xen-linux-2.6.10curacaoxenu"
nics=1
disk=[ 'phy:mainvg/strugglersroot,sda1,w', 'phy:mainvg/strugglersswap,sda2,w' ]
root="/dev/sda1 ro"

Free up memory

Currently I have domain 0 taking up all spare memory on the machine. Therefore to free up 128M for a new domain, I have to reduce domain 0's total by the same amount:

# xm balloon 0 1792

The above tells Xen to set the total memory allocated to domain 0 to 1792M.

Start it!

The moment of truth:

# xm create /etc/xen/strugglers.conf -c
Using config file "/etc/xen/strugglers.conf".
Started domain strugglers, console on port 9602
************ REMOTE CONSOLE: CTRL-] TO QUIT ********
Linux version 2.6.10curacaoxenu (root@curacao.strugglers.net)
 (gcc version 3.3.5 (Debian 1:3.3.5-8)) #1 Sat Apr 23 21:06:19 UTC 2005
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000008000000 (usable)
0MB HIGHMEM available.
128MB LOWMEM available.
DMI not present.
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  root=/dev/sda1 ro
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 16384 bytes)
Xen reported: 3000.261 MHz processor.
Using tsc for high-res timesource
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 126080k/131072k available (2228k kernel code, 4804k reserved,
574k data, 120k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 04
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
SCSI subsystem initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
loop: loaded (max 8 devices)
elevator: using anticipatory as default io scheduler
nbd: registered device at major 43
Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky
Xen virtual console successfully installed as tty
Event-channel device installed.
xen_blk: Initialising virtual block device driver
xen_net: Initialising virtual ethernet driver.
register_blkdev: cannot get major 8 for sd
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 8Kbytes
TCP: Hash tables configured (established 8192 bind 16384)
ip_conntrack version 2.1 (1024 buckets, 8192 max) - 212 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>.
 http://snowman.net/projects/ipt_recent/
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 10
IPv6 over IPv4 tunneling driver
ip6_tables: (C) 2000-2002 Netfilter core team
NET: Registered protocol family 17
NET: Registered protocol family 15
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 120k freed
INIT: version 2.86 booting
Activating swap.
Adding 262136k swap on /dev/sda2.  Priority:-1 extents:1
Checking root file system...
fsck 1.35 (28-Feb-2004)
/dev/sda1: clean, 13617/131072 files, 48382/262144 blocks
EXT3 FS on sda1, internal journal
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System time was Mon Apr 25 12:02:00 UTC 2005.
Setting the System Clock using the Hardware Clock as reference...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. System local time is now Mon Apr 25 12:02:00 UTC 2005.
Cleaning up ifupdown...done.
Checking all file systems...
fsck 1.35 (28-Feb-2004)
Setting kernel variables ...
... done.
Mounting local filesystems...
Cleaning /tmp /var/run /var/lock.
Running 0dns-down to make sure resolv.conf is ok...done.
Setting up networking.../dev/shm/network/...done.
Setting up IP spoofing protection: rp_filter.
Configuring network interfaces...
Disabled Privacy Extensions on device c0383b20(lo)
done.

Setting the System Clock using the Hardware Clock as reference...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. Local time: Mon Apr 25 12:02:01 UTC 2005

Initializing random number generator...done.
Recovering nvi editor sessions... done.
INIT: Entering runlevel: 2
Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting MTA: exim4.
Starting internet superserver: inetd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 3.1 strugglers tty1

strugglers login:

All of the hwclock complaints are because an unprivileged domain hasn't got access to the system's realtime clock hardware. They can be ignored, as all the time-related settings can be done once in domain 0.

Issuing CTRL-] exits from the xen console back to domain 0.

Logging in to a domain

Normally you'd use ssh, but the minimal Sarge install from above doesn't have that to begin with. You can connect to the xen console of a domain like this:

$ xm list
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0     1787    0  r----   2176.5
strugglers         2      127    1  -b---      1.7    9602
$ xm console 2
************ REMOTE CONSOLE: CTRL-] TO QUIT ********

Debian GNU/Linux 3.1 strugglers tty1

strugglers login: root
Last login: Mon Apr 25 14:29:50 2005 on tty1
Linux strugglers 2.6.10curacaoxenu #1 Sat Apr 23 21:06:19 UTC 2005 i686 GNU/Linux

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
strugglers:~# uname -a
Linux strugglers 2.6.10curacaoxenu #1 Sat Apr 23 21:06:19 UTC 2005 i686 GNU/Linux
# top
top - 15:18:55 up  3:17,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  24 total,   1 running,  23 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 99.3% id,  0.7% wa,  0.0% hi,  0.0% si
Mem:    126388k total,    13764k used,   112624k free,     1308k buffers
Swap:   262136k total,        0k used,   262136k free,     6816k cached

Note that even a normal user in dom0 can connect to the console of an unprivileged domain, although there would normally be a root password!

You can even reboot them from inside:

strugglers:~# reboot

Broadcast message from root (tty1) (Mon Apr 25 15:20:07 2005):

The system is going down for reboot NOW!
INIT: Sending processes the TERM signal
strugglers:~#
INIT: Sending procesStopping periodic command scheduler: cron.
Stopping MTA: exim4.
Stopping internet superserver: inetd.
Saving the System Clock time to the Hardware Clock...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
Hardware Clock updated to Mon Apr 25 15:20:14 UTC 2005.
Stopping deferred execution scheduler: atd.
Stopping kernel log daemon: klogd.
Stopping system log daemon: syslogd.
Sending all processes the TERM signal...done.
Sending all processes the KILL signal...done.
Saving random seed...done.
Unmounting remote and non-toplevel virtual filesystems...done.
Deconfiguring network interfaces...done.
Cleaning up ifupdown...done.
Deactivating swap...done.
Unmounting local filesystems...done.
Rebooting... Restarting system.
Linux version 2.6.10curacaoxenu (root@curacao.strugglers.net)
 (gcc version 3.3.5 (Debian 1:3.3.5-8)) #1 Sat Apr 23 21:06:19 UTC 2005
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000008000000 (usable)
0MB HIGHMEM available.
128MB LOWMEM available.
DMI not present.
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  root=/dev/sda1 ro
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 16384 bytes)
Xen reported: 3000.261 MHz processor.
Using tsc for high-res timesource
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 126080k/131072k available (2228k kernel code, 4804k reserved,
 574k data, 120k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 04
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
SCSI subsystem initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
loop: loaded (max 8 devices)
elevator: using anticipatory as default io scheduler
nbd: registered device at major 43
Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky
Xen virtual console successfully installed as tty
Event-channel device installed.
xen_blk: Initialising virtual block device driver
xen_net: Initialising virtual ethernet driver.
register_blkdev: cannot get major 8 for sd
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 8Kbytes
TCP: Hash tables configured (established 8192 bind 16384)
ip_conntrack version 2.1 (1024 buckets, 8192 max) - 212 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>.
 http://snowman.net/projects/ipt_recent/
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 10
IPv6 over IPv4 tunneling driver
ip6_tables: (C) 2000-2002 Netfilter core team
NET: Registered protocol family 17
NET: Registered protocol family 15
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 120k freed
INIT: version 2.86 booting
Activating swap.
Adding 262136k swap on /dev/sda2.  Priority:-1 extents:1
Checking root file system...
fsck 1.35 (28-Feb-2004)
/dev/sda1: clean, 13637/131072 files, 48407/262144 blocks
EXT3 FS on sda1, internal journal
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System time was Mon Apr 25 15:20:25 UTC 2005.
Setting the System Clock using the Hardware Clock as reference...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. System local time is now Mon Apr 25 15:20:25 UTC 2005.
Cleaning up ifupdown...done.
Checking all file systems...
fsck 1.35 (28-Feb-2004)
Setting kernel variables ...
... done.
Mounting local filesystems...
Cleaning /tmp /var/run /var/lock.
Running 0dns-down to make sure resolv.conf is ok...done.
Setting up networking.../dev/shm/network/...done.
Setting up IP spoofing protection: rp_filter.
Configuring network interfaces...
Disabled Privacy Extensions on device c0383b20(lo)
done.

Setting the System Clock using the Hardware Clock as reference...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. Local time: Mon Apr 25 15:20:26 UTC 2005

Initializing random number generator...done.
Recovering nvi editor sessions... done.
INIT: Entering runlevel: 2
Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting MTA: exim4.
Starting internet superserver: inetd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 3.1 strugglers tty1

strugglers login: