User:Andy/Xen
The Xen virtual machine monitor is a set of tools and a patch to the x86 Linux kernel to enable it to host multiple virtual machines with close to native performance. Any x86 operating system can be ported to run as a Xen guest, and ports already exist for Linux, FreeBSD and NetBSD.
Contents
- 1 Why?
- 2 Installation
- 2.1 Xen kernel patch
- 2.2 Domain 0 kernel
- 2.3 Unprivileged domain (domU) kernel
- 2.4 Xen packages and dependencies
- 2.5 Grub
- 2.6 Pray and reboot into the Xen kernel
- 2.7 xend
- 2.8 Filesystems for first unprivileged domain
- 2.9 debootstrap
- 2.10 Domain config
- 2.11 Xen domain configuration file
- 2.12 Free up memory
- 2.13 Start it!
- 3 Logging in to a domain
Why?
I've started playing with Xen for my own interests and because it has the potential to help with hosting issues of Linux User Groups UK. This page is for notes of my experiences with Xen.
Installation
We've already set Xen up once for lug.org.uk, on a Fedora Core host, but it seems rather buggy. This could be due to Xen, or the kernel used. I've recently installed Xen on my own Debian Sarge machine and this seems to be working much better, with a total so far of 6 unprivileged domains. Here's how I did that.
Xen kernel patch
Downloaded a snapshot of xen-testing from http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0-testing-src.tgz and unpacked it into /opt/xen. This archive contains (amongst other things) a number of trees of kernel source where only files changed from the stock kernels are present. These are the so-called "sparse" trees. I deleted all spares trees that I was not interested in:
$ rm -fr freebsd-5.3-xen-sparse linux-2.4.30-xen-sparse netbsd-2.0-xen-sparse
and then generated a patch against stock 2.6.11 kernel:
[andy@curacao xen-2.0-testing]$ make mkpatches for i in linux-2.6.11 ; do make $i-xen.patch; done make[1]: Entering directory `/opt/xen/xen-2.0-testing' Cannot find linux-2.6.11.tar.bz2 in path .:.. wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2 -O./linux-2.6.11.tar.bz2 --14:09:14-- http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2 => `./linux-2.6.11.tar.bz2' Resolving www.kernel.org... 204.152.191.37, 204.152.191.5 Connecting to www.kernel.org[204.152.191.37]:80... connected. HTTP request sent, awaiting response... 200 OK Length: 37,075,679 [application/x-bzip2] 100%[====================================>] 37,075,679 143.88K/s ETA 00:00 14:13:06 (156.07 KB/s) - `./linux-2.6.11.tar.bz2' saved [37075679/37075679] rm -rf tmp-pristine-linux-2.6.11 pristine-linux-2.6.11 mkdir -p tmp-pristine-linux-2.6.11 tar -C tmp-pristine-linux-2.6.11 -jxf linux-2.6.11.tar.bz2 mv tmp-pristine-linux-2.6.11/* pristine-linux-2.6.11 touch pristine-linux-2.6.11/.valid-pristine # update timestamp to avoid rebuild rm -rf ref-linux-2.6.11 cp -al pristine-linux-2.6.11 ref-linux-2.6.11 ([ -d patches/linux-2.6.11 ] && \ for i in patches/linux-2.6.11/*.patch ; \ do ( cd ref-linux-2.6.11 ; patch -p1 <../$i || exit 1 ) ; done) || true patching file drivers/char/agp/agp.h patching file drivers/char/agp/ali-agp.c patching file drivers/char/agp/amd-k7-agp.c patching file drivers/char/agp/amd64-agp.c patching file drivers/char/agp/ati-agp.c patching file drivers/char/agp/backend.c patching file drivers/char/agp/efficeon-agp.c patching file drivers/char/agp/generic.c patching file drivers/char/agp/hp-agp.c patching file drivers/char/agp/i460-agp.c patching file drivers/char/agp/intel-agp.c patching file drivers/char/agp/intel-mch-agp.c patching file drivers/char/agp/sworks-agp.c patching file drivers/char/agp/uninorth-agp.c patching file include/asm-alpha/agp.h patching file include/asm-i386/agp.h patching file include/asm-ia64/agp.h patching file include/asm-ppc/agp.h patching file include/asm-sparc64/agp.h patching file include/asm-x86_64/agp.h patching file drivers/char/agp/frontend.c patching file drivers/char/drm/drm_vm.c patching file drivers/char/drm/i810_dma.c patching file drivers/char/drm/i830_dma.c patching file drivers/char/hpet.c patching file drivers/sbus/char/flash.c patching file include/linux/mm.h patching file Documentation/SecurityBugs patching file MAINTAINERS patching file Makefile patching file REPORTING-BUGS patching file arch/ia64/kernel/fsys.S patching file arch/ia64/kernel/signal.c patching file arch/ppc/oprofile/op_model_fsl_booke.c patching file arch/ppc/platforms/4xx/ebony.h patching file arch/ppc/platforms/4xx/luan.h patching file arch/ppc/platforms/4xx/ocotea.h patching file arch/ppc64/kernel/pSeries_iommu.c patching file arch/sparc/kernel/ptrace.c patching file arch/sparc64/kernel/ptrace.c patching file arch/sparc64/kernel/signal32.c patching file arch/sparc64/kernel/systbls.S patching file arch/um/include/sysdep-i386/syscalls.h patching file arch/um/include/sysdep-x86_64/syscalls.h patching file arch/um/kernel/skas/uaccess.c patching file arch/um/kernel/sys_call_table.c patching file arch/x86_64/kernel/apic.c patching file arch/x86_64/kernel/ptrace.c patching file arch/x86_64/kernel/smpboot.c patching file arch/x86_64/mm/fault.c patching file arch/x86_64/mm/ioremap.c patching file drivers/block/ioctl.c patching file drivers/block/pktcdvd.c patching file drivers/char/drm/drm_ioctl.c patching file drivers/char/raw.c patching file drivers/i2c/chips/eeprom.c patching file drivers/i2c/chips/it87.c patching file drivers/i2c/chips/via686a.c patching file drivers/ide/ide-disk.c patching file drivers/input/serio/i8042-x86ia64io.h patching file drivers/md/raid6altivec.uc patching file drivers/media/video/adv7170.c patching file drivers/media/video/adv7175.c patching file drivers/media/video/bt819.c patching file drivers/media/video/bttv-cards.c patching file drivers/media/video/saa7110.c patching file drivers/media/video/saa7114.c patching file drivers/media/video/saa7185.c patching file drivers/net/3c59x.c patching file drivers/net/amd8111e.c patching file drivers/net/ppp_async.c patching file drivers/net/r8169.c patching file drivers/net/sis900.c patching file drivers/net/tun.c patching file drivers/net/via-rhine.c patching file drivers/net/wan/hd6457x.c patching file drivers/pci/hotplug/pciehp_ctrl.c patching file drivers/usb/serial/visor.c patching file drivers/video/matrox/matroxfb_accel.c patching file drivers/video/matrox/matroxfb_base.h patching file fs/binfmt_elf.c patching file fs/cramfs/inode.c patching file fs/eventpoll.c patching file fs/exec.c patching file fs/ext2/dir.c patching file fs/ext3/balloc.c patching file fs/hfs/mdb.c patching file fs/hfs/super.c patching file fs/hfsplus/super.c patching file fs/isofs/inode.c patching file fs/isofs/rock.c patching file fs/jbd/checkpoint.c patching file fs/jbd/transaction.c patching file include/asm-x86_64/processor.h patching file include/linux/err.h patching file kernel/exit.c patching file kernel/signal.c patching file lib/rwsem-spinlock.c patching file lib/rwsem.c patching file mm/mmap.c patching file mm/rmap.c patching file net/bluetooth/af_bluetooth.c patching file net/bridge/br_input.c patching file net/bridge/br_stp_bpdu.c patching file net/bridge/netfilter/ebtables.c patching file net/ipv4/fib_hash.c patching file net/ipv4/netfilter/ip_queue.c patching file net/ipv4/tcp_input.c patching file net/ipv4/tcp_timer.c patching file net/ipv4/xfrm4_output.c patching file net/ipv6/xfrm6_output.c patching file net/netrom/nr_in.c patching file net/rose/rose_route.c patching file net/sched/sch_netem.c patching file net/xfrm/xfrm_state.c patching file security/keys/key.c patching file sound/core/timer.c patching file sound/pci/ac97/ac97_codec.c patching file sound/usb/usbaudio.c patching file sound/usb/usx2y/usbusx2y.c patching file drivers/mtd/maps/nettel.c patching file kernel/rcupdate.c patching file net/ipv4/udp.c Hunk #1 succeeded at 737 (offset -1 lines). Hunk #2 succeeded at 747 (offset -1 lines). Hunk #3 succeeded at 847 (offset -1 lines). Hunk #4 succeeded at 1331 (offset -3 lines). Hunk #5 succeeded at 1342 (offset -3 lines). touch ref-linux-2.6.11/.valid-ref # update timestamp to avoid rebuild rm -rf tmp-linux-2.6.11-xen.patch cp -al ref-linux-2.6.11 tmp-linux-2.6.11-xen.patch ( cd linux-2.6.11-xen-sparse && ./mkbuildtree ../tmp-linux-2.6.11-xen.patch ) diff -Nurp ref-linux-2.6.11 tmp-linux-2.6.11-xen.patch > linux-2.6.11-xen.patch || true rm -rf tmp-linux-2.6.11-xen.patch make[1]: Leaving directory `/opt/xen/xen-2.0-testing' [andy@curacao xen-2.0-testing]$
That left me with a pristine 2.6.11 kernel archive in linux-2.6.11.tar.bz2 and Xen's patch to that in linux-2.6.11-xen.patch.
I then unpacked the kernel to /usr/src and applied the patch:
[andy@curacao xen-2.0-testing]$ cd /usr/src [andy@curacao src]$ sudo tar jxf /opt/xen/xen-2.0-testing/linux-2.6.11.tar.bz2 [andy@curacao src]$ sudo mv linux-2.6.11 linux-2.6.11-xen [andy@curacao src]$ cd linux-2.6.11-xen [andy@curacao linux-2.6.11-xen]$ sudo patch -p1 < /opt/xen/xen-2.0-testing/linux-2.6.11-xen.patch patching file arch/xen/boot/Makefile patching file arch/xen/configs/xen0_defconfig patching file arch/xen/configs/xenU_defconfig patching file arch/xen/i386/Kconfig patching file arch/xen/i386/kernel/cpu/common.c patching file arch/xen/i386/kernel/cpu/Makefile patching file arch/xen/i386/kernel/cpu/mtrr/main.c patching file arch/xen/i386/kernel/cpu/mtrr/Makefile patching file arch/xen/i386/kernel/entry.S patching file arch/xen/i386/kernel/head.S patching file arch/xen/i386/kernel/i386_ksyms.c patching file arch/xen/i386/kernel/ioport.c patching file arch/xen/i386/kernel/ldt.c patching file arch/xen/i386/kernel/Makefile patching file arch/xen/i386/kernel/microcode.c patching file arch/xen/i386/kernel/pci-dma.c patching file arch/xen/i386/kernel/process.c patching file arch/xen/i386/kernel/setup.c patching file arch/xen/i386/kernel/signal.c patching file arch/xen/i386/kernel/time.c patching file arch/xen/i386/kernel/timers/Makefile patching file arch/xen/i386/kernel/timers/timer_tsc.c patching file arch/xen/i386/kernel/traps.c patching file arch/xen/i386/kernel/vsyscall.S patching file arch/xen/i386/Makefile patching file arch/xen/i386/mm/fault.c patching file arch/xen/i386/mm/highmem.c patching file arch/xen/i386/mm/hypervisor.c patching file arch/xen/i386/mm/init.c patching file arch/xen/i386/mm/ioremap.c patching file arch/xen/i386/mm/Makefile patching file arch/xen/i386/mm/pageattr.c patching file arch/xen/i386/mm/pgtable.c patching file arch/xen/i386/pci/direct.c patching file arch/xen/i386/pci/irq.c patching file arch/xen/i386/pci/Makefile patching file arch/xen/Kconfig patching file arch/xen/Kconfig.drivers patching file arch/xen/kernel/ctrl_if.c patching file arch/xen/kernel/devmem.c patching file arch/xen/kernel/evtchn.c patching file arch/xen/kernel/fixup.c patching file arch/xen/kernel/Makefile patching file arch/xen/kernel/reboot.c patching file arch/xen/kernel/skbuff.c patching file arch/xen/kernel/xen_proc.c patching file arch/xen/Makefile patching file arch/xen/x86_64/kernel/early_printk.c patching file drivers/char/mem.c patching file drivers/char/tty_io.c patching file drivers/Makefile patching file drivers/xen/balloon/balloon.c patching file drivers/xen/balloon/Makefile patching file drivers/xen/blkback/blkback.c patching file drivers/xen/blkback/common.h patching file drivers/xen/blkback/control.c patching file drivers/xen/blkback/interface.c patching file drivers/xen/blkback/Makefile patching file drivers/xen/blkback/vbd.c patching file drivers/xen/blkfront/blkfront.c patching file drivers/xen/blkfront/block.h patching file drivers/xen/blkfront/Kconfig patching file drivers/xen/blkfront/Makefile patching file drivers/xen/blkfront/vbd.c patching file drivers/xen/console/console.c patching file drivers/xen/console/Makefile patching file drivers/xen/evtchn/evtchn.c patching file drivers/xen/evtchn/Makefile patching file drivers/xen/Makefile patching file drivers/xen/netback/common.h patching file drivers/xen/netback/control.c patching file drivers/xen/netback/interface.c patching file drivers/xen/netback/loopback.c patching file drivers/xen/netback/Makefile patching file drivers/xen/netback/netback.c patching file drivers/xen/netfront/Kconfig patching file drivers/xen/netfront/Makefile patching file drivers/xen/netfront/netfront.c patching file drivers/xen/privcmd/Makefile patching file drivers/xen/privcmd/privcmd.c patching file include/asm-generic/pgtable.h patching file include/asm-xen/asm-i386/agp.h patching file include/asm-xen/asm-i386/bugs.h patching file include/asm-xen/asm-i386/desc.h patching file include/asm-xen/asm-i386/dma-mapping.h patching file include/asm-xen/asm-i386/fixmap.h patching file include/asm-xen/asm-i386/floppy.h patching file include/asm-xen/asm-i386/highmem.h patching file include/asm-xen/asm-i386/io.h patching file include/asm-xen/asm-i386/mach-xen/irq_vectors.h patching file include/asm-xen/asm-i386/mach-xen/setup_arch_post.h patching file include/asm-xen/asm-i386/mach-xen/setup_arch_pre.h patching file include/asm-xen/asm-i386/mmu_context.h patching file include/asm-xen/asm-i386/msr.h patching file include/asm-xen/asm-i386/page.h patching file include/asm-xen/asm-i386/param.h patching file include/asm-xen/asm-i386/pci.h patching file include/asm-xen/asm-i386/pgalloc.h patching file include/asm-xen/asm-i386/pgtable-2level-defs.h patching file include/asm-xen/asm-i386/pgtable-2level.h patching file include/asm-xen/asm-i386/pgtable.h patching file include/asm-xen/asm-i386/processor.h patching file include/asm-xen/asm-i386/ptrace.h patching file include/asm-xen/asm-i386/segment.h patching file include/asm-xen/asm-i386/setup.h patching file include/asm-xen/asm-i386/synch_bitops.h patching file include/asm-xen/asm-i386/system.h patching file include/asm-xen/asm-i386/tlbflush.h patching file include/asm-xen/asm-i386/vga.h patching file include/asm-xen/asm-i386/xor.h patching file include/asm-xen/balloon.h patching file include/asm-xen/ctrl_if.h patching file include/asm-xen/evtchn.h patching file include/asm-xen/foreign_page.h patching file include/asm-xen/hypervisor.h patching file include/asm-xen/linux-public/privcmd.h patching file include/asm-xen/linux-public/suspend.h patching file include/asm-xen/multicall.h patching file include/asm-xen/queues.h patching file include/asm-xen/xen_proc.h patching file include/asm-xen/xen-public/arch-x86_32.h patching file include/asm-xen/xen-public/arch-x86_64.h patching file include/asm-xen/xen-public/COPYING patching file include/asm-xen/xen-public/dom0_ops.h patching file include/asm-xen/xen-public/event_channel.h patching file include/asm-xen/xen-public/grant_table.h patching file include/asm-xen/xen-public/io/blkif.h patching file include/asm-xen/xen-public/io/domain_controller.h patching file include/asm-xen/xen-public/io/netif.h patching file include/asm-xen/xen-public/physdev.h patching file include/asm-xen/xen-public/sched_ctl.h patching file include/asm-xen/xen-public/trace.h patching file include/asm-xen/xen-public/xen.h patching file include/linux/gfp.h patching file include/linux/highmem.h patching file include/linux/irq.h patching file kernel/irq/manage.c patching file mm/highmem.c patching file mm/memory.c patching file mm/page_alloc.c [andy@curacao linux-2.6.11-xen]$
At this point I had a 2.6.11 kernel with Xen patches in usr/src/linux-2.6.11-xen.
Domain 0 kernel
Then it was time to build a dom0 kernel.
I copied my old kernel config file from /boot as /usr/src/linux-2.6.11-xen/.config
I needed to add the following at the top of my .config otherwise menuconfig wouldn't work properly:
CONFIG_XEN=y CONFIG_ARCH_XEN=y CONFIG_NO_IDLE_HZ=y # # XEN # CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PHYSDEV_ACCESS=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_BACKEND=y # CONFIG_XEN_BLKDEV_FRONTEND is not set # CONFIG_XEN_NETDEV_FRONTEND is not set CONFIG_XEN_WRITABLE_PAGETABLES=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_X86=y # CONFIG_X86_64 is not set CONFIG_HAVE_ARCH_DEV_ALLOC_SKB=y
I configured and compiled a new kernel:
[andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \ --arch=xen --revision=1 --append-to-version=curacaoxen0 kernel_image
This brought up a menuconfig as normal, based on my normal kernel's config, but with some extra Xen options. I made sure to have the following settings:
- XEN
- Privileged Guest
- X86 Processor Configuration
- Kernel hacking
- Magic SysRq key
- Kernel hacking
- Device Drivers
- Multi-device support (RAID and LVM)
- Device mapper support
- Snapshot target
- Networking support
- Networking options
- Network packet filtering
- Networking options
- Multi-device support (RAID and LVM)
I later found it was very important to also disable anything related to AGP.
After this had finished compiling I was left with /usr/src/kernel-xen0-2.6.11curacaoxen0_1_i386.deb
Finally I copied .config to /usr/src/config-2.6.11-xen0 for safe keeping.
Unprivileged domain (domU) kernel
Cleaned out old compile and started another:
[andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \ --arch=xen --revision=1 --append-to-version=curacaotestxenu clean [andy@curacao linux-2.6.11-xen]$ sudo make-kpkg --config=menuconfig \ --arch=xen --revision=1 --append-to-version=curacaotestxenu kernel_image
The unprivileged kernel can be very stripped down, with no support for any physical devices. It shouldn't have loadable module support. My domUs would not have loadable module support, although that is possible. Aside from general stripping-down, the following options in menuconfig definitely needed to be changed:
- XEN
- (DISABLE) Privileged Guest
- Network-device frontend driver
- Block-device frontend driver
- (DISABLE) Loadable module support
- File systems
- Pseudo filesystems
- (DISABLE) /dev filesystem support
- Pseudo filesystems
make-kpkg does not like the loadable module support being toggled while it is running, so the first build will fail. This is documented in the man page. I just did the clean and kernel_image steps again and it was fine. I was left with /usr/src/kernel-xen0-2.6.11curacaoxenu_1_i386.deb.
Finally I copied .config to /usr/src/config-2.6.11-xenu for safe keeping.
Xen packages and dependencies
I installed required dependencies and dom0 kernel:
[andy@curacao andy]$ sudo apt-get install iproute libatm1 \ python2.3-twisted python2.3-twisted-bin libcurl3 bridge-utils [andy@curacao andy]$ sudo dpkg -i /usr/src/kernel-xen0-2.6.11curacaoxen0_1_i386.deb
Disabled thread-local storage as recommended in Xen docs:
[andy@curacao andy]$ sudo mv /lib/tls /lib/tls.disabled [andy@curacao andy]$ sudo touch /lib/tls [andy@curacao andy]$ sudo chmod 0 /lib/tls [andy@curacao andy]$ sudo chattr +i /lib/tls
Installed the xen kernel itself, and xen tools:
[andy@curacao andy]$ cd /opt/xen/xen-2.0-testing [andy@curacao xen-2.0-testing]$ sudo make xen tools
That installed xen.gz into /boot, and various other things around the system including xend and libxen.
Grub
Added a static stanza to my /boot/grub/menu.lst:
title Debian GNU/Linux, Xen 2.6.11xen0, testing 25/5/2005 kernel /xen.gz dom0_mem=1966080 root (hd0,0) module /xen-linux-2.6.11curacaoxen0 root=/dev/sda2 ro console=tty0 console=ttyS0
Pray and reboot into the Xen kernel
First time around my praying wasn't enough as I forgot to remove the AGP stuff. This caused a nice kernel oops which locked up the boot process, and I needed to go and power cycle the machine. After building a kernel without AGP support it seemed to work fine.
xend
xend is the Xen control daemon. It should have already been started by sysv init. Commands are issued to it with the xm command:
# xm list Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 1915 0 r---- 1785.2
Filesystems for first unprivileged domain
I already had one LVM volume group with free space (mainvg), so I just created two new logical volumes within this; one for root and one for swap, then initialised them:
# lvcreate -L1024M -n strugglersroot mainvg # lvcreate -L256M -n strugglersswap mainvg # mke2fs -j /dev/mainvg/strugglersroot # mkswap /dev/mainvg/strugglersswap
debootstrap
For my first domain I decided to just install a minimal Debian Sarge. This was easily done with debootstrap:
# apt-get install debootstrap # mkdir /mnt/xen # mount /dev/mainvg/strugglersroot /mnt/xen # debootstrap --arch i386 sarge /mnt/xen http://www.uk.debian.org/debian/
Domain config
Before doing anything I took an archive of the Sarge install to keep for any other domains I might want to make:
# mkdir /data/xen-images # cd /mnt/xen # tar jpcf /data/xen-images/debian-sarge-root.tar.bz2 # ls -sh /data/xen-images/debian-sarge-root.tar.bz2 65M /data/xen-images/debian-sarge-root.tar.bz2
Before the files in /mnt/xen can be turned into a bootable Linux install there are a few things that need to be configured. Note that all filenames are relative to /mnt/xen:
etc/fstab
We will be exporting /dev/mainvg/strugglersroot as /dev/sda1 and /dev/mainvg/strugglersswap as /dev/sda2:
/dev/sda1 / ext3 defaults 0 1 /dev/sda2 swap swap defaults 0 0 proc /proc proc defaults 0 0
etc/hostname
I haven't quite decided on a naming scheme yet but since this domain will be for strugglers stuff I decided to call it strugglers, and its hostname will be strugglers too. Its domain name will probably be something like domu.curacao.strugglers.net.
strugglers
etc/hosts
Should contain at least:
127.0.0.1 localhost
etc/network/interfaces
I decided to give the IP 212.13.198.70 to this domain's eth0, which will be bridged to a virtual interface in domain 0:
auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 212.13.198.70 netmask 255.255.255.224 gateway 212.13.198.65 dns-search strugglers.net dns-nameservers 212.13.198.69
etc/apt/sources.list
Eventually I will probably run my own apt mirror, but for now this is just a copy of what's in domain 0's file:
deb http://the.earth.li/debian/ testing main deb-src http://the.earth.li/debian/ testing main deb http://ftp.uk.debian.org/debian/ testing main deb-src http://ftp.uk.debian.org/debian/ testing main deb http://security.debian.org/ testing/updates main
lib/tls
This needs to be renamed to lib/tls.disabled like in domain 0.
/mnt/xen should now be unmounted.
Xen domain configuration file
I wanted to:
- create a domain called "strugglers"
- with 128M RAM
- 1 network interface
- the two LVM logical volumes available as /dev/sda1 and /dev/sda2
The following file saved as /etc/xen/strugglers.conf achieves this:
name="strugglers" memory=128 kernel="/boot/xen-linux-2.6.10curacaoxenu" nics=1 disk=[ 'phy:mainvg/strugglersroot,sda1,w', 'phy:mainvg/strugglersswap,sda2,w' ] root="/dev/sda1 ro"
Free up memory
Currently I have domain 0 taking up all spare memory on the machine. Therefore to free up 128M for a new domain, I have to reduce domain 0's total by the same amount:
# xm balloon 0 1792
The above tells Xen to set the total memory allocated to domain 0 to 1792M.
Start it!
The moment of truth:
# xm create /etc/xen/strugglers.conf -c Using config file "/etc/xen/strugglers.conf". Started domain strugglers, console on port 9602 ************ REMOTE CONSOLE: CTRL-] TO QUIT ******** Linux version 2.6.10curacaoxenu (root@curacao.strugglers.net) (gcc version 3.3.5 (Debian 1:3.3.5-8)) #1 Sat Apr 23 21:06:19 UTC 2005 BIOS-provided physical RAM map: Xen: 0000000000000000 - 0000000008000000 (usable) 0MB HIGHMEM available. 128MB LOWMEM available. DMI not present. IRQ lockup detection disabled Built 1 zonelists Kernel command line: root=/dev/sda1 ro Initializing CPU#0 PID hash table entries: 1024 (order: 10, 16384 bytes) Xen reported: 3000.261 MHz processor. Using tsc for high-res timesource Dentry cache hash table entries: 32768 (order: 5, 131072 bytes) Inode-cache hash table entries: 16384 (order: 4, 65536 bytes) Memory: 126080k/131072k available (2228k kernel code, 4804k reserved, 574k data, 120k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Mount-cache hash table entries: 512 (order: 0, 4096 bytes) CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 1024K CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 04 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... disabled NET: Registered protocol family 16 xen_mem: Initialising balloon driver. SCSI subsystem initialized Total HugeTLB memory allocated, 0 VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Initializing Cryptographic API io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered loop: loaded (max 8 devices) elevator: using anticipatory as default io scheduler nbd: registered device at major 43 Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky Xen virtual console successfully installed as tty Event-channel device installed. xen_blk: Initialising virtual block device driver xen_net: Initialising virtual ethernet driver. register_blkdev: cannot get major 8 for sd NET: Registered protocol family 2 IP: routing cache hash table of 1024 buckets, 8Kbytes TCP: Hash tables configured (established 8192 bind 16384) ip_conntrack version 2.1 (1024 buckets, 8192 max) - 212 bytes per conntrack ip_tables: (C) 2000-2002 Netfilter core team ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>. http://snowman.net/projects/ipt_recent/ Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 10 IPv6 over IPv4 tunneling driver ip6_tables: (C) 2000-2002 Netfilter core team NET: Registered protocol family 17 NET: Registered protocol family 15 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Freeing unused kernel memory: 120k freed INIT: version 2.86 booting Activating swap. Adding 262136k swap on /dev/sda2. Priority:-1 extents:1 Checking root file system... fsck 1.35 (28-Feb-2004) /dev/sda1: clean, 13617/131072 files, 48382/262144 blocks EXT3 FS on sda1, internal journal hwclock is unable to get I/O port access: the iopl(3) call failed. System time was Mon Apr 25 12:02:00 UTC 2005. Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. System local time is now Mon Apr 25 12:02:00 UTC 2005. Cleaning up ifupdown...done. Checking all file systems... fsck 1.35 (28-Feb-2004) Setting kernel variables ... ... done. Mounting local filesystems... Cleaning /tmp /var/run /var/lock. Running 0dns-down to make sure resolv.conf is ok...done. Setting up networking.../dev/shm/network/...done. Setting up IP spoofing protection: rp_filter. Configuring network interfaces... Disabled Privacy Extensions on device c0383b20(lo) done. Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. Local time: Mon Apr 25 12:02:01 UTC 2005 Initializing random number generator...done. Recovering nvi editor sessions... done. INIT: Entering runlevel: 2 Starting system log daemon: syslogd. Starting kernel log daemon: klogd. Starting MTA: exim4. Starting internet superserver: inetd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron. Debian GNU/Linux 3.1 strugglers tty1 strugglers login:
All of the hwclock complaints are because an unprivileged domain hasn't got access to the system's realtime clock hardware. They can be ignored, as all the time-related settings can be done once in domain 0.
Issuing CTRL-] exits from the xen console back to domain 0.
Logging in to a domain
Normally you'd use ssh, but the minimal Sarge install from above doesn't have that to begin with. You can connect to the xen console of a domain like this:
$ xm list Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 1787 0 r---- 2176.5 strugglers 2 127 1 -b--- 1.7 9602 $ xm console 2 ************ REMOTE CONSOLE: CTRL-] TO QUIT ******** Debian GNU/Linux 3.1 strugglers tty1 strugglers login: root Last login: Mon Apr 25 14:29:50 2005 on tty1 Linux strugglers 2.6.10curacaoxenu #1 Sat Apr 23 21:06:19 UTC 2005 i686 GNU/Linux The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. strugglers:~# uname -a Linux strugglers 2.6.10curacaoxenu #1 Sat Apr 23 21:06:19 UTC 2005 i686 GNU/Linux # top top - 15:18:55 up 3:17, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 24 total, 1 running, 23 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 99.3% id, 0.7% wa, 0.0% hi, 0.0% si Mem: 126388k total, 13764k used, 112624k free, 1308k buffers Swap: 262136k total, 0k used, 262136k free, 6816k cached
Note that even a normal user in dom0 can connect to the console of an unprivileged domain, although there would normally be a root password!
You can even reboot them from inside:
strugglers:~# reboot Broadcast message from root (tty1) (Mon Apr 25 15:20:07 2005): The system is going down for reboot NOW! INIT: Sending processes the TERM signal strugglers:~# INIT: Sending procesStopping periodic command scheduler: cron. Stopping MTA: exim4. Stopping internet superserver: inetd. Saving the System Clock time to the Hardware Clock... hwclock is unable to get I/O port access: the iopl(3) call failed. Hardware Clock updated to Mon Apr 25 15:20:14 UTC 2005. Stopping deferred execution scheduler: atd. Stopping kernel log daemon: klogd. Stopping system log daemon: syslogd. Sending all processes the TERM signal...done. Sending all processes the KILL signal...done. Saving random seed...done. Unmounting remote and non-toplevel virtual filesystems...done. Deconfiguring network interfaces...done. Cleaning up ifupdown...done. Deactivating swap...done. Unmounting local filesystems...done. Rebooting... Restarting system. Linux version 2.6.10curacaoxenu (root@curacao.strugglers.net) (gcc version 3.3.5 (Debian 1:3.3.5-8)) #1 Sat Apr 23 21:06:19 UTC 2005 BIOS-provided physical RAM map: Xen: 0000000000000000 - 0000000008000000 (usable) 0MB HIGHMEM available. 128MB LOWMEM available. DMI not present. IRQ lockup detection disabled Built 1 zonelists Kernel command line: root=/dev/sda1 ro Initializing CPU#0 PID hash table entries: 1024 (order: 10, 16384 bytes) Xen reported: 3000.261 MHz processor. Using tsc for high-res timesource Dentry cache hash table entries: 32768 (order: 5, 131072 bytes) Inode-cache hash table entries: 16384 (order: 4, 65536 bytes) Memory: 126080k/131072k available (2228k kernel code, 4804k reserved, 574k data, 120k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Mount-cache hash table entries: 512 (order: 0, 4096 bytes) CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 1024K CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz stepping 04 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... disabled NET: Registered protocol family 16 xen_mem: Initialising balloon driver. SCSI subsystem initialized Total HugeTLB memory allocated, 0 VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Initializing Cryptographic API io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered loop: loaded (max 8 devices) elevator: using anticipatory as default io scheduler nbd: registered device at major 43 Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky Xen virtual console successfully installed as tty Event-channel device installed. xen_blk: Initialising virtual block device driver xen_net: Initialising virtual ethernet driver. register_blkdev: cannot get major 8 for sd NET: Registered protocol family 2 IP: routing cache hash table of 1024 buckets, 8Kbytes TCP: Hash tables configured (established 8192 bind 16384) ip_conntrack version 2.1 (1024 buckets, 8192 max) - 212 bytes per conntrack ip_tables: (C) 2000-2002 Netfilter core team ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>. http://snowman.net/projects/ipt_recent/ Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 10 IPv6 over IPv4 tunneling driver ip6_tables: (C) 2000-2002 Netfilter core team NET: Registered protocol family 17 NET: Registered protocol family 15 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Freeing unused kernel memory: 120k freed INIT: version 2.86 booting Activating swap. Adding 262136k swap on /dev/sda2. Priority:-1 extents:1 Checking root file system... fsck 1.35 (28-Feb-2004) /dev/sda1: clean, 13637/131072 files, 48407/262144 blocks EXT3 FS on sda1, internal journal hwclock is unable to get I/O port access: the iopl(3) call failed. System time was Mon Apr 25 15:20:25 UTC 2005. Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. System local time is now Mon Apr 25 15:20:25 UTC 2005. Cleaning up ifupdown...done. Checking all file systems... fsck 1.35 (28-Feb-2004) Setting kernel variables ... ... done. Mounting local filesystems... Cleaning /tmp /var/run /var/lock. Running 0dns-down to make sure resolv.conf is ok...done. Setting up networking.../dev/shm/network/...done. Setting up IP spoofing protection: rp_filter. Configuring network interfaces... Disabled Privacy Extensions on device c0383b20(lo) done. Setting the System Clock using the Hardware Clock as reference... hwclock is unable to get I/O port access: the iopl(3) call failed. System Clock set. Local time: Mon Apr 25 15:20:26 UTC 2005 Initializing random number generator...done. Recovering nvi editor sessions... done. INIT: Entering runlevel: 2 Starting system log daemon: syslogd. Starting kernel log daemon: klogd. Starting MTA: exim4. Starting internet superserver: inetd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron. Debian GNU/Linux 3.1 strugglers tty1 strugglers login: