CentOS/RHEL and Xen ^
As of the release of CentOS 8 / RHEL8, Red Hat disabled kernel support for running as a Xen PV or PVH guest, even though such support is enabled by default in the upstream Linux kernel.
As a result—unlike with all previous versions of CentOS/RHEL—you cannot boot the installer in Xen PV or PVH mode. You can still boot it in Xen HVM mode, or under KVM, but that is not very helpful if you don’t want to run HVM or KVM.
At BitFolk ever since the release of CentOS 8 we’ve had to tell customers to use the Rescue VM (a kind of live system) to unpack CentOS into a chroot.
Fortunately there is now a better way.
This method was worked out by Jon Fautley. Jon emailed me instructions and I was able to replicate them. Several people have since asked me how it was done and Jon was happy for me to write it up, but this was all worked out by Jon, not me.
The basic idea here is to:
- take the installer initrd.img
- unpack it
- shove the modules from a Debian kernel into it
- repack it
- use a Debian kernel and this new frankeninitrd as the installer kernel and initrd
- switch the installed OS to kernel-ml package from ELRepo so it has a working kernel when it boots
Detailed process ^
I’ll go into enough detail that you should be able to exactly replicate what I did to end up with something that works. This is quite a lot but it only needs to be done each time the real installer initrd.img changes, which isn’t that often. The resulting kernel and initrd.img can be used to install many guests.
Throughout the rest of this article I’ll refer to CentOS, but Jon initially made this work for RHEL 8. I’ve replicated it for CentOS 8 and will soon do so for RHEL 8 as well.
Extract the CentOS initrd.img ^
You will find this in the install ISO or on mirrors as images/pxeboot/initrd.img.
$ mkdir /var/tmp/frankeninitrd/initrd $ cd /var/tmp/frankeninitrd/initrd $ xz -dc /path/to/initrd.img > ../initrd.cpio $ # root needed because this will do some mknod/mkdev. $ sudo cpio -idv < ../initrd.cpio
Copy modules from a working Xen guest ^
I’m going to use the Xen guest that I’m doing this on, which at the time of writing is a Debian buster system running kernel 4.19.0-13. Even a system that is not currently running as a Xen guest will probably work, as they usually have modules available for everything.
At the time of writing the kernel version in the installer is 4.18.0-240.
If you’ve got different, adjust filenames accordingly.
$ sudo cp -r /lib/modules/4.19.0-13-amd64 lib/modules/ $ # You're not going to use the original modules $ # so may as well delete them to save space. $ sudo rm -vr lib/modules/4.18*
Add dracut hook to copy fs modules ^
$ cat > usr/lib/dracut/hooks/pre-pivot/99-move-modules.sh <<__EOF__ #!/bin/sh mkdir -p /sysroot/lib/modules/$(uname -r)/kernel/fs rm -r /sysroot/lib/modules/4.18* cp -r /lib/modules/$(uname -r)/kernel/fs/* /sysroot/lib/modules/$(uname -r)/kernel/fs cp /lib/modules/$(uname -r)/modules.builtin /sysroot/lib/modules/$(uname -r)/ depmod -a -b /sysroot exit 0 __EOF__ $ chmod +x usr/lib/dracut/hooks/pre-pivot/99-move-modules.sh
Repack initrd ^
This will take a really long time because
xz -9 is sloooooow.
$ sudo find . 2>/dev/null | \ sudo cpio -o -H newc -R root:root | \ xz -9 --format=lzma > ../centos8-initrd.img
Use the Debian kernel ^
Put the matching kernel next to your initrd.
$ cp /boot/vmlinuz-4.19.0-13-amd64 ../centos8-vmlinuz $ ls -lah ../centos* -rw-r--r-- 1 andy andy 81M Feb 1 04:43 ../centos8-initrd.img -rw-r--r-- 1 andy andy 5.1M Feb 1 04:04 ../centos8-vmlinuz
Boot this kernel/initrd as a Xen guest ^
Copy the kernel and initrd to somewhere on your dom0 and create a guest config file that looks a bit like this:
name = "centostest" # CentOS 8 installer requires at least 2.5G RAM. # OS will run with a lot less though. memory = 2560 vif = [ "mac=00:16:5e:00:02:39, ip=192.168.82.225, vifname=v-centostest" ] type = "pvh" kernel = "/var/tmp/frankeninitrd/centos8-vmlinuz" ramdisk = "/var/tmp/frankeninitrd/centos8-initrd.img" extra = "console=hvc0 ip=192.168.82.225::192.168.82.1:255.255.255.0:centostest:eth0:none nameserver=220.127.116.11 inst.stage2=http://www.mirrorservice.org/sites/mirror.centos.org/8/BaseOS/x86_64/os/ inst.ks=http://example.com/yourkickstart.ks" disk = [ "phy:/dev/vg/centostest_xvda,xvda,w", "phy:/dev/vg/centostest_xvdb,xvdb,w" ]
Assumptions in the above:
- vif and disk settings will be however you usually do that.
- “extra” is for the kernel command line and here gives the installer static networking with the
ip=IP address::default gateway:netmask:hostname:interface name:auto configuration typeoption.
inst.stage2here goes to a public mirror but could be an unpacked installer iso file instead.
inst.kspoints to a minimal kickstart file you’ll have to create (see below).
Minimal kickstart file ^
This kickstart file will:
- Automatically wipe disks and partition. I use
xvdafor the OS and
xvdbfor swap. Adjust accordingly.
- Install only minimal package set.
- Switch the installed system over to kernel-ml from EPEL.
- Force an SELinux autorelabel at first boot.
The only thing it doesn’t do is create any users. The installer will wait for you to do that. If you want an entirely automated install just add the user creation stuff to your kickstart file.
url --url="http://www.mirrorservice.org/sites/mirror.centos.org/8/BaseOS/x86_64/os" text # Clear all the disks. clearpart --all --initlabel zerombr # A root filesystem that takes up all of xvda. part / --ondisk=xvda --fstype=xfs --size=1 --grow # A swap partition that takes up all of xvdb. part swap --ondisk=xvdb --size=1 --grow bootloader --location=mbr --driveorder=xvda --append="console=hvc0" firstboot --disabled timezone --utc Etc/UTC --ntpservers="0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org" keyboard --vckeymap=gb --xlayouts='gb' lang en_GB.UTF-8 skipx firewall --enabled --ssh halt %packages @^Minimal install %end %post --interpreter=/usr/bin/bash --log=/root/ks-post.log --erroronfail # Switch to kernel-ml from EPEL. Necessary for Xen PV/PVH boot support. rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org yum -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm yum --enablerepo=elrepo-kernel -y install kernel-ml yum -y remove kernel-tools kernel-core kernel-modules sed -i -e 's/DEFAULTKERNEL=.*/DEFAULTKERNEL=kernel-ml/' /etc/sysconfig/kernel grub2-mkconfig -o /boot/grub2/grub.cfg # Force SELinux autorelabel on first boot. touch /.autorelabel %end
Launch the guest ^
$ sudo xl create -c /etc/xen/centostest.conf
Obviously this guest config can only boot the installer. Once it’s actually installed and halts you’ll want to make a guest config suitable for normal booting. The kernel-ml does work in PVH mode so at BitFolk we use pvhgrub to boot these.
A better way? ^
The actual modifications needed to the stock installer kernel are quite small: just enable
CONFIG_XEN_PVH kernel option and build. I don’t know the process to build a CentOS or RHEL installer kernel though, so that wasn’t an option for me.
If you do know how to do it please do send me any information you have.