If your Ubuntu Gutsy chroot/domU segfaults weirdly under Xen…

If your Ubuntu Gutsy chroot/domU segfaults weirdly under Xen, maybe you forgot to move /lib/tls out of the way?

I encountered this one yesterday as one of my customers asked me about problems debootstrapping under Xen. I have done several debootstraps before so wasn’t aware of any obvious problem, and this irritated me enough to make me delve into it further.

It was actually the dpkg --configure <package> step done for each package after debootstrap that was failing with a segfault. Having a look at the post-installaton script for that first package, the part where it was segfaulting was when it tried to do:

update-rc.d module-init-tools start 15 S .

a quick strace of that revealed:

open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
getdents64(3, /* 11 entries */, 4096)   = 368
getdents64(3, /* 0 entries */, 4096)    = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---

Seeing how low-level this had become I started to consider where the bug would have to be, and soon thought “what about libc?” There are well-known issues with 32 bit Xen and the TLS features of GNU libc, which is why Debian has provided a libc6-xen package since Etch. Sure enough, doing:

# mv /lib/tls /lib/tls.disabled

made this problem go away and resulted in a perfectly usable chroot environment.

TLS is normally one of the first things I check when doing things under Xen, but I didn’t think for one minute that Gutsy would still be shipping with a non-fixed libc.

I’m not really an Ubuntu user and am not aware of their packaging strategies for Xen stuff but I was also surprised to see that there is apparently no libc6-xen package in Gutsy. What is the recommended way for Ubuntu domUs to deal with this problem?

Moving the directory out of the way works, but it sucks as anything but a workaround as any libc update will replace the files. That happens to my users with older distributions all the time and they only find out because of the masses of warnings in the kernel log. Fair enough, but if it’s going to start causing weird segfaults as well then I don’t really want that support burden.

(Update: it was only because I didn’t have all the repositories in my /etc/apt/sources.list. Gutsy does have a libc6-xen package.)

It is perhaps time for me to increase the priority of moving to 64bit. (most of my servers already do 64bit but I run them 32bit as some of my customers’ chosen distributions have iffy 64bit architecture support — that may not be the case any longer)

12 thoughts on “If your Ubuntu Gutsy chroot/domU segfaults weirdly under Xen…

  1. Dear Andy,

    You’ve just saved my life. I was trying to dist-upgrade a virtual machine running Ubuntu Feisty to Ubuntu Gutsy on Amazon EC2 and I hit the TLS bug. Your solution worked perfectly.

    Thank you so much!
    Keep it up man!

  2. I’ve tried installing libc6-xen, but it doesn’t seem to want to coexist with ubuntu-minimal (ubuntu-minimal wants libc6-i686, which I assume conflicts with libc6-xen, though I can’t seem to confirm that). Is there a way to install ubuntu-minimal (which I’d imagine some packages would depend on), or is it not a big issue?

  3. Hi Alex,

    As far as I am aware libc6-xen does not conflict with libc6, so I would be interested to see what exact errors you get when you try to install libc6-xen.

  4. I cant do mv /lib/tls /lib/tls.disabled as tls is an entire directory for me containing:
    i686/ libdl-2.3.6.so libnss_files-2.3.6.so libresolv-2.3.6.so
    libanl-2.3.6.so libdl.so.2 libnss_files.so.2 libresolv.so.2
    libanl.so.1 libm-2.3.6.so libnss_hesiod-2.3.6.so librt-2.3.6.so
    libBrokenLocale-2.3.6.so libmemusage.so libnss_hesiod.so.2 librt.so.1
    libBrokenLocale.so.1 libm.so.6 libnss_nis-2.3.6.so libSegFault.so
    libc-2.3.6.so libnsl-2.3.6.so libnss_nisplus-2.3.6.so libthread_db-1.0.so
    libcidn-2.3.6.so libnsl.so.1 libnss_nisplus.so.2 libthread_db.so.1
    libcidn.so.1 libnss_compat-2.3.6.so libnss_nis.so.2 libutil-2.3.6.so
    libcrypt-2.3.6.so libnss_compat.so.2 libpcprofile.so libutil.so.1
    libcrypt.so.1 libnss_dns-2.3.6.so libpthread-2.3.6.so
    libc.so.6 libnss_dns.so.2 libpthread.so.0

    Shall I move the whole directory itself?
    I am using debian 4.0 with Xen 3.1

  5. Thanks Andy. It already works.
    But I have some other questions regarding xen:
    My setup is like this: 2 1TB disks with each with /boot and LVM. These two are in RAID1 configuration on the two disks.Also, I have installed debian 4.0r5 (etch) with kernel 2.6.18.
    So here are some concerns:
    Shall I create a new user, chroot it and then create the VMs under this user? I need to check if this is possible. If it would, then would it provide additional security?
    If not, would there be other ways to secure my server and the virtual machines?
    Also is it good to configure /etc/network/interfaces as
    auto lo
    iface lo inet loopback

    auto xenbr0
    iface xenbr0 inet static
    address 192.168.0.100
    netmask 255.255.255.0
    gateway 192.168.0.1
    bridge_ports eth0
    bridge_maxwait 0

    Finally, what is the best process to migrate an exisiting linux server into a VM?
    For our first vm we made an image of the partitions sans /boot using dd.
    I then created a partition in the LV and mounted it under /mnt, and formatted this to reiserfs. then did a mount -o loop old.img /mnt/X
    But is there a standard way to do it?

    Thanks,
    Regards,
    Amit

Leave a Reply to Nathaniel Eliot Cancel reply

Your email address will not be published. Required fields are marked *