My root file system doesn’t show up in “df” anymore!

Earlier tonight I had a strange bug report from a customer. Ever since I’d moved his VPS from one host to another, he’d stopped being able to see how much disk space he had free.

At first I thought it was simply because when I had moved his VPS I had taken the opportunity to reconfigure it to the new way I was setting them up, which meant that his root file system would be mounted from /dev/xvda instead of /dev/xvda1 (or /dev/sda1). That would have accounted for it if his monitoring tool had been doing it by device name, but it turned out that it was more fundamental than that — neither mount nor df were showing his root file system at all!

This was highly confusing at first. /proc/mounts looked correct and anyway how does a machine boot if it doesn’t know where its root file system is?

The answer to that question was a bit of a clue really: the boot loader tells the kernel what device the root file system is on, and in this case it was doing it by UUID. The UUID in the boot loader configuration was not the same as the UUID listed in /etc/fstab. I had forgotten to update the customer’s /etc/fstab. 🙁

The machine was able to boot because the boot loader was correctly configured, but then after it had already mounted the root file system it was trying to mount everything in /etc/fstab and failing on a line for a UUID that wasn’t present. That line then never made it to /etc/mtab which is what mount and df are reading from.

After correcting the /etc/fstab, it is fixable without a reboot by just mounting / again over the top of the existing one. Or you could probably just edit /etc/mtab.