- The Initial Problem
- Ancient History
- What’s Gone Wrong?
- Possible Fixes
- Rebuilding My /boot
- Take a backup
- Unmount /boot and stop the RAID array that it’s on
- Delete and recreate first partition on each drive
- Create md0 array again
- Get the Array UUID
- Make a new filesystem on /dev/md0
- Mount it and put your files back
- Reinstall grub-pc
- My Kingdom For 7 Bytes
The Initial Problem ^
The recent security update of the GRUB bootloader did not want to install on my fileserver at home:
$ sudo apt dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: grub-common grub-pc grub-pc-bin grub2-common 4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 4,067 kB of archives. After this operation, 72.7 kB of additional disk space will be used. Do you want to continue? [Y/n] … Setting up grub-pc (2.02+dfsg1-20+deb10u4) ... Installing for i386-pc platform. grub-install: warning: your core.img is unusually large. It won't fit in the embedding area. grub-install: error: embedding is not possible, but this is required for RAID and LVM install. Installing for i386-pc platform. grub-install: warning: your core.img is unusually large. It won't fit in the embedding area. grub-install: error: embedding is not possible, but this is required for RAID and LVM install. Installing for i386-pc platform. grub-install: warning: your core.img is unusually large. It won't fit in the embedding area. grub-install: error: embedding is not possible, but this is required for RAID and LVM install. Installing for i386-pc platform. grub-install: warning: your core.img is unusually large. It won't fit in the embedding area. grub-install: error: embedding is not possible, but this is required for RAID and LVM install.
Four identical error messages, because this server has four drives upon which the operating system is installed, and I’d decided to do a four way RAID-1 of a small first partition to make up /boot. This error is coming from grub-install.
Ancient History ^
This system came to life in 2006, so it’s 15 years old. It’s always been Debian stable, so right now it runs Debian buster and during those 15 years it’s been transplanted into several different iterations of hardware.
Choices were made in 2006 that were reasonable for 2006, but it’s not 2006 now. Some of these choices are now causing problems.
Aside: four way RAID-1 might seem excessive, but we’re only talking about the small /boot partition. Back in 2006 I chose a ~256M one so if I did the minimal thing of only having a RAID-1 pair I’d have 2x 256M spare on the two other drives, which isn’t very useful. I’d honestly rather have all four system drives with the same partition table and there’s hardly ever writes to /boot anyway.
Here’s what the identical partition tables of the drives
/dev/sd[abcd] look like:
$ sudo fdisk -u -l /dev/sda Disk /dev/sda: 298.1 GiB, 320069031424 bytes, 625134827 sectors Disk model: ST3320620AS Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type /dev/sda1 * 63 514079 514017 251M fd Linux raid autodetect /dev/sda2 514080 6393869 5879790 2.8G fd Linux raid autodetect /dev/sda3 6393870 625121279 618727410 295G fd Linux raid autodetect
Note that the first partition starts at sector 63, 32,256 bytes into the disk. Modern partition tools tend to start partitions at sector 2,048 (1,024KiB in), but this was acceptable in 2006 for me and worked up until a few days ago.
Those four partitions
/dev/sd[abcd]1 make up an mdadm RAID-1 with metadata version
0.90. This was purposefully chosen because at the time of install GRUB did not have RAID support. This metadata version lives at the end of the member device so anything that just reads the device can pretend it’s an
ext2 filesystem. That’s what people did many years ago to boot off of software RAID.
What’s Gone Wrong? ^
The last successful update of grub-pc seems to have been done on 7 February 2021:
$ ls -la /boot/grub/i386-pc/core.img -rw-r--r-- 1 root root 31082 Feb 7 17:19 /boot/grub/i386-pc/core.img
I’ve got 62 sectors available for the core.img so that’s 31,744 bytes – just 662 bytes more than what is required.
The update of grub-pc appears to be detecting that my /boot partition is on a software RAID and is now including MD RAID support even though I don’t strictly require it. This makes the core.img larger than the space I have available for it.
I don’t think it is great that such a major change has been introduced as a security update, and it doesn’t seem like there is any easy way to tell it not to include the MD RAID support, but I’m sure everyone is doing their best here and it’s more important to get the security update out.
Possible Fixes ^
So, how to fix? It seems to me the choices are:
- Ignore the problem and stay on the older grub-pc
- Create a core.img with only the modules I need
- Rebuild my /boot partition
Option #1 is okay short term, especially if you don’t use Secure Boot as that’s what the security update was about.
Option #2 doesn’t seem that feasible as I can’t find a way to influence how Debian’s upgrade process calls grub-install. I don’t want that to become a manual process.
Option #3 seems like the easiest thing to do, as shaving ~1MiB off the size of my /boot isn’t going to cause me any issues.
Rebuilding My /boot ^
Take a backup ^
/boot is only relatively small so it seemed easiest just to tar it up ready to put it back later.
$ sudo tar -C /boot -cvf ~/boot.tar .
Unmount /boot and stop the RAID array that it’s on ^
I’ve already checked in /etc/fstab that /boot is on
$ sudo umount /boot $ sudo mdadm --stop md0 mdadm: stopped md0
At this point I would also recommend doing a
wipefs -a on each of the partitions in order to remove the MD superblocks. I didn’t and it caused me a slight problem later as we shall see.
Delete and recreate first partition on each drive ^
I chose to use parted, but should be doable with fdisk or sfdisk or whatever you prefer.
I know from the fdisk output way above that the new partition needs to start at sector 2048 and end at sector 514,079.
$ sudo parted /dev/sda GNU Parted 3.2 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) unit s (parted) rm 1 (parted) mkpart primary ext4 2048 514079s (parted) set 1 raid on (parted) set 1 boot on (parted) p Model: ATA ST3320620AS (scsi) Disk /dev/sda: 625134827s Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 2048s 514079s 512032s primary ext4 boot, raid, lba 2 514080s 6393869s 5879790s primary raid 3 6393870s 625121279s 618727410s primary raid (parted) q Information: You may need to update /etc/fstab.
Do that for each drive in turn. When I got to
/dev/sdd, this happened:
Error: Partition(s) 1 on /dev/sdd have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes. Ignore/Cancel?
The reason for this seems to be that something has decided that there is still a RAID signature on
/dev/sdd1 and so it will try to incrementally assemble the RAID-1 automatically in the background. This is why I recommend a
wipefs of each member device.
To get out of this situation without rebooting I needed to repeat my
mdadm --stop /dev/md0 command and then do a
wipefs -a /dev/sdd1. I was then able to partition it with parted.
md0 array again ^
I’m going to stick with metadata format 0.90 for this one even though it may not be strictly necessary.
$ sudo mdadm --create /dev/md0 \ --metadata 0.9 \ --level=1 \ --raid-devices=4 \ /dev/sd[abcd]1 mdadm: array /dev/md0 started.
Again, if you did not do a
wipefs earlier then mdadm will complain that these devices already have a RAID array on them and ask for confirmation.
Get the Array UUID ^
$ sudo mdadm --detail /dev/md0 /dev/md0: Version : 0.90 Creation Time : Sat Mar 6 03:20:10 2021 Raid Level : raid1 Array Size : 255936 (249.94 MiB 262.08 MB) Used Dev Size : 255936 (249.94 MiB 262.08 MB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat Mar 6 03:20:16 2021 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Consistency Policy : resync UUID : e05aa2fc:91023169:da7eb873:22131b12 (local to host specialbrew.localnet) Events : 0.18 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1
Change your /etc/mdadm/mdadm.conf for the updated UUID of
$ grep md0 /etc/mdadm/mdadm.conf ARRAY /dev/md0 level=raid1 num-devices=4 UUID=e05aa2fc:91023169:da7eb873:22131b12
Make a new filesystem on
$ sudo mkfs.ext4 -m0 -L boot /dev/md0 mke2fs 1.44.5 (15-Dec-2018) Creating filesystem with 255936 1k blocks and 64000 inodes Filesystem UUID: fdc611f2-e82a-4877-91d3-0f5f8a5dd31d Superblock backups stored on blocks: 8193, 24577, 40961, 57345, 73729, 204801, 221185 Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done
My /etc/fstab didn’t need a change because it mounted by device name, i.e.
/dev/md0, but if yours uses UUID or label then you’ll need to update that now, too.
Mount it and put your files back ^
$ sudo mount /boot $ sudo tar -C /boot -xvf ~/boot.tar
Reinstall grub-pc ^
$ sudo apt reinstall grub-pc … Setting up grub-pc (2.02+dfsg1-20+deb10u4) ... Installing for i386-pc platform. Installation finished. No error reported. Installing for i386-pc platform. Installation finished. No error reported. Installing for i386-pc platform. Installation finished. No error reported. Installing for i386-pc platform. Installation finished. No error reported.
You probably should reboot now to make sure it all works when you have time to fix any problems, as opposed to risking issues when you least expect it.
$ uprecords # Uptime | System Boot up ----------------------------+--------------------------------------------------- 1 392 days, 16:45:55 | Linux 4.7.0 Thu Jun 14 16:13:52 2018 2 325 days, 03:20:18 | Linux 3.16.0-0.bpo.4-amd Wed Apr 1 14:43:32 2015 -> 3 287 days, 16:03:12 | Linux 4.19.0-9-amd64 Fri May 22 12:33:27 2020 4 257 days, 07:31:42 | Linux 4.19.0-6-amd64 Sun Sep 8 05:00:38 2019 5 246 days, 14:45:10 | Linux 4.7.0 Sat Aug 6 06:27:52 2016 6 165 days, 01:24:22 | Linux 4.5.0-rc4-specialb Sat Feb 20 18:18:47 2016 7 131 days, 18:27:51 | Linux 3.16.0 Tue Sep 16 08:01:05 2014 8 89 days, 16:01:40 | Linux 4.7.0 Fri May 26 18:28:40 2017 9 85 days, 17:33:51 | Linux 4.7.0 Mon Feb 19 17:17:39 2018 10 63 days, 18:57:12 | Linux 3.16.0-0.bpo.4-amd Mon Jan 26 02:33:47 2015 ----------------------------+--------------------------------------------------- 1up in 37 days, 11:17:07 | at Mon Apr 12 15:53:46 2021 no1 in 105 days, 00:42:44 | at Sat Jun 19 05:19:23 2021 up 2362 days, 06:33:25 | since Tue Sep 16 08:01:05 2014 down 0 days, 14:02:09 | since Tue Sep 16 08:01:05 2014 %up 99.975 | since Tue Sep 16 08:01:05 2014
My Kingdom For 7 Bytes ^
My new core.img is 7 bytes too big to fit before my original /boot:
$ ls -la /boot/grub/i386-pc/core.img -rw-r--r-- 1 root root 31751 Mar 6 03:24 /boot/grub/i386-pc/core.img