I recently rebuilt my file server at home on a Linux software RAID array, but after rebooting last night, it no longer boots. It hangs at "grub loading stage2". When I boot rescue mode from the Fedora install CD, I can see my RAID1 ext3 /boot partition under /mnt/sysimage, and everything in it looks fine (kernels, initrds, etc. are all there) but it's not booting.

My first thought was that the boot sector got corrupted, but, I don't know if/how the boot sector comes into play on a RAID array. I tried running grub-install /dev/md0 but it doesn't recognize md0 as a block device so it won't put the boot sector there.

Was I naive to expect a RAID1 boot parititon to work properly? Should I just have a plain old primary boot partition and not worry about making it redundant?

Grub doesn't grok RAID, so all files that Grub accesses (itself, kernels, initrd images..) must all be where it can find them without having to understand RAID.

For RAID1, this can be simple, but often isn't.

The usual method is a static non-RAID boot partition, that is present (and bit-for-bit identical) and on all drives.

Normally Grub would just boot from the first drive, but if that drive dies, then one can tell it to try the next drive.

You might be able to get it working again with simply:

grub-install /dev/XXXX

where XXXX is the /boot partition on any SINGLE drive of your RAID1.

Cheers

I've been having a similar problem with grub on a debian and kubuntu install. It seems to install perfectly but won't boot and gives a loading stage 1.5 error, I think. I was going to try fedora on the box in hopes it would install, perhaps there's something bigger at work.

Matthew

OK, that makes sense. I'll give the grub-install /dev/sda thing a shot tonight, and if that fails, I'll create non-RAID partitions.

Thanks much for the explanation.

RAID1 mirrors under Linux aren't just exactly the same if they were standalone filesystems? That sucks.

FWIW, Solaris's software RAID is bootable because each half of a mirror looks exactly like a non-RAIDed device, then, during boot, the other mirror is joined. The metadata is kept elsewhere.

Well, pointing grub to the specific drive ended up working, so all is well. I guess the RAID partitions are normal enough to be bootable.

I'm sorry I missed this thread earlier because I'm fairly intimate with the RAID1 implementation in linux. I built my first RAID1 server back in 2000 when getting root and boot onto RAID1 was a non-trivial excercise and lilo was the bootloader du jour.

Anyway...

Yes, you need to install grub on both disks individually.

Remember that /dev/md0 doesn't itself have an MBR. One could argue that grub should be clever enough to install to the MBR of all drives of a RAID array, but one could also argue that since it isn't necessary to use the entirety of the drives to build an array, that it might not make sense. You should install grub to both drives of your mirror so that if your hda drive goes south, grub still exists on the second drive and you won't need to hunt down a rescue disk.

And yes, a partition that is part of a RAID1 array can be mounted outside of the array if necessary, and this is what makes boot on raid1 possible. Both partitions in a mirrored array are identical (assuming the array is clean).

When the box boots, the 512byte MBR of the BIOS-specified drive is examined, and the boot loader found there is run. In this case it will be grub stage1. Stage1 doesn't understand anything about filesystems - it does a raw block lookup for Stage1.5 to use to load Stage2. (There are several Stage1.5 files to support various filesystems - the appropriate one is installed when you install grub.) Stage1.5 understands the filesystem in use, it locates Stage2 which does the brunt of the work - determining kernel images, configurations, menus etc. Stage1.5 can recognise partitions of type 'fd' (Linux RAID Autodetect) in order to read them.

Once grub has done its stuff, it boots the kernel. Very early in the boot process, the kernel loads the md driver. The md driver scans the hard drives for partitions of type 'fd' (Linux Raid Auto). When it has built the list of such partitions it starts matching them up. It examines the persistent superblock of the partition, which contains UUID and Event Count information. Each partition in an array has the same UUID. When the md driver writes to an array it increments the Event Count. So the md driver assembles all the partitions in the array. In the event that the Event Count doesn't match the array is dirty and gets resynchronised (syncing to the partition with the highest Count).

Since the MBR isn't part of any partition per se, it never gets mirrored. It is possible to install grub to the boot sector of a partition, and one would presume that it would then get mirrored, but then you'd need to chainload that boot sector from another bootloader on the MBR (which could be grub, lilo or whatever). It's not worth the pain to do that - most people just use the MBR. (And you'd need that bootloader on the MBRs of both drives still anyway).

Remember that it is only Stage1 that isn't automatically mirrored - Stage1.5, Stage2 and all the config lives within your /boot md device. Since Stage1 only loads Stage1.5 it's basically static. Once you've installed grub on both drives then you're not going to need to touch it.

You might want to look at http://www.linuxsa.org.au/mailing-list/2003-07/1270.html for a hint about the grub config file too - getting grub to look for kernels on both drives, with automatic fallback.

HTH.

yep.. that's exactly how it works.. I've done several raid1 software mirror installs.. Normaly I do this with an initrd, and not with putting md directly in the kernel. The initrd is loaded by grub and that loads the correct kernel modules required for booting the arrays

A couple of other pitfalls that people should pay attention to:
If you use LVM with software raid.. make sure you edit the lvm.conf to force it to ignore the direct access partitions.. bad things can happen if LVM writes to one of the devices and not the raid device.

damn.. that's all I can think of right now..