Hi everyone,
I'm experimenting with Fedora 33 & BTRFS on VM for a setup I'd like to do in my desktop in the near future.
I'd like to have my system drive with BTRFS on a RAID1 profile. I was experminenting with a simple / (root filesystem) subvolume but it appears GRUB2 can't work properly with it. I get the famous:
"Sparse file is not allowed" during boot (and no way to access GRUB2 menu). It boots fine but I can't longer edit the GRUB2 menu during boot.
Fine, I read somewhere that /boot wasn't allowed with BTRFS in Fedora?
ok, I then tried to create a separate ext4 /boot partition (using Anaconda) but it created a mess (partition-wise) so I'm going to install it again with just one disk as /dev/vda1 for /boot and vda2 for BTRFS , then afterwards I'll copy partition tables to /dev/vdb, rsync /boot and balance to RAID1.
I'd like my desktop to have a RAID1 setup for my main drive and be able to remove either disk and boot seamlessly. Is this possible with BTFS & GRUB2 these days?
I tried to followed link [1] but it doesn't work with Fedora33. Does anyone know of any online guide to accomplish this with F33? Also , is it really necessary to have a separate /boot partition? Aren't there any workarounds? I'd like to keep stuff simple and have only BTRFS / & /home subvolumes.
Thanks!
On 11/2/20 11:31 AM, Jorge Fábregas wrote:
I tried to followed link [1] but it doesn't work with Fedora33.
Sorry, here it is:
http://www.gattis.org/Work-and-Tech/operating-systems-and-applications/unix/...
-- Jorge
On Mon, Nov 2, 2020 at 8:32 AM Jorge Fábregas jorge.fabregas@gmail.com wrote:
I'd like to have my system drive with BTRFS on a RAID1 profile. I was experminenting with a simple / (root filesystem) subvolume but it appears GRUB2 can't work properly with it. I get the famous:
"Sparse file is not allowed" during boot (and no way to access GRUB2 menu). It boots fine but I can't longer edit the GRUB2 menu during boot.
I'm not exactly sure what condition results in this error, but I suspect it's a bug. The message we should see when grubenv is on Btrfs is that grubenv writes on btrfs are disallowed. This is because GRUB (in the pre-boot environment) writes to grubenv not via file system writes but by directly overwriting the two 512 byte blocks that make up grubenv. On btrfs, this is indistinguishable from corruption because the checksum isn't also updated. Further, this 1KiB grubenv file is so small it tends to be an inline extent, i.e. it's sorted inside the 16KiB leaf alone with its inode. If the file were overwritten, it'd invalidate the entire 16KiB leaf. It's possibly a very serious corruption. But GRUB has known this since forever and doesn't do writes to files on Btrfs (or LUKS, or mdraid volumes, and maybe not to LVM).
So I suspect this message is just an artifact of having put your /boot on Btrfs on a BIOS system. On BIOS systems, the grubenv is located in /boot/grub2/grubenv which means if you put /boot on Btrfs, it's on Btrfs and can't use GRUB_DEFAULT=saved or the variable GRUB hidden menu feature where it checks for successful boots. Basically, GRUB can't reset the count, so it sees boots as always successful.
You can reveal the GRUB menu by F8 (often this includes using Fn key). Or you can do it permanently by
`grub2-editenv - unset menu_auto_hide`
Meanwhile on UEFI systems, grubenv is always on the FAT formatted EFI System partition.
Fine, I read somewhere that /boot wasn't allowed with BTRFS in Fedora?
It's allowed but not the default.
I'd like my desktop to have a RAID1 setup for my main drive and be able to remove either disk and boot seamlessly. Is this possible with BTFS & GRUB2 these days?
Short version: If you want unattended degraded RAID boot, use mdadm and put Btrfs on top of it.
Long version:
GRUB is not a factor at all in this. It has had Btrfs support for 11 years, and knows about all the profiles. It supports all the raid types including the new raid1c3 and raid1c4, and knows how to find things even when degraded, and can even reconstruct from parity if /boot were on a raid5 or raid6 with missing disks.
The issue is that there's no such thing as automatic degraded mounts in Btrfs. If a device has failed or is missing, you will get a long delay followed by being dropped to a dracut prompt. (At least it used to, I vaguely recall one or two issues where systemd waits for sysroot indefinitely, which seems like it should be a bug.)
Why? Missing features to make automatic degraded mounts safe. i.e. if the missing device reappears, it needs to be "caught up" with all the changes since it went missing. Right now you have to do a scrub of the entire volume, and there's no partial scrubs. Where with mdadm raid, there is a write intent bitmap that tells mdadm how to kick off a partial resync.
There is some bad advice on the internet (imagine that), suggesting removing the btrfs udev rule that waits for all Btrfs devices to show up before attempting to mount, and also add 'degraded' mount option to fstab. The problem is, any small delay by any device during boot means an immediate degraded mount. And without a scrub to catch up the late device, the mirrors can end up in kind of "split brain" situation, and that's not recoverable or repairable.
-- Chris Murphy
On 11/2/20 3:48 PM, Chris Murphy wrote:
Short version: If you want unattended degraded RAID boot, use mdadm and put Btrfs on top of it.
Hi Chris,
I get it now. Thanks. I see it's not that straightforward at this point :(
I guess I'll continue to use ext4 but now I'll consider it over mdadm (RAID 1) for my next setup as suggested. I'll miss the self-healing properties of a BTRFS RAID-1 setup but don't want the hassle at this point when ext4 has served me well.
I've been using BTRFS for 7 years now without issues (for my internal backup drive) and will continue to do so. The only major "enhacement" I'll do is to convert its profile from SINGLE to DUP when I get to replace the HDD soon (for a 2X larger one).
Thanks for that wonderful feedback Chris. It was very useful!
Best regards.
On Mon, Nov 2, 2020 at 2:14 PM Jorge Fábregas jorge.fabregas@gmail.com wrote:
On 11/2/20 3:48 PM, Chris Murphy wrote:
Short version: If you want unattended degraded RAID boot, use mdadm and put Btrfs on top of it.
Hi Chris,
I get it now. Thanks. I see it's not that straightforward at this point :(
I guess I'll continue to use ext4 but now I'll consider it over mdadm (RAID 1) for my next setup as suggested. I'll miss the self-healing properties of a BTRFS RAID-1 setup but don't want the hassle at this point when ext4 has served me well.
You don't have those issues with Btrfs on mdadm raid1. You'll still get metadata self-healing from DUP metadata profile, and you'll get error detection for data. That's not something mdadm does on its own.