This is a followup to: https://bugzilla.redhat.com/show_bug.cgi?id=1101359 https://bugzilla.redhat.com/show_bug.cgi?id=1100938
While it's not strictly related to Anaconda the codebase, this is a good mailing list which has the relevant participants, and I find email to be better for architectural discussions than Bugzilla.
Now, unfortunately the OSTree model definitely impacts the bootloader situation badly. The fundamental promise of it is that upgrades are atomic, and the way that's implemented is for it to be in control of the bootloader configuration, specifically to perform an atomic replacement. That bootloader configuration then has an ostree= kernel argument which is used in the initramfs.
The original design was to use the http://www.freedesktop.org/wiki/Specifications/BootLoaderSpec/ - it had the promise of bootloader-independent configuration. I really liked that idea, as it got me out of the grubby-like model of having to keep up with lots of file formats for the many bootloaders.
Peter did add support for the BLS to grub2. However at the time, syslinux didn't implement BLS, and rather than hack it, I cowardly decided to take the easy route and just generate syslinux configs as well.
Now fast forward a bit, and mainly Chris convinced me in: https://bugzilla.redhat.com/show_bug.cgi?id=1101359 that the BLS design of having kernels/initramfs on FAT isn't right. See also: https://lists.fedoraproject.org/pipermail/desktop/2014-June/009962.html
There's another thread here, which is that I recently discovered grub2 has support for reading syslinux config files: http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/commands/syslinuxcf...
This dovetails with a Fedora feature Dennis created to use syslinux config files for u-boot: http://fedoraproject.org/wiki/Changes/u-boot_syslinux (Incidentally OSTree has code to generate u-boot too: https://git.gnome.org/browse/ostree/tree/src/libostree/ostree-bootloader-ubo... that could go away)
Now, this offers the opportunity to basically have the unified configuration aspect of the BLS, without anything else. We retain the current way UEFI boots work. OSTree and grubby remain the tools to read/write bootloader configuration (we could entertain some code merging but I don't see that as a priority).
One thing to work out with this I think would be how probed entries were merged with static syslinux config entries. Or would we just do probing at install time?
Does that all make sense?
On Thu, Jul 03, 2014 at 03:29:07PM -0700, Colin Walters wrote:
Does that all make sense?
One of the benefits of the BLS approach is that it makes it more straightforward to handle multi-OS scenarios - people can drop config fragments into a defined location and still have them picked up by any conforming bootloader. That seems desirable, but syslinux doesn't appear to have a straightforward way to handle that.
I agree that there are issues with the BLS as it currently stands (no support for chaining legacy OSes, the idea of using FAT for /boot) but it's more attractive to simply fix those bits than toss them out. I'm planning on submitting an F21 change covering this before the deadline (although, to be fair, the motivation here is mostly to get rid of the worst of os-prober rather than helping ostree - but if it works for you, that works for me)
On Fri, Jul 4, 2014, at 10:51 AM, Matthew Garrett wrote:
One of the benefits of the BLS approach is that it makes it more straightforward to handle multi-OS scenarios - people can drop config fragments into a defined location and still have them picked up by any conforming bootloader. That seems desirable, but syslinux doesn't appear to have a straightforward way to handle that.
Right, though it is an easy file to edit.
I agree that there are issues with the BLS as it currently stands (no support for chaining legacy OSes,
Yeah, I hadn't considered chainloading in my proposal...leaving it in the bootloader-specific config as the BLS suggests leaves us with os-prober for GRUB, but is that really that bad? I don't have a lot of experience with the non-Linux multiboot scenario though I admit.
the idea of using FAT for /boot) but it's more attractive to simply fix those bits than toss them out.
I agree this is worth pursuing.
I'm planning on submitting an F21 change covering this before the deadline (although, to be fair, the motivation here is mostly to get rid of the worst of os-prober rather than helping ostree - but if it works for you, that works for me)
Yep, I'll comment once I see it.
Ok, here are my first attempts at modifying the spec to be more generally useful. Note that this means that gummiboot can no longer implement the full spec, but since gummiboot is also unable to do several other things that we need to do I don't see that as a problem in Fedora.
https://secure.freedesktop.org/cgit/www/log/MatthewGarrett/BootLoaderSpec.md...
On Jul 7, 2014, at 2:43 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
Ok, here are my first attempts at modifying the spec to be more generally useful. Note that this means that gummiboot can no longer implement the full spec, but since gummiboot is also unable to do several other things that we need to do I don't see that as a problem in Fedora.
https://secure.freedesktop.org/cgit/www/log/MatthewGarrett/BootLoaderSpec.md...
* What's the distinction between $boot and $root in grub-core/commands/blscfg.c? GRUB_BOOT_DEVICE is $boot when GRUB_MACHINE_EFI. While GRUB_BOOT_DEVICE is $root on non-UEFI. Yet the revised spec says /org/freedesktop/bls goes on the ESP as $boot whether UEFI or non-EFI.
* Changes, 3rd bullet, "There's no reason to enforce VFAT as long as the filesystem is able to read it." I'm thinking "filesystem" was meant to be "firmware".
* What happens to BIOS Boot on BIOS systems where GPT is either preferred, or necessary to support > 2.2TB drives? Looks like such systems get BIOS Boot still for core.img, a FAT formatted ESP for /org/freedesktop/bls/entries, and a Linux boot partition for kernel+initramfs, grub, etc. And I'm not sure really what's gained by changing the legacy layout with one more partition, formatted as FAT, for just bootloader snippets.
* Spec doesn't address how a spec compliant installer should behave when installing along side an existing installation: Should bootability of the previous OS must be preserved? That seems a critical point of contention with most distros erring on the side of "oh, whatever" <hand waive> "user obviously doesn't care about the other OS that much, they're installing ours." I think as a default behavior it's pernicious.
* A consequence of BLS is multiple device resilient boot isn't possible because the configuration files go on a single ESP disk, which can't be on md raid (or btrfs). So it becomes a single point of failure.
On legacy BIOS systems, we've had support for this use case for years. The GUI installer also supports it out of the box, the system can boot degraded. Fedora's UEFI implementation breaks this by putting the grub.cfg on the ESP, rather than /boot/grub2 [1]; and BLS extends this to non-UEFI.
If it's an important use case, and I think it is for Workstation for sure and maybe even Server, on all archs, there needs to be a change. Alternatively we end up with long term BLS and non-BLS installs being supported, which I think is untenable both to the adoption of a BLS-like thing, as well as maintaining it.
* How about this: $bootload = The ESP, BIOS Boot, and MBR gap (or 0xEF partition). $bootbls = A neutral territory boot volume that definitely contains /org/freedesktop/bls/entries, optionally may contain kernel+initramfs. $boot = Distro specific boot (i.e. what's normally mounted at /boot on Fedora by default). $root = Distro root (i.e. what's normally mounted at /).
$bootload must be firmware readable, contains bootloader code and the least amount of information needed to find and read $bootbls. Any $bootload configuration file is static, shouldn't need modification, and is therefore easily propagated at install time to multiple disks; and to replacements when recovering from device failure.
$bootbls can be any filesystem and logical device supported by bootloader, installer, kernel. Rather than the spec proscribing filesystems, it leaves this question up to the use case and the parts needed to support it. So it could be FAT32, it could be md raid1 on XFS. The former would permit use of a future gummiboot [2], while the latter would probably require GRUB. Aside from the multiple device boot support, I'm troubled by FAT16/32 because of the corruptions I keep experiencing so I'm definitely not a fan of critical boot information being stored on this volume right now[3].
$boot may be superfluous because kernels are better suited for either $bootbls or $root, but the spec probably doesn't need to proscribe a distro specific /boot.
[1] RFE bug 1048999. Just today I realized this has been implemented by Ubuntu 14.04 LTS. Their ESP grub.cfg reads in its entirety: search.fs_uuid <volumeuuid> root hd0,gpt2 set prefix=($root) '/boot/grub' configfile $prefix/grub.cfg
[2] GRUB2 is both a floorwax and a dessert topping. Gummiboot is neither, but when it decides what it wants to be when it grows up it very clearly needs to support different volumes, even now it can be on the ESP while kernels are on the Extended Boot Partition defined by the discoverable partitions spec and he upstream BLS.
[3] Bug 1077917. Linux OS's seem to always mount it rw, which neither Windows nor OS X do. Fedora isn't fsck'ing it at mount time either, so I don't think anyone knows they've got a pending problem until boot breaks. And about 30% of my corruptions aren't fixable by fsck.vfat -a which means an fs_passno of 2 often still wouldn't fix things. I've lost count how many unrecoverable FAT32 ESP volumes I've collected while viciously testing over the last year, maybe 8-10? On the same configurations, one or two Btrfs corruptions only one of which was totally fatal, none for ext4 or XFS. So… something isn't right here. If FAT is this unreliable we probably shouldn't be persistently mounting it rw either, let alone it being used for dynamic changes to bootloader information.
Chris Murphy
Chris Murphy
So, 3 months later. It turned out that for OSTree, using extlinux was not going to work for modern UEFI machines, and we needed to support grub2.
That work is in progress now; see:
https://github.com/projectatomic/rpm-ostree/commit/15ecaacd36bfcb91181400a73... https://git.gnome.org/browse/ostree/commit/?id=d546abfa2a8c527d548b1be0ceea9... https://lists.fedorahosted.org/pipermail/anaconda-patches/2014-October/01416...
What I ended up doing was a "15_ostree" that hooks into GRUB2's config. This gets us something working today.
It'd clearly be better though to unify bootloader config files - the key decision is: 1) Push forward with mjg59's fork of the BLS 2) Have bootloaders implement syslinux 3) Live with the current setup (which I know ostree makes somewhat worse unfortunately)
anaconda-devel@lists.stg.fedoraproject.org