On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk with a
valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or btrfs
volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases. While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases.
I'm bringing this to anaconda, kernel and test teams initially to kick around; if we agree on an approach we should then probably loop in devel@ for wider review, unless the choice is 1).
Thanks for any thoughts, folks!
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Fri, Aug 23, 2019 at 3:48 PM Justin Forbes jmforbes@linuxtx.org wrote:
On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk with a
valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or btrfs
volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
Getting btrfs in Fedora to be in a state where it *could* be the default is something I am working towards. However, it is *very* hard when people keep shutting down discussions that I try to have about enablement related to it. The situation with btrfs today is many orders of magnitude better than before, and yet I've mostly been improving Btrfs support in Fedora in tiny ways because the bigger things to do (improving kickstart, Anaconda, etc.) are impossible due to how difficult it is to contribute to those projects.
The *only* remaining "major" issue in Btrfs itself is the RAID 5/6 feature, which does not provide write hole protection without additional work (similar to mdraid). There was some work last year by David Sterba to rework the the RAID code for the SUSE Hackweek 17, but it has not been completed yet. Some work was done again to try to land this for the 5.3 cycle, but some last minute issues got that postponed. It's definitely on the radar to fix, though.
I've been watching and using Btrfs since May of 2015, and the development has drastically improved. I know for a fact no one has asked the upstream developers in at least the last two years, because I've gotten "cautionary" recommendations that it'd be okay to do so since early last year, and last week I've gotten much more enthusiastic responses when I met some of the Facebook folks (like David, who I've CC'd to this email).
The question I have is what things do we need to target to make Btrfs better for Fedora? I've already got some work done for boot to snapshot support, and I'm looking at how to adapt that for the BLS work being done today.
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
Would it help if we could get a kernel engineer who works on Btrfs to join the Fedora kernel team to help with Btrfs-specific issues? If the issue is the inability to work on that code, then getting one who does would help, right?
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases.
I would prefer option 1, provided that people stop shutting down discussions for improving support for it. As the btrfs-progs maintainer, I'm also trying to improve the quality of btrfs support in Fedora. Even if I'm mostly doing it alone right now, I'm hoping to have that change soon.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 8/23/19 1:10 PM, Neal Gompa wrote:
On Fri, Aug 23, 2019 at 3:48 PM Justin Forbes jmforbes@linuxtx.org wrote:
On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk with a
valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or btrfs
volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
Getting btrfs in Fedora to be in a state where it *could* be the default is something I am working towards. However, it is *very* hard when people keep shutting down discussions that I try to have about enablement related to it. The situation with btrfs today is many orders of magnitude better than before, and yet I've mostly been improving Btrfs support in Fedora in tiny ways because the bigger things to do (improving kickstart, Anaconda, etc.) are impossible due to how difficult it is to contribute to those projects.
The *only* remaining "major" issue in Btrfs itself is the RAID 5/6 feature, which does not provide write hole protection without additional work (similar to mdraid). There was some work last year by David Sterba to rework the the RAID code for the SUSE Hackweek 17, but it has not been completed yet. Some work was done again to try to land this for the 5.3 cycle, but some last minute issues got that postponed. It's definitely on the radar to fix, though.
I've been watching and using Btrfs since May of 2015, and the development has drastically improved. I know for a fact no one has asked the upstream developers in at least the last two years, because I've gotten "cautionary" recommendations that it'd be okay to do so since early last year, and last week I've gotten much more enthusiastic responses when I met some of the Facebook folks (like David, who I've CC'd to this email).
The question I have is what things do we need to target to make Btrfs better for Fedora? I've already got some work done for boot to snapshot support, and I'm looking at how to adapt that for the BLS work being done today.
We need a person to respond to bug reports and deal with them in a timely fashion.
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
Would it help if we could get a kernel engineer who works on Btrfs to join the Fedora kernel team to help with Btrfs-specific issues? If the issue is the inability to work on that code, then getting one who does would help, right?
I don't think we need someone to join the team per se. All we need is someone who we can assign bugs to and have them work through the issues, whether that's development or working with upstream to test. We have a fedora-btrfs bug alias and we can add whoever we want on here.
I'm okay with keeping btrfs alive if there's enough of a community who is willing to actually fix bugs and work through the issues. We do this with other parts of the kernel too.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases.
I would prefer option 1, provided that people stop shutting down discussions for improving support for it. As the btrfs-progs maintainer, I'm also trying to improve the quality of btrfs support in Fedora. Even if I'm mostly doing it alone right now, I'm hoping to have that change soon.
I think 3-5 are the best options right now with a focus on having btrfs be available but not "supported". If we had a group of people who were willing to actively debug issues like the one Adam reported, I'd be okay with #1.
Thanks, Laura
On Fri, Aug 23, 2019 at 2:26 PM Laura Abbott labbott@redhat.com wrote:
I don't think we need someone to join the team per se. All we need is someone who we can assign bugs to and have them work through the issues, whether that's development or working with upstream to test. We have a fedora-btrfs bug alias and we can add whoever we want on here.
I'm okay with keeping btrfs alive if there's enough of a community who is willing to actually fix bugs and work through the issues. We do this with other parts of the kernel too.
Past Fedora kernel team statements officially recommended against Btrfs. I think it would be weird to make Btrfs the default file system, were that still the case. And one way to alleviate that, it sounds like, is if there were a Btrfs developer on the Fedora kernel team in some capacity, even if it is strictly Btrfs bugs. But I'm open to other ideas.
And if it's just a case of release criteria violating Btrfs bugs remaining blocker worthy, then I think it can go either way.
I think 3-5 are the best options right now with a focus on having btrfs be available but not "supported". If we had a group of people who were willing to actively debug issues like the one Adam reported, I'd be okay with #1.
I'm on the fedora-kernel-btrfs@ since February, and also supposedly get btrfs-progs bug notifications. And I've been on linux-btrfs@ since early days, they know who I am, even though I can't code my way out of a hat. They've always been responsive when I show them bugs I can reproduce.
-- Chris Murphy
On Fri, Aug 23, 2019 at 1:44 PM Justin Forbes jmforbes@linuxtx.org wrote:
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it.
Josef Bacik alone pushed for it. And it was Fedora that wasn't ready for Btrfs, not the other way around. In Josef's last couple emails to devel@ he stated the decision would need to be made by others, not him. He pretty much gave up once SUSE got there first.
I'm not aware of any upstream developers recommending Fedora not use it. A significant chunk of upstream are at SUSE and by this time had moved to Btrfs by default, so it'd be a little weird if they're recommending against the thing.
At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better.
This is ambiguous. One possible way to read this is: no matter what resources are put into supporting it in Fedora, it's safe to say it won't be the default. Another possibility: the support resources necessary haven't materialized, therefore it won't be the default.
I would like to better understand why it is "safe to say" it won't be the default.
Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
The thread is ostensibly about whether it's appropriate to block release on Btrfs related bugs, not whether it should be the default file system. But as it's been brought up, I'd like to know if there's any difference in the expected support resources between it remaining "blocker worthy" versus becoming "a default file system" somewhere in the Fedora ecosystem in a release blocking capacity (i.e. presumably a Fedora Spin could choose to make Btrfs its default file system, but that wouldn't be release blocking).
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
That's understandable and reasonable. I don't think anyone uninterested in supporting Btrfs should be made to feel like they ought to. Life is short, do what you're interested in doing, no more.
On Mon, Aug 26, 2019 at 10:48 PM Chris Murphy lists@colorremedies.com wrote:
On Fri, Aug 23, 2019 at 1:44 PM Justin Forbes jmforbes@linuxtx.org wrote:
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it.
Josef Bacik alone pushed for it. And it was Fedora that wasn't ready for Btrfs, not the other way around. In Josef's last couple emails to devel@ he stated the decision would need to be made by others, not him. He pretty much gave up once SUSE got there first.
Indeed. In fact, I more or less took over the reigns of trying to improve Fedora's Btrfs support after Josef gave up.
I'm not aware of any upstream developers recommending Fedora not use it. A significant chunk of upstream are at SUSE and by this time had moved to Btrfs by default, so it'd be a little weird if they're recommending against the thing.
If anything, the Btrfs stack developers at SUSE have been nothing but helpful whenever I've talked to them about working on bringing this to Fedora. I've literally *never* heard them say that Btrfs isn't ready. However, they have told me in the past to be cautious with some features for "enterprise supportability". But they've never said anything about Btrfs not being ready as a whole.
At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better.
This is ambiguous. One possible way to read this is: no matter what resources are put into supporting it in Fedora, it's safe to say it won't be the default. Another possibility: the support resources necessary haven't materialized, therefore it won't be the default.
I would like to better understand why it is "safe to say" it won't be the default.
I really didn't want this to come up, and I ignored this the first time, but I can't anymore...
I'm pretty sure this is another variation of the "shutdowns" I keep getting when trying to work on Btrfs support in Fedora. It's surprisingly annoying that Fedora as a community distribution has touch points where community members have basically zero chance at being successful at anything.
Honestly, I'm surprised we've kept Anaconda as the installer for Fedora, given the attitude I and other community members get from the Anaconda developers and sheer amount of pain and agony we have to deal with to even attempt simple contributions, much less complex ones. Amazingly, I do have a single commit in Anaconda[0], but the amount of effort and time it took to get it in was ridiculous.
And while I like the Anaconda installer, I don't like that there can never be a community around it. It's one of those projects that has a stranglehold around it that is designed to discourage you.
And that's not even with things like Btrfs. There's been a request (with code even!) from the Qubes OS folks to add GPG key support to kickstart[1] and Anaconda[2] for literally years. I've wanted to accept the changes for livecd-tools[3] for years, but I can't until pykickstart and anaconda support it. I'm not even going to get into the repo --priority option from the FedBerry guys.
By any meaningful measure, Anaconda has less community than Btrfs, and it remains the "blessed" installer. Moreover, it gets to singlehandedly dictate what the distribution supports. And its developers basically get to shut down any discussion of anything related to Btrfs, in any form or fashion[4]. And you wonder why Btrfs support is so weak in Fedora? Because of all this, I was discouraged from finishing and submitting my change proposal to fully enable Btrfs with full system snapshots[5].
I'm honestly not surprised that Josef gave up. I'm surprised that I'm still stubborn enough to try to make it happen, but perhaps there's a part of me that hates giving up...
[0]: https://github.com/rhinstaller/anaconda/commit/c8c5589e4a4c5451d91ae8c47bf2f... [1]: https://github.com/bcl/pykickstart/pull/32 [2]: https://github.com/rhinstaller/anaconda/pull/375 [3]: https://github.com/livecd-tools/livecd-tools/pull/14 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1717728#c9 [5]: https://fedoraproject.org/wiki/Changes/BtrfsWithFullSystemSnapshots
Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
The thread is ostensibly about whether it's appropriate to block release on Btrfs related bugs, not whether it should be the default file system. But as it's been brought up, I'd like to know if there's any difference in the expected support resources between it remaining "blocker worthy" versus becoming "a default file system" somewhere in the Fedora ecosystem in a release blocking capacity (i.e. presumably a Fedora Spin could choose to make Btrfs its default file system, but that wouldn't be release blocking).
I can unequivocally say if I were allowed to, I would have a Fedora Spin that used Btrfs by default. But for the reasons outlined above, I don't think I'll ever be allowed to.
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
That's understandable and reasonable. I don't think anyone uninterested in supporting Btrfs should be made to feel like they ought to. Life is short, do what you're interested in doing, no more.
At the same time, actively strangling others' ability to do what they want is unhelpful and against the community spirit that Fedora is supposed to have.
kernel@lists.fedoraproject.org
-
Chris Murphy -
Justin Forbes -
Laura Abbott -
Neal Gompa