Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
* Correctly interpret, and modify as described below, any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions * Create mount points backed by ext4 partitions, LVM volumes or btrfs volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
So...what should we do? Here are the options as I see 'em:
1. Keep supporting btrfs 2. Just modify the criterion with a btrfs exception, even if it's weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I'm bringing this to anaconda, kernel and test teams initially to kick around; if we agree on an approach we should then probably loop in devel@ for wider review, unless the choice is 1).
Thanks for any thoughts, folks!
On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk with a
valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or btrfs
volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases. While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases.
I'm bringing this to anaconda, kernel and test teams initially to kick around; if we agree on an approach we should then probably loop in devel@ for wider review, unless the choice is 1).
Thanks for any thoughts, folks!
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Fri, Aug 23, 2019 at 3:48 PM Justin Forbes jmforbes@linuxtx.org wrote:
On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk with a
valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or btrfs
volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
Getting btrfs in Fedora to be in a state where it *could* be the default is something I am working towards. However, it is *very* hard when people keep shutting down discussions that I try to have about enablement related to it. The situation with btrfs today is many orders of magnitude better than before, and yet I've mostly been improving Btrfs support in Fedora in tiny ways because the bigger things to do (improving kickstart, Anaconda, etc.) are impossible due to how difficult it is to contribute to those projects.
The *only* remaining "major" issue in Btrfs itself is the RAID 5/6 feature, which does not provide write hole protection without additional work (similar to mdraid). There was some work last year by David Sterba to rework the the RAID code for the SUSE Hackweek 17, but it has not been completed yet. Some work was done again to try to land this for the 5.3 cycle, but some last minute issues got that postponed. It's definitely on the radar to fix, though.
I've been watching and using Btrfs since May of 2015, and the development has drastically improved. I know for a fact no one has asked the upstream developers in at least the last two years, because I've gotten "cautionary" recommendations that it'd be okay to do so since early last year, and last week I've gotten much more enthusiastic responses when I met some of the Facebook folks (like David, who I've CC'd to this email).
The question I have is what things do we need to target to make Btrfs better for Fedora? I've already got some work done for boot to snapshot support, and I'm looking at how to adapt that for the BLS work being done today.
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
Would it help if we could get a kernel engineer who works on Btrfs to join the Fedora kernel team to help with Btrfs-specific issues? If the issue is the inability to work on that code, then getting one who does would help, right?
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases.
I would prefer option 1, provided that people stop shutting down discussions for improving support for it. As the btrfs-progs maintainer, I'm also trying to improve the quality of btrfs support in Fedora. Even if I'm mostly doing it alone right now, I'm hoping to have that change soon.
-- 真実はいつも一つ!/ Always, there's only one truth!
On 8/23/19 1:10 PM, Neal Gompa wrote:
On Fri, Aug 23, 2019 at 3:48 PM Justin Forbes jmforbes@linuxtx.org wrote:
On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk with a
valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or btrfs
volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
Getting btrfs in Fedora to be in a state where it *could* be the default is something I am working towards. However, it is *very* hard when people keep shutting down discussions that I try to have about enablement related to it. The situation with btrfs today is many orders of magnitude better than before, and yet I've mostly been improving Btrfs support in Fedora in tiny ways because the bigger things to do (improving kickstart, Anaconda, etc.) are impossible due to how difficult it is to contribute to those projects.
The *only* remaining "major" issue in Btrfs itself is the RAID 5/6 feature, which does not provide write hole protection without additional work (similar to mdraid). There was some work last year by David Sterba to rework the the RAID code for the SUSE Hackweek 17, but it has not been completed yet. Some work was done again to try to land this for the 5.3 cycle, but some last minute issues got that postponed. It's definitely on the radar to fix, though.
I've been watching and using Btrfs since May of 2015, and the development has drastically improved. I know for a fact no one has asked the upstream developers in at least the last two years, because I've gotten "cautionary" recommendations that it'd be okay to do so since early last year, and last week I've gotten much more enthusiastic responses when I met some of the Facebook folks (like David, who I've CC'd to this email).
The question I have is what things do we need to target to make Btrfs better for Fedora? I've already got some work done for boot to snapshot support, and I'm looking at how to adapt that for the BLS work being done today.
We need a person to respond to bug reports and deal with them in a timely fashion.
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
Would it help if we could get a kernel engineer who works on Btrfs to join the Fedora kernel team to help with Btrfs-specific issues? If the issue is the inability to work on that code, then getting one who does would help, right?
I don't think we need someone to join the team per se. All we need is someone who we can assign bugs to and have them work through the issues, whether that's development or working with upstream to test. We have a fedora-btrfs bug alias and we can add whoever we want on here.
I'm okay with keeping btrfs alive if there's enough of a community who is willing to actually fix bugs and work through the issues. We do this with other parts of the kernel too.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases.
I would prefer option 1, provided that people stop shutting down discussions for improving support for it. As the btrfs-progs maintainer, I'm also trying to improve the quality of btrfs support in Fedora. Even if I'm mostly doing it alone right now, I'm hoping to have that change soon.
I think 3-5 are the best options right now with a focus on having btrfs be available but not "supported". If we had a group of people who were willing to actively debug issues like the one Adam reported, I'd be okay with #1.
Thanks, Laura
On Fri, Aug 23, 2019 at 2:26 PM Laura Abbott labbott@redhat.com wrote:
I don't think we need someone to join the team per se. All we need is someone who we can assign bugs to and have them work through the issues, whether that's development or working with upstream to test. We have a fedora-btrfs bug alias and we can add whoever we want on here.
I'm okay with keeping btrfs alive if there's enough of a community who is willing to actually fix bugs and work through the issues. We do this with other parts of the kernel too.
Past Fedora kernel team statements officially recommended against Btrfs. I think it would be weird to make Btrfs the default file system, were that still the case. And one way to alleviate that, it sounds like, is if there were a Btrfs developer on the Fedora kernel team in some capacity, even if it is strictly Btrfs bugs. But I'm open to other ideas.
And if it's just a case of release criteria violating Btrfs bugs remaining blocker worthy, then I think it can go either way.
I think 3-5 are the best options right now with a focus on having btrfs be available but not "supported". If we had a group of people who were willing to actively debug issues like the one Adam reported, I'd be okay with #1.
I'm on the fedora-kernel-btrfs@ since February, and also supposedly get btrfs-progs bug notifications. And I've been on linux-btrfs@ since early days, they know who I am, even though I can't code my way out of a hat. They've always been responsive when I show them bugs I can reproduce.
-- Chris Murphy
On Fri, 2019-08-23 at 16:10 -0400, Neal Gompa wrote:
On Fri, Aug 23, 2019 at 3:48 PM Justin Forbes jmforbes@linuxtx.org wrote:
On Fri, Aug 23, 2019 at 2:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk
with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or
btrfs volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it. At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better. Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
Getting btrfs in Fedora to be in a state where it *could* be the default is something I am working towards. However, it is *very* hard when people keep shutting down discussions that I try to have about enablement related to it. The situation with btrfs today is many orders of magnitude better than before, and yet I've mostly been improving Btrfs support in Fedora in tiny ways because the bigger things to do (improving kickstart, Anaconda, etc.) are impossible due to how difficult it is to contribute to those projects.
Sorry about this. One of the major reasons why contribution to Anaconda is hard is because of the insane amount of possibilities which can go wrong with the simple patch and the really hard testing of those patches. This is especially true on the bootloader part and storage. We are working hard on making this better with our QE but we're not there yet.
The *only* remaining "major" issue in Btrfs itself is the RAID 5/6 feature, which does not provide write hole protection without additional work (similar to mdraid). There was some work last year by David Sterba to rework the the RAID code for the SUSE Hackweek 17, but it has not been completed yet. Some work was done again to try to land this for the 5.3 cycle, but some last minute issues got that postponed. It's definitely on the radar to fix, though.
I've been watching and using Btrfs since May of 2015, and the development has drastically improved. I know for a fact no one has asked the upstream developers in at least the last two years, because I've gotten "cautionary" recommendations that it'd be okay to do so since early last year, and last week I've gotten much more enthusiastic responses when I met some of the Facebook folks (like David, who I've CC'd to this email).
The question I have is what things do we need to target to make Btrfs better for Fedora? I've already got some work done for boot to snapshot support, and I'm looking at how to adapt that for the BLS work being done today.
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
Would it help if we could get a kernel engineer who works on Btrfs to join the Fedora kernel team to help with Btrfs-specific issues? If the issue is the inability to work on that code, then getting one who does would help, right?
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I would opt for 4 or 5, and would be in full support of 5. I do not think that it can (or should) be dropped from the kernel, because we don't want to cut off existing users, and it can still be a useful filesystem for specific cases.
I would prefer option 1, provided that people stop shutting down discussions for improving support for it. As the btrfs-progs maintainer, I'm also trying to improve the quality of btrfs support in Fedora. Even if I'm mostly doing it alone right now, I'm hoping to have that change soon.
-- 真実はいつも一つ!/ Always, there's only one truth!
Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/anaconda-devel-list
On Fri, Aug 23, 2019 at 1:44 PM Justin Forbes jmforbes@linuxtx.org wrote:
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it.
Josef Bacik alone pushed for it. And it was Fedora that wasn't ready for Btrfs, not the other way around. In Josef's last couple emails to devel@ he stated the decision would need to be made by others, not him. He pretty much gave up once SUSE got there first.
I'm not aware of any upstream developers recommending Fedora not use it. A significant chunk of upstream are at SUSE and by this time had moved to Btrfs by default, so it'd be a little weird if they're recommending against the thing.
At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better.
This is ambiguous. One possible way to read this is: no matter what resources are put into supporting it in Fedora, it's safe to say it won't be the default. Another possibility: the support resources necessary haven't materialized, therefore it won't be the default.
I would like to better understand why it is "safe to say" it won't be the default.
Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
The thread is ostensibly about whether it's appropriate to block release on Btrfs related bugs, not whether it should be the default file system. But as it's been brought up, I'd like to know if there's any difference in the expected support resources between it remaining "blocker worthy" versus becoming "a default file system" somewhere in the Fedora ecosystem in a release blocking capacity (i.e. presumably a Fedora Spin could choose to make Btrfs its default file system, but that wouldn't be release blocking).
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
That's understandable and reasonable. I don't think anyone uninterested in supporting Btrfs should be made to feel like they ought to. Life is short, do what you're interested in doing, no more.
On Mon, Aug 26, 2019 at 10:48 PM Chris Murphy lists@colorremedies.com wrote:
On Fri, Aug 23, 2019 at 1:44 PM Justin Forbes jmforbes@linuxtx.org wrote:
All of this, the criteria, and the UI support for btrfs are from the many years old proposal to make btrfs the default filesystem. In the beginning, it was not ready, but did show promise. This proposal came up for several releases in a row, and at the end of it, even the upstream developers recommended against it.
Josef Bacik alone pushed for it. And it was Fedora that wasn't ready for Btrfs, not the other way around. In Josef's last couple emails to devel@ he stated the decision would need to be made by others, not him. He pretty much gave up once SUSE got there first.
Indeed. In fact, I more or less took over the reigns of trying to improve Fedora's Btrfs support after Josef gave up.
I'm not aware of any upstream developers recommending Fedora not use it. A significant chunk of upstream are at SUSE and by this time had moved to Btrfs by default, so it'd be a little weird if they're recommending against the thing.
If anything, the Btrfs stack developers at SUSE have been nothing but helpful whenever I've talked to them about working on bringing this to Fedora. I've literally *never* heard them say that Btrfs isn't ready. However, they have told me in the past to be cautious with some features for "enterprise supportability". But they've never said anything about Btrfs not being ready as a whole.
At this point, it is safe to say that btrfs will not be the default. Since that time, things have not gotten better.
This is ambiguous. One possible way to read this is: no matter what resources are put into supporting it in Fedora, it's safe to say it won't be the default. Another possibility: the support resources necessary haven't materialized, therefore it won't be the default.
I would like to better understand why it is "safe to say" it won't be the default.
I really didn't want this to come up, and I ignored this the first time, but I can't anymore...
I'm pretty sure this is another variation of the "shutdowns" I keep getting when trying to work on Btrfs support in Fedora. It's surprisingly annoying that Fedora as a community distribution has touch points where community members have basically zero chance at being successful at anything.
Honestly, I'm surprised we've kept Anaconda as the installer for Fedora, given the attitude I and other community members get from the Anaconda developers and sheer amount of pain and agony we have to deal with to even attempt simple contributions, much less complex ones. Amazingly, I do have a single commit in Anaconda[0], but the amount of effort and time it took to get it in was ridiculous.
And while I like the Anaconda installer, I don't like that there can never be a community around it. It's one of those projects that has a stranglehold around it that is designed to discourage you.
And that's not even with things like Btrfs. There's been a request (with code even!) from the Qubes OS folks to add GPG key support to kickstart[1] and Anaconda[2] for literally years. I've wanted to accept the changes for livecd-tools[3] for years, but I can't until pykickstart and anaconda support it. I'm not even going to get into the repo --priority option from the FedBerry guys.
By any meaningful measure, Anaconda has less community than Btrfs, and it remains the "blessed" installer. Moreover, it gets to singlehandedly dictate what the distribution supports. And its developers basically get to shut down any discussion of anything related to Btrfs, in any form or fashion[4]. And you wonder why Btrfs support is so weak in Fedora? Because of all this, I was discouraged from finishing and submitting my change proposal to fully enable Btrfs with full system snapshots[5].
I'm honestly not surprised that Josef gave up. I'm surprised that I'm still stubborn enough to try to make it happen, but perhaps there's a part of me that hates giving up...
[0]: https://github.com/rhinstaller/anaconda/commit/c8c5589e4a4c5451d91ae8c47bf2f... [1]: https://github.com/bcl/pykickstart/pull/32 [2]: https://github.com/rhinstaller/anaconda/pull/375 [3]: https://github.com/livecd-tools/livecd-tools/pull/14 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1717728#c9 [5]: https://fedoraproject.org/wiki/Changes/BtrfsWithFullSystemSnapshots
Yes, there is active btrfs development upstream. It is fairly narrowly focused, and not something we can rely upon for a supported default among the Fedora use cases.
The thread is ostensibly about whether it's appropriate to block release on Btrfs related bugs, not whether it should be the default file system. But as it's been brought up, I'd like to know if there's any difference in the expected support resources between it remaining "blocker worthy" versus becoming "a default file system" somewhere in the Fedora ecosystem in a release blocking capacity (i.e. presumably a Fedora Spin could choose to make Btrfs its default file system, but that wouldn't be release blocking).
I can unequivocally say if I were allowed to, I would have a Fedora Spin that used Btrfs by default. But for the reasons outlined above, I don't think I'll ever be allowed to.
While Fedora does enable it in the kernel, and plans to continue doing so, it is enabled in the "if you break it, you get to keep the pieces" method of many other options. Sure, we will be happy to bring in a patch that is headed upstream if it fixes a bug, and someone points us to it. No, we aren't going to spend time debugging issues with it ourselves. There is no shortage of issues in more "core" kernel pieces that require attention.
That's understandable and reasonable. I don't think anyone uninterested in supporting Btrfs should be made to feel like they ought to. Life is short, do what you're interested in doing, no more.
At the same time, actively strangling others' ability to do what they want is unhelpful and against the community spirit that Fedora is supposed to have.
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
-- Chris Murphy
On Fri, 2019-08-23 at 19:00 -0600, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
Nothing. The kernel team's concern is not related to this bug or the handling of this bug in any way. The only relevance of the bug is that it alerted the kernel team to the fact that we currently block on btrfs, which they think we should not.
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer,
You're misunderstanding here. It's not the fact that a bug showed up which caused the concern.
On Sat, 2019-08-24 at 07:31 -0700, Adam Williamson wrote:
On Fri, 2019-08-23 at 19:00 -0600, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
Nothing. The kernel team's concern is not related to this bug or the handling of this bug in any way. The only relevance of the bug is that it alerted the kernel team to the fact that we currently block on btrfs, which they think we should not.
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
From the Anaconda team PoV it would make our lives easier to not
support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer,
You're misunderstanding here. It's not the fact that a bug showed up which caused the concern.
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote:
On Sat, 2019-08-24 at 07:31 -0700, Adam Williamson wrote:
On Fri, 2019-08-23 at 19:00 -0600, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
Nothing. The kernel team's concern is not related to this bug or the handling of this bug in any way. The only relevance of the bug is that it alerted the kernel team to the fact that we currently block on btrfs, which they think we should not.
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
From the Anaconda team PoV it would make our lives easier to not
support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote:
On Sat, 2019-08-24 at 07:31 -0700, Adam Williamson wrote:
On Fri, 2019-08-23 at 19:00 -0600, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
Nothing. The kernel team's concern is not related to this bug or the handling of this bug in any way. The only relevance of the bug is that it alerted the kernel team to the fact that we currently block on btrfs, which they think we should not.
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
From the Anaconda team PoV it would make our lives easier to not
support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote:
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote:
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
From the Anaconda team PoV it would make our lives easier to not
support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
From the way you describe it, Fedora is just something occasionally
give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote:
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote:
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
From the Anaconda team PoV it would make our lives easier to not
support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past".
From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin?
My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done.
Before you think I'm being naive or disingenuous, I'm truly not. I understand the frustration of wanting to get code into a project or product that isn't willing to take that code. However, it's not only about the code. The code review and acceptance isn't the end of the time investment. It is the start. There's test cases, test infrastructure, debugging support issues, on-going code changes and refactoring, etc etc. One of the ways to limit these impacts is to be selective in what enablement you choose to bring into a project. That is what the anaconda team is doing here. The beauty (and ugly!) of open source is that you can still take the code and do what you want if you are willing to make that investment.
josh
On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote:
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote:
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
From the Anaconda team PoV it would make our lives easier to not
support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past".
Until there's a branch in Anaconda's git for the next version of RHEL, it doesn't exist yet. I'm sure people are *thinking* about it, but it's obviously on the back burner for a little while. I would expect to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now it doesn't exist.
From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin?
I have seriously contemplated it. It isn't the first time I've done a fork because I had to[1].
But the main reason I don't do it is because it will cause more damage in the Fedora community by doing so. Two sets of packages for Anaconda that all the things that depend on Anaconda could cause a huge level of breakage because the two versions must *always* be drop-in replacements for each other. If they're not, it makes it impossible to leverage the Fedora tooling to do things like making spins and such. I don't even *know* what kind of work it would entail if I wanted it to be an official spin composed through pungi. I'm pretty sure the releng folks would kill me for doing that, as now pungi would have be aware and switch anaconda packages for lorax...
Additionally, it would require a fork of pykickstart so that further enhancements to the Btrfs partitioning can be defined. However, *that* causes bigger problems because now there's incompatible grammar. Given how poorly the pykickstart project is run right now, it might even make sense to fully fork it, except that it fragments a specification defined in implementation (kickstart files).
I've looked at writing automation to watch Anaconda and pykickstart and continuously integrate patchsets on top for a forked package set. But in the end, it would be too destructive for the Fedora community to do so.
[1]: https://lists.fedoraproject.org/archives/list/livecd@lists.fedoraproject.org...
My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done.
I *know* the Anaconda team barely has bandwidth right now. That is a function of them being too closed for a community to develop around it. That is *entirely* their fault. The Anaconda engineers, the team leads, the managers, and the project/product owners for Anaconda do not value the community enough to allow one to develop.
(I've met a fair number of Anaconda developers and manager folks, I know this is the case, even if they don't entirely realize it themselves.)
There are several distributions in the RH ecosystem that use Anaconda, and yet only a team of ~3 people are allowed to do anything on it? That seems brutally useless. Then again, we have the same problem with Koji. Terminally understaffing is not useful. And because the community cannot contribute to Anaconda, if there are bugs, there's no way for the community to help fix them.
Before you think I'm being naive or disingenuous, I'm truly not. I understand the frustration of wanting to get code into a project or product that isn't willing to take that code. However, it's not only about the code. The code review and acceptance isn't the end of the time investment. It is the start. There's test cases, test infrastructure, debugging support issues, on-going code changes and refactoring, etc etc. One of the ways to limit these impacts is to be selective in what enablement you choose to bring into a project. That is what the anaconda team is doing here. The beauty (and ugly!) of open source is that you can still take the code and do what you want if you are willing to make that investment.
Honestly, this goes back to Anaconda being a horrifically managed project. Where is the contribution criteria? Where is the onboarding process for more people to be involved?
I'm proficient in Python. I work in plumbing code all the time. Tell me what I need to do to make my feature possible! Let me see the test feedback in my pull request so that I can act on it! Give me (and other interested folks) the ability to contribute to the success of Anaconda!
On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote:
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote:
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
> From the Anaconda team PoV it would make our lives easier to not support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past".
Until there's a branch in Anaconda's git for the next version of RHEL, it doesn't exist yet. I'm sure people are *thinking* about it, but it's obviously on the back burner for a little while. I would expect to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now it doesn't exist.
In 1.5 years is entirely too late. Massively so. I know people think we're paying lip-service to upstream when discussing Fedora and RHEL, but even with a terrible "import once" model the code being developed in Fedora right now will materially land in RHEL 9. Your presumption on a branch needing to exist is simply incorrect.
From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin?
I have seriously contemplated it. It isn't the first time I've done a fork because I had to[1].
But the main reason I don't do it is because it will cause more damage in the Fedora community by doing so. Two sets of packages for Anaconda that all the things that depend on Anaconda could cause a huge level of breakage because the two versions must *always* be drop-in replacements for each other. If they're not, it makes it impossible to leverage the Fedora tooling to do things like making spins and such. I don't even *know* what kind of work it would entail if I wanted it to be an official spin composed through pungi. I'm pretty sure the releng folks would kill me for doing that, as now pungi would have be aware and switch anaconda packages for lorax...
So... more time investment. Yep.
Additionally, it would require a fork of pykickstart so that further enhancements to the Btrfs partitioning can be defined. However, *that* causes bigger problems because now there's incompatible grammar. Given how poorly the pykickstart project is run right now, it might even make sense to fully fork it, except that it fragments a specification defined in implementation (kickstart files).
I agree on this one. It would be great if someone created a canonical specification for both kickstarts and comps.
I've looked at writing automation to watch Anaconda and pykickstart and continuously integrate patchsets on top for a forked package set. But in the end, it would be too destructive for the Fedora community to do so.
I'll disagree, but only in severity. It would certainly be inconvenient.
My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done.
I *know* the Anaconda team barely has bandwidth right now. That is a function of them being too closed for a community to develop around it. That is *entirely* their fault. The Anaconda engineers, the team leads, the managers, and the project/product owners for Anaconda do not value the community enough to allow one to develop.
(I've met a fair number of Anaconda developers and manager folks, I know this is the case, even if they don't entirely realize it themselves.)
I think you're conflating "selective in what they accept" and "do not value community".
There are several distributions in the RH ecosystem that use Anaconda, and yet only a team of ~3 people are allowed to do anything on it? That seems brutally useless. Then again, we have the same problem with Koji. Terminally understaffing is not useful. And because the community cannot contribute to Anaconda, if there are bugs, there's no way for the community to help fix them.
There certainly is. You submit a PR. It's what they would do. That PR being merged or not is a completely different topic than the ability to submit it in the first place.
Before you think I'm being naive or disingenuous, I'm truly not. I understand the frustration of wanting to get code into a project or product that isn't willing to take that code. However, it's not only about the code. The code review and acceptance isn't the end of the time investment. It is the start. There's test cases, test infrastructure, debugging support issues, on-going code changes and refactoring, etc etc. One of the ways to limit these impacts is to be selective in what enablement you choose to bring into a project. That is what the anaconda team is doing here. The beauty (and ugly!) of open source is that you can still take the code and do what you want if you are willing to make that investment.
Honestly, this goes back to Anaconda being a horrifically managed project. Where is the contribution criteria? Where is the onboarding process for more people to be involved?
I mean, if you want to materially help free up time for that team to do project on-boarding, developer education, and code review you could always volunteer to do all their bug triage. Nobody wants to do the crap work though, so you're just asking for more things they don't have time to properly document.
Or you could document it for them and see what they think?
I'm proficient in Python. I work in plumbing code all the time. Tell me what I need to do to make my feature possible! Let me see the test feedback in my pull request so that I can act on it! Give me (and other interested folks) the ability to contribute to the success of Anaconda!
This is not hard. You're trending to hyperbole, which is always fun. So let's bring it back to concrete steps: You submit PRs. Submit them with testcases. Submit them with documentation. If it's something the project wants to accept, great. If it isn't, they'll tell you. Remember, determining what contributes to the success of the project is up to the project owners. Dealing with rejection can be hard for anyone, but it's not a statement on you or your work. It normally winds up being about what the project is trying to do and what it wants to support.
josh
On Tue, Aug 27, 2019 at 8:30 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote:
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote: > > I understand them. The point is, for them and even us (the > installer) > is work on BTRFS not a priority. It's something we can't benefit on > RHEL and it could be almost completely replaced by LVM + xfs > solution. > However, it still giving us bugs and making our test surface > bigger. > > > From the Anaconda team PoV it would make our lives easier to not > support BTRFS at all. I'm not saying that we should drop BTRFS in > Fedora, only that it would be easier for Anaconda team to be > without > that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past".
Until there's a branch in Anaconda's git for the next version of RHEL, it doesn't exist yet. I'm sure people are *thinking* about it, but it's obviously on the back burner for a little while. I would expect to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now it doesn't exist.
In 1.5 years is entirely too late. Massively so. I know people think we're paying lip-service to upstream when discussing Fedora and RHEL, but even with a terrible "import once" model the code being developed in Fedora right now will materially land in RHEL 9. Your presumption on a branch needing to exist is simply incorrect.
For us non RH people, there's nothing for us to do about RHEL 9 right now. Your planning is done in secret, and there's little to no community feedback loop for RHEL.next. I agree that in ordinary circumstances, it's too late once the branch has happened, because usually that means it's the stabilizing phase. But there's nothing I can do before then.
From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin?
I have seriously contemplated it. It isn't the first time I've done a fork because I had to[1].
But the main reason I don't do it is because it will cause more damage in the Fedora community by doing so. Two sets of packages for Anaconda that all the things that depend on Anaconda could cause a huge level of breakage because the two versions must *always* be drop-in replacements for each other. If they're not, it makes it impossible to leverage the Fedora tooling to do things like making spins and such. I don't even *know* what kind of work it would entail if I wanted it to be an official spin composed through pungi. I'm pretty sure the releng folks would kill me for doing that, as now pungi would have be aware and switch anaconda packages for lorax...
So... more time investment. Yep.
Additionally, it would require a fork of pykickstart so that further enhancements to the Btrfs partitioning can be defined. However, *that* causes bigger problems because now there's incompatible grammar. Given how poorly the pykickstart project is run right now, it might even make sense to fully fork it, except that it fragments a specification defined in implementation (kickstart files).
I agree on this one. It would be great if someone created a canonical specification for both kickstarts and comps.
Comps isn't *quite* so bad off. There's a DTD for it, at least. I've collected most of the RPM-MD repodata specification files that I could find into a single repo: https://pagure.io/rpm-metadata
Someday, maybe those documents can be rationalized into a formal spec, but there are bigger fish to fry in that realm right now...
I've looked at writing automation to watch Anaconda and pykickstart and continuously integrate patchsets on top for a forked package set. But in the end, it would be too destructive for the Fedora community to do so.
I'll disagree, but only in severity. It would certainly be inconvenient.
It sends a bad message as well: "Fedora developers have to fork their own installer because installer developers can't work with Fedora developers."
My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done.
I *know* the Anaconda team barely has bandwidth right now. That is a function of them being too closed for a community to develop around it. That is *entirely* their fault. The Anaconda engineers, the team leads, the managers, and the project/product owners for Anaconda do not value the community enough to allow one to develop.
(I've met a fair number of Anaconda developers and manager folks, I know this is the case, even if they don't entirely realize it themselves.)
I think you're conflating "selective in what they accept" and "do not value community".
There are several distributions in the RH ecosystem that use Anaconda, and yet only a team of ~3 people are allowed to do anything on it? That seems brutally useless. Then again, we have the same problem with Koji. Terminally understaffing is not useful. And because the community cannot contribute to Anaconda, if there are bugs, there's no way for the community to help fix them.
There certainly is. You submit a PR. It's what they would do. That PR being merged or not is a completely different topic than the ability to submit it in the first place.
There *was* a PR submitted. It was even a one-liner because the contributor fixed the underlying problem elsewhere. It's been in limbo for over a year: https://github.com/rhinstaller/anaconda/pull/1375
You seem to think that I'm just shouting without any effort to back it up. There was originally four of us working on this two years ago. It's dwindled over time as the roadblocks were thrown up time and time again.
On Tue, Aug 27, 2019 at 8:48 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 8:30 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote:
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote: > > > > I understand them. The point is, for them and even us (the > > installer) > > is work on BTRFS not a priority. It's something we can't benefit on > > RHEL and it could be almost completely replaced by LVM + xfs > > solution. > > However, it still giving us bugs and making our test surface > > bigger. > > > > > From the Anaconda team PoV it would make our lives easier to not > > support BTRFS at all. I'm not saying that we should drop BTRFS in > > Fedora, only that it would be easier for Anaconda team to be > > without > > that on Fedora. > > This is flat-out a trap. This is what makes Anaconda such a failure > as > a community project. Why does the past (RHEL) affect the present and > future (Fedora)? There's basically no way whatsoever to make anything > better with this logic. The Anaconda releases that any improvements > would be going into aren't even landing into the RHEL 8 branch that > governs the latest iteration of Fedora's past. From any reasonable > person's perspective, this answer makes no sense unless you're using > RHEL as an excuse to not support Fedora. >
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past".
Until there's a branch in Anaconda's git for the next version of RHEL, it doesn't exist yet. I'm sure people are *thinking* about it, but it's obviously on the back burner for a little while. I would expect to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now it doesn't exist.
In 1.5 years is entirely too late. Massively so. I know people think we're paying lip-service to upstream when discussing Fedora and RHEL, but even with a terrible "import once" model the code being developed in Fedora right now will materially land in RHEL 9. Your presumption on a branch needing to exist is simply incorrect.
For us non RH people, there's nothing for us to do about RHEL 9 right now. Your planning is done in secret, and there's little to no community feedback loop for RHEL.next. I agree that in ordinary circumstances, it's too late once the branch has happened, because usually that means it's the stabilizing phase. But there's nothing I can do before then.
I think that's a problem and one which we're trying to address. However, whether code lands due to internal planning or due to community contribution, it all still impacts any future OS release, Fedora and enterprise alike.
From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin?
I have seriously contemplated it. It isn't the first time I've done a fork because I had to[1].
But the main reason I don't do it is because it will cause more damage in the Fedora community by doing so. Two sets of packages for Anaconda that all the things that depend on Anaconda could cause a huge level of breakage because the two versions must *always* be drop-in replacements for each other. If they're not, it makes it impossible to leverage the Fedora tooling to do things like making spins and such. I don't even *know* what kind of work it would entail if I wanted it to be an official spin composed through pungi. I'm pretty sure the releng folks would kill me for doing that, as now pungi would have be aware and switch anaconda packages for lorax...
So... more time investment. Yep.
Additionally, it would require a fork of pykickstart so that further enhancements to the Btrfs partitioning can be defined. However, *that* causes bigger problems because now there's incompatible grammar. Given how poorly the pykickstart project is run right now, it might even make sense to fully fork it, except that it fragments a specification defined in implementation (kickstart files).
I agree on this one. It would be great if someone created a canonical specification for both kickstarts and comps.
Comps isn't *quite* so bad off. There's a DTD for it, at least. I've collected most of the RPM-MD repodata specification files that I could find into a single repo: https://pagure.io/rpm-metadata
Someday, maybe those documents can be rationalized into a formal spec, but there are bigger fish to fry in that realm right now...
So you've made a priority call? :)
I've looked at writing automation to watch Anaconda and pykickstart and continuously integrate patchsets on top for a forked package set. But in the end, it would be too destructive for the Fedora community to do so.
I'll disagree, but only in severity. It would certainly be inconvenient.
It sends a bad message as well: "Fedora developers have to fork their own installer because installer developers can't work with Fedora developers."
Generalization again.
My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done.
I *know* the Anaconda team barely has bandwidth right now. That is a function of them being too closed for a community to develop around it. That is *entirely* their fault. The Anaconda engineers, the team leads, the managers, and the project/product owners for Anaconda do not value the community enough to allow one to develop.
(I've met a fair number of Anaconda developers and manager folks, I know this is the case, even if they don't entirely realize it themselves.)
I think you're conflating "selective in what they accept" and "do not value community".
There are several distributions in the RH ecosystem that use Anaconda, and yet only a team of ~3 people are allowed to do anything on it? That seems brutally useless. Then again, we have the same problem with Koji. Terminally understaffing is not useful. And because the community cannot contribute to Anaconda, if there are bugs, there's no way for the community to help fix them.
There certainly is. You submit a PR. It's what they would do. That PR being merged or not is a completely different topic than the ability to submit it in the first place.
There *was* a PR submitted. It was even a one-liner because the contributor fixed the underlying problem elsewhere. It's been in limbo for over a year: https://github.com/rhinstaller/anaconda/pull/1375
You seem to think that I'm just shouting without any effort to back it up. There was originally four of us working on this two years ago. It's dwindled over time as the roadblocks were thrown up time and time again.
No, I don't think that at all. I think you've taken that PR, which was rejected because the project doesn't want to do more btrfs enablement, and generalized it to "nobody can contribute to anaconda". That's my point. You're using hyperbole as an argument for something specific.
If you have other PRs that were general bug fixes or unrelated to btrfs which were rejected, I think it would be refreshing for all involved to look at why again. That would indicate a contribution model problem to me. Not a single feature/PR the project doesn't want to include.
josh
On Tue, Aug 27, 2019 at 8:57 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 8:48 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 8:30 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 8:10 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote: > > On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote: > > On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote: > > > > > > I understand them. The point is, for them and even us (the > > > installer) > > > is work on BTRFS not a priority. It's something we can't benefit on > > > RHEL and it could be almost completely replaced by LVM + xfs > > > solution. > > > However, it still giving us bugs and making our test surface > > > bigger. > > > > > > > From the Anaconda team PoV it would make our lives easier to not > > > support BTRFS at all. I'm not saying that we should drop BTRFS in > > > Fedora, only that it would be easier for Anaconda team to be > > > without > > > that on Fedora. > > > > This is flat-out a trap. This is what makes Anaconda such a failure > > as > > a community project. Why does the past (RHEL) affect the present and > > future (Fedora)? There's basically no way whatsoever to make anything > > better with this logic. The Anaconda releases that any improvements > > would be going into aren't even landing into the RHEL 8 branch that > > governs the latest iteration of Fedora's past. From any reasonable > > person's perspective, this answer makes no sense unless you're using > > RHEL as an excuse to not support Fedora. > > > > RHEL is not the past. Everything we do we have to think that it will go > to RHEL and if it is Fedora specific we have to create a way to disable > the functionality for another RHEL branching. And yes, we have a few > things (not only a BTRFS) specific to Fedora the same way as a few > things specific to RHEL which are disabled on Fedora. > > And as I wrote before, I'm not saying that we will remove the BTRFS > support from Fedora. The point is that making the list specific to > releases smaller will make our live easier. >
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past".
Until there's a branch in Anaconda's git for the next version of RHEL, it doesn't exist yet. I'm sure people are *thinking* about it, but it's obviously on the back burner for a little while. I would expect to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now it doesn't exist.
In 1.5 years is entirely too late. Massively so. I know people think we're paying lip-service to upstream when discussing Fedora and RHEL, but even with a terrible "import once" model the code being developed in Fedora right now will materially land in RHEL 9. Your presumption on a branch needing to exist is simply incorrect.
For us non RH people, there's nothing for us to do about RHEL 9 right now. Your planning is done in secret, and there's little to no community feedback loop for RHEL.next. I agree that in ordinary circumstances, it's too late once the branch has happened, because usually that means it's the stabilizing phase. But there's nothing I can do before then.
I think that's a problem and one which we're trying to address. However, whether code lands due to internal planning or due to community contribution, it all still impacts any future OS release, Fedora and enterprise alike.
From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin?
I have seriously contemplated it. It isn't the first time I've done a fork because I had to[1].
But the main reason I don't do it is because it will cause more damage in the Fedora community by doing so. Two sets of packages for Anaconda that all the things that depend on Anaconda could cause a huge level of breakage because the two versions must *always* be drop-in replacements for each other. If they're not, it makes it impossible to leverage the Fedora tooling to do things like making spins and such. I don't even *know* what kind of work it would entail if I wanted it to be an official spin composed through pungi. I'm pretty sure the releng folks would kill me for doing that, as now pungi would have be aware and switch anaconda packages for lorax...
So... more time investment. Yep.
Additionally, it would require a fork of pykickstart so that further enhancements to the Btrfs partitioning can be defined. However, *that* causes bigger problems because now there's incompatible grammar. Given how poorly the pykickstart project is run right now, it might even make sense to fully fork it, except that it fragments a specification defined in implementation (kickstart files).
I agree on this one. It would be great if someone created a canonical specification for both kickstarts and comps.
Comps isn't *quite* so bad off. There's a DTD for it, at least. I've collected most of the RPM-MD repodata specification files that I could find into a single repo: https://pagure.io/rpm-metadata
Someday, maybe those documents can be rationalized into a formal spec, but there are bigger fish to fry in that realm right now...
So you've made a priority call? :)
I've looked at writing automation to watch Anaconda and pykickstart and continuously integrate patchsets on top for a forked package set. But in the end, it would be too destructive for the Fedora community to do so.
I'll disagree, but only in severity. It would certainly be inconvenient.
It sends a bad message as well: "Fedora developers have to fork their own installer because installer developers can't work with Fedora developers."
Generalization again.
My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done.
I *know* the Anaconda team barely has bandwidth right now. That is a function of them being too closed for a community to develop around it. That is *entirely* their fault. The Anaconda engineers, the team leads, the managers, and the project/product owners for Anaconda do not value the community enough to allow one to develop.
(I've met a fair number of Anaconda developers and manager folks, I know this is the case, even if they don't entirely realize it themselves.)
I think you're conflating "selective in what they accept" and "do not value community".
There are several distributions in the RH ecosystem that use Anaconda, and yet only a team of ~3 people are allowed to do anything on it? That seems brutally useless. Then again, we have the same problem with Koji. Terminally understaffing is not useful. And because the community cannot contribute to Anaconda, if there are bugs, there's no way for the community to help fix them.
There certainly is. You submit a PR. It's what they would do. That PR being merged or not is a completely different topic than the ability to submit it in the first place.
There *was* a PR submitted. It was even a one-liner because the contributor fixed the underlying problem elsewhere. It's been in limbo for over a year: https://github.com/rhinstaller/anaconda/pull/1375
You seem to think that I'm just shouting without any effort to back it up. There was originally four of us working on this two years ago. It's dwindled over time as the roadblocks were thrown up time and time again.
No, I don't think that at all. I think you've taken that PR, which was rejected because the project doesn't want to do more btrfs enablement, and generalized it to "nobody can contribute to anaconda". That's my point. You're using hyperbole as an argument for something specific.
If you have other PRs that were general bug fixes or unrelated to btrfs which were rejected, I think it would be refreshing for all involved to look at why again. That would indicate a contribution model problem to me. Not a single feature/PR the project doesn't want to include.
If it was just that, I'd agree with you.
But other people have had similar "limbo PR" problems for other features. The gpgkey and repo priority support for repositories in pykickstart and anaconda are other examples.
And even my attempt to proactively fix Anaconda to exclude packages correctly took quite a long time to get through. Over six months before getting feedback, and that only happened because I nagged people on IRC to look at it to fix other problems. Trying to fix something *before* it becomes a problem is an exercise in futility. It only got looked at once we started having problems in media creation and net installer.
Anaconda definitely has a problem.
On Tue, 2019-08-27 at 08:57 -0400, Josh Boyer wrote:
There *was* a PR submitted. It was even a one-liner because the contributor fixed the underlying problem elsewhere. It's been in limbo for over a year: https://github.com/rhinstaller/anaconda/pull/1375
You seem to think that I'm just shouting without any effort to back it up. There was originally four of us working on this two years ago. It's dwindled over time as the roadblocks were thrown up time and time again.
No, I don't think that at all. I think you've taken that PR, which was rejected because the project doesn't want to do more btrfs enablement, and generalized it to "nobody can contribute to anaconda". That's my point. You're using hyperbole as an argument for something specific.
If you have other PRs that were general bug fixes or unrelated to btrfs which were rejected, I think it would be refreshing for all involved to look at why again. That would indicate a contribution model problem to me. Not a single feature/PR the project doesn't want to include.
Josh, to be fair, I can see Neal's point here. In a way you seem to be kinda sending him in circles here: "anaconda team doesn't have the time/resources to invest in btrfs development, but you can help by sending them pull requests. Oh, you sent them a pull request and they rejected it? Well, it's because they don't have time/resources for btrfs development..." You're right that one rejected PR doesn't necessarily indicate a contribution model problem, but putting the wider issue aside, a very simple one-liner with a major impact on btrfs functionality being rejected *does* seem to say a lot about the prospects of any btrfs-related work being accepted.
Putting myself in Neal's shoes, given my experience with that PR and other attempts to help out with btrfs in anaconda, why would I invest my time and effort to write another one when it seems extremely likely it would be rejected?
I think we at least owe it to people to be clear about whether there is any point in them investing time and effort *trying* to contribute to btrfs support in anaconda, and if the answer is currently "no", whether there would realistically be any prospect of there being any way to change this.
I also think there are other perspectives that might at least potentially be useful here. Right now we've mainly heard from a couple of community folks who are very passionate about btrfs, and Red Hat folks from anaconda/kernel development and RHEL management who essentially see it as a burden that is not aligned with their priorities. Is that all the perspectives we have to make a decision with? I think we should at least talk to the Facebook team that apparently uses btrfs on Fedora extensively about whether they'd be sad to see installer btrfs support die and, if so, whether they'd perhaps be interested in helping support it. And more broadly, community folks on all the lists this is going to: are there more people who actually are interested in this functionality and would be sad to see it go? If btrfs isn't a part of Red Hat's product roadmaps but is important to lots of folks in the wider Fedora community, maybe that's another indicator we can use....or indeed, if it turns out not many folks actually care.
On 8/27/19 12:07 PM, Adam Williamson wrote:
On Tue, 2019-08-27 at 08:57 -0400, Josh Boyer wrote:
There *was* a PR submitted. It was even a one-liner because the contributor fixed the underlying problem elsewhere. It's been in limbo for over a year: https://github.com/rhinstaller/anaconda/pull/1375
You seem to think that I'm just shouting without any effort to back it up. There was originally four of us working on this two years ago. It's dwindled over time as the roadblocks were thrown up time and time again.
No, I don't think that at all. I think you've taken that PR, which was rejected because the project doesn't want to do more btrfs enablement, and generalized it to "nobody can contribute to anaconda". That's my point. You're using hyperbole as an argument for something specific.
If you have other PRs that were general bug fixes or unrelated to btrfs which were rejected, I think it would be refreshing for all involved to look at why again. That would indicate a contribution model problem to me. Not a single feature/PR the project doesn't want to include.
Josh, to be fair, I can see Neal's point here. In a way you seem to be kinda sending him in circles here: "anaconda team doesn't have the time/resources to invest in btrfs development, but you can help by sending them pull requests. Oh, you sent them a pull request and they rejected it? Well, it's because they don't have time/resources for btrfs development..." You're right that one rejected PR doesn't necessarily indicate a contribution model problem, but putting the wider issue aside, a very simple one-liner with a major impact on btrfs functionality being rejected *does* seem to say a lot about the prospects of any btrfs-related work being accepted.
Putting myself in Neal's shoes, given my experience with that PR and other attempts to help out with btrfs in anaconda, why would I invest my time and effort to write another one when it seems extremely likely it would be rejected?
There's an assumption here that when someone is asked to send a PR, it will always be accepted. Enabling and/or fixing btrfs functionality in anaconda is not a zero cost. There's also the issue that anaconda has always aimed to not break systems. Does the project have the resources to guarantee that and fix problems that show up? What might appear at first as a "one line patch" comes with a lot of other assumptions and expectations from users.
I think we at least owe it to people to be clear about whether there is any point in them investing time and effort *trying* to contribute to btrfs support in anaconda, and if the answer is currently "no", whether there would realistically be any prospect of there being any way to change this.
I also think there are other perspectives that might at least potentially be useful here. Right now we've mainly heard from a couple of community folks who are very passionate about btrfs, and Red Hat folks from anaconda/kernel development and RHEL management who essentially see it as a burden that is not aligned with their priorities. Is that all the perspectives we have to make a decision with? I think we should at least talk to the Facebook team that apparently uses btrfs on Fedora extensively about whether they'd be sad to see installer btrfs support die and, if so, whether they'd perhaps be interested in helping support it. And more broadly, community folks on all the lists this is going to: are there more people who actually are interested in this functionality and would be sad to see it go? If btrfs isn't a part of Red Hat's product roadmaps but is important to lots of folks in the wider Fedora community, maybe that's another indicator we can use....or indeed, if it turns out not many folks actually care.
The installer team rejecting btrfs patches is going to be based on their resources to support the functionality. I would say "btrfs in Fedora" needs a FESCo decision to set expectations and policy for the project. Is it something that Fedora wants to offer and if so, what does that look like? If it's a best effort thing, then that makes it easier for projects and contributors. Going back to Adam's original list, I would suggest a FESCo decision like this should require explicit opt-in by the user to enable btrfs functionality in the application in question. For example, in the installer that could be enabled via a boot parameter (we did this initially when btrfs functionality was first enabled in anaconda).
I'm not advocating one way or another for btrfs. But it seems we as a project need a larger decision and policy around btrfs in general so we can set expectations for users and developers.
Thanks,
On Tue, Aug 27, 2019 at 11:25 AM David Cantrell dcantrell@redhat.com wrote:
The installer team rejecting btrfs patches is going to be based on their resources to support the functionality. I would say "btrfs in Fedora" needs a FESCo decision to set expectations and policy for the project. Is it something that Fedora wants to offer and if so, what does that look like?
FESCo already voted 8 years ago to make Btrfs the default file system, and then allowed that to wither and become moot rather than revert the decision. Then later when the editions were created, part of Fedora.next, the decision of default file systems was handed to the working groups to decide. And the Fedora kernel team has also said this is a working group decision.
The Fedora working group's technical specification states Btrfs is to be the default. Yet the working group has said it's uncomfortable taking action on this decision expressly because the Federal kernel team's official recommendation is to not recommend Btrfs. And I agree. I trust the Fedora kernel team as they've clearly stated limited resources and interest in Btrfs, the expectations and parameters for properly supporting Btrfs either as bug blocker worthy, and as a default file system from a user advocacy point of view.
If it's a best effort thing, then that makes it easier for projects and contributors. Going back to Adam's original list, I would suggest a FESCo decision like this should require explicit opt-in by the user to enable btrfs functionality in the application in question. For example, in the installer that could be enabled via a boot parameter (we did this initially when btrfs functionality was first enabled in anaconda).
That can only be considered to be a remarkable regression, not just in the context of Fedora, but in the context of the top 10 linux distributions all of which have visible Btrfs support in their GUI installers. Fedora's installer being the first to make Btrfs invisible by default would be a remarkable first indeed.
I'm not advocating one way or another for btrfs. But it seems we as a project need a larger decision and policy around btrfs in general so we can set expectations for users and developers.
That decision and policy has already been made. Do you want it reverted?
On Tue, Aug 27, 2019 at 12:00 PM Chris Murphy lists@colorremedies.com wrote:
The Fedora working group's technical specification states Btrfs is to be the default. Yet the working group has said it's uncomfortable taking action on this decision expressly because the Federal kernel team's official recommendation is to not recommend Btrfs. And I agree.
Points of clarity:
- it is the Workstation working group's technical spec that says this (the Server working group decided on XFS)
- I agree with Workstation working group's reticence to actually deploy Btrfs as the default file system, while the Fedora kernel team recommends against Btrfs as the default. And I would like someone on the Fedora kernel team who will recommend and support it.
References:
FESCo, make Btrfs the default https://meetbot.fedoraproject.org/fedora-meeting/2011-06-08/fesco.2011-06-08...
Workstation working group's technical specification, make btrfs default https://fedoraproject.org/wiki/Workstation/Technical_Specification
A point of reference that when LVM thinp was made release blocking, it was stated that usability issues are to be treated as bugs to be worked out https://meetbot.fedoraproject.org/fedora-meeting/2013-07-24/fesco.2013-07-24...
File system choice is up to the working group https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
--- Chris Murphy
On 8/27/19 2:00 PM, Chris Murphy wrote:
On Tue, Aug 27, 2019 at 11:25 AM David Cantrell dcantrell@redhat.com wrote:
The installer team rejecting btrfs patches is going to be based on their resources to support the functionality. I would say "btrfs in Fedora" needs a FESCo decision to set expectations and policy for the project. Is it something that Fedora wants to offer and if so, what does that look like?
FESCo already voted 8 years ago to make Btrfs the default file system, and then allowed that to wither and become moot rather than revert the decision. Then later when the editions were created, part of Fedora.next, the decision of default file systems was handed to the working groups to decide. And the Fedora kernel team has also said this is a working group decision.
The Fedora working group's technical specification states Btrfs is to be the default. Yet the working group has said it's uncomfortable taking action on this decision expressly because the Federal kernel team's official recommendation is to not recommend Btrfs. And I agree. I trust the Fedora kernel team as they've clearly stated limited resources and interest in Btrfs, the expectations and parameters for properly supporting Btrfs either as bug blocker worthy, and as a default file system from a user advocacy point of view.
OK, so 8 years has gone by and the landscape around btrfs looks different in Fedora. Given the kernel team's position, is it worth still having the FESCo decision and kernel team's recommendation at odds?
If it's a best effort thing, then that makes it easier for projects and contributors. Going back to Adam's original list, I would suggest a FESCo decision like this should require explicit opt-in by the user to enable btrfs functionality in the application in question. For example, in the installer that could be enabled via a boot parameter (we did this initially when btrfs functionality was first enabled in anaconda).
That can only be considered to be a remarkable regression, not just in the context of Fedora, but in the context of the top 10 linux distributions all of which have visible Btrfs support in their GUI installers. Fedora's installer being the first to make Btrfs invisible by default would be a remarkable first indeed.
I'm merely offering an example scenario.
This does illustrate a problem with expectations among users. It's visible now, but not a priority, which leads to frustration.
I'm not advocating one way or another for btrfs. But it seems we as a project need a larger decision and policy around btrfs in general so we can set expectations for users and developers.
That decision and policy has already been made. Do you want it reverted?
I guess I meant to say "FESCo needs to revisit this decision for a potential change".
Thanks,
On Tue, Aug 27, 2019 at 1:02 PM David Cantrell dcantrell@redhat.com wrote:
On 8/27/19 2:00 PM, Chris Murphy wrote:
The Fedora working group's technical specification states Btrfs is to be the default. Yet the working group has said it's uncomfortable taking action on this decision expressly because the Federal kernel team's official recommendation is to not recommend Btrfs. And I agree. I trust the Fedora kernel team as they've clearly stated limited resources and interest in Btrfs, the expectations and parameters for properly supporting Btrfs either as bug blocker worthy, and as a default file system from a user advocacy point of view.
OK, so 8 years has gone by and the landscape around btrfs looks different in Fedora. Given the kernel team's position, is it worth still having the FESCo decision and kernel team's recommendation at odds?
They aren't at odds.
8 years ago FESCo decision on Btrfs 5 years ago Workstation working group decision on Btrfs (their requirements and specs docs were signed off by FESCo) 3 years ago kernel team said while they don't recommend it, they acknowledged it was ultimately up to the working group to decide, and noted they were sick of having this conversation every release
The Workstation working group has, correctly in my view, put their own decision in abeyance, as a result of the kernel team's official recommendation. How does the Btrfs landscape in Fedora look different to you in three years?
The way the landscape looks different to me:
One, the possibility of getting a kernel engineer who works on Btrfs to join the Fedora kernel team, so that the team has the capacity to support and recommend (at least as a team, not that any individual needs to give up one tiny bit of their Btrfs scepticism) is an important development.
Two, that in the interim, Red Hat is where the landscape has really changed has occurred with respect to Btrfs, and I see indications of this bias being injected back into Fedora: suggesting that a Red Hat shift away from Btrfs somehow is actually a Fedora shift away from Btrfs, suggesting a do over vote in the Workstation working group, or even an undo vote by FESCo to set aside the Workstation working group vote.
And to what end? All that's needed is for the Anaconda team to provided the same courtesy of clear expectations and parameters, as the kernel team has done.
If it's a best effort thing, then that makes it easier for projects and contributors. Going back to Adam's original list, I would suggest a FESCo decision like this should require explicit opt-in by the user to enable btrfs functionality in the application in question. For example, in the installer that could be enabled via a boot parameter (we did this initially when btrfs functionality was first enabled in anaconda).
That can only be considered to be a remarkable regression, not just in the context of Fedora, but in the context of the top 10 linux distributions all of which have visible Btrfs support in their GUI installers. Fedora's installer being the first to make Btrfs invisible by default would be a remarkable first indeed.
I'm merely offering an example scenario.
This does illustrate a problem with expectations among users. It's visible now, but not a priority, which leads to frustration.
Just like today's kernel team, Anaconda today are not obligated to make it a priority. But qualifying what "not a priority" actually means is necessary. The question is what are the preconditions for others who will make it a priority? And whether Anaconda is going to slow walk every PR that tries to improve Btrfs support? What about simplifying the Btrfs support in Anaconda to make it easier to maintain with priority parity with other file systems? Gut all the multiple device stuff, perhaps? Btrfs can easily do quite a lot of adjustments post-install, it's one of it's most central features. Few options are mkfs only.
I'm not advocating one way or another for btrfs. But it seems we as a project need a larger decision and policy around btrfs in general so we can set expectations for users and developers.
That decision and policy has already been made. Do you want it reverted?
I guess I meant to say "FESCo needs to revisit this decision for a potential change".
I can't parse this. Which decision, and what potential change?
-- Chris Murphy
On Tue, 2019-08-27 at 16:54 -0600, Chris Murphy wrote:
On Tue, Aug 27, 2019 at 1:02 PM David Cantrell dcantrell@redhat.com wrote:
On 8/27/19 2:00 PM, Chris Murphy wrote:
The Fedora working group's technical specification states Btrfs is to be the default. Yet the working group has said it's uncomfortable taking action on this decision expressly because the Federal kernel team's official recommendation is to not recommend Btrfs. And I agree. I trust the Fedora kernel team as they've clearly stated limited resources and interest in Btrfs, the expectations and parameters for properly supporting Btrfs either as bug blocker worthy, and as a default file system from a user advocacy point of view.
OK, so 8 years has gone by and the landscape around btrfs looks different in Fedora. Given the kernel team's position, is it worth still having the FESCo decision and kernel team's recommendation at odds?
They aren't at odds.
8 years ago FESCo decision on Btrfs 5 years ago Workstation working group decision on Btrfs (their requirements and specs docs were signed off by FESCo) 3 years ago kernel team said while they don't recommend it, they acknowledged it was ultimately up to the working group to decide, and noted they were sick of having this conversation every release
The Workstation working group has, correctly in my view, put their own decision in abeyance, as a result of the kernel team's official recommendation. How does the Btrfs landscape in Fedora look different to you in three years?
The way the landscape looks different to me:
One, the possibility of getting a kernel engineer who works on Btrfs to join the Fedora kernel team, so that the team has the capacity to support and recommend (at least as a team, not that any individual needs to give up one tiny bit of their Btrfs scepticism) is an important development.
Two, that in the interim, Red Hat is where the landscape has really changed has occurred with respect to Btrfs, and I see indications of this bias being injected back into Fedora: suggesting that a Red Hat shift away from Btrfs somehow is actually a Fedora shift away from Btrfs, suggesting a do over vote in the Workstation working group, or even an undo vote by FESCo to set aside the Workstation working group vote.
And to what end? All that's needed is for the Anaconda team to provided the same courtesy of clear expectations and parameters, as the kernel team has done.
If it's a best effort thing, then that makes it easier for projects and contributors. Going back to Adam's original list, I would suggest a FESCo decision like this should require explicit opt- in by the user to enable btrfs functionality in the application in question. For example, in the installer that could be enabled via a boot parameter (we did this initially when btrfs functionality was first enabled in anaconda).
That can only be considered to be a remarkable regression, not just in the context of Fedora, but in the context of the top 10 linux distributions all of which have visible Btrfs support in their GUI installers. Fedora's installer being the first to make Btrfs invisible by default would be a remarkable first indeed.
I'm merely offering an example scenario.
This does illustrate a problem with expectations among users. It's visible now, but not a priority, which leads to frustration.
Just like today's kernel team, Anaconda today are not obligated to make it a priority. But qualifying what "not a priority" actually means is necessary. The question is what are the preconditions for others who will make it a priority? And whether Anaconda is going to slow walk every PR that tries to improve Btrfs support? What about simplifying the Btrfs support in Anaconda to make it easier to maintain with priority parity with other file systems? Gut all the multiple device stuff, perhaps? Btrfs can easily do quite a lot of adjustments post-install, it's one of it's most central features. Few options are mkfs only.
Removing something from Anaconda to replace it by BTRFS functionality doesn't really help. It would even make our lives harder. The point is that the same code has to be there because of the RHEL so it would only diverge our effort that everything have to be done twice.
I'm not advocating one way or another for btrfs. But it seems we as a project need a larger decision and policy around btrfs in general so we can set expectations for users and developers.
That decision and policy has already been made. Do you want it reverted?
I guess I meant to say "FESCo needs to revisit this decision for a potential change".
I can't parse this. Which decision, and what potential change?
-- Chris Murphy
Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/anaconda-devel-list
On Tue, 2019-08-27 at 13:25 -0400, David Cantrell wrote:
Josh, to be fair, I can see Neal's point here. In a way you seem to be kinda sending him in circles here: "anaconda team doesn't have the time/resources to invest in btrfs development, but you can help by sending them pull requests. Oh, you sent them a pull request and they rejected it? Well, it's because they don't have time/resources for btrfs development..." You're right that one rejected PR doesn't necessarily indicate a contribution model problem, but putting the wider issue aside, a very simple one-liner with a major impact on btrfs functionality being rejected *does* seem to say a lot about the prospects of any btrfs-related work being accepted.
Putting myself in Neal's shoes, given my experience with that PR and other attempts to help out with btrfs in anaconda, why would I invest my time and effort to write another one when it seems extremely likely it would be rejected?
There's an assumption here that when someone is asked to send a PR, it will always be accepted.
No, that's not what I'm assuming, but if we (Red Hat) tell people that it would be a good idea to send PRs, we should at least be *potentially* willing to accept them. We should not be saying 'send PRs' if there is no chance of them being accepted. And it would be nicest if we gave people a roadmap: here are the specific things you can do that would be acceptable to us as a way to continue including btrfs handling in the installer.
Just as a for instance - if the anaconda team would find it acceptable to maintain btrfs installer support if some person or some group came up with a way to modularize filesystem support in the installer such that that person or group could maintain it and the existing anaconda devs would not have to worry about it (and perhaps even such that images could easily be built with or without support for specific filesystems), then we should tell people that that is the kind of work we're looking for and would accept if it was done.
I know the folks on both sides here and I understand both their frustrations: the anaconda team is trying to maintain a very large project with very limited resources and very specific demands from the people who write their paychecks, which is absolutely a difficult position to be in. But also, the community folks here are trying their best to contribute - not just to demand that the RH folks do stuff for them, but actually to contribute - but feel like they are being given no guidance at all as to what kind of contribution would actually be welcomed or acceptable, they feel like the message is "just keep coming up with stuff and we'll keep rejecting it until we see something we like". I know it's a tough position for both sides, but I really hope we can get somewhere more positive than here with it.
Enabling and/or fixing btrfs functionality in anaconda is not a zero cost. There's also the issue that anaconda has always aimed to not break systems. Does the project have the resources to guarantee that and fix problems that show up? What might appear at first as a "one line patch" comes with a lot of other assumptions and expectations from users.
I think we at least owe it to people to be clear about whether there is any point in them investing time and effort *trying* to contribute to btrfs support in anaconda, and if the answer is currently "no", whether there would realistically be any prospect of there being any way to change this.
I also think there are other perspectives that might at least potentially be useful here. Right now we've mainly heard from a couple of community folks who are very passionate about btrfs, and Red Hat folks from anaconda/kernel development and RHEL management who essentially see it as a burden that is not aligned with their priorities. Is that all the perspectives we have to make a decision with? I think we should at least talk to the Facebook team that apparently uses btrfs on Fedora extensively about whether they'd be sad to see installer btrfs support die and, if so, whether they'd perhaps be interested in helping support it. And more broadly, community folks on all the lists this is going to: are there more people who actually are interested in this functionality and would be sad to see it go? If btrfs isn't a part of Red Hat's product roadmaps but is important to lots of folks in the wider Fedora community, maybe that's another indicator we can use....or indeed, if it turns out not many folks actually care.
The installer team rejecting btrfs patches is going to be based on their resources to support the functionality.
But then why not think about whether those resources are available outside just the people Red Hat pay to work on anaconda? If there are other folks banging on our door and telling us they really want to work on supporting btrfs, wouldn't it be cool to at least try and see how that could work?
I would say "btrfs in Fedora" needs a FESCo decision to set expectations and policy for the project. Is it something that Fedora wants to offer and if so, what does that look like?
I agree, but I also think this discussion is kind of an input to that decision. I don't know that FESCo can simply make it in a vacuum.
If it's a best effort thing, then that makes it easier for projects and contributors. Going back to Adam's original list, I would suggest a FESCo decision like this should require explicit opt-in by the user to enable btrfs functionality in the application in question. For example, in the installer that could be enabled via a boot parameter (we did this initially when btrfs functionality was first enabled in anaconda).
That's exactly the kind of thing that would be helpful to the community, indeed. If everyone can agree on this, that gives them an entry point.
On Tue, 2019-08-27 at 14:59 -0700, Adam Williamson wrote:
On Tue, 2019-08-27 at 13:25 -0400, David Cantrell wrote:
Josh, to be fair, I can see Neal's point here. In a way you seem to be kinda sending him in circles here: "anaconda team doesn't have the time/resources to invest in btrfs development, but you can help by sending them pull requests. Oh, you sent them a pull request and they rejected it? Well, it's because they don't have time/resources for btrfs development..." You're right that one rejected PR doesn't necessarily indicate a contribution model problem, but putting the wider issue aside, a very simple one-liner with a major impact on btrfs functionality being rejected *does* seem to say a lot about the prospects of any btrfs-related work being accepted.
Putting myself in Neal's shoes, given my experience with that PR and other attempts to help out with btrfs in anaconda, why would I invest my time and effort to write another one when it seems extremely likely it would be rejected?
There's an assumption here that when someone is asked to send a PR, it will always be accepted.
No, that's not what I'm assuming, but if we (Red Hat) tell people that it would be a good idea to send PRs, we should at least be *potentially* willing to accept them. We should not be saying 'send PRs' if there is no chance of them being accepted. And it would be nicest if we gave people a roadmap: here are the specific things you can do that would be acceptable to us as a way to continue including btrfs handling in the installer.
Sorry but I have to react on this. It's killing me how many of you people here are telling Anaconda does not accept patches. That is really not true. We have smaller amount of contributions from community that is partially our fault because of a lack of documentation but mainly it's really hard to develop & test Anaconda and most users see it just once in a few years so it's not bothering them so much. I guess most of the installers are in the same situation.
Please before any of you will tell again that Anaconda team is not accepting patches, please look on the last few years history. How many patches were "rejected" and how many were accepted. There are just a few patches which weren't accepted and basically only a few PR (I would even say the only one) for BTRFS. That is unfortunate but it's not all our contributions.
Please stop saying all the time that Anaconda is not accepting contributions or that users doesn't have a chance to get the contributions in.
Just as a for instance - if the anaconda team would find it acceptable to maintain btrfs installer support if some person or some group came up with a way to modularize filesystem support in the installer such that that person or group could maintain it and the existing anaconda devs would not have to worry about it (and perhaps even such that images could easily be built with or without support for specific filesystems), then we should tell people that that is the kind of work we're looking for and would accept if it was done.
I know the folks on both sides here and I understand both their frustrations: the anaconda team is trying to maintain a very large project with very limited resources and very specific demands from the people who write their paychecks, which is absolutely a difficult position to be in. But also, the community folks here are trying their best to contribute - not just to demand that the RH folks do stuff for them, but actually to contribute - but feel like they are being given no guidance at all as to what kind of contribution would actually be welcomed or acceptable, they feel like the message is "just keep coming up with stuff and we'll keep rejecting it until we see something we like". I know it's a tough position for both sides, but I really hope we can get somewhere more positive than here with it.
Enabling and/or fixing btrfs functionality in anaconda is not a zero cost. There's also the issue that anaconda has always aimed to not break systems. Does the project have the resources to guarantee that and fix problems that show up? What might appear at first as a "one line patch" comes with a lot of other assumptions and expectations from users.
I think we at least owe it to people to be clear about whether there is any point in them investing time and effort *trying* to contribute to btrfs support in anaconda, and if the answer is currently "no", whether there would realistically be any prospect of there being any way to change this.
I also think there are other perspectives that might at least potentially be useful here. Right now we've mainly heard from a couple of community folks who are very passionate about btrfs, and Red Hat folks from anaconda/kernel development and RHEL management who essentially see it as a burden that is not aligned with their priorities. Is that all the perspectives we have to make a decision with? I think we should at least talk to the Facebook team that apparently uses btrfs on Fedora extensively about whether they'd be sad to see installer btrfs support die and, if so, whether they'd perhaps be interested in helping support it. And more broadly, community folks on all the lists this is going to: are there more people who actually are interested in this functionality and would be sad to see it go? If btrfs isn't a part of Red Hat's product roadmaps but is important to lots of folks in the wider Fedora community, maybe that's another indicator we can use....or indeed, if it turns out not many folks actually care.
The installer team rejecting btrfs patches is going to be based on their resources to support the functionality.
But then why not think about whether those resources are available outside just the people Red Hat pay to work on anaconda? If there are other folks banging on our door and telling us they really want to work on supporting btrfs, wouldn't it be cool to at least try and see how that could work?
I would say "btrfs in Fedora" needs a FESCo decision to set expectations and policy for the project. Is it something that Fedora wants to offer and if so, what does that look like?
I agree, but I also think this discussion is kind of an input to that decision. I don't know that FESCo can simply make it in a vacuum.
If it's a best effort thing, then that makes it easier for projects and contributors. Going back to Adam's original list, I would suggest a FESCo decision like this should require explicit opt-in by the user to enable btrfs functionality in the application in question. For example, in the installer that could be enabled via a boot parameter (we did this initially when btrfs functionality was first enabled in anaconda).
That's exactly the kind of thing that would be helpful to the community, indeed. If everyone can agree on this, that gives them an entry point.
On Thu, Aug 29, 2019 at 4:23 AM jkonecny@redhat.com wrote:
On Tue, 2019-08-27 at 14:59 -0700, Adam Williamson wrote:
On Tue, 2019-08-27 at 13:25 -0400, David Cantrell wrote:
Josh, to be fair, I can see Neal's point here. In a way you seem to be kinda sending him in circles here: "anaconda team doesn't have the time/resources to invest in btrfs development, but you can help by sending them pull requests. Oh, you sent them a pull request and they rejected it? Well, it's because they don't have time/resources for btrfs development..." You're right that one rejected PR doesn't necessarily indicate a contribution model problem, but putting the wider issue aside, a very simple one-liner with a major impact on btrfs functionality being rejected *does* seem to say a lot about the prospects of any btrfs-related work being accepted.
Putting myself in Neal's shoes, given my experience with that PR and other attempts to help out with btrfs in anaconda, why would I invest my time and effort to write another one when it seems extremely likely it would be rejected?
There's an assumption here that when someone is asked to send a PR, it will always be accepted.
No, that's not what I'm assuming, but if we (Red Hat) tell people that it would be a good idea to send PRs, we should at least be *potentially* willing to accept them. We should not be saying 'send PRs' if there is no chance of them being accepted. And it would be nicest if we gave people a roadmap: here are the specific things you can do that would be acceptable to us as a way to continue including btrfs handling in the installer.
Sorry but I have to react on this. It's killing me how many of you people here are telling Anaconda does not accept patches. That is really not true. We have smaller amount of contributions from community that is partially our fault because of a lack of documentation but mainly it's really hard to develop & test Anaconda and most users see it just once in a few years so it's not bothering them so much. I guess most of the installers are in the same situation.
I sat on this for a few days, as I wanted to cool down and think about how to reply to this. As of this moment, I have directly and indirectly contributed to both the Red Hat and SUSE installers, and yes it's definitely true that both installers have some of the worst interaction models for contributors I've ever seen.
YaST's contribution model and process seems to be somewhat better, because their team has components being used by other people. Their libyui components are used by Fedora and Mageia for dnfdragora, and the rest of ManaTools in Mageia uses it too. The YaST control center is a cornerstone in the user experience in SUSE distributions, so there are contributors from their community who do work on the main YaST components.
The Calamares installer stack has a pretty healthy community, but our community has an interesting aversion to all things Qt/KDE even though the installer works fine on Fedora...
But the point I'm trying to make is there is no reason for Anaconda to be so difficult. Unfortunately, it is.
Please before any of you will tell again that Anaconda team is not accepting patches, please look on the last few years history. How many patches were "rejected" and how many were accepted. There are just a few patches which weren't accepted and basically only a few PR (I would even say the only one) for BTRFS. That is unfortunate but it's not all our contributions.
I also took the opportunity look back at over the last 400 merged pull requests by paging through GitHub. Now I don't exactly have firm numbers, but in the past 400 pull requests, I saw less than a dozen pull requests merged from non-RHers. Half of them were from Pat from the Scientific Linux team. One of them was a documentation fix from some person I don't recognize with no clear affiliation. If I page back *slightly more* then the next PR from a non Red Hatter that was merged was *mine* fixing Anaconda's package exclusion feature for kickstarts (which was nearly two years old when it was merged).
Of the 42 currently open pull requests, 4 are from people I could clearly identify as not from Red Hatters. Of the ones that are from Red Hatters, 7 appear to not be from members of the Red Hat Installer team or the Red Hat Bootloader team as I know of them.
The lag time to response *is* improving. It's gone down from years to months, with the most recent pull request getting a comment within a week, and then stalling out after that. So it may even improve to weeks!
As Adam pointed out, there's literally no reason for me to attempt more sophisticated changes to Anaconda if a simple one-liner can't get merged. And I didn't even make that fix, I've just been trying to shepherd it in for over a year! I only gave up trying and focusing on other things when it became clear I'd be dogged with non-answers forever.
Please stop saying all the time that Anaconda is not accepting contributions or that users doesn't have a chance to get the contributions in.
You guys brought it upon yourselves by having terrible documentation and contribution policies, unclear ownership or responsibilities of projects that are clearly related (pykickstart, which by the way guys, changed hands *again* and now is basically in a brand new black hole!), oh, and lets not forget the oh-so-joyous aspect of the CI that is forbidden to be accessed by non-Red Hatters.
Honestly, I shouldn't have expected better. When I noticed that the Anaconda mailing lists never moved from the Red Hat lists to lists.fedorahosted.org, I should have realized that Red Hat wasn't treating it like a community project. Every dysfunctional RH/Fedora ecosystem project seems to have that as a common property. It's emblematic of the deeper problem that it's not treated as a project where the community is valued. OpenShift, Spacewalk, and Anaconda are all projects where I've suffered these problems. That's not to say RH projects not hosted on the Red Hat lists can't be dysfunctional too, but people seem to try to be better when they aren't. Maybe it's an attitude difference? A different approach to the project? I don't know.
But something is wrong and it needs to be fixed.
----- Original Message -----
From: "Neal Gompa" ngompa13@gmail.com To: "Josh Boyer" jwboyer@fedoraproject.org Cc: "For testing and quality assurance of Fedora releases" test@lists.fedoraproject.org, "kernel" kernel@lists.fedoraproject.org, "Discussion of Development and Customization of the Red Hat Linux Installer" anaconda-devel-list@redhat.com Sent: Tuesday, August 27, 2019 2:09:41 PM Subject: Re: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 7:41 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Aug 27, 2019 at 7:19 AM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Aug 27, 2019 at 5:55 AM jkonecny@redhat.com wrote:
On Mon, 2019-08-26 at 23:54 -0400, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 7:16 AM jkonecny@redhat.com wrote:
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
> From the Anaconda team PoV it would make our lives easier to not support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
This is flat-out a trap. This is what makes Anaconda such a failure as a community project. Why does the past (RHEL) affect the present and future (Fedora)? There's basically no way whatsoever to make anything better with this logic. The Anaconda releases that any improvements would be going into aren't even landing into the RHEL 8 branch that governs the latest iteration of Fedora's past. From any reasonable person's perspective, this answer makes no sense unless you're using RHEL as an excuse to not support Fedora.
RHEL is not the past. Everything we do we have to think that it will go to RHEL and if it is Fedora specific we have to create a way to disable the functionality for another RHEL branching. And yes, we have a few things (not only a BTRFS) specific to Fedora the same way as a few things specific to RHEL which are disabled on Fedora.
And as I wrote before, I'm not saying that we will remove the BTRFS support from Fedora. The point is that making the list specific to releases smaller will make our live easier.
By definition, RHEL *is* the past from a Fedora context. It's forked from an old version of Fedora that's not supported anymore. It is the result of decisions that aren't supposed to apply to Fedora. And it is the result of a different bias that should never apply to Fedora if the RH ecosystem is supposed to be able to evolve.
There is always the *next* RHEL. Or, if you want to remove a product context, the next enterprise operating system derived from Fedora. RHEL/enterprise is both past and future and Fedora focuses on the future. You cannot dismiss enterprise as a target by waiving it away as "past".
Until there's a branch in Anaconda's git for the next version of RHEL, it doesn't exist yet. I'm sure people are *thinking* about it, but it's obviously on the back burner for a little while. I would expect to start to see a rhel-9 branch in Anaconda in 1.5 years, but for now it doesn't exist.
From the way you describe it, Fedora is just something occasionally give lip service to while your main focus is RHEL. That's fine, but that is a problem for the Fedora context.
Neal, I don't understand. The source code to anaconda is available. What is preventing you from taking it and making a micro-fork that does better btrfs enablement, and packaging that in Fedora and using it in a spin?
I have seriously contemplated it. It isn't the first time I've done a fork because I had to[1].
But the main reason I don't do it is because it will cause more damage in the Fedora community by doing so. Two sets of packages for Anaconda that all the things that depend on Anaconda could cause a huge level of breakage because the two versions must *always* be drop-in replacements for each other. If they're not, it makes it impossible to leverage the Fedora tooling to do things like making spins and such. I don't even *know* what kind of work it would entail if I wanted it to be an official spin composed through pungi. I'm pretty sure the releng folks would kill me for doing that, as now pungi would have be aware and switch anaconda packages for lorax...
Additionally, it would require a fork of pykickstart so that further enhancements to the Btrfs partitioning can be defined. However, *that* causes bigger problems because now there's incompatible grammar. Given how poorly the pykickstart project is run right now, it might even make sense to fully fork it, except that it fragments a specification defined in implementation (kickstart files).
I've looked at writing automation to watch Anaconda and pykickstart and continuously integrate patchsets on top for a forked package set. But in the end, it would be too destructive for the Fedora community to do so.
My guess would be that you perhaps don't have the time to do that. That's reasonable. I can say, with no uncertainty, that the anaconda team doesn't have time to do it either. The day to day things that team is working on far outweigh btrfs as a priority, even in Fedora. This team literally gets dozens of completely unrelated bugs they have to look at and triage simply because it is the first thing people interact with when they install the OS. It's a catch-all. Btrfs enablement would continually sit on the back burner waiting to get done.
I *know* the Anaconda team barely has bandwidth right now. That is a function of them being too closed for a community to develop around it. That is *entirely* their fault. The Anaconda engineers, the team leads, the managers, and the project/product owners for Anaconda do not value the community enough to allow one to develop.
(I've met a fair number of Anaconda developers and manager folks, I know this is the case, even if they don't entirely realize it themselves.)
There are several distributions in the RH ecosystem that use Anaconda, and yet only a team of ~3 people are allowed to do anything on it? That seems brutally useless. Then again, we have the same problem with Koji. Terminally understaffing is not useful. And because the community cannot contribute to Anaconda, if there are bugs, there's no way for the community to help fix them.
Before you think I'm being naive or disingenuous, I'm truly not. I understand the frustration of wanting to get code into a project or product that isn't willing to take that code. However, it's not only about the code. The code review and acceptance isn't the end of the time investment. It is the start. There's test cases, test infrastructure, debugging support issues, on-going code changes and refactoring, etc etc. One of the ways to limit these impacts is to be selective in what enablement you choose to bring into a project. That is what the anaconda team is doing here. The beauty (and ugly!) of open source is that you can still take the code and do what you want if you are willing to make that investment.
Honestly, this goes back to Anaconda being a horrifically managed project. Where is the contribution criteria?
We have contribution guidelines as part of the Anaconda documentation on Read the Docs:
https://anaconda-installer.readthedocs.io/en/latest/contributing.html
Please let us know if you see something there that is incorrect or out of date (that can always happen).
Where is the onboarding process for more people to be involved?
I'm proficient in Python. I work in plumbing code all the time. Tell me what I need to do to make my feature possible! Let me see the test feedback in my pull request so that I can act on it! Give me (and other interested folks) the ability to contribute to the success of Anaconda!
-- 真実はいつも一つ!/ Always, there's only one truth!
Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/anaconda-devel-list
On Mon, Aug 26, 2019 at 5:16 AM jkonecny@redhat.com wrote:
I understand them. The point is, for them and even us (the installer) is work on BTRFS not a priority. It's something we can't benefit on RHEL and it could be almost completely replaced by LVM + xfs solution. However, it still giving us bugs and making our test surface bigger.
From the Anaconda team PoV it would make our lives easier to not
support BTRFS at all. I'm not saying that we should drop BTRFS in Fedora, only that it would be easier for Anaconda team to be without that on Fedora.
It would also be easier if Custom and Advanced partitioning UIs were dropped entirely.
Most Linux distros now support Btrfs. All the top 10 do. One, currently ranked #7 Solus, supports it via a point and shoot installer, deferring to Gparted to actually set it up. All the others have a custom interface that supports Btrfs directly.
Meanwhile, #23 Parrot uses Btrfs by default for home and root. And so does openSUSE for a while now.
And the idea being floated, is that Fedora shouldn't have a sense of adventure, but to maybe drop Btrfs from the installer. Fedora would be the first, if it did.
It is completely reasonable for Red Hat to have maintainability concerns about Btrfs for RHEL, and it's entirely fair for Red Hat to have a bias against it. If it were true that Red Hat is, however unintentionally, injecting its Btrfs bias into Fedora, that would be troubling.
On Fri, Aug 23, 2019 at 8:00 PM Chris Murphy lists@colorremedies.com wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
From my standpoint, ext4 and xfs are the primary supported root
filesystems. I don't think that anything else should be release blocking. As for whether the installer exposes other options, it is really more of an installer and QA question. We are certainly not even discussing turning them off in the kernel at this point, and I don't think that we should.
-- Chris Murphy _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
On Mon, Aug 26, 2019 at 2:42 PM Justin Forbes jmforbes@linuxtx.org wrote:
From my standpoint, ext4 and xfs are the primary supported root filesystems. I don't think that anything else should be release blocking.
If this is the case, we can explicitly list the supported file systems in criteria. The list would need to be extended with at least vfat, which is used for ESP, though.
If we go this route, it would be nice to communicate this somehow to the end user, directly in anaconda interface. Either by showing a warning when a "not officially supported" filesystem is selected, or by hiding those filesystems in dialogs when creating a new partition (with a documented override). Existing partitions still need to be handled somehow, so the warning bar might need to be implemented in any case (warn that the existing partition is unsupported by allow to use it, or warn that the existing partition can't be used unless the override is activated).
On Mon, Aug 26, 2019 at 4:53 PM Kamil Paral kparal@redhat.com wrote:
On Mon, Aug 26, 2019 at 2:42 PM Justin Forbes jmforbes@linuxtx.org wrote:
From my standpoint, ext4 and xfs are the primary supported root filesystems. I don't think that anything else should be release blocking.
If this is the case, we can explicitly list the supported file systems in criteria. The list would need to be extended with at least vfat, which is used for ESP, though.
If we go this route, it would be nice to communicate this somehow to the end user, directly in anaconda interface. Either by showing a warning when a "not officially supported" filesystem is selected, or by hiding those filesystems in dialogs when creating a new partition (with a documented override).
Hmm, I don't see this as necessary. I think changing criterions on what file systems are blocking doesn't mean we need to hide things or add some ugly warnings. Anybody who uses advanced partitioning should know what is doing, we can just update criterions so not everything visible in advanced partitioning must work and is supported.
Existing partitions still need to be handled somehow, so the warning bar might need to be implemented in any case (warn that the existing partition is unsupported by allow to use it, or warn that the existing partition can't be used unless the override is activated).
I am -1 on this. I just somehow hate the idea of showing warnings and/or adding some blocks and overrides. We weren't testing on unsupported/other file systems anyway (correct me if I am mistaken), so what's the difference now?
On Mon, 2019-08-26 at 19:33 +0200, Frantisek Zatloukal wrote:
On Mon, Aug 26, 2019 at 4:53 PM Kamil Paral kparal@redhat.com wrote:
On Mon, Aug 26, 2019 at 2:42 PM Justin Forbes jmforbes@linuxtx.org wrote:
From my standpoint, ext4 and xfs are the primary supported root filesystems. I don't think that anything else should be release blocking.
If this is the case, we can explicitly list the supported file systems in criteria. The list would need to be extended with at least vfat, which is used for ESP, though.
If we go this route, it would be nice to communicate this somehow to the end user, directly in anaconda interface. Either by showing a warning when a "not officially supported" filesystem is selected, or by hiding those filesystems in dialogs when creating a new partition (with a documented override).
Hmm, I don't see this as necessary. I think changing criterions on what file systems are blocking doesn't mean we need to hide things or add some ugly warnings. Anybody who uses advanced partitioning should know what is doing, we can just update criterions so not everything visible in advanced partitioning must work and is supported.
I mean, I don't see that there's necessarily an equivalence between "knowing what you're doing" and "expecting it to be broken". That's the reason I quite like the current criterion: it commits us to not just throwing a bunch of hand grenades at the user in the installer. If we're going to do that it should at least be communicated *somehow* outside of just being in the release criteria.
On 8/23/19 9:00 PM, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case.
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases.
Thanks, Laura
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott labbott@redhat.com wrote:
On 8/23/19 9:00 PM, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case.
Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of?
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases.
Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat.
There are plenty of examples of Red Hat doing crazy/experimental things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is...
On 8/26/19 11:39 PM, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott labbott@redhat.com wrote:
On 8/23/19 9:00 PM, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case.
Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of?
Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues.
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases.
Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat.
There are plenty of examples of Red Hat doing crazy/experimental
things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is...
btrfs still doesn't give me the warm fuzzies and I also think this is a bigger issue than other features simply because user data is at stake. We do need to consider that the failure case is not "I can't do X" but "my precious data which I have been trying to snapshot is now inaccessible" in a way that's even worse than say rpm database corruption. Even if it is in the advanced partitioning or not the default, we can still end up with people clicking in because they read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and btrfs as release criteria. I think you are correct that there's enough community support to justify keeping btrfs around at least in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident about that if we could have a file system developer on the btrfs alias. I'm glad to hear the btrfs upstream community has been receptive to bugs but it's still much easier to make things happen if we have contributors who are active in the Fedora community, especially if we want the advanced features that btrfs has (which is why people want it anyway). So, who would you suggest to work with us in Fedora?
Thanks, Laura
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
On 8/26/19 11:39 PM, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott labbott@redhat.com wrote:
On 8/23/19 9:00 PM, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case.
Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of?
Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues.
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases.
Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat.
There are plenty of examples of Red Hat doing crazy/experimental
things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is...
btrfs still doesn't give me the warm fuzzies and I also think this is a bigger issue than other features simply because user data is at stake. We do need to consider that the failure case is not "I can't do X" but "my precious data which I have been trying to snapshot is now inaccessible" in a way that's even worse than say rpm database corruption. Even if it is in the advanced partitioning or not the default, we can still end up with people clicking in because they read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and btrfs as release criteria. I think you are correct that there's enough community support to justify keeping btrfs around at least in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident about that if we could have a file system developer on the btrfs alias. I'm glad to hear the btrfs upstream community has been receptive to bugs but it's still much easier to make things happen if we have contributors who are active in the Fedora community, especially if we want the advanced features that btrfs has (which is why people want it anyway). So, who would you suggest to work with us in Fedora?
You can always CC me, if I get an email from you or anybody else I recognize from the fedora kernel team I'm going to pay attention to it.
Facebook runs more btrfs file systems than Fedora has installs, so we're pretty happy with how it works stability wise. That being said we're slightly more fault tolerant than most users. If you guys are hitting problems chances are we'll hit them eventually as well, so it makes sense for us to be on top of them.
I agree it would be better if somebody inside Fedora was able to help out, but again I'm only an email away. Thanks,
Josef
On 8/28/19 1:58 PM, Josef Bacik wrote:
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
On 8/26/19 11:39 PM, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott labbott@redhat.com wrote:
On 8/23/19 9:00 PM, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case.
Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of?
Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues.
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases.
Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat.
There are plenty of examples of Red Hat doing crazy/experimental
things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is...
btrfs still doesn't give me the warm fuzzies and I also think this is a bigger issue than other features simply because user data is at stake. We do need to consider that the failure case is not "I can't do X" but "my precious data which I have been trying to snapshot is now inaccessible" in a way that's even worse than say rpm database corruption. Even if it is in the advanced partitioning or not the default, we can still end up with people clicking in because they read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and btrfs as release criteria. I think you are correct that there's enough community support to justify keeping btrfs around at least in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident about that if we could have a file system developer on the btrfs alias. I'm glad to hear the btrfs upstream community has been receptive to bugs but it's still much easier to make things happen if we have contributors who are active in the Fedora community, especially if we want the advanced features that btrfs has (which is why people want it anyway). So, who would you suggest to work with us in Fedora?
You can always CC me, if I get an email from you or anybody else I recognize from the fedora kernel team I'm going to pay attention to it.
Facebook runs more btrfs file systems than Fedora has installs, so we're pretty happy with how it works stability wise. That being said we're slightly more fault tolerant than most users. If you guys are hitting problems chances are we'll hit them eventually as well, so it makes sense for us to be on top of them.
I agree it would be better if somebody inside Fedora was able to help out, but again I'm only an email away. Thanks,
So it appears you are on the btrfs alias already:
fedora-kernel-btrfs: fs-maint@redhat.com,josef@toxicpanda.com,bugzilla@colorremedies.com
This technically meets the requirements if you are willing to stay on this alias and (continue) to help with requests as needed. I would feel more confident if we had a few more people involved as well. Even better would be proactively going through the bugzillas to help find the btrfs ones.
Fedora is ultimately a community project and as far as the kernel goes, there does seem to be enough interest from the community if Josef et. al. are willing to be involved. I hope to see this continue.
Thanks, Laura
On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote:
On 8/28/19 1:58 PM, Josef Bacik wrote:
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
On 8/26/19 11:39 PM, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott labbott@redhat.com wrote:
On 8/23/19 9:00 PM, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
> So, there was recently a Thing where btrfs installs were broken, and > this got accepted as a release blocker: > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388
Summary: This bug was introduced and discovered in linux-next, it started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug resulted in a somewhat transient deadlock which caused installs to hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly how much faster should it have been fixed in order to avoid worrying about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@ 7/25 22:33 utc bug was first reported in Fedora bugzilla 7/26 19:20 utc I confirmed upstream's patch related to this bug with upstream and updated the Fedora bug 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option in the installer and if there are bugs they Beta blocking, how could or should this have been fixed sooner? What about the handling should have been different?
That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case.
Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of?
Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues.
I note here that ext2 and ext3 are offered as file systems in Custom/Advanced partitioning and in this sense have parity with Btrfs. If this same bug occurred in ext2 or ext3 would or should that cause discussion to drop them from the installer, even if the bug were fixed within 24 hours of discovery and patch? What about vfat? That's literally the only truly required filesystem that must work, for the most commonly supported hardware so it can't be dropped, we'd just be stuck until it got fixed. That work would have to be done upstream, yes?
I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases.
Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat.
There are plenty of examples of Red Hat doing crazy/experimental
things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is...
btrfs still doesn't give me the warm fuzzies and I also think this is a bigger issue than other features simply because user data is at stake. We do need to consider that the failure case is not "I can't do X" but "my precious data which I have been trying to snapshot is now inaccessible" in a way that's even worse than say rpm database corruption. Even if it is in the advanced partitioning or not the default, we can still end up with people clicking in because they read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and btrfs as release criteria. I think you are correct that there's enough community support to justify keeping btrfs around at least in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident about that if we could have a file system developer on the btrfs alias. I'm glad to hear the btrfs upstream community has been receptive to bugs but it's still much easier to make things happen if we have contributors who are active in the Fedora community, especially if we want the advanced features that btrfs has (which is why people want it anyway). So, who would you suggest to work with us in Fedora?
You can always CC me, if I get an email from you or anybody else I recognize from the fedora kernel team I'm going to pay attention to it.
Facebook runs more btrfs file systems than Fedora has installs, so we're pretty happy with how it works stability wise. That being said we're slightly more fault tolerant than most users. If you guys are hitting problems chances are we'll hit them eventually as well, so it makes sense for us to be on top of them.
I agree it would be better if somebody inside Fedora was able to help out, but again I'm only an email away. Thanks,
So it appears you are on the btrfs alias already:
fedora-kernel-btrfs: fs-maint@redhat.com,josef@toxicpanda.com,bugzilla@colorremedies.com
This technically meets the requirements if you are willing to stay on this alias and (continue) to help with requests as needed. I would feel more confident if we had a few more people involved as well. Even better would be proactively going through the bugzillas to help find the btrfs ones.
Yeah that goes into a bucket that basically is ignored. The only time I'll peek in there is if somebody specifically pokes me, because generally speaking we hit the problems and fix them welllllll before Fedora users start to notice them.
If it'll make people feel more comfortable I'll start sticking my head in there every week or so and see if something needs my attention. Thanks,
Josef
On Wed, Aug 28, 2019 at 2:40 PM Josef Bacik josef@toxicpanda.com wrote:
On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote:
On 8/28/19 1:58 PM, Josef Bacik wrote:
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
On 8/26/19 11:39 PM, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott labbott@redhat.com wrote:
On 8/23/19 9:00 PM, Chris Murphy wrote: > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > adamwill@fedoraproject.org wrote: > > > So, there was recently a Thing where btrfs installs were broken, and > > this got accepted as a release blocker: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > Summary: This bug was introduced and discovered in linux-next, it > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug > resulted in a somewhat transient deadlock which caused installs to > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 > deletions (1/2 the insertions are comments). > > How remarkable or interesting is this bug? And in particular, exactly > how much faster should it have been fixed in order to avoid worrying > about it being a blocker bug? > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > 7/25 22:33 utc bug was first reported in Fedora bugzilla > 7/26 19:20 utc I confirmed upstream's patch related to this bug with > upstream and updated the Fedora bug > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug > > So in the context of status quo, where Btrfs is presented as an option > in the installer and if there are bugs they Beta blocking, how could > or should this have been fixed sooner? What about the handling should > have been different? >
That's a fair question. This bug actually represents how this _should_ work. The concern is that in the past we haven't seen a lot engagement in the past. Maybe today that has changed as demonstrated by this thread. I'm still concerned about having this be a blocker vs. just keeping it as an option, simply because a blocker stops the entire release and it can be a last minute scramble to get things fixed. This was the ideal case for a blocker bugs and I'm skeptical about all bugs going this well. If we had a few more people who were willing to be on the btrfs alias and do the work for blocker bugs it would be a much stronger case.
Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of?
Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues.
> I note here that ext2 and ext3 are offered as file systems in > Custom/Advanced partitioning and in this sense have parity with Btrfs. > If this same bug occurred in ext2 or ext3 would or should that cause > discussion to drop them from the installer, even if the bug were fixed > within 24 hours of discovery and patch? What about vfat? That's > literally the only truly required filesystem that must work, for the > most commonly supported hardware so it can't be dropped, we'd just be > stuck until it got fixed. That work would have to be done upstream, > yes? >
I don't think that's really a fair comparison. Just because options are presented doesn't mean all of them are equal. ext2/ext3 and vfat have been in development for much longer than btrfs and length of development is something that's particularly important for file system stability from talking with file system developers. It's not impossible for there to be bugs in ext4 for example (we've certainly seen them before) but btrfs is only now gaining overall stability and we're still more likely to see bugs, especially with custom setups where people are likely to find edge cases.
Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat.
There are plenty of examples of Red Hat doing crazy/experimental
things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is...
btrfs still doesn't give me the warm fuzzies and I also think this is a bigger issue than other features simply because user data is at stake. We do need to consider that the failure case is not "I can't do X" but "my precious data which I have been trying to snapshot is now inaccessible" in a way that's even worse than say rpm database corruption. Even if it is in the advanced partitioning or not the default, we can still end up with people clicking in because they read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and btrfs as release criteria. I think you are correct that there's enough community support to justify keeping btrfs around at least in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident about that if we could have a file system developer on the btrfs alias. I'm glad to hear the btrfs upstream community has been receptive to bugs but it's still much easier to make things happen if we have contributors who are active in the Fedora community, especially if we want the advanced features that btrfs has (which is why people want it anyway). So, who would you suggest to work with us in Fedora?
You can always CC me, if I get an email from you or anybody else I recognize from the fedora kernel team I'm going to pay attention to it.
Facebook runs more btrfs file systems than Fedora has installs, so we're pretty happy with how it works stability wise. That being said we're slightly more fault tolerant than most users. If you guys are hitting problems chances are we'll hit them eventually as well, so it makes sense for us to be on top of them.
I agree it would be better if somebody inside Fedora was able to help out, but again I'm only an email away. Thanks,
So it appears you are on the btrfs alias already:
fedora-kernel-btrfs: fs-maint@redhat.com,josef@toxicpanda.com,bugzilla@colorremedies.com
This technically meets the requirements if you are willing to stay on this alias and (continue) to help with requests as needed. I would feel more confident if we had a few more people involved as well. Even better would be proactively going through the bugzillas to help find the btrfs ones.
Yeah that goes into a bucket that basically is ignored. The only time I'll peek in there is if somebody specifically pokes me, because generally speaking we hit the problems and fix them welllllll before Fedora users start to notice them.
Fedora chugs along at the rate of daily upstream Linus snapshots. If you're hitting and fixing issues before Fedora users see them, I'm curious why Fedora users would ever see them.
Where does the lag come from? Are the fixes queued internally? Staged in an upstream subsystem tree? Is there a way for interested btrfs people to proactively just get those fixed in Fedora before users hit them?
josh
On Wed, Aug 28, 2019 at 03:01:16PM -0400, Josh Boyer wrote:
On Wed, Aug 28, 2019 at 2:40 PM Josef Bacik josef@toxicpanda.com wrote:
On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote:
On 8/28/19 1:58 PM, Josef Bacik wrote:
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
On 8/26/19 11:39 PM, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott labbott@redhat.com wrote: > > On 8/23/19 9:00 PM, Chris Murphy wrote: > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson > > adamwill@fedoraproject.org wrote: > > > > > So, there was recently a Thing where btrfs installs were broken, and > > > this got accepted as a release blocker: > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388 > > > > Summary: This bug was introduced and discovered in linux-next, it > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch > > appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug > > resulted in a somewhat transient deadlock which caused installs to > > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8 > > deletions (1/2 the insertions are comments). > > > > How remarkable or interesting is this bug? And in particular, exactly > > how much faster should it have been fixed in order to avoid worrying > > about it being a blocker bug? > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@ > > 7/25 22:33 utc bug was first reported in Fedora bugzilla > > 7/26 19:20 utc I confirmed upstream's patch related to this bug with > > upstream and updated the Fedora bug > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug > > > > So in the context of status quo, where Btrfs is presented as an option > > in the installer and if there are bugs they Beta blocking, how could > > or should this have been fixed sooner? What about the handling should > > have been different? > > > > That's a fair question. This bug actually represents how this _should_ > work. The concern is that in the past we haven't seen a lot engagement > in the past. Maybe today that has changed as demonstrated by this thread. > I'm still concerned about having this be a blocker vs. just keeping it > as an option, simply because a blocker stops the entire release and it > can be a last minute scramble to get things fixed. This was the ideal > case for a blocker bugs and I'm skeptical about all bugs going this well. > If we had a few more people who were willing to be on the btrfs alias and > do the work for blocker bugs it would be a much stronger case. >
Out of curiosity, how many such issues have we had in the past 2 years? I personally can't recall any monumental occasions where people were scrambling over *Btrfs* in Fedora. If anything, we continue to inherit the work that SUSE and Facebook are doing upstream as part of us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached out to btrfs folks in Fedora? Chris and myself are the current ones, but there have been others in the past. Both of us are subscribed to the linux-btrfs mailing list, and Chris has a decent rapport with most of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't have any for the majority of filesystems Fedora supports, only XFS. Is there some kind of problem with communicating with the upstream kernel developers about Fedora bugs that I'm not aware of?
Again, it's about length of overall development. ext and XFS have a much longer history in general which is something that's important for file system stability in general. It's also a bit of a catch-22 where the rate of btrfs use in Fedora is so low we don't actually see issues.
> > I note here that ext2 and ext3 are offered as file systems in > > Custom/Advanced partitioning and in this sense have parity with Btrfs. > > If this same bug occurred in ext2 or ext3 would or should that cause > > discussion to drop them from the installer, even if the bug were fixed > > within 24 hours of discovery and patch? What about vfat? That's > > literally the only truly required filesystem that must work, for the > > most commonly supported hardware so it can't be dropped, we'd just be > > stuck until it got fixed. That work would have to be done upstream, > > yes? > > > > I don't think that's really a fair comparison. Just because options > are presented doesn't mean all of them are equal. ext2/ext3 and vfat > have been in development for much longer than btrfs and length of development > is something that's particularly important for file system stability > from talking with file system developers. It's not impossible for there > to be bugs in ext4 for example (we've certainly seen them before) but > btrfs is only now gaining overall stability and we're still more likely to see > bugs, especially with custom setups where people are likely to find > edge cases. >
Nope. We can totally use this because LVM has not existed as long (we use LVM + filesystem by default, not plain partitions), and we still encounter quirks with things like thinp LVM combined with these filesystems. OverlayFS is mostly hot garbage (kernel people know it, container people know it, filesystem people know it, etc.), and yet we continue to try to use it in more places. Stratis is in an odd state of limbo now, since its main developer and advocate left Red Hat. > There are plenty of examples of Red Hat doing crazy/experimental things... I'd like to think Red Hat isn't supposed to be special here, but in this realm, it seems like it is...
btrfs still doesn't give me the warm fuzzies and I also think this is a bigger issue than other features simply because user data is at stake. We do need to consider that the failure case is not "I can't do X" but "my precious data which I have been trying to snapshot is now inaccessible" in a way that's even worse than say rpm database corruption. Even if it is in the advanced partitioning or not the default, we can still end up with people clicking in because they read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and btrfs as release criteria. I think you are correct that there's enough community support to justify keeping btrfs around at least in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident about that if we could have a file system developer on the btrfs alias. I'm glad to hear the btrfs upstream community has been receptive to bugs but it's still much easier to make things happen if we have contributors who are active in the Fedora community, especially if we want the advanced features that btrfs has (which is why people want it anyway). So, who would you suggest to work with us in Fedora?
You can always CC me, if I get an email from you or anybody else I recognize from the fedora kernel team I'm going to pay attention to it.
Facebook runs more btrfs file systems than Fedora has installs, so we're pretty happy with how it works stability wise. That being said we're slightly more fault tolerant than most users. If you guys are hitting problems chances are we'll hit them eventually as well, so it makes sense for us to be on top of them.
I agree it would be better if somebody inside Fedora was able to help out, but again I'm only an email away. Thanks,
So it appears you are on the btrfs alias already:
fedora-kernel-btrfs: fs-maint@redhat.com,josef@toxicpanda.com,bugzilla@colorremedies.com
This technically meets the requirements if you are willing to stay on this alias and (continue) to help with requests as needed. I would feel more confident if we had a few more people involved as well. Even better would be proactively going through the bugzillas to help find the btrfs ones.
Yeah that goes into a bucket that basically is ignored. The only time I'll peek in there is if somebody specifically pokes me, because generally speaking we hit the problems and fix them welllllll before Fedora users start to notice them.
Fedora chugs along at the rate of daily upstream Linus snapshots. If you're hitting and fixing issues before Fedora users see them, I'm curious why Fedora users would ever see them.
Where does the lag come from? Are the fixes queued internally? Staged in an upstream subsystem tree? Is there a way for interested btrfs people to proactively just get those fixed in Fedora before users hit them?
For this particular example we saw the problem in testing and had a patch on the mailinglist before you hit the problem. It was in a tree and sent to Linus, and was merged the day after the bugzilla was reported. So yes before users see them, unless they are subscribed to the daily snapshots, which I assume is just for testing right? Or were you guys going to ship 5.3-rc0?
On one hand I understand all of the consternation around making btrfs bugs blockers for Fedora, but on the other hand it seems a bit silly to be having this conversation at all based on hitting a bug that went into the merge window and then was fixed before rc1 was even cut. Thanks,
Josef
On Wed, Aug 28, 2019 at 1:07 PM Josef Bacik josef@toxicpanda.com wrote:
On Wed, Aug 28, 2019 at 03:01:16PM -0400, Josh Boyer wrote:
Fedora chugs along at the rate of daily upstream Linus snapshots. If you're hitting and fixing issues before Fedora users see them, I'm curious why Fedora users would ever see them.
Where does the lag come from? Are the fixes queued internally? Staged in an upstream subsystem tree? Is there a way for interested btrfs people to proactively just get those fixed in Fedora before users hit them?
For this particular example we saw the problem in testing and had a patch on the mailinglist before you hit the problem. It was in a tree and sent to Linus, and was merged the day after the bugzilla was reported. So yes before users see them, unless they are subscribed to the daily snapshots, which I assume is just for testing right? Or were you guys going to ship 5.3-rc0?
On one hand I understand all of the consternation around making btrfs bugs blockers for Fedora, but on the other hand it seems a bit silly to be having this conversation at all based on hitting a bug that went into the merge window and then was fixed before rc1 was even cut. Thanks,
It is silly, if it's really safe to say that Btrfs won't be the default file system in any release blocking deliverable. Having blocker status was always a means to that end. But right now it's maybe three (?) automated tests that depend on Btrfs working. If Fedora Workstation defaulted to Btrfs, that's dozens or even hundreds of tests? Adam?
Bug fix was merged in rc2. The patch on linux-btrfs@
25 Jul 2019 11:27:29 +0300 which is just before kernel-5.3.0-0.rc1.git3.1.fc31, 2019-07-25 21:01:20
Plausibly it was in all rc0 and rc1 kernels. What if this deadlock were happening in ext4 for all rc0 and rc1 kernels? What questions get asked? How did the bug not get caught by xfstests before it got into linux-next? Does anyone pop a gasket on lkml? Is this just a weird sometimes it happens rarely kind of thing?
I don't know the exact nature of the bug, which goes to the kernel team's point that someone who does know needs to tell them the autopsy summary so they can really assess the default fs question.
And another question for QA. If it were Btrfs by default for Workstation, would you just convert all the tests that rely only on ext4 now to Btrfs? Or duplicate those tests so you can run them in parallel? How much more testing is that and what's the impact on time and resources?
-- Chris Murphy
On Wed, 2019-08-28 at 15:59 -0600, Chris Murphy wrote:
On one hand I understand all of the consternation around making btrfs bugs blockers for Fedora, but on the other hand it seems a bit silly to be having this conversation at all based on hitting a bug that went into the merge window and then was fixed before rc1 was even cut. Thanks,
It is silly, if it's really safe to say that Btrfs won't be the default file system in any release blocking deliverable. Having blocker status was always a means to that end. But right now it's maybe three (?) automated tests that depend on Btrfs working. If Fedora Workstation defaulted to Btrfs, that's dozens or even hundreds of tests? Adam?
Well, yeah, but they all use the same filesystem layout. So either they all work or they're all broken, so far as any filesystem bug is concerned.
But anyway, I've said this once already, but just to say it again: this discussion isn't happening because of any specific concern related to the bug that was linked in the original mail. The bug simply alerted the kernel team to the fact that we currently block on btrfs, which is something they thought we shouldn't do *anyway*. It's nothing to do with that particular bug at all. They just hadn't yelled about it before because, until an actual blocker bug showed up, they weren't aware of it.
And another question for QA. If it were Btrfs by default for Workstation, would you just convert all the tests that rely only on ext4 now to Btrfs? Or duplicate those tests so you can run them in parallel? How much more testing is that and what's the impact on time and resources?
I mean, there wouldn't be any 'conversion'. That's just sort of not how the tests work. The tests work by running the installer and clicking stuff (talking about the openQA tests, here). If the default filesystem is different most of the tests will run the same - they'll just happen to be using a different default filesystem.
I wouldn't duplicate all the tests and run them all on different filesystems, no, there isn't really much justification for that. We already theoretically block on about eight different filesystems, we don't re-run every single blocking test on each of those filesystems. If you start trying to do that kind of combinatorial testing you're going to blow up quite rapidly - do we do every single test on each blocking desktop on each blocking arch with each blocking filesystem on both BIOS and UEFI with three different graphics cards? By the time you multiply all those factors by each other you're probably already looking at several billion tests, and I haven't even thrown in another half-dozen factors I easily could throw in.
On Fri, Aug 23, 2019 at 9:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hey folks!
So, there was recently a Thing where btrfs installs were broken, and this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
The bug was fixed, so that's fine, but along the way, Laura said this:
"I'm strongly against anything with btrfs being a blocker. If that's in the criteria I think we should see about removing btrfs simply because we don't have the resources to actually deal with btrfs besides reporting bugs upstream."
and Justin followed up with:
"Agreed, btrfs has been a gamble pretty much always. See previous discussion around proposals to make btrfs default. Ext4 and xfs should be the only release blocking."
So, that's the whole kernel team 'strongly against' blocking on btrfs. Which means we should talk about not doing that any more!
This is a bit complicated, though, because of how the Final criteria are phrased. Basic does not include btrfs at all, and Beta includes a laundry list we can just remove btrfs from:
"When using both the installer-native and the blivet-gui-based custom partitioning flow, the installer must be able to:
- Correctly interpret, and modify as described below, any disk with a
valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions
- Create mount points backed by ext4 partitions, LVM volumes or btrfs
volumes, or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions ..."
so those two are easy. However, the Final criterion is not laundry list-style. The relevant Final criterion is this:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration."
with a somewhat apologetic explanatory footnote:
"Wait, what? Yeah, we know. This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing. We are trying to write more specific criteria covering this area, but it's not easy. Patches welcome, as the kids say..."
so as the footnote says, the rule is basically "if the installer lets you do it, it ought to work". It seems a bit awkward to craft an exception for btrfs from that. I mean, technically it's easy:
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration, except btrfs."
but that's odd. Why is btrfs, alone, an exception? It kinda goes against the fundamental idea of the criterion: that we stand behind everything the UI offers.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I'm bringing this to anaconda, kernel and test teams initially to kick around; if we agree on an approach we should then probably loop in devel@ for wider review, unless the choice is 1).
This thread has diverged wildly into a lot of delightful invective-slinging at my team. In the interest of getting back to the original topic at hand: I would be in support of three -- five.
Thanks for bringing this topic up, Adam, and for presenting this list of options.
Thanks for any thoughts, folks!
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Tue, Aug 27, 2019 at 7:49 AM Samantha Bueno sbueno@redhat.com wrote:
On Fri, Aug 23, 2019 at 9:17 PM Adam Williamson adamwill@fedoraproject.org wrote:
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's weird
- Rewrite the criterion entirely
- Keep btrfs support in the installer (and blivet-gui) but hide it as
we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
This thread has diverged wildly into a lot of delightful invective-slinging at my team. In the interest of getting back to the original topic at hand: I would be in support of three -- five.
This clearly means Btrfs related bugs in Fedora's Anaconda will not block release, and by extension Btrfs could not be the default file system either.
"Btrfs is not a priority" does not at all answer the central question of what level of support and resources are expected to exist in the Fedora community to maintain Btrfs in both the bug blocking, and default file system contexts. I think Laura Abbott's emails on the kernel side are quite clear ground rules for serious Btrfs bugs to remain blocker worthy, along with a path to discuss any additional expectations for Btrfs to be a default file system in some capacity.
Unqualified statements like "it is safe to say btrfs will not be the default" and "Btrfs is not a priority" and this 'zero chance Btrfs' comment [1] are completely compatible with assuming that no matter how much work is done by others, those things will not appear in Anaconda because the decision has been made, it is a fait accompli.
It is very important to have clarity exactly to what degree Btrfs is not a priority. If the Anaconda team cannot state the terms and expectations for the #1 option in Adam's list, that's troubling. Again, I think it's completely fine if Red Hat Anaconda folks are disinterested in Btrfs, but it's a very different thing if there's resistance to it, and I'm getting a lot of language that is compatible with resisting Btrfs no matter what.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1717728#c10
The language being used by Anaconda team suggests the "Btrfs is not a priority" and should be unsupported, is a decision that has already happened. This discussion, in this thread, is about how to handle that decision in UI/Ux. When and where did this decision get made? How do outside contributors get the information they need to know their efforts are worthwhile? I have hundreds of hours invested in Anaconda testing, perhaps 1/2 not related to Btrfs, over ~8 years. I would like answers to these questions.
On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote:
Hey folks!
Hi Adam! Thanks for bringing this up again.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I like option 3 most. The current criteria have always seemed, to me, too vague. I'd be happy to help hash out the details if/when it happens.
Option 4 is also somewhat appealing.
David
I'm bringing this to anaconda, kernel and test teams initially to kick around; if we agree on an approach we should then probably loop in devel@ for wider review, unless the choice is 1).
Thanks for any thoughts, folks!
On Tue, 2019-09-03 at 13:38 -0400, David Lehman wrote:
On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote:
Hey folks!
Hi Adam! Thanks for bringing this up again.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I like option 3 most. The current criteria have always seemed, to me, too vague. I'd be happy to help hash out the details if/when it happens.
Thanks for the offer.
So aside from the 'fun' of drafting very specific rules, my concern with #3 is we would then potentially be shipping an installer that presents things as roughly equal choices which are not in fact equally supported. You can pick 'btrfs' or 'ext4' from the dropdown...but one of those we commit to making sure is working, one of them we don't.
That to me is concerning; in this scenario I'd prefer we indicate somehow, somewhere, that all the choices are not equally guaranteed to be reliable. WDYT?
On Fri, Sep 6, 2019 at 3:02 AM Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2019-09-03 at 13:38 -0400, David Lehman wrote:
On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote:
Hey folks!
Hi Adam! Thanks for bringing this up again.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I like option 3 most. The current criteria have always seemed, to me, too vague. I'd be happy to help hash out the details if/when it happens.
Thanks for the offer.
So aside from the 'fun' of drafting very specific rules, my concern with #3 is we would then potentially be shipping an installer that presents things as roughly equal choices which are not in fact equally supported. You can pick 'btrfs' or 'ext4' from the dropdown...but one of those we commit to making sure is working, one of them we don't.
That to me is concerning; in this scenario I'd prefer we indicate somehow, somewhere, that all the choices are not equally guaranteed to be reliable. WDYT?
Hi Adam,
I think that the best option is to add a new storage validation check that will report a warning if a user wants to use a file system that is not recommended by the installed product. The list of recommended file systems would be provided by the Anaconda configuration files, so products and variants could override it.
We already show warnings with recommendations, for example for too small root partition or missing swap. The storage validation checks are run for every type of partitioning, results are logged and warnings have to be waved by the user in the interactive mode.
Vendy
-- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/anaconda-devel-list
On Mon, 2019-09-09 at 15:19 +0200, Vendula Poncova wrote:
On Fri, Sep 6, 2019 at 3:02 AM Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2019-09-03 at 13:38 -0400, David Lehman wrote:
On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote:
Hey folks!
Hi Adam! Thanks for bringing this up again.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I like option 3 most. The current criteria have always seemed, to me, too vague. I'd be happy to help hash out the details if/when it happens.
Thanks for the offer.
So aside from the 'fun' of drafting very specific rules, my concern with #3 is we would then potentially be shipping an installer that presents things as roughly equal choices which are not in fact equally supported. You can pick 'btrfs' or 'ext4' from the dropdown...but one of those we commit to making sure is working, one of them we don't.
That to me is concerning; in this scenario I'd prefer we indicate somehow, somewhere, that all the choices are not equally guaranteed to be reliable. WDYT?
Hi Adam,
I think that the best option is to add a new storage validation check that will report a warning if a user wants to use a file system that is not recommended by the installed product. The list of recommended file systems would be provided by the Anaconda configuration files, so products and variants could override it.
We already show warnings with recommendations, for example for too small root partition or missing swap. The storage validation checks are run for every type of partitioning, results are logged and warnings have to be waved by the user in the interactive mode.
This could work, however it's what I'd call "backwards UI" (I'm sure there's a real term for it, but I'm not an expert so I don't know what it is) - it's a pattern I find annoying because it makes you make choices *before you know what the constraints are*, then tells you you broke the mystery rules you didn't know about. ;) It's like password systems that just say 'enter a password', then you enter one, and *then* it says 'oh BTW it's meant to have more than 8 characters', so you enter one with more than 8 characters and it says 'oh yeah and one of them has to be upper case', so you upper case one, then it says 'oh yeah and one has to be a special character', then you shoot the PC and go herd yaks...:P
On Mon, Sep 9, 2019 at 5:10 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2019-09-09 at 15:19 +0200, Vendula Poncova wrote:
On Fri, Sep 6, 2019 at 3:02 AM Adam Williamson <
adamwill@fedoraproject.org>
wrote:
On Tue, 2019-09-03 at 13:38 -0400, David Lehman wrote:
On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote:
Hey folks!
Hi Adam! Thanks for bringing this up again.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I like option 3 most. The current criteria have always seemed, to me, too vague. I'd be happy to help hash out the details if/when it happens.
Thanks for the offer.
So aside from the 'fun' of drafting very specific rules, my concern with #3 is we would then potentially be shipping an installer that presents things as roughly equal choices which are not in fact equally supported. You can pick 'btrfs' or 'ext4' from the dropdown...but one of those we commit to making sure is working, one of them we don't.
That to me is concerning; in this scenario I'd prefer we indicate somehow, somewhere, that all the choices are not equally guaranteed to be reliable. WDYT?
Hi Adam,
I think that the best option is to add a new storage validation check
that
will report a warning if a user wants to use a file system that is not recommended by the installed product. The list of recommended file
systems
would be provided by the Anaconda configuration files, so products and variants could override it.
We already show warnings with recommendations, for example for too small root partition or missing swap. The storage validation checks are run for every type of partitioning, results are logged and warnings have to be waved by the user in the interactive mode.
This could work, however it's what I'd call "backwards UI" (I'm sure there's a real term for it, but I'm not an expert so I don't know what it is) - it's a pattern I find annoying because it makes you make choices *before you know what the constraints are*, then tells you you broke the mystery rules you didn't know about. ;) It's like password systems that just say 'enter a password', then you enter one, and *then* it says 'oh BTW it's meant to have more than 8 characters', so you enter one with more than 8 characters and it says 'oh yeah and one of them has to be upper case', so you upper case one, then it says 'oh yeah and one has to be a special character', then you shoot the PC and go herd yaks...:P
Warnings inform you about potential risks of your choices, but you are allowed to ignore them unlike the error messages you are talking about. It is like, when you enter a password with non-ASCII characters, the password system can say 'be careful, you might not be able to switch a keyboard layout when typing it', but you can still go ahead and use your password anyway.
I would expect that users in general know what they are doing (I thought that is the premise of the Custom Partitioning Spoke), so I understand the warning about unsupported file systems as a disclaimer.
On Thu, 2019-09-05 at 18:01 -0700, Adam Williamson wrote:
On Tue, 2019-09-03 at 13:38 -0400, David Lehman wrote:
On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote:
Hey folks!
Hi Adam! Thanks for bringing this up again.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I like option 3 most. The current criteria have always seemed, to me, too vague. I'd be happy to help hash out the details if/when it happens.
Thanks for the offer.
So aside from the 'fun' of drafting very specific rules, my concern with #3 is we would then potentially be shipping an installer that presents things as roughly equal choices which are not in fact equally supported. You can pick 'btrfs' or 'ext4' from the dropdown...but one of those we commit to making sure is working, one of them we don't.
That to me is concerning; in this scenario I'd prefer we indicate somehow, somewhere, that all the choices are not equally guaranteed to be reliable. WDYT?
Two off-the-cuff ideas:
1. every fs (and device?) type in the combo/dropdown has "(Supported)" or "(Unsupported)" postfix, eg: "xfs (Supported)" or "BTRFS (Unsupported)" 2. every unsupported fs type has a corresponding kernel arg, eg: "inst.btrfs" or "inst.fs.btrfs" which is required to get that unsupported fs type in the GUI list
I could live with either, personally.
David
On Mon, Sep 09, 2019 at 01:43:00PM -0400, David Lehman wrote:
On Thu, 2019-09-05 at 18:01 -0700, Adam Williamson wrote:
On Tue, 2019-09-03 at 13:38 -0400, David Lehman wrote:
On Fri, 2019-08-23 at 12:16 -0700, Adam Williamson wrote:
Hey folks!
Hi Adam! Thanks for bringing this up again.
So...what should we do? Here are the options as I see 'em:
- Keep supporting btrfs
- Just modify the criterion with a btrfs exception, even if it's
weird 3. Rewrite the criterion entirely 4. Keep btrfs support in the installer (and blivet-gui) but hide it as we used to - require a special boot argument for it to be visible 5. Drop btrfs support from the installer
I like option 3 most. The current criteria have always seemed, to me, too vague. I'd be happy to help hash out the details if/when it happens.
Thanks for the offer.
So aside from the 'fun' of drafting very specific rules, my concern with #3 is we would then potentially be shipping an installer that presents things as roughly equal choices which are not in fact equally supported. You can pick 'btrfs' or 'ext4' from the dropdown...but one of those we commit to making sure is working, one of them we don't.
That to me is concerning; in this scenario I'd prefer we indicate somehow, somewhere, that all the choices are not equally guaranteed to be reliable. WDYT?
Two off-the-cuff ideas:
- every fs (and device?) type in the combo/dropdown has "(Supported)"
or "(Unsupported)" postfix, eg: "xfs (Supported)" or "BTRFS (Unsupported)" 2. every unsupported fs type has a corresponding kernel arg, eg: "inst.btrfs" or "inst.fs.btrfs" which is required to get that unsupported fs type in the GUI list
I could live with either, personally.
Why bother hiding it at all? I get that there's no btrfs expertise at Red Hat, but there's not any reason to indicate that it's bad for the user to choose it. Suse ships it, all of Facebook is on it, it's not like we're talking about hfs+ here or anything. I get the discussions around wether it's a release blocker, I don't really understand why the installer needs to be changed? Thanks,
Josef
anaconda-devel@lists.stg.fedoraproject.org