Hello.
Recently, a couple hundred packages were retired from rawhide (Fedora 31 at that time) based on the Fedora Failed to Build From Source Policy [1]. From various reactions over several threads it seems this policy is not ideal. This is an attempt to collect feedback and make the policy better serve Fedora's needs.
Fedora has a policy for a long time that can be simplified as:
1. Mass rebuild for Fedora N happens by releng 2. Packages that fail to build get open bugzillas by releng 3. Packages with NEW bugizllas are orphaned after 8 weeks with weekly reminders by releng 4. A week before Fedora N+1 branching any packages which still have open Fedora N FTBFS bug are retired by releng
However, 3 or 4 haven't happened since Fedora ~26, because the automation was not working any more for various reasons I don't understand.
The policy was then updated by FESCo to allow anybody to step up and do 3. manually. However it needs 2 e-mails to devel directed at the package owners and that may be understood as a personal attack by some. So nobody ever did that but me (twice) IIRC (and I got my share of shame for that).
Currently, the FTBFS retirement is problematic due to various things:
1) Bugzilla spam: Maintainers are spammed weekly by releng, some of them find that annoying and simply switch the bug to ASSIGNED to make it stop.
The problem is, how do we both keep notifying the maintainers that action is needed and not spam them with stuff that will make them filter all Fedora e-mails to /dev/null.
2) Retirement out of the blue. When releng executes 4., packagers that stopped the bugzilla spam by setting the bug to ASSIGNED are suddenly surprised the package was retired. OTOH arguably they made a deliberate action to stop the notifications. However, most importantly, any dependent packages were not notified at all, which is **not fair**.
This state is broken mostly because there is no automatic orphaning of packages that have NEW bugzillas and there is no orphaning at all for packages where the bugzillas are ASSIGNED for months. For the second bit, everything indicates that packagers are aware of the problem and will fix the bug in time before 4., but they don't and things blow up.
3) Questionable importance of the FTBFS bug.
Repeatedly, it has been stated by some, the FTBFS bug is not important and we should not retire packages at all based on the fact that they "simply" fail to build. I personally don't agree with this for various reasons, but I can imagine a situation, where it is reasonable to say so and delay the problem. Obvious argument is: Better to have 1 package nonbuildable than have 100 packages nonisntallable. So we need a way to opt-out from the retirement, however simply setting the bugzilla to ASSIGNED is not it, because we will end up with packages last built 6 years ago (and I believe this is not what we want).
I'm starting this thread to collect the ideas about how to make this better. If you see more problems, share them. I promise I'll do my best to make the policy work better for both Fedora and the people who make Fedora.
[1] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_...
* Miro Hrončok:
Recently, a couple hundred packages were retired from rawhide (Fedora 31 at that time) based on the Fedora Failed to Build From Source Policy [1]. From various reactions over several threads it seems this policy is not ideal. This is an attempt to collect feedback and make the policy better serve Fedora's needs.
Debian treats FTBFS bugs as release-critical. They either have to be fixed, or the package gets removed from the release. However, this is not an automated process.
I wonder if something similar could work for Fedora: The package would remain available in rawhide, but would be removed from the release composes.
In the end, someone has to fix the packages eventually, and the package maintainers are probably best qualified to deal with that. If they lack the resources for that, it points to a much more significant problem that needs solving separately, I think.
Thanks, Florian
"FW" == Florian Weimer fweimer@redhat.com writes:
FW> Debian treats FTBFS bugs as release-critical. They either have to FW> be fixed, or the package gets removed from the release. However, FW> this is not an automated process.
Of course, Debian works on a slightly different release schedule, so it's not exactly a direct comparison.
FW> I wonder if something similar could work for Fedora: The package FW> would remain available in rawhide, but would be removed from the FW> release composes.
That's an interesting option, I suppose. In part I think it depends on just why some people have been upset over the recent orphaning. Is it the removal from the distribution, the shock of having the project say "we don't want this package any longer", the fact that user's won't be able to access the package any longer, the annoyance with process for getting the package into the distribution if it's fixed, or something else? (Certainly those aren't mutually exclusive and the true answer is more complicated and differs between people.)
Technical solutions to some of these are possible, though I don't know how feasible they would be. Procedural solutions (such as making it easier for such packages to get back into the distribution) could also help.
FW> In the end, someone has to fix the packages eventually, and the FW> package maintainers are probably best qualified to deal with that. FW> If they lack the resources for that, it points to a much more FW> significant problem that needs solving separately, I think.
Yes, this is fundamental.
- J<
Dne 14. 08. 19 v 14:23 Miro Hrončok napsal(a):
Hello.
Recently, a couple hundred packages were retired from rawhide (Fedora 31 at that time) based on the Fedora Failed to Build From Source Policy [1]. From various reactions over several threads it seems this policy is not ideal. This is an attempt to collect feedback and make the policy better serve Fedora's needs.
Fedora has a policy for a long time that can be simplified as:
1. Mass rebuild for Fedora N happens by releng 2. Packages that fail to build get open bugzillas by releng 3. Packages with NEW bugizllas are orphaned after 8 weeks with weekly reminders by releng
I think it would be probably enough to stop here. Orphaned packages gets garbage collected ATM. The step 4 was a bit unexpected for packages with bugs in ASSIGNED state especially.
Also the timing with mass rebuild and shifting packages from Rawhide to F31 is unfortunate. I saw a lot of packages, which were reported as FTBFS in BZ, then they were retired but later the bugs were moved from Rawhide to F31, that was strange.
Vít
4. A week before Fedora N+1 branching any packages which still have open Fedora N FTBFS bug are retired by releng
However, 3 or 4 haven't happened since Fedora ~26, because the automation was not working any more for various reasons I don't understand.
The policy was then updated by FESCo to allow anybody to step up and do 3. manually. However it needs 2 e-mails to devel directed at the package owners and that may be understood as a personal attack by some. So nobody ever did that but me (twice) IIRC (and I got my share of shame for that).
Currently, the FTBFS retirement is problematic due to various things:
- Bugzilla spam: Maintainers are spammed weekly by releng, some of
them find that annoying and simply switch the bug to ASSIGNED to make it stop.
The problem is, how do we both keep notifying the maintainers that action is needed and not spam them with stuff that will make them filter all Fedora e-mails to /dev/null.
- Retirement out of the blue. When releng executes 4., packagers that
stopped the bugzilla spam by setting the bug to ASSIGNED are suddenly surprised the package was retired. OTOH arguably they made a deliberate action to stop the notifications. However, most importantly, any dependent packages were not notified at all, which is **not fair**.
This state is broken mostly because there is no automatic orphaning of packages that have NEW bugzillas and there is no orphaning at all for packages where the bugzillas are ASSIGNED for months. For the second bit, everything indicates that packagers are aware of the problem and will fix the bug in time before 4., but they don't and things blow up.
- Questionable importance of the FTBFS bug.
Repeatedly, it has been stated by some, the FTBFS bug is not important and we should not retire packages at all based on the fact that they "simply" fail to build. I personally don't agree with this for various reasons, but I can imagine a situation, where it is reasonable to say so and delay the problem. Obvious argument is: Better to have 1 package nonbuildable than have 100 packages nonisntallable. So we need a way to opt-out from the retirement, however simply setting the bugzilla to ASSIGNED is not it, because we will end up with packages last built 6 years ago (and I believe this is not what we want).
I'm starting this thread to collect the ideas about how to make this better. If you see more problems, share them. I promise I'll do my best to make the policy work better for both Fedora and the people who make Fedora.
[1] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_...
On 14. 08. 19 14:55, Vít Ondruch wrote:
I think it would be probably enough to stop here. Orphaned packages gets garbage collected ATM. The step 4 was a bit unexpected for packages with bugs in ASSIGNED state especially.
If we stop here, the current "setting to ASSIGNED to stop this" remains a problem.
Also the timing with mass rebuild and shifting packages from Rawhide to F31 is unfortunate. I saw a lot of packages, which were reported as FTBFS in BZ, then they were retired but later the bugs were moved from Rawhide to F31, that was strange.
IMHO we should probably do this after branching and in rawhide only, to avoid breakage few weeks before the beta freeze.
Dne 14. 08. 19 v 15:20 Miro Hrončok napsal(a):
On 14. 08. 19 14:55, Vít Ondruch wrote:
I think it would be probably enough to stop here. Orphaned packages gets garbage collected ATM. The step 4 was a bit unexpected for packages with bugs in ASSIGNED state especially.
If we stop here, the current "setting to ASSIGNED to stop this" remains a problem.
It depends. I agree that this might prevent some packages from removing immediately. OTOH, the FTBFS package is reported with every mass rebuild, i.e. every 6 months. So if something changes in these 6+ months, the new ticket will eventually stay in NEW and the package will be orphaned and later garbage collected ...
Vít
"MH" == Miro Hrončok mhroncok@redhat.com writes:
MH> If we stop here, the current "setting to ASSIGNED to stop this" MH> remains a problem.
Let's think about why this is perceived as a problem. The maintainer has performed an affirmative act that shows they noticed. Can't we just accept that as some statement of intent and stop bugging them at that point? Further emails have utility only as periodic reminders, and experience has shown that we can't predict whether those would be perceived positively or negatively.
Certainly the _real_ problem here (that the packages fail to build) isn't solved by continuing to send bug spam mail. And similarly, we should spend time to evaluate why that is perceived as a problem.
If a package is installable and works, certainly it meets some acceptability criteria for packages in the distribution and fails others. So let's list a few (not a comprehensive list, I'm sure):
1. Can end users install and use the package properly?
2. Does the package have unresolved security issues which would prevent end users from using it safely?
3. Does the package somehow prevent progress or cause additional maintenance burden elsewhere in Fedora?
4. Can those packages be consumed by those who want to modify or rebuild them locally?
I think the last two points are often missed in the discussion.
If someone keeps having to maintain some old compatibility package because packages which use it can't be rebuilt for a new version, then that's a problem (but it's an issue that goes beyond FTBFS). Still, people who maintain such compatibility packages should still be able to drop them, under the doctrine that nobody can force them to maintain them. And then point #1 would fail, which we all agree should force the removal of a package.
And if someone goes to check out a package from git or grabs an SRPM and finds that they can't actually build it, even after spending time setting up a proper build environment (which I know isn't terribly difficult, but still), then that's not great. I know I do this all the time, but maybe that's atypical. It still looks a bit sloppy in any case. I do think our duty to people who want to do that goes beyond simply complying with licenses and handing out source.
So in summary, I guess I mostly support allowing packages which can't be rebuilt to stay in the distribution as long as they actually work and aren't causing maintenance burden elsewhere (which needs input from the release engineering folks and such as to whether these things waste their time). But I do think that everyone who advocates for that position needs to consider the negatives. These things do have nonzero impact even if it's not immediately obvious. And everyone needs to be aware that unbuildable packages are more prone to being removed pretty much as soon as they impede work elsewhere in the distribution.
- J<
I want to publicly express my appreciation for Miro's efforts to enforce our policy and his willingness to take the hits from people being rightly upset at its flaws. I also appreciate that the community has done a good job of understanding that the policy is the problem and not making it a personal attack on Miro.
On Wed, Aug 14, 2019 at 12:24 PM Jason L Tibbitts III tibbs@math.uh.edu wrote:
"MH" == Miro Hrončok mhroncok@redhat.com writes:
MH> If we stop here, the current "setting to ASSIGNED to stop this" MH> remains a problem.
Let's think about why this is perceived as a problem. The maintainer has performed an affirmative act that shows they noticed. Can't we just accept that as some statement of intent and stop bugging them at that point?
This is a reasonable compromise to make, IMO. In a perfect world, we'd have enough active packagers to handle things in a timely manner. But in this world, people have a lot going on and there's only so much we can do. People setting a bug to ASSIGNED is a problem I'm willing to accept. If there's an exceptional case, we can say FESCo has the ability to step in and remove it. We should assume positive intent with maintainers and trust that they're doing the right thing.
On Wed, 2019-08-14 at 13:22 -0400, Ben Cotton wrote:
I want to publicly express my appreciation for Miro's efforts to enforce our policy and his willingness to take the hits from people being rightly upset at its flaws. I also appreciate that the community has done a good job of understanding that the policy is the problem and not making it a personal attack on Miro.
On Wed, Aug 14, 2019 at 12:24 PM Jason L Tibbitts III tibbs@math.uh.edu wrote:
> "MH" == Miro Hrončok mhroncok@redhat.com writes:
MH> If we stop here, the current "setting to ASSIGNED to stop this" MH> remains a problem.
Let's think about why this is perceived as a problem. The maintainer has performed an affirmative act that shows they noticed. Can't we just accept that as some statement of intent and stop bugging them at that point?
This is a reasonable compromise to make, IMO. In a perfect world, we'd have enough active packagers to handle things in a timely manner. But in this world, people have a lot going on and there's only so much we can do. People setting a bug to ASSIGNED is a problem I'm willing to accept. If there's an exceptional case, we can say FESCo has the ability to step in and remove it. We should assume positive intent with maintainers and trust that they're doing the right thing.
Alternate perspective, entirely a personal one:
I think the process is actually great. I kinda prefer the direction of travel where we expect that packages are actively maintained and quite aggressively throw them out if they aren't, to the direction where we accumulate cruft and only throw it out after extremely longwinded and easily-subverted processes.
If a package hasn't built for months that's a *problem*. I don't think a packager should be allowed to get away with just setting a bug to ASSIGNED to have the package duck auto-orphaning and eventual removal, possibly forever. That shouldn't be a thing. Packages need to be taken care of.
Exceptions should be for things that really ought to be removed but which we need to keep around. Removing unmaintained things shouldn't be the exception.
I actually think the consequences of the revival of the old policy have been fine. We are throwing out tons of cruft. Occasionally we find something very crufty yet important: this is a *good* outcome of the process. It alerts us to the fact that important stuff depends on something which is not being properly maintained and allows us to address that.
No actions here are reversible, retired packages can be and have been unretired.
On Wed, Aug 14, 2019 at 1:31 PM Adam Williamson adamwill@fedoraproject.org wrote:
I actually think the consequences of the revival of the old policy have been fine. We are throwing out tons of cruft. Occasionally we find something very crufty yet important: this is a *good* outcome of the process. It alerts us to the fact that important stuff depends on something which is not being properly maintained and allows us to address that.
I'm sympathetic to this argument, and I agree that it produces a better output in a vaccuum. My concern is the effect that this has on the community. We rely on the volunteer labor of a lot of people, and I think that obligates us to make compromises. Package retirements are easily reversible; packager retirements are less so.
Part of the problem is that the policy went unenforced for so long. I wonder if we've started enforcement too quickly. Leaving some loopholes in place—and acknowledging that some people will take advantage of them—may be a way to keep the impact on packagers low for now. Then perhaps some of the packager experience initiatives that are in the works can have time to come in and make a more aggressive enforcement palatable.
On Wed, Aug 14, 2019 at 10:30:36AM -0700, Adam Williamson wrote:
I think the process is actually great. I kinda prefer the direction of travel where we expect that packages are actively maintained and quite aggressively throw them out if they aren't, to the direction where we accumulate cruft and only throw it out after extremely longwinded and easily-subverted processes.
I think if you make it easy to "throw out" packages then you must also make it easy to add them back later. People do a lot of work adding and maintaining packages and requiring a full re-review for a package that was retired a few days ago is too much.
Rich.
On 16. 08. 19 16:21, Richard W.M. Jones wrote:
I think if you make it easy to "throw out" packages then you must also make it easy to add them back later. People do a lot of work adding and maintaining packages and requiring a full re-review for a package that was retired a few days ago is too much.
Re-review is only required after 8 weeks.
On Fri, 2019-08-16 at 15:21 +0100, Richard W.M. Jones wrote:
On Wed, Aug 14, 2019 at 10:30:36AM -0700, Adam Williamson wrote:
I think the process is actually great. I kinda prefer the direction of travel where we expect that packages are actively maintained and quite aggressively throw them out if they aren't, to the direction where we accumulate cruft and only throw it out after extremely longwinded and easily-subverted processes.
I think if you make it easy to "throw out" packages then you must also make it easy to add them back later. People do a lot of work adding and maintaining packages and requiring a full re-review for a package that was retired a few days ago is too much.
We don't. There's a grace period of several weeks where a full re- review is not required.
On 14. 08. 19 19:22, Ben Cotton wrote:
I want to publicly express my appreciation for Miro's efforts to enforce our policy and his willingness to take the hits from people being rightly upset at its flaws. I also appreciate that the community has done a good job of understanding that the policy is the problem and not making it a personal attack on Miro.
Thanks. Support like this is greatly appreciated.
On Wed, Aug 14, 2019 at 12:24 PM Jason L Tibbitts III tibbs@math.uh.edu wrote:
> "MH" == Miro Hrončok mhroncok@redhat.com writes:
MH> If we stop here, the current "setting to ASSIGNED to stop this" MH> remains a problem.
Let's think about why this is perceived as a problem. The maintainer has performed an affirmative act that shows they noticed. Can't we just accept that as some statement of intent and stop bugging them at that point?
This is a reasonable compromise to make, IMO. In a perfect world, we'd have enough active packagers to handle things in a timely manner. But in this world, people have a lot going on and there's only so much we can do. People setting a bug to ASSIGNED is a problem I'm willing to accept. If there's an exceptional case, we can say FESCo has the ability to step in and remove it. We should assume positive intent with maintainers and trust that they're doing the right thing.
What if... we only allow swaying FTBFS bugs under the carpet for a certain period of time?
E.g.
1. Mass rebuild for Fedora N happens 2. Packages that fail to build get open bugzillas 3. Packages with NEW bugizllas are orphaned after 8 weeks with weekly reminders 4. If the package hasn't rebuilt for a certain number of releases, it is flagged for removal despite the bug status.
Of course the removal would still need to be communicated properly, but I suspect only a couple of packages would fall into that category.
I suppose that at a time of a release of Fedora version, all shipped packages should have been rebuilt on a platform that was supported on the time of the release.
E.g. when we release Fedora 32, Fedora 28 is already EOL for 5 months. It is IMHO reasonable to expect the packages were rebuilt at least on Fedora 29.
That would effectively mean:
0. package built with .fc29 release tag before the mass rebuild 1. Fedora 29 mass rebuild, package FTBFS, is ASSIGNED, nothing happens 2. Fedora 30 mass rebuild, package FTBFS, is ASSIGNED, nothing happens 3. Fedora 31 mass rebuild, package FTBFS, is ASSIGNED, kind warning 4. Fedora 32 mass rebuild, package FTBFS, is ASSIGNED, big fat warning 5. Fedora 32 branches, package still fails to build, we retire it
Or:
0. package built with .fc28 release tag before F28 branching 1. Fedora 29 mass rebuild, package FTBFS, is ASSIGNED, nothing happens 2. Fedora 30 mass rebuild, package FTBFS, is ASSIGNED, kind warning 3. Fedora 31 mass rebuild, package FTBFS, is ASSIGNED, big fat warning 4. Fedora 31 branches, package still fails to build, we retire it
That gives 1.5 years minimum (F28 branching to F31 branching) to fix a FTBFS. Is that reasonable?
(With a possibility to request an exception in exceptional cases.)
The policy is also easy enough to define:
"Rawhide packages with latest successful build made in Fedora N are retired from master after Fedora N+3 branches."
----- Original Message -----
From: "Miro Hrončok" mhroncok@redhat.com To: devel@lists.fedoraproject.org Sent: Wednesday, August 14, 2019 7:43:13 PM Subject: Re: Let's revisit the FTBFS policy
On 14. 08. 19 19:22, Ben Cotton wrote:
I want to publicly express my appreciation for Miro's efforts to enforce our policy and his willingness to take the hits from people being rightly upset at its flaws. I also appreciate that the community has done a good job of understanding that the policy is the problem and not making it a personal attack on Miro.
Thanks. Support like this is greatly appreciated.
On Wed, Aug 14, 2019 at 12:24 PM Jason L Tibbitts III tibbs@math.uh.edu wrote:
>> "MH" == Miro Hrončok mhroncok@redhat.com writes:
MH> If we stop here, the current "setting to ASSIGNED to stop this" MH> remains a problem.
Let's think about why this is perceived as a problem. The maintainer has performed an affirmative act that shows they noticed. Can't we just accept that as some statement of intent and stop bugging them at that point?
This is a reasonable compromise to make, IMO. In a perfect world, we'd have enough active packagers to handle things in a timely manner. But in this world, people have a lot going on and there's only so much we can do. People setting a bug to ASSIGNED is a problem I'm willing to accept. If there's an exceptional case, we can say FESCo has the ability to step in and remove it. We should assume positive intent with maintainers and trust that they're doing the right thing.
What if... we only allow swaying FTBFS bugs under the carpet for a certain period of time?
E.g.
- Mass rebuild for Fedora N happens
- Packages that fail to build get open bugzillas
- Packages with NEW bugizllas are orphaned after 8 weeks with weekly
reminders 4. If the package hasn't rebuilt for a certain number of releases, it is flagged for removal despite the bug status.
Of course the removal would still need to be communicated properly, but I suspect only a couple of packages would fall into that category.
I suppose that at a time of a release of Fedora version, all shipped packages should have been rebuilt on a platform that was supported on the time of the release.
E.g. when we release Fedora 32, Fedora 28 is already EOL for 5 months. It is IMHO reasonable to expect the packages were rebuilt at least on Fedora 29.
That would effectively mean:
- package built with .fc29 release tag before the mass rebuild
- Fedora 29 mass rebuild, package FTBFS, is ASSIGNED, nothing happens
- Fedora 30 mass rebuild, package FTBFS, is ASSIGNED, nothing happens
- Fedora 31 mass rebuild, package FTBFS, is ASSIGNED, kind warning
- Fedora 32 mass rebuild, package FTBFS, is ASSIGNED, big fat warning
- Fedora 32 branches, package still fails to build, we retire it
Or:
- package built with .fc28 release tag before F28 branching
- Fedora 29 mass rebuild, package FTBFS, is ASSIGNED, nothing happens
- Fedora 30 mass rebuild, package FTBFS, is ASSIGNED, kind warning
- Fedora 31 mass rebuild, package FTBFS, is ASSIGNED, big fat warning
- Fedora 31 branches, package still fails to build, we retire it
That gives 1.5 years minimum (F28 branching to F31 branching) to fix a FTBFS. Is that reasonable?
(With a possibility to request an exception in exceptional cases.)
The policy is also easy enough to define:
"Rawhide packages with latest successful build made in Fedora N are retired from master after Fedora N+3 branches."
Yes, that sounds great!
Thanks for communicating this, Pavel
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok
On 14. 08. 19 19:43, Miro Hrončok wrote:
E.g. when we release Fedora 32, Fedora 28 is already EOL for 5 months. It is IMHO reasonable to expect the packages were rebuilt at least on Fedora 29.
Oh. When we release Fedora 32, Fedora 29 is already EOL for 5 months. But the rest checks out. So the "rebuilt on a platform that was supported on the time of the release" doesn't really stand, but I still think the rest of the proposal makes sense. It gives a reasonable time to sway things while it makes sure:
- FTBFS packages where the bug remains NEW are orphaned - We don't ship FTBFS packages forever - We don't "rip off" packages after "just 6 months"
Dne 14. 08. 19 v 19:43 Miro Hrončok napsal(a):
On 14. 08. 19 19:22, Ben Cotton wrote:
I want to publicly express my appreciation for Miro's efforts to enforce our policy and his willingness to take the hits from people being rightly upset at its flaws. I also appreciate that the community has done a good job of understanding that the policy is the problem and not making it a personal attack on Miro.
Thanks. Support like this is greatly appreciated.
On Wed, Aug 14, 2019 at 12:24 PM Jason L Tibbitts III tibbs@math.uh.edu wrote:
>> "MH" == Miro Hrončok mhroncok@redhat.com writes:
MH> If we stop here, the current "setting to ASSIGNED to stop this" MH> remains a problem.
Let's think about why this is perceived as a problem. The maintainer has performed an affirmative act that shows they noticed. Can't we just accept that as some statement of intent and stop bugging them at that point?
This is a reasonable compromise to make, IMO. In a perfect world, we'd have enough active packagers to handle things in a timely manner. But in this world, people have a lot going on and there's only so much we can do. People setting a bug to ASSIGNED is a problem I'm willing to accept. If there's an exceptional case, we can say FESCo has the ability to step in and remove it. We should assume positive intent with maintainers and trust that they're doing the right thing.
What if... we only allow swaying FTBFS bugs under the carpet for a certain period of time?
E.g.
1. Mass rebuild for Fedora N happens 2. Packages that fail to build get open bugzillas 3. Packages with NEW bugizllas are orphaned after 8 weeks with weekly reminders 4. If the package hasn't rebuilt for a certain number of releases, it is flagged for removal despite the bug status.
Of course the removal would still need to be communicated properly, but I suspect only a couple of packages would fall into that category.
I suppose that at a time of a release of Fedora version, all shipped packages should have been rebuilt on a platform that was supported on the time of the release.
E.g. when we release Fedora 32, Fedora 28 is already EOL for 5 months. It is IMHO reasonable to expect the packages were rebuilt at least on Fedora 29.
That would effectively mean:
- package built with .fc29 release tag before the mass rebuild
- Fedora 29 mass rebuild, package FTBFS, is ASSIGNED, nothing happens
- Fedora 30 mass rebuild, package FTBFS, is ASSIGNED, nothing happens
- Fedora 31 mass rebuild, package FTBFS, is ASSIGNED, kind warning
- Fedora 32 mass rebuild, package FTBFS, is ASSIGNED, big fat warning
- Fedora 32 branches, package still fails to build, we retire it
Or:
- package built with .fc28 release tag before F28 branching
- Fedora 29 mass rebuild, package FTBFS, is ASSIGNED, nothing happens
- Fedora 30 mass rebuild, package FTBFS, is ASSIGNED, kind warning
- Fedora 31 mass rebuild, package FTBFS, is ASSIGNED, big fat warning
- Fedora 31 branches, package still fails to build, we retire it
That gives 1.5 years minimum (F28 branching to F31 branching) to fix a FTBFS. Is that reasonable?
(With a possibility to request an exception in exceptional cases.)
The policy is also easy enough to define:
"Rawhide packages with latest successful build made in Fedora N are retired from master after Fedora N+3 branches."
Well, really, I don't see the ASSIGNED as that big deal. That at least means the maintainer is alive and paying (some) attention and that is important. In the end, you can't prevent such maintainer to unretire the FTBFS package and tag it back into Fedora.
Also, in my case "rubygems" package was retired. The interesting part here is that the binary package build from "rubygems" SRPM was not even part of compose, so some of the proposals to remove such packages from compose would not apply here.
Now you might wonder why we had "rubygems" package in Fedora. Partly, it is historic reason. It used to be independent project you probably wanted to install alongside Ruby. As the time goes, the RubyGems were bundled into Ruby. So now "ruby" package provides "rubygems" package. But RubyGems are still doing independent releases out of Ruby schedule. So the "rubygems" package gives us easy way to update RubyGems in Ruby without updating Ruby itself and adding some additional sources etc. There was no reason to do this in more than year, that's why I did not bother to fix the FTBFS. Now the package is retired.
Of course you might consider this special case, but apparently all the other people who speak up had different special cases.
Vít
On 15. 08. 19 7:39, Vít Ondruch wrote:
Of course you might consider this special case, but apparently all the other people who speak up had different special cases.
"special cases aren't special enough to break the rules"
I still think that if somebody would need to keep package unretired for 1.5+ years, they have options:
- let it be retired, unretire, retag (as in: "I don't give a damn") - request an exception with proper reasons (as in: "I have proper reasons")
Just being able to let the package rot for 3+ releases is not good enough reason IMHO.
Dne 15. 08. 19 v 9:33 Miro Hrončok napsal(a):
On 15. 08. 19 7:39, Vít Ondruch wrote:
Of course you might consider this special case, but apparently all the other people who speak up had different special cases.
"special cases aren't special enough to break the rules"
They had either "special" or "unique" cases. As you want. The point is if there is no rule as retiring packages which has the last FTBFS bug in ASSIGNED state for whatever reason, there is no rule to break. IOW trying to define rules for this case is overkill.
At the end, if somebody cares about such cases, it should not be hard to discover and act upon them, i.e. bugging the maintainer, fixing them, taking over the maintenance etc.
I still think that if somebody would need to keep package unretired for 1.5+ years, they have options:
- let it be retired, unretire, retag (as in: "I don't give a damn") - request an exception with proper reasons (as in: "I have proper reasons")
Yes, right, but the maintainer already have taken action, they switched their bugs from NEW to ASSIGNED. They had to evaluate it is worth of the action. Why we should explicitly believe they did not do it in good faith?
Just being able to let the package rot for 3+ releases is not good enough reason IMHO.
There needs to be taken action prior the package is left rotting!
Actually, do you happen to know how many components were retired? According to compose report from 20190811 [1], I guess it was ~570 packages. How many of them had associated FTBFS BZs in "ASSIGNED" state and for which version of Fedora? This would be interesting statistics to know. My guess is that it was 100 BZs at most, but probably much lower number.
Vít
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On 15. 08. 19 12:06, Vít Ondruch wrote:
At the end, if somebody cares about such cases, it should not be hard to discover and act upon them, i.e. bugging the maintainer, fixing them, taking over the maintenance etc.
This part is problematic. Because it requires human action that can be seen as toxic by some.
According to compose report from 20190811 [1], I guess it was ~570 packages. How many of them had associated FTBFS BZs in "ASSIGNED" state and for which version of Fedora? This would be interesting statistics to know. My guess is that it was 100 BZs at most, but probably much lower number.
"for which version of Fedora" doesn't apply really. Most of the bugs were just "rawhide" since the latest rawhide -> 30 only happened partially.
The status data should be visible in Bugzilla, however no idea how to query them grammatically:
- get CLOSED EOL bugzillas blocking the F30FTBFS tracker - fetch their previous state (this is visible in the bug, but no idea how to query it)
----- Original Message -----
Sent: Thursday, August 15, 2019 12:42:02 PM Subject: Re: Let's revisit the FTBFS policy
On 15. 08. 19 12:06, Vít Ondruch wrote:
At the end, if somebody cares about such cases, it should not be hard to discover and act upon them, i.e. bugging the maintainer, fixing them, taking over the maintenance etc.
This part is problematic. Because it requires human action that can be seen as toxic by some.
Only if they're present to notice. In the end, they're late and it was fixed for them, or at least someone cares for their work, so they should be... grateful?
According to compose report from 20190811 [1], I guess it was ~570 packages. How many of them had associated FTBFS BZs in "ASSIGNED" state and for which version of Fedora? This would be interesting statistics to know. My guess is that it was 100 BZs at most, but probably much lower number.
"for which version of Fedora" doesn't apply really. Most of the bugs were just "rawhide" since the latest rawhide -> 30 only happened partially.
The status data should be visible in Bugzilla, however no idea how to query them grammatically:
- get CLOSED EOL bugzillas blocking the F30FTBFS tracker
This should be it: https://bugzilla.redhat.com/buglist.cgi?bug_id=1674516&bug_id_type=andde...
- fetch their previous state (this is visible in the bug, but no idea how to query it)
Sorry, I have no idea for this one.
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok
Regards,
Dne 15. 08. 19 v 13:36 Pavel Valena napsal(a):
----- Original Message -----
Sent: Thursday, August 15, 2019 12:42:02 PM Subject: Re: Let's revisit the FTBFS policy
On 15. 08. 19 12:06, Vít Ondruch wrote:
At the end, if somebody cares about such cases, it should not be hard to discover and act upon them, i.e. bugging the maintainer, fixing them, taking over the maintenance etc.
This part is problematic. Because it requires human action that can be seen as toxic by some.
Only if they're present to notice. In the end, they're late and it was fixed for them, or at least someone cares for their work, so they should be... grateful?
According to compose report from 20190811 [1], I guess it was ~570 packages. How many of them had associated FTBFS BZs in "ASSIGNED" state and for which version of Fedora? This would be interesting statistics to know. My guess is that it was 100 BZs at most, but probably much lower number.
"for which version of Fedora" doesn't apply really. Most of the bugs were just "rawhide" since the latest rawhide -> 30 only happened partially.
The status data should be visible in Bugzilla, however no idea how to query them grammatically:
- get CLOSED EOL bugzillas blocking the F30FTBFS tracker
This should be it: https://bugzilla.redhat.com/buglist.cgi?bug_id=1674516&bug_id_type=andde...
So this lists 656 components for F30.
There are 16 components for F29:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1602938&bug_id_type=andde...
- fetch their previous state (this is visible in the bug, but no idea how to query it)
Sorry, I have no idea for this one.
Ah, you beat me to do this:
F30 - 41 components:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1674516&bug_id_type=andde...
F29 - 2 components:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1602938&bug_id_type=andde...
F28 - 25 components:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1555378&bug_id_type=andde...
Interestingly enough, some people who complains the most about the process are too busy to even switch the component to assigned ...
Vít
Dne 15. 08. 19 v 14:40 Vít Ondruch napsal(a):
Dne 15. 08. 19 v 13:36 Pavel Valena napsal(a):
----- Original Message -----
Sent: Thursday, August 15, 2019 12:42:02 PM Subject: Re: Let's revisit the FTBFS policy
On 15. 08. 19 12:06, Vít Ondruch wrote:
At the end, if somebody cares about such cases, it should not be hard to discover and act upon them, i.e. bugging the maintainer, fixing them, taking over the maintenance etc.
This part is problematic. Because it requires human action that can be seen as toxic by some.
Only if they're present to notice. In the end, they're late and it was fixed for them, or at least someone cares for their work, so they should be... grateful?
According to compose report from 20190811 [1], I guess it was ~570 packages. How many of them had associated FTBFS BZs in "ASSIGNED" state and for which version of Fedora? This would be interesting statistics to know. My guess is that it was 100 BZs at most, but probably much lower number.
"for which version of Fedora" doesn't apply really. Most of the bugs were just "rawhide" since the latest rawhide -> 30 only happened partially.
The status data should be visible in Bugzilla, however no idea how to query them grammatically:
- get CLOSED EOL bugzillas blocking the F30FTBFS tracker
This should be it: https://bugzilla.redhat.com/buglist.cgi?bug_id=1674516&bug_id_type=andde...
So this lists 656 components for F30.
There are 16 components for F29:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1602938&bug_id_type=andde...
- fetch their previous state (this is visible in the bug, but no idea how to query it)
Sorry, I have no idea for this one.
Ah, you beat me to do this:
F30 - 41 components:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1674516&bug_id_type=andde...
F29 - 2 components:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1602938&bug_id_type=andde...
F28 - 25 components:
https://bugzilla.redhat.com/buglist.cgi?bug_id=1555378&bug_id_type=andde...
Interestingly enough, some people who complains the most about the process are too busy to even switch the component to assigned ...
Checking more of the tickets, I want to apologize for the last remark, which was not necessary.
Vít
Vít
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 15. 08. 19 14:40, Vít Ondruch wrote:
Interestingly enough, some people who complains the most about the process are too busy to even switch the component to assigned ...
¯_(ツ)_/¯
To rephrase: People have real work to do, so we should stop bothering them.
On Thu, 2019-08-15 at 09:33 +0200, Miro Hrončok wrote:
On 15. 08. 19 7:39, Vít Ondruch wrote:
Of course you might consider this special case, but apparently all the other people who speak up had different special cases.
"special cases aren't special enough to break the rules"
I still think that if somebody would need to keep package unretired for 1.5+ years, they have options:
- let it be retired, unretire, retag (as in: "I don't give a damn")
- request an exception with proper reasons (as in: "I have proper reasons")
Just being able to let the package rot for 3+ releases is not good enough reason IMHO.
Or just fix it so it damn well builds. Even if *you* don't need to use it. I mean, is it so hard? I get *itchy* if I have an FTBFS bug on one of my packages for three days. I can't imagine letting one sit there for six months!
On Thu, Aug 15, 2019 at 1:33 PM Adam Williamson adamwill@fedoraproject.org wrote:
Or just fix it so it damn well builds. Even if *you* don't need to use it. I mean, is it so hard? I get *itchy* if I have an FTBFS bug on one of my packages for three days. I can't imagine letting one sit there for six months!
I don't like it either but I have to admit my free cycles have been much fewer as of late both for personal and professional reasons.
Perhaps a partial solution is encouraging people to ask for help. Sure it's easy to post to the devel list but sometimes it's difficult to admit you need help :)
I am sometimes guilty of beating my head against the wall to fix something. I usually can in the long run and I have learned a lot doing it but sometimes it would be better if things got fixed more quickly.
Just thinking out loud but perhaps some sort of flag in bugzilla that says "If you know how to fix this please do!"
Thanks, Richard
Richard Shaw wrote:
Perhaps a partial solution is encouraging people to ask for help. Sure it's easy to post to the devel list but sometimes it's difficult to admit you need help :)
IMHO, it should be the job of those people who broke the packages to fix them. E.g., if yet another incompatible GCC update breaks dozens of C and/or C++ packages, it should be up to the GCC maintainers to make them build again. If some policy change requires a specfile update (e.g., the addition of explicit BuildRequires: gcc-c++), it should be up to the people who mandated the policy change to do this update (which was at least partially done in the aforementioned example, but there were still dozens of packages left to the individual package maintainers to fix for various reasons). The current situation where you can break hundreds of packages and then expect somebody else to fix them is really antisocial and unfair.
Kevin Kofler
On Wed, 14 Aug 2019 11:23:53 -0500, you wrote:
So in summary, I guess I mostly support allowing packages which can't be rebuilt to stay in the distribution as long as they actually work and aren't causing maintenance burden elsewhere
On the other hand, unbuildable packages could be viewed as a security risk.
If you can't just fix the security issue and rebuild, but instead have to also fix the issue(s) that prevent the package from rebuilding this could cause delays in getting a security update out.
On Thu, 2019-08-15 at 09:50 -0400, Gerald Henriksen wrote:
On Wed, 14 Aug 2019 11:23:53 -0500, you wrote:
So in summary, I guess I mostly support allowing packages which can't be rebuilt to stay in the distribution as long as they actually work and aren't causing maintenance burden elsewhere
On the other hand, unbuildable packages could be viewed as a security risk.
If you can't just fix the security issue and rebuild, but instead have to also fix the issue(s) that prevent the package from rebuilding this could cause delays in getting a security update out.
Not to mention packages with compiled code not picking up all the hardening flags introduced since they have last been build - that could be a security issue by itself.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
"GH" == Gerald Henriksen ghenriks@gmail.com writes:
GH> On the other hand, unbuildable packages could be viewed as a GH> security risk.
I mentioned security explicitly in my message. Just not in the portion you quoted.
GH> If you can't just fix the security issue and rebuild, but instead GH> have to also fix the issue(s) that prevent the package from GH> rebuilding this could cause delays in getting a security update out.
I mean, nothing currently guarantees that security fixes go out in a timely manner, for all sorts of reasons. If we're going to get serious about reducing that time, I would think there's a more productive way to do that than dumping all FTBFS packages because they _might_ one day have a security issue that needs fixing. But yes, certainly dump all that have open security bugs.
- J<
On Wed, 2019-08-14 at 14:55 +0200, Vít Ondruch wrote:
Dne 14. 08. 19 v 14:23 Miro Hrončok napsal(a):
Hello.
Recently, a couple hundred packages were retired from rawhide (Fedora 31 at that time) based on the Fedora Failed to Build From Source Policy [1]. From various reactions over several threads it seems this policy is not ideal. This is an attempt to collect feedback and make the policy better serve Fedora's needs.
Fedora has a policy for a long time that can be simplified as:
- Mass rebuild for Fedora N happens by releng
- Packages that fail to build get open bugzillas by releng
- Packages with NEW bugizllas are orphaned after 8 weeks with weekly
reminders by releng
I think it would be probably enough to stop here. Orphaned packages gets garbage collected ATM. The step 4 was a bit unexpected for packages with bugs in ASSIGNED state especially.
Also the timing with mass rebuild and shifting packages from Rawhide to F31 is unfortunate. I saw a lot of packages, which were reported as FTBFS in BZ, then they were retired but later the bugs were moved from Rawhide to F31, that was strange.
While not directly policy related, I strongly suggest, if possible, to do a test compose with the packages removed to see how it fares & ideally run some tests (Open QA ?) on the result, to prevent avoidable breakage.
Droping such big batches of packages without testing we can still produce out blocking deliverables & checking the deliverables appear to work simply seems too invasive to me.
Vít
- A week before Fedora N+1 branching any packages which still have
open Fedora N FTBFS bug are retired by releng
However, 3 or 4 haven't happened since Fedora ~26, because the automation was not working any more for various reasons I don't understand.
The policy was then updated by FESCo to allow anybody to step up and do 3. manually. However it needs 2 e-mails to devel directed at the package owners and that may be understood as a personal attack by some. So nobody ever did that but me (twice) IIRC (and I got my share of shame for that).
Currently, the FTBFS retirement is problematic due to various things:
- Bugzilla spam: Maintainers are spammed weekly by releng, some of
them find that annoying and simply switch the bug to ASSIGNED to make it stop.
The problem is, how do we both keep notifying the maintainers that action is needed and not spam them with stuff that will make them filter all Fedora e-mails to /dev/null.
- Retirement out of the blue. When releng executes 4., packagers that
stopped the bugzilla spam by setting the bug to ASSIGNED are suddenly surprised the package was retired. OTOH arguably they made a deliberate action to stop the notifications. However, most importantly, any dependent packages were not notified at all, which is **not fair**.
This state is broken mostly because there is no automatic orphaning of packages that have NEW bugzillas and there is no orphaning at all for packages where the bugzillas are ASSIGNED for months. For the second bit, everything indicates that packagers are aware of the problem and will fix the bug in time before 4., but they don't and things blow up.
- Questionable importance of the FTBFS bug.
Repeatedly, it has been stated by some, the FTBFS bug is not important and we should not retire packages at all based on the fact that they "simply" fail to build. I personally don't agree with this for various reasons, but I can imagine a situation, where it is reasonable to say so and delay the problem. Obvious argument is: Better to have 1 package nonbuildable than have 100 packages nonisntallable. So we need a way to opt-out from the retirement, however simply setting the bugzilla to ASSIGNED is not it, because we will end up with packages last built 6 years ago (and I believe this is not what we want).
I'm starting this thread to collect the ideas about how to make this better. If you see more problems, share them. I promise I'll do my best to make the policy work better for both Fedora and the people who make Fedora.
[1] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_...
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 14. 08. 19 14:23, Miro Hrončok wrote:
Hello.
Recently, a couple hundred packages were retired from rawhide (Fedora 31 at that time) based on the Fedora Failed to Build From Source Policy [1]. From various reactions over several threads it seems this policy is not ideal. This is an attempt to collect feedback and make the policy better serve Fedora's needs.
Fedora has a policy for a long time that can be simplified as:
1. Mass rebuild for Fedora N happens by releng 2. Packages that fail to build get open bugzillas by releng 3. Packages with NEW bugizllas are orphaned after 8 weeks with weekly reminders by releng 4. A week before Fedora N+1 branching any packages which still have open Fedora N FTBFS bug are retired by releng
However, 3 or 4 haven't happened since Fedora ~26, because the automation was not working any more for various reasons I don't understand.
The policy was then updated by FESCo to allow anybody to step up and do 3. manually. However it needs 2 e-mails to devel directed at the package owners and that may be understood as a personal attack by some. So nobody ever did that but me (twice) IIRC (and I got my share of shame for that).
Currently, the FTBFS retirement is problematic due to various things:
- Bugzilla spam: Maintainers are spammed weekly by releng, some of them find
that annoying and simply switch the bug to ASSIGNED to make it stop.
The problem is, how do we both keep notifying the maintainers that action is needed and not spam them with stuff that will make them filter all Fedora e-mails to /dev/null.
- Retirement out of the blue. When releng executes 4., packagers that stopped
the bugzilla spam by setting the bug to ASSIGNED are suddenly surprised the package was retired. OTOH arguably they made a deliberate action to stop the notifications. However, most importantly, any dependent packages were not notified at all, which is **not fair**.
This state is broken mostly because there is no automatic orphaning of packages that have NEW bugzillas and there is no orphaning at all for packages where the bugzillas are ASSIGNED for months. For the second bit, everything indicates that packagers are aware of the problem and will fix the bug in time before 4., but they don't and things blow up.
- Questionable importance of the FTBFS bug.
Repeatedly, it has been stated by some, the FTBFS bug is not important and we should not retire packages at all based on the fact that they "simply" fail to build. I personally don't agree with this for various reasons, but I can imagine a situation, where it is reasonable to say so and delay the problem. Obvious argument is: Better to have 1 package nonbuildable than have 100 packages nonisntallable. So we need a way to opt-out from the retirement, however simply setting the bugzilla to ASSIGNED is not it, because we will end up with packages last built 6 years ago (and I believe this is not what we want).
I'm starting this thread to collect the ideas about how to make this better. If you see more problems, share them. I promise I'll do my best to make the policy work better for both Fedora and the people who make Fedora.
[1] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_...
Policy changes proposed at:
https://pagure.io/fesco/fesco-docs/pull-request/18 https://pagure.io/fesco/issue/2244
devel@lists.stg.fedoraproject.org
-
Adam Williamson -
Ben Cotton -
Florian Weimer -
Gerald Henriksen -
Jason L Tibbitts III -
Kevin Kofler -
Martin Kolman -
Miro Hrončok -
Panu Matilainen -
Pavel Valena -
Richard Shaw -
Richard W.M. Jones -
Vít Ondruch