I talked to several people over the last couple days about what we can do to try to avoid the "hero testing" treadmill that we've been on during every Freeze in recent memory (specifically that we're usually fixing Blocker bugs until the day before the Go/No-Go meeting and that means that our QA team is pulling all-night test runs basically every week).
I made the following suggestion to both Adam Williamson and David Cantrell over the last couple of days and both seemed to think that this is a reasonable approach (but for different reasons, interestingly).
Beginning with Alpha Freeze in Fedora 22, the policy would be amended to include the following: 1. As currently, the Go/No-Go meeting must be held on the Thursday preceding the planned Tuesday of public release. 2. At 1600 UTC on the Monday preceding Go/No-Go, a check of the known Blocker list must be performed. If there are any blocker bugs that are not marked as MODIFIED or ON_QA (meaning that they are filed as an update in Bodhi), then the Go/No-Go meeting that week is canceled (or converted to a Blocker Bug review) and a slip is *automatic*. 3. Relevant to the above, a Release Candidate compose must be started immediately after the above blocker review and must complete successfully by 1300 UTC on Tuesday. Otherwise, the Go/No-Go meeting that week is canceled (or converted to a Blocker Bug review) and a slip is *automatic*. 4. If *new* blockers are discovered (or regressed) between the Monday blocker review and Thursday Go/No-Go, it is *permissible* for another compose to be created if the following conditions are met: a. The fix is capable of being produced and built quickly. b. There is at least 24 hours remaining between the expected completion of the compose and the Go/No-Go meeting.
The idea here is to ensure that there is a clear engineering deadline in order to guarantee that the QA team has a reasonable time period in which to perform validation tests. I think this approach is too risky this late in the F21 process, which is why I propose this only for Fedora 22 and later.
On Fri, Oct 31, 2014 at 11:17:39 -0400, Stephen Gallagher sgallagh@redhat.com wrote:
The idea here is to ensure that there is a clear engineering deadline in order to guarantee that the QA team has a reasonable time period in which to perform validation tests. I think this approach is too risky this late in the F21 process, which is why I propose this only for Fedora 22 and later.
Are the automatic slips always going to be a full week? While we normally slip full week, there have been cases recently where slips of less than a week have been contemplated.
On Fri, 2014-10-31 at 10:34 -0500, Bruno Wolff III wrote:
On Fri, Oct 31, 2014 at 11:17:39 -0400, Stephen Gallagher sgallagh@redhat.com wrote:
The idea here is to ensure that there is a clear engineering deadline in order to guarantee that the QA team has a reasonable time period in which to perform validation tests. I think this approach is too risky this late in the F21 process, which is why I propose this only for Fedora 22 and later.
Are the automatic slips always going to be a full week? While we normally slip full week, there have been cases recently where slips of less than a week have been contemplated.
Well, we learned during those questions that it's a matter of coordination with the mirrors. They expect to have our master trees prepared at certain times or they can't be mirrored out in time for the Tuesday launches. So really, we need to keep to the same schedule wherever possible.
On Fri, Oct 31, 2014 at 14:44:06 -0400, Stephen Gallagher sgallagh@redhat.com wrote:
Well, we learned during those questions that it's a matter of coordination with the mirrors. They expect to have our master trees prepared at certain times or they can't be mirrored out in time for the Tuesday launches. So really, we need to keep to the same schedule wherever possible.
I was really questioning that the policy didn't say how long the slip is, rather than what the amount is. Though my opinion is that it is better to always slip a week as the one or two day slips will just encourage more of what you are trying to avoid.
I think there is real risk of burn out in the QA volunteers, with not only the quick turn around commonly needed for testing right before release, but also with the really long blocker meetings leading up to that.
This policy change seems like a really good idea. Thanks for working on it, Stephen.
Stephen Gallagher wrote:
I talked to several people over the last couple days about what we can do to try to avoid the "hero testing" treadmill that we've been on during every Freeze in recent memory (specifically that we're usually fixing Blocker bugs until the day before the Go/No-Go meeting and that means that our QA team is pulling all-night test runs basically every week).
Your proposed changes will only cause us to slip more often. Those "hero" runs have been done for a reason: to prevent a slip that would otherwise have been inevitable! Your proposed changes effectively ban such "hero" actions, actively preventing volunteers from helping releasing Fedora on time. I see no benefit whatsoever coming from those changes.
Kevin Kofler
On Tue, Nov 04, 2014 at 03:26:50AM +0100, Kevin Kofler wrote:
I talked to several people over the last couple days about what we can do to try to avoid the "hero testing" treadmill that we've been on during every Freeze in recent memory (specifically that we're usually fixing Blocker bugs until the day before the Go/No-Go meeting and that means that our QA team is pulling all-night test runs basically every week).
Your proposed changes will only cause us to slip more often. Those "hero" runs have been done for a reason: to prevent a slip that would otherwise have been inevitable! Your proposed changes effectively ban such "hero" actions, actively preventing volunteers from helping releasing Fedora on time. I see no benefit whatsoever coming from those changes.
Kevin, I'm missing something here. You're right that the QA hero runs are done to prevent slips, but I'm not seeing the logical connection between what Stephen suggests and _banning_ them. The idea is to make them less likely to be necessary with the cooparation of packagers as we go up to the release.
Matthew Miller wrote:
Kevin, I'm missing something here. You're right that the QA hero runs are done to prevent slips, but I'm not seeing the logical connection between what Stephen suggests and _banning_ them. The idea is to make them less likely to be necessary with the cooparation of packagers as we go up to the release.
Well, Stephen Gallagher wrote:
- At 1600 UTC on the Monday preceding Go/No-Go, a check of the known
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ new strict deadline
Blocker list must be performed. If there are any blocker bugs that are
^^^^ MUST
not marked as MODIFIED or ON_QA (meaning that they are filed as an update in Bodhi), then the Go/No-Go meeting that week is canceled (or converted to a Blocker Bug review) and a slip is *automatic*.
^^^^^^^^^^^^^^^^^^^^^ says it all
- Relevant to the above, a Release Candidate compose must be started
^^^^ MUST
immediately after the above blocker review and must complete successfully by 1300 UTC on Tuesday. Otherwise, the Go/No-Go meeting
^^^^^^^^^^^^^^^^^^^ another new strict deadline
that week is canceled (or converted to a Blocker Bug review) and a slip is *automatic*.
^^^^^^^^^^^^^^^^^^^^^ says it all, again
- If *new* blockers are discovered (or regressed) between the Monday
blocker review and Thursday Go/No-Go, it is *permissible* for another compose to be created if the following conditions are met: a. The fix is capable of being produced and built quickly. b. There is at least 24 hours remaining between the expected completion
^^^^^^^^^^^^^^^^^^^^^^^^^^^ yet another new strict deadline
of the compose and the Go/No-Go meeting.
So how do those new strict rules NOT ban preventing a slip? They spell out 3 strict deadlines which, if they are not met, AUTOMATICALLY trigger a slip.
Kevin Kofler
On Sun, 2014-11-09 at 23:45 +0100, Kevin Kofler wrote:
Matthew Miller wrote:
Kevin, I'm missing something here. You're right that the QA hero runs are done to prevent slips, but I'm not seeing the logical connection between what Stephen suggests and _banning_ them. The idea is to make them less likely to be necessary with the cooparation of packagers as we go up to the release.
Well, Stephen Gallagher wrote:
- At 1600 UTC on the Monday preceding Go/No-Go, a check of the known
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ new strict deadlineBlocker list must be performed. If there are any blocker bugs that are
^^^^ MUSTnot marked as MODIFIED or ON_QA (meaning that they are filed as an update in Bodhi), then the Go/No-Go meeting that week is canceled (or converted to a Blocker Bug review) and a slip is *automatic*.
^^^^^^^^^^^^^^^^^^^^^ says it all
- Relevant to the above, a Release Candidate compose must be started
^^^^ MUSTimmediately after the above blocker review and must complete successfully by 1300 UTC on Tuesday. Otherwise, the Go/No-Go meeting
^^^^^^^^^^^^^^^^^^^ another new strict deadlinethat week is canceled (or converted to a Blocker Bug review) and a slip is *automatic*.
^^^^^^^^^^^^^^^^^^^^^ says it all, again
- If *new* blockers are discovered (or regressed) between the Monday
blocker review and Thursday Go/No-Go, it is *permissible* for another compose to be created if the following conditions are met: a. The fix is capable of being produced and built quickly. b. There is at least 24 hours remaining between the expected completion
^^^^^^^^^^^^^^^^^^^^^^^^^^^ yet another new strict deadlineof the compose and the Go/No-Go meeting.So how do those new strict rules NOT ban preventing a slip? They spell out 3 strict deadlines which, if they are not met, AUTOMATICALLY trigger a slip.
Apologies for the late response; I've been at a conference all last week.
These new rules don't ban "preventing a slip", they attempt to eliminate the unreasonable demands we're putting on our volunteer QA team *every week during Freeze*. It's gotten out of hand and it's burning people out.
The primary problem is that when we slip, there has never been a clear statement made about when exactly when the deadline is for devs to get their fixes in. Historically, devs have been operating under the assumption that as long as a package lands before the next Go/No-Go meeting, but that has failed to account for the time needed to create a new Test Compose (which takes approx. 8 hours right now) as well as time to have the QA team re-run the Release Validation tests (which takes an absolute minimum of 20 hours fueled by caffeine and adrenaline). This constant pause-then-panic situation is untenable and needs to be addressed.
By instituting the above plan, we will be much more transparent about what the deadlines are for all participants (dev/maintainers, rel-eng and QA) and we relieve the latter two of some of their panicked efforts if we get to the Monday Blocker Review and it's clear that there is no realistic chance that the Thursday Go/No-Go will rule in favor.
I considered making a modification to the slip-rule for the Monday blocker meeting that would allow us to *on rare occasions* add a single day to the devel cycle (if, for example, the last blocker fix is known and being prepared already), but I hesitated to include it in the policy for fear that it would be misused.
Instead, perhaps we can just add the following line to the policy as written:
"A one-time exception to the policy may be made if there is a unanimous agreement between the development (FESCo) representative, the rel-eng representative and the QA representative at the Monday Blocker Review meeting".
Stephen Gallagher wrote:
These new rules don't ban "preventing a slip", they attempt to eliminate the unreasonable demands we're putting on our volunteer QA team *every week during Freeze*. It's gotten out of hand and it's burning people out.
The primary problem is that when we slip, there has never been a clear statement made about when exactly when the deadline is for devs to get their fixes in. Historically, devs have been operating under the assumption that as long as a package lands before the next Go/No-Go meeting, but that has failed to account for the time needed to create a new Test Compose (which takes approx. 8 hours right now) as well as time to have the QA team re-run the Release Validation tests (which takes an absolute minimum of 20 hours fueled by caffeine and adrenaline). This constant pause-then-panic situation is untenable and needs to be addressed.
By instituting the above plan, we will be much more transparent about what the deadlines are for all participants (dev/maintainers, rel-eng and QA) and we relieve the latter two of some of their panicked efforts if we get to the Monday Blocker Review and it's clear that there is no realistic chance that the Thursday Go/No-Go will rule in favor.
I think our fundamental disagreement is that you believe that the rules will make developers come up with fixes faster, whereas I believe that we developers are already fixing things as fast as we can and the rules will only make Fedora releases slip more often.
Kevin Kofler
On Sun, 2014-11-23 at 10:02 +0100, Kevin Kofler wrote:
Stephen Gallagher wrote:
These new rules don't ban "preventing a slip", they attempt to eliminate the unreasonable demands we're putting on our volunteer QA team *every week during Freeze*. It's gotten out of hand and it's burning people out.
The primary problem is that when we slip, there has never been a clear statement made about when exactly when the deadline is for devs to get their fixes in. Historically, devs have been operating under the assumption that as long as a package lands before the next Go/No-Go meeting, but that has failed to account for the time needed to create a new Test Compose (which takes approx. 8 hours right now) as well as time to have the QA team re-run the Release Validation tests (which takes an absolute minimum of 20 hours fueled by caffeine and adrenaline). This constant pause-then-panic situation is untenable and needs to be addressed.
By instituting the above plan, we will be much more transparent about what the deadlines are for all participants (dev/maintainers, rel-eng and QA) and we relieve the latter two of some of their panicked efforts if we get to the Monday Blocker Review and it's clear that there is no realistic chance that the Thursday Go/No-Go will rule in favor.
I think our fundamental disagreement is that you believe that the rules will make developers come up with fixes faster, whereas I believe that we developers are already fixing things as fast as we can and the rules will only make Fedora releases slip more often.
Yeah, that's a valid concern and one I'm not ignoring. I'm just concerned that (going by F21 Alpha and Beta) the "hero testing" doesn't result in avoiding a slip most of the time. In the case of Alpha, that was going on for a month before we finally were able to release. That's not fair to QA and it *certainly* doesn't make it seem like something new contributors would want to put themselves through.
On Mon, Nov 24, 2014 at 8:46 PM, Stephen Gallagher sgallagh@redhat.com wrote:
Yeah, that's a valid concern and one I'm not ignoring. I'm just concerned that (going by F21 Alpha and Beta) the "hero testing" doesn't result in avoiding a slip most of the time. In the case of Alpha, that was going on for a month before we finally were able to release. That's not fair to QA and it *certainly* doesn't make it seem like something new contributors would want to put themselves through.
If your goal is preventing slips you are doing it wrong (tm). Your proposal would as Kevin said just result into *more* slips. What we should do is to find out *why* we slip every time and address that. The handling of the Go/NoGo meeting isn't really the problem, you are fighting the symptoms instead of the disease.
So you'd have to 1) find out what causes us to slip so often (*cough* anaconda *cough* [1]) and 2) talk to the related developers / involved parties to find a way to solve it in a way that is acceptable to both sides (in that example rel eng / qa / anaconda devs).
1: Ok I didn't check the data but my impression is that most blocker bugs are in that area I might be wrong though ... but the data is available to check that.
On Tue, 2014-11-25 at 10:21 +0100, drago01 wrote:
On Mon, Nov 24, 2014 at 8:46 PM, Stephen Gallagher sgallagh@redhat.com wrote:
Yeah, that's a valid concern and one I'm not ignoring. I'm just concerned that (going by F21 Alpha and Beta) the "hero testing" doesn't result in avoiding a slip most of the time. In the case of Alpha, that was going on for a month before we finally were able to release. That's not fair to QA and it *certainly* doesn't make it seem like something new contributors would want to put themselves through.
If your goal is preventing slips you are doing it wrong (tm). Your proposal would as Kevin said just result into *more* slips. What we should do is to find out *why* we slip every time and address that. The handling of the Go/NoGo meeting isn't really the problem, you are fighting the symptoms instead of the disease.
So you'd have to 1) find out what causes us to slip so often (*cough* anaconda *cough* [1]) and 2) talk to the related developers / involved parties to find a way to solve it in a way that is acceptable to both sides (in that example rel eng / qa / anaconda devs).
Well, that's actually one piece this is trying to address. By publicizing and making clear that "Developers must have their packages *submitted for stable*" at a specific time, we help those developers schedule their time more accurately.
As I said before, part of the problem is that most developers (anaconda included, I suspect) have been operating under the impression that as long as the package is prepped before *Thursday*, it's all good. But this doesn't allow time for adequate testing and discovery of remaining issues.
1: Ok I didn't check the data but my impression is that most blocker bugs are in that area I might be wrong though ... but the data is available to check that.
----- Original Message -----
On Tue, 2014-11-25 at 10:21 +0100, drago01 wrote:
On Mon, Nov 24, 2014 at 8:46 PM, Stephen Gallagher sgallagh@redhat.com wrote:
Yeah, that's a valid concern and one I'm not ignoring. I'm just concerned that (going by F21 Alpha and Beta) the "hero testing" doesn't result in avoiding a slip most of the time. In the case of Alpha, that was going on for a month before we finally were able to release. That's not fair to QA and it *certainly* doesn't make it seem like something new contributors would want to put themselves through.
If your goal is preventing slips you are doing it wrong (tm). Your proposal would as Kevin said just result into *more* slips. What we should do is to find out *why* we slip every time and address that. The handling of the Go/NoGo meeting isn't really the problem, you are fighting the symptoms instead of the disease.
So you'd have to 1) find out what causes us to slip so often (*cough* anaconda *cough* [1]) and 2) talk to the related developers / involved parties to find a way to solve it in a way that is acceptable to both sides (in that example rel eng / qa / anaconda devs).
Well, that's actually one piece this is trying to address. By publicizing and making clear that "Developers must have their packages *submitted for stable*" at a specific time, we help those developers schedule their time more accurately.
As I said before, part of the problem is that most developers (anaconda included, I suspect) have been operating under the impression that as long as the package is prepped before *Thursday*, it's all good. But this doesn't allow time for adequate testing and discovery of remaining issues.
From my experience, many misunderstandings comes from "release is next
Tuesday, it will be ready on Monday". So even one milestone later;-) The good thing is, Adam is usually sending email to devel list as reminder. And it actually follows what you proposed :). Asking devels to fix stuff by Tuesday, info we would have to slip in case it's not etc. Just it never was strict deadline and many times it was in the way - hey, we're so close, let's try to be crazy heroes, we can make it. So yes, we already do almost what you proposed, just it's not enforced. If there's demand for it, I'm not against it. It's a bit less flexible but can lower pressure on many folks (but prolong release -> more pressure but spread in more time).
Jaroslav
1: Ok I didn't check the data but my impression is that most blocker bugs are in that area I might be wrong though ... but the data is available to check that.
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On Fri, 2014-10-31 at 11:17 -0400, Stephen Gallagher wrote:
I talked to several people over the last couple days about what we can do to try to avoid the "hero testing" treadmill that we've been on during every Freeze in recent memory (specifically that we're usually fixing Blocker bugs until the day before the Go/No-Go meeting and that means that our QA team is pulling all-night test runs basically every week).
I made the following suggestion to both Adam Williamson and David Cantrell over the last couple of days and both seemed to think that this is a reasonable approach (but for different reasons, interestingly).
I honestly don't recall that you did, though I don't know that it's you or me that's mistaken. In any case I have no recollection of having seen or commented on this idea before, positively or negatively.
In any case, my personal opinion should of course never be mistaken for QA's official opinion.
-
Adam Williamson -
Bruno Wolff III -
drago01 -
Jaroslav Reznik -
Kevin Kofler -
Matthew Miller -
Michael Catanzaro -
Stephen Gallagher