== Preview Release ==
Known issues:
* PPC had a variety of issues o oversized o installed the wrong kernel o failed to install a bootloader * assorted anaconda partitioning issues
Discussed maybe using a separate config for PPC to keep it under size constraints, but it was decided to stay with one config.
== Deltarpm for F11 ==
Work needs done to either compose updates in a chroot (which has the F11 deltarpm support) or to backport it to the OS release used to generate updates. Seth Vidal is going to investigate which of these makes more sense. Given the timeframe, this is tight for F11 final. rawhide will continue to have deltas, as that's a separate compose process.
== F12 schedule ==
The schedule proposed by John Poelstra for Fedora 12 in https://fedorahosted.org/rel-eng/ticket/1271 was reviewed. The following changes were approved:
* the alpha milestone was removed entirely * due to conferences such as the Red Hat Summit, LinuxCon, and Linux Plumber's Conference, each milestone from 'Final freeze: development' (2009-09-15) should be shifted out one week.
This pushes GA from 2009-10-27 to 2009-11-03. The schedule will be presented for FESCo discussion at the 2009-05-08 meeting.
For more information on any of these, see the full transcript at: https://fedoraproject.org/wiki/ReleaseEngineering/Meetings/2009-may-04
Bill Nottingham said the following on 05/04/2009 01:01 PM Pacific Time:
== Preview Release ==
Known issues:
* PPC had a variety of issues o oversized o installed the wrong kernel o failed to install a bootloader * assorted anaconda partitioning issuesDiscussed maybe using a separate config for PPC to keep it under size constraints, but it was decided to stay with one config.
== Deltarpm for F11 ==
Work needs done to either compose updates in a chroot (which has the F11 deltarpm support) or to backport it to the OS release used to generate updates. Seth Vidal is going to investigate which of these makes more sense. Given the timeframe, this is tight for F11 final. rawhide will continue to have deltas, as that's a separate compose process.
== F12 schedule ==
The schedule proposed by John Poelstra for Fedora 12 in https://fedorahosted.org/rel-eng/ticket/1271 was reviewed. The following changes were approved:
* the alpha milestone was removed entirely
Reading the IRC log am I correct in understanding that a more detailed summary is: "Remove all alpha release tasks from the schedule. There will be no alpha release because it does not provide enough value for the effort required to create it. There is little public testing value from it either." ?
1) What dates are we proposing for releasing "development snapshots" before the beta? We should put these on the schedule now.
2) The alpha release has always been a good first opportunity to start marketing our next release, sending out press releases, etc. Basically, drawing attention to the fact with the general public that a new release is in the works. Without an Alpha the first general press releases would be a month later. Is this okay?
The Alpha also naturally gets the release notes process and other parts of Fedora going (not development focused tasks) early which is a good thing. We'd be losing that too.
3) If we do away with Alpha as we know it, leaving two test releases, can we simply call them "Alpha" and "Beta"? I've always thought "Preview Release" was a funny name for a test release and I think the terms "Alpha" and "Beta" are more familiar to the general public.
Thanks, John
* due to conferences such as the Red Hat Summit, LinuxCon, and Linux Plumber's Conference, each milestone from 'Final freeze: development' (2009-09-15) should be shifted out one week.This pushes GA from 2009-10-27 to 2009-11-03. The schedule will be presented for FESCo discussion at the 2009-05-08 meeting.
For more information on any of these, see the full transcript at: https://fedoraproject.org/wiki/ReleaseEngineering/Meetings/2009-may-04
John Poelstra (poelstra@redhat.com) said:
* the alpha milestone was removed entirelyReading the IRC log am I correct in understanding that a more detailed summary is: "Remove all alpha release tasks from the schedule. There will be no alpha release because it does not provide enough value for the effort required to create it. There is little public testing value from it either." ?
- What dates are we proposing for releasing "development snapshots"
before the beta? We should put these on the schedule now.
Not yet determined.
(Skipping over marketing)
The Alpha also naturally gets the release notes process and other parts of Fedora going (not development focused tasks) early which is a good thing. We'd be losing that too.
Is there no way for these to be started without a milestone?
- If we do away with Alpha as we know it, leaving two test releases,
can we simply call them "Alpha" and "Beta"? I've always thought "Preview Release" was a funny name for a test release and I think the terms "Alpha" and "Beta" are more familiar to the general public.
Maybe 'beta 1' and 'beta 2'. Given that we're feature frozen, calling the first milestone 'alpha' seems odd; similarly, given the tree is frozen, calling the second one 'beta' doesn't quite fit.
Bill
On Wed, 2009-05-06 at 05:45 -0700, John Poelstra wrote:
Reading the IRC log am I correct in understanding that a more detailed summary is: "Remove all alpha release tasks from the schedule. There will be no alpha release because it does not provide enough value for the effort required to create it. There is little public testing value from it either." ?
Not fully known yet. Basically because 12 is such a short cycle, I was looking for ways to maximize uninterrupted development time. The best way I could do that was to drop the Alpha cycle. While already a non-blocking freeze, it still drew too much attention away from ongoing development in order to deliver something that was weeks old and already irrelevant. The alpha has had dubious quality to the development cycle, at least from the developer and tester POV. About the only thing of quality it provided was a "known good starting point" for which to install and then update to rawhide, and even that hasn't been true for large swaths of folks in recent releases.
- What dates are we proposing for releasing "development snapshots"
before the beta? We should put these on the schedule now.
I honestly hadn't planned on doing regular snapshots during this period. Instead I was hoping for some test-days to drive the need for live images and/or full media images for a particular test target. For people looking for a good "jumping off" point, they can install the GA of F11 and yum update to rawhide.
- The alpha release has always been a good first opportunity to start
marketing our next release, sending out press releases, etc. Basically, drawing attention to the fact with the general public that a new release is in the works. Without an Alpha the first general press releases would be a month later. Is this okay?
I think so. Alpha rarely had any features in a state that we'd want to start showing people anyway. Traditionally Alpha releases are internal only and useful for whitebox testing. We can get Alpha level testing out of the rawhide snapshots.
The Alpha also naturally gets the release notes process and other parts of Fedora going (not development focused tasks) early which is a good thing. We'd be losing that too.
Yep, that is indeed something to consider.
- If we do away with Alpha as we know it, leaving two test releases,
can we simply call them "Alpha" and "Beta"? I've always thought "Preview Release" was a funny name for a test release and I think the terms "Alpha" and "Beta" are more familiar to the general public.
Honestly, I don't think Alpha fits there. Our existing Beta release is when we want to start blackbox public testing of our features, and that isn't changing. That's still the point where we want wide public usage and testing of our features and release, to gather bugs and feedback for the second half of development, the bug fixing.
I tend to agree that "Preview" is a bit of an odd name, however I refuse to call it "Release Candidate" like some other projects do, as it really isn't such a thing. There is 0 chance that our Preview release could become the final release. So we have to call it something. Maybe Gamma or Delta or Omega or Zenith would fit here, although not as widely used. It's the last snapshot mirrored before release candidates are actually created and a release candidate is picked as is to "release to manufacturing" (which in our case means stage for the mirrors).
Frankly we can play with the names a bit as we go or after we pick the dates, but the rough dates are what we're trying to pass through now.
On Wed, May 06, 2009 at 08:56:36 -0700, Jesse Keating jkeating@redhat.com wrote:
I honestly hadn't planned on doing regular snapshots during this period. Instead I was hoping for some test-days to drive the need for live images and/or full media images for a particular test target. For people looking for a good "jumping off" point, they can install the GA of F11 and yum update to rawhide.
One thing that would help compensate for no alpha, is having usable boot.iso images almost allt of the time. The last couple rawhides there have been significant stretches when there were either no boot.iso images being produced or the current ones weren't usable (or at least not for some types of installs).
I think there was talk at one time of keeping usable ones around for a while in case of either failed or unusable builds.
Bruno Wolff III (bruno@wolff.to) said:
On Wed, May 06, 2009 at 08:56:36 -0700, Jesse Keating jkeating@redhat.com wrote:
I honestly hadn't planned on doing regular snapshots during this period. Instead I was hoping for some test-days to drive the need for live images and/or full media images for a particular test target. For people looking for a good "jumping off" point, they can install the GA of F11 and yum update to rawhide.
One thing that would help compensate for no alpha, is having usable boot.iso images almost allt of the time. The last couple rawhides there have been significant stretches when there were either no boot.iso images being produced or the current ones weren't usable (or at least not for some types of installs).
I think there was talk at one time of keeping usable ones around for a while in case of either failed or unusable builds.
Generally¹, using the prior release's boot.iso and changing/editing the repos to point at rawhide should work reasonably well.
Bill
¹: modulo rpm hash changes, etc.
Jesse Keating said the following on 05/06/2009 08:56 AM Pacific Time:
On Wed, 2009-05-06 at 05:45 -0700, John Poelstra wrote:
Reading the IRC log am I correct in understanding that a more detailed summary is: "Remove all alpha release tasks from the schedule. There will be no alpha release because it does not provide enough value for the effort required to create it. There is little public testing value from it either." ?
Not fully known yet. Basically because 12 is such a short cycle, I was looking for ways to maximize uninterrupted development time. The best
The overall Fedora 12 schedule with a GA date of 2009-11-03 is 21 days shorter than Fedora 11 and exactly the same length as Fedora 8 so it is not unusually short.
way I could do that was to drop the Alpha cycle. While already a non-blocking freeze, it still drew too much attention away from ongoing development in order to deliver something that was weeks old and already irrelevant. The alpha has had dubious quality to the development cycle, at least from the developer and tester POV. About the only thing of quality it provided was a "known good starting point" for which to install and then update to rawhide, and even that hasn't been true for large swaths of folks in recent releases.
I'd be curious to see our torrent and download numbers for the F11 Alpha to understand how insignificant it was. Where can we find them?
Although it is simple to just "not do the Alpha" it will take more coordination across the other Fedora teams AND good messaging in the press and wider world that no alpha is coming for Fedora 12 along with the reasons why. Have we considered the cost of this trade-off to our brand and community?
- What dates are we proposing for releasing "development snapshots"
before the beta? We should put these on the schedule now.
I honestly hadn't planned on doing regular snapshots during this period. Instead I was hoping for some test-days to drive the need for live images and/or full media images for a particular test target. For people looking for a good "jumping off" point, they can install the GA of F11 and yum update to rawhide.
No snapshots at all or just not before Beta? Here was the original draft which I can change to reflect the "no alpha" scenario if it goes forward: http://poelstra.fedorapeople.org/schedules/f-12/f-12-releng-tasks.html
If we plan to push out final freeze and corresponding GA by one week are we proposing to start the beta a week later too? Or have one week longer between public beta release and final freeze?
Thanks, John
On Wed, 2009-05-06 at 16:52 -0700, John Poelstra wrote:
The overall Fedora 12 schedule with a GA date of 2009-11-03 is 21 days shorter than Fedora 11 and exactly the same length as Fedora 8 so it is not unusually short.
Well 21 days is a lot of days to take in, particularly when a lot of them seemed to come between the GA of 11 and the Alpha freeze of 12.
way I could do that was to drop the Alpha cycle. While already a non-blocking freeze, it still drew too much attention away from ongoing development in order to deliver something that was weeks old and already irrelevant. The alpha has had dubious quality to the development cycle, at least from the developer and tester POV. About the only thing of quality it provided was a "known good starting point" for which to install and then update to rawhide, and even that hasn't been true for large swaths of folks in recent releases.
I'd be curious to see our torrent and download numbers for the F11 Alpha to understand how insignificant it was. Where can we find them?
Torrent numbers are at http://spins.fedoraproject.org:6969/ but downloads alone don't show the whole picture. By the time people got to the Alpha bits, the general feeling I have is that most of the bugs in alpha were already known, and the solution was to update to rawhide. Even if the bugs weren't fixed, the first suggestion was always update to rawhide, so rawhide is really the target we're after, alpha just appeared to be an easy way to get there.
Although it is simple to just "not do the Alpha" it will take more coordination across the other Fedora teams AND good messaging in the press and wider world that no alpha is coming for Fedora 12 along with the reasons why. Have we considered the cost of this trade-off to our brand and community?
Hard to judge the cost, however our "three tests and a release" which turned into "Alpha Beta Preview Release" was really designed before we had things such as Test Days and the easily available live images, or the ability to easily compose regular install images with slightly different package sets. The prohibitive cost to doing images on demand has gone down considerably, and combined with test days provides a better harness for gathering feedback as we go than the date based quickly stagnant snapshots.
- What dates are we proposing for releasing "development snapshots"
before the beta? We should put these on the schedule now.
I honestly hadn't planned on doing regular snapshots during this period. Instead I was hoping for some test-days to drive the need for live images and/or full media images for a particular test target. For people looking for a good "jumping off" point, they can install the GA of F11 and yum update to rawhide.
No snapshots at all or just not before Beta?
Just none planned before Beta.
Here was the original draft which I can change to reflect the "no alpha" scenario if it goes forward: http://poelstra.fedorapeople.org/schedules/f-12/f-12-releng-tasks.html
If we plan to push out final freeze and corresponding GA by one week are we proposing to start the beta a week later too? Or have one week longer between public beta release and final freeze?
One week longer between public beta release and final release. The dates leading up to the Beta are fine. I'd still like to only have 3 snapshots, so we can start snapshot 1 a week later as well.
Jesse Keating said the following on 05/06/2009 05:04 PM Pacific Time:
On Wed, 2009-05-06 at 16:52 -0700, John Poelstra wrote:
The overall Fedora 12 schedule with a GA date of 2009-11-03 is 21 days shorter than Fedora 11 and exactly the same length as Fedora 8 so it is not unusually short.
Well 21 days is a lot of days to take in, particularly when a lot of them seemed to come between the GA of 11 and the Alpha freeze of 12.
I still don't understand why that matters considering the amount of development and changes that happen up until GA no matter where we are in the release process. Even now after final freeze lots of changes are being ACKd on releng-list and very few if any are getting NAKd.
way I could do that was to drop the Alpha cycle. While already a non-blocking freeze, it still drew too much attention away from ongoing development in order to deliver something that was weeks old and already irrelevant. The alpha has had dubious quality to the development cycle, at least from the developer and tester POV. About the only thing of quality it provided was a "known good starting point" for which to install and then update to rawhide, and even that hasn't been true for large swaths of folks in recent releases.
I'd be curious to see our torrent and download numbers for the F11 Alpha to understand how insignificant it was. Where can we find them?
Torrent numbers are at http://spins.fedoraproject.org:6969/ but downloads alone don't show the whole picture. By the time people got to the Alpha bits, the general feeling I have is that most of the bugs in alpha were already known, and the solution was to update to rawhide. Even if the bugs weren't fixed, the first suggestion was always update to rawhide, so rawhide is really the target we're after, alpha just appeared to be an easy way to get there.
If I'm reading the information correctly there, 10,000+ people got the Fedora 11 alpha.
I would think that giving 10,000 people an easy starting place for a release would be a good thing.
How will we measure--not assert or subjectively decide as we are doing here and have done in the past--whether a change we have made helps our releases or if time is better spent somewhere else?
It seems that we continue to experiment and tweak the schedule with different durations, freezes, etc. but we rarely collect any hard data to evaluate whether or not past tweaks were beneficial or not.
If we can't really measure our progress or lack thereof in tangible terms, how do we really know we should continue to tweak things?
Although it is simple to just "not do the Alpha" it will take more coordination across the other Fedora teams AND good messaging in the press and wider world that no alpha is coming for Fedora 12 along with the reasons why. Have we considered the cost of this trade-off to our brand and community?
Hard to judge the cost, however our "three tests and a release" which turned into "Alpha Beta Preview Release" was really designed before we had things such as Test Days and the easily available live images, or the ability to easily compose regular install images with slightly different package sets. The prohibitive cost to doing images on demand has gone down considerably, and combined with test days provides a better harness for gathering feedback as we go than the date based quickly stagnant snapshots.
This sounds good, but how do we really know this is true? What data can we point to? Maybe we are focusing on the wrong problem and should instead focus on the amount of churn and changes during development.
John
On Thu, 2009-05-07 at 13:15 -0700, John Poelstra wrote:
Jesse Keating said the following on 05/06/2009 05:04 PM Pacific Time:
On Wed, 2009-05-06 at 16:52 -0700, John Poelstra wrote:
The overall Fedora 12 schedule with a GA date of 2009-11-03 is 21 days shorter than Fedora 11 and exactly the same length as Fedora 8 so it is not unusually short.
Well 21 days is a lot of days to take in, particularly when a lot of them seemed to come between the GA of 11 and the Alpha freeze of 12.
I still don't understand why that matters considering the amount of development and changes that happen up until GA no matter where we are in the release process. Even now after final freeze lots of changes are being ACKd on releng-list and very few if any are getting NAKd.
It's a sad but true state of things. Just because we freeze doesn't mean people stop finding bugs, and if we have time to take in the bugfixes, we will, as long as they aren't threatening stability. The freeze is there to prevent the unstable things from accidentally making it in. This made even more sense when we didn't branch for the release until just before GA, now we branch a lot earlier and there is less risk of unstable things slipping in, but there are still plenty of builds that we'd rather not see hit the tree.
The freeze override process exists to add a filter between developer and repo to catch the obvious and to help maintainers think a moment about what they're doing and whether it's acceptable to do in a freeze period. it would make a lot more sense if developers would treat updates to a release with the same care as they treat updates during a freeze period, but that's a fight for another day.
way I could do that was to drop the Alpha cycle. While already a non-blocking freeze, it still drew too much attention away from ongoing development in order to deliver something that was weeks old and already irrelevant. The alpha has had dubious quality to the development cycle, at least from the developer and tester POV. About the only thing of quality it provided was a "known good starting point" for which to install and then update to rawhide, and even that hasn't been true for large swaths of folks in recent releases.
I'd be curious to see our torrent and download numbers for the F11 Alpha to understand how insignificant it was. Where can we find them?
Torrent numbers are at http://spins.fedoraproject.org:6969/ but downloads alone don't show the whole picture. By the time people got to the Alpha bits, the general feeling I have is that most of the bugs in alpha were already known, and the solution was to update to rawhide. Even if the bugs weren't fixed, the first suggestion was always update to rawhide, so rawhide is really the target we're after, alpha just appeared to be an easy way to get there.
If I'm reading the information correctly there, 10,000+ people got the Fedora 11 alpha.
I would think that giving 10,000 people an easy starting place for a release would be a good thing.
That's assuming that those 10,000 people couldn't easily use F11 GA and then upgrade to rawhide. I don't think you can draw the conclusion that "10K people got Alpha, if we drop Alpha we'll lose 10K people".
How will we measure--not assert or subjectively decide as we are doing here and have done in the past--whether a change we have made helps our releases or if time is better spent somewhere else?
We just have to go on gut feeling, like just about everything else in Fedora. We have to listen to maintainer feedback, and our QA team feedback and make a judgment call. If you have better ideas, I'm all ears. The message I've consistently gotten from our developers is less freeze points, and the non-blocking freeze of alpha wasn't good enough. QA has expressed that Alpha isn't really helpful too. So now we're talking about dropping it, both to save some downtime in F12 schedule and to evaluate if we really need an Alpha snapshot.
It seems that we continue to experiment and tweak the schedule with different durations, freezes, etc. but we rarely collect any hard data to evaluate whether or not past tweaks were beneficial or not.
What data is there to collect? So many other things change at the same time, there is really no scientific way to experiment, measure, tweak, and experiment and measure again, showing a difference with only one variable change.
If we can't really measure our progress or lack thereof in tangible terms, how do we really know we should continue to tweak things?
Feedback from maintainers and users.
Although it is simple to just "not do the Alpha" it will take more coordination across the other Fedora teams AND good messaging in the press and wider world that no alpha is coming for Fedora 12 along with the reasons why. Have we considered the cost of this trade-off to our brand and community?
Hard to judge the cost, however our "three tests and a release" which turned into "Alpha Beta Preview Release" was really designed before we had things such as Test Days and the easily available live images, or the ability to easily compose regular install images with slightly different package sets. The prohibitive cost to doing images on demand has gone down considerably, and combined with test days provides a better harness for gathering feedback as we go than the date based quickly stagnant snapshots.
This sounds good, but how do we really know this is true? What data can we point to? Maybe we are focusing on the wrong problem and should instead focus on the amount of churn and changes during development.
John
Amount of churn and change is something I can't directly change. I can't even get it to change for a stable release, there is no way I can get it to change for rawhide. Given our sharp upward trend in new packages and number of packages, the change and churn is only going to get worse. So I focus on changing the things I can control, which involve schedules, freeze strategies, etc..
On Thu, 2009-05-07 at 13:56 -0700, Jesse Keating wrote:
The freeze override process exists to add a filter between developer and repo to catch the obvious and to help maintainers think a moment about what they're doing and whether it's acceptable to do in a freeze period.
I think this works well. It forces one briefly explain the benefit, and justify the risk, of the update. That alone makes it less likely you're going to screw things up with a hugely risky, or low value, update.
I'd still like to make the process a little more visible, though. These freeze break requests are fairly central to the development process in the lead up to GA. It should be easier for folks to follow what's going on than running a trac query.
it would make a lot more sense if developers would treat updates to a release with the same care as they treat updates during a freeze period, but that's a fight for another day.
Agreed. It seems logical that post-GA updates would go through a similar process.
It could be as simple as two new fields in bodhi - "Benefit to users" and "Risk of regressions" - and the ability for folks to jump in and say "wait, that doesn't make sense" before an update gets pushed.
Cheers, Mark.
On Friday, May 08 2009, Mark McLoughlin said:
On Thu, 2009-05-07 at 13:56 -0700, Jesse Keating wrote:
The freeze override process exists to add a filter between developer and repo to catch the obvious and to help maintainers think a moment about what they're doing and whether it's acceptable to do in a freeze period.
I think this works well. It forces one briefly explain the benefit, and justify the risk, of the update. That alone makes it less likely you're going to screw things up with a hugely risky, or low value, update.
I'd still like to make the process a little more visible, though. These freeze break requests are fairly central to the development process in the lead up to GA. It should be easier for folks to follow what's going on than running a trac query.
it would make a lot more sense if developers would treat updates to a release with the same care as they treat updates during a freeze period, but that's a fight for another day.
Agreed. It seems logical that post-GA updates would go through a similar process.
It could be as simple as two new fields in bodhi - "Benefit to users" and "Risk of regressions" - and the ability for folks to jump in and say "wait, that doesn't make sense" before an update gets pushed.
This makes a fair bit of sense to me. Although in theory, the benefit should be captured by the description, having it be more explicit can't hurt.
Another nice thing is that might then be a step in the direction of having bodhi used to propose freeze updates which has been something that has been talked about off and on for a while. And perhaps that helps to kill two threads with one stone. This one and also it then would change "stable" updates for an unreleased release be the release stream... and by making testing available and using bodhi, it would help with your visibility point above.
Jeremy
On Thu, 2009-05-07 at 13:56 -0700, Jesse Keating wrote:
How will we measure--not assert or subjectively decide as we are
doing
here and have done in the past--whether a change we have made helps
our
releases or if time is better spent somewhere else?
We just have to go on gut feeling, like just about everything else in Fedora. We have to listen to maintainer feedback, and our QA team feedback and make a judgment call. If you have better ideas, I'm all ears. The message I've consistently gotten from our developers is less freeze points, and the non-blocking freeze of alpha wasn't good enough. QA has expressed that Alpha isn't really helpful too.
From a QA perspective, I don't know if I would say that the Alpha isn't helpful. I do think that each milestone puts a stake in the ground and provides project-wide focus around a shared goal that daily rawhide doesn't demand now.
So now we're talking about dropping it, both to save some downtime in F12 schedule and to evaluate if we really need an Alpha snapshot.
I'm uncertain as to whether removing the Alpha milestone will result in improved quality without some thinking/planning on how we should invest the extra time.
One thought was ... in the absence of an Alpha, QA can schedule several test days. However, experience shows that not all test days have the same starting conditions. During F10 and F11, several test days experienced 'launch failure'. That is, we either couldn't generate a usable live image, or the steps required to prepare the test environment required enough forks in the road that it became a barrier to participation.
Another investment for QA would be to provide more data on the health of rawhide. This doesn't directly influence quality, but more a foot in the door when it comes to measuring quality.
Without the Alpha, how do we entice the distro to come together? How do we ensure that we aren't shifting the bugs we find in an Alpha now, to the Beta? Should we hold the F12 beta to the same/higher/lower standards that we hold the current Beta?
Thanks, James
On Fri, 2009-05-08 at 12:26 -0400, James Laska wrote:
One thought was ... in the absence of an Alpha, QA can schedule several test days. However, experience shows that not all test days have the same starting conditions. During F10 and F11, several test days experienced 'launch failure'. That is, we either couldn't generate a usable live image, or the steps required to prepare the test environment required enough forks in the road that it became a barrier to participation.
Having more visibility and more coordinated planning on upcoming test days can help us focus to get the required bits in good shape for those days, be it anaconda, be it livecd-tools, whatever. These are lower cost then grinding everything to a halt (as far as the freeze is concerned) for a week or more.
Another investment for QA would be to provide more data on the health of rawhide. This doesn't directly influence quality, but more a foot in the door when it comes to measuring quality.
Without the Alpha, how do we entice the distro to come together? How do we ensure that we aren't shifting the bugs we find in an Alpha now, to the Beta? Should we hold the F12 beta to the same/higher/lower standards that we hold the current Beta?
Yes, we should hold Beta to the same or higher standard.
On Fri, May 08, 2009 at 09:46:36AM -0700, Jesse Keating wrote:
On Fri, 2009-05-08 at 12:26 -0400, James Laska wrote:
One thought was ... in the absence of an Alpha, QA can schedule several test days. However, experience shows that not all test days have the same starting conditions. During F10 and F11, several test days experienced 'launch failure'. That is, we either couldn't generate a usable live image, or the steps required to prepare the test environment required enough forks in the road that it became a barrier to participation.
Having more visibility and more coordinated planning on upcoming test days can help us focus to get the required bits in good shape for those days, be it anaconda, be it livecd-tools, whatever. These are lower cost then grinding everything to a halt (as far as the freeze is concerned) for a week or more.
Another investment for QA would be to provide more data on the health of rawhide. This doesn't directly influence quality, but more a foot in the door when it comes to measuring quality.
Without the Alpha, how do we entice the distro to come together? How do we ensure that we aren't shifting the bugs we find in an Alpha now, to the Beta? Should we hold the F12 beta to the same/higher/lower standards that we hold the current Beta?
Yes, we should hold Beta to the same or higher standard.
What standard should that be?
Are there some metrics we could use to measure against that standard?
There must be some factors we can identify that constitute a "successful" Alpha vs. Beta (or Beta vs. Preview, or...). For instance, you could say a successful test release results in filing of bugs. When it comes to those bugs, what would interest us to know about their distribution between one test phase and the next, or from test release to GA?
On Fri, 2009-05-08 at 15:54 -0400, Paul W. Frields wrote:
What standard should that be?
Is installable by the majority of people who try it. Is capable of being upgraded. Is useful for meaningful testing of the proposed features. Is released in a timely manner without (too much) delay. Is "well received".
Are there some metrics we could use to measure against that standard?
Some are measurable, some aren't. We've been making judgment calls on "success/failure" of test releases since long before Fedora existed. We continue to make those judgment calls based on reception and perception.
There must be some factors we can identify that constitute a "successful" Alpha vs. Beta (or Beta vs. Preview, or...). For instance, you could say a successful test release results in filing of bugs. When it comes to those bugs, what would interest us to know about their distribution between one test phase and the next, or from test release to GA?
That's really tough to measure. Finding bugs in a alpha or beta is a good and bad thing. Good that they were found, bad that they existed or weren't found in rawhide earlier. Definitely bad at GA but then again Fedora is a bleeding edge distro where things are tested out in a wide fashion, so finding bugs in a Fedora GA release is actually a good thing for upstreams and later downstreams.
-
Bill Nottingham -
Bruno Wolff III -
James Laska -
Jeremy Katz -
Jesse Keating -
John Poelstra -
Mark McLoughlin -
Paul W. Frields