Our current upgrade criterion says: "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed." https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Upgrade_requi...
Currently we have no criterion that would cover upgrading across 2 releases, e.g. from F21 to F23 (directly, not one by one). But in the real world this very often happens. It's even one of the reasons we support our releases until N+2 release is available + 1 month (i.e. F21 is supported until F23 is out + 1 month). The often cited reason is for people to be upgrading just once per year (and have one month to do that). And of course many (probably most) of them don't upgrade one by one, but skip a release.
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
"For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of any of the two previous stable Fedora releases with that package set installed." (language corrections very welcome)
We can discuss whether N+2 upgrading should be a separate Final criterion, not joined with the Beta one. I don't feel strongly either way.
I'd also set up a new test case in our installation matrix in the upgrade section: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade Something like QA:Testcase_upgrade_dnf_skip_release. The question is whether to have just a single test case and let people choose which package set they test, or whether to pick some particular package set. We probably don't want to test all combinations, at least not manually. Just a single "please test something" test case would be satisfactory here, I think. Something will get tested, and we will block on important bugs we discover, that's the important change.
If we decide to not go this route for some reason, I think we should adjust our tools (system-upgrade) and documentation (wiki, fedora docs) and provide very clear and visible warning that the only officially supported means of upgrading is to go up releases one by one. And that skipping releases might be dangerous (considerably more than doing it the recommended way). Because I feel we would be doing our users a disservice if we neither tested skipping releases nor warned them against doing that.
Thoughts?
Kamil
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
"For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of any of the two previous stable Fedora releases with that package set installed." (language corrections very welcome)
During QA meeting, people were generally in favor of this idea. I was asked to check with Will Woods, system-upgrade maintainer. I did that, and he has no objections. He says it's no extra work for him and that the most work is needed from package maintainers when designing dependencies, scriptlets, etc. This is something to consider as well, but I think we implicitly want all packages to be upgradable forward (even when skipping a release). And I don't remember any issues with this lately.
Does someone have some additional concerns about this?
Thanks, Kamil
On Mon, 2015-11-23 at 13:26 -0500, Kamil Paral wrote:
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
"For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of any of the two previous stable Fedora releases with that package set installed." (language corrections very welcome)
During QA meeting, people were generally in favor of this idea. I was asked to check with Will Woods, system-upgrade maintainer. I did that, and he has no objections. He says it's no extra work for him and that the most work is needed from package maintainers when designing dependencies, scriptlets, etc. This is something to consider as well, but I think we implicitly want all packages to be upgradable forward (even when skipping a release). And I don't remember any issues with this lately.
Does someone have some additional concerns about this?
Nope - as I said in the meeting, if tools folks are OK, so am I. It may be worth floating on devel@, though, to see if any packagers are concerned about maintaining upgradability on that level.
I believe a failure to upgrade from N-2 to N should not block the N release. The reason is limited resources, both for tests and for changes to fix problems. These resources are more valuable applied to the N release than to something two releases in the past.
If someone wants to test a release-skipping upgrade, fine. Report problems? Sure. If someone wants to fix these problems, OK; if not, the policy should be "Upgrade one release at a time. Release-skipping upgrades may work, but are not guaranteed."
If "upgrade N-2 -> N-1" works, and "upgrade N-1 -> N" works, then "upgrade N-2 -> N" also works, right? Maybe not, and I think it profligate to insist we fix a broken two-release upgrade when two single-release upgrades successfully reach the desired target. Document, do not block.
I may hold a minority opinion, but this seems like another call for QA to do somebody else's job. Who should decide that release-skip upgrade is a Fedora imperative?
I believe a failure to upgrade from N-2 to N should not block the N release. The reason is limited resources, both for tests and for changes to fix problems. These resources are more valuable applied to the N release than to something two releases in the past.
If someone wants to test a release-skipping upgrade, fine. Report problems? Sure. If someone wants to fix these problems, OK; if not, the policy should be "Upgrade one release at a time. Release-skipping upgrades may work, but are not guaranteed."
That's exactly what I'm trying to improve here. Either make it a reliable feature, or warn the users against it very visibly, because this is an operation that can easily break their system heavily. We don't currently warn the users much, especially the system-upgrade tool doesn't contain any warnings like that. If we decide we're not going to support release skipping officially, users should always do a conscious and informed decision about this and be aware of the risks.
If "upgrade N-2 -> N-1" works, and "upgrade N-1 -> N" works, then "upgrade N-2 -> N" also works, right? Maybe not, and I think it profligate to insist we fix a broken two-release upgrade when two single-release upgrades successfully reach the desired target. Document, do not block.
I may hold a minority opinion, but this seems like another call for QA to do somebody else's job. Who should decide that release-skip upgrade is a Fedora imperative?
That would be FESCo. If more people hold your opinion, we should probably file a ticket and ask them. It's definitely a valid opinion. And it would definitely be easier for QA. But in this case, I believe the work is worth it, because I personally know many people who don't want to upgrade twice a year. And offering people a way to upgrade just once a year (easily, i.e. not going through consecutive upgrades) is something I consider essential for an operating system.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/24/2015 06:48 AM, Kamil Paral wrote:
That would be FESCo. If more people hold your opinion, we should probably file a ticket and ask them. It's definitely a valid opinion. And it would definitely be easier for QA. But in this case, I believe the work is worth it, because I personally know many people who don't want to upgrade twice a year. And offering people a way to upgrade just once a year (easily, i.e. not going through consecutive upgrades) is something I consider essential for an operating system.
With my FESCo hat on, I'll say this: I'd prefer that Fedora officially supports the two-release upgrade path. From anecdotal data, a significant percentage of our non-developer user-base is doing this already and is comfortable with the existing risks. Improving the situation can only serve to enhance our standing and reputation.
Historically, the reason we have not done this was twofold:
1) Way back when we were doing Anaconda-based upgrades, I think it amounted to a technological limitation (please correct me if my information is wrong here).
2) The QA impact is non-zero when attempting to support two-version upgrades, but since this recommendation is coming *from* prominent QA team members, I'm assuming that this is a non-issue.
Also, to add to the anecdotes, I've upgraded several family members directly from Fedora 21 to Fedora 23 in the last week with no issues whatsoever (of course, I also curate their repository selection, so they don't end up with incompatible packages). So I'd argue that we're already at the point where this is a real functional capability that we should just ensure doesn't get broken in the future (as opposed to needing to add new capabilities to enable two-version upgrades).
On Tue, 2015-11-24 at 07:52 -0500, Stephen Gallagher wrote:
- Way back when we were doing Anaconda-based upgrades, I think it
amounted to a technological limitation (please correct me if my information is wrong here).
I think it wasn't exactly technically *impossible*, but it was significantly more likely that some kind of problem would emerge, with that much heavier system.
- The QA impact is non-zero when attempting to support two-version
upgrades, but since this recommendation is coming *from* prominent QA team members, I'm assuming that this is a non-issue.
Not exactly a non-issue, but we at least have the ability to automate this testing quite effectively now.
On Tue, 24 Nov 2015 07:52:47 -0500, Stephen Gallagher wrote:
I've upgraded several family members directly from Fedora 21 to Fedora 23 in the last week with no issues whatsoever (of course, I also curate their repository selection, so they don't end up with incompatible packages).
Aye, there's the rub. It is not just update from an "original" installation of N-2 to N, it is update from N-2 plus whatever additional repository packages were installed plus additional non-repository software and user configuration that you want to upgrade. The starting condition is quite variable, and may include old versions of applications because these are familiar to the user or incompatible with more up-to-date software.
When system-upgrade fails (if you are lucky, this will be obvious; if unlucky, it may not be recognized until after the event), the only way back to the starting condition is a full system restore. Many users will back up their own data, and, if upgrade ends badly, plan to recover with a new installation (of N, N-1, or N-2) and necessary software, followed by restoration of their personal data. This is the most reliable upgrade strategy: no exposure to problems with old stuff, and it defines how to get from a well-known initial state to the user's working environment.
In this case it may be impossible to fix an upgrade problem, because there is no way to reconstruct the original state and verify successful upgrade is possible.
Of course, these considerations also apply to the single-release upgrade. The situation just gets worse the farther back one goes. And sometimes there simply is no upgrade path: the new software is just incompatible with the old. To use the new system, you must adapt to it. (Remember the agony voiced by some GNOME2 users when GNOME3 came along?)
Unlike new installation with its well-defined result, system-upgrade (with dependencies on user configuration and possibly incompatible pieces of software) necessarily has some vague result. It is undeniably valuable and extremely convenient, but unavoidably a second-class facility. We may insist system-upgrade of a newly installed N-1 to N works, and further insist system-upgrade of a newly installed N-2 to N works, but this is deceptive. Once a user configures and uses his system, system-upgrade is unpredictable.
To add my own to Mr. Gallagher's anecdote, my single attempt at "dnf system-upgrade" from 21 to 23 failed. After upgrade, my userid was not displayed on the login screen. Login in text mode was possible. After an hour or two tinkering with gdm configuration, I gave up, did a new install of 23, and configured it without difficulty. Several times I have used "dnf system-upgrade" from 22 to 23 without problems.
Conclusion: release-skipping upgrade is more difficult than single-release upgrade (and probably not worth the effort to fix.)
If anyone would like to investigate the problem I experienced, I am willing to install a new 21 system and see if I can reproduce the error.
On Tue, 2015-11-24 at 13:28 -0500, Richard Ryniker wrote:
On Tue, 24 Nov 2015 07:52:47 -0500, Stephen Gallagher wrote:
I've upgraded several family members directly from Fedora 21 to Fedora 23 in the last week with no issues whatsoever (of course, I also curate their repository selection, so they don't end up with incompatible packages).
Aye, there's the rub.  It is not just update from an "original" installation of N-2 to N, it is update from N-2 plus whatever additional repository packages were installed plus additional non-repository software and user configuration that you want to upgrade.
Not really. We have never really gone to any lengths to test or support N-1 upgrades with 3rd party repositories or non-repo software either. That's a different question.
Of course, these considerations also apply to the single-release upgrade. The situation just gets worse the farther back one goes.
I'm not really sure that logically follows. Why does the situation inevitably 'get worse' for N-2 upgrade?
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
On Tue, 24 Nov 2015 10:46:55 -0800, Adam Williamson wrote:
We have never really gone to any lengths to test or support N-1 upgrades with 3rd party repositories or non-repo software either. That's a different question.
From a user's perspective, the value of system-upgrade depends on its ability to move his system "as configured" to a new release. All of his personalization, including any non-fedora software, remains in place.
You are certainly right to except non-Fedora software from QA tests. The problem I perceive is the vast number of possible combinations of packages from just the Fedora repositories. There may be no need to test exhaustively, but any heuristic is likely to miss problems from time to time. It is likely only enough cases can be tested to distinguish "some system-upgrades work" from "system-upgrade is often impossible."
This is valuable to know, and I have no quibble with a QA criterion that says "system-upgrade is never possible" will block a release. I suspect there is no precise, useful definition that will tell a user whether his particular system will succeed with system-upgrade. All he can do is try and hope for the best. There might be documentation that says "these circumstances are likely to prevent successful system-upgrade" - the "common bugs" approach.
With any general claim "You can use system-upgrade to advance to the next (or next+1) release" there will be triage issues (how many users will experience problem 1; how many will encounter problem 2?) How many susceptible users must we estimate in order to block a release? I fear software developers will rightly claim they are busy with new features or current bugs and they cannot spend time to investigate compatibility problems from system-upgrade that simply do not appear when their sofware is newly installed.
I do not want a Fedora user to understand "You can use system-upgrade to migrate to a newer release" has the same confidence as "The new Fedora meets all release criteria."
Why does the situation inevitably 'get worse' for N-2 upgrade?
There are more possible initial conditions. Even a very sparse test plan requires more effort. Analysis of "How important is this case?" may be more difficult, and there are more cases. Instead of problems from six months of software evolution, there will be problems from 12 months, and the increase in number of problems over time is likely greater than linear.
I certainly do not want to abandon system-upgrade. I want users to understand we cannot test it well, probably cannot fix many failures, and, when it fails, we reccommend they report the problem then undertake a new installation of Fedora, configure it, and reload applications to meet their requirements.
"System-upgrade must work before release" is too vague ("work" is too difficult to define and test), too severe (important new function could be delayed for reasons no one is willing to fix), and will tell users who experience problems that Fedora is lower in quality than they hoped.
If users are told up front that system-upgrade is useful but cannot be tested or guaranteed for all cases, their expectations will be more realistic, therefore satisfaction with Fedora will be greater.
"dnf system-upgrade" is rather new. Maybe the Fedora organization should observe user experience for another release or two before it decides this is a critical, and therefore release-blocking, part of Fedora. I understand enthusiasm for the new, offline upgrade mechanism. More history will yield a better understanding of what problems can occur, and what costs must be borne to fix them.
On Tue, 2015-11-24 at 21:57 -0500, Richard Ryniker wrote:
On Tue, 24 Nov 2015 10:46:55 -0800, Adam Williamson wrote:
We have never really gone to any lengths to test or support N-1 upgrades with 3rd party repositories or non-repo software either.  That's a different question.
From a user's perspective, the value of system-upgrade depends on its ability to move his system "as configured" to a new release.  All of his personalization, including any non-fedora software, remains in place.
You are certainly right to except non-Fedora software from QA tests.  The problem I perceive is the vast number of possible combinations of packages from just the Fedora repositories.  There may be no need to test exhaustively, but any heuristic is likely to miss problems from time to time.  It is likely only enough cases can be tested to distinguish "some system-upgrades work" from "system-upgrade is often impossible."
This is valuable to know, and I have no quibble with a QA criterion that says "system-upgrade is never possible" will block a release.  I suspect there is no precise, useful definition that will tell a user whether his particular system will succeed with system-upgrade.
Yes, this is basically the situation. We can only do so much by literally testing actual upgrades, by hand or automated. Things like the daily dependency checks and the taskotron depcheck test can help more. But ultimately, there are so many permutations that we can never really offer a guaranteed smooth upgrade path (and in fact, AFAIK, no OS vendor - even those with far more resources than us - does this).
"dnf system-upgrade" is rather new.  Maybe the Fedora organization should observe user experience for another release or two before it decides this is a critical, and therefore release-blocking, part of Fedora.  I understand enthusiasm for the new, offline upgrade mechanism.  More history will yield a better understanding of what problems can occur, and what costs must be borne to fix them.
dnf-system-upgrade itself is new, but Fedora has always had a theoretically 'official supported' upgrade mechanism; it was anaconda, then fedup, now it's dnf-system-upgrade. dnf-system-upgrade was introduced specifically with the understanding that it replaced fedup at the same level of support.
I don't think this is enthusiasm for dnf-system-upgrade per se, just a kind of recognition of a gap in coverage that's been around for a while. We could probably have plausibly supported N+2 upgrades with fedup; certainly in practice we put effort into fixing them.
System-upgrade across two releases raises an interesting End-of-Life policy issue. If system-upgrade from N-2 to N is so important it will block release of N until it works, how do we explain why it is no longer important after four weeks when N-2 reaches End of Life?
Four weeks is little time to allow users who chose not to upgrade to N-1 to move from N-2 to N. Do we say "After four weeks, the only supported mechanismm is new installation of N"? That is clear and simple, consistent with the EOL schedule, but denies system-upgrade is such a critical facility.
Should system-upgrade be formally supported for three releases (18 months?) That would tell users they may count on it for their upgrade plan, but it is a significant change that could involve FESCo.
On Wed, 2015-11-25 at 09:46 -0500, Richard Ryniker wrote:
System-upgrade across two releases raises an interesting End-of-Life policy issue.  If system-upgrade from N-2 to N is so important it will block release of N until it works, how do we explain why it is no longer important after four weeks when N-2 reaches End of Life?
It's not like dnf-system-upgrade would magically stop working when N-2 reaches EOL, so honestly, overall, I just don't really see the problem you're describing in this mail. We've had the current release scheme for years, people are fairly used to it; I don't honestly see that officially supporting N-2 upgrades changes the overall story much, we're just giving people another tool to use in the one month transition period (and of course, in practice, dnf-system-upgrade should usually continue to work just fine after EOL).
It's not like dnf-system-upgrade would magically stop working when N-2 reaches EOL, so honestly, overall, I just don't really see the problem you're describing
Like most of Fedora, dnf-system-upgrade gets limited testing before release. When N is released, a large number of users who did not install N-1 will think it is time to upgrade before their current N-2 reaches End-of-Life in four weeks. These users will try system-upgrade from a much larger set of initial states than could be tested before release. This is the time the greatest number of (N-2 to N) problems will be found.
It is also the time when problems in the brand-new release N are most likely, as users of N-1 eagerly try N.
With an abundance of bug reports in the just-released N, and some new reports of problems with N-2 that will be closed by EOL in a handful of days, where do you think maintenance should focus?
I admit this view is based on what is probable, not on actual fact. My point is not do this or avoid that, I want the Fedora user to have an accurate understanding of upgrade choices.
My reading of the EOL policy says "If it isn't fixed before EOL, it is unlikely to be fixed." If I encounter a problem with release-skipping system-upgrade, too bad. There is probably no time to fix it before EOL.
In effect, release-skipping system-upgrade is not supported. Either I upgrade every release (when system-upgrade is supported and bug fixes are likely), I plan to do a new install, or I hope to be lucky with EOL code.
That is not a bad set of choices. Bad is a user in trouble because he reasonably thought they were different. "Release-skipping system-upgrade is a release-blocking requirement" sounds likely to obscure the support risks that attend its use.
Fedora is certainly better for a robust system-upgrade facility. No point is filing bugs now for my 21-23 failure, though. 21 is EOL, and the failure did not occur with 22.
On Wed, 2015-11-25 at 13:36 -0500, Richard Ryniker wrote:
My reading of the EOL policy says "If it isn't fixed before EOL, it is unlikely to be fixed." If I encounter a problem with release-skipping system-upgrade, too bad. There is probably no time to fix it before EOL.
That's not really accurate. A month is a pretty long time in Fedora development, certainly long enough to diagnose and fix typical upgrade issues.
A month is a pretty long time in Fedora development
True, but a month is only available if the problem is reported on day 1. If it takes a week or two for a user to report a problem, that interval lessens the remaining time to EOL.
On the other hand, there is no prohibition against a fix after EOL.
We also do not know there will be serious problems. We know it is impossible to test more than a miniscule fraction of possible upgrade cases. That does not mean there will be lots of bugs, it means we have little confidence the test plan has found most bugs. We might say there is no reason to believe experience with the current release will be any worse than the last.
As far as I know, this is the first time "dnf system-upgrade" has been generally available for a release-skip upgrade. It is way too early to have any historical perspective.
On Wed, 2015-11-25 at 16:51 -0500, Richard Ryniker wrote:
A month is a pretty long time in Fedora development
True, but a month is only available if the problem is reported on day 1. If it takes a week or two for a user to report a problem, that interval lessens the remaining time to EOL.
On the other other hand, you can test upgrades at any time, with dnf- system-upgrade. We're already testing 22-24 upgrades daily in openQA at present.
On the other hand, there is no prohibition against a fix after EOL.
There is, in fact; you can't push updates for EOL releases, it's simply disabled by policy.
We also do not know there will be serious problems.  We know it is impossible to test more than a miniscule fraction of possible upgrade cases.  That does not mean there will be lots of bugs, it means we have little confidence the test plan has found most bugs.  We might say there is no reason to believe experience with the current release will be any worse than the last.
This is basically what I'm saying: we can't actually make any guarantees about *most* upgrade experiences, and N-2 upgrades aren't really some special case which is massively more likely to go wrong than any other. Will we have people who have issues doing N-2 upgrades? Probably, sure. Do we have people who have issues doing N-1 upgrades? yep! This isn't something new. We *do*, I think, mention in all the relevant documentation that upgrades are complex operations and success is never guaranteed, that's not what 'supported' means, especially in this context.
As far as I know, this is the first time "dnf system-upgrade" has been generally available for a release-skip upgrade.  It is way too early to have any historical perspective.
F23 is the first release where the dnf-system-upgrade mechanism has been available, yes. But it's really not hugely different from fedup - it's just an even lighter implementation of the same basic idea (just boot to a minimal environment and run a dnf transaction). The experience of fedup should be a reasonable indicator of future dnf- system-upgrade experiences.
This change in release criteria might also affect the plan for this functionality being integrated into GNOME Software. My understanding is that for Fedora 24 it's expected GNOME Software will be able to do upgrades (major version change) in addition to updates. I don't know if that's Fedora 24 -> 25 upgrade functionality, or if it'll be brought into Fedora 23 so that its GNOME Software can upgrade to Fedora 24.
Anyway, GNOME Software leverages mainly PackageKit, so Richard Hughes should probably be solicited for concerns as well.
--- Chris Murphy
On Mon, 23 Nov 2015 13:26:54 -0500 (EST) Kamil Paral kparal@redhat.com wrote:
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
"For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of any of the two previous stable Fedora releases with that package set installed." (language corrections very welcome)
During QA meeting, people were generally in favor of this idea. I was asked to check with Will Woods, system-upgrade maintainer. I did that, and he has no objections. He says it's no extra work for him and that the most work is needed from package maintainers when designing dependencies, scriptlets, etc. This is something to consider as well, but I think we implicitly want all packages to be upgradable forward (even when skipping a release). And I don't remember any issues with this lately.
Does someone have some additional concerns about this?
I'm +1 on N-2 upgrades blocking N's release. It'd be nice to have more automated upgrade tests, maybe with a few different configs but that's more into the details.
Tim
On Mon, 2015-12-07 at 09:46 -0700, Tim Flink wrote:
On Mon, 23 Nov 2015 13:26:54 -0500 (EST) Kamil Paral kparal@redhat.com wrote:
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
"For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of any of the two previous stable Fedora releases with that package set installed." (language corrections very welcome)
During QA meeting, people were generally in favor of this idea. I was asked to check with Will Woods, system-upgrade maintainer. I did that, and he has no objections. He says it's no extra work for him and that the most work is needed from package maintainers when designing dependencies, scriptlets, etc. This is something to consider as well, but I think we implicitly want all packages to be upgradable forward (even when skipping a release). And I don't remember any issues with this lately.
Does someone have some additional concerns about this?
I'm +1 on N-2 upgrades blocking N's release. It'd be nice to have more automated upgrade tests, maybe with a few different configs but that's more into the details.
As of ~5 minutes ago, openQA tests both N-1 and N-2 upgrades; it does 'minimal' and 'desktop' upgrades on x86_64, and just 'desktop' on i686. Note that up until now, openQA's F24 upgrade tests have actually been N-2 upgrades, because we have to manually rebuild the base install images and edit the openQA test config to update the upgrade tests, and we hadn't done that yet (so it was still using F22 base images for the upgrade tests). I've now bumped the original upgrade tests to F23 and added new 'upgrade_2' tests for N-2, which are currently set to F22 of course.
On Mon, 2015-11-23 at 10:52 -0500, Kamil Paral wrote:
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
So, what's the status of this proposal? I'd say we should go ahead and do it or not before the New Year, ahead of the F24 Alpha ramp-up. Does anyone have further +-1s?
As mentioned, we do now have openQA testing minimal and Workstation upgrades from F22 and F23 each day (and in fact it looks like it caught a bug in Workstation upgrades, today...)
On Wed, Dec 16, 2015 at 10:29 AM, Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2015-11-23 at 10:52 -0500, Kamil Paral wrote:
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
So, what's the status of this proposal? I'd say we should go ahead and do it or not before the New Year, ahead of the F24 Alpha ramp-up. Does anyone have further +-1s?
So I can't find this thread, including this post, in the new list web UI doing a search for the subject. I did send a "oh by the way" email suggested to ping Richard Hughes since the policy likely affects whatever is happening with GNOME Software to eventually do the major version upgrades. But I can't find that email either.
On Wed, Dec 16, 2015 at 10:44 AM, Chris Murphy lists@colorremedies.com wrote:
On Wed, Dec 16, 2015 at 10:29 AM, Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2015-11-23 at 10:52 -0500, Kamil Paral wrote:
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
So, what's the status of this proposal? I'd say we should go ahead and do it or not before the New Year, ahead of the F24 Alpha ramp-up. Does anyone have further +-1s?
So I can't find this thread, including this post, in the new list web UI doing a search for the subject. I did send a "oh by the way" email suggested to ping Richard Hughes since the policy likely affects whatever is happening with GNOME Software to eventually do the major version upgrades. But I can't find that email either.
Found mine, but not yours from today. https://lists.fedoraproject.org/archives/list/test%40lists.fedoraproject.org...
Anyway, I'm a +1.
I am a bit concerned about how GRUB gets stale on BIOS computers with upgrades rather than new installations, since grub2-install isn't called with any of the upgrade methods we've had.
http://hmarco.org/bugs/CVE-2015-8370-Grub2-authentication-bypass.html
That is fixed today in grub2-2.02-0.25.fc23. On UEFI, grubx64.efi is replaced so it's fixed there just by updating the package. But on BIOS it'll require a manual grub2-install.
On Wed, 2015-12-16 at 10:54 -0700, Chris Murphy wrote:
Found mine, but not yours from today. https://lists.fedoraproject.org/archives/list/test%40lists.fedoraproject.org...
And yeah, the other thing I noticed with the new shiny is that mails take a bit longer to show up in the web archive. With the old mailman it was virtually instantaneous, it seems like it can take 15-30 mins with the new one.
On Wed, 2015-12-16 at 10:44 -0700, Chris Murphy wrote:
On Wed, Dec 16, 2015 at 10:29 AM, Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2015-11-23 at 10:52 -0500, Kamil Paral wrote:
I feel that for something as important as system upgrade, we should provide a better level of quality and assurance for upgrading across 2 releases. Currently we have no criterion and testing it is just an afterthought, not even tracked anywhere. I'd like to amend the existing criterion to include N-2 release as well, i.e.:
So, what's the status of this proposal? I'd say we should go ahead and do it or not before the New Year, ahead of the F24 Alpha ramp-up. Does anyone have further +-1s?
So I can't find this thread, including this post, in the new list web UI doing a search for the subject. I did send a "oh by the way" email suggested to ping Richard Hughes since the policy likely affects whatever is happening with GNOME Software to eventually do the major version upgrades. But I can't find that email either.
Last I checked, for some reason, the test@ archives from before the mailman3 switch hadn't been imported to the new web UI, so you would only see mails from after the switch there. Possibly that's why. The old archive is still around -Â https://lists.fedoraproject.org/pipermail/test/%C2%A0- with all the emails up to the switchover.
So, what's the status of this proposal? I'd say we should go ahead and do it or not before the New Year, ahead of the F24 Alpha ramp-up. Does anyone have further +-1s?
We have +1 from QA, +1 from dnf-plugin-system-upgrade maintainer, and +1 from gnome-software maintainers (at least that's how I chose to interpret it). There were not many responses from general audience (I hoped for some package maintainers feedback). From my POV, we should just do it, or have it blessed by FESCo if we want to be extra safe and correct. I would do the former.
So I can't find this thread, including this post, in the new list web UI doing a search for the subject. I did send a "oh by the way" email suggested to ping Richard Hughes since the policy likely affects whatever is happening with GNOME Software to eventually do the major version upgrades. But I can't find that email either.
Here's Kalev's reply: https://lists.fedoraproject.org/archives/list/devel%40lists.fedoraproject.or...
Richard's answer doesn't seem to be indexed by hyperkitty, but it was: "Well, we're not calling into DNF for several technical reasons, but we're doing basically the same thing with libhif to what dnf does. I don't see any technical reason why we can't do n+2 upgrades, it just makes it harder to test all the permutations."
We have +1 from QA, +1 from dnf-plugin-system-upgrade maintainer, and +1 from gnome-software maintainers (at least that's how I chose to interpret it). There were not many responses from general audience (I hoped for some package maintainers feedback). From my POV, we should just do it, or have it blessed by FESCo if we want to be extra safe and correct. I would do the former.
I forgot two things. There's this quote from Kalev that's worth mentioning:
With my packager hat on, it would be great if we could get this in the packaging guidelines as well, so that there's a canonical source that says that obsoletes/conflicts etc must be preserved to support upgrades across 2 releases. And also maybe make some noise in devel-announce and in the fedora magazine so that packagers are aware that this is something everybody needs to support.
What do you think about pushing this into packaging guidelines?
Second, I'd like to highlight that gnome-software is not going to use dnf-plugin-system-upgrade, but libhif instead. So we will need to test both approaches (that's not specific to skip-release upgrades, but the combinations multiply). So this is going to need more resources in OpenQA, and possibly some more human resources when debugging issues. I'm not too happy about it, after all the effort we put into dnf-plugin-system-upgrade. Still, I think that supporting skip-release upgrades doesn't add that much overhead, and it's worth it).
On Thu, 2015-12-17 at 05:16 -0500, Kamil Paral wrote:
We have +1 from QA, +1 from dnf-plugin-system-upgrade maintainer, and +1 from gnome-software maintainers (at least that's how I chose to interpret it). There were not many responses from general audience (I hoped for some package maintainers feedback). From my POV, we should just do it, or have it blessed by FESCo if we want to be extra safe and correct. I would do the former.
I forgot two things. There's this quote from Kalev that's worth mentioning:
With my packager hat on, it would be great if we could get this in the packaging guidelines as well, so that there's a canonical source that says that obsoletes/conflicts etc must be preserved to support upgrades across 2 releases. And also maybe make some noise in devel-announce and in the fedora magazine so that packagers are aware that this is something everybody needs to support.
What do you think about pushing this into packaging guidelines?
Second, I'd like to highlight that gnome-software is not going to use dnf-plugin-system-upgrade, but libhif instead. So we will need to test both approaches (that's not specific to skip-release upgrades, but the combinations multiply). So this is going to need more resources in OpenQA, and possibly some more human resources when debugging issues. I'm not too happy about it, after all the effort we put into dnf-plugin-system-upgrade. Still, I think that supporting skip-release upgrades doesn't add that much overhead, and it's worth it).
I suppose what we could do is test upgrades of some package sets with DNF and upgrades of others with gnome-software - I guess minimal and maybe Server with DNF, Workstation with gnome-software.
Adding that text to the packaging guidelines seems like a sensible idea, yep. You'd have to propose it in an FPC ticket, that's the procedure for packaging guideline changes IIRC.
Adding that text to the packaging guidelines seems like a sensible idea, yep. You'd have to propose it in an FPC ticket, that's the procedure for packaging guideline changes IIRC.
After discussing this on the packaging mailing list, it seems we don't need any modifications of the packaging guidelines: https://lists.fedoraproject.org/archives/list/packaging%40lists.fedoraprojec...
After further consideration, I've also created a FESCo ticket to officially approve this: https://fedorahosted.org/fesco/ticket/1534
Adding that text to the packaging guidelines seems like a sensible idea, yep. You'd have to propose it in an FPC ticket, that's the procedure for packaging guideline changes IIRC.
After discussing this on the packaging mailing list, it seems we don't need any modifications of the packaging guidelines: https://lists.fedoraproject.org/archives/list/packaging%40lists.fedoraprojec...
After further consideration, I've also created a FESCo ticket to officially approve this: https://fedorahosted.org/fesco/ticket/1534
The ticket is not yet updated, but the request was approved: https://meetbot.fedoraproject.org/teams/fesco/fesco.2016-01-22-17.02.log.htm...
The new test cases are already in place, we just need to change milestone: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
They use our updated terminology of "current" and "previous" Fedora release. To keep things consistent, I propose to change the current criterion [1]: "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed. " into "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the current and previous stable Fedora release with that package set installed. "
References section would get updated with a link to this thread. The criterion itself does not mention that by upgrade we mean a direct upgrade (not one by one release). If you think it's not clear enough, I can add some wording to the criterion (phrasing suggestions welcome) or add another description section below it.
[1] https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Upgrade_requi...
On Mon, 2016-01-25 at 09:18 -0500, Kamil Paral wrote:
Adding that text to the packaging guidelines seems like a sensible idea, yep. You'd have to propose it in an FPC ticket, that's the procedure for packaging guideline changes IIRC.
After discussing this on the packaging mailing list, it seems we don't need any modifications of the packaging guidelines: https://lists.fedoraproject.org/archives/list/packaging%40lists.fedoraprojec...
After further consideration, I've also created a FESCo ticket to officially approve this: https://fedorahosted.org/fesco/ticket/1534
The ticket is not yet updated, but the request was approved: https://meetbot.fedoraproject.org/teams/fesco/fesco.2016-01-22-17.02.log.htm...
The new test cases are already in place, we just need to change milestone: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
They use our updated terminology of "current" and "previous" Fedora release. To keep things consistent, I propose to change the current criterion [1]: "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed. " into "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the current and previous stable Fedora release with that package set installed. "
References section would get updated with a link to this thread. The criterion itself does not mention that by upgrade we mean a direct upgrade (not one by one release). If you think it's not clear enough, I can add some wording to the criterion (phrasing suggestions welcome) or add another description section below it.
[1] https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Upgrade_requi...
I think the only problem with that is that in the criteria pages I tend to use 'current release' to refer to the release under test. You might need to search through the criteria pages for other occurrences of terms like "current" and "previous" and reconcile those, too - basically come up with some consistent terms and make sure they're all used consistently throughout all three pages.
They use our updated terminology of "current" and "previous" Fedora release. To keep things consistent, I propose to change the current criterion [1]: "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed. " into "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the current and previous stable Fedora release with that package set installed. "
References section would get updated with a link to this thread. The criterion itself does not mention that by upgrade we mean a direct upgrade (not one by one release). If you think it's not clear enough, I can add some wording to the criterion (phrasing suggestions welcome) or add another description section below it.
[1] https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Upgrade_requi...
I think the only problem with that is that in the criteria pages I tend to use 'current release' to refer to the release under test. You might need to search through the criteria pages for other occurrences of terms like "current" and "previous" and reconcile those, too - basically come up with some consistent terms and make sure they're all used consistently throughout all three pages.
Yay for overloaded terms. What we can do is to use "current stable release", "previous stable release" and "Branched release". Do you think that makes sense or do you see a better way?
I went through the wiki pages and this would have to change:
https://fedoraproject.org/wiki/Fedora_24_Alpha_Release_Criteria#Desktop_back... Change "The default desktop background must be different from that of the two previous stable releases. " to "The default desktop background must be different from the current stable and previous stable release. " (or maybe the original sentence is clear enough?)
https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Guest_on_prev... Change " Guest on previous release The release must install and boot successfully as a virtual guest in a situation where the virtual host is running the previous stable Fedora release. " to " Guest on current stable release The release must install and boot successfully as a virtual guest in a situation where the virtual host is running the current stable Fedora release. "
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process This is tricky. We agreed on "AcceptedPreviousRelease" flag name. I wouldn't change that. There's also: "AcceptedPreviousRelease is used for cases where the fix must appear as an update for one or more previous stable releases. " which can be "AcceptedPreviousRelease is used for cases where the fix must appear as an update for the current or previous stable release. " And "this will usually need to be fixed in the previous stable releases, not the new release" which can be "this will usually need to be fixed in the current or previous stable release, not the new release" But I'm not fully convince we really need to be that strict about the choice of words here.
Apart from these, I don't see any further edits needed.
On Mon, 2016-02-08 at 10:38 -0500, Kamil Paral wrote:
I think the only problem with that is that in the criteria pages I tend to use 'current release' to refer to the release under test. You might need to search through the criteria pages for other occurrences of terms like "current" and "previous" and reconcile those, too - basically come up with some consistent terms and make sure they're all used consistently throughout all three pages.
Yay for overloaded terms. What we can do is to use "current stable release", "previous stable release" and "Branched release". Do you think that makes sense or do you see a better way?
I might suggest "the release being tested" instead of "Branched release", but of course it kinda depends on context.
But I'm not fully convince we really need to be that strict about the choice of words here.
Apart from these, I don't see any further edits needed.
I'd say we should try to be consistent but not at the cost of making the language sound incredibly awkward in context. I'd say go ahead and do it as best you can, we can always tweak it later :) Thanks!
But I'm not fully convince we really need to be that strict about the choice of words here.
Apart from these, I don't see any further edits needed.
I'd say we should try to be consistent but not at the cost of making the language sound incredibly awkward in context. I'd say go ahead and do it as best you can, we can always tweak it later :) Thanks!
OK, I tried to improve the clarity while still keeping the language natural. Patches welcome, of course :)
These are my edits:
https://fedoraproject.org/w/index.php?title=Fedora_24_Beta_Release_Criteria&...
https://fedoraproject.org/w/index.php?title=Fedora_24_Alpha_Release_Criteria...
https://fedoraproject.org/w/index.php?title=QA%3ASOP_blocker_bug_process&...
On Mon, Feb 8, 2016 at 8:38 AM, Kamil Paral kparal@redhat.com wrote:
Yay for overloaded terms. What we can do is to use "current stable release", "previous stable release" and "Branched release". Do you think that makes sense or do you see a better way?
Tricky. For ~2 months each year there are three supported current stable releases at the same time.
rawhide next-release (or branched) current-release previous-release expiring-release expired
"Branched" is good because it ties the release to the branching process, which is an existing familiar term. I don't like combining "branched" with the word release though, because this isn't a release yet. Whereas 'next-release' suggests it's not yet a release but will be. Another plus for next-release is that before branch, it's also rawhide, whereas branched(-release) doesn't exist until branch happens.
previous and expiring could just be referred to as previous.
stable vs release Does using both help? Or is this just wordy? I'm not thinking how the combination helps. Is there an unstable release? Are there previous unstable releases? Ostensibly Fedora releases stable software, so I think stable release is redundant. I'd pick one: i.e. current-stable or current-release.
This way it's possible refer to supported releases as just "release" or as *-release rather than "one or more previous stable" or "current or previous". Just call them Fedora *-release. It's less wordy.
Yeah there is a bit of secret decoder ring with this too, but off hand I think it's easier to explain/document, remember, and write out than long handing everything. In any case, the terms chosen should be useful to those who use them the most.
On Mon, Feb 8, 2016 at 8:38 AM, Kamil Paral kparal@redhat.com wrote:
Yay for overloaded terms. What we can do is to use "current stable release", "previous stable release" and "Branched release". Do you think that makes sense or do you see a better way?
Tricky. For ~2 months each year there are three supported current stable releases at the same time.
rawhide next-release (or branched) current-release previous-release expiring-release expired
"Branched" is good because it ties the release to the branching process, which is an existing familiar term. I don't like combining "branched" with the word release though, because this isn't a release yet. Whereas 'next-release' suggests it's not yet a release but will be. Another plus for next-release is that before branch, it's also rawhide, whereas branched(-release) doesn't exist until branch happens.
Thanks for the ideas, it helped me to think about this. I didn't use it in my current edit, but I quite like "next release".
previous and expiring could just be referred to as previous.
stable vs release Does using both help? Or is this just wordy? I'm not thinking how the combination helps. Is there an unstable release? Are there previous unstable releases? Ostensibly Fedora releases stable software, so I think stable release is redundant. I'd pick one: i.e. current-stable or current-release.
In this case I think it's better to use "current stable release", because we also have development releases, and "current release" does not make it clear which one it is.
This way it's possible refer to supported releases as just "release" or as *-release rather than "one or more previous stable" or "current or previous". Just call them Fedora *-release. It's less wordy.
Yeah there is a bit of secret decoder ring with this too, but off hand I think it's easier to explain/document, remember, and write out than long handing everything. In any case, the terms chosen should be useful to those who use them the most.
-- Chris Murphy
-
Adam Williamson -
Chris Murphy -
Kamil Paral -
Richard Ryniker -
Stephen Gallagher -
Tim Flink