Hi, folks.
So, we're scheduled for Alpha TC1 tomorrow. We had a nice happy co-operative plan where QA and the WGs would collaborate on revising the release validation test process for Fedora.next...
...which, well, didn't really happen. As of this morning we were nowhere near having a viable validation process. So I went for plan B: I spent today more or less pulling the entire thing out of my ass.
It's a bit rough around the edges, but I think we more or less have something workable now. I have skipped the draft stage for a lot of documents just in the interest of having something vaguely workable in time for TC1; of course, the pages can be revised as much as necessary as we work with them.
It's a bit hard to remember everything I've done, but we now have a draft Alpha Release Criteria page which should cover all release-blocking media, which for now I'm assuming includes the Workstation live media, Server minimal and offline install media, Cloud and ARM disk images, and possibly some kind of generic network install image. That draft is at https://fedoraproject.org/wiki/User:Adamwill/Draft_F21_Alpha_criteria and is based on the stuff from https://fedoraproject.org/wiki/User:Adamwill/Draft_server_release_criteria , https://fedoraproject.org/wiki/User:Adamwill/Draft_workstation_release_crite... and https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Release_... , plus some adjustments to the templates that handle the preamble.
We have a new validation matrix, for Server product-specific tests: https://fedoraproject.org/wiki/QA:Server_validation_results_template
The Desktop matrix has been adjusted to cover - not quite elegantly, but at least cover - both the Workstation product and the KDE spin: https://fedoraproject.org/wiki/QA:Desktop_validation_results_template
The Base matrix has been extended to add a couple of new test cases that came out of the Product criteria drafting process, but actually aren't really product specific, and has had its columns adjusted to be product-y: https://fedoraproject.org/wiki/QA:Base_validation_results_template
The installation matrix has similarly had a couple of new criteria wedged in, but much more importantly, I ripped the netinst, DVD and live image 'sanity test' sections and replaced them with an ARM-style table where a single 'generic' test case is run against several images on several platforms - that's the "Default boot and install" section, so please cast an eye over it: https://fedoraproject.org/wiki/QA:Fedora_21_Install_Results_Template
and I made a small change to the release validation test event SOP to list the server matrix as one to create:
https://fedoraproject.org/wiki/QA/SOP_Release_Validation_Test_Event
Here are links to all (I think) of the new test cases I had to write as I went along:
https://fedoraproject.org/wiki/QA:Testcase_Boot_default_install https://fedoraproject.org/wiki/QA:Testcase_kickstart_user_creation https://fedoraproject.org/wiki/QA:Testcase_base_service_manipulation https://fedoraproject.org/wiki/QA:Testcase_base_selinux https://fedoraproject.org/wiki/QA:Testcase_Server_firewall_default https://fedoraproject.org/wiki/QA:Testcase_kickstart_firewall https://fedoraproject.org/wiki/QA:Testcase_Server_cockpit_default
The following test cases already existed, but are newly included in the release validation process (they were written for test days):
https://fedoraproject.org/wiki/QA:Testcase_FreeIPA_realmd_join https://fedoraproject.org/wiki/QA:Testcase_realmd_join_kickstart https://fedoraproject.org/wiki/QA:Testcase_realmd_join_server
There are still quite a few i's to dot and t's to cross. There are some release criteria and test cases that explicitly reference 'the DVD' image that will need to be adjusted. We need to apply the 'associated release criterion' template to all the new test cases, and probably clean up some categorizations. Various other process documentation pages may need to be updated, we'll have to check through all of them. But I think now we at least have the broad strokes of what's needed for .next validation testing.
All feedback on the above changes is of course welcome! Please do cast your eye over and point out anything I missed, anything that looks silly, any possible improvements and so on. Remember, though, this is really *test process* design: we're not actually doing product design here, if you think there are issues with the Fedora.next changes themselves, that goes to the WGs or FESCo. As far as this work is concerned, we're just trying to test the new Products as they're designed (all of the above is based off of the Product PRDs and tech specs).
I'll take time tomorrow to do some polishing, and of course look at any and all feedback on the stuff I did today.
Hey Adam,
I've looked over the workstation testcases quickly.
Some comments:
https://fedoraproject.org/wiki/QA:Testcase_desktop_updates talks about yum and PackageKit - should it include dnf ?
https://fedoraproject.org/wiki/QA:Testcase_desktop_panel_basic could benefit from some outline of what desktop 'panel' elements we expect to see. Sticking with the most common laptop case, I would mention network, sound and battery status as expected
https://fedoraproject.org/wiki/QA:Testcase_desktop_login seems to be a bit of a misnomer; the test is really about keyboard layouts.
https://fedoraproject.org/wiki/QA:Testcase_audio_basic talks about selection hw profiles. I really think that the expectation for 'common hw' (ie your typical laptop) should be that sound works out of the box.
https://fedoraproject.org/wiki/QA:Testcase_desktop_updates same comment as above - should this use dnf instead of yum ? I think it would be good to make this testcase more specific wrt to the way offline updates are integrated in the desktop now. Test that we notify (once per day) about available updates. Test that the logout/poweroff dialog offers to install pending updates. Test that we notify about successful and unsuccessful offline updates, and show details.
https://fedoraproject.org/wiki/QA:Testcase_workstation_theming would benefit from adding some mention of accessibility - we expect the HighContrast theme to work and be of similar quality to the default theme.
Things that would be good to capture in new testcases:
- Screen locking. Explicit locking via keyboard shortcut or menu should work, as well as lock on idle. Screen blanking after lock is something that frequently has issues. Can also test that inhibiting works (when watching video in totem, or in the browser.
----- Original Message -----
Hey Adam,
I've looked over the workstation testcases quickly.
Some comments:
https://fedoraproject.org/wiki/QA:Testcase_desktop_updates talks about yum and PackageKit - should it include dnf ?
Dnf is still not default in Fedora, so we don't expect 100% functionality that's needed for yum. It could be mentioned as optional step but let's try to avoid that complexity at least in the beginning.
Jaroslav
On Fri, 2014-07-11 at 13:23 -0400, Matthias Clasen wrote:
Hey Adam,
I've looked over the workstation testcases quickly.
I just found this mail a half year later, so some very belated replies...
Some comments:
https://fedoraproject.org/wiki/QA:Testcase_desktop_updates talks about yum and PackageKit - should it include dnf ?
As Jaro said at the time, yum is still the default/supported package manager for F21. For F22 we're currently slated to change to dnf, and if that *happens* we'll change the test case, but there's still some discussion about it at present.
https://fedoraproject.org/wiki/QA:Testcase_desktop_panel_basic could benefit from some outline of what desktop 'panel' elements we expect to see. Sticking with the most common laptop case, I would mention network, sound and battery status as expected
So yeah, a thing to bear in mind is that most of the cases were written around F13 time (IIRC), with the GNOME 2 and KDE 3 of those days in mind, when everything looked more or less like Windows 98 and everyone's 'panel' actually had some bits running in it that they didn't write (like nm-applet was a pretty separate thing from gnome- panel, for e.g.)
When I'm running this test (and the 'advanced' version) against Shell I usually check all the bits of the Shell top bar, user menu, and overview. Re-wording it to be too specific about the expected elements runs the risk of no longer being applicable to the other desktops, but of course if it's necessary we can have two (or more) cases, or some conditional language.
I think this just about scrapes by as 'OK' at present, but if I can find the time I'll see if I can improve it for the Modern World.
https://fedoraproject.org/wiki/QA:Testcase_desktop_login seems to be a bit of a misnomer; the test is really about keyboard layouts.
Hum, not really, it really *does* test login/DM stuff - it just also tests that there aren't screwups with the keyboard layout not being consistent between installer, DM, and desktop. IIRC, I initially started writing a separate 'keyboard layouts' test case at some point, then realized that what it was actually doing was duplicating a bunch of existing test cases but with 'oh, and do it with a different keyboard layout' - so instead of doing that, I just wrote 'check it with a different keyboard layout' into each of the existing tests. There are similar bits in the disk encryption test case, for e.g., to make sure the right keyboard layout is used for entering the passphrase on boot.
https://fedoraproject.org/wiki/QA:Testcase_audio_basic talks about selection hw profiles. I really think that the expectation for 'common hw' (ie your typical laptop) should be that sound works out of the box.
Reasonable point, I'll have to check on the current actual status of this with different hardware; I *think* at least on modern systems the drivers/PA should be able to sense what's connected and choose the appropriate output profile, but I'm not 100% sure.
I think the note for multiple *devices* is still valid, though. If you have multiple output devices, PA can't magically tell which one you want sound to come out of, you do have to tell it.
https://fedoraproject.org/wiki/QA:Testcase_desktop_updates same comment as above - should this use dnf instead of yum ? I think it would be good to make this testcase more specific wrt to the way offline updates are integrated in the desktop now. Test that we notify (once per day) about available updates. Test that the logout/poweroff dialog offers to install pending updates. Test that we notify about successful and unsuccessful offline updates, and show details.
There's definitely some ancient text in there ("observing whether the 'checking for updates' and then 'updates available' icons appear in the notification area", no, that's not going to happen). The same problem as I mentioned above applies to making this *too* specific to Workstation - the same test case is applied to KDE and other lives too - but I think the same approach will work, first try and 'modernize' it as a generic test case, and then evaluate whether it's necessary/worthwhile to split out a Workstation-specific case from it.
https://fedoraproject.org/wiki/QA:Testcase_workstation_theming would benefit from adding some mention of accessibility - we expect the HighContrast theme to work and be of similar quality to the default theme.
Thanks, I'm not sure whether that'd work best as an addon to this test case or a new one, but either way, I'll try and add something to cover it.
Things that would be good to capture in new testcases:
- Screen locking. Explicit locking via keyboard shortcut or menu
should work, as well as lock on idle. Screen blanking after lock is something that frequently has issues. Can also test that inhibiting works (when watching video in totem, or in the browser.
Noted, thanks!
desktop@lists.fedoraproject.org
-
Adam Williamson -
Jaroslav Reznik -
Matthias Clasen