Validation proposal: separate out artwork / release identification checks to their own test case
Hi folks! Time for another proposal :)
We currently kinda bundle the checks for correct artwork and 'release identification' (stuff that identifies the release, like 'Welcome to Fedora 26!' etc.) into two test cases that also include functionality testing:
https://fedoraproject.org/wiki/QA:Testcase_base_startup https://fedoraproject.org/wiki/QA:Testcase_base_initial_setup
This is a bit awkward, and also is a bit of a problem for automation. We could actually automate the functional parts of those tests in openQA (I've just finished adding openQA tests that install without a user account and check gnome-initial-setup and initial-setup work OK), but we can't really automate the 'release identification' / artwork checks. So we can't have openQA report wiki passes for these tests, so long as they're combined like this.
So I'm proposing we separate out a new 'artwork and release identification' test case. I've written a draft of it:
https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_artwork_release_...
We would remove the relevant bits from the other two test cases.
I'm a bit in two minds about what to do with the matrices. I'm thinking we could either add this test case to the Server, Cloud and Desktop pages, or we could put it in the Base page (as the current tests are). Looking at this, it occurs to me that not just these checks but most of the other Base tests would actually be relevant to things beyond the 'environments' currently in the Base page (Workstation, Server, KDE, ARM and Cloud). I'm thinking perhaps we could rejig the page a bit, and add another table with non-blocking columns for other desktops and stuff.
Thoughts? Thanks!
I agree with the change, good thinking.
I think that we might check changing artwork by creating needles for old release and then checking that this specific needle DOESN'T match.
On Thu, 2017-04-06 at 13:19 +0200, Jan Sedlak wrote:
I agree with the change, good thinking.
I think that we might check changing artwork by creating needles for old release and then checking that this specific needle DOESN'T match.
Yeah, we could do something along those lines, but I don't think we'll ever be able to assert a full pass of the test case, easy for something to slip through. But we could indeed just add a check that no previous release's wallpaper appears on the 'clean desktop' state, it'd at least point out the obvious cases where we didn't update the damn wallpaper *again*.
On Wed, Apr 05, 2017 at 03:38:00PM -0700, Adam Williamson wrote:
Hi folks! Time for another proposal :)
We currently kinda bundle the checks for correct artwork and 'release identification' (stuff that identifies the release, like 'Welcome to Fedora 26!' etc.) into two test cases that also include functionality testing:
https://fedoraproject.org/wiki/QA:Testcase_base_startup https://fedoraproject.org/wiki/QA:Testcase_base_initial_setup
This is a bit awkward, and also is a bit of a problem for automation. We could actually automate the functional parts of those tests in openQA (I've just finished adding openQA tests that install without a user account and check gnome-initial-setup and initial-setup work OK), but we can't really automate the 'release identification' / artwork checks. So we can't have openQA report wiki passes for these tests, so long as they're combined like this.
So I'm proposing we separate out a new 'artwork and release identification' test case. I've written a draft of it:
https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_artwork_release_...
We would remove the relevant bits from the other two test cases.
I'm a bit in two minds about what to do with the matrices. I'm thinking we could either add this test case to the Server, Cloud and Desktop pages, or we could put it in the Base page (as the current tests are). Looking at this, it occurs to me that not just these checks but most of the other Base tests would actually be relevant to things beyond the 'environments' currently in the Base page (Workstation, Server, KDE, ARM and Cloud). I'm thinking perhaps we could rejig the page a bit, and add another table with non-blocking columns for other desktops and stuff.
Thoughts? Thanks!
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
Sounds good to me.
// Mike -- Fedora QA
On Wed, 2017-04-05 at 15:38 -0700, Adam Williamson wrote:
Hi folks! Time for another proposal :)
We currently kinda bundle the checks for correct artwork and 'release identification' (stuff that identifies the release, like 'Welcome to Fedora 26!' etc.) into two test cases that also include functionality testing:
https://fedoraproject.org/wiki/QA:Testcase_base_startup https://fedoraproject.org/wiki/QA:Testcase_base_initial_setup
This is a bit awkward, and also is a bit of a problem for automation. We could actually automate the functional parts of those tests in openQA (I've just finished adding openQA tests that install without a user account and check gnome-initial-setup and initial-setup work OK), but we can't really automate the 'release identification' / artwork checks. So we can't have openQA report wiki passes for these tests, so long as they're combined like this.
So I'm proposing we separate out a new 'artwork and release identification' test case. I've written a draft of it:
https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_artwork_release_...
We would remove the relevant bits from the other two test cases.
I'm a bit in two minds about what to do with the matrices. I'm thinking we could either add this test case to the Server, Cloud and Desktop pages, or we could put it in the Base page (as the current tests are). Looking at this, it occurs to me that not just these checks but most of the other Base tests would actually be relevant to things beyond the 'environments' currently in the Base page (Workstation, Server, KDE, ARM and Cloud). I'm thinking perhaps we could rejig the page a bit, and add another table with non-blocking columns for other desktops and stuff.
Thoughts? Thanks!
Thanks for the feedback so far. So I've moved the test case itself to a 'production' name:
https://fedoraproject.org/wiki/QA:Testcase_base_artwork_release_identificati...
and created a draft of a revised Base matrix:
https://fedoraproject.org/wiki/User:Adamwill/Draft_Base_test_matrix
It looks a lot bigger, but don't be scared! Two of the tables are completely optional. The ARM table is really just the old 'ARM' column from the single table split out into its own table and with a split between the two ARM release-blocking environments, Xfce and Minimal - it seems reasonable that we could have different results for those, so an 'ARM' column really didn't seem to make much sense. I added the new test to each table, with the milestone set to 'Alpha / Final'.
What do people think of this (especially ARM folks - paging pwhalen)? Does it look reasonable? Thanks!
If this draft looks OK, I'll put it into production soon, and remove the artwork-y bits from the startup and initial_setup tests.
On Wed, 2017-04-19 at 17:07 -0700, Adam Williamson wrote:
Thanks for the feedback so far. So I've moved the test case itself to a 'production' name:
https://fedoraproject.org/wiki/QA:Testcase_base_artwork_release_identificati...
and created a draft of a revised Base matrix:
https://fedoraproject.org/wiki/User:Adamwill/Draft_Base_test_matrix
It looks a lot bigger, but don't be scared! Two of the tables are completely optional. The ARM table is really just the old 'ARM' column from the single table split out into its own table and with a split between the two ARM release-blocking environments, Xfce and Minimal - it seems reasonable that we could have different results for those, so an 'ARM' column really didn't seem to make much sense. I added the new test to each table, with the milestone set to 'Alpha / Final'.
What do people think of this (especially ARM folks - paging pwhalen)? Does it look reasonable? Thanks!
If this draft looks OK, I'll put it into production soon, and remove the artwork-y bits from the startup and initial_setup tests.
As no-one had any objections to this, I've gone ahead and deployed it to production now. The Base_test_matrix page is revised to the new layout, the test case pages are adjusted, and I've updated the openQA wiki reporting code to work with the new setup. Please let me know if you see any problems! Thanks.
-
Adam Williamson -
Jan Sedlak -
Mike Ruckman