Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like this:
Windows macOS Fedora 24 Fedora 25 QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
Thoughts?
On Thu, Jan 12, 2017 at 4:32 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like this:
Windows macOS Fedora 24 Fedora 25QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
Thoughts?
Well I don't really want to eat a hat, because I've had my fill for a lifetime, but I'd eat my hat if FMW works on Windows 10 but does not work on Windows 7. Or vice versa. So may be ask mbriza which one to test on, and hope the numbers game fills in the rest on its own. The API/ABI stability on Windows is pretty extreme. In order of market share though, by a long shot it's Windows 7, then 10, and a distant third is 8.1 and then 8(.0) barely registers. So weirdly enough, chances are it'll work on 8/8.1 if it works on 7 and 10. So a thorough test would be testing both 7 and 10. Less thorough, but sane, is testing just on 10.
(For those who don't know, it's possible to get a free copy of Windows Enterprise, the installer iSO will install a copy of Windows that'll work for 90 days, the timer starts from the time of installation; i.e. it's not the download ISO that's time limited.)
FWIW at the moment on Fedora 25 I'm running into these two bugs and can't write images at all with FMW. https://bugzilla.redhat.com/show_bug.cgi?id=1412063 https://bugzilla.redhat.com/show_bug.cgi?id=1412057
On Fri, 13 Jan 2017 04:30:38 +0100, Chris Murphy lists@colorremedies.com wrote:
On Thu, Jan 12, 2017 at 4:32 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like this:
Windows macOS Fedora 24 Fedora 25QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
Thoughts?
Well I don't really want to eat a hat, because I've had my fill for a lifetime, but I'd eat my hat if FMW works on Windows 10 but does not work on Windows 7. Or vice versa. So may be ask mbriza which one to test on, and hope the numbers game fills in the rest on its own. The API/ABI stability on Windows is pretty extreme. In order of market share though, by a long shot it's Windows 7, then 10, and a distant third is 8.1 and then 8(.0) barely registers. So weirdly enough, chances are it'll work on 8/8.1 if it works on 7 and 10. So a thorough test would be testing both 7 and 10. Less thorough, but sane, is testing just on 10.
(For those who don't know, it's possible to get a free copy of Windows Enterprise, the installer iSO will install a copy of Windows that'll work for 90 days, the timer starts from the time of installation; i.e. it's not the download ISO that's time limited.)
FWIW at the moment on Fedora 25 I'm running into these two bugs and can't write images at all with FMW. https://bugzilla.redhat.com/show_bug.cgi?id=1412063 https://bugzilla.redhat.com/show_bug.cgi?id=1412057
It does not work on Windows 7 for some folks. On my laptop with Win 7, it works. I've got a virtual machine where it doesn't work for testing though. I suspect there's a problem with ANGLE (which is a library that translates OpenGL calls to DirectX on Windows) combined with hardware drivers not supporting recent enough OpenGL. This is probably a problem with Qt packaging on Fedora but as of now, I don't know how to fix that. There's a bug [1] open for this specific issue for a while now.
I'm currently waiting for the release of Qt 5.8 where there is a more advanced level of deciding which rendering backend to use for QML and it can even fallback to a software renderer now which could be sufficient for FMW. However, I need to test that first.
So in the light of this, I agree with Chris that testing on Windows 7 and 10 would probably be what we should do.
Chris, regarding the issues you mentioned: #1412063 should be fixed in 4.0.8 (now in updates-testing) #1412057 I have a fix for this using a different method in the UDisks2 API - now FMW will seem to be "benchmarking" a drive instead of writing (especially in polkit dialogs). OpenForRestore doesn't restore writing AND reading the written data, hence I needed to use a method that allows me to do so. The source of your problem was using OpenForRestore and then OpenForBackup - in case of large images or slow flash drives, a timeout somewhere in polkit ran out and it requested authentication again.
Well I don't really want to eat a hat, because I've had my fill for a lifetime, but I'd eat my hat if FMW works on Windows 10 but does not work on Windows 7. Or vice versa. ...
It does not work on Windows 7 for some folks. On my laptop with Win 7, it works. I've got a virtual machine where it doesn't work for testing though. I suspect there's a problem with ANGLE (which is a library that translates OpenGL calls to DirectX on Windows) combined with hardware drivers not supporting recent enough OpenGL.
Chris, a picture or didn't happen ;-)
On Fri, 13 Jan 2017 10:36:37 +0100, Kamil Paral kparal@redhat.com wrote:
Well I don't really want to eat a hat, because I've had my fill for a lifetime, but I'd eat my hat if FMW works on Windows 10 but does not work on Windows 7. Or vice versa. ...
It does not work on Windows 7 for some folks. On my laptop with Win 7, it works. I've got a virtual machine where it doesn't work for testing though. I suspect there's a problem with ANGLE (which is a library that translates OpenGL calls to DirectX on Windows) combined with hardware drivers not supporting recent enough OpenGL.
Chris, a picture or didn't happen ;-)
Don't need a picture or anything, I know there's a problem. A complex one at that. Let's see what happens when Qt 5.8 comes out (in a few days actually, I may as well start packaging it for mingw already).
Well I don't really want to eat a hat, because I've had my fill for a lifetime, but I'd eat my hat if FMW works on Windows 10 but does not work on Windows 7. Or vice versa. ...
It does not work on Windows 7 for some folks. On my laptop with Win 7, it works. I've got a virtual machine where it doesn't work for testing though. I suspect there's a problem with ANGLE (which is a library that translates OpenGL calls to DirectX on Windows) combined with hardware drivers not supporting recent enough OpenGL.
Chris, a picture or didn't happen ;-)
Don't need a picture or anything, I know there's a problem. A complex one at that. Let's see what happens when Qt 5.8 comes out (in a few days actually, I may as well start packaging it for mingw already).
I was talking about *the hat*.
On Fri, Jan 13, 2017 at 2:36 AM, Kamil Paral kparal@redhat.com wrote:
Well I don't really want to eat a hat, because I've had my fill for a lifetime, but I'd eat my hat if FMW works on Windows 10 but does not work on Windows 7. Or vice versa. ...
It does not work on Windows 7 for some folks. On my laptop with Win 7, it works. I've got a virtual machine where it doesn't work for testing though. I suspect there's a problem with ANGLE (which is a library that translates OpenGL calls to DirectX on Windows) combined with hardware drivers not supporting recent enough OpenGL.
Chris, a picture or didn't happen ;-)
I am not going to eat another goddamn hat just because of some Windows edge cases :-P
Besides, we live in an era now of merely saying things makes them true/untrue, and I wouldn't want to impune that by suggesting a photo enhances my hat eating claim. But it should be completely believable because I've switched to hat eating due to my former target of self punishment for being wrong was having noticeable effects on billy goat herd population.
On Thu, Jan 12, 2017 at 08:30:38PM -0700, Chris Murphy wrote:
API/ABI stability on Windows is pretty extreme. In order of market share though, by a long shot it's Windows 7, then 10, and a distant third is 8.1 and then 8(.0) barely registers. So weirdly enough, chances are it'll work on 8/8.1 if it works on 7 and 10. So a thorough test would be testing both 7 and 10. Less thorough, but sane, is testing just on 10.
From http://stackoverflow.com/research/developer-survey-2016#technology-desktop-o...
22.5% Windows 7 20.8% Windows 10 8.4% Windows 8 0.4% Windows XP 0.1% Windows Vista
and 26.2% OS X, FWIW.
(Fedora, at 1.4%, beats XP and Vista *combined*. Whoo!)
This backs up what you're saying with numbers, and specifically from our target userbase for Workstation.
To my deep sadness, they did not ask this question on the 2017 survey (which is open now).
Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like this:
Windows macOS Fedora 24 Fedora 25QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
After reading Martin Briza's reply, we could have a Win 7 and Win 10 column, yeah. But e.g. for dual-boot, I always test just with Win 10 and hope the community does the rest :-) I'd probably do it the same here - it should be OK if at least one of those fields is populated, not require both. Alternatively, we can have a single column and recommend to primarily test on Win 7 and Win 10 in the test case description (and hope that people will submit multiple results for this from time to time, and add a note to indicate what they tested with). I don't have a real preference here.
What do we do if one or both of the Windows fields fail? Or the macOS one? Our release criteria don't specify a host system at the moment.
Also, should we split Fedora systems per arch (x86_64, armhfp)? I don't want to have a too much exploded matrix here, I'd keep it architecture-less I guess.
On Fri, 13 Jan 2017 00:32:06 +0100, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like this:
Windows macOS Fedora 24 Fedora 25QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
Thoughts?
I'd consider focusing on using checkisomd5 as the main way to test if FMW works ok. First the offline one and then online one running from the flash drive before bootup.
If the integrity check passes, all other errors are either in the compose process or the packages. Anything above this I would consider testing the actual written image, not FMW.
I'd consider focusing on using checkisomd5 as the main way to test if FMW works ok. First the offline one and then online one running from the flash drive before bootup.
That's actually a very good advice for the test case. As Martin showed me recently, running simply "checkisomd5 --verbose /dev/sdX" after you've written the image to /dev/sdX is actually a fully satisfactory verification that the image was written properly (or you could also do "cmp /dev/sdX image.iso"). Of course booting the thumb stick and letting it verify consistency from the syslinux menu is a bonus point.
There's a potential issue here. Sometimes it can happen that the written partitions are automounted and some filesystem metadata are changed before unplugging (at least that's my guess). This happened to me several times during the last cycle, and the checksum verification then fails (but the installation proceeds completely OK). This happens also on Windows, according to Martin, and it's quite difficult to prevent it. We should mention it in the test case, but I'm not sure what advice we can give. We don't want to of course handwaive such issues completely, because they can also mean some data writing corruption (I found one such bug a few months back, and it was FMW's fault).
If the integrity check passes, all other errors are either in the compose process or the packages. Anything above this I would consider testing the actual written image, not FMW.
Yes. In our particular case, in the "Default boot and install" section, we also test default installation (anaconda), so I guess we can't optimize there. But for that particular test case, the offline/online verification should be enough, I believe.
On Fri, 13 Jan 2017 11:07:13 +0100, Kamil Paral kparal@redhat.com wrote:
I'd consider focusing on using checkisomd5 as the main way to test if FMW works ok. First the offline one and then online one running from the flash drive before bootup.
That's actually a very good advice for the test case. As Martin showed me recently, running simply "checkisomd5 --verbose /dev/sdX" after you've written the image to /dev/sdX is actually a fully satisfactory verification that the image was written properly (or you could also do "cmp /dev/sdX image.iso"). Of course booting the thumb stick and letting it verify consistency from the syslinux menu is a bonus point.
There's a potential issue here. Sometimes it can happen that the written partitions are automounted and some filesystem metadata are changed before unplugging (at least that's my guess). This happened to me several times during the last cycle, and the checksum verification then fails (but the installation proceeds completely OK). This happens also on Windows, according to Martin, and it's quite difficult to prevent it. We should mention it in the test case, but I'm not sure what advice we can give. We don't want to of course handwaive such issues completely, because they can also mean some data writing corruption (I found one such bug a few months back, and it was FMW's fault).
If the integrity check passes, all other errors are either in the compose process or the packages. Anything above this I would consider testing the actual written image, not FMW.
Yes. In our particular case, in the "Default boot and install" section, we also test default installation (anaconda), so I guess we can't optimize there. But for that particular test case, the offline/online verification should be enough, I believe.
I think I have fixed the issue on Windows. However, I think there's still a possibility the OS will mess with your drive if you reinsert it to the system. Especially on Mac, where I have worked around this by blocking all media mounting while FMW is running.
To be sure this doesn't happen and to included the case in our test, I proposed using the online media check too.
This is related to validation tests, but different enough that I wanted a new thread. While we *can* release new versions of FMW on getfedora.org at Fedora GA, there's no reason to tie the two together — and really, some good ones to *not* double up:
* we can fix bugs without waiting * we can add new functionality whenever * and we don't have to worry about surprise failures due to regressions on GA release day affecting the rush of people
I have a feature request (use a distinct user-agent for ISO downloads so we can tell when FMW is being used) which I'd love to not wait for June/July to get live.
Do we want to go through validation and update the website whenever Martin cuts a new release upstream? Or do we want to have regularly-scheduled times to do it? Or whenever there's a particular reason to do it and leave it alone otherwise? Or something else?
On Fri, Jan 13, 2017 at 8:12 AM, Matthew Miller mattdm@fedoraproject.org wrote:
This is related to validation tests, but different enough that I wanted a new thread. While we *can* release new versions of FMW on getfedora.org at Fedora GA, there's no reason to tie the two together — and really, some good ones to *not* double up:
- we can fix bugs without waiting
- we can add new functionality whenever
- and we don't have to worry about surprise failures due to regressions on GA release day affecting the rush of people
I have a feature request (use a distinct user-agent for ISO downloads so we can tell when FMW is being used) which I'd love to not wait for June/July to get live.
Do we want to go through validation and update the website whenever Martin cuts a new release upstream? Or do we want to have regularly-scheduled times to do it? Or whenever there's a particular reason to do it and leave it alone otherwise? Or something else?
I like the idea of decoupling this from the release schedule.
Maybe mbriza "nominates" a release for validation (not every release needs to go to the web site ASAP perhaps?) and schedules something like a Test Day with QA. And that test day can act as validation, out of band. And then after all the major bugs are found to be fixed, it can be handed off to web folks for updating.
This is related to validation tests, but different enough that I wanted a new thread. While we *can* release new versions of FMW on getfedora.org at Fedora GA, there's no reason to tie the two together — and really, some good ones to *not* double up:
- we can fix bugs without waiting
- we can add new functionality whenever
- and we don't have to worry about surprise failures due to regressions on GA release day affecting the rush of people
Sure, makes sense.
I have a feature request (use a distinct user-agent for ISO downloads so we can tell when FMW is being used) which I'd love to not wait for June/July to get live.
Do we want to go through validation and update the website whenever Martin cuts a new release upstream? Or do we want to have regularly-scheduled times to do it? Or whenever there's a particular reason to do it and leave it alone otherwise? Or something else?
Martin should be probably the one to tell websites team when to pull new version (once it is tested, not every one needs to be pulled).
We'll probably not be able to test every each upstream release he does, especially on all platforms. During Branched release cycle, it will be easier, since we're testing FMW all the time anyway. Outside of the release cycle, he can ping us if we have time to do that, or he can wait until some feedback is available in Bodhi and act based on that (however, that does not include feedback for non-Fedora systems, but I think he said he performs a few smoke tests every time on his test machines, so that should be probably fine).
On Mon, 16 Jan 2017 10:52:28 +0100, Kamil Paral kparal@redhat.com wrote:
This is related to validation tests, but different enough that I wanted a new thread. While we *can* release new versions of FMW on getfedora.org at Fedora GA, there's no reason to tie the two together — and really, some good ones to *not* double up:
- we can fix bugs without waiting
- we can add new functionality whenever
- and we don't have to worry about surprise failures due to regressions on GA release day affecting the rush of people
Sure, makes sense.
I have a feature request (use a distinct user-agent for ISO downloads so we can tell when FMW is being used) which I'd love to not wait for June/July to get live.
Do we want to go through validation and update the website whenever Martin cuts a new release upstream? Or do we want to have regularly-scheduled times to do it? Or whenever there's a particular reason to do it and leave it alone otherwise? Or something else?
Martin should be probably the one to tell websites team when to pull new version (once it is tested, not every one needs to be pulled).
We'll probably not be able to test every each upstream release he does, especially on all platforms. During Branched release cycle, it will be easier, since we're testing FMW all the time anyway. Outside of the release cycle, he can ping us if we have time to do that, or he can wait until some feedback is available in Bodhi and act based on that (however, that does not include feedback for non-Fedora systems, but I think he said he performs a few smoke tests every time on his test machines, so that should be probably fine).
I'm testing the releases myself, yes. Usually I focus on the stuff I changed and then just try writing and erasing a drive, so nothing too complicated.
I see another problem with the release process now though: pulling new non-Fedora releases. There is no rigid process of getting the releases to websites. The process now is I put up a new release and ping dgilmore to compile it. He then, when he has time, compiles it and puts it somewhere and notifies websites to pull it or he puts it there himself.
For example for 4.0.8 which was released about a week ago there has been no activity yet - meanwhile, f25 package is already stable.
All I can do is ping the people over and over and hope one day the version get pushed there. There is very limited feedback in regards to what the status is now, if it compiles and there is no testing besides mine.
On Mon, Jan 16, 2017 at 11:28:50AM +0100, Martin Bříza wrote:
new non-Fedora releases. There is no rigid process of getting the releases to websites. The process now is I put up a new release and ping dgilmore to compile it. He then, when he has time, compiles it and puts it somewhere and notifies websites to pull it or he puts it there himself.
I think changing "ping dgilmore" to "file a ticket with rel-eng" would be an improvement. Not that Dennis doesn't do a bunch of stuff -- in fact, exactly because he does, a ticket will help keep it from getting lost.
Ideally, we'd have something so you could trigger the build automatically, but I can see the work/reward of that being low if this is infrequent.
For example for 4.0.8 which was released about a week ago there has been no activity yet - meanwhile, f25 package is already stable.
A week and waiting for the F25 package to go stable seems fine in most cases - but we'd want it to be much faster in the event of a serious bugfix.
All I can do is ping the people over and over and hope one day the version get pushed there. There is very limited feedback in regards to what the status is now, if it compiles and there is no testing besides mine.
A ticket will help, yeah.
And there needs to be a step in between "build made and available for download $somewhere" and "build pushed to web server" where QA can do the QAing.
On Fri, Jan 13, 2017 at 12:32 AM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like this:
Windows macOS Fedora 24 Fedora 25QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
Thoughts?
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org
Current media writer supports ARM, I would propose to add a row for it, so it could look like this:
Windows macOS Fedora 24 Fedora 25 x86_64 iso arm image
where both "x86_64 iso" and "arm image" links to QA:Testcase_USB_fmw
On Tue, Apr 18, 2017 at 2:50 PM, Lukas Brabec lbrabec@redhat.com wrote:
Current media writer supports ARM, I would propose to add a row for it, so it could look like this:
Windows macOS Fedora 24 Fedora 25x86_64 iso arm image
We'll also probably want "Fedora 26" (Branched at that time) column.
On Tue, 2017-04-18 at 14:55 +0200, Kamil Paral wrote:
On Tue, Apr 18, 2017 at 2:50 PM, Lukas Brabec lbrabec@redhat.com wrote:
Current media writer supports ARM, I would propose to add a row for it, so it could look like this:
Windows macOS Fedora 24 Fedora 25x86_64 iso arm image
We'll also probably want "Fedora 26" (Branched at that time) column.
Eh, I intentionally left it out because we've always held that being able to write images from the new release is much less important than being able to write images from the existing stable releases. The point was to try and keep a lid on the required testing, remember? :)
On Tue, Apr 18, 2017 at 6:13 PM, Adam Williamson <adamwill@fedoraproject.org
wrote:
We'll also probably want "Fedora 26" (Branched at that time) column.
Eh, I intentionally left it out because we've always held that being able to write images from the new release is much less important than being able to write images from the existing stable releases. The point was to try and keep a lid on the required testing, remember? :)
That's the question though - is it required? I thought the test case would be marked as optional. We already require FMW in "Default boot and install" matrix, so Fedora is covered (we just don't know *what* Fedora they used for that). And are any FMW crashes on Win/macOS blocking? I don't think we have a criterion for that (the question is whether we should).
So as long as we keep that table optional, I think it's useful to split out the Fedora version, so that we clearly see what people tried and what they didn't. Also, for the same reason it would probably help to see Windows 7 and Windows 10 separated.
But if we want to mark it as required, I wouldn't want yet another exploded matrix, and I agree Fedora 26 could be left out (and Windows merged together).
On Wed, Apr 19, 2017 at 2:25 PM, Kamil Paral kparal@redhat.com wrote:
That's the question though - is it required? I thought the test case would be marked as optional. We already require FMW in "Default boot and install" matrix, so Fedora is covered (we just don't know *what* Fedora they used for that). And are any FMW crashes on Win/macOS blocking? I don't think we have a criterion for that (the question is whether we should).
So as long as we keep that table optional, I think it's useful to split out the Fedora version, so that we clearly see what people tried and what they didn't. Also, for the same reason it would probably help to see Windows 7 and Windows 10 separated.
But if we want to mark it as required, I wouldn't want yet another exploded matrix, and I agree Fedora 26 could be left out (and Windows merged together).
I figured it wouldn't hurt if I add the table marked as optional: https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_mat...
We can now continue discussing how we want it to look like, but at least it's there already.
As a footnote, I hardcoded Fedora 24/25/26 numbers, because I wasn't sure how to do that better. I can use a template, but I don't want the numbers to change once it is converted to a standard install matrix, and I'm not sure what method relval uses to do so. (So this is mainly a question for Adam, whether we can use templates in there or not).
\ On 04/21/2017 05:16 AM, Kamil Paral wrote:
On Wed, Apr 19, 2017 at 2:25 PM, Kamil Paral <kparal@redhat.com mailto:kparal@redhat.com> wrote:
That's the question though - is it required? I thought the test case would be marked as optional. We already require FMW in "Default boot and install" matrix, so Fedora is covered (we just don't know *what* Fedora they used for that). And are any FMW crashes on Win/macOS blocking? I don't think we have a criterion for that (the question is whether we should). So as long as we keep that table optional, I think it's useful to split out the Fedora version, so that we clearly see what people tried and what they didn't. Also, for the same reason it would probably help to see Windows 7 and Windows 10 separated. But if we want to mark it as required, I wouldn't want yet another exploded matrix, and I agree Fedora 26 could be left out (and Windows merged together).I figured it wouldn't hurt if I add the table marked as optional: https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_mat...
We can now continue discussing how we want it to look like, but at least it's there already.
As a footnote, I hardcoded Fedora 24/25/26 numbers, because I wasn't sure how to do that better. I can use a template, but I don't want the numbers to change once it is converted to a standard install matrix, and I'm not sure what method relval uses to do so. (So this is mainly a question for Adam, whether we can use templates in there or not).
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org
I added download links and test results to this https://fedoraproject.org/w/index.php?title=Template:Installation_test_matri...
Please edit these changes if this is in error
satellit
On Fri, 2017-04-21 at 06:46 -0700, Thomas Gilliard wrote:
I added download links and test results to this https://fedoraproject.org/w/index.php?title=Template:Installation_test_matri...
Please edit these changes if this is in error
Sorry, but yes, it is. As the note on the page says, please don't edit it unless you're sure you understand how it works. I will revert it once the site's up again.
On Fri, Apr 21, 2017 at 3:46 PM, Thomas Gilliard satellitgo@gmail.com wrote:
I added download links and test results to this https://fedoraproject.org/w/index.php?title=Template: Installation_test_matrix&diff=next&oldid=491518#Fedora_Media_Writer Please edit these changes if this is in error
satellit
Adam reverted those (thanks Adam). This is a template page, individual results are not supposed to be written there. Results are written into standard test matrices, which are generated from the matrix template. If you find this difficult to distinguish, it's easy - only fill out matrices which are linked in test-announce list :) Thanks.
On Fri, 2017-04-21 at 14:16 +0200, Kamil Paral wrote:
but I don't want the numbers to change once it is converted to a standard install matrix, and I'm not sure what method relval uses to do so.
just for the record, relval / python-wikitcms per se don't do anything magic in this regard, it's all done in the wiki templates. you can (still) create the validation pages correctly by hand, just by using the correct template invocation. the text that relval actually uses to create a validation page is just something like this:
{{subst:Validation_results|testtype=Installation|release=26|milestone=Beta|compose=1.1}}
or:
{{subst:Validation_results|testtype=Installation|release=26|milestone=Beta|date=20170420.n.0}}
and thanks to a heavy dose of wiki template magic, that 'seed text' (as I call it) produces the entire clean validation page. relval's job (some of which is done in python-wikitcms) is knowing what pages to create for a given 'validation event', and what to call them, and what categories they should be in.
On Fri, 2017-04-21 at 14:16 +0200, Kamil Paral wrote:
On Wed, Apr 19, 2017 at 2:25 PM, Kamil Paral kparal@redhat.com wrote:
That's the question though - is it required? I thought the test case would be marked as optional. We already require FMW in "Default boot and install" matrix, so Fedora is covered (we just don't know *what* Fedora they used for that). And are any FMW crashes on Win/macOS blocking? I don't think we have a criterion for that (the question is whether we should).
So as long as we keep that table optional, I think it's useful to split out the Fedora version, so that we clearly see what people tried and what they didn't. Also, for the same reason it would probably help to see Windows 7 and Windows 10 separated.
But if we want to mark it as required, I wouldn't want yet another exploded matrix, and I agree Fedora 26 could be left out (and Windows merged together).
I figured it wouldn't hurt if I add the table marked as optional: https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_mat...
We can now continue discussing how we want it to look like, but at least it's there already.
As a footnote, I hardcoded Fedora 24/25/26 numbers, because I wasn't sure how to do that better. I can use a template, but I don't want the numbers to change once it is converted to a standard install matrix, and I'm not sure what method relval uses to do so. (So this is mainly a question for Adam, whether we can use templates in there or not).
That weird empty mail was a long reply to this which something ate. *sob*
So here's the short version: yeah, you can do it. It involves advanced- level wiki magic, though. I did it:
https://fedoraproject.org/w/index.php?title=Template%3AInstallation_test_mat...
What's going on there is documented at https://www.mediawi ki.org/wiki/Help:Substitution , it's basically all about setting up template invocations within a template so that a substitution occurs when the parent template is being substituted, but doesn't happen in any other circumstance (when the parent template is viewed or saved, or when it's transcluded). (That page explains the difference between transclusion and substitution).
The #expr stuff is basically a reinvention of the FedoraVersionNumber template; that template transcludes CurrentFedoraVersion, so if we just substitute FedoraVersionNumber (or FedoraVersion) it doesn't solve the problem, as the underlying CurrentFedoraVersion usage is still a *transclusion*, so the number would still change in result pages. Which is what we're trying to avoid.
If you're interested in all the details, ask and I'll rewrite the long answer. :)
On Sat, Apr 22, 2017 at 8:22 AM, Adam Williamson <adamwill@fedoraproject.org
wrote:
The #expr stuff is basically a reinvention of the FedoraVersionNumber template; that template transcludes CurrentFedoraVersion, so if we just substitute FedoraVersionNumber (or FedoraVersion) it doesn't solve the problem, as the underlying CurrentFedoraVersion usage is still a *transclusion*, so the number would still change in result pages. Which is what we're trying to avoid.
Eh. Thanks! That's exactly what I wanted to achieve without studying that crazy syntax.
If you're interested in all the details, ask and I'll rewrite the long answer. :)
No, no, I'm good, thanks :)
Definitely split windows. I've had writer crash on 10 but not 7, trying to get a fedora spin (lxde)
On 18 April 2017 15:50:06 GMT+03:00, Lukas Brabec lbrabec@redhat.com wrote:
On Fri, Jan 13, 2017 at 12:32 AM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like
this:
Windows macOS Fedora 24 Fedora 25QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
Thoughts?
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin .
net
http://www.happyassassin.net _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org
Current media writer supports ARM, I would propose to add a row for it, so it could look like this:
Windows macOS Fedora 24 Fedora 25x86_64 iso arm image
where both "x86_64 iso" and "arm image" links to QA:Testcase_USB_fmw _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org
On Wed, 19 Apr 2017 06:51:00 +0200, Allan Mwenda allanitomwesh@gmail.com wrote:
Definitely split windows. I've had writer crash on 10 but not 7, trying to get a fedora spin (lxde)
On 18 April 2017 15:50:06 GMT+03:00, Lukas Brabec lbrabec@redhat.com wrote:
On Fri, Jan 13, 2017 at 12:32 AM, Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! So I finally got around to that 'think about USB test coverage' item that's been on my todo list forever.
I propose we add a table to the Installation Validation page. The purpose is simply to check that writing images is working with mediawriter in the major supported environments: Windows, macOS , and the supported stable Fedora releases. So it could just look like
this:
Windows macOS Fedora 24 Fedora 25QA:Testcase_USB_fmw
with the intent being that we at least check that writing any one release blocking image with mediawriter in each environment.
We could split Windows into 7, 8 and 10 or something, but not sure if it's really necessary...
Thoughts?
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin .
net
http://www.happyassassin.net _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org
Current media writer supports ARM, I would propose to add a row for it, so it could look like this:
Windows macOS Fedora 24 Fedora 25x86_64 iso arm image
where both "x86_64 iso" and "arm image" links to QA:Testcase_USB_fmw _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org
Yeah I mentioned this already, Win10 is a bit different story than anything else when testing FMW.
-
Adam Williamson -
Allan Mwenda -
Chris Murphy -
Kamil Paral -
Lukas Brabec -
Martin Bříza -
Matthew Miller -
Thomas Gilliard