On Mon, Feb 17, 2020 at 3:20 PM Kamil Paral kparal@redhat.com wrote:
This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test.
*= Background =*
The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.
For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:
web browser (e.g. firefox) [4]
file browser (e.g. nautilus)
package manager (e.g. gnome-software)
image viewer (e.g. eog)
document viewer (e.g. evince)
text editor (e.g. gedit)
archive manager (e.g. file-roller)
terminal emulator (e.g. gnome-terminal) [4]
problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not. Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).
Please comment, thanks.
[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_appl...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)
[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
I haven't received much feedback on this proposal. On test list and kde list, there was a short follow-up regarding the list of release-blocking applications. On kde list, Kevin disagreed because it lowers the guaranteed quality bar for KDE, but he didn't continue discussing it with me. I haven't received any alternative proposals either. On desktop list, there was silence, but Chris Murphy gave me a thumbs up on IRC.
I'll give it a few more days, and unless somebody yells hard (and ideally also provides some constructive ways how to optimize desktop testing, because we need to cut it down /somehow/), I'll put the change into effect. I'll also probably move the help viewer into the list of release-blocking applications, based on the discussion I saw.
Thanks, Kamil Fedora QA
On Tue, Mar 3, 2020 at 2:44 PM Kamil Paral kparal@redhat.com wrote:
On Mon, Feb 17, 2020 at 3:20 PM Kamil Paral kparal@redhat.com wrote:
This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test.
*= Background =*
The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.
For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:
web browser (e.g. firefox) [4]
file browser (e.g. nautilus)
package manager (e.g. gnome-software)
image viewer (e.g. eog)
document viewer (e.g. evince)
text editor (e.g. gedit)
archive manager (e.g. file-roller)
terminal emulator (e.g. gnome-terminal) [4]
problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not. Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).
Please comment, thanks.
[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_appl...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)
[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
I haven't received much feedback on this proposal. On test list and kde list, there was a short follow-up regarding the list of release-blocking applications. On kde list, Kevin disagreed because it lowers the guaranteed quality bar for KDE, but he didn't continue discussing it with me. I haven't received any alternative proposals either. On desktop list, there was silence, but Chris Murphy gave me a thumbs up on IRC.
I'll give it a few more days, and unless somebody yells hard (and ideally also provides some constructive ways how to optimize desktop testing, because we need to cut it down /somehow/), I'll put the change into effect. I'll also probably move the help viewer into the list of release-blocking applications, based on the discussion I saw.
Hi Kamil,
I think this change makes sense. +1 from me as well.
Kalev
On Tue, Mar 3, 2020 at 2:43 PM Kamil Paral kparal@redhat.com wrote:
I haven't received much feedback on this proposal. On test list and kde list, there was a short follow-up regarding the list of release-blocking applications. On kde list, Kevin disagreed because it lowers the guaranteed quality bar for KDE, but he didn't continue discussing it with me. I haven't received any alternative proposals either. On desktop list, there was silence, but Chris Murphy gave me a thumbs up on IRC.
I'll give it a few more days, and unless somebody yells hard (and ideally also provides some constructive ways how to optimize desktop testing, because we need to cut it down /somehow/), I'll put the change into effect. I'll also probably move the help viewer into the list of release-blocking applications, based on the discussion I saw.
I talked to Rex Dieter and Kevin Kofler on #fedora-kde IRC, and they raised a concern about about this sentence: "If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements."
Their worry is that with this approach it is problematic to have a primary application for the majority of use cases, and then a secondary application for other use cases/different users, because if the secondary app works fine, the release will not be stopped even in case of serious issues in the primary app. Another potential problem is when another application is pulled into the distribution e.g. by a dependency, and even when it is not seen as a big issue, it can still make the original app be completely ignored in terms of release blocking. We might have found a way to avoid this by changing the rule this way:
""" If there are multiple applications of the same type (e.g. several web browsers), the primary/default one must satisfy the requirements. If the primary/default application can't be determined, only one of said applications must satisfy the requirements.
Note: Determining primary applications The usual way to determine a primary/default application is to look into system configuration where default applications are set (e.g. gnome-control-center -> Default Applications), or to launch the default application on an appropriate file type (e.g. double click on an .odt file in a file browser), or to look into a favorites menu section (e.g. KDE menu -> Favorites). """
I think the change is reasonable and I adjust my proposal to include it. If you don't like it, or like it, or think it should be changed somehow, please comment. Thanks!
On Wed, Mar 4, 2020 at 1:55 PM Kamil Paral kparal@redhat.com wrote:
""" If there are multiple applications of the same type (e.g. several web browsers), the primary/default one must satisfy the requirements. If the primary/default application can't be determined, only one of said applications must satisfy the requirements.
Note: Determining primary applications The usual way to determine a primary/default application is to look into system configuration where default applications are set (e.g. gnome-control-center -> Default Applications), or to launch the default application on an appropriate file type (e.g. double click on an .odt file in a file browser), or to look into a favorites menu section (e.g. KDE menu -> Favorites). """
I like it. This seems like a reasonable way to address their concerns. Another approach could be to have the maintaining SIG provide their preferred default and in cases where none is provided, at least one has to work. So maybe, e.g. Llama Spin doesn't care *which* web browser works, but Alpaca Spin says Firefox has to work even if another browser is functional.
This essentially lets us go with the original proposal as a default case instead of having to go through and figure it out (because it may change from release-to-release, so it has to be verified every cycle)
On Wed, Mar 4, 2020 at 9:28 PM Ben Cotton bcotton@fedoraproject.org wrote:
On Wed, Mar 4, 2020 at 1:55 PM Kamil Paral kparal@redhat.com wrote:
""" If there are multiple applications of the same type (e.g. several web
browsers), the primary/default one must satisfy the requirements. If the primary/default application can't be determined, only one of said applications must satisfy the requirements.
Note: Determining primary applications The usual way to determine a primary/default application is to look into
system configuration where default applications are set (e.g. gnome-control-center -> Default Applications), or to launch the default application on an appropriate file type (e.g. double click on an .odt file in a file browser), or to look into a favorites menu section (e.g. KDE menu -> Favorites).
"""
I like it. This seems like a reasonable way to address their concerns. Another approach could be to have the maintaining SIG provide their preferred default and in cases where none is provided, at least one has to work. So maybe, e.g. Llama Spin doesn't care *which* web browser works, but Alpaca Spin says Firefox has to work even if another browser is functional.
This essentially lets us go with the original proposal as a default case instead of having to go through and figure it out (because it may change from release-to-release, so it has to be verified every cycle)
Yes, that's also an option. However, I intentionally tried to stay away from hard-coding concrete application names in release criteria (doesn't matter who maintains them, whether QA or some SIG). There's a maintenance cost I'd like to avoid. I'm afraid that people will forget to update the criteria when there are changes in the desktop apps. Also, if the list grows and people can't remember it, it might actually be easier to figure out the default application from the desktop environment than open the criteria, find the relevant section, and learn the application names. So I'd rather go with the generic solution, and if it doesn't work well, we can always change it to what you said or something similar.
It has been 7 days since the last response to the thread and I received no further complaints, so I've put the criterion into effect: https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria...
I've done slight tweaks compared to the original proposal: * Help viewer has been included in the covered application types. * Primary/default application is the one that must satisfy requirements in case of multiple same-type applications present (per the proposal by KDE folks mentioned in this thread). * "only one of said applications must satisfy the requirements" has been changed to "at least one of said applications...", which has the same meaning but is clearer (I hope) * "file browser" was renamed to "file manager", which seems to be the more official term (when looking at wikipedia) * application type examples were put into a footnote, to make the criterion itself more readable (and avoid some potential confusion when people see exact application names in there and miss the "e.g.")
I'll work on adjusting associated test cases, to put them in line with the new criterion.
desktop@lists.stg.fedoraproject.org
-
Ben Cotton -
Kalev Lember -
Kamil Paral -
Zbigniew Jędrzejewski-Szmek