Yesterday I sent out a QA-related proposal that directly affects the release criteria for KDE x86_64 and Workstation x86_64 and (probably) aarch64 images. The proposal is attached below and also available in the test list (including some follow-ups which are recommended to read): https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/t...

Can you please review that proposal and post your thoughts, if you have any? (Responding to test list is preferred, so that we can discuss it in a single place, but responding here is fine as well). Thank you.

---------- Forwarded message --------- From: Kamil Paral kparal@redhat.com Date: Mon, Feb 17, 2020 at 3:20 PM Subject: proposal: Default application functionality criterion reduction To: test test@lists.fedoraproject.org

This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:

All applications that can be launched using the standard graphical

mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test.

*= Background =*

The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.

*= Proposal =*

Change the criterion to something along these lines:

All applications that can be launched using the standard graphical

mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.

For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:

• web browser (e.g. firefox) [4]

• file browser (e.g. nautilus)

• package manager (e.g. gnome-software)

• image viewer (e.g. eog)

• document viewer (e.g. evince)

• text editor (e.g. gedit)

• archive manager (e.g. file-roller)

• terminal emulator (e.g. gnome-terminal) [4]

• problem reporter (e.g. abrt)

If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.

As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:

* word editor (e.g. libreoffice-writer)

* spreadsheet editor (e.g. libreoffice-calc)

* video player (e.g. totem)

* help viewer (e.g. yelp)

I’d like to hear your thoughts on whether they should be included or not. Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).

I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).

Please comment, thanks.

[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_appl...

[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking

[2] https://pagure.io/fedora-iot/issue/23

[3] https://pagure.io/fesco/issue/2339

[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)

[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications