Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
What does this mean for Fedora and CentOS? This discussion is in part to figure that out. Based on some very brief analysis, the following should hold:
- RHEL customers should continue to file support cases through the Red Hat Customer portal, which will remain consistent regardless of the backend tooling used.
- There is no imminent retirement of the Red Hat Bugzilla instance being planned at this time. RHEL 7, 8, and 9 will continue to use bugzilla in some form and RHEL 9 has a very long lifecycle.
- Fedora Linux and EPEL have their own Bugzilla product families and are not directly impacted in their own workflows by the choice to use only issues.redhat.com for RHEL. - There will be impacts on existing documentation that provide guidance on requesting things from RHEL in various places like EPEL. We will be happy to help adjust these.
- CentOS Stream contribution and bug reporting workflows will be adjusted to use issues.redhat.com instead of bugzilla in the relevant places. This should apply to all versions of CentOS Stream for a unified contributor workflow. This will happen gradually as we discover the best workflow to use.
If there are other impacts that you can think of, please raise them on this thread. We’d like to ensure we’re covering as much as possible as this rolls out.
josh
On Mon, 2022-03-07 at 12:44 -0500, Josh Boyer wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
Thanks for sharing this Josh. Would you be able to expand on why Red Hat chose to use Jira specifically here, and what benefits do you forsee this switch will bring to CentOS down the road?
Cheers Davide
On Wed, Mar 9, 2022 at 5:05 PM Davide Cavalca via CentOS-devel centos-devel@centos.org wrote:
On Mon, 2022-03-07 at 12:44 -0500, Josh Boyer wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
Thanks for sharing this Josh. Would you be able to expand on why Red Hat chose to use Jira specifically here, and what benefits do you forsee this switch will bring to CentOS down the road?
Red Hat has long used Jira in various parts of the company, with JBoss and other products being heavy users for quite some time. Within the past few years we've consolidated a number of different instances into the single issues.redhat.com instance. That has enabled us to more easily plan and coordinate across our product portfolio.
RHEL's decision is certainly informed by that same overall direction, but also specifically influenced by the limiting factors of using multiple tools to accomplish similar things. We've been using Bugzilla since Red Hat Linux days, and while it has served us well as a defect tracking backend, it was never designed to handle the complexity we have for planning and coordinating the varying scope of work we have. Aligning to a single tool will help us refine our processes internally.
As for benefits to CentOS, or any of the other upstream projects we interact with, we'll have to see. I think the intention is to minimize overall impact to start with, and then evolve our workflows to bring further benefit where we can. That being said, we are certainly aware that we need to default to open issues to allow us to collaborate directly in the open source manner we've long held to be fundamental to our communities. I expect that approach to behave very similarly to how Bugzilla is used today.
josh
On Mon, Mar 7, 2022, at 12:44 PM, Josh Boyer wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
I think it's unfortunate to replace the FOSS Bugzilla with proprietary software. I am eternally conflicted about this with respect to GitHub (xref https://blog.verbum.org/2020/12/03/still-on-github/ ) but...Jira is not as compelling of a user experience upgrade.
A continual challenge related to this I feel is using the same software to track product bugs with potentially sensitive customer data in it, and public open development.
To link these things, I quite commonly move Bugzilla discussion that has no need to be private to Github, because I know Github is always public (from our PoV).
One thing that may help is to at least use different themes (e.g. blue colors for public CentOS issues, red for RHEL?) on issues.redhat.com.
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Yes, I personally think gitlab.com would be a good replacement for src.fedoraproject.org and bugzilla.redhat.com.
- Ken
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Yes, I personally think gitlab.com would be a good replacement for src.fedoraproject.org and bugzilla.redhat.com.
I disagree as the interface for agile development like sprints, kanban and other team tools is rudimentary at best and really not suitable for anything more than the basics and is not a patch on the functionality and plugin ecosystem that is available in Jira. The gitlab functionality there is a toy in comparison.
On Fri, Mar 11, 2022, 8:53 AM Peter Robinson pbrobinson@gmail.com wrote:
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org
wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Yes, I personally think gitlab.com would be a good replacement for src.fedoraproject.org and bugzilla.redhat.com.
I disagree as the interface for agile development like sprints, kanban and other team tools is rudimentary at best and really not suitable for anything more than the basics and is not a patch on the functionality and plugin ecosystem that is available in Jira. The gitlab functionality there is a toy in comparison.
Toys are fun, one of the four Fs for Fedora. I'm not interested in bringing sprints or kanban to Fedora. I am interested in making it fun for volunteers :)
- Ken
On Fri, 2022-03-11 at 13:52 +0000, Peter Robinson wrote:
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Yes, I personally think gitlab.com would be a good replacement for src.fedoraproject.org and bugzilla.redhat.com.
I disagree as the interface for agile development like sprints, kanban and other team tools is rudimentary at best and really not suitable for anything more than the basics and is not a patch on the functionality and plugin ecosystem that is available in Jira. The gitlab functionality there is a toy in comparison.
Peter, Bugzilla has no support for any sprints or kanban at all, so I do not see how this is relevant. Unless your claim is that Fedora absolutely needs a sprint/kanban based organizational tool which would surprise me.
I personally do not think Fedora should tie itself to Red Hat's Jira. For community oriented development Jira is simply not a great tool IMO, because it is built with a "traditional" organization in mind, not for community use.
Keeping it simple is actually an existential requirement for Fedora where we have so many drive-by volunteers. Complex on-boarding becomes quickly an impenetrable barrier to participation. This is another reason I think common forges issue systems, while limited are a better fit for community issue reporting than either Bugzilla or Jira, because they are simple and immediately available.
You can easily write automation to pull in/mirror issues into your own Jira projects if that's what you use for work and feel the need for, IMO.
And just to be clear I am both a *heavy* Jira and Bugzilla user (including writing automation for both and other stuff via bots) for work, so I think I can say I know what I am talking about.
Simo.
* Simo Sorce:
On Fri, 2022-03-11 at 13:52 +0000, Peter Robinson wrote:
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Yes, I personally think gitlab.com would be a good replacement for src.fedoraproject.org and bugzilla.redhat.com.
I disagree as the interface for agile development like sprints, kanban and other team tools is rudimentary at best and really not suitable for anything more than the basics and is not a patch on the functionality and plugin ecosystem that is available in Jira. The gitlab functionality there is a toy in comparison.
Bugzilla has no support for any sprints or kanban at all, so I do not see how this is relevant.
These features are disabled for the Fedora product in bugzilla.redhat.com, but the code is there. Perhaps you have different views what tool support for these processes should look like?
Thanks, Florian
On Sat, 2022-03-12 at 10:15 +0100, Florian Weimer wrote:
- Simo Sorce:
On Fri, 2022-03-11 at 13:52 +0000, Peter Robinson wrote:
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Yes, I personally think gitlab.com would be a good replacement for src.fedoraproject.org and bugzilla.redhat.com.
I disagree as the interface for agile development like sprints, kanban and other team tools is rudimentary at best and really not suitable for anything more than the basics and is not a patch on the functionality and plugin ecosystem that is available in Jira. The gitlab functionality there is a toy in comparison.
Bugzilla has no support for any sprints or kanban at all, so I do not see how this is relevant.
These features are disabled for the Fedora product in bugzilla.redhat.com, but the code is there. Perhaps you have different views what tool support for these processes should look like?
This was exactly my point, Fedora does not use sprints/kanban with bugzilla now, so it does not seem like a fair comment to complain that another bug tracker proposed as replacement does not have them either, unless suddenly there is a need for them (I personally see no need to have a fedora specific sprint/kanban system).
Simo.
On Fri, Mar 11, 2022 at 01:52:53PM +0000, Peter Robinson wrote:
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Yes, I personally think gitlab.com would be a good replacement for src.fedoraproject.org and bugzilla.redhat.com.
I disagree as the interface for agile development like sprints, kanban and other team tools is rudimentary at best and really not suitable for anything more than the basics and is not a patch on the functionality and plugin ecosystem that is available in Jira. The gitlab functionality there is a toy in comparison.
The comparison is reasonable, but I would ask if support for things like sprints, kanban, etc is actually important for Fedora's bug reporting needs ? Personally I can't see that it is given the way I see that maintainers work in Fedora in an adhoc way, usually independantly, and light on any formally required processes or bureaucracy.
When I look at how we've evolved & used Bugzilla over the years for RHEL, I feel like much of the unpleasantness has been because we've tried to take a bug tracking tool, and bend it into being a project management tool. Thankfully Fedora kept its use of bugzilla simple, and didn't try to make it apply specific dev practices/workflows.
IOW to me the appeal of the git forge bug trackers is precisely that they are very simple, to both reporters and maintainers. They may not be suitable for RHEL, but they feel like a decent fit for Fedora.
IMHO if there are times when some group of Fedora maintainers is closely collaborating and wants to use agile dev practice, it ought to be possible to use a separate tool for those tasks, linking to the bug tracker if & when relevant.
With regards, Daniel
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
I'm not sure how active the use of Red Hat Bugzilla is outside of the distribution space. I have no concerns about Bugzilla being yanked away from underneath us, but this does give us an opportunity to evaluate what we want in a bug tracker and whether or not Bugzilla meets the needs of the Fedora Project in 2022. Later this year, I'll be conducting a survey to see what features we want from a bug tracker so that we can start thinking about what best suits our needs.
To be clear: this is not a "we're moving off of Bugzilla next year" statement. It's a "let's use this opportunity to make sure what we're doing works for us" statement. I am not concerned about Red Hat Bugzilla going away in the near- or medium-term. And perhaps not even in the long-term (although nothing lasts forever).
I guess what I'm saying for now is let's not spend time speculating about what might happen or making rushed plans to move our bug tracking. I know that's not what Colin is doing, I just want to preempt the discussion. Look for a survey sometime after the F36 final release.
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis
On 10. 03. 22 17:33, Ben Cotton wrote:
On Thu, Mar 10, 2022 at 9:45 AM Colin Walters walters@verbum.org wrote:
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
I'm not sure how active the use of Red Hat Bugzilla is outside of the distribution space. I have no concerns about Bugzilla being yanked away from underneath us, but this does give us an opportunity to evaluate what we want in a bug tracker and whether or not Bugzilla meets the needs of the Fedora Project in 2022. Later this year, I'll be conducting a survey to see what features we want from a bug tracker so that we can start thinking about what best suits our needs.
To be clear: this is not a "we're moving off of Bugzilla next year" statement. It's a "let's use this opportunity to make sure what we're doing works for us" statement. I am not concerned about Red Hat Bugzilla going away in the near- or medium-term. And perhaps not even in the long-term (although nothing lasts forever).
I guess what I'm saying for now is let's not spend time speculating about what might happen or making rushed plans to move our bug tracking. I know that's not what Colin is doing, I just want to preempt the discussion. Look for a survey sometime after the F36 final release.
Hey Ben,
I volunteer to help you review the questions before the survey is actually conducted, if you are interested.
On Thu, 2022-03-10 at 09:44 -0500, Colin Walters wrote:
On Mon, Mar 7, 2022, at 12:44 PM, Josh Boyer wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
I think it's unfortunate to replace the FOSS Bugzilla with proprietary software. I am eternally conflicted about this with respect to GitHub (xref https://blog.verbum.org/2020/12/03/still-on-github/ ) but...Jira is not as compelling of a user experience upgrade.
A continual challenge related to this I feel is using the same software to track product bugs with potentially sensitive customer data in it, and public open development.
To link these things, I quite commonly move Bugzilla discussion that has no need to be private to Github, because I know Github is always public (from our PoV).
One thing that may help is to at least use different themes (e.g. blue colors for public CentOS issues, red for RHEL?) on issues.redhat.com.
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Given how we use bugzilla (we do not really use any big feature there) in Fedora I would give a big +1 to use an issue tracker embedded in the forge we use to store the packages (whether that is pagure, gitlab, github, or something else).
Also I always resented that I need two separate accounts to deal with Fedora packages, I think reducing that to host all in the same place with the same authentication will also be a positive factor in fostering collaboration, less barriers. It will also reduce administrative overhead of having to configure components/ownership/etc in multiple places and what not.
Finally by having issues and code in the same place it means we can easily connect commits/PRs/MRs to the issues meaning our issue tracker a lot more useful, and will allow us to have better content also in our updates, where today associating an update to an issue (a bz) is not happening as well as it could.
HTH, Simo.
On Thursday, 10 March 2022 at 17:51, Simo Sorce wrote: [...]
Also I always resented that I need two separate accounts to deal with Fedora packages,
It's been possible to log in with FAS credentials (automatically if you have an active Kerberos ticket) into bugzilla for quite some time now. I still have my old bugzilla account but I'm not sure it's required anymore.
Regards, Dominik
On Thu, 2022-03-10 at 19:28 +0100, Dominik 'Rathann' Mierzejewski wrote:
On Thursday, 10 March 2022 at 17:51, Simo Sorce wrote: [...]
Also I always resented that I need two separate accounts to deal with Fedora packages,
It's been possible to log in with FAS credentials (automatically if you have an active Kerberos ticket) into bugzilla for quite some time now. I still have my old bugzilla account but I'm not sure it's required anymore.
Ah thanks, I missed that change, obviously my experience is now almost 2 decades long since I had to create my account, but I still think that having bugs and code in the same place is a big win for a project like Fedora, it is the same model followed by most upstream projects at this point and seem to be working well.
Regards, Dominik -- Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Mar 10, 2022 at 01:41:15PM -0500, Simo Sorce wrote:
On Thu, 2022-03-10 at 19:28 +0100, Dominik 'Rathann' Mierzejewski wrote:
On Thursday, 10 March 2022 at 17:51, Simo Sorce wrote: [...]
Also I always resented that I need two separate accounts to deal with Fedora packages,
It's been possible to log in with FAS credentials (automatically if you have an active Kerberos ticket) into bugzilla for quite some time now. I still have my old bugzilla account but I'm not sure it's required anymore.
Ah thanks, I missed that change, obviously my experience is now almost 2 decades long since I had to create my account, but I still think that having bugs and code in the same place is a big win for a project like Fedora, it is the same model followed by most upstream projects at this point and seem to be working well.
Yes, I find having the issue tracker alongside the code repo is a good thing for usability. For a start you have a list of bugs to browse, visible from a single click. No need to do a search, excluding countless products/projects that are unrelated to Fedora and component you want. Similarly, you get the link to file a new bug directly against the code you're looking at, compared to bugzilla where you must navigate through several pages to even get to selecting Fedora, and then again select the component in the form.
The Red Hat Jira instance mentioned earlier is just as bad as Bugzilla in this respect, actually probably worse. It wants to show you all bugs in the system straight away, until you type in the right search query terms to restrict it down to a particular product and component you are actually looking for bugs in.
With regards, Daniel
On Fri, 2022-03-11 at 08:59 +0000, Daniel P. Berrangé wrote:
On Thu, Mar 10, 2022 at 01:41:15PM -0500, Simo Sorce wrote:
On Thu, 2022-03-10 at 19:28 +0100, Dominik 'Rathann' Mierzejewski wrote:
On Thursday, 10 March 2022 at 17:51, Simo Sorce wrote: [...]
Also I always resented that I need two separate accounts to deal with Fedora packages,
It's been possible to log in with FAS credentials (automatically if you have an active Kerberos ticket) into bugzilla for quite some time now. I still have my old bugzilla account but I'm not sure it's required anymore.
Ah thanks, I missed that change, obviously my experience is now almost 2 decades long since I had to create my account, but I still think that having bugs and code in the same place is a big win for a project like Fedora, it is the same model followed by most upstream projects at this point and seem to be working well.
Yes, I find having the issue tracker alongside the code repo is a good thing for usability. For a start you have a list of bugs to browse, visible from a single click. No need to do a search, excluding countless products/projects that are unrelated to Fedora and component you want. Similarly, you get the link to file a new bug directly against the code you're looking at, compared to bugzilla where you must navigate through several pages to even get to selecting Fedora, and then again select the component in the form.
But what about bugs filled on the wrong component (I maintain both initial-setup & firstboot, lets say some users have a different idea about what these project names mean) or bugs that are filled on one component but turn out to be caused by another one ?
If all the bugs are effectively attached to a forge ditgit repo, what to do you effectively need to take them a attach them to a different repo ? Close the old one and open a new one on the correct repo ? If so, there should be at least some automation for this.
Another thing, tracking bugs or bugs/RFEs impacting multiple componenets, where to put those ? Wikipa pages like with the change process or some empty forge repo perhaps ?
Also, while searching in Bugzilla can be quite a mess due to everything being one one big pile by default, it can also be very useful to search for say error messages, stack traces and other similar behavior across bugs. It would be good to check if the forge solution also supports that natively, not just searching the bugs attached to a project/repo.
The Red Hat Jira instance mentioned earlier is just as bad as Bugzilla in this respect, actually probably worse. It wants to show you all bugs in the system straight away, until you type in the right search query terms to restrict it down to a particular product and component you are actually looking for bugs in.
With regards, Daniel --
: https://berrange.com%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0 -o- https://www.flickr.com/photos/dberrange%C2%A0:%7C : https://libvirt.org%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0 -o- https://fstop138.berrange.com%C2%A0:%7C : https://entangle-photo.org%C2%A0%C2%A0%C2%A0 -o- https://www.instagram.com/dberrange%C2%A0:%7C
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Fri, Mar 11, 2022 at 02:34:29PM +0100, mkolman@redhat.com wrote:
On Fri, 2022-03-11 at 08:59 +0000, Daniel P. Berrangé wrote:
On Thu, Mar 10, 2022 at 01:41:15PM -0500, Simo Sorce wrote:
On Thu, 2022-03-10 at 19:28 +0100, Dominik 'Rathann' Mierzejewski wrote:
On Thursday, 10 March 2022 at 17:51, Simo Sorce wrote: [...]
Also I always resented that I need two separate accounts to deal with Fedora packages,
It's been possible to log in with FAS credentials (automatically if you have an active Kerberos ticket) into bugzilla for quite some time now. I still have my old bugzilla account but I'm not sure it's required anymore.
Ah thanks, I missed that change, obviously my experience is now almost 2 decades long since I had to create my account, but I still think that having bugs and code in the same place is a big win for a project like Fedora, it is the same model followed by most upstream projects at this point and seem to be working well.
Yes, I find having the issue tracker alongside the code repo is a good thing for usability. For a start you have a list of bugs to browse, visible from a single click. No need to do a search, excluding countless products/projects that are unrelated to Fedora and component you want. Similarly, you get the link to file a new bug directly against the code you're looking at, compared to bugzilla where you must navigate through several pages to even get to selecting Fedora, and then again select the component in the form.
But what about bugs filled on the wrong component (I maintain both initial-setup & firstboot, lets say some users have a different idea about what these project names mean) or bugs that are filled on one component but turn out to be caused by another one ?
If all the bugs are effectively attached to a forge ditgit repo, what to do you effectively need to take them a attach them to a different repo ? Close the old one and open a new one on the correct repo ? If so, there should be at least some automation for this.
GitLab at least has a "Move issue" button that lets you move issues between repositories. Behind the scenes it is essentially re-creating the issue in the new repo and closing the old one.
Another thing, tracking bugs or bugs/RFEs impacting multiple componenets, where to put those ? Wikipa pages like with the change process or some empty forge repo perhaps ?
Bugzilla was no good at tracking stuff across components either. All the cross-component cloning and linking people did turned into a big mess, which is a large part of why people increasingly disliked Bugzilla for RHEL.
At least from Fedora's POV, the Feature page acts as a central point of information linking off to many different places. I don't know if that's the best answer, but bending a component level bug tracker to that does is not it, no matter what bug tracker we choose, because the granularity is a mismatch.
Also, while searching in Bugzilla can be quite a mess due to everything being one one big pile by default, it can also be very useful to search for say error messages, stack traces and other similar behavior across bugs. It would be good to check if the forge solution also supports that natively, not just searching the bugs attached to a project/repo.
GitLab lets you search across multiple repos at the group level
Regards, Daniel
On Thu, 2022-03-10 at 11:51 -0500, Simo Sorce wrote:
Given how we use bugzilla (we do not really use any big feature there)
Uhhh! Whoah! Yes, we certainly do!
I think people may be underestimating how much of Bugzilla we use. Replacing it would *certainly* not be a drop-in operation.
We use the dependency tracking extensively, as part of core Fedora processes (the release blocker process and the Change process at least). We use flags for package review. We use keywords and whiteboards for various housekeeping and documentation processes. I know a lot of people use Bugzilla saved searches.
I'm not saying we need to keep Bugzilla forever, but we need to be realistic about how disruptive a change moving off it would be. It is not a simple thing to do.
On Thu, Mar 10, 2022 at 10:51 AM Simo Sorce simo@redhat.com wrote:
On Thu, 2022-03-10 at 09:44 -0500, Colin Walters wrote:
On Mon, Mar 7, 2022, at 12:44 PM, Josh Boyer wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
My only real concern here from a Fedora standpoint is security tracking. As it stands, the security team files fedora bugs for us linking back to the umbrella issue where most discussion happens. I would really miss that workflow if it were to break as unfortunately the kernel does get a number of CVEs to address (we are at 43 so far in 2022, we had 211 in 2021).
I think it's unfortunate to replace the FOSS Bugzilla with proprietary software. I am eternally conflicted about this with respect to GitHub (xref https://blog.verbum.org/2020/12/03/still-on-github/ ) but...Jira is not as compelling of a user experience upgrade.
A continual challenge related to this I feel is using the same software to track product bugs with potentially sensitive customer data in it, and public open development.
To link these things, I quite commonly move Bugzilla discussion that has no need to be private to Github, because I know Github is always public (from our PoV).
One thing that may help is to at least use different themes (e.g. blue colors for public CentOS issues, red for RHEL?) on issues.redhat.com.
Long term if Bugzilla slowly morphs into only being used by Fedora, personally I'd prefer to have bugs/issues in gitlab instead.
Given how we use bugzilla (we do not really use any big feature there) in Fedora I would give a big +1 to use an issue tracker embedded in the forge we use to store the packages (whether that is pagure, gitlab, github, or something else).
I would be very much against this move to integrate with the source forge. While It would probably work well for large parts of the distribution, it would be a bit of a nightmare for something like kernel with the number of bugs that it gets, and the rate of moving through things. I am not advocating for Jira here either, but at least what I currently see in gitlab and pagure would be a bit of a nightmare for me. I do very much like the idea of being able to perhaps tie bugs to kernel versions rather than Fedora releases though. It would be a useful feature in whatever we might choose if we were to move.
Justin
Also I always resented that I need two separate accounts to deal with Fedora packages, I think reducing that to host all in the same place with the same authentication will also be a positive factor in fostering collaboration, less barriers. It will also reduce administrative overhead of having to configure components/ownership/etc in multiple places and what not.
Finally by having issues and code in the same place it means we can easily connect commits/PRs/MRs to the issues meaning our issue tracker a lot more useful, and will allow us to have better content also in our updates, where today associating an update to an issue (a bz) is not happening as well as it could.
HTH, Simo.
-- Simo Sorce RHEL Crypto Team Red Hat, Inc
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
* Josh Boyer:
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
Thanks for posting this publicly.
- Fedora Linux and EPEL have their own Bugzilla product families and
are not directly impacted in their own workflows by the choice to use only issues.redhat.com for RHEL. - There will be impacts on existing documentation that provide guidance on requesting things from RHEL in various places like EPEL. We will be happy to help adjust these.
There is already an “FC” project on issues.redhat.com, into which Fedora bugs can be mirrored from bugzilla.redhat.com. Should we expose the mirror+ Bugzilla flag publicly, and make the FC project public, so that people can experiment with that?
If there are other impacts that you can think of, please raise them on this thread. We’d like to ensure we’re covering as much as possible as this rolls out.
What is going to happen to the CentOS Mantis instance https://bugs.centos.org/? From the looks of it, it probably should just be switched off?
Thanks, Florian
On Fri, Mar 11, 2022, at 04:55, Florian Weimer wrote:
- Josh Boyer:
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
Thanks for posting this publicly.
- Fedora Linux and EPEL have their own Bugzilla product families and
are not directly impacted in their own workflows by the choice to use only issues.redhat.com for RHEL. - There will be impacts on existing documentation that provide guidance on requesting things from RHEL in various places like EPEL. We will be happy to help adjust these.
There is already an “FC” project on issues.redhat.com, into which Fedora bugs can be mirrored from bugzilla.redhat.com. Should we expose the mirror+ Bugzilla flag publicly, and make the FC project public, so that people can experiment with that?
If there are other impacts that you can think of, please raise them on this thread. We’d like to ensure we’re covering as much as possible as this rolls out.
What is going to happen to the CentOS Mantis instance https://bugs.centos.org/? From the looks of it, it probably should just be switched off?
Since we've moved CentOS Stream to bugzilla/jira I think it will make more and more sense to look at deduplicating the number of bug trackers we have. CentOS Linux 7 bugs are still nominally tracked in Mantis, though.
We do not have active plans to retire bugs.centos.org yet.
--Brian
Thanks, Florian _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Mon, 2022-03-07 at 12:44 -0500, Josh Boyer wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
What does this mean for Fedora and CentOS? This discussion is in part to figure that out. Based on some very brief analysis, the following should hold:
- RHEL customers should continue to file support cases through the Red
Hat Customer portal, which will remain consistent regardless of the backend tooling used.
- There is no imminent retirement of the Red Hat Bugzilla instance
being planned at this time. RHEL 7, 8, and 9 will continue to use bugzilla in some form and RHEL 9 has a very long lifecycle.
- Fedora Linux and EPEL have their own Bugzilla product families and
are not directly impacted in their own workflows by the choice to use only issues.redhat.com for RHEL. - There will be impacts on existing documentation that provide guidance on requesting things from RHEL in various places like EPEL. We will be happy to help adjust these.
- CentOS Stream contribution and bug reporting workflows will be
adjusted to use issues.redhat.com instead of bugzilla in the relevant places. This should apply to all versions of CentOS Stream for a unified contributor workflow. This will happen gradually as we discover the best workflow to use.
If there are other impacts that you can think of, please raise them on this thread. We’d like to ensure we’re covering as much as possible as this rolls out.
So if I'm understanding this correctly, the ultimate consequence here is "Red Hat Bugzilla might go away, or stop being maintained, at whatever point it's no longer needed for RHEL 9", right?
That could obviously have pretty significant consequences for Fedora. Bugzilla isn't only an issue tracker for Fedora; we run some significant processes through it, notably the Change process, the blocker/FE bug process, and the prioritized bug process. In A World Without Bugzilla all of those would need adapting (and their documentation updating). There's fairly tight integration between Bodhi and Bugzilla, which would need to be redesigned. Those are just things I can think of off the top of my head. There are also a couple of decades worth of internet links to Fedora issues on RH Bugzilla, of course.
I guess the two big choices for Fedora if RH said "we're not maintaining Bugzilla any more" would be 1) take over maintaining Bugzilla or 2) switch to something else. 1) would probably be the path of least resistance, I guess.
This does also kinda lead to a larger question for me, trying to wear both Red Hat and Fedora hats at the same time[0]. I wonder if we're kind of lacking a...mechanism, for want of a better word, to handle the *generic* case here. Let's rewind to Ye Olde Days, when "the Fedora project" first started. At that point Fedora and Red Hat shared a lot of tooling and infrastructure, and this was useful to both sides in many ways; it saves on development costs and it makes it easy for people to work in both worlds. With my Red Hat on, I think I'm allowed to say that internally we often talk about this being desirable - having consistency between how X is done in Fedora and how it's done for RHEL - and it obviously has benefits to Fedora too (it means we don't have to find the resources to do that same work at Fedora level).
However, situations like this make me wonder if we might have an issue with keeping shared infra/tooling where it's desirable. It seems like this is a decision/conversation that's been happening within RH, about what makes sense for RH in terms of RHEL development. AFAIK this is the first time it's been formally talked about in a Fedora context, and the messaging is "RH has already decided to stop using Bugzilla for RHEL after 9". In other words, RH has decided on its own to move away from something that is part of the shared RH/Fedora "heritage way of doing things".
I'm not saying that's wrong, but as I said it does make me wonder whether, if both sides do find shared tooling/approaches beneficial, we might want to approach this kind of potential change differently in future. Otherwise it does seem like we could sort of gradually drift apart, with no explicit intention to do so, and lose the benefits of shared tooling and process. Unless the ultimate outcome of this is "Fedora adopts issues.redhat.com for bug tracking" - which would be a possibility, but doesn't seem like a certainty - the result will be that we go from having a shared bug tracker, with the benefits of shared maintenance and being able to easily clone or reference bugs between Fedora and RHEL, to each maintaining our own bug tracker and not having those benefits.
Of course, there would be sensitivities in developing such a process - it could look a lot like Red Hat telling Fedora how to do stuff, which I think isn't exactly the relationship we want to have. But at the same time I'm not sure "Red Hat or Fedora just deciding unilaterally to stop using this thing they'd previously both used" is always the best choice either.
[0]: http://trenchescomic.com/comic/post/hats-off
Hi Adam,
Adam Williamson adamwill@fedoraproject.org writes:
snip
That could obviously have pretty significant consequences for Fedora. Bugzilla isn't only an issue tracker for Fedora; we run some significant processes through it, notably the Change process, the blocker/FE bug process, and the prioritized bug process. In A World Without Bugzilla all of those would need adapting (and their documentation updating). There's fairly tight integration between Bodhi and Bugzilla, which would need to be redesigned. Those are just things I can think of off the top of my head. There are also a couple of decades worth of internet links to Fedora issues on RH Bugzilla, of course.
I guess the two big choices for Fedora if RH said "we're not maintaining Bugzilla any more" would be 1) take over maintaining Bugzilla or 2) switch to something else. 1) would probably be the path of least resistance, I guess.
Short term it is the path of the least resistance, but at least what I've heard from $dayjob, maintaining a Bugzilla instance is no easy task, as they are often customized (via non-upstream patches) and this all needs to be maintained. I cannot speak for our infra team, but I really don't know if they'd like yet another huge service, because this effectively means they'd have to take over maintenance of bugzilla.redhat.com...
This does also kinda lead to a larger question for me, trying to wear both Red Hat and Fedora hats at the same time[0]. I wonder if we're kind of lacking a...mechanism, for want of a better word, to handle the *generic* case here. Let's rewind to Ye Olde Days, when "the Fedora project" first started. At that point Fedora and Red Hat shared a lot of tooling and infrastructure, and this was useful to both sides in many ways; it saves on development costs and it makes it easy for people to work in both worlds. With my Red Hat on, I think I'm allowed to say that internally we often talk about this being desirable - having consistency between how X is done in Fedora and how it's done for RHEL - and it obviously has benefits to Fedora too (it means we don't have to find the resources to do that same work at Fedora level).
However, situations like this make me wonder if we might have an issue with keeping shared infra/tooling where it's desirable. It seems like this is a decision/conversation that's been happening within RH, about what makes sense for RH in terms of RHEL development. AFAIK this is the first time it's been formally talked about in a Fedora context, and the messaging is "RH has already decided to stop using Bugzilla for RHEL after 9". In other words, RH has decided on its own to move away from something that is part of the shared RH/Fedora "heritage way of doing things".
I'm not saying that's wrong, but as I said it does make me wonder whether, if both sides do find shared tooling/approaches beneficial, we might want to approach this kind of potential change differently in future. Otherwise it does seem like we could sort of gradually drift apart, with no explicit intention to do so, and lose the benefits of shared tooling and process. Unless the ultimate outcome of this is "Fedora adopts issues.redhat.com for bug tracking" - which would be a possibility, but doesn't seem like a certainty - the result will be that we go from having a shared bug tracker, with the benefits of shared maintenance and being able to easily clone or reference bugs between Fedora and RHEL, to each maintaining our own bug tracker and not having those benefits.
Of course, there would be sensitivities in developing such a process - it could look a lot like Red Hat telling Fedora how to do stuff, which I think isn't exactly the relationship we want to have. But at the same time I'm not sure "Red Hat or Fedora just deciding unilaterally to stop using this thing they'd previously both used" is always the best choice either.
No, certainly not. I think it would have been nice if the tooling discussion happened before RH decided to drop Bugzilla so that we can all use a common tooling. However, I also know that in a business sometimes reaching such a consensus is everything but easy. It would have been nice if someone at least tried though.
Cheers,
Dan
On Mon, Mar 14, 2022 at 2:59 PM Dan Čermák dan.cermak@cgc-instruments.com wrote:
Hi Adam,
Adam Williamson adamwill@fedoraproject.org writes:
snip
That could obviously have pretty significant consequences for Fedora. Bugzilla isn't only an issue tracker for Fedora; we run some significant processes through it, notably the Change process, the blocker/FE bug process, and the prioritized bug process. In A World Without Bugzilla all of those would need adapting (and their documentation updating). There's fairly tight integration between Bodhi and Bugzilla, which would need to be redesigned. Those are just things I can think of off the top of my head. There are also a couple of decades worth of internet links to Fedora issues on RH Bugzilla, of course.
I guess the two big choices for Fedora if RH said "we're not maintaining Bugzilla any more" would be 1) take over maintaining Bugzilla or 2) switch to something else. 1) would probably be the path of least resistance, I guess.
Short term it is the path of the least resistance, but at least what I've heard from $dayjob, maintaining a Bugzilla instance is no easy task, as they are often customized (via non-upstream patches) and this all needs to be maintained. I cannot speak for our infra team, but I really don't know if they'd like yet another huge service, because this effectively means they'd have to take over maintenance of bugzilla.redhat.com...
This does also kinda lead to a larger question for me, trying to wear both Red Hat and Fedora hats at the same time[0]. I wonder if we're kind of lacking a...mechanism, for want of a better word, to handle the *generic* case here. Let's rewind to Ye Olde Days, when "the Fedora project" first started. At that point Fedora and Red Hat shared a lot of tooling and infrastructure, and this was useful to both sides in many ways; it saves on development costs and it makes it easy for people to work in both worlds. With my Red Hat on, I think I'm allowed to say that internally we often talk about this being desirable - having consistency between how X is done in Fedora and how it's done for RHEL - and it obviously has benefits to Fedora too (it means we don't have to find the resources to do that same work at Fedora level).
However, situations like this make me wonder if we might have an issue with keeping shared infra/tooling where it's desirable. It seems like this is a decision/conversation that's been happening within RH, about what makes sense for RH in terms of RHEL development. AFAIK this is the first time it's been formally talked about in a Fedora context, and the messaging is "RH has already decided to stop using Bugzilla for RHEL after 9". In other words, RH has decided on its own to move away from something that is part of the shared RH/Fedora "heritage way of doing things".
I'm not saying that's wrong, but as I said it does make me wonder whether, if both sides do find shared tooling/approaches beneficial, we might want to approach this kind of potential change differently in future. Otherwise it does seem like we could sort of gradually drift apart, with no explicit intention to do so, and lose the benefits of shared tooling and process. Unless the ultimate outcome of this is "Fedora adopts issues.redhat.com for bug tracking" - which would be a possibility, but doesn't seem like a certainty - the result will be that we go from having a shared bug tracker, with the benefits of shared maintenance and being able to easily clone or reference bugs between Fedora and RHEL, to each maintaining our own bug tracker and not having those benefits.
Of course, there would be sensitivities in developing such a process - it could look a lot like Red Hat telling Fedora how to do stuff, which I think isn't exactly the relationship we want to have. But at the same time I'm not sure "Red Hat or Fedora just deciding unilaterally to stop using this thing they'd previously both used" is always the best choice either.
No, certainly not. I think it would have been nice if the tooling discussion happened before RH decided to drop Bugzilla so that we can all use a common tooling. However, I also know that in a business
RHEL is choosing not to use Bugzilla for future versions of RHEL. I need to be clear in wording there, because Red Hat is a company, RHEL is one of its products, and we're only talking about newer versions of that product. I am not aware of any plans for Red Hat to drop Bugzilla. I am aware of plans for newer versions of RHEL to no longer use Bugzilla.
sometimes reaching such a consensus is everything but easy. It would have been nice if someone at least tried though.
Tried what, to be precise? If you mean try to find common tooling between Fedora and RHEL, well we have off and on for years. Several things work. Many didn't.
If you mean try to use bugzilla, we've been trying for the last 5 years internally to make it work in conjunction with issues.redhat.com. It's not working and it's time to consolidate to a single tool. That decision has no direct bearing on Fedora though.
If you mean try having a conversation with the community before something goes into effect... that's what this thread is. Depending on how you count, at least a year in advance if not 3.
If you meant something else, I've missed it.
josh
On Tue, Mar 15, 2022 at 07:17:16AM -0400, Josh Boyer wrote:
If you mean try having a conversation with the community before something goes into effect... that's what this thread is. Depending on how you count, at least a year in advance if not 3.
I was going to reply last week, but last week kind of turned out to be One Of Those Weeks. Here's what I'm thinking: we should plan to migrate to something else in the next three years. There are two reasons for this, and only one of them is that Red Hat is moving away. The second is that Bugzilla isn't a great tool for what we need anyway.
On the first part: yes, there's a long, slow sunset, but I think realistically we're going to see business-side attention drop significantly, and we'll have correspondingly worse and worse service. Right now, there are basically only two people keeping it all going, which is heroic. I don't think it's too much pulling-back-the-corporate-curtain to guess that if one or both get tired of that and decide to go start a yak farm, Red Hat won't prioritize hiring backfill dedicated in the same way. We could ask CPE to keep it going, but... there's so much more I'd like to ask CPE. (And non-CPE volunteers? There's so much that's more interesting!)
So to the second part: Bugzilla isn't what we really need anyway.
I'm not saying Bugzilla has been terrible — it's served us well, in fact. And I have personal fondness for it that, which I do not for, say, the wiki. :) Buzilla is it's deeply integrated in a lot of our processes, and we've got a lot of automations and so on. It's important. But... there's a lot that could be better. I think specifically:
1. It doesn't serve as a single place to track everything. We've got stuff scattered around Bugzilla; issue trackers in Pagure GitLab, and GitHub; pull requests in all of those places; and more. Not to mention upstreams (and inconsistent approaches to tracking upstream issues). I'm not arguing that we need ALL things in one place, but it's important that Bugzilla isn't that now anyway.
2. Bugzilla a terrible experience for end users. Usually it's the _Wrong Place_. When a user has a problem, that may or may not be caused by a software defect. This is a whole topic of its own, but in short, it's really better for most users to report problems at Ask Fedora, and then possibly after triage a bug should be filed and tracked somewhere.
Many of our users _are_ advanced and recognize real defects and file good reports, but this leads to even more frustration, because our response is inconsistent. Maybe that report should actually be upstream. Maybe it's in a dependency package that's really only loosely maintained. For whatever reason, a lot of perfectly good reports end up closed EOL, which is never a good outcome.
3. We're inconsistent with PRs vs issues, which is confusing and makes more duplication. I have seen cases where a packager complained that someone filed a PR that they never noticed, saying "this should be a bug so I'll see it", while others close bugs with "please send a PR". We could make stronger statements about policies, but as long as these two things exist in separate places, that tension will keep coming back.
Plus plenty of minor things: Our workflow still is shoehorned around a bunch of RH-centric stuff (lots of fields, flags, and statuses that we don't really use or want). It's not great for the review workflow, or for some of the other things we've twisted it to. A component-centric approach makes it hard to track larger issues. The SCM workflow is ... not good.
And I'm sure there's more. But I'm not really here to _complain_ about Bugzilla. It's just actually not filling our needs. I therefore think that Jira / issues.redhat.com wouldn't either — although it's got a ton of features on top, it's fundamentally the same kind of thing.
We need define exactly what we do need, and figure out how to get _that_, in a sustainable way going forward. And fortunately, we DO have a few years, so for once we could do this _before_ it's a crisis.
I think we should create a project to figure this out. In fact, I think it should be a Fedora Objective. But, it also shouldn't be a completely blue-sky initiative — we should avoid trying to develop a new gigantic piece of software that we own. (See past results on that, *sigh*.)
We do have some pieces already in place that should be part of the foundation (or at least other metaphorical bricks in the construction):
1. Ask Fedora is the place for users to report and discuss problems. This is our first-line of support, and it's where we can do triage. Then, software defects may or may not be tracked relating to those conversations.
2. Package-specific issues should be next to the PRs. Let's enable issue tracking on src.fedoraproject.org (with whatever gitforge we end up with that on). I think this makes sense for initial package review too, although that would need some workflow changes.
3. Bodhi and the CI results system and all of that. This is update-centric, but does also have a lot of issue-tracker-like characteristcs in the "karma" system, and it is inherently close to our release process. Maybe some of the process that currently goes through Bugzilla could move here.
4. The Fedora account system and Fedora message bus, plus the packager dashboard, the (under significant update!) Fedora notifications system, and so on. We have a nice underlying infrastructure for tying things together, and we can do more integrations. (Look at how Copr uses Discourse, for example. Or the Blocker Bugs app and its connections with Pagure and, um, Bugzilla.)
Obviously that's not nearly sufficient. But we should look at all of our needs around tracking issues, plans, projects, blockers, and defects; consider different packaging cases with various relationships with upstreams; and imagine ideal workflows for users, packagers, project managers, spin and edition teams, QA, and all the other stakeholders. The last time we did this was in 2013, so... it's a good 10-years-later exercise! (https://fedoraproject.org/wiki/Infrastructure_Fedora_bug_tracker)
Then, once we have a good shared understanding of what we want, start fitting pieces like the above into that picture, define the gaps, and then find exactly what we need to fill them.
On Tue, 2022-03-22 at 11:55 -0400, Matthew Miller wrote:
On the first part: yes, there's a long, slow sunset, but I think realistically we're going to see business-side attention drop significantly, and we'll have correspondingly worse and worse service. Right now, there are basically only two people keeping it all going, which is heroic. I don't think it's too much pulling-back-the-corporate-curtain to guess that if one or both get tired of that and decide to go start a yak farm, Red Hat won't prioritize hiring backfill dedicated in the same way.
There is one thing I worry about here - and maybe we need to take this up internally within Red Hat, but whatever - I strongly believe that *at a minimum*, going to https://bugzilla.redhat.com/show_bug.cgi?id=NNN should show you the report, comments, and attachments for bug NNN. *Forever*. RH Bugzilla is effectively a huge, 20-year long knowledge base at this point. Probably hundreds of thousands of places on the internet link to RH bugs. Taking that offline would be effectively an act of vandalism.
It may not need to keep actually working as a dynamic web app if we do retire BZ - we could reduce it to a server that just serves up static snapshots of the content - but I'd really hate to see the content lost.
On Tue, Mar 22, 2022 at 08:06:49PM -0700, Adam Williamson wrote:
There is one thing I worry about here - and maybe we need to take this up internally within Red Hat, but whatever - I strongly believe that *at a minimum*, going to https://bugzilla.redhat.com/show_bug.cgi?id=NNN should show you the report, comments, and attachments for bug NNN. *Forever*. RH Bugzilla is effectively a huge, 20-year long knowledge base at this point. Probably hundreds of thousands of places on the internet link to RH bugs. Taking that offline would be effectively an act of vandalism.
It may not need to keep actually working as a dynamic web app if we do retire BZ - we could reduce it to a server that just serves up static snapshots of the content - but I'd really hate to see the content lost.
Yeah, agreed. There's a lot of useful history!
For what it's worth, Wayback Machine links like https://web.archive.org/web/20120719062752/https://bugzilla.redhat.com/show_... seem to generally work. I wonder if we could officially "donate" it all to them.
On Tue, Mar 22, 2022 at 08:06:49PM -0700, Adam Williamson wrote:
On Tue, 2022-03-22 at 11:55 -0400, Matthew Miller wrote:
On the first part: yes, there's a long, slow sunset, but I think realistically we're going to see business-side attention drop significantly, and we'll have correspondingly worse and worse service. Right now, there are basically only two people keeping it all going, which is heroic. I don't think it's too much pulling-back-the-corporate-curtain to guess that if one or both get tired of that and decide to go start a yak farm, Red Hat won't prioritize hiring backfill dedicated in the same way.
There is one thing I worry about here - and maybe we need to take this up internally within Red Hat, but whatever - I strongly believe that *at a minimum*, going to https://bugzilla.redhat.com/show_bug.cgi?id=NNN should show you the report, comments, and attachments for bug NNN. *Forever*. RH Bugzilla is effectively a huge, 20-year long knowledge base at this point. Probably hundreds of thousands of places on the internet link to RH bugs. Taking that offline would be effectively an act of vandalism.
It would be an act of massive self-harm to Red Hat too, not merely community projects using it.
As a RHEL package maintainer, I frequently consult old bugs, from links we have embedded in patches / changelogs / mailing lists / commits. It is not unknown for me to look at bugs *10-15* years old to understand the history of why something was done the way it is.
IOW, even if RHEL changes to use Jira for all future bug tracking tomorrow, the need to reference bugzilla tickets could remain for literally decades to come. Jira has been mirroring some bugzilla content for a while, but that's by no means a complete archive of relevant history, and still leaves the problem if figuring out the matching link.
It may not need to keep actually working as a dynamic web app if we do retire BZ - we could reduce it to a server that just serves up static snapshots of the content - but I'd really hate to see the content lost.
With regards, Daniel
On 3/22/22 11:55, Matthew Miller wrote:
On Tue, Mar 15, 2022 at 07:17:16AM -0400, Josh Boyer wrote:
If you mean try having a conversation with the community before something goes into effect... that's what this thread is. Depending on how you count, at least a year in advance if not 3.
I was going to reply last week, but last week kind of turned out to be One Of Those Weeks. Here's what I'm thinking: we should plan to migrate to something else in the next three years. There are two reasons for this, and only one of them is that Red Hat is moving away. The second is that Bugzilla isn't a great tool for what we need anyway.
On the first part: yes, there's a long, slow sunset, but I think realistically we're going to see business-side attention drop significantly, and we'll have correspondingly worse and worse service. Right now, there are basically only two people keeping it all going, which is heroic. I don't think it's too much pulling-back-the-corporate-curtain to guess that if one or both get tired of that and decide to go start a yak farm, Red Hat won't prioritize hiring backfill dedicated in the same way. We could ask CPE to keep it going, but... there's so much more I'd like to ask CPE. (And non-CPE volunteers? There's so much that's more interesting!)
So to the second part: Bugzilla isn't what we really need anyway.
I'm not saying Bugzilla has been terrible — it's served us well, in fact. And I have personal fondness for it that, which I do not for, say, the wiki. :) Buzilla is it's deeply integrated in a lot of our processes, and we've got a lot of automations and so on. It's important. But... there's a lot that could be better. I think specifically:
It doesn't serve as a single place to track everything. We've got stuff scattered around Bugzilla; issue trackers in Pagure GitLab, and GitHub; pull requests in all of those places; and more. Not to mention upstreams (and inconsistent approaches to tracking upstream issues). I'm not arguing that we need ALL things in one place, but it's important that Bugzilla isn't that now anyway.
Bugzilla a terrible experience for end users. Usually it's the _Wrong Place_. When a user has a problem, that may or may not be caused by a software defect. This is a whole topic of its own, but in short, it's really better for most users to report problems at Ask Fedora, and then possibly after triage a bug should be filed and tracked somewhere.
Many of our users _are_ advanced and recognize real defects and file good reports, but this leads to even more frustration, because our response is inconsistent. Maybe that report should actually be upstream. Maybe it's in a dependency package that's really only loosely maintained. For whatever reason, a lot of perfectly good reports end up closed EOL, which is never a good outcome.
We're inconsistent with PRs vs issues, which is confusing and makes more duplication. I have seen cases where a packager complained that someone filed a PR that they never noticed, saying "this should be a bug so I'll see it", while others close bugs with "please send a PR". We could make stronger statements about policies, but as long as these two things exist in separate places, that tension will keep coming back.
Plus plenty of minor things: Our workflow still is shoehorned around a bunch of RH-centric stuff (lots of fields, flags, and statuses that we don't really use or want). It's not great for the review workflow, or for some of the other things we've twisted it to. A component-centric approach makes it hard to track larger issues. The SCM workflow is ... not good.
And I'm sure there's more. But I'm not really here to _complain_ about Bugzilla. It's just actually not filling our needs. I therefore think that Jira / issues.redhat.com wouldn't either — although it's got a ton of features on top, it's fundamentally the same kind of thing.
We need define exactly what we do need, and figure out how to get _that_, in a sustainable way going forward. And fortunately, we DO have a few years, so for once we could do this _before_ it's a crisis.
I think we should create a project to figure this out. In fact, I think it should be a Fedora Objective. But, it also shouldn't be a completely blue-sky initiative — we should avoid trying to develop a new gigantic piece of software that we own. (See past results on that, *sigh*.)
We do have some pieces already in place that should be part of the foundation (or at least other metaphorical bricks in the construction):
Ask Fedora is the place for users to report and discuss problems. This is our first-line of support, and it's where we can do triage. Then, software defects may or may not be tracked relating to those conversations.
Package-specific issues should be next to the PRs. Let's enable issue tracking on src.fedoraproject.org (with whatever gitforge we end up with that on). I think this makes sense for initial package review too, although that would need some workflow changes.
Bodhi and the CI results system and all of that. This is update-centric, but does also have a lot of issue-tracker-like characteristcs in the "karma" system, and it is inherently close to our release process. Maybe some of the process that currently goes through Bugzilla could move here.
The Fedora account system and Fedora message bus, plus the packager dashboard, the (under significant update!) Fedora notifications system, and so on. We have a nice underlying infrastructure for tying things together, and we can do more integrations. (Look at how Copr uses Discourse, for example. Or the Blocker Bugs app and its connections with Pagure and, um, Bugzilla.)
Obviously that's not nearly sufficient. But we should look at all of our needs around tracking issues, plans, projects, blockers, and defects; consider different packaging cases with various relationships with upstreams; and imagine ideal workflows for users, packagers, project managers, spin and edition teams, QA, and all the other stakeholders. The last time we did this was in 2013, so... it's a good 10-years-later exercise! (https://fedoraproject.org/wiki/Infrastructure_Fedora_bug_tracker)
Then, once we have a good shared understanding of what we want, start fitting pieces like the above into that picture, define the gaps, and then find exactly what we need to fill them.
It’s worth noting that Bugzilla is used for far more than just RHEL, CentOS Stream, and Fedora. Bugzilla is also used for LVM2, libguestfs, and many other community projects. So I don’t think it will be going away any time soon. That said, it does need to be modernized significantly; for instance, upstream Bugzilla supports Markdown IIRC.
On Mon, Mar 14, 2022 at 11:12 AM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2022-03-07 at 12:44 -0500, Josh Boyer wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
What does this mean for Fedora and CentOS? This discussion is in part to figure that out. Based on some very brief analysis, the following should hold:
- RHEL customers should continue to file support cases through the Red
Hat Customer portal, which will remain consistent regardless of the backend tooling used.
- There is no imminent retirement of the Red Hat Bugzilla instance
being planned at this time. RHEL 7, 8, and 9 will continue to use bugzilla in some form and RHEL 9 has a very long lifecycle.
- Fedora Linux and EPEL have their own Bugzilla product families and
are not directly impacted in their own workflows by the choice to use only issues.redhat.com for RHEL. - There will be impacts on existing documentation that provide guidance on requesting things from RHEL in various places like EPEL. We will be happy to help adjust these.
- CentOS Stream contribution and bug reporting workflows will be
adjusted to use issues.redhat.com instead of bugzilla in the relevant places. This should apply to all versions of CentOS Stream for a unified contributor workflow. This will happen gradually as we discover the best workflow to use.
If there are other impacts that you can think of, please raise them on this thread. We’d like to ensure we’re covering as much as possible as this rolls out.
So if I'm understanding this correctly, the ultimate consequence here is "Red Hat Bugzilla might go away, or stop being maintained, at whatever point it's no longer needed for RHEL 9", right?
I have no idea, to be honest. There's a lot of assumption in that statement and it certainly could be an outcome, but I'm not aware of any plans towards that directly.
That could obviously have pretty significant consequences for Fedora. Bugzilla isn't only an issue tracker for Fedora; we run some significant processes through it, notably the Change process, the blocker/FE bug process, and the prioritized bug process. In A World Without Bugzilla all of those would need adapting (and their documentation updating). There's fairly tight integration between Bodhi and Bugzilla, which would need to be redesigned. Those are just things I can think of off the top of my head. There are also a couple of decades worth of internet links to Fedora issues on RH Bugzilla, of course.
Those all sound like the things I'm familiar with.
I guess the two big choices for Fedora if RH said "we're not maintaining Bugzilla any more" would be 1) take over maintaining Bugzilla or 2) switch to something else. 1) would probably be the path of least resistance, I guess.
This does also kinda lead to a larger question for me, trying to wear both Red Hat and Fedora hats at the same time[0]. I wonder if we're kind of lacking a...mechanism, for want of a better word, to handle the *generic* case here. Let's rewind to Ye Olde Days, when "the Fedora project" first started. At that point Fedora and Red Hat shared a lot of tooling and infrastructure, and this was useful to both sides in many ways; it saves on development costs and it makes it easy for people to work in both worlds. With my Red Hat on, I think I'm allowed to say that internally we often talk about this being desirable - having consistency between how X is done in Fedora and how it's done for RHEL - and it obviously has benefits to Fedora too (it means we don't have to find the resources to do that same work at Fedora level).
Fedora and RHEL process and tooling has deviated significantly over the years. So much so that by the time I joined the RHEL team, it was already very different. That was almost 5 years ago, which means the individual decisions that led to it were even earlier. I don't really want to revisit those decisions because I wasn't around and can't speak to why they were made, but the connection between Fedora and RHEL via bugzilla is minimal at best.
The commonality that brings the most shared benefit is the activities of our communities, the quality and rigor they bring into Fedora, and the sources we share. Tooling and process are orthogonal to those. Important, because they certainly lend themselves to aiding that quality and rigor, but orthogonal.
However, situations like this make me wonder if we might have an issue with keeping shared infra/tooling where it's desirable. It seems like this is a decision/conversation that's been happening within RH, about what makes sense for RH in terms of RHEL development. AFAIK this is the first time it's been formally talked about in a Fedora context, and the messaging is "RH has already decided to stop using Bugzilla for RHEL after 9". In other words, RH has decided on its own to move away from something that is part of the shared RH/Fedora "heritage way of doing things".
I don't think this is the first instance of those kinds of decisions. It is perhaps one of the few instances where we've been up-front about it well in advance.
I'm not saying that's wrong, but as I said it does make me wonder whether, if both sides do find shared tooling/approaches beneficial, we
I think we do in some cases, and not in others. I don't find bugzilla to be one of those. Even the clone function is questionable.
might want to approach this kind of potential change differently in future. Otherwise it does seem like we could sort of gradually drift apart, with no explicit intention to do so, and lose the benefits of shared tooling and process. Unless the ultimate outcome of this is
From my perspective, we minimally share tooling. Process is common in concepts, but RHEL process is far more additive and heavy-weight than anything Fedora does.
"Fedora adopts issues.redhat.com for bug tracking" - which would be a possibility, but doesn't seem like a certainty - the result will be
To be clear, it is not an explicit goal for Fedora to adopt issues.redhat.com. That's up to the Fedora project, much as adoption of gitlab was.
that we go from having a shared bug tracker, with the benefits of shared maintenance and being able to easily clone or reference bugs between Fedora and RHEL, to each maintaining our own bug tracker and not having those benefits.
Personally, I don't think we're taking advantage of perceived benefits at all. They are effectively separate.
Of course, there would be sensitivities in developing such a process - it could look a lot like Red Hat telling Fedora how to do stuff, which I think isn't exactly the relationship we want to have. But at the same time I'm not sure "Red Hat or Fedora just deciding unilaterally to stop using this thing they'd previously both used" is always the best choice either.
The Fedora decision is up to Fedora. RHEL is driven by product constraints and will evaluate based on those needs. Where we can align, we should. For a recent example of a decision to align more, look at the CKI project. Both Fedora and RHEL benefit greatly from maintaining the kernel in the CKI manner, and this is the closest the Fedora and RHEL kernels have ever been.
josh
Hi,
I am trying to request a package for RHEL 9, but I cannot find RHEL under Projects at issues.redhat.com. What is the correct project for RHEL 9 ?
Thanks
--- Lee
On Mon, Mar 7, 2022 at 11:14 PM Josh Boyer jwboyer@redhat.com wrote:
Hi Fedora, CentOS, and EPEL Communities!
As part of our continued 3 year major Red Hat Enterprise Linux release cadence, RHEL 9 development is starting to wrap up with the spring 2022 release coming soon. That means planning for the next release will start in earnest in the very near future. As some of you may know, Red Hat has been using both Bugzilla and Jira via issues.redhat.com for RHEL development for several years. Our intention is to move to using issues.redhat.com only for the major RHEL releases after RHEL 9.
What does this mean for Fedora and CentOS? This discussion is in part to figure that out. Based on some very brief analysis, the following should hold:
- RHEL customers should continue to file support cases through the Red
Hat Customer portal, which will remain consistent regardless of the backend tooling used.
- There is no imminent retirement of the Red Hat Bugzilla instance
being planned at this time. RHEL 7, 8, and 9 will continue to use bugzilla in some form and RHEL 9 has a very long lifecycle.
- Fedora Linux and EPEL have their own Bugzilla product families and
are not directly impacted in their own workflows by the choice to use only issues.redhat.com for RHEL. - There will be impacts on existing documentation that provide guidance on requesting things from RHEL in various places like EPEL. We will be happy to help adjust these.
- CentOS Stream contribution and bug reporting workflows will be
adjusted to use issues.redhat.com instead of bugzilla in the relevant places. This should apply to all versions of CentOS Stream for a unified contributor workflow. This will happen gradually as we discover the best workflow to use.
If there are other impacts that you can think of, please raise them on this thread. We’d like to ensure we’re covering as much as possible as this rolls out.
josh _______________________________________________ epel-devel mailing list -- epel-devel@lists.fedoraproject.org To unsubscribe send an email to epel-devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
devel@lists.stg.fedoraproject.org
-
Adam Williamson -
Ben Cotton -
Brian Stinson -
Colin Walters -
Dan Čermák -
Daniel P. Berrangé -
Davide Cavalca -
Demi Marie Obenour -
Dominik 'Rathann' Mierzejewski -
Florian Weimer -
Josh Boyer -
Justin Forbes -
Ken Dreyer -
Matthew Miller -
Miro Hrončok -
mkolman＠redhat.com -
Peter Robinson -
Simo Sorce -
Thomas Stephen Lee