We've been planning on doing one of these forever, but never seeming to get around to it.
The kernel gets a lot of bugs (possibly more than any other package), and as such, we've got nearly a thousand bugs open right now, and just three people working on it full-time.
The problem we've faced with triage efforts in the past is that for many bugs, the person doing the triage really needs to have at least some kernel knowledge to know what information to ask the reporter.
However, there are some basic tasks that would help us out a lot, like making sure bugs are assigned to the right people etc. There's a first pass at some 'how to' ideas at https://fedoraproject.org/wiki/KernelTriage
We'll be doing this in #fedora-kernel next Monday (22nd) I expect that the wiki page will continue to evolve as we start working on this, and perhaps this can even become a regular thing.
Dave
On Thu, 2011-08-18 at 12:32 -0400, Dave Jones wrote:
We've been planning on doing one of these forever, but never seeming to get around to it.
The kernel gets a lot of bugs (possibly more than any other package), and as such, we've got nearly a thousand bugs open right now, and just three people working on it full-time.
The problem we've faced with triage efforts in the past is that for many bugs, the person doing the triage really needs to have at least some kernel knowledge to know what information to ask the reporter.
However, there are some basic tasks that would help us out a lot, like making sure bugs are assigned to the right people etc. There's a first pass at some 'how to' ideas at https://fedoraproject.org/wiki/KernelTriage
We'll be doing this in #fedora-kernel next Monday (22nd) I expect that the wiki page will continue to evolve as we start working on this, and perhaps this can even become a regular thing.
apologies for not helping with this; turned out to be bad timing for me, I didn't even get to decompress from Alpha last week before I had to spend three days at Linuxcon, and spent most of the weekend in a dazed heap on the couch...sorry again!
On Mon, Aug 22, 2011 at 03:33:24PM -0700, Adam Williamson wrote:
We'll be doing this in #fedora-kernel next Monday (22nd) I expect that the wiki page will continue to evolve as we start working on this, and perhaps this can even become a regular thing.
apologies for not helping with this; turned out to be bad timing for me, I didn't even get to decompress from Alpha last week before I had to spend three days at Linuxcon, and spent most of the weekend in a dazed heap on the couch...sorry again!
You didn't miss much. Turned out to be a non-event. Maybe we'll give it another shot in a few weeks.
Dave
On Mon, 2011-08-22 at 22:24 -0400, Dave Jones wrote:
On Mon, Aug 22, 2011 at 03:33:24PM -0700, Adam Williamson wrote:
We'll be doing this in #fedora-kernel next Monday (22nd) I expect that the wiki page will continue to evolve as we start working on this, and perhaps this can even become a regular thing.
apologies for not helping with this; turned out to be bad timing for me, I didn't even get to decompress from Alpha last week before I had to spend three days at Linuxcon, and spent most of the weekend in a dazed heap on the couch...sorry again!
You didn't miss much. Turned out to be a non-event. Maybe we'll give it another shot in a few weeks.
well, I was planning to help spread the word a bit, which might have resulted in a few more people showing up and complaining =)
On Mon, Aug 22, 2011 at 10:24:45PM -0400, Dave Jones wrote:
On Mon, Aug 22, 2011 at 03:33:24PM -0700, Adam Williamson wrote:
We'll be doing this in #fedora-kernel next Monday (22nd) I expect that the wiki page will continue to evolve as we start working on this, and perhaps this can even become a regular thing.
apologies for not helping with this; turned out to be bad timing for me, I didn't even get to decompress from Alpha last week before I had to spend three days at Linuxcon, and spent most of the weekend in a dazed heap on the couch...sorry again!
You didn't miss much. Turned out to be a non-event. Maybe we'll give it another shot in a few weeks.
So we're thinking of trying this again this thursday with a focus on 16, (but triage work on older releases is welcomed too).
Dave
On Mon, Oct 03, 2011 at 15:36:27 -0400, Dave Jones davej@redhat.com wrote:
So we're thinking of trying this again this thursday with a focus on 16, (but triage work on older releases is welcomed too).
I have a bug (36242 at the unavailable kernel bugzilla and 684424 in Fedora) where sound being played using my motherboard sound chip and high I/O (it seems like network traffic more so than disk) results in a hang. I haven't been able to get a crash dump or traceback once the hang occurs. I'd be interested in getting this bug looked at, but I think I need some suggestions for how to get a traceback, or there isn't going to be enough info to track this down. I tried some kernel boot parameters to get watchdog timeouts, but that didn't work. Maybe I need to rebuild the kernel with a different config to really enable that feature though?
On Wed, Oct 05, 2011 at 07:33:22AM -0500, Bruno Wolff III wrote:
On Mon, Oct 03, 2011 at 15:36:27 -0400, Dave Jones davej@redhat.com wrote:
So we're thinking of trying this again this thursday with a focus on 16, (but triage work on older releases is welcomed too).
I have a bug (36242 at the unavailable kernel bugzilla and 684424 in Fedora) where sound being played using my motherboard sound chip and high I/O (it seems like network traffic more so than disk) results in a hang. I haven't been able to get a crash dump or traceback once the hang occurs. I'd be interested in getting this bug looked at, but I think I need some suggestions for how to get a traceback, or there isn't going to be enough info to track this down. I tried some kernel boot parameters to get watchdog timeouts, but that didn't work. Maybe I need to rebuild the kernel with a different config to really enable that feature though?
The kernel-debug rpm should have everything you'd need.
Hardware specific problems like this are a nightmare for us to diagnose. It might even come down to you needing to do a bisect to find the individual kernel change that caused the problem. (assuming you know a 'good' version to start from)
Dave
Once upon a time, Dave Jones davej@redhat.com said:
Hardware specific problems like this are a nightmare for us to diagnose. It might even come down to you needing to do a bisect to find the individual kernel change that caused the problem. (assuming you know a 'good' version to start from)
I know suspend is another "fun" area, but are there any good tools to figure out what is wrong when suspend/resume doesn't work right? I've got a problem system (RH BZ 548593) that I don't know what else I can do to try to fix.
On Wed, Oct 05, 2011 at 10:38:18 -0400, Dave Jones davej@redhat.com wrote:
The kernel-debug rpm should have everything you'd need.
But I can't get the system to crash to get a traceback. It just hangs. I tried using the sysrq commands and NMI timeouts and I haven't been able to get a dump or traceback.
Hardware specific problems like this are a nightmare for us to diagnose. It might even come down to you needing to do a bisect to find the individual kernel change that caused the problem. (assuming you know a 'good' version to start from)
The issue goes back quite a ways (probably it last worked correctly with 2.6.29.1-68.fc11.i686.PAE). I have a USB headset now so I have been less concerned about the issue. But several months ago there were some USB issues and I switched back to motherboard sound during that time and had a number of hangs, so I opened a new bug (since I am not 100% sure it is really the same bug as 496536).
Some details about the triage day we are holding tomorrow.
Where: #fedora-kernel on irc.freenode.net
When: October 6th 2011
What: The primary focus is going to be on getting things in the best shape possible for Fedora 16's release. However there are some useful things that can be done for all releases.
* Fedora 14 At this point in 14's lifecycle, many open bugs are not going to be fixed, due to the amount of work necessary to identify and backport individual fixes. Because of this, identifying open bugs that are still relevant on 15/16 is important, so we don't perpetually have to keep dragging these bugs forward every release.
* Fedora 15 F15 bugs that are likely to affect F16 are obviously of importance, so triage assistance on those bugs is useful. If a bug is confirmed to be still present in Fedora 16, add 'F16' to the whiteboard field.
* Fedora 16: With the release of the F16 beta, the primary focus will be to make sure the existing bugs are all properly assigned, and triaged.
* Rawhide: At the moment, Rawhide is essentially in-sync with F16. The main difference is that Rawhide kernels have debugging enabled by default, whereas F16 doesn't (except for the kernel-debug builds). Because F16 is the upcoming release, we're focusing there but bugs found in Rawhide will likely still occur in F16 as well. However, there are a number of 'perpetual' bugs that are stuck in rawhide (especially Feature requests and WIP items) and those should probably be avoided for now. When in doubt, focus on F16.
General info: * See https://fedoraproject.org/wiki/KernelTriage for further 'howto' info. * Of particular importance are bugs that will prevent installation, break networking, or cause system hangs. These bugs should be marked as blocking the f16blocker bug. To do this, you can use the 'f16blocker' alias or the actual bug number 713568. Confirm in IRC before adding them to the tracker bug. If you don't have the Bugzilla powers to add them to the tracker bug, ask in IRC and someone else will be able to do it for you. * Bugs that aren't serious enough to be blockers but might warrant special effort to fix might be added to the F16-accepted tracker (bug number 713566). Again, check in the channel before adding a bug to this tracker.
Useful links: All open f14 kernel bugs: http://bit.ly/nmkC8m All open f15 kernel bugs: http://bit.ly/pIpDdM All open f16 kernel bugs: http://bit.ly/ouhRBY
On Mon, 2011-10-03 at 15:36 -0400, Dave Jones wrote:
On Mon, Aug 22, 2011 at 10:24:45PM -0400, Dave Jones wrote:
On Mon, Aug 22, 2011 at 03:33:24PM -0700, Adam Williamson wrote:
We'll be doing this in #fedora-kernel next Monday (22nd) I expect that the wiki page will continue to evolve as we start working on this, and perhaps this can even become a regular thing.
apologies for not helping with this; turned out to be bad timing for me, I didn't even get to decompress from Alpha last week before I had to spend three days at Linuxcon, and spent most of the weekend in a dazed heap on the couch...sorry again!
You didn't miss much. Turned out to be a non-event. Maybe we'll give it another shot in a few weeks.
So we're thinking of trying this again this thursday with a focus on 16, (but triage work on older releases is welcomed too).
It's a bit late to note now, but Thursday isn't a great choice of day as it's the regular Test Day slot: tomorrow is printing test day, so people who are particularly inclined to join in with test day type events will have to pick one or the other...
On Thu, Oct 6, 2011 at 12:14 AM, Bruno Wolff III bruno@wolff.to wrote:
On Wed, Oct 05, 2011 at 10:38:18 -0400, Dave Jones davej@redhat.com wrote:
The kernel-debug rpm should have everything you'd need.
But I can't get the system to crash to get a traceback. It just hangs. I tried using the sysrq commands and NMI timeouts and I haven't been able to get a dump or traceback.
Maybe you can try "nmi_watchdog=panic"?
On Sat, Oct 08, 2011 at 18:30:44 +0800, Américo Wang xiyou.wangcong@gmail.com wrote:
On Thu, Oct 6, 2011 at 12:14 AM, Bruno Wolff III bruno@wolff.to wrote:
On Wed, Oct 05, 2011 at 10:38:18 -0400, Dave Jones davej@redhat.com wrote:
The kernel-debug rpm should have everything you'd need.
But I can't get the system to crash to get a traceback. It just hangs. I tried using the sysrq commands and NMI timeouts and I haven't been able to get a dump or traceback.
Maybe you can try "nmi_watchdog=panic"?
I am not sure if I tried that particular varient, but I did try several different watchdog related kernel settings. I can got back and test that.
devel@lists.stg.fedoraproject.org
-
Adam Williamson -
Bruno Wolff III -
Chris Adams -
Cong Wang -
Dave Jones