If netconsole is built as a module, you can't use it to capture output before the module gets loaded. Rebuilding Fedora srpms (to get Fedora patches) takes pretty long, which makes testing slow.
I have something particular I am testing now (probably a nouveau related crash) that crashes for netconsole gets loaded (though I tried to add it to the initramfs to get it loaded fairly early). I am not sure if it will be early enough even if it is built in. I won't know until sometime tomorrow. But if it does get loaded early enough to handle video card related stuff, it seems like it be worth having built into debug kernels.
On Sat, Jun 28, 2014 at 13:37:44 -0500, Bruno Wolff III bruno@wolff.to wrote:
I have something particular I am testing now (probably a nouveau related crash) that crashes for netconsole gets loaded (though I tried to add it to the initramfs to get it loaded fairly early). I am not sure if it will be early enough even if it is built in. I won't know until sometime tomorrow. But if it does get loaded early enough to handle video card related stuff, it seems like it be worth having built into debug kernels.
The crash appears appears to be happening too early to send netconsole output even when netconsole is builtin to the kernel.
Also I may have been wrong about it not being built in, as I am not finding a netconsole.ko in /usr/lib/modules. The config file for Fedora normally has: config-3.15.0-1.fc21.i686+PAE:CONFIG_NETCONSOLE=m config-3.15.0-1.fc21.i686+PAE:CONFIG_NETCONSOLE_DYNAMIC=y
But maybe enabling the dynamic feature forces it to be built in now?
It looks like I'll need to try something else to get the beginning of the traceback from the crashes.
On Sun, Jun 29, 2014 at 08:34:49AM -0500, Bruno Wolff III wrote:
On Sat, Jun 28, 2014 at 13:37:44 -0500, Bruno Wolff III bruno@wolff.to wrote:
I have something particular I am testing now (probably a nouveau related crash) that crashes for netconsole gets loaded (though I tried to add it to the initramfs to get it loaded fairly early). I am not sure if it will be early enough even if it is built in. I won't know until sometime tomorrow. But if it does get loaded early enough to handle video card related stuff, it seems like it be worth having built into debug kernels.
The crash appears appears to be happening too early to send netconsole output even when netconsole is builtin to the kernel.
iirc, you also need to build-in your NIC driver for this to work this early
Dave
On Sun, Jun 29, 2014 at 17:29:36 -0400, Dave Jones davej@redhat.com wrote:
On Sun, Jun 29, 2014 at 08:34:49AM -0500, Bruno Wolff III wrote:
On Sat, Jun 28, 2014 at 13:37:44 -0500, Bruno Wolff III bruno@wolff.to wrote:
I have something particular I am testing now (probably a nouveau related crash) that crashes for netconsole gets loaded (though I tried to add it to the initramfs to get it loaded fairly early). I am not sure if it will be early enough even if it is built in. I won't know until sometime tomorrow. But if it does get loaded early enough to handle video card related stuff, it seems like it be worth having built into debug kernels.
The crash appears appears to be happening too early to send netconsole output even when netconsole is builtin to the kernel.
iirc, you also need to build-in your NIC driver for this to work this early
Thanks, I'll try doing that. It doesn't seem I have the right set of connectors to make a serial connection between two of my computers and my current thought had been to slow the connection down to 110 and take pictures of the screen of an old terminal I can connect to.
On Sat, Jun 28, 2014 at 01:37:44PM -0500, Bruno Wolff III wrote:
If netconsole is built as a module, you can't use it to capture output before the module gets loaded. Rebuilding Fedora srpms (to get Fedora patches) takes pretty long, which makes testing slow.
I have something particular I am testing now (probably a nouveau related crash) that crashes for netconsole gets loaded (though I tried to add it to the initramfs to get it loaded fairly early). I am not sure if it will be early enough even if it is built in. I won't know until sometime tomorrow. But if it does get loaded early enough to handle video card related stuff, it seems like it be worth having built into debug kernels. _______________________________________________ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
Even building it all (netconsole+dependent network drivers) isn't typically enough, since you also need to bring the network interface up and assign an ip address to it, which happens fairly late in the boot cycle. If you have a crash that occurs so early that you're thinking about doing this, a RAC device with serial console support is usually a better bet.
That said, it would be nice if dracut had a module that allowed you to include netconsole so that you could set it up before you did a switch_root.
Neil
On Mon, Jun 30, 2014 at 09:25:01 -0400, Neil Horman nhorman@redhat.com wrote:
Even building it all (netconsole+dependent network drivers) isn't typically enough, since you also need to bring the network interface up and assign an ip address to it, which happens fairly late in the boot cycle. If you have a crash that occurs so early that you're thinking about doing this, a RAC device with serial console support is usually a better bet.
The documentation implies that you don't need an IP assigned before you can use netconsole, since that information can be provided with kernel parameters. I don't know whether or not it really works that way though.
That said, it would be nice if dracut had a module that allowed you to include netconsole so that you could set it up before you did a switch_root.
The confir for dracut allows for you to include kernel drivers in the initramfs. So you should be able to use netconsole before root pivet. You might need to worry about the interface name changing part way through the boot.
On Sun, Jun 29, 2014 at 08:34:49 -0500, Bruno Wolff III bruno@wolff.to wrote:
The crash appears appears to be happening too early to send netconsole output even when netconsole is builtin to the kernel.
I am going to do Dave's suggestion of also building in the network driver.
I am also just rebuilding the srpm on my machine. I am doing an arch specific scratch build to rebuild the rpms. That should cut down the time needed for tests by quite a lot.
Another note on this is that netconsole was getting built as a module, but I was confused by the switch to compressed modules. So while netconsole.ko was the name for older kernerls, netconsole.ko.xz is what I should have been looking for.
Hopefully I'll be able to capture the traceback when I get home.
On 28.06.2014 20:37, Bruno Wolff III wrote:
If netconsole is built as a module, you can't use it to capture output before the module gets loaded. Rebuilding Fedora srpms (to get Fedora patches) takes pretty long, which makes testing slow.
I have something particular I am testing now (probably a nouveau related crash) that crashes for netconsole gets loaded (though I tried to add it to the initramfs to get it loaded fairly early). I am not sure if it will be early enough even if it is built in. I won't know until sometime tomorrow. But if it does get loaded early enough to handle video card related stuff, it seems like it be worth having built into debug kernels.
This is how it can be done;
1. /etc/modprobe.d/blacklist-foo.conf blacklist nouveau
2. # reboot
4. When the boot reaches any of the multi-user.target or graphical.target probably network is already set up so you just have to: - check whether [tgt-port] is opened. # modprobe [-v] netconsole netconsole=[src-port]@[src-ip]/[dev],[tgt-port]@<tgt-ip>/[tgt-macaddr] # modprobe [-v] nouveau
3. At the "tgt" with: # nc -l -u [tgt-port] after '4.' you'll see something like: [ monotonic timestamps] [drm] Initialized drm 1.1.0 20060810 [ monotonic timestamps] fb: switching to nouveaufb from VESA VGA [ monotonic timestamps] Console: switching to colour dummy device 80x25 [ monotonic timestamps] nouveau [ DEVICE][0000:02:00.0] BOOT0 : ... ...
There may be additional steps related to the specific set up, however it is more or less all.
poma
On Mon, Jun 30, 2014 at 13:29:52 -0500, Bruno Wolff III bruno@wolff.to wrote:
Hopefully I'll be able to capture the traceback when I get home.
No joy on getting the traceback via netconsole. Looks like I'll need to try the picture taking route for this problem.
On Tue, Jul 01, 2014 at 00:44:41 +0200, poma pomidorabelisima@gmail.com wrote:
This is how it can be done;
/etc/modprobe.d/blacklist-foo.conf blacklist nouveau
# reboot
When the boot reaches any of the multi-user.target or graphical.target probably network is already set up so you just have to:
- check whether [tgt-port] is opened.
# modprobe [-v] netconsole netconsole=[src-port]@[src-ip]/[dev],[tgt-port]@<tgt-ip>/[tgt-macaddr] # modprobe [-v] nouveau
At the "tgt" with: # nc -l -u [tgt-port] after '4.' you'll see something like: [ monotonic timestamps] [drm] Initialized drm 1.1.0 20060810 [ monotonic timestamps] fb: switching to nouveaufb from VESA VGA [ monotonic timestamps] Console: switching to colour dummy device 80x25 [ monotonic timestamps] nouveau [ DEVICE][0000:02:00.0] BOOT0 : ... ...
There may be additional steps related to the specific set up, however it is more or less all.
I'll try testing coming up with vesa and then switching to nouveau. If that works (well breaks), hopefully I'll get a useful traceback. If it doesn't work, I'll fallback to taking pictures.
kernel@lists.fedoraproject.org
-
Bruno Wolff III -
Dave Jones -
Neil Horman -
poma