Hi,
Currently we have coredumpctl installed by default, but it doesn't work because core dumps are consumed by ABRT instead. I'd like to get coredumpctl working out-of-the-box. We can do this by changing our systemd presets [1] to disable abrt-ccpp.service.
ABRT will continue to function thanks to the ABRT developers' work on coredumpctl integration via abrt-vmcore.service. There is a somewhat- reduced feature set, but I've been running this for several months happily and I haven't noticed any issues; my crashes are being reported to FAF, and I can manually report to Bugzilla. See [2] for details.
This might be slightly controversial for two reasons. First, the big change here is that core dumps will appear in the magical and super- awesome coredumpctl tool in addition to ABRT, but would no longer be created in the cwd when ulimit is set appropriately (we would need to feature that in the release notes). This aligns us with upstream systemd behavior, but diverges from Debian and Ubuntu, which still create the old-style coredumps in the cwd. Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl. But as coredumpctl is very mature (nicer) and cross-distro so it gets many more contributors, I definitely prefer to instruct people to use coredumpctl instead for C/C++ crashes.
We could totally do this change for F24 as well as it should be just a one-line change, but in the spirit of beta freeze I figure it's probably best left for F25.
Michael
[1] https://pagure.io/fedora-release/blob/master/f/90-default.preset [2] https://github.com/abrt/abrt/issues/1108
On Tue, Apr 12, 2016 at 03:49:09PM -0500, Michael Catanzaro wrote:
Hi,
Currently we have coredumpctl installed by default, but it doesn't work because core dumps are consumed by ABRT instead. I'd like to get coredumpctl working out-of-the-box. We can do this by changing our systemd presets [1] to disable abrt-ccpp.service.
ABRT will continue to function thanks to the ABRT developers' work on coredumpctl integration via abrt-vmcore.service. There is a somewhat- reduced feature set, but I've been running this for several months happily and I haven't noticed any issues; my crashes are being reported to FAF, and I can manually report to Bugzilla. See [2] for details.
+1 to integrating abrt to use coredump as a "backend".
This might be slightly controversial for two reasons. First, the big change here is that core dumps will appear in the magical and super- awesome coredumpctl tool in addition to ABRT, but would no longer be created in the cwd when ulimit is set appropriately (we would need to feature that in the release notes).
I don't think that's a problem in practice. For most people coredumps are annoying, and use up disk space unless you remember to clean them up. And coredumps stored by systemd-coredump are still easily accessible (although they might be compressed, but this can be easily turned off it desired).
Having the coredumps go through systemd-coredump gives us a robust and convenient way to gather coredumps, also during boot.
This aligns us with upstream systemd behavior, but diverges from Debian and Ubuntu, which still create the old-style coredumps in the cwd. Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl. But as coredumpctl is very mature (nicer) and cross-distro so it gets many more contributors, I definitely prefer to instruct people to use coredumpctl instead for C/C++ crashes.
We could totally do this change for F24 as well as it should be just a one-line change, but in the spirit of beta freeze I figure it's probably best left for F25.
Zbyszek
On Tue, Apr 12, 2016 at 10:49 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl.
Any particular reason for that preference? As coredumpctl gets more and more widespread, having the same tool as other distros should make Fedora more approachable which I think is a good thing?
Rui
----- Original Message -----
On Tue, Apr 12, 2016 at 10:49 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl.
Any particular reason for that preference? As coredumpctl gets more and more widespread, having the same tool as other distros should make Fedora more approachable which I think is a good thing?
The upstream systemd developers I talked to weren't interested in adapting coredumpctl to be usable for use cases other than for developers.
It doesn't claim to be usable for end-users reporting bugs, and the original developers have no interest in making changes that would allow that.
On Wed, 13.04.16 08:44, Bastien Nocera (bnocera@redhat.com) wrote:
On Tue, Apr 12, 2016 at 10:49 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl.
Any particular reason for that preference? As coredumpctl gets more and more widespread, having the same tool as other distros should make Fedora more approachable which I think is a good thing?
The upstream systemd developers I talked to weren't interested in adapting coredumpctl to be usable for use cases other than for developers.
What precisely do you need?
It doesn't claim to be usable for end-users reporting bugs, and the original developers have no interest in making changes that would allow that.
I indeed think coredumpctl should probably be something for more professional users. But I'd be curious what you are missing in coredumpctl.
Lennart
----- Original Message -----
On Wed, 13.04.16 08:44, Bastien Nocera (bnocera@redhat.com) wrote:
On Tue, Apr 12, 2016 at 10:49 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl.
Any particular reason for that preference? As coredumpctl gets more and more widespread, having the same tool as other distros should make Fedora more approachable which I think is a good thing?
The upstream systemd developers I talked to weren't interested in adapting coredumpctl to be usable for use cases other than for developers.
What precisely do you need?
I asked, through private mail, and after we discussed this face-to-face: " Could you please send me a piece of code that would: - monitor the journal for new coredumps - gather information about the crashed binary (abrt collects things like envvars and more that allows us to determine which application or component crashed, whether it's packaged or not, etc.) - gather information about kernel oopses/lockdep warnings - store additional metadata about the report, if journald can support that (whether it was reported and where for example) "
It doesn't claim to be usable for end-users reporting bugs, and the original developers have no interest in making changes that would allow that.
I indeed think coredumpctl should probably be something for more professional users. But I'd be curious what you are missing in coredumpctl.
Some help implementing one of: https://bugs.freedesktop.org/show_bug.cgi?id=82507 So that we can eventually move that down a level from ABRT.
To be fair, I misrepresented the discussion, I was actually told it would be too much work, and that you didn't have the manpower.
Bastien, do I get it right you want to ditch ABRT and teach systemd to detect Python, Ruby, Java, Node.js exceptions?
On 10/11/2016 03:16 PM, Bastien Nocera wrote:
----- Original Message -----
On Wed, 13.04.16 08:44, Bastien Nocera (bnocera@redhat.com) wrote:
On Tue, Apr 12, 2016 at 10:49 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl.
Any particular reason for that preference? As coredumpctl gets more and more widespread, having the same tool as other distros should make Fedora more approachable which I think is a good thing?
The upstream systemd developers I talked to weren't interested in adapting coredumpctl to be usable for use cases other than for developers.
What precisely do you need?
I asked, through private mail, and after we discussed this face-to-face: " Could you please send me a piece of code that would:
- monitor the journal for new coredumps
- gather information about the crashed binary (abrt collects things like envvars and more that allows us to determine which application or component crashed, whether it's packaged or not, etc.)
- gather information about kernel oopses/lockdep warnings
- store additional metadata about the report, if journald can support that (whether it was reported and where for example)
"
It doesn't claim to be usable for end-users reporting bugs, and the original developers have no interest in making changes that would allow that.
I indeed think coredumpctl should probably be something for more professional users. But I'd be curious what you are missing in coredumpctl.
Some help implementing one of: https://bugs.freedesktop.org/show_bug.cgi?id=82507 So that we can eventually move that down a level from ABRT.
To be fair, I misrepresented the discussion, I was actually told it would be too much work, and that you didn't have the manpower.
On Fri, 2016-10-14 at 09:06 +0200, Jakub Filak wrote:
Bastien, do I get it right you want to ditch ABRT and teach systemd to detect Python, Ruby, Java, Node.js exceptions?
That's my understanding... but I'm not sure why, and I doubt the systemd developers would want to do that. coredumpctl is a tool for handling core dumps, nothing more or less. It's for Python interpreter crashes, not Python exceptions....
Michael
On Wed, 2016-04-13 at 08:44 -0400, Bastien Nocera wrote:
The upstream systemd developers I talked to weren't interested in adapting coredumpctl to be usable for use cases other than for developers.
It doesn't claim to be usable for end-users reporting bugs, and the original developers have no interest in making changes that would allow that.
coredumpctl is not an end user bug reporting tool. We don't need it to be -- we have ABRT that. coredumpctl is a tool for developers. It's awesome and upstream and cross-distro. Now that we can have coredumpctl and ABRT at the same time, enabling it seems like an easy choice to me.
Michael
On 04/13/2016 02:12 PM, Rui Tiago Cação Matos wrote:
On Tue, Apr 12, 2016 at 10:49 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl.
Any particular reason for that preference? As coredumpctl gets more and more widespread, having the same tool as other distros should make Fedora more approachable which I think is a good thing?
Rui
User experience is the reason why I prefer improving abrt CLI. coredumpctl handles only coredump files, abrt-cli (or rather 'abrt' - a brand new tool that should replace abrt-cli) handles more problem types - coredumps, Kernel oopses, Python exceptions, Java exceptions, Ruby exceptions, ...
abrt can do pretty much everything as coredumpctl can do (BTW it can run 'gdb'). Besides, it can install missing debuginfo packages, generate backtrace using a remote server, report the problem to Bugzilla, upload it , etc., because it can re-use ABRT infrastructure.
I don't say abrt is perfect and it desperately needs more love.
ABRT's purpose has changed. It used to be a set of tools for reporting problems caused by system packages, but, now, we are looking on how to make ABRT more useful for developers. Perhaps, you do not see much progress on this goal, but that is only because we have very limited resources.
Regards, Jakub ABRT
On Tue, Apr 12, 2016 at 9:49 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
Hi,
Currently we have coredumpctl installed by default, but it doesn't work because core dumps are consumed by ABRT instead. I'd like to get coredumpctl working out-of-the-box. We can do this by changing our systemd presets [1] to disable abrt-ccpp.service.
ABRT will continue to function thanks to the ABRT developers' work on coredumpctl integration via abrt-vmcore.service. There is a somewhat- reduced feature set, but I've been running this for several months happily and I haven't noticed any issues; my crashes are being reported to FAF, and I can manually report to Bugzilla. See [2] for details.
This might be slightly controversial for two reasons. First, the big change here is that core dumps will appear in the magical and super- awesome coredumpctl tool in addition to ABRT, but would no longer be created in the cwd when ulimit is set appropriately (we would need to feature that in the release notes). This aligns us with upstream systemd behavior, but diverges from Debian and Ubuntu, which still create the old-style coredumps in the cwd. Second, my understanding is that the ABRT developers prefer to improve abrt-cli and tell people to use that rather than coredumpctl. But as coredumpctl is very mature (nicer) and cross-distro so it gets many more contributors, I definitely prefer to instruct people to use coredumpctl instead for C/C++ crashes.
We could totally do this change for F24 as well as it should be just a one-line change, but in the spirit of beta freeze I figure it's probably best left for F25.
What is the impact on other non Workstation parts of Fedora, like Server, Cloud and the Spins?
Peter
On Thu, 2016-05-12 at 23:01 +0100, Peter Robinson wrote:
What is the impact on other non Workstation parts of Fedora, like Server, Cloud and the Spins?
I think it's just a matter of changing our systemd presets. The impact could be none, if we make the change in the Workstation presets, or it could be identical, if we make the change in the default presets. I think the Server product would probably like to have coredumpctl as well. Cloud, I'm not so sure. But it's really up to those WGs.
Michael
On Thu, May 12, 2016 at 11:12 PM, Michael Catanzaro mcatanzaro@gnome.org wrote:
On Thu, 2016-05-12 at 23:01 +0100, Peter Robinson wrote:
What is the impact on other non Workstation parts of Fedora, like Server, Cloud and the Spins?
I think it's just a matter of changing our systemd presets. The impact could be none, if we make the change in the Workstation presets, or it could be identical, if we make the change in the default presets. I think the Server product would probably like to have coredumpctl as well. Cloud, I'm not so sure. But it's really up to those WGs.
Yet further up in the thread there are people saying that it's not suitable for outside of developers. I think if it's going to be wider ranging that just impact on workstation it needs to be discussed on a wider channel than just here.
Peter
On Thu, 2016-05-12 at 23:26 +0100, Peter Robinson wrote:
Yet further up in the thread there are people saying that it's not suitable for outside of developers.
It's not suitable for non-developers in the sense that no command line program is ever suitable for nontechnical users, but it's not somehow inappropriate to have enabled on all systems. It's a tool that makes debugging and bug reporting easier, but it's not a distro bug reporting tool. ABRT remains our distro bug reporting tool.
Basically: if you want to work with core dumps, you probably want to use coredumpctl (though understandably the ABRT folks prefer abrt-cli). If not, you don't care. Status quo is that we have coredumpctl installed and broken by default.
Michael
On 05/13/2016 01:05 AM, Michael Catanzaro wrote:
On Thu, 2016-05-12 at 23:26 +0100, Peter Robinson wrote:
Yet further up in the thread there are people saying that it's not suitable for outside of developers.
It's not suitable for non-developers in the sense that no command line program is ever suitable for nontechnical users, but it's not somehow inappropriate to have enabled on all systems. It's a tool that makes debugging and bug reporting easier, but it's not a distro bug reporting tool. ABRT remains our distro bug reporting tool.
Basically: if you want to work with core dumps, you probably want to use coredumpctl (though understandably the ABRT folks prefer abrt-cli). If not, you don't care. Status quo is that we have coredumpctl installed and broken by default.
Michael
Well, if I need to analyze core dump files during development I prefer no intermediate tool - no ABRT, no systemd-coredump. I just set 'ulimit -c unlimited' and run gdb on the file generated in CWD. Actually, that's not true, I run my programs in gdb, so I don't need to analyze local core dump files most of the time. I usually need to analyze core dump files generated on foreign systems.
One benefit of ABRT for some developers was that it could create the old way core dump files in CWD but starting with systemd-229 we had to disable this feature by default. Because systemd changed the default RLIMIT_CORE from 0 to unlimited ('ulimit -c').
I do prefer abrt-cli (or abrt - which can run gdb too) when I need to work with errors/crashes/problems of any source - Python, Java, C/C++, Kernel oops, Ruby, etc. I can analyze them and eventually report them.
systedm-coredump makes ABRT's job harder. Currently, ABRT fails to get core dump files from systemd-journal because systemd-coredump switched from xz compression to lz4 compression. My bad, I should have been watching systemd development.
Best regards, Jakub ABRT folk
On Sat, 2016-05-14 at 07:09 +0200, Jakub Filak wrote:
systedm-coredump makes ABRT's job harder. Currently, ABRT fails to get core dump files from systemd-journal because systemd-coredump switched from xz compression to lz4 compression. My bad, I should have been watching systemd development.
I think you should view this as a positive, not as a burden: abrt gets features (such as lz-compressed coredumps) for free by relying systemd.
On Sat, 2016-05-14 at 07:09 +0200, Jakub Filak wrote:
Well, if I need to analyze core dump files during development I prefer no intermediate tool - no ABRT, no systemd-coredump. I just set 'ulimit -c unlimited' and run gdb on the file generated in CWD. Actually, that's not true, I run my programs in gdb, so I don't need to analyze local core dump files most of the time. I usually need to analyze core dump files generated on foreign systems.
I prefer coredumpctl, because:
* It's always listening. You don't need to set ulimit, run a program from a terminal, and then try reproducing a tough to hit crash. You can just use your computer as normal, and wait for it to crash. * It works for all processes on your system, including daemons which are otherwise tough to get coredumps for. * Simply 'coredumpctl gdb' to open the most recent crash in gdb. * No need to remember to delete the coredumps; core dumps are stored in one place and removed automatically based on reasonable disk space limits. (ABRT does this too when not changing ulimit.) * It's upstream. Users coming to Fedora from Arch and other distros already know how to use it, and will not be frustrated when it doesn't work as expected. (Currently we have it installed by default, but totally broken out of the box.)
Grabbing coredumps from cwd feels like stone age to me. Having to set ulimit is really inconvenient and frustrating when using an old system that doesn't have systemd.
I used to have ABRT disabled in favor of coredumpctl, but now that ABRT and coredumpctl work nicely together, I don't see any inherent conflict anymore. So I still want to proceed with this for F25.
I do prefer abrt-cli (or abrt - which can run gdb too) when I need to work with errors/crashes/problems of any source - Python, Java, C/C++, Kernel oops, Ruby, etc. I can analyze them and eventually report them.
No doubt ABRT is better for automatic bug reports and for handling non- C/C++ crashes. Fortunately, we can have both!
systedm-coredump makes ABRT's job harder. Currently, ABRT fails to get core dump files from systemd-journal because systemd-coredump switched from xz compression to lz4 compression. My bad, I should have been watching systemd development.
OK, this is a problem. :( You're planning to fix it?
Michael
On Mon, May 16, 2016 at 09:55:55AM -0500, Michael Catanzaro wrote:
On Sat, 2016-05-14 at 07:09 +0200, Jakub Filak wrote:
systedm-coredump makes ABRT's job harder. Currently, ABRT fails to get core dump files from systemd-journal because systemd-coredump switched from xz compression to lz4 compression. My bad, I should have been watching systemd development.
OK, this is a problem. :( You're planning to fix it?
The compression algorithm changed in two places: - in the journal itself - in core dump files in /var/lib/systemd/coredump/ The first should be invisible to abrt, because sd_journal_get_data() uncompresses the data, so the compression method is completely hidden. The second is indeed visible, but it should be a matter of simply switching xzcat to lz4cat in the consumers. The switch is a big improvement for users because xz was just too slow for large coredumps.
Zbyszek
On 05/16/2016 04:55 PM, Michael Catanzaro wrote:
systedm-coredump makes ABRT's job harder. Currently, ABRT fails to get core dump files from systemd-journal because systemd-coredump switched from xz compression to lz4 compression. My bad, I should have been watching systemd development.
OK, this is a problem. :( You're planning to fix it?
Of course, I fix bugs causing ABRT to stop working ASAP. Updated ABRT packages will land in Fedora repositories today or tomorrow.
Jakub
On Tue, 2016-04-12 at 15:49 -0500, Michael Catanzaro wrote:
Currently we have coredumpctl installed by default, but it doesn't work because core dumps are consumed by ABRT instead. I'd like to get coredumpctl working out-of-the-box. We can do this by changing our systemd presets [1] to disable abrt-ccpp.service.
A quick update on this. The above is true for F23. Unfortunately coredumpctl in F24 is broken in two different ways:
https://bugzilla.redhat.com/show_bug.cgi?id=1331926 https://bugzilla.redhat.com/show_bug.cgi?id=1341829
So we'll have to wait until these are resolved.
Michael
One of those links was wrong; correct links:
https://bugzilla.redhat.com/show_bug.cgi?id=1317927 https://bugzilla.redhat.com/show_bug.cgi?id=1341829
On Sat, Jun 04, 2016 at 08:49:22AM -0500, Michael Catanzaro wrote:
One of those links was wrong; correct links:
https://bugzilla.redhat.com/show_bug.cgi?id=1317927 https://bugzilla.redhat.com/show_bug.cgi?id=1341829
It's also broken in rawhide (appears to be a kernel issue): https://bugzilla.redhat.com/show_bug.cgi?id=1342671
Oh, bugs everywhere :(
Zbyszek
desktop@lists.fedoraproject.org
-
Bastien Nocera -
Jakub Filak -
Lennart Poettering -
Matthias Clasen -
Michael Catanzaro -
Peter Robinson -
Rui Tiago Cação Matos -
Zbigniew Jędrzejewski-Szmek