On Thu, 2018-03-22 at 04:40 +0000, Peter Robinson wrote:

... to improve performance on low resource devices ...

Hi, I only recently, like three weeks ago, had been asked to update Fedora on a very old Dell notebook. I installed there Fedora 20 years ago and when I've been asked to get there a newer Firefox, because that ancient doesn't work for sites like Facebook (do not ask, it's not my machine), then I decided to update to the recent Fedora 27. Then the user gets all the security fixes and so on, right? Everyone wins. After successful 'distro-sync', which was quite impressive on its own, the gdm didn't boot quicker than in like 5 minutes, when I was lucky. Booting to GNOME took even longer. The rescue mode booted quicker, but that's not the production. There were caught some ABRT reports about kernel crash, but it was just about: "hey, kernel crashed, but you've bad luck, because from this crash kernel developers won't get anything, thus I'll not let you report it". What are such ABRT reports good for? The ancient Fedora 20 didn't have this issue, it just worked. Long story short, I spent half night and quite few hours the next day trying fresh install of Fedora 27, 26, 25, but none worked that well as Fedora 20, thus I resulted in getting the ancient Fedora 20 from the archives, install it and bring Firefox straight from Mozilla. The task accomplished, but...

Maybe there's some regression there, kernel should not crash, but I do not have that machine anymore, I've it only borrowed to get there new- enough browser and then I returned it back. It's not helpful for the community, I know, but it's the user experience I've got with one low resource device (I'm pretty sure it had the CPU and HDD higher than the minimum, the RAM at least 1GB too, but I'm not sure right now; did Fedora 20 have the same requirements? It works there reasonably well...).

Maybe you can compare such ancient Fedora, like the Fedora 20, with the current Fedora on those low resource devices to see (and feel) the difference. With classic "rotated disks", not SSD, that might be important too. Bye, Milan

On Thu, 2018-03-22 at 09:52 +0100, Matej Marusak wrote:

Now you knew that there is something wrong with kernel and you have a good starting point for you later investigation.

Hi, yes, I agree, informing is good, but this one is not useless only for developers, it's useless also for the user. It's a starting point, but also the final point. I do not know how to debug kernel crashes and I'd bet the "Details" button was disabled (it's a long time, I can be wrong), not talking that getting information from the ABRT GUI (like the most important is the backtrace of the crash for me) is a problem, as it's not available there at all or it cannot be copied from the GUI (not everything in the GUI of ABRT can be copied out).

My options, as I see them:

- ABRT tells me kernel crashed during boot, but doesn't let me file

the bug

As a regular user: - file a bug in probably Red Hat bugzilla and hope kernel folks monitor it, thus they can help/guide me and so on - read about the crash the next boot, until...

As a developer: - oops, kernel crashed. I've no idea how to debug kernel crashes during boot, thus let's file a bug and continue as a regular user above.

So I end up with a bug and with a hope. What's the main difference between the bug filled by ABRT and the one filled by me? As long as the anonymous bug reports are disabled (it's the FAF for, right, bugs cannot be filled anonymously), then the bug I fill manually is even less useful than the one which ABRT would fill. Why? ABRT may include basic information about the issue, like what crashed, when/where it crashed, what versions of packages had been involved and all that information which the developer would ask for anyway, but which I'd not always be able to give him/her easily (see above, cannot copy everything from the ABRT GUI). Thus ABRT-filled bug would be more accurate and contain much more information than what the user would do. Frankly, my initial bug text would probably look like:

ABRT tells me about a crash of kernel after boot, but ABRT claims the report would be useless for kernel developers, thus it doesn't let me to report it. Could you guide me what to do now to investigate the crash, please? It's there after each restart of the machine.

I even do not know what heuristic is done to make ABRT think that the human kernel developer would consider the report useless.

(Or still, if you are experienced enough you can take closer look into the problem details)

This is kernel/boot related. Most users are not experienced enough, I'm pretty sure of it.

Bye, Milan

2018-03-22 1:44 GMT-06:00 Milan Crha mcrha@redhat.com:

On Thu, 2018-03-22 at 04:40 +0000, Peter Robinson wrote:

...

... to improve performance on low resource devices ...

    Hi,

I only recently, like three weeks ago, had been asked to update Fedora on a very old Dell notebook. I installed there Fedora 20 years ago and when I've been asked to get there a newer Firefox, because that ancient doesn't work for sites like Facebook (do not ask, it's not my machine), then I decided to update to the recent Fedora 27. Then the user gets all the security fixes and so on, right? Everyone wins. After successful 'distro-sync', which was quite impressive on its own, the gdm didn't boot quicker than in like 5 minutes, when I was lucky. Booting to GNOME took even longer. The rescue mode booted quicker, but that's not the production. There were caught some ABRT reports about kernel crash, but it was just about: "hey, kernel crashed, but you've bad luck, because from this crash kernel developers won't get anything, thus I'll not let you report it". What are such ABRT reports good for? The ancient Fedora 20 didn't have this issue, it just worked. Long story short, I spent half night and quite few hours the next day trying fresh install of Fedora 27, 26, 25, but none worked that well as Fedora 20, thus I resulted in getting the ancient Fedora 20 from the archives, install it and bring Firefox straight from Mozilla. The task accomplished, but...

I a computer without a SSD Plasma and Gnome Shell do not work well, so in the current state os those desktop enviroments the page also should recomend to use a SSD disck, is not use a alternative desktop like Cinnamon or a more clasical option like Mate.

On Thu, Mar 22, 2018 at 07:07:12AM -0400, Bastien Nocera wrote:

----- Original Message -----

...

On Wed, Mar 21, 2018 at 10:30:09PM -0600, Chris Murphy wrote:

...

For Fedora 27 it says this:

Fedora Workstation to a laptop or desktop computer that has at least 1 GHz processor, 1 GB RAM, and 10 GB space available

https://getfedora.org/en/workstation/download/

When I set a VM to 1G it's always unusable. Sometimes it's PackageKit that crashes first, which causes systemd-coredump to spin up and run out of memory next, and then oom killers start killing off various processes. Sometimes the oom killing just starts happening without a prior crash. In any case I never get a desktop.

With 1.5G RAM I usually get a desktop but then if anything crashes, and it seems like PackageKit is crashing often when under memory pressure (?), it kicks off systemd-coredump which makes the memory problem worse

systemd-coredump should probably short-circuit the coredump in oom conditions. I'm not sure what the exact conditions should be, maybe if the amount of free RAM is lower then the core size?

Why does the kernel even make that information available? I regularly have my machine made unusable when the kernel decides to kill the largest memory users (usually the mail client, or a browser tab), and it OOM kills the largest memory users, meaning the ones that will take the longest to dump to disk.

We already have records of the application being killed through OOM in the journal, we should be spared even trying to pass the core to user-space.

That depends. Sometimes you want the core on oom to diagnose a memory leak. There are two different cases: - a single program went awry and used all memory and was killed. After it is gone, the machine will be usable again. It's potentially useful to dump the core. - the machine is generally starved for resources, and even after the biggest consumer of memory has been killed, there still is not enough. Dump of the core puts further strain on the system and should be avoided.

The question is how to distinguish those two cases.

Zbyszek

On Thu, Mar 22, 2018 at 09:45:39AM -0400, Bastien Nocera wrote:

I can't think of a single time of when I'd want to wait for the core dumping to finish (which can make the machine unusable for tens of minutes) to debug a memory usage problem.

I can think of many times — but none of them are individual desktop use cases. This could be very valuable in server, HPC, or even enterprise desktop contexts.

----- Original Message -----

On Thu, Mar 22, 2018 at 09:45:39AM -0400, Bastien Nocera wrote:

...

I can't think of a single time of when I'd want to wait for the core dumping to finish (which can make the machine unusable for tens of minutes) to debug a memory usage problem.

I can think of many times — but none of them are individual desktop use cases. This could be very valuable in server, HPC, or even enterprise desktop contexts.

Dumping the core of the application that might not even be the one responsible for eating all the RAM? Please expand.

Enterprise desktop users are the same as desktop users, except that they can't afford to take a 30-minute break when their machine is thrashing. If you know any different, I'm all for an expanded explanation here.

----- Original Message -----

On Wed, Mar 28, 2018 at 04:53:20AM -0400, Bastien Nocera wrote:

...

Dumping the core of the application that might not even be the one responsible for eating all the RAM? Please expand.

Enterprise desktop users are the same as desktop users, except that they can't afford to take a 30-minute break when their machine is thrashing. If you know any different, I'm all for an expanded explanation here.

If I'm having a persistent but difficult to trace problem across my set of X hundred or thousand managed workstations, I want all the information I can get. The downtime would be no different than if a hard drive failed or something — if it's half an hour, sorry. If it's more than that, that's why we have a pool of loaner machines.

How is that a different behaviour from setting, say, a system through which you can dump kernel cores, or any other root causing done by an admin?

I said:

I'd say making this a kernel tunable that's disabled by default would cover the vast majority of uses, development machines included.

And that's still what we would want. Especially given that the core that's dumped in the end might not even be that of the culprit.

On Do, 22.03.18 08:46, Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) wrote:

On Wed, Mar 21, 2018 at 10:30:09PM -0600, Chris Murphy wrote:

...

For Fedora 27 it says this:

Fedora Workstation to a laptop or desktop computer that has at least 1 GHz processor, 1 GB RAM, and 10 GB space available

https://getfedora.org/en/workstation/download/

When I set a VM to 1G it's always unusable. Sometimes it's PackageKit that crashes first, which causes systemd-coredump to spin up and run out of memory next, and then oom killers start killing off various processes. Sometimes the oom killing just starts happening without a prior crash. In any case I never get a desktop.

With 1.5G RAM I usually get a desktop but then if anything crashes, and it seems like PackageKit is crashing often when under memory pressure (?), it kicks off systemd-coredump which makes the memory problem worse

systemd-coredump should probably short-circuit the coredump in oom conditions. I'm not sure what the exact conditions should be, maybe if the amount of free RAM is lower then the core size?

Hmm, to be frank, I wasn't aware the kernel would trigger coredump handling in case of OOM. This feels a bit like making things better and then making things worse again ;-)

I wonder if there's any way to detect the reason the kernel calls the coredump handlers, i.e. that it can tell us whether OOM triggered it or anything else.

Lennart

On Thu, Mar 22, 2018 at 3:15 AM, Richard Hughes hughsient@gmail.com wrote:

On 22 March 2018 at 04:30, Chris Murphy lists@colorremedies.com wrote:

...

With 1.5G RAM I usually get a desktop but then if anything crashes, and it seems like PackageKit is crashing often when under memory pressure (?)

PackageKit-the-supervisor uses very little memory indeed by design, but PackageKit does also load libdnf which loads libsolv etc which is probably where the problem lays. Do you have a massif run handy on a small machine? I can do one myself if not, although I fear the problem is going to be libsolv rather than much we can fix in "Fedora".

Never used massif, don't know how to use it, but I'll do whatever I'm told on any machine I have, real or VM.

What I'm noticing is if the VM is set to 1G RAM I pretty much always see packagekitd crash. And it's too memory limited of an environment for a live to then get any data out of it, even serial console with 'virsh console' hangs up for tens of minutes at a time. If I go to 1.5G RAM the packagekitd crash I'd say is rare, but the sample size so far is maybe 10 boots. I don't know if what you need is a crashed packagekitd to find out why it's crashing, or something else.

On 03/23/2018 04:06 AM, Chris Murphy wrote:

On Thu, Mar 22, 2018 at 3:15 AM, Richard Hughes hughsient@gmail.com wrote:

...

On 22 March 2018 at 04:30, Chris Murphy lists@colorremedies.com wrote:

...

With 1.5G RAM I usually get a desktop but then if anything crashes, and it seems like PackageKit is crashing often when under memory pressure (?)

PackageKit-the-supervisor uses very little memory indeed by design, but PackageKit does also load libdnf which loads libsolv etc which is probably where the problem lays. Do you have a massif run handy on a small machine? I can do one myself if not, although I fear the problem is going to be libsolv rather than much we can fix in "Fedora".

OK I got it to crash in a way that I was able to collect some minimal info...

https://bugzilla.redhat.com/show_bug.cgi?id=1559684

There's a link to the coredump there. For whatever reason the retrace server will not accept it with abrt. https://github.com/abrt/retrace-server/issues/180

Thanks Chris. The coredump seems to be just various things starting to go wrong in packagekitd when we start running out of resources and I don't think there's much we can do there, HOWEVER:

The main issue seems to be that we're starting gnome-software on the live media, which in turn is starting packagekitd and asking it to download metadata. This is 100 MB download which eats up precious RAM as the file system overlay on the live media is backed by memory.

This all adds up with gnome-software using RAM, packagekitd using RAM, and downloaded metadata using RAM.

Working on fixing it now to avoid all that on live media.

On Sat, 24 Mar 2018, 01:23 Adam Williamson, adamwill@fedoraproject.org wrote:

On Fri, 2018-03-23 at 11:37 +0100, Kalev Lember wrote:

...

On 03/23/2018 04:06 AM, Chris Murphy wrote:

...

On Thu, Mar 22, 2018 at 3:15 AM, Richard Hughes hughsient@gmail.com

wrote:

...

...

...

On 22 March 2018 at 04:30, Chris Murphy lists@colorremedies.com

wrote:

...

...

...

...

With 1.5G RAM I usually get a desktop but then if anything crashes, and it seems like PackageKit is crashing often when under memory pressure (?)

PackageKit-the-supervisor uses very little memory indeed by design, but PackageKit does also load libdnf which loads libsolv etc which is probably where the problem lays. Do you have a massif run handy on a small machine? I can do one myself if not, although I fear the

problem

...

...

...

is going to be libsolv rather than much we can fix in "Fedora".

OK I got it to crash in a way that I was able to collect some minimal

info...

...

...

https://bugzilla.redhat.com/show_bug.cgi?id=1559684

There's a link to the coredump there. For whatever reason the retrace server will not accept it with abrt. https://github.com/abrt/retrace-server/issues/180

Thanks Chris. The coredump seems to be just various things starting to go wrong in packagekitd when we start running out of resources and I don't think there's much we can do there, HOWEVER:

The main issue seems to be that we're starting gnome-software on the live media, which in turn is starting packagekitd and asking it to download metadata. This is 100 MB download which eats up precious RAM as the file system overlay on the live media is backed by memory.

This all adds up with gnome-software using RAM, packagekitd using RAM, and downloaded metadata using RAM.

Working on fixing it now to avoid all that on live media.

Can you or Chris propose a bug for this as a Beta freeze exception and give it an accurate description (like 'GNOME Software runs unnecessarily on live images' or so?) It'd be good to get a fix for this into Beta I guess.

I do still suspect anaconda is using more memory than before as well, but will have to look into that, and it may not affect liveinst as the live installer doesn't use the dnf payload.

I agree here, I suspect, although I've not investigated, that the initrd contents has bloated due some random increase in deps causing memory used by the ram disk to increase.