dracut and grub2-mkconfig hang indefinitely in rawhide, in uninterruptable sleep - 20180128
Just compiled a custom kernel. When I tried to create a custom initramfs, the job seemed to create the file, but hung indefinitely in D status, with call_r. Trying to recreate the grub.cfg file by running grub2-mkconfig also went into uninterruptable sleep. When I ran this two weeks ago (20180114), both were working fine. Not sure what is at fault here. Anyone else seeing this?
On Sun, Jan 28, 2018 at 11:49 AM, stan stanl-fedorauser@vfemail.net wrote:
Just compiled a custom kernel. When I tried to create a custom initramfs, the job seemed to create the file, but hung indefinitely in D status, with call_r. Trying to recreate the grub.cfg file by running grub2-mkconfig also went into uninterruptable sleep. When I ran this two weeks ago (20180114), both were working fine. Not sure what is at fault here. Anyone else seeing this?
I did a bunch of kernel bisecting Friday and Saturday and haven't run into this. I'm not totally certain I had the same versions you're running, what I have now:
4.14.14-300.fc27.x86_64 dracut-046-92.git20180118.fc28.x86_64 grub2-tools-2.02-24.fc28.x86_64
Also I'm using enforcing=0 because I'm getting so many avc and audit related messages it's plugging up the journal files, which went over 500MB in like two days. And due to a new kernel bug in 4.15, hence the bisecting, I'm running kernel 4.14. I just did dracut -f and grub2-mkconfig with this combination and it works fine.
On Sun, 28 Jan 2018 16:33:13 -0700 Chris Murphy lists@colorremedies.com wrote:
Thanks for the response.
I did a bunch of kernel bisecting Friday and Saturday and haven't run into this. I'm not totally certain I had the same versions you're running, what I have now:
4.14.14-300.fc27.x86_64 dracut-046-92.git20180118.fc28.x86_64 grub2-tools-2.02-24.fc28.x86_64
Name : kernel Version : 4.15.0 Release : 0.rc9.git4.1.20180128.fc28 Architecture: x86_64 Install Date: Sun 28 Jan 2018 10:49:34 AM MST
Name : dracut Version : 046 Release : 92.git20180118.fc28 Architecture: x86_64 Install Date: Thu 25 Jan 2018 02:44:52 PM MST
Name : grub2-tools Epoch : 1 Version : 2.02 Release : 24.fc28 Architecture: x86_64 Install Date: Fri 26 Jan 2018 01:55:49 PM MST
The same except for the kernel version. But there were hundreds of updates in the last few days, so it could be a lot of things.
Also I'm using enforcing=0 because I'm getting so many avc and audit related messages it's plugging up the journal files, which went over 500MB in like two days. And due to a new kernel bug in 4.15, hence the bisecting, I'm running kernel 4.14. I just did dracut -f and grub2-mkconfig with this combination and it works fine.
I have selinux in enforcing mode, so that could be it. I'll try setting enforcing to 0 and booting into an earlier kernel.
On Sun, 28 Jan 2018 18:49:41 -0700 stan stanl-fedorauser@vfemail.net wrote:
On Sun, 28 Jan 2018 16:33:13 -0700 Chris Murphy lists@colorremedies.com wrote:
I did a bunch of kernel bisecting Friday and Saturday and haven't run into this. I'm not totally certain I had the same versions you're running, what I have now:
4.14.14-300.fc27.x86_64 dracut-046-92.git20180118.fc28.x86_64 grub2-tools-2.02-24.fc28.x86_64
Also I'm using enforcing=0 because I'm getting so many avc and audit related messages it's plugging up the journal files, which went over 500MB in like two days. And due to a new kernel bug in 4.15, hence the bisecting, I'm running kernel 4.14. I just did dracut -f and grub2-mkconfig with this combination and it works fine.
I have selinux in enforcing mode, so that could be it. I'll try setting enforcing to 0 and booting into an earlier kernel.
Mystery solved. It's an artifact of the other problem, the piping of command output to less automatically. I was redirecting the output of dracut into files. When I just ran it without the redirects, it hit several pauses while waiting for less. I just hit space to get rid of them when they occurred, and dracut completed normally. As did grub2-mkconfig when dracut wasn't in uninterruptable sleep.
Still no idea why commands in bash would be automatically routed to less the way systemd does it with journalctl, etc.
On Sun, 2018-01-28 at 11:49 -0700, stan wrote:
Just compiled a custom kernel. When I tried to create a custom initramfs, the job seemed to create the file, but hung indefinitely in D status, with call_r. Trying to recreate the grub.cfg file by running grub2-mkconfig also went into uninterruptable sleep. When I ran this two weeks ago (20180114), both were working fine. Not sure what is at fault here. Anyone else seeing this?
Hi Stan! One *possible* source of this might be a recent glibc change:
* Fri Jan 19 2018 Björn Esser besser82@fedoraproject.org - 2.26.9000-46 - Remove deprecated libcrypt, gets replaced by libxcrypt
you could try downgrading to a glibc from before that and see if it helps? Only a possibility, though. Thanks!
On Mon, 29 Jan 2018 01:34:38 +0100 Adam Williamson adamwill@fedoraproject.org wrote:
Thanks for the reply.
Hi Stan! One *possible* source of this might be a recent glibc change:
- Fri Jan 19 2018 Björn Esser besser82@fedoraproject.org -
2.26.9000-46
- Remove deprecated libcrypt, gets replaced by libxcrypt
you could try downgrading to a glibc from before that and see if it helps? Only a possibility, though. Thanks!
It fits the time frame.
Name : libxcrypt Version : 4.0.0 Release : 0.204.20180120git3436e7b.fc28 Architecture: x86_64 Install Date: Thu 25 Jan 2018 02:36:09 PM MST
Name : glibc Version : 2.26.9000 Release : 48.fc28 Architecture: x86_64 Install Date: Thu 25 Jan 2018 02:36:05 PM MST
The libxcrypt info says
... programs linked against libxcrypt will not work with glibc's libcrypt ...
I think that means an earlier glibc might cause problems with libxcrypt installed, since programs must be linked against it, though I'll still try it.
On Sun, 2018-01-28 at 18:57 -0700, stan wrote:
On Mon, 29 Jan 2018 01:34:38 +0100 Adam Williamson adamwill@fedoraproject.org wrote:
Thanks for the reply.
Hi Stan! One *possible* source of this might be a recent glibc change:
- Fri Jan 19 2018 Björn Esser besser82@fedoraproject.org -
2.26.9000-46
- Remove deprecated libcrypt, gets replaced by libxcrypt
you could try downgrading to a glibc from before that and see if it helps? Only a possibility, though. Thanks!
It fits the time frame.
Name : libxcrypt Version : 4.0.0 Release : 0.204.20180120git3436e7b.fc28 Architecture: x86_64 Install Date: Thu 25 Jan 2018 02:36:09 PM MST
Name : glibc Version : 2.26.9000 Release : 48.fc28 Architecture: x86_64 Install Date: Thu 25 Jan 2018 02:36:05 PM MST
The libxcrypt info says
... programs linked against libxcrypt will not work with glibc's libcrypt ...
I think that means an earlier glibc might cause problems with libxcrypt installed, since programs must be linked against it, though I'll still try it.
Yeah, indeed - I didn't mean it as a permanent solution, but as a debugging step (i.e. if it works with the old glibc, we know the bug's in glibc). Thanks!
On Sun, 28 Jan 2018 18:57:01 -0700 stan stanl-fedorauser@vfemail.net wrote:
On Mon, 29 Jan 2018 01:34:38 +0100 Adam Williamson adamwill@fedoraproject.org wrote:
Hi Stan! One *possible* source of this might be a recent glibc change:
- Fri Jan 19 2018 Björn Esser besser82@fedoraproject.org -
2.26.9000-46
- Remove deprecated libcrypt, gets replaced by libxcrypt
you could try downgrading to a glibc from before that and see if it helps? Only a possibility, though. Thanks!
The libxcrypt info says
... programs linked against libxcrypt will not work with glibc's libcrypt ...
I think that means an earlier glibc might cause problems with libxcrypt installed, since programs must be linked against it, though I'll still try it.
I had second thoughts about this, worried about bricking my machine. But, as I wrote Chris, the problem is in the automatic routing of command output to less, like systemd does. Not sure why bash would suddenly start doing this. There is no issue with dracut or grub2-mkconfig themselves.
On Tue, 2018-01-30 at 09:49 -0700, stan wrote:
I had second thoughts about this, worried about bricking my machine. But, as I wrote Chris, the problem is in the automatic routing of command output to less, like systemd does. Not sure why bash would suddenly start doing this. There is no issue with dracut or grub2-mkconfig themselves.
That's definitely weird! My first thought would be that it's possibly a profile bug; check ~/.bash_profile , ~/.bashrc and /etc/profile.d/ for anything that looks odd. Try as a different user; does it happen there?
On Wed, 31 Jan 2018 07:40:43 +0100 Adam Williamson adamwill@fedoraproject.org wrote:
That's definitely weird! My first thought would be that it's possibly a profile bug; check ~/.bash_profile , ~/.bashrc and /etc/profile.d/ for anything that looks odd. Try as a different user; does it happen there?
The behavior seems to have changed after yesterday's updates. It occurs on the first login after boot in the virtual consoles. But any logins by any user in a virtual console after that is fine. (root and 2 users, including the first user to log in). However, if I use less to look at a file, it still has the behavior for every user - I have to hit space to get past the message to see the file. In X, doing an su to a user other than the session belongs to still has the behavior. I looked at the bash files, and didn't see anything that would cause this. It started occurring with no changes to any of those files. Perhaps there was a configuration redefinition, and an option behaves differently?
These are the last changes within a reasonable timeframe to /etc/profile.d. Nothing there jumps out at me as a possible cause.
-rw-r--r--. 1 0 0 81 Jan 13 10:22 sh.local -rw-r--r--. 1 0 0 80 Jan 13 10:22 csh.local -rw-r--r--. 1 0 0 1000 Jan 14 06:46 ccache.sh -rw-r--r--. 1 0 0 987 Jan 14 06:46 ccache.csh -rw-r--r--. 1 0 0 248 Jan 19 05:01 vim.sh -rw-r--r--. 1 0 0 106 Jan 19 05:01 vim.csh -rw-r--r--. 1 0 0 78 Jan 20 15:58 cvs.sh -rw-r--r--. 1 0 0 92 Jan 20 15:58 cvs.csh -rw-r--r--. 1 0 0 56 Jan 22 08:21 colorsysstat.sh -rw-r--r--. 1 0 0 69 Jan 22 08:21 colorsysstat.csh -rw-r--r--. 1 0 0 1606 Jan 23 09:06 colorls.sh -rw-r--r--. 1 0 0 1741 Jan 23 09:06 colorls.csh lrwxrwxrwx. 1 0 0 28 Jan 25 14:57 modules.sh -> /etc/alternatives/modules.sh lrwxrwxrwx. 1 0 0 29 Jan 25 14:57 modules.csh -> /etc/alternatives/modules.csh -rw-r--r--. 1 0 0 264 Jan 26 05:21 guestfish.sh -rw-r--r--. 1 0 0 70 Jan 26 08:32 gnome-ssh-askpass.sh -rw-r--r--. 1 0 0 58 Jan 26 08:32 gnome-ssh-askpass.csh
It's annoying, but not a show stopper now that I know about it. I'll keep investigating.
Hey stan.
I ran into this same issue here. It seems it's caused by environment-modules. There's a bug on it:
https://bugzilla.redhat.com/show_bug.cgi?id=1539856
kevin
On Wed, 31 Jan 2018 13:58:52 -0800 Kevin Fenzi kevin@scrye.com wrote:
Hey stan.
I ran into this same issue here. It seems it's caused by environment-modules. There's a bug on it:
Thanks, that's it! I checked, and that package was updated on the day that the problem started. I don't think I'll do the workaround, I'll just wait for the fix. In the meantime I'll downgrade to the previous version from koji, environment-modules-4.0.0-2.fc28.x86_64.rpm, from November of last year.
-
Adam Williamson -
Chris Murphy -
Kevin Fenzi -
stan