https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Bug ID: 1853261 Summary: sssd fails to start on read-only filesystem Product: Fedora Version: rawhide Status: NEW Component: sssd Assignee: sssd-maintainers@lists.fedoraproject.org Reporter: zbyszek@in.waw.pl QA Contact: extras-qa@fedoraproject.org CC: abokovoy@redhat.com, atikhono@redhat.com, jhrozek@redhat.com, lslebodn@redhat.com, mzidek@redhat.com, pbrezina@redhat.com, rharwood@redhat.com, sbose@redhat.com, ssorce@redhat.com, sssd-maintainers@lists.fedoraproject.org Target Milestone: --- Classification: Fedora
Description of problem: sssd refuses to start when /var/log is read-only. It is obviously a problem when trying to recover from a file system issue. Since logging is non-essential, sssd should just continue. At most is should emit a warning.
$ touch /var/tmp/foo touch: cannot touch '/var/tmp/foo': Read-only file system $ sudo systemctl start sssd.service Job for sssd.service failed because the control process exited with error code. See "systemctl status sssd.service" and "journalctl -xe" for details. [fedora@workstation-uefi ~]$ Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02 11:16:46 CEST): sssd[900]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02 11:16:47 CEST): sssd[903]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02 11:16:47 CEST): sssd[905]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02 11:16:47 CEST): sssd[906]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only file system]
Broadcast message from systemd-journald@workstation-uefi (Thu 2020-07-02 11:16:47 CEST): sssd[907]: Could not open file [/var/log/sssd/sssd.log]. Error: [30][Read-only file system]
We see multiple issues here: - the main one is that sssd should not fail to start as described above - but also, why is sssd not just logging to the journal? Why is it spamming with broadcast messages? If sssd would just log to the journal like any modern service, all those problems would be avoided.
Version-Release number of selected component (if applicable): sssd-2.3.0-2.fc33.x86_64
How reproducible: Deterministic
Steps to Reproduce: 1. Make /var/log read-only, either by not remounting the filesystems rw, or by using udev.blockdev-read-only with systemd-246+ 2. Try to start sssd
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Pavel Březina pbrezina@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|sssd-maintainers@lists.fedo |ppolawsk@redhat.com |raproject.org | Doc Type|--- |If docs needed, set a value
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #1 from Lukas Slebodnik lslebodn@redhat.com --- (In reply to Zbigniew Jędrzejewski-Szmek from comment #0)
We see multiple issues here:
- the main one is that sssd should not fail to start as described above
I would say you should use root to recover from a file system issue and root user is not handled by sssd
- but also, why is sssd not just logging to the journal? Why is it spamming
with broadcast messages?
By default it try to log to files in /var/log/sssd. Obviously it failed due to read-only file system. that's the reason of broadcast emssage.
If sssd would just log to the journal like any modern service, all those problems would be avoided.
sssd can log to journald you just need to explicitely configure it.
sh$ systemctl cat sssd | head # /usr/lib/systemd/system/sssd.service [Unit] Description=System Security Services Daemon # SSSD must be running before we permit user sessions Before=systemd-user-sessions.service nss-user-lookup.target Wants=nss-user-lookup.target
[Service] Environment=DEBUG_LOGGER=--logger=files EnvironmentFile=-/etc/sysconfig/sssd
Just set DEBUG_LOGGER in /etc/sysconfig/sssd (details in `man sssd`)
But I doubt it will help if system is read-only. sssd will still try to open rw files in /var/lib/sss/
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #2 from Alexey Tikhonov atikhono@redhat.com --- Hi,
1) "basic" logs by default are written to files (/var/log/sssd) and one can control this via "--logger" option (including option to redirect to journald or stderr completely) 2) "important" messages are also (additionally) logged to journald What you see as "Broadcast message from systemd-journald@workstation-uefi" is broadcast from journald, not directly from SSSD. You can control those broadcast messages via journald settings (namely `ForwardToWall` and `MaxLevelWall`)
So the questions here are: 1) Does this message have appropriate log level (*perhaps* emergency level of this specific message should be decreased)? 2) as you wrote, perhaps SSSD should fallback to "/dev/null" for "basic" logs output in case of fail to open files...
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #3 from Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl --- The goal of my bug report is to nudge Fedora toward a system which is resilient to failure. In particular this is motivated by making it easier for users to recover from disk corruption and other file system issues. Users are confused and unhappy when they boot to a tty prompt. I'm going through all services which reasonably should run fine with readonly fs. Obviously, in really bad cases, logging in at the tty will be necessary. But read-only root is not one of them.
There is really no reason to log to a private log file nowadays. Please just log to the journal by default. This is much nicer for users because they expect 'journalctl' to give them system logs, and not to have to hunt for some obscure log file. This also makes this *much* more robust. In particular, journald is completely fine with read-only root. Log files are stored in /run, and if /var/log becomes available, logs are flushed there. Logs should "just work" from user's POV.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #4 from Pavel Březina pbrezina@redhat.com --- Well, I don't think users would be happy given the amount of messages sssd produce. But I agree that being unable to produce logs should not be considered fatal.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #5 from Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl --- (In reply to Pavel Březina from comment #4)
Well, I don't think users would be happy given the amount of messages sssd produce.
I didn't consider this... On my machine sssd seems to be very quiet, but that's probably because I have just one local user. Users hate "chatty" services with a passion, so if sssd were to log the journal, it indeed shouldn't log too much (by default).
But I agree that being unable to produce logs should not be considered fatal.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #6 from Pavel Březina pbrezina@redhat.com --- Since you have only one local users you use sssd only marginally.
We use journald (well syslog) for the most important error messages (like something that requires admin attention), but the rest goes to files when logging is enabled (unless --logger journald).
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Michael Romero romerom@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |romerom@gmail.com
--- Comment #8 from Michael Romero romerom@gmail.com --- Any change this is going to be addressed? I'm trying to get us completely away from an older LDAP environment, but whenever a filesystem goes read-only, the only account we can get in to investigate with is either the "root" user via console, or a legacy LDAP user via "SSH". While these r/o FS issues don't happen that often, when they do - it would be GREAT if authentication could continue to function and not require logging in as root over console to resolve.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #9 from Alexey Tikhonov atikhono@redhat.com --- (In reply to Michael Romero from comment #8)
Any change this is going to be addressed? I'm trying to get us completely away from an older LDAP environment, but whenever a filesystem goes read-only, the only account we can get in to investigate with is either the "root" user via console, or a legacy LDAP user via "SSH". While these r/o FS issues don't happen that often, when they do - it would be GREAT if authentication could continue to function and not require logging in as root over console to resolve.
Solving this for logs won't help: sssd need writable /var/lib/sss/ anyway.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski ppolawsk@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |WONTFIX Last Closed| |2021-04-08 13:31:48
--- Comment #10 from Paweł Poławski ppolawsk@redhat.com --- After some discussion it looks like there is no point of SSSD working on read-only filesystem. User can already redirect logs to journal using existing SSSD options. The problem is our databases located at /var/lib/sss/db/* still needs to be RW for SSSD to work.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |NEW Resolution|WONTFIX |--- Keywords| |Reopened
--- Comment #11 from Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl --- Sorry, but that is not an acceptable answer.
After some discussion it looks like there is no point of SSSD working on read-only filesystem.
Let me quote the previous previous message:
but whenever a filesystem goes read-only, the only account we can get in to investigate with is either the "root" user via console, or a legacy LDAP user via "SSH".
If sssd was an opt-in thing, throwing up your hands and saying this is too hard to support would fine. But it is a default component in Fedora. This means that it also needs to solve the hard edge cases. The default authentication mechanism simply cannot stop working in a scenario which is a fairly common failure mode.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski ppolawsk@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Triaged
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski ppolawsk@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #12 from Ben Cotton bcotton@redhat.com --- This message is a reminder that Fedora 33 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '33'.
Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 33 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #13 from Ben Cotton bcotton@redhat.com --- This message is a reminder that Fedora 33 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '33'.
Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 33 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
--- Comment #14 from Ben Cotton bcotton@redhat.com --- This message is a reminder that Fedora 33 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '33'.
Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 33 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Paweł Poławski ppolawsk@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|ppolawsk@redhat.com |sssd-maint@redhat.com
https://bugzilla.redhat.com/show_bug.cgi?id=1853261
Ben Cotton bcotton@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |EOL Last Closed|2021-04-08 13:31:48 |2021-11-30 16:08:05
--- Comment #15 from Ben Cotton bcotton@redhat.com --- Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug.
Thank you for reporting this bug and we are sorry it could not be fixed.
sssd-maintainers@lists.fedoraproject.org
-
bugzilla＠redhat.com