Sssd experts,
We have a problem with two RHEL7 servers, where one particular account is logged into a great many times, fairly frequently. Can be as much as 2-3 concurrent sessions, but generally it’s just a bunch of sequential logins. All from that one account.
We have just converted this user authentication to sssd (AD back-end) from a non-Kerberos-based user authentication.
After an hour or so, the account can no longer log in. It reports an error from pam_sss.so.
It says this in /var/log/secure:
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
In /var/log/sssd/sssd_pam.log file we see:
(Thu Oct 15 13:53:36 2020) [sssd[pam]] [pam_dp_process_reply] (0x0200): received: [4 (System error)][amer.dell.com]
In /var/log/sssd/sssd_amer.dell.com.log, we see:
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [read_pipe_handler] (0x0400): EOF received, client finished
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [krb5_auth_done] (0x0040): The krb5_child process returned an error. Please inspect the krb5_child.log file or the journal for more information
And finally, in /var/log/sssd/krb5_child.log, we see:
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Initializing ccache of type [KEYRING]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): CC supports switch
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Match not found
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): krb5_cc_cache_match failed
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x0020): 1016: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [map_krb5_error] (0x0020): 1817: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x0200): Received error code 1432158209
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [pack_response_packet] (0x2000): response packet size: [4]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x4000): Response sent.
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [main] (0x0400): krb5_child completed successfully
So we think the problem is that all the keyrings are consumed, or this user has exceeded its quota of keyrings. Faster than Kerberos garbage collects them.
In googling, we see a similar problem but it’s specific to GNOME.
https://bugzilla.redhat.com/show_bug.cgi?id=1230750
Our problem does not involve GNOME, just raw SSH connections.
We bumped up */proc/sys/kernel/keys/maxkeys *(from default of 200 to 1000) and it
seemed to help. Just wondering what is the correct fix? And if this is the correct
fix, some guidance as to best value for maxkeys.
BTW, even when we specify :
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf, sssd’s krb5_child.log seems to want to use KEYRING. What’s up
with that?
Spike
On Thu, Oct 15, 2020 at 9:57 PM Spike White spikewhitetx@gmail.com wrote:
Sssd experts,
We have a problem with two RHEL7 servers, where one particular account is logged into a great many times, fairly frequently. Can be as much as 2-3 concurrent sessions, but generally it’s just a bunch of sequential logins. All from that one account.
We have just converted this user authentication to sssd (AD back-end) from a non-Kerberos-based user authentication.
After an hour or so, the account can no longer log in. It reports an error from pam_sss.so.
It says this in /var/log/secure:
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
In /var/log/sssd/sssd_pam.log file we see:
(Thu Oct 15 13:53:36 2020) [sssd[pam]] [pam_dp_process_reply] (0x0200): received: [4 (System error)][amer.dell.com]
In /var/log/sssd/sssd_amer.dell.com.log, we see:
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [read_pipe_handler] (0x0400): EOF received, client finished
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [krb5_auth_done] (0x0040): The krb5_child process returned an error. Please inspect the krb5_child.log file or the journal for more information
And finally, in /var/log/sssd/krb5_child.log, we see:
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Initializing ccache of type [KEYRING]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): CC supports switch
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Match not found
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): krb5_cc_cache_match failed
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x0020): 1016: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [map_krb5_error] (0x0020): 1817: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x0200): Received error code 1432158209
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [pack_response_packet] (0x2000): response packet size: [4]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x4000): Response sent.
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [main] (0x0400): krb5_child completed successfully
So we think the problem is that all the keyrings are consumed, or this user has exceeded its quota of keyrings. Faster than Kerberos garbage collects them.
In googling, we see a similar problem but it’s specific to GNOME.
https://bugzilla.redhat.com/show_bug.cgi?id=1230750
Our problem does not involve GNOME, just raw SSH connections.
We bumped up */proc/sys/kernel/keys/maxkeys *(from default of 200 to 1000) and it
seemed to help. Just wondering what is the correct fix? And if this is the correct
fix, some guidance as to best value for maxkeys.
BTW, even when we specify :
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf, sssd’s krb5_child.log seems to want to use KEYRING. What’s up
with that?
Don't you have this option redefined in /etc/krb5.conf.d/* by any chance? Or via `krb5_ccname_template` in sssd.conf?
Spike _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Alexey,
Thanks for responding. I checked /etc/krb5.conf.d/*. Good idea, but I checked. No settings of krb5_ccname_template there.
Yes, in sssd.conf there is a krb5_ccname_template. But we don't set it. In the sssd-krb5 man page, it says for thie parameter:
krb5_ccname_template (string) Location of the user's credential cache. Three credential cache types are currently supported: “FILE”, “DIR” and “KEYRING:persistent”. The cache can be specified either as TYPE:RESIDUAL, or as an absolute path, which implies the “FILE” type. In the template, the following sequences are substituted:
... Default: (from libkrb5)
The way I interpret that statement about default, is that if not set, it does whatever Kerberos libs are doing. Which is set in /etc/krb5.conf file or /etc/krb5.conf.d/*. That's why I thought it was so strange that when I set default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf, sssd still continued to store Kerberos creds in KEYRING.
We've worked around the problem by increasing *kernel.keys.maxkeys * to 1000. So at this point, it's just a curiosity question.
Thanks,
Spike
On Tue, Oct 20, 2020 at 3:27 AM Alexey Tikhonov atikhono@redhat.com wrote:
On Thu, Oct 15, 2020 at 9:57 PM Spike White spikewhitetx@gmail.com wrote:
Sssd experts,
We have a problem with two RHEL7 servers, where one particular account is logged into a great many times, fairly frequently. Can be as much as 2-3 concurrent sessions, but generally it’s just a bunch of sequential logins. All from that one account.
We have just converted this user authentication to sssd (AD back-end) from a non-Kerberos-based user authentication.
After an hour or so, the account can no longer log in. It reports an error from pam_sss.so.
It says this in /var/log/secure:
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
In /var/log/sssd/sssd_pam.log file we see:
(Thu Oct 15 13:53:36 2020) [sssd[pam]] [pam_dp_process_reply] (0x0200): received: [4 (System error)][amer.dell.com]
In /var/log/sssd/sssd_amer.dell.com.log, we see:
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [read_pipe_handler] (0x0400): EOF received, client finished
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [krb5_auth_done] (0x0040): The krb5_child process returned an error. Please inspect the krb5_child.log file or the journal for more information
And finally, in /var/log/sssd/krb5_child.log, we see:
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Initializing ccache of type [KEYRING]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): CC supports switch
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Match not found
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): krb5_cc_cache_match failed
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x0020): 1016: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [map_krb5_error] (0x0020): 1817: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x0200): Received error code 1432158209
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [pack_response_packet] (0x2000): response packet size: [4]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x4000): Response sent.
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [main] (0x0400): krb5_child completed successfully
So we think the problem is that all the keyrings are consumed, or this user has exceeded its quota of keyrings. Faster than Kerberos garbage collects them.
In googling, we see a similar problem but it’s specific to GNOME.
https://bugzilla.redhat.com/show_bug.cgi?id=1230750
Our problem does not involve GNOME, just raw SSH connections.
We bumped up */proc/sys/kernel/keys/maxkeys *(from default of 200 to 1000) and it
seemed to help. Just wondering what is the correct fix? And if this is the correct
fix, some guidance as to best value for maxkeys.
BTW, even when we specify :
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf, sssd’s krb5_child.log seems to want to use KEYRING. What’s up
with that?
Don't you have this option redefined in /etc/krb5.conf.d/* by any chance? Or via `krb5_ccname_template` in sssd.conf?
Spike _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
On Tue, Oct 20, 2020 at 4:16 PM Spike White spikewhitetx@gmail.com wrote:
Alexey,
Thanks for responding. I checked /etc/krb5.conf.d/*. Good idea, but I checked. No settings of krb5_ccname_template there.
Yes, in sssd.conf there is a krb5_ccname_template. But we don't set it. In the sssd-krb5 man page, it says for thie parameter:
krb5_ccname_template (string) Location of the user's credential cache. Three credential cachetypes are currently supported: “FILE”, “DIR” and “KEYRING:persistent”. The cache can be specified either as TYPE:RESIDUAL, or as an absolute path, which implies the “FILE” type. In the template, the following sequences are substituted:
... Default: (from libkrb5)The way I interpret that statement about default, is that if not set, it does whatever Kerberos libs are doing.
That would be my expectation as well.
What kind of ticket yields running `kinit` manually when you have ```
BTW, even when we specify :
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf
```
?
And is "kr5.conf" merely a mistype?
Which is set in /etc/krb5.conf file or /etc/krb5.conf.d/*. That's why I thought it was so strange that when I set default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf, sssd still continued to store Kerberos creds in KEYRING.
We've worked around the problem by increasing *kernel.keys.maxkeys * to 1000. So at this point, it's just a curiosity question.
Thanks,
Spike
On Tue, Oct 20, 2020 at 3:27 AM Alexey Tikhonov atikhono@redhat.com wrote:
On Thu, Oct 15, 2020 at 9:57 PM Spike White spikewhitetx@gmail.com wrote:
Sssd experts,
We have a problem with two RHEL7 servers, where one particular account is logged into a great many times, fairly frequently. Can be as much as 2-3 concurrent sessions, but generally it’s just a bunch of sequential logins. All from that one account.
We have just converted this user authentication to sssd (AD back-end) from a non-Kerberos-based user authentication.
After an hour or so, the account can no longer log in. It reports an error from pam_sss.so.
It says this in /var/log/secure:
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): received for user svc_npesrsnonprd: 4 (System error)
In /var/log/sssd/sssd_pam.log file we see:
(Thu Oct 15 13:53:36 2020) [sssd[pam]] [pam_dp_process_reply] (0x0200): received: [4 (System error)][amer.dell.com]
In /var/log/sssd/sssd_amer.dell.com.log, we see:
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [read_pipe_handler] (0x0400): EOF received, client finished
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [krb5_auth_done] (0x0040): The krb5_child process returned an error. Please inspect the krb5_child.log file or the journal for more information
And finally, in /var/log/sssd/krb5_child.log, we see:
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Initializing ccache of type [KEYRING]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): CC supports switch
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): Match not found
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x4000): krb5_cc_cache_match failed
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache] (0x0020): 1016: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [map_krb5_error] (0x0020): 1817: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x0200): Received error code 1432158209
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [pack_response_packet] (0x2000): response packet size: [4]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data] (0x4000): Response sent.
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [main] (0x0400): krb5_child completed successfully
So we think the problem is that all the keyrings are consumed, or this user has exceeded its quota of keyrings. Faster than Kerberos garbage collects them.
In googling, we see a similar problem but it’s specific to GNOME.
https://bugzilla.redhat.com/show_bug.cgi?id=1230750
Our problem does not involve GNOME, just raw SSH connections.
We bumped up */proc/sys/kernel/keys/maxkeys *(from default of 200 to 1000) and it
seemed to help. Just wondering what is the correct fix? And if this is the correct
fix, some guidance as to best value for maxkeys.
BTW, even when we specify :
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf, sssd’s krb5_child.log seems to want to use KEYRING. What’s up
with that?
Don't you have this option redefined in /etc/krb5.conf.d/* by any chance? Or via `krb5_ccname_template` in sssd.conf?
Spike _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users@lists.fedorahosted.org
-
Alexey Tikhonov -
Spike White