Hi,
I have a weird situation when a user launches a Slurm interactive job on a new compute node with empty sssd cache. I'm not sure if its an issue with Slurm or SSSD/NSS. Below is the output of the Slurm session originated on node ip-0A0B0004 and starting interactive node ip-0A0B0006 by user1 with uid 705601104:
[user1@ip-0A0B0004 ~]$ srun -p lowmem -w lowmem-2 --pty bash
/usr/bin/id: cannot find name for group ID 705600513 /usr/bin/id: cannot find name for user ID 705601104 [I have no name!@ip-0A0B0006 ~]$
From this node I can successfully id user1 or any other AD user (this populates my cache and subsequent Slurm sessions for any user will resolve):
[I have no name!@ip-0A0B0006 ~]$ id user1 uid=705601104(user1) gid=705600513 groups=705600513,705601103(group1) [I have no name!@ip-0A0B0006 ~]$
Here is the relevant session from my sssd_nss.log:
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_send] (0x0400): CR #0: New request 'User by ID' (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_select_domains] (0x0400): CR #0: Performing a multi-domain search (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_domains] (0x0400): CR #0: Search will check the cache and check the data provider (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_set_domain] (0x0400): CR #0: Using domain [jmorey.net] (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_send] (0x0400): CR #0: Looking up UID:705601104@jmorey.net (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #0: Checking negative cache for [UID:705601104@jmorey.net] (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #0: [UID:705601104@jmorey.net] is not present in negative cache (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Looking up [UID:705601104@jmorey.net] in cache (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Object [UID:705601104@jmorey.net] was not found in cache (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_dp] (0x0400): CR #0: Looking up [UID:705601104@jmorey.net] in data provider (Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x562848473820:1:705601104@jmorey.net] (Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [jmorey.net][0x1][BE_REQ_USER][idnumber=705601104:-] (Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x562848473820:1:705601104@jmorey.net] (Fri Jul 10 22:47:58 2020) (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_common_dp_recv] (0x0040): CR #0: Data Provider Error: 3, 0, Success (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_common_dp_recv] (0x0400): CR #0: Due to an error we will return cached data (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Looking up [UID:705601104@jmorey.net] in cache (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #0: Object [UID:705601104@jmorey.net] was not found in cache (Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_process_result] (0x0400): CR #0: Finished: Not found
To my inexperienced eye it looks like it is communicating with the backend but receives some kind of error so it reverts to the cache, which is empty. The other oddity is the 'User by id' request that is trying to resolve based on the uid not the name. While this is happening I can successfully resolve user1 using SSH to the same node.
The SSH session:
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_send] (0x0400): CR #6: New request 'User by name' (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_process_input] (0x0400): CR #6: Parsing input name [user1] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'user1' matched without domain, user is user1 (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_set_name] (0x0400): CR #6: Setting name [user1] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_select_domains] (0x0400): CR #6: Performing a multi-domain search (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_domains] (0x0400): CR #6: Search will check the cache and check the data provider (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_set_domain] (0x0400): CR #6: Using domain [jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_prepare_domain_data] (0x0400): CR #6: Preparing input data for domain [jmorey.net] rules (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_send] (0x0400): CR #6: Looking up user1@jmorey.net (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #6: Checking negative cache for [user1@jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache] (0x0400): CR #6: [user1@jmorey.net] is not present in negative cache (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #6: Looking up [user1@jmorey.net] in cache (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #6: Object [user1@jmorey.net] was not found in cache (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_dp] (0x0400): CR #6: Looking up [user1@jmorey.net] in data provider (Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x564629422820:1:user1@jmorey.net@jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [jmorey.net][0x1][BE_REQ_USER][name=user1@jmorey.net:-] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x564629422820:1:user1@jmorey.net@jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #6: Looking up [user1@jmorey.net] in cache (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache_filter] (0x0400): CR #6: This request type does not support filtering result by negative cache (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_done] (0x0400): CR #6: Returning updated object [user1@jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_create_and_add_result] (0x0400): CR #6: Found 1 entries in domain jmorey.net (Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_req_destructor] (0x0400): Deleting request: [0x564629422820:1:user1@jmorey.net@jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_done] (0x0400): CR #6: Finished: Success
Here is my sssd.conf:
[sssd] config_file_version = 2 services = nss,pam domains = jmorey.net
[nss] filter_users = root filter_groups = root debug_level = 7
[pam]
[domain/jmorey.net] id_provider = ldap auth_provider = ldap access_provider = ldap debug_level = 7 dns_discovery_domain = jmorey.net enumerate = False cache_credentials = True case_sensitive = false ldap_schema = ad
ldap_uri = ldaps://jmorey-novo-ad.jmorey.net ldap_user_search_base = cn=Users,dc=jmorey,dc=net ldap_group_search_base = cn=group1,cn=Users,dc=jmorey,dc=net ldap_referrals = False ldap_tls_reqcert = never ldap_use_tokengroups = True ldap_id_mapping = True override_homedir = /mnt/exports/shared/home/%u fallback_homedir = /shared/home/%u default_shell = /bin/bash ldap_access_order = filter, expire ldap_account_expire_policy = ad ldap_access_filter = (|(memberOf=cn=group1,cn=Users,dc=jmorey,dc=net)) ldap_default_bind_dn = cn=user 1,cn=Users,dc=jmorey,dc=net ldap_default_authtok_type = password ldap_default_authtok =
thanks, Jerry