Issues with Free IPA (Red Hat IDM). Sporadic lookup results. Different results in EL 6 and 7.
All,
First the deets of the setup: 3 IDM servers on RHEL 7.7 ipa version VERSION: 4.6.5, API_VERSION: 2.231 sssd version 1.16.4 389 directory server version 1.3.9.1-10
Clients: EL7: ipa version 5.6.5, sssd version EL6: ipa version 3.0.0.51, sssd 1.13.3.60
Servers are setup in an AD trust ipa-ad-trust-posix. I have done the performance tweaks for sssd as described at https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-i... and we use the accounts/groups in AD for login, authorization, and file ownership.
There are 3 main issues we are having. 1. On ipa clients on EL 7 servers we are running into sporadic issues. If you totally clear the sssd cache and do an ls -la on let's say /home where there are 12 unique owners of directories usually between 8 to 10 of the UID numbers come back with the the user found, but you have to wait 1 to 5 minutes before the rest of the uids owning the other directories come back as found.
2. Also on ipa clients on EL 7 servers we are running into an issue where occasionally, at what seems like totally random times, AD users that normally can access a client suddenly can't. Someone will have to go in and clear the SSSD cache after which the user will once again be able to access the system.
3. There are some users that are just not visible on the EL 6 clients. On the IDM servers and on EL 7 clients the AD users are able to be found by id and the users can login. On EL 6 those AD users just do not resolve and cannot be seen.
Anyway, we have had Red Hat support looking at problem 3 for almost 2 months now with no luck. We have been poking around at problems 1 and 2 but no eureka moments as of yet. I'm hoping someone else on this list has encountered these same issues and found a solution. I would greatly appreciate any insight and help that anyone could provide.
Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services University of Virginia rwj5d@virginia.edu
On 9/26/19 2:24 AM, Bob Jones via FreeIPA-users wrote:
All,
First the deets of the setup: 3 IDM servers on RHEL 7.7 ipa version VERSION: 4.6.5, API_VERSION: 2.231 sssd version 1.16.4 389 directory server version 1.3.9.1-10
Clients: EL7: ipa version 5.6.5, sssd version EL6: ipa version 3.0.0.51, sssd 1.13.3.60
Servers are setup in an AD trust ipa-ad-trust-posix. I have done the performance tweaks for sssd as described at https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-i... and we use the accounts/groups in AD for login, authorization, and file ownership.
There are 3 main issues we are having.
On ipa clients on EL 7 servers we are running into sporadic issues. If you totally clear the sssd cache and do an ls -la on let's say /home where there are 12 unique owners of directories usually between 8 to 10 of the UID numbers come back with the the user found, but you have to wait 1 to 5 minutes before the rest of the uids owning the other directories come back as found.
Also on ipa clients on EL 7 servers we are running into an issue where occasionally, at what seems like totally random times, AD users that normally can access a client suddenly can't. Someone will have to go in and clear the SSSD cache after which the user will once again be able to access the system.
This issue looks like BZ 1717008 [1] User incorrectly added to negative cache when backend is reconnecting to IPA service / timed out: error code 32 'No such object'
It's been fixed on ipa-4-6 branch but not shipped yet.
flo [1] https://bugzilla.redhat.com/show_bug.cgi?id=1717008
- There are some users that are just not visible on the EL 6 clients. On the IDM servers and on EL 7 clients the AD users are able to be found by id and the users can login. On EL 6 those AD users just do not resolve and cannot be seen.
Anyway, we have had Red Hat support looking at problem 3 for almost 2 months now with no luck. We have been poking around at problems 1 and 2 but no eureka moments as of yet. I'm hoping someone else on this list has encountered these same issues and found a solution. I would greatly appreciate any insight and help that anyone could provide.
Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services University of Virginia rwj5d@virginia.edu _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Thank you for the answer. My guess was it had something to do with the negative cache, but wasn’t sure. Unfortunately I’m not authorized to access bug #1717008 so cannot view the details in order to potentially confirm this is my issue. Are there any log messages I should be looking for in order to confirm this is the problem I’m seeing?
Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On Sep 26, 2019, at 4:25 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 9/26/19 2:24 AM, Bob Jones via FreeIPA-users wrote:
All, First the deets of the setup: 3 IDM servers on RHEL 7.7 ipa version VERSION: 4.6.5, API_VERSION: 2.231 sssd version 1.16.4 389 directory server version 1.3.9.1-10 Clients: EL7: ipa version 5.6.5, sssd version EL6: ipa version 3.0.0.51, sssd 1.13.3.60 Servers are setup in an AD trust ipa-ad-trust-posix. I have done the performance tweaks for sssd as described at https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-i... and we use the accounts/groups in AD for login, authorization, and file ownership. There are 3 main issues we are having.
- On ipa clients on EL 7 servers we are running into sporadic issues. If you totally clear the sssd cache and do an ls -la on let's say /home where there are 12 unique owners of directories usually between 8 to 10 of the UID numbers come back with the the user found, but you have to wait 1 to 5 minutes before the rest of the uids owning the other directories come back as found.
- Also on ipa clients on EL 7 servers we are running into an issue where occasionally, at what seems like totally random times, AD users that normally can access a client suddenly can't. Someone will have to go in and clear the SSSD cache after which the user will once again be able to access the system.
This issue looks like BZ 1717008 [1] User incorrectly added to negative cache when backend is reconnecting to IPA service / timed out: error code 32 'No such object'
It's been fixed on ipa-4-6 branch but not shipped yet.
flo [1] https://bugzilla.redhat.com/show_bug.cgi?id=1717008
- There are some users that are just not visible on the EL 6 clients. On the IDM servers and on EL 7 clients the AD users are able to be found by id and the users can login. On EL 6 those AD users just do not resolve and cannot be seen.
Anyway, we have had Red Hat support looking at problem 3 for almost 2 months now with no luck. We have been poking around at problems 1 and 2 but no eureka moments as of yet. I'm hoping someone else on this list has encountered these same issues and found a solution. I would greatly appreciate any insight and help that anyone could provide. Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services University of Virginia rwj5d@virginia.edu _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 9/26/19 3:44 PM, Jones, Bob (rwj5d) via FreeIPA-users wrote:
Thank you for the answer. My guess was it had something to do with the negative cache, but wasn’t sure. Unfortunately I’m not authorized to access bug #1717008 so cannot view the details in order to potentially confirm this is my issue. Are there any log messages I should be looking for in order to confirm this is the problem I’m seeing?
The corresponding upstream ticket is https://pagure.io/freeipa/issue/8044. The LDAP server returns 32 (no such object) instead of an appropriate error code when there is a timeout or operations error.
When the issue happens, the sssd logs display something similar to: May 17 15:29:04 [sdap_id_conn_data_expire_handler] (0x0080): connection is about to expire, releasing it May 17 15:29:11 [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA' May 17 15:29:11 [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA' May 17 15:29:11 [sdap_cli_auth_step] (0x0100): expire timeout is 900 May 17 15:29:11 [sasl_bind_send] (0x0100): Executing sasl bind mech: GSSAPI, user: host/XXXXXX May 17 15:29:11 [child_sig_handler] (0x0100): child [34594] finished successfully. May 17 15:29:11 [fo_set_port_status] (0x0100): Marking port 389 of server 'XXXXXX' as 'working' May 17 15:29:11 [set_server_common_status] (0x0100): Marking server 'XXXXXX' as 'working'
And failure to find a valid user: May 17 15:29:12 [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: No such object(32), (null). May 17 15:29:12 [sysdb_get_real_name] (0x0040): Cannot find user [XXX@XXX] in cache May 17 15:29:12 [ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending request
HTH, flo
Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On Sep 26, 2019, at 4:25 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 9/26/19 2:24 AM, Bob Jones via FreeIPA-users wrote:
All, First the deets of the setup: 3 IDM servers on RHEL 7.7 ipa version VERSION: 4.6.5, API_VERSION: 2.231 sssd version 1.16.4 389 directory server version 1.3.9.1-10 Clients: EL7: ipa version 5.6.5, sssd version EL6: ipa version 3.0.0.51, sssd 1.13.3.60 Servers are setup in an AD trust ipa-ad-trust-posix. I have done the performance tweaks for sssd as described at https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-i... and we use the accounts/groups in AD for login, authorization, and file ownership. There are 3 main issues we are having.
- On ipa clients on EL 7 servers we are running into sporadic issues. If you totally clear the sssd cache and do an ls -la on let's say /home where there are 12 unique owners of directories usually between 8 to 10 of the UID numbers come back with the the user found, but you have to wait 1 to 5 minutes before the rest of the uids owning the other directories come back as found.
- Also on ipa clients on EL 7 servers we are running into an issue where occasionally, at what seems like totally random times, AD users that normally can access a client suddenly can't. Someone will have to go in and clear the SSSD cache after which the user will once again be able to access the system.
This issue looks like BZ 1717008 [1] User incorrectly added to negative cache when backend is reconnecting to IPA service / timed out: error code 32 'No such object'
It's been fixed on ipa-4-6 branch but not shipped yet.
flo [1] https://bugzilla.redhat.com/show_bug.cgi?id=1717008
- There are some users that are just not visible on the EL 6 clients. On the IDM servers and on EL 7 clients the AD users are able to be found by id and the users can login. On EL 6 those AD users just do not resolve and cannot be seen.
Anyway, we have had Red Hat support looking at problem 3 for almost 2 months now with no luck. We have been poking around at problems 1 and 2 but no eureka moments as of yet. I'm hoping someone else on this list has encountered these same issues and found a solution. I would greatly appreciate any insight and help that anyone could provide. Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services University of Virginia rwj5d@virginia.edu _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
I am seeing entries for “ldap_extended_operation result: No such object(32), (null)” when some users that I know exist and can look up on other systems cannot be resolved on a system. Until the version of IPA with this fix finds it’s way into RHEL 7, is there a workaround? If the issue is that objects are getting erroneously added to negative cache would setting entry_negative_timeout to 0 mitigate this (an actual real need for negative cache on these systems should be minimal).
Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On Sep 27, 2019, at 5:09 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 9/26/19 3:44 PM, Jones, Bob (rwj5d) via FreeIPA-users wrote:
Thank you for the answer. My guess was it had something to do with the negative cache, but wasn’t sure. Unfortunately I’m not authorized to access bug #1717008 so cannot view the details in order to potentially confirm this is my issue. Are there any log messages I should be looking for in order to confirm this is the problem I’m seeing?
The corresponding upstream ticket is https://pagure.io/freeipa/issue/8044. The LDAP server returns 32 (no such object) instead of an appropriate error code when there is a timeout or operations error.
When the issue happens, the sssd logs display something similar to: May 17 15:29:04 [sdap_id_conn_data_expire_handler] (0x0080): connection is about to expire, releasing it May 17 15:29:11 [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA' May 17 15:29:11 [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA' May 17 15:29:11 [sdap_cli_auth_step] (0x0100): expire timeout is 900 May 17 15:29:11 [sasl_bind_send] (0x0100): Executing sasl bind mech: GSSAPI, user: host/XXXXXX May 17 15:29:11 [child_sig_handler] (0x0100): child [34594] finished successfully. May 17 15:29:11 [fo_set_port_status] (0x0100): Marking port 389 of server 'XXXXXX' as 'working' May 17 15:29:11 [set_server_common_status] (0x0100): Marking server 'XXXXXX' as 'working'
And failure to find a valid user: May 17 15:29:12 [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: No such object(32), (null). May 17 15:29:12 [sysdb_get_real_name] (0x0040): Cannot find user [XXX@XXX] in cache May 17 15:29:12 [ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending request
HTH, flo
Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On Sep 26, 2019, at 4:25 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 9/26/19 2:24 AM, Bob Jones via FreeIPA-users wrote:
All, First the deets of the setup: 3 IDM servers on RHEL 7.7 ipa version VERSION: 4.6.5, API_VERSION: 2.231 sssd version 1.16.4 389 directory server version 1.3.9.1-10 Clients: EL7: ipa version 5.6.5, sssd version EL6: ipa version 3.0.0.51, sssd 1.13.3.60 Servers are setup in an AD trust ipa-ad-trust-posix. I have done the performance tweaks for sssd as described at https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-i... and we use the accounts/groups in AD for login, authorization, and file ownership. There are 3 main issues we are having.
- On ipa clients on EL 7 servers we are running into sporadic issues. If you totally clear the sssd cache and do an ls -la on let's say /home where there are 12 unique owners of directories usually between 8 to 10 of the UID numbers come back with the the user found, but you have to wait 1 to 5 minutes before the rest of the uids owning the other directories come back as found.
- Also on ipa clients on EL 7 servers we are running into an issue where occasionally, at what seems like totally random times, AD users that normally can access a client suddenly can't. Someone will have to go in and clear the SSSD cache after which the user will once again be able to access the system.
This issue looks like BZ 1717008 [1] User incorrectly added to negative cache when backend is reconnecting to IPA service / timed out: error code 32 'No such object'
It's been fixed on ipa-4-6 branch but not shipped yet.
flo [1] https://bugzilla.redhat.com/show_bug.cgi?id=1717008
- There are some users that are just not visible on the EL 6 clients. On the IDM servers and on EL 7 clients the AD users are able to be found by id and the users can login. On EL 6 those AD users just do not resolve and cannot be seen.
Anyway, we have had Red Hat support looking at problem 3 for almost 2 months now with no luck. We have been poking around at problems 1 and 2 but no eureka moments as of yet. I'm hoping someone else on this list has encountered these same issues and found a solution. I would greatly appreciate any insight and help that anyone could provide. Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services University of Virginia rwj5d@virginia.edu _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hello all,
Florence has graciously helped me determine the underlying problem for issue 2 which I also believe is part of the problem with issue 1 as well. Has anyone experienced or have any idea about issue 3? I have to believe there is some difference in how sssd 1.13.3 and ipa 3.0.0 is handling the information returned from Active Directory (by way of the IDM server ipa version 4.6.5) that is causing the behavior being seen in issue 3.
Thanks, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services
On Sep 25, 2019, at 8:24 PM, Bob Jones via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
All,
First the deets of the setup: 3 IDM servers on RHEL 7.7 ipa version VERSION: 4.6.5, API_VERSION: 2.231 sssd version 1.16.4 389 directory server version 1.3.9.1-10
Clients: EL7: ipa version 5.6.5, sssd version EL6: ipa version 3.0.0.51, sssd 1.13.3.60
Servers are setup in an AD trust ipa-ad-trust-posix. I have done the performance tweaks for sssd as described at https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-i... and we use the accounts/groups in AD for login, authorization, and file ownership.
There are 3 main issues we are having.
On ipa clients on EL 7 servers we are running into sporadic issues. If you totally clear the sssd cache and do an ls -la on let's say /home where there are 12 unique owners of directories usually between 8 to 10 of the UID numbers come back with the the user found, but you have to wait 1 to 5 minutes before the rest of the uids owning the other directories come back as found.
Also on ipa clients on EL 7 servers we are running into an issue where occasionally, at what seems like totally random times, AD users that normally can access a client suddenly can't. Someone will have to go in and clear the SSSD cache after which the user will once again be able to access the system.
There are some users that are just not visible on the EL 6 clients. On the IDM servers and on EL 7 clients the AD users are able to be found by id and the users can login. On EL 6 those AD users just do not resolve and cannot be seen.
Anyway, we have had Red Hat support looking at problem 3 for almost 2 months now with no luck. We have been poking around at problems 1 and 2 but no eureka moments as of yet. I'm hoping someone else on this list has encountered these same issues and found a solution. I would greatly appreciate any insight and help that anyone could provide.
Sincerely, — Bob Jones Lead Linux Services Engineer ITS ECP - Linux Services University of Virginia rwj5d@virginia.edu _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Bob Jones -
Florence Blanc-Renaud -
Jones, Bob (rwj5d)