So yesterday we upgrade our older IPA 3.x servers (RHEL 6.8) to the latest and greatest (RHEL 6.9) and it seemed to be working as expected. Came in the next day and older IPA 4.2 server (RHEL 7.2) was having issues so thought it would be a good time patch it up to the latest (IPA 4.4 and RHEL 7.3) seemed okay after it came back up but after a little while hosts configured to use IPA stopped allowing access via ssh.
I can explicitly put in sssd.conf to use the patched up 4.4 server and I seem to be able to log in via ssh but when a server is pointing to one of the 3.x servers it just gives me an access denied. The strange part is some accounts (like my test account) can log in to some servers via ssh and can even sudo to root where available regardless of which IPA server the host is using.
Any help would be appreciated because I'm stumped on this one. Just let me know what logs would be helpful in troubleshooting.
Thanks!
After upping the log levels on sssd on one of the failing servers I saw this in one of the sssd log files:
from sssd_pamd.log:
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/domain.tld/jbowman] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_issue_request] (0x0400): Issuing request for [0x41b5c0:3:jbowman@domain.tld] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_account_msg] (0x0400): Creating request for [domain.tld][3][1][name=jbowman] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_add_timeout] (0x2000): 0x20ef8a0 (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_internal_get_send] (0x0400): Entering request [0x41b5c0:3:jbowman@domain.tld] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_remove_timeout] (0x2000): 0x20ef8a0 (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 3 errno: 22 error message: Init Groups Failed (Wed Jun 14 23:16:05 2017) [sssd[pam]] [pam_check_user_dp_callback] (0x0040): Unable to get information from Data Provider Error: 3, 22, Init Groups Failed
from sssd_domain.tld.log (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_eval_user_element] (0x0080): Parse error on [cn=system: manage service principals+nsuniqueid=e8d2f834-512111e7-9205b5bf-43202000,cn=permissions,cn=pbac,dc=domain,dc=tld] (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_ctx_to_rules] (0x0020): Could not construct eval request (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [ipa_hbac_evaluate_rules] (0x0020): Could not construct HBAC rules (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_id_op_destroy] (0x4000): releasing operation connection (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100): Backend returned: (3, 4, <NULL>) [Internal Error (System error)] (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100): Sending result [4][domain.tld] (Wed Jun 14 22:55:37 2017) [sssd[be[domainn.tld]]] [be_pam_handler_callback] (0x0100): Sent result [4][domain.tld] (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000): Trace: sh[0x7ea6b0], connected[1], ops[(nil)], ldap[0x844de0] (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000): Trace: ldap_result found nothing! (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000): dbus conn: 7B2A00 (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000): Dispatching. (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_message_handler] (0x4000): Received SBUS method [ping]
I saw a similar issue in a previous posting to the list: https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html
I was wondering if these errors might be related to the issues I'm seeing currently since they seem very similar so far...
On Thu, Jun 15, 2017 at 04:28:13AM -0000, john.bowman--- via FreeIPA-users wrote:
After upping the log levels on sssd on one of the failing servers I saw this in one of the sssd log files:
from sssd_pamd.log:
(Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/domain.tld/jbowman] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_issue_request] (0x0400): Issuing request for [0x41b5c0:3:jbowman@domain.tld] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_account_msg] (0x0400): Creating request for [domain.tld][3][1][name=jbowman] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_add_timeout] (0x2000): 0x20ef8a0 (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_internal_get_send] (0x0400): Entering request [0x41b5c0:3:jbowman@domain.tld] (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sbus_remove_timeout] (0x2000): 0x20ef8a0 (Wed Jun 14 23:16:05 2017) [sssd[pam]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 3 errno: 22 error message: Init Groups Failed (Wed Jun 14 23:16:05 2017) [sssd[pam]] [pam_check_user_dp_callback] (0x0040): Unable to get information from Data Provider Error: 3, 22, Init Groups Failed
from sssd_domain.tld.log (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_eval_user_element] (0x0080): Parse error on [cn=system: manage service principals+nsuniqueid=e8d2f834-512111e7-9205b5bf-43202000,cn=permissions,cn=pbac,dc=domain,dc=tld]
Yes, only recent vresions of sssd can skip over the replication conflicts. I would recommend to clear the conflicts manually.
(Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [hbac_ctx_to_rules] (0x0020): Could not construct eval request (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [ipa_hbac_evaluate_rules] (0x0020): Could not construct HBAC rules (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_id_op_destroy] (0x4000): releasing operation connection (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100): Backend returned: (3, 4, <NULL>) [Internal Error (System error)] (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [be_pam_handler_callback] (0x0100): Sending result [4][domain.tld] (Wed Jun 14 22:55:37 2017) [sssd[be[domainn.tld]]] [be_pam_handler_callback] (0x0100): Sent result [4][domain.tld] (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000): Trace: sh[0x7ea6b0], connected[1], ops[(nil)], ldap[0x844de0] (Wed Jun 14 22:55:37 2017) [sssd[be[domain.tld]]] [sdap_process_result] (0x2000): Trace: ldap_result found nothing! (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000): dbus conn: 7B2A00 (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_dispatch] (0x4000): Dispatching. (Wed Jun 14 22:55:38 2017) [sssd[be[domain.tld]]] [sbus_message_handler] (0x4000): Received SBUS method [ping]
I saw a similar issue in a previous posting to the list: https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html
I was wondering if these errors might be related to the issues I'm seeing currently since they seem very similar so far... _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
You'll have to forgive my ignorance here since I'm still fairly new to IPA and fortunately haven't run in to many issues as of yet.
The three IPA 3.0 servers all have what look to be following conflicts:
$ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld" "nsds5ReplConflict=*" * nsds5ReplConflict | grep nsds5ReplConflict # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict nsds5ReplConflict: namingConflict cn=ipa4-4.domain.tld,cn=masters,cn=ipa,cn nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum= nsds5ReplConflict: namingConflict cn=ipaservers,cn=hostgroups,cn=accounts,dc=z nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=domain,dc=tld nsds5ReplConflict: namingConflict cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain, nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=domain,dc=tld nsds5ReplConflict: namingConflict cn=dns administrators,cn=privileges,cn=pbac, nsds5ReplConflict: namingConflict cn=dns servers,cn=privileges,cn=pbac,dc=domain nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=domain,dc=tld nsds5ReplConflict: namingConflict cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=domain, nsds5ReplConflict: namingConflict cn=ca,cn=topology,cn=ipa,cn=etc,dc=domain,dc=u nsds5ReplConflict: namingConflict cn=system: add ca,cn=permissions,cn=pbac,dc= nsds5ReplConflict: namingConflict cn=system: delete ca,cn=permissions,cn=pbac, nsds5ReplConflict: namingConflict cn=system: modify ca,cn=permissions,cn=pbac, nsds5ReplConflict: namingConflict cn=system: read cas,cn=permissions,cn=pbac,d nsds5ReplConflict: namingConflict cn=system: modify dns servers configuration, nsds5ReplConflict: namingConflict cn=system: read dns servers configuration,cn nsds5ReplConflict: namingConflict cn=system: add ipa locations,cn=permissions, nsds5ReplConflict: namingConflict cn=system: modify ipa locations,cn=permissio nsds5ReplConflict: namingConflict cn=system: read ipa locations,cn=permissions nsds5ReplConflict: namingConflict cn=system: remove ipa locations,cn=permissio nsds5ReplConflict: namingConflict cn=system: read locations of ipa servers,cn= nsds5ReplConflict: namingConflict cn=system: read status of services on ipa se nsds5ReplConflict: namingConflict cn=system: manage service principals,cn=perm nsds5ReplConflict: namingConflict cn=system: manage user principals,cn=permiss nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum=
While the IPA 4.4 server shows no conflicts: $ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld" "nsds5ReplConflict=*" * nsds5ReplConflict | grep nsds5ReplConflict # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict
So I would need to delete/modify the conflicts on the IPA 3.0 servers but the IPA 4.4 server should be okay, correct? Is there any impact to running the ldapmodify command to remove these conflicts while services are running? Would I need to do this on each of the IPA 3.x servers?
Looking at one of the conflicts on one of the IPA 3.0: $ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld" "cn=domain" # extended LDIF # # LDAPv3 # base <dc=domain,dc=tld> with scope subtree # filter: cn=domain # requesting: ALL #
# domain, topology, ipa, etc, domain.us dn: cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain,dc=tld cn: domain nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount objectClass: top objectClass: iparepltopoconf ipaReplTopoConfRoot: dc=domain,dc=tld nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in ternalModifyTimestamp
# domain + e8d2f70e-512111e7-9205b5bf-43202000, topology, ipa, etc, domain.us dn: cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000,cn=topology,cn=ip a,cn=etc,dc=domain,dc=tld nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in ternalModifyTimestamp ipaReplTopoConfRoot: dc=domain,dc=tld objectClass: top objectClass: iparepltopoconf nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount cn: domain
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2
Would I need to remove the "dn: cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000" entry in this case? And do that removal on each server?
Thank you and any help is appreciated!
On Thu, Jun 15, 2017 at 01:07:27PM -0000, john.bowman--- via FreeIPA-users wrote:
You'll have to forgive my ignorance here since I'm still fairly new to IPA and fortunately haven't run in to many issues as of yet.
The three IPA 3.0 servers all have what look to be following conflicts:
$ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld" "nsds5ReplConflict=*" * nsds5ReplConflict | grep nsds5ReplConflict # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict nsds5ReplConflict: namingConflict cn=ipa4-4.domain.tld,cn=masters,cn=ipa,cn nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum= nsds5ReplConflict: namingConflict cn=ipaservers,cn=hostgroups,cn=accounts,dc=z nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=domain,dc=tld nsds5ReplConflict: namingConflict cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain, nsds5ReplConflict: namingConflict cn=locations,cn=etc,dc=domain,dc=tld nsds5ReplConflict: namingConflict cn=dns administrators,cn=privileges,cn=pbac, nsds5ReplConflict: namingConflict cn=dns servers,cn=privileges,cn=pbac,dc=domain nsds5ReplConflict: namingConflict cn=cas,cn=ca,dc=domain,dc=tld nsds5ReplConflict: namingConflict cn=dogtag,cn=custodia,cn=ipa,cn=etc,dc=domain, nsds5ReplConflict: namingConflict cn=ca,cn=topology,cn=ipa,cn=etc,dc=domain,dc=u nsds5ReplConflict: namingConflict cn=system: add ca,cn=permissions,cn=pbac,dc= nsds5ReplConflict: namingConflict cn=system: delete ca,cn=permissions,cn=pbac, nsds5ReplConflict: namingConflict cn=system: modify ca,cn=permissions,cn=pbac, nsds5ReplConflict: namingConflict cn=system: read cas,cn=permissions,cn=pbac,d nsds5ReplConflict: namingConflict cn=system: modify dns servers configuration, nsds5ReplConflict: namingConflict cn=system: read dns servers configuration,cn nsds5ReplConflict: namingConflict cn=system: add ipa locations,cn=permissions, nsds5ReplConflict: namingConflict cn=system: modify ipa locations,cn=permissio nsds5ReplConflict: namingConflict cn=system: read ipa locations,cn=permissions nsds5ReplConflict: namingConflict cn=system: remove ipa locations,cn=permissio nsds5ReplConflict: namingConflict cn=system: read locations of ipa servers,cn= nsds5ReplConflict: namingConflict cn=system: read status of services on ipa se nsds5ReplConflict: namingConflict cn=system: manage service principals,cn=perm nsds5ReplConflict: namingConflict cn=system: manage user principals,cn=permiss nsds5ReplConflict: namingConflict dnahostname=ipa4-4.domain.tld+dnaportnum=
While the IPA 4.4 server shows no conflicts: $ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld" "nsds5ReplConflict=*" * nsds5ReplConflict | grep nsds5ReplConflict # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict
Depends on whether you need to keep the data on the v3 machine and whether the data on the v4 machine is correct...
But the general guide to managing replication conflicts is: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/htm...
So I would need to delete/modify the conflicts on the IPA 3.0 servers but the IPA 4.4 server should be okay, correct? Is there any impact to running the ldapmodify command to remove these conflicts while services are running? Would I need to do this on each of the IPA 3.x servers?
Looking at one of the conflicts on one of the IPA 3.0: $ ldapsearch -D "cn=directory manager" -w secret -b "dc=domain,dc=tld" "cn=domain" # extended LDIF # # LDAPv3 # base <dc=domain,dc=tld> with scope subtree # filter: cn=domain # requesting: ALL #
# domain, topology, ipa, etc, domain.us dn: cn=domain,cn=topology,cn=ipa,cn=etc,dc=domain,dc=tld cn: domain nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount objectClass: top objectClass: iparepltopoconf ipaReplTopoConfRoot: dc=domain,dc=tld nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in ternalModifyTimestamp
# domain + e8d2f70e-512111e7-9205b5bf-43202000, topology, ipa, etc, domain.us dn: cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000,cn=topology,cn=ip a,cn=etc,dc=domain,dc=tld nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in ternalModifyTimestamp ipaReplTopoConfRoot: dc=domain,dc=tld objectClass: top objectClass: iparepltopoconf nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount cn: domain
# search result search: 2 result: 0 Success
# numResponses: 3 # numEntries: 2
Would I need to remove the "dn: cn=domain+nsuniqueid=e8d2f70e-512111e7-9205b5bf-43202000" entry in this case? And do that removal on each server?
Thank you and any help is appreciated! _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Which path would be better? Upgrading sssd on the older machines or attempting to delete the ldap entries? Both eventually? Does having the namingConflict entries pose a threat to the system stability?
On Thu, Jun 15, 2017 at 05:15:41PM -0000, john.bowman--- via FreeIPA-users wrote:
Which path would be better? Upgrading sssd on the older machines or attempting to delete the ldap entries?
I think you want to fix the server side, upgrading sssd is just a quick kludge to let you access the systems.
freeipa-users@lists.fedorahosted.org
-
Jakub Hrozek -
John Bowman -
john.bowman＠zayo.com