When I first installed our replica, it worked just fine - I could add a user and see it on the master server. And vice versa.
I recently went back to take a look and make sure everything was working - and it's not.
ipactl status shows everything is ok. Munge is up. I can ssh hostname between machines.
When I look at the ID Views in the interface, I get an "IPA Error 903: InternalError".
When I do an id <username> I get nosuch user.
I did some googling. In /var log/dirsrv/domain/errors I found this:
[26/Oct/2017:12:31:23.454702287 +1100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ vmdr-linuxidm.unix.domain.com@UNIX.DOMAIN.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
I can get `kinit admin` working fine. But there's something wrong. I don't know where to look exactly.
/var/log/httpd/error has this
RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty
Which is interesting. There's no file /usr/share/ipa/smb.conf.empty but there is a /usr/share/ipa/smb.conf.template?
Ok, I think I've found the problem:
ipa-replica-conncheck -c -m <master> Failed to connect to port 7389 tcp on 10.126.18.73 PKI-CA: Directory Service port (7389): FAILED ERROR: Port check failed! Inaccessible port(s): 7389 (TCP)
On the master, pki-tomcatd is showing as OK, although nmap -sT -O localhost doesn't show 7389 open.
Where can I look next?
ipa -version VERSION: 4.5.0, API_VERSION: 2.228
cheers L.
------ "The antidote to apocalypticism is *apocalyptic civics*. Apocalyptic civics is the insistence that we cannot ignore the truth, nor should we panic about it. It is a shared consciousness that our institutions have failed and our ecosystem is collapsing, yet we are still here — and we are creative agents who can shape our destinies. Apocalyptic civics is the conviction that the only way out is through, and the only way through is together. "
*Greg Bloom* @greggish https://twitter.com/greggish/status/873177525903609857
Lachlan Musicman via FreeIPA-users wrote:
When I first installed our replica, it worked just fine - I could add a user and see it on the master server. And vice versa.
I recently went back to take a look and make sure everything was working
- and it's not.
ipactl status shows everything is ok. Munge is up. I can ssh hostname between machines.
When I look at the ID Views in the interface, I get an "IPA Error 903: InternalError".
See /var/log/httpd/error_log for details, there may be a python backtrace.
When I do an id <username> I get nosuch user.
https://docs.pagure.org/SSSD.sssd/users/troubleshooting.html
I did some googling. In /var log/dirsrv/domain/errors I found this:
[26/Oct/2017:12:31:23.454702287 +1100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/vmdr-linuxidm.unix.domain.com@UNIX.DOMAIN.COM mailto:vmdr-linuxidm.unix.domain.com@UNIX.DOMAIN.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
I can get `kinit admin` working fine. But there's something wrong. I don't know where to look exactly.
KRB5_TRACE=/dev/stdout kinit admin
See what KDC kinit is using. It should be using the local box because masters should point only to themselves.
/var/log/httpd/error has this
RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty
Which is interesting. There's no file /usr/share/ipa/smb.conf.empty but there is a /usr/share/ipa/smb.conf.template?
Probably need more context.
Ok, I think I've found the problem:
ipa-replica-conncheck -c -m <master> Failed to connect to port 7389 tcp on 10.126.18.73 PKI-CA: Directory Service port (7389): FAILED ERROR: Port check failed! Inaccessible port(s): 7389 (TCP)
On the master, pki-tomcatd is showing as OK, although nmap -sT -O localhost doesn't show 7389 open.
Where can I look next?
ipa -version VERSION: 4.5.0, API_VERSION: 2.228
It shouldn't be even trying port 7389 with v4.5.0. Very old versions of IPA used to use two separate 389-ds instances, one for the IPA data and one for the CA data. They were combined long ago. This could just be a check in case you had a very old master in which case this is a red herring.
rob
On 27 October 2017 at 07:38, Rob Crittenden rcritten@redhat.com wrote:
Lachlan Musicman via FreeIPA-users wrote:
When I look at the ID Views in the interface, I get an "IPA Error 903: InternalError".
See /var/log/httpd/error_log for details, there may be a python backtrace.
Sure do!
[Thu Oct 26 12:57:25.413102 2017] [:error] [pid 1316] ipa: ERROR: non-public: RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty [Thu Oct 26 12:57:25.413118 2017] [:error] [pid 1316] Traceback (most recent call last): [Thu Oct 26 12:57:25.413121 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 367, in wsgi_execute [Thu Oct 26 12:57:25.413124 2017] [:error] [pid 1316] result = command(*args, **options) [Thu Oct 26 12:57:25.413126 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 447, in __call__ [Thu Oct 26 12:57:25.413128 2017] [:error] [pid 1316] return self.__do_call(*args, **options) [Thu Oct 26 12:57:25.413130 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 475, in __do_call [Thu Oct 26 12:57:25.413133 2017] [:error] [pid 1316] ret = self.run(*args, **options) [Thu Oct 26 12:57:25.413135 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 797, in run [Thu Oct 26 12:57:25.413137 2017] [:error] [pid 1316] return self.execute(*args, **options) [Thu Oct 26 12:57:25.413139 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/baseldap.py", line 2050, in execute [Thu Oct 26 12:57:25.413141 2017] [:error] [pid 1316] truncated = callback(self, ldap, entries, truncated, *args, **options) [Thu Oct 26 12:57:25.413144 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 1123, in post_callback [Thu Oct 26 12:57:25.413146 2017] [:error] [pid 1316] ldap, entries, truncated, *args, **options) [Thu Oct 26 12:57:25.413148 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 829, in post_callback [Thu Oct 26 12:57:25.413151 2017] [:error] [pid 1316] self.obj.convert_anchor_to_human_readable_form(entry, **options) [Thu Oct 26 12:57:25.413153 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 733, in convert_anchor_to_human_readable_form [Thu Oct 26 12:57:25.413156 2017] [:error] [pid 1316] anchor [Thu Oct 26 12:57:25.413158 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 632, in resolve_anchor_to_object_name [Thu Oct 26 12:57:25.413161 2017] [:error] [pid 1316] name = domain_validator.get_trusted_domain_object_from_sid(sid) [Thu Oct 26 12:57:25.413163 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 503, in get_trusted_domain_object_from_sid [Thu Oct 26 12:57:25.413165 2017] [:error] [pid 1316] attrs=attrs) [Thu Oct 26 12:57:25.413167 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 380, in get_trusted_domain_objects [Thu Oct 26 12:57:25.413170 2017] [:error] [pid 1316] entries = self.search_in_dc(domain, filter, attrs, scope, basedn) [Thu Oct 26 12:57:25.413172 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 689, in search_in_dc [Thu Oct 26 12:57:25.413174 2017] [:error] [pid 1316] info = self.__retrieve_trusted_domain_gc_list(domain) [Thu Oct 26 12:57:25.413176 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 763, in __retrieve_trusted_domain_gc_list [Thu Oct 26 12:57:25.413179 2017] [:error] [pid 1316] os.path.join(paths.USR_SHARE_IPA_DIR, "smb.conf.empty")) [Thu Oct 26 12:57:25.413181 2017] [:error] [pid 1316] RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty
[26/Oct/2017:12:31:23.454702287 +1100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/vmdr-linuxidm.unix.domain.com@UNIX.DOMAIN.COM mailto:vmdr-linuxidm.unix.domain.com@UNIX.DOMAIN.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
I can get `kinit admin` working fine. But there's something wrong. I don't know where to look exactly.
KRB5_TRACE=/dev/stdout kinit admin
See what KDC kinit is using. It should be using the local box because masters should point only to themselves.
Yes, that command makes reference to it's own ip, eg: "Sending TCP request to stream 10.126.18.129:88"
/var/log/httpd/error has this
RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty
Which is interesting. There's no file /usr/share/ipa/smb.conf.empty but there is a /usr/share/ipa/smb.conf.template?
Probably need more context.
I've only just realised this is the above error - when I go to ID View->Default Trust View in the WebUI, I get the above python stacktrace, but I also get
[Fri Oct 27 10:03:43.466674 2017] [:warn] [pid 5686] [client 10.126.160.47:53715] failed to set perms (3140) on file (/var/run/ipa/ccaches/admin@UNIX.DOMAIN.COM)!, referer: https://vmdr-linuxidm.unix.domain.com/ipa/ui/
Ok, I think I've found the problem:
ipa-replica-conncheck -c -m <master> Failed to connect to port 7389 tcp on 10.126.18.73 PKI-CA: Directory Service port (7389): FAILED ERROR: Port check failed! Inaccessible port(s): 7389 (TCP)
On the master, pki-tomcatd is showing as OK, although nmap -sT -O localhost doesn't show 7389 open.
Where can I look next?
ipa -version VERSION: 4.5.0, API_VERSION: 2.228
It shouldn't be even trying port 7389 with v4.5.0. Very old versions of IPA used to use two separate 389-ds instances, one for the IPA data and one for the CA data. They were combined long ago. This could just be a check in case you had a very old master in which case this is a red herring.
Ok - I'll ignore then.
cheers L.
------ "The antidote to apocalypticism is *apocalyptic civics*. Apocalyptic civics is the insistence that we cannot ignore the truth, nor should we panic about it. It is a shared consciousness that our institutions have failed and our ecosystem is collapsing, yet we are still here — and we are creative agents who can shape our destinies. Apocalyptic civics is the conviction that the only way out is through, and the only way through is together. "
*Greg Bloom* @greggish https://twitter.com/greggish/status/873177525903609857
On 27 October 2017 at 10:32, Lachlan Musicman datakid@gmail.com wrote:
On 27 October 2017 at 07:38, Rob Crittenden rcritten@redhat.com wrote:
Lachlan Musicman via FreeIPA-users wrote:
When I look at the ID Views in the interface, I get an "IPA Error 903: InternalError".
See /var/log/httpd/error_log for details, there may be a python backtrace.
Sure do!
[Thu Oct 26 12:57:25.413102 2017] [:error] [pid 1316] ipa: ERROR: non-public: RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty [Thu Oct 26 12:57:25.413118 2017] [:error] [pid 1316] Traceback (most recent call last): [Thu Oct 26 12:57:25.413121 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 367, in wsgi_execute [Thu Oct 26 12:57:25.413124 2017] [:error] [pid 1316] result = command(*args, **options) [Thu Oct 26 12:57:25.413126 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 447, in __call__ [Thu Oct 26 12:57:25.413128 2017] [:error] [pid 1316] return self.__do_call(*args, **options) [Thu Oct 26 12:57:25.413130 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 475, in __do_call [Thu Oct 26 12:57:25.413133 2017] [:error] [pid 1316] ret = self.run(*args, **options) [Thu Oct 26 12:57:25.413135 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 797, in run [Thu Oct 26 12:57:25.413137 2017] [:error] [pid 1316] return self.execute(*args, **options) [Thu Oct 26 12:57:25.413139 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/baseldap.py", line 2050, in execute [Thu Oct 26 12:57:25.413141 2017] [:error] [pid 1316] truncated = callback(self, ldap, entries, truncated, *args, **options) [Thu Oct 26 12:57:25.413144 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 1123, in post_callback [Thu Oct 26 12:57:25.413146 2017] [:error] [pid 1316] ldap, entries, truncated, *args, **options) [Thu Oct 26 12:57:25.413148 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 829, in post_callback [Thu Oct 26 12:57:25.413151 2017] [:error] [pid 1316] self.obj.convert_anchor_to_human_readable_form(entry, **options) [Thu Oct 26 12:57:25.413153 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 733, in convert_anchor_to_human_readable_form [Thu Oct 26 12:57:25.413156 2017] [:error] [pid 1316] anchor [Thu Oct 26 12:57:25.413158 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/idviews.py", line 632, in resolve_anchor_to_object_name [Thu Oct 26 12:57:25.413161 2017] [:error] [pid 1316] name = domain_validator.get_trusted_domain_object_from_sid(sid) [Thu Oct 26 12:57:25.413163 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 503, in get_trusted_domain_object_from_sid [Thu Oct 26 12:57:25.413165 2017] [:error] [pid 1316] attrs=attrs) [Thu Oct 26 12:57:25.413167 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 380, in get_trusted_domain_objects [Thu Oct 26 12:57:25.413170 2017] [:error] [pid 1316] entries = self.search_in_dc(domain, filter, attrs, scope, basedn) [Thu Oct 26 12:57:25.413172 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 689, in search_in_dc [Thu Oct 26 12:57:25.413174 2017] [:error] [pid 1316] info = self.__retrieve_trusted_domain_gc_list(domain) [Thu Oct 26 12:57:25.413176 2017] [:error] [pid 1316] File "/usr/lib/python2.7/site-packages/ipaserver/dcerpc.py", line 763, in __retrieve_trusted_domain_gc_list [Thu Oct 26 12:57:25.413179 2017] [:error] [pid 1316] os.path.join(paths.USR_SHARE_IPA_DIR, "smb.conf.empty")) [Thu Oct 26 12:57:25.413181 2017] [:error] [pid 1316] RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty
[26/Oct/2017:12:31:23.454702287 +1100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/vmdr-linuxidm.unix.domain.com@UNIX.DOMAIN.COM mailto:vmdr-linuxidm.unix.domain.com@UNIX.DOMAIN.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
I can get `kinit admin` working fine. But there's something wrong. I don't know where to look exactly.
KRB5_TRACE=/dev/stdout kinit admin
See what KDC kinit is using. It should be using the local box because masters should point only to themselves.
Yes, that command makes reference to it's own ip, eg: "Sending TCP request to stream 10.126.18.129:88"
/var/log/httpd/error has this
RuntimeError: Unable to load file /usr/share/ipa/smb.conf.empty
Which is interesting. There's no file /usr/share/ipa/smb.conf.empty but there is a /usr/share/ipa/smb.conf.template?
Probably need more context.
I've only just realised this is the above error - when I go to ID View->Default Trust View in the WebUI, I get the above python stacktrace, but I also get
[Fri Oct 27 10:03:43.466674 2017] [:warn] [pid 5686] [client 10.126.160.47:53715] failed to set perms (3140) on file (/var/run/ipa/ccaches/admin@UNIX.DOMAIN.COM)!, referer: https://vmdr-linuxidm.unix.domain.com/ipa/ui/
I looked at the notes I made from the out put created when I set the replica up. It makes reference to https://vmdr-linuxidm.unix.domain.com/ipa/session/json. When I go to that page in a browser, I see:
{"result": null, "version": "4.5.0", "error": {"message": "Missing or invalid HTTP Referer, missing", "code": 911, "data": {"referer": "missing"}, "name": "RefererError"}, "id": null, "principal": "admin@UNIX.DOMAIN.COM"}
cheers L.
------ "The antidote to apocalypticism is *apocalyptic civics*. Apocalyptic civics is the insistence that we cannot ignore the truth, nor should we panic about it. It is a shared consciousness that our institutions have failed and our ecosystem is collapsing, yet we are still here — and we are creative agents who can shape our destinies. Apocalyptic civics is the conviction that the only way out is through, and the only way through is together. "
*Greg Bloom* @greggish https://twitter.com/greggish/status/873177525903609857
On 27 October 2017 at 07:38, Rob Crittenden rcritten@redhat.com wrote:
Lachlan Musicman via FreeIPA-users wrote:
ipa -version VERSION: 4.5.0, API_VERSION: 2.228
It shouldn't be even trying port 7389 with v4.5.0. Very old versions of IPA used to use two separate 389-ds instances, one for the IPA data and one for the CA data. They were combined long ago. This could just be a check in case you had a very old master in which case this is a red herring.
I went back to take another look at the dirsrv logs after you said this, and I saw something I didn't see yesterday. I notice that the cn has "meTo" appended to the start of the master server name.
Is that meant to be that way, or have I mistyped in the wrong window at the wrong time somewhere?
ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn= meTovmpr-linuxidm.unix.domain.com" (vmpr-linuxidm:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
Cheers L.
------ "The antidote to apocalypticism is *apocalyptic civics*. Apocalyptic civics is the insistence that we cannot ignore the truth, nor should we panic about it. It is a shared consciousness that our institutions have failed and our ecosystem is collapsing, yet we are still here — and we are creative agents who can shape our destinies. Apocalyptic civics is the conviction that the only way out is through, and the only way through is together. "
*Greg Bloom* @greggish https://twitter.com/greggish/status/873177525903609857
freeipa-users@lists.fedorahosted.org
-
Lachlan Musicman -
Rob Crittenden