DNS and kerberos seem to be working fine (and have been for a long while). All `ipa` commands fail:
``` # kinit admin Password for admin@$REALM:
# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
# ipa help topics ipa: ERROR: cannot connect to 'any of the configured servers': https://$MASTER/ipa/json, https://$REPLICA/ipa/json ```
(yes, the firewall is open)
Attempting to login via the WebUI with user/pass, says `Authenticating...`, then prints red text: An unknown error occurred. (or something to that effect).
The apache error log shows: ``` [Tue Nov 06 07:46:46.388297 2018] [:error] [pid 23816] ipa: INFO: *** PROCESS START *** [Tue Nov 06 07:46:46.862410 2018] [:error] [pid 23815] ipa: INFO: *** PROCESS START *** [Tue Nov 06 07:48:55.510961 2018] [:error] [pid 23816] ipa: ERROR: 500 Internal Server Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not defined in HTTP request environment [Tue Nov 06 07:48:55.512943 2018] [:error] [pid 23816] [remote $MASTER_IP:52342] mod_wsgi (pid=23816): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Tue Nov 06 07:48:55.513207 2018] [:error] [pid 23816] [remote $MASTER_IP:52342] RuntimeError: response has not been started [Tue Nov 06 17:09:21.111120 2018] [:error] [pid 23815] ipa: ERROR: 500 Internal Server Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not defined in HTTP request environment [Tue Nov 06 17:09:21.113133 2018] [:error] [pid 23815] [remote $MASTER_IP:52342] mod_wsgi (pid=23815): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Tue Nov 06 17:09:21.113410 2018] [:error] [pid 23815] [remote $MASTER_IP:52342] RuntimeError: response has not been started [Tue Nov 06 17:17:28.498098 2018] [auth_gssapi:error] [pid 23819] [client $CLIENT:36060] NO AUTH DATA Client did not send any authentication headers, referer: https://$MASTER/ipa/ui/ [Tue Nov 06 17:17:28.522306 2018] [auth_gssapi:error] [pid 23819] [client $CLIENT:36060] NO AUTH DATA Client did not send any authentication headers, referer: https://$MASTER/ipa/ui/ [Tue Nov 06 17:17:35.408453 2018] [:error] [pid 23815] [remote $CLIENT:24687] mod_wsgi (pid=23815): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Tue Nov 06 17:17:35.408776 2018] [:error] [pid 23815] [remote $CLIENT:24687] Traceback (most recent call last): [Tue Nov 06 17:17:35.408944 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/share/ipa/wsgi.py", line 51, in application [Tue Nov 06 17:17:35.409572 2018] [:error] [pid 23815] [remote $CLIENT:24687] return api.Backend.wsgi_dispatch(environ, start_response) [Tue Nov 06 17:17:35.409666 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 262, in __call__ [Tue Nov 06 17:17:35.471519 2018] [:error] [pid 23815] [remote $CLIENT:24687] return self.route(environ, start_response) [Tue Nov 06 17:17:35.471701 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route [Tue Nov 06 17:17:35.471923 2018] [:error] [pid 23815] [remote $CLIENT:24687] return app(environ, start_response) [Tue Nov 06 17:17:35.472027 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 929, in __call__ [Tue Nov 06 17:17:35.472163 2018] [:error] [pid 23815] [remote $CLIENT:24687] self.kinit(user_principal, password, ipa_ccache_name) [Tue Nov 06 17:17:35.472244 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 965, in kinit [Tue Nov 06 17:17:35.472378 2018] [:error] [pid 23815] [remote $CLIENT:24687] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Tue Nov 06 17:17:35.472461 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 125, in kinit_armor [Tue Nov 06 17:17:35.474208 2018] [:error] [pid 23815] [remote $CLIENT:24687] run(args, env=env, raiseonerr=True, capture_error=True) [Tue Nov 06 17:17:35.474308 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 512, in run [Tue Nov 06 17:17:35.480086 2018] [:error] [pid 23815] [remote $CLIENT:24687] raise CalledProcessError(p.returncode, arg_string, str(output)) [Tue Nov 06 17:17:35.480364 2018] [:error] [pid 23815] [remote $CLIENT:24687] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_23815 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 ```
I'm not above trying to troubleshoot this a little, but honestly it's probably faster to reinstall both master and replica. The problem isn't a bug, it was most certainly my blundering.
Being able to recover the 20-30 DNS entries (somehow) would be super nice. If I could recover the 5-10 host-details, even better. I don't care too much about my three users, they can just be told to re-enter their passwords :D
In case it's important, this is Centos 7, 32-bit, running on a Raspberry Pi 3. I had to use the Oracle Java, and hand-edit a pki-related-file.py (somewhere) to tweak a startup timeout. Otherwise it was working brilliantly for a long time, until I screwed it up.