Hey all,
About a year ago I did a really, really stupid thing. I updated IPA on one CentOS 7 host, then before being really sure things were working, I did the replica. Turned out the first upgrade only 'mostly' worked[*], meaning both hosts are now partially wrecked :S
The good news is, DNS and PKI seems mostly in-tact and functional (why I haven't done anything for a year). The bad news is, the web interface and API-access (ipa cmdline) is non-functional. Meaning I have no way to maintain the setup, add new replicas/hosts, etc. :(
Both kerberos and ldapsearch are working, so I'm wondering if there's a way I can "save" my DNS and user/group/kerberos records, to make a re-build/re-install less painful? I don't have anything worth saving PKI-wise.
Thoughts?
[*] The damage was caused by running out of disk-space after the package install, while the upgrade or schema-update script was running. I'm not above trying to repair the API, but so far my attempts have all been fruitless. I tried 'yum reinstall' and manually running the upgrade scripts. The damage seems to be inside the databases, since restoring from backup also restores API-breakage.
Chris Evich via FreeIPA-users wrote:
Hey all,
About a year ago I did a really, really stupid thing. I updated IPA on one CentOS 7 host, then before being really sure things were working, I did the replica. Turned out the first upgrade only 'mostly' worked[*], meaning both hosts are now partially wrecked :S
The good news is, DNS and PKI seems mostly in-tact and functional (why I haven't done anything for a year). The bad news is, the web interface and API-access (ipa cmdline) is non-functional. Meaning I have no way to maintain the setup, add new replicas/hosts, etc. :(
Both kerberos and ldapsearch are working, so I'm wondering if there's a way I can "save" my DNS and user/group/kerberos records, to make a re-build/re-install less painful? I don't have anything worth saving PKI-wise.
Thoughts?
[*] The damage was caused by running out of disk-space after the package install, while the upgrade or schema-update script was running. I'm not above trying to repair the API, but so far my attempts have all been fruitless. I tried 'yum reinstall' and manually running the upgrade scripts. The damage seems to be inside the databases, since restoring from backup also restores API-breakage.
We need more information on why your definition of wrecked is. What isn't working? What logs can you provide?
rob
DNS and kerberos seem to be working fine (and have been for a long while). All `ipa` commands fail:
``` # kinit admin Password for admin@$REALM:
# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
# ipa help topics ipa: ERROR: cannot connect to 'any of the configured servers': https://$MASTER/ipa/json, https://$REPLICA/ipa/json ```
(yes, the firewall is open)
Attempting to login via the WebUI with user/pass, says `Authenticating...`, then prints red text: An unknown error occurred. (or something to that effect).
The apache error log shows: ``` [Tue Nov 06 07:46:46.388297 2018] [:error] [pid 23816] ipa: INFO: *** PROCESS START *** [Tue Nov 06 07:46:46.862410 2018] [:error] [pid 23815] ipa: INFO: *** PROCESS START *** [Tue Nov 06 07:48:55.510961 2018] [:error] [pid 23816] ipa: ERROR: 500 Internal Server Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not defined in HTTP request environment [Tue Nov 06 07:48:55.512943 2018] [:error] [pid 23816] [remote $MASTER_IP:52342] mod_wsgi (pid=23816): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Tue Nov 06 07:48:55.513207 2018] [:error] [pid 23816] [remote $MASTER_IP:52342] RuntimeError: response has not been started [Tue Nov 06 17:09:21.111120 2018] [:error] [pid 23815] ipa: ERROR: 500 Internal Server Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not defined in HTTP request environment [Tue Nov 06 17:09:21.113133 2018] [:error] [pid 23815] [remote $MASTER_IP:52342] mod_wsgi (pid=23815): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Tue Nov 06 17:09:21.113410 2018] [:error] [pid 23815] [remote $MASTER_IP:52342] RuntimeError: response has not been started [Tue Nov 06 17:17:28.498098 2018] [auth_gssapi:error] [pid 23819] [client $CLIENT:36060] NO AUTH DATA Client did not send any authentication headers, referer: https://$MASTER/ipa/ui/ [Tue Nov 06 17:17:28.522306 2018] [auth_gssapi:error] [pid 23819] [client $CLIENT:36060] NO AUTH DATA Client did not send any authentication headers, referer: https://$MASTER/ipa/ui/ [Tue Nov 06 17:17:35.408453 2018] [:error] [pid 23815] [remote $CLIENT:24687] mod_wsgi (pid=23815): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Tue Nov 06 17:17:35.408776 2018] [:error] [pid 23815] [remote $CLIENT:24687] Traceback (most recent call last): [Tue Nov 06 17:17:35.408944 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/share/ipa/wsgi.py", line 51, in application [Tue Nov 06 17:17:35.409572 2018] [:error] [pid 23815] [remote $CLIENT:24687] return api.Backend.wsgi_dispatch(environ, start_response) [Tue Nov 06 17:17:35.409666 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 262, in __call__ [Tue Nov 06 17:17:35.471519 2018] [:error] [pid 23815] [remote $CLIENT:24687] return self.route(environ, start_response) [Tue Nov 06 17:17:35.471701 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route [Tue Nov 06 17:17:35.471923 2018] [:error] [pid 23815] [remote $CLIENT:24687] return app(environ, start_response) [Tue Nov 06 17:17:35.472027 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 929, in __call__ [Tue Nov 06 17:17:35.472163 2018] [:error] [pid 23815] [remote $CLIENT:24687] self.kinit(user_principal, password, ipa_ccache_name) [Tue Nov 06 17:17:35.472244 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 965, in kinit [Tue Nov 06 17:17:35.472378 2018] [:error] [pid 23815] [remote $CLIENT:24687] pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM], [Tue Nov 06 17:17:35.472461 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 125, in kinit_armor [Tue Nov 06 17:17:35.474208 2018] [:error] [pid 23815] [remote $CLIENT:24687] run(args, env=env, raiseonerr=True, capture_error=True) [Tue Nov 06 17:17:35.474308 2018] [:error] [pid 23815] [remote $CLIENT:24687] File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 512, in run [Tue Nov 06 17:17:35.480086 2018] [:error] [pid 23815] [remote $CLIENT:24687] raise CalledProcessError(p.returncode, arg_string, str(output)) [Tue Nov 06 17:17:35.480364 2018] [:error] [pid 23815] [remote $CLIENT:24687] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_23815 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 ```
I'm not above trying to troubleshoot this a little, but honestly it's probably faster to reinstall both master and replica. The problem isn't a bug, it was most certainly my blundering.
Being able to recover the 20-30 DNS entries (somehow) would be super nice. If I could recover the 5-10 host-details, even better. I don't care too much about my three users, they can just be told to re-enter their passwords :D
In case it's important, this is Centos 7, 32-bit, running on a Raspberry Pi 3. I had to use the Oracle Java, and hand-edit a pki-related-file.py (somewhere) to tweak a startup timeout. Otherwise it was working brilliantly for a long time, until I screwed it up.
...uggg, crap, tried replying twice and hyperkitty seems to just eat all my text...
...oh, it says "Your reply has been sent, and is being processed'...maybe that means it will eventually show up. I guess I'll wait :S
freeipa-users@lists.fedorahosted.org
-
Chris Evich -
Rob Crittenden