Hi all,
We run IPA 3.0.0 and have a cert on the CA master expiring in about 10 days. The problem is that we mistakenly provisioned the last cert using an old hostname which means that automatically renewing the cert fails, and the IPA cert checks we run fails with...
ca-error: Server at "http://correct.hostname:9180/ca/ee/ca/profileSubmit" replied: 1: Server Internal Error.
I also get a java NPE error when curling that endpoint.
Is it possible to zero out the existing cert and resubmit it with the correct hostname? This is a production environment supporting several thousand hosts which means I want to test whatever solution I come up with. We have a few staging environments but they're all configured correctly, so I'm wondering if we can intentionally put one into a similar bad state and revert it.
Happy to provide clarifying information if I'm not making sense here.
thx,
Scott Stevson via FreeIPA-users wrote:
Hi all,
We run IPA 3.0.0 and have a cert on the CA master expiring in about 10 days. The problem is that we mistakenly provisioned the last cert using an old hostname which means that automatically renewing the cert fails, and the IPA cert checks we run fails with...
ca-error: Server at "http://correct.hostname:9180/ca/ee/ca/profileSubmit" replied: 1: Server Internal Error.
I also get a java NPE error when curling that endpoint.
Is it possible to zero out the existing cert and resubmit it with the correct hostname? This is a production environment supporting several thousand hosts which means I want to test whatever solution I come up with. We have a few staging environments but they're all configured correctly, so I'm wondering if we can intentionally put one into a similar bad state and revert it.
Happy to provide clarifying information if I'm not making sense here.
Yeah, more details are needed. What cert is provisioned with an old hostname and how did someone manage to do that?
What does the CA debug log say when it is failing?
rob
Hey Rob,
It's the NSSDB cert. Here's some console output that might be helpful.
PROD [root@server-ns-1 var]# getcert list | grep -A10 20150827000358 Request ID '20150827000358': status: MONITORING ca-error: Server at "http://server-ns-1.our.domain.local:9180/ca/ee/ca/profileSubmit" replied: 1: Server Internal Error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=COMPANY.LOCAL subject: CN=IPA RA,O=COMPANY.LOCAL expires: 2017-08-15 20:17:52 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
As for how this happened: We're not entirely sure yet but the working theory is the SRE who provisioned the new CA master didn't fully remove all references to the old one. I need to track down a few more people and details in order to answer this fully.
As for the CA debug log. I don't see any references to failures per se, and I'm wondering if there's something I can do to force an error that'll be useful to you. Again, the failure I referenced earlier is a our Nagios check for certmonger that simply reacts to the output `getcert list` returns. Versions of this are all I see in the debug logs.
[08/Aug/2017:15:39:31][TP-Processor9]: CMSServlet: curDate=Tue Aug 08 15:39:31 UTC 2017 id=caProfileSubmitSSLClient time=62
Scott Stevson via FreeIPA-users wrote:
Hey Rob,
It's the NSSDB cert. Here's some console output that might be helpful.
PROD [root@server-ns-1 var]# getcert list | grep -A10 20150827000358 Request ID '20150827000358': status: MONITORING ca-error: Server at "http://server-ns-1.our.domain.local:9180/ca/ee/ca/profileSubmit" replied: 1: Server Internal Error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=COMPANY.LOCAL subject: CN=IPA RA,O=COMPANY.LOCAL expires: 2017-08-15 20:17:52 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
As for how this happened: We're not entirely sure yet but the working theory is the SRE who provisioned the new CA master didn't fully remove all references to the old one. I need to track down a few more people and details in order to answer this fully.
As for the CA debug log. I don't see any references to failures per se, and I'm wondering if there's something I can do to force an error that'll be useful to you. Again, the failure I referenced earlier is a our Nagios check for certmonger that simply reacts to the output `getcert list` returns. Versions of this are all I see in the debug logs.
[08/Aug/2017:15:39:31][TP-Processor9]: CMSServlet: curDate=Tue Aug 08 15:39:31 UTC 2017 id=caProfileSubmitSSLClient time=62
certmonger doesn't use SRV records to lookup an IPA master. Update the xmlrpc_server entry in /etc/ipa/default.conf to point to a working IPA server and that should fix this for you after a certmonger restart.
There is a bug open on this we just haven't gotten to it yet.
rob
Thanks, Rob.
Unfortunately my test in staging resulted in an expired dogtag cert. The staging environment didn't have any certificates that were due to expire soon so I updated the xmlrpc_server variable on one of the four IPA hosts we have to another one in the same AWS region and restarted certmonger. I then resubmitted the cert request for one of the ID's I have and suddenly a cert that was due to expire later this year is now expired as of 2016.
STAGING Request ID '20170124164909': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=COMPANY.LOCAL subject: CN=IPA RA,O=COMPANY.LOCAL expires: 2016-12-07 03:35:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes
And for completeness, I'm pasting the output of getcert list on prod so you can see the cert that's due to expire in its entirety. PROD Request ID '20150827000358': status: MONITORING ca-error: Server at "http://server-ns-1.our.domain.local:9180/ca/ee/ca/profileSubmit" replied: 1: Server Internal Error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=COMPANY.LOCAL subject: CN=IPA RA,O=COMPANY.LOCAL expires: 2017-08-15 20:17:52 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
Apologies if this is painful. We appreciate the back and forth here.
Scott Stevson via FreeIPA-users wrote:
Thanks, Rob.
Unfortunately my test in staging resulted in an expired dogtag cert. The staging environment didn't have any certificates that were due to expire soon so I updated the xmlrpc_server variable on one of the four IPA hosts we have to another one in the same AWS region and restarted certmonger. I then resubmitted the cert request for one of the ID's I have and suddenly a cert that was due to expire later this year is now expired as of 2016.
STAGING Request ID '20170124164909': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=COMPANY.LOCAL subject: CN=IPA RA,O=COMPANY.LOCAL expires: 2016-12-07 03:35:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes
For some things in IPA, like the Highlander, there can be only one.
An example of this is the CA responsible for renewing the CA subsystem certificates (OCSP, the RA cert, etc).
During the renewal the updated certificates are stored in LDAP which is replicated. The non-renewing masters monitor that location and fetch updated certs from there.
So I'm guessing that the old cert is in LDAP and that got pulled down for some reason, why I have no idea as certmonger should know better. You can check to see what's in there by:
$ ldapsearch -Y GSSAPI -b 'cn=ca_renewal,cn=ipa,cn=etc,dc=example,dc=com
This will show the current ipaCert in more detail:
# certutil -L -d /etc/httpd/alias -n ipaCert
You can compare this to the output of the various other staging masters.
And for completeness, I'm pasting the output of getcert list on prod so you can see the cert that's due to expire in its entirety. PROD Request ID '20150827000358': status: MONITORING ca-error: Server at "http://server-ns-1.our.domain.local:9180/ca/ee/ca/profileSubmit" replied: 1: Server Internal Error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=COMPANY.LOCAL subject: CN=IPA RA,O=COMPANY.LOCAL expires: 2017-08-15 20:17:52 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
I still think updating the xmlrpc_server is the way forward. I can't explain the mix-up.
ipa config-show should show you the currently configured renewal master.
rob
Hey Rob,
You may recall earlier when I said that we wound up pulling an expired cert on one of our staging IPA replicas after updating the xmlrpc_server variable to point to a different host. It's not clear to us how best to fix that cert (although I suppose we could roll back time on the box), so we're wondering if we can update the certificate using openssl and then adding the entry using something like this:
certutil -A -d /etc/httpd/alias -n 'ipaCert' -t u,u,u -a -i /root/renew/new_ipaCert.crt
Thoughts? We don't need to go this route but we're gaming out recovery/alternate solutions in the event our efforts to fix prod fail.
I'm on IRC now if responses there would be faster or easier for you.
thx,
Scott Stevson via FreeIPA-users wrote:
Hey Rob,
You may recall earlier when I said that we wound up pulling an expired cert on one of our staging IPA replicas after updating the xmlrpc_server variable to point to a different host. It's not clear to us how best to fix that cert (although I suppose we could roll back time on the box), so we're wondering if we can update the certificate using openssl and then adding the entry using something like this:
certutil -A -d /etc/httpd/alias -n 'ipaCert' -t u,u,u -a -i /root/renew/new_ipaCert.crt
Thoughts? We don't need to go this route but we're gaming out recovery/alternate solutions in the event our efforts to fix prod fail.
I'm on IRC now if responses there would be faster or easier for you.
I was with you until you mentioned openssl. The current cert should already exist on the current IPA CA renewal master. You can export the cert from there with:
certutil -L -d /etc/httpd/alias -n ipaCert -a > /path/to/somewhere
Then use the certutil command you mentioned to import it.
Once imported restart httpd and I'd confirm that the master can talk to the CA by running:
ipa cert-show 1
The actual contents of the cert don't matter but this will show that end-to-end connectivity is there and that the master has the right RA cert.
rob
Yeah, I was referring to the instructions in https://www.freeipa.org/page/Certmonger#Manually_renew_a_certificate which discuss manual renewal of a certificate which is interesting to us since the all the nodes in the IPA cluster on prod have the same cert that's expiring on Tuesday.
For what it's worth, we were able to recover the staging IPA node that we managed to put into a bad state a few days ago. The export / import instructions you pasted above worked.
Hey Rob,
I have an update that'll close out this thread.
We discovered that the code in the pki-ca was looking for a CN of the IPA RA's serial number in ou=certificateRepository,ou=ca,o=ipaca. This didn't exist and we realized it might be part of the problem. It turns out that it was which helps explain the NPE we saw in the original error.
We ultimately had to create a local ldif for the current IPA RA certificate in production, add the new cn entry to "ou=certificateRepository,ou=ca,o=ipaca", and attempt a resubmit operation. We had a little trouble deciphering some of the metaInfo, specifically the "requestId" as ou=ca,ou=requests,o=ipaca was also missing a request entry for our IPA RA certificate. After testing in staging, we felt comfortable pushing to production pointing at the previous certificates ou=ca,ou=requests,o=ipaca entry. The resubmit worked late last week.
Thanks for your help.
Scott
freeipa-users@lists.fedorahosted.org