On Fri, 26 May 2017, Rob Crittenden wrote:
Rob Foehl via FreeIPA-users wrote:
On Fri, 26 May 2017, Fraser Tweedale wrote:
What is the validity of the leaf certificates? Is the notAfter time of the leaf certificate pegged to the notAfter time of the CA certificate? If so, this is (IMO) a bug.
The leaf certs' expiration is pegged to that of the CA cert that was used to issue them -- the old one, in this case -- but that is expected behavior for any CA. It wouldn't be semantically valid otherwise, and there's no guarantee that the CA cert will actually be renewed without changing the key.
The odd behavior here is that certmonger woke up, noticed that every IPA cert including the externally-signed IPA CA needed to be renewed, and immediately caused the CA to renew them all. The IPA CA cert itself yielded a log entry like this:
May 25 00:25:21 ipa.example.com dogtag-ipa-ca-renew-agent-submit[868]: Certificate with subject 'CN=Certificate Authority,O=EXAMPLE.COM' is about to expire, use ipa-cacert-manage to renew it
The other 7 or so IPA-generated certificates (host, RA, OCSP, etc.) were renewed using the existing CA cert, with new validity periods tied to that cert. As mentioned, certmonger would likely figure this out and renew them all again using the since-replaced CA cert within the ~2 week period until they all expire again, but this seems like unexpected behavior when the IPA CA cert is signed by an external CA and can't be auto-renewed.
(Actually, based on the order the renewals were submitted, this seems like it'd be an issue even if the CA cert were automatically renewed -- it wasn't the first one to be submitted, either. Incidentally, the certs which were renewed aren't a complete list -- both the "CN=ipa-ca-agent" and "CN=Object Signing Cert" certs weren't renewed and aren't tracked by certmonger.)
certmonger doesn't have the context to know internal vs external. It just knows a cert is expiring within its window so it renews it. IMHO this is completely expected.
Right, I wouldn't expect it to know the provenance of the CA cert... I am wondering whether it should be able to recognize the dependency between the certs, though -- it should be able to recognize the chain, at least.
I believe that certmonger will renew it again as the final day approaches.
So, out of curiosity, I left the VM running through the original CA expiration date to see what would happen. The results aren't pretty:
- the running httpd kept using the old certificate (and CA chain), which broke https sessions to the UI/API (as might be expected);
- certmonger thinks it's renewed everything, with the new expiration dates lined up with that of the replaced external CA;
- none of the services recognize that they have new certs installed, for example the same httpd issue as seen in another thread:
[Fri Jun 09 01:07:37.413789 2017] [:error] [pid 14616] SSL Library Error: -8162 The certificate issuer's certificate has expired. Check your system date and time [Fri Jun 09 01:07:37.413828 2017] [:error] [pid 14616] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved.
vs.
# getcert list -d /etc/httpd/alias -n Server-Cert Number of certificates and requests being tracked: 8. Request ID '20170508063315': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=ipa1.example.com,O=EXAMPLE.COM expires: 2017-06-24 04:32:24 UTC dns: ipa1.example.com principal name: HTTP/ipa1.example.com@EXAMPLE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes
- neither httpd nor pki-tomcatd will (re)start as a result, the former fails and the latter just spews stack traces into the logs every minute if forcibly started (even if httpd is band-aided first):
Jun 09 01:14:24 ipa1.example.com server[15236]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@43523130 background process Jun 09 01:14:24 ipa1.example.com server[15236]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Jun 09 01:14:24 ipa1.example.com server[15236]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:130) Jun 09 01:14:24 ipa1.example.com server[15236]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154) Jun 09 01:14:24 ipa1.example.com server[15236]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5707) Jun 09 01:14:24 ipa1.example.com server[15236]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377) Jun 09 01:14:24 ipa1.example.com server[15236]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381) Jun 09 01:14:24 ipa1.example.com server[15236]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381) Jun 09 01:14:24 ipa1.example.com server[15236]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349) Jun 09 01:14:24 ipa1.example.com server[15236]: at java.lang.Thread.run(Thread.java:748)
- certmonger never actually logged anything about replacing these certs a second time, and the first round were signed with the now-expired CA cert instead of the new one;
- the UI is only partially functional, after clicking through the certificate warnings in the browser;
- pki-tomcatd is non-functional even if forcibly started, leading to repeated "IPA Error 4301: CertificateOperationError" in the UI when trying to access anything CA-related;
- possibly other issues I haven't discovered yet.
In short, that didn't go particularly well at all, which in some ways brings me back to the original as-yet-unanswered deployment question:
Is trying to do this with an external CA worth the pain?
If I go that route, at some point I'm going to have to replace the CA cert -- and between this test VM and the "phase 2" mentioned in https://www.freeipa.org/page/V4/Distribution_of_CA_certificates_to_clients and the linked Pagure issue 4322, I have basically zero confidence in any of this...
-Rob