Hi all,
I was referred to this place by Florence. I'm hoping to get some help in the right direction with this issue I've been having. I have a FreeIPA system that I inherited from a previous coworker with no install notes so I'm trying to figure out heads/tails out of this thing. From what I can tell its a 4 node deployment with all of them as CA servers and 1 of them was the CA master.
The issue is the LDAP and HTTP certs expired but from my knowledge they were supposed to auto-renew. I've tried auto-renewing the certs by rolling back time and restarting certmonger but it keeps returning this error:
--- status: CA_UNREACHABLE ca-error: Server at https://$hostname/ipa/xml failed request, will retry: 4002 (RPC failed at server. service with name “HTTP/$hostname@DOMAIN.COM” already exists). ... pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd ---
Similar with the LDAP cert:
ca-error: Server at https://$hostname/ipa/xml failed request, will retry: 4002 (RPC failed at server. service with name "HTTP/$hostname@DOMAIN.COM" already exists). ... pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM
I've also seen it appear on the other nodes before the certs expired so I'm guessing certmonger was trying to renew it but something snuck into LDAP ?
---
Some diagnostics:
[root@a01-n ~]# ipa --version VERSION: 4.4.0, API_VERSION: 2.213
[root@a01-n ~]# getcert list Number of certificates and requests being tracked: 8. Request ID '20170315010441': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=CA Audit,O=DOMAIN.COM expires: 2021-01-25 01:52:48 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010442': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=OCSP Subsystem,O=DOMAIN.COM expires: 2021-01-25 01:53:28 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010443': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=CA Subsystem,O=DOMAIN.COM expires: 2021-01-25 01:53:48 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010444': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=Certificate Authority,O=DOMAIN.COM expires: 2039-03-15 01:45:42 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010445': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=IPA RA,O=DOMAIN.COM expires: 2021-01-25 01:53:18 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20170315010446': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2021-01-25 01:52:44 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20190203000836': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2019-03-16 01:05:25 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190329001401': status: CA_UNREACHABLE ca-error: Server at https://a01-n.fqdn/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.). stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DOMAIN-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2019-03-16 01:05:03 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM track: yes auto-renew: yes
---------
[root@a01-n ~]# ipa config-show ... IPA masters: a01-e.fqdn, a01-n.fqdn, a02-e.fqdn, a02-n.fqdn IPA CA servers: a01-e.fqdn, a01-n.fqdn, a02-e.fqdn, a02-n.fqdn IPA NTP servers: a01-e.fqdn, a01-n.fqdn, a02-e.fqdn, a02-n.fqdn IPA CA renewal master: a01-n.fqdn
----------
Any help would be greatly appreciated.
John Aquino via FreeIPA-users wrote:
Hi all,
I was referred to this place by Florence. I'm hoping to get some help in the right direction with this issue I've been having. I have a FreeIPA system that I inherited from a previous coworker with no install notes so I'm trying to figure out heads/tails out of this thing. From what I can tell its a 4 node deployment with all of them as CA servers and 1 of them was the CA master.
The issue is the LDAP and HTTP certs expired but from my knowledge they were supposed to auto-renew. I've tried auto-renewing the certs by rolling back time and restarting certmonger but it keeps returning this error:
status: CA_UNREACHABLE ca-error: Server at https://$hostname/ipa/xml failed request, will retry: 4002 (RPC failed at server. service with name “HTTP/$hostname@DOMAIN.COM” already exists). ... pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd
Similar with the LDAP cert:
ca-error: Server at https://$hostname/ipa/xml failed request, will retry: 4002 (RPC failed at server. service with name "HTTP/$hostname@DOMAIN.COM" already exists). ... pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM
I've also seen it appear on the other nodes before the certs expired so I'm guessing certmonger was trying to renew it but something snuck into LDAP ?
What date are you going back to?
Look in /var/log/httpd/error_log around the time of the attempt.
journalctl -u certmonger to see if that is logging anything.
rob
Some diagnostics:
[root@a01-n ~]# ipa --version VERSION: 4.4.0, API_VERSION: 2.213
[root@a01-n ~]# getcert list Number of certificates and requests being tracked: 8. Request ID '20170315010441': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=CA Audit,O=DOMAIN.COM expires: 2021-01-25 01:52:48 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010442': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=OCSP Subsystem,O=DOMAIN.COM expires: 2021-01-25 01:53:28 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010443': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=CA Subsystem,O=DOMAIN.COM expires: 2021-01-25 01:53:48 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010444': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=Certificate Authority,O=DOMAIN.COM expires: 2039-03-15 01:45:42 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170315010445': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=IPA RA,O=DOMAIN.COM expires: 2021-01-25 01:53:18 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20170315010446': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2021-01-25 01:52:44 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20190203000836': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2019-03-16 01:05:25 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190329001401': status: CA_UNREACHABLE ca-error: Server at https://a01-n.fqdn/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.). stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DOMAIN-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2019-03-16 01:05:03 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM track: yes auto-renew: yes
[root@a01-n ~]# ipa config-show ... IPA masters: a01-e.fqdn, a01-n.fqdn, a02-e.fqdn, a02-n.fqdn IPA CA servers: a01-e.fqdn, a01-n.fqdn, a02-e.fqdn, a02-n.fqdn IPA NTP servers: a01-e.fqdn, a01-n.fqdn, a02-e.fqdn, a02-n.fqdn IPA CA renewal master: a01-n.fqdn
Any help would be greatly appreciated. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi John,
Looks like the Certmonger tracking requests are missing the principal name. So here's the first thing to try: wind back the clock again, restart IPA, and then issue the following certmonger commands:
- getcert resubmit -i 20190203000836 -K "HTTP/<domain>@<realm>" - getcert resubmit -i 20190329001401 -K "ldap/<domain>@<realm>"
With the replacement of <domain> and <realm> as appropriate.
If that does not resolve the issue, could you please supply:
- exact version of FreeIPA packages
- /var/log/httpd/error snippet at the time the error occurred: "service with name "HTTP/$hostname@DOMAIN.COM" already exists"
- output of `ipa service-show HTTP/<hostname>@<realm> --all --raw` and `ipa service-show ldap/<hostname>@<realm> --all --raw`
And preferably do not "sanitise" those outputs (you can send to me off-list if you prefer).
Cheers, Fraser
On Thu, Apr 11, 2019 at 11:28:40PM -0000, John Aquino via FreeIPA-users wrote:
Hi all,
I was referred to this place by Florence. I'm hoping to get some help in the right direction with this issue I've been having. I have a FreeIPA system that I inherited from a previous coworker with no install notes so I'm trying to figure out heads/tails out of this thing. From what I can tell its a 4 node deployment with all of them as CA servers and 1 of them was the CA master.
The issue is the LDAP and HTTP certs expired but from my knowledge they were supposed to auto-renew. I've tried auto-renewing the certs by rolling back time and restarting certmonger but it keeps returning this error:
status: CA_UNREACHABLE ca-error: Server at https://$hostname/ipa/xml failed request, will retry: 4002 (RPC failed at server. service with name “HTTP/$hostname@DOMAIN.COM” already exists). ... pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd
Similar with the LDAP cert:
ca-error: Server at https://$hostname/ipa/xml failed request, will retry: 4002 (RPC failed at server. service with name "HTTP/$hostname@DOMAIN.COM" already exists). ... pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM
[root@a01-n ~]# getcert list Request ID '20190203000836': status: SUBMITTING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2019-03-16 01:05:25 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190329001401': status: CA_UNREACHABLE ca-error: Server at https://a01-n.fqdn/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining: Peer's Certificate has expired.). stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DOMAIN-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOMAIN.COM subject: CN=a01-n.fqdn,O=DOMAIN.COM expires: 2019-03-16 01:05:03 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM track: yes auto-renew: yes
freeipa-users@lists.fedorahosted.org
-
Fraser Tweedale -
John Aquino -
Rob Crittenden