On 10/19/18 6:49 AM, Z D via FreeIPA-users wrote:
Hi there,
This is el7.3 running ipa-server 4.4.0 release 12.0.1.el7.
After reboot I couldn't start ipa service via systemctl, hence I run "ipactl start --ignore-service-failures" and this was kind of successful. I still have some discrepancies, and looking for troubleshooting ideas.
- "systemctl status ipa.service" reads that service failed
- "systemctl status pki-tomcatd.target" reads that PKI Tomcat Server is running.
Hi,
The PKI service status can be found using "systemctl status pki-tomcatd@pki-tomcat.service". More details on the differences between targets and units can be found in the man pages for systemd.unit(5) and systemd.target(5).
- # ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING ipa_memcached Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED <---- !! ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
To troubleshoot, you can have a look at the output of # systemctl status pki-tomcatd@pki-tomcat.service and the logs in /var/log/pki/pki-tomcat/ca/debug.
I would start by checking if some certificates expired with getcert list (check the status, should be MONITORING, and the expires: <date>).
HTH, flo
Well, why pki-tomcatd reads 'stopped' and how to make systemctl to recognize that ipa service is running, thanks in advance,
Zarko
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Thanks Flo. [1] Service pki-tomcatd@pki-tomcat.service is active (running) [2] /var/log/pki/pki-tomcat/ca/debug reads among others:
- SSL handshake happened - Could not connect to LDAP server host ca-ldap03.us.domain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48) - Internal Database Error encountered: Could not connect to LDAP server host ca-ldap03.us.domain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
I believe 48 is LDAP_INAPPROPRIATE_AUTH
[3] /var/log/ipa/ipactl.log reads:
-Failed to check CA status: cannot connect to 'http://ca-ldap03.us.domain.com:8080 /ca/admin/ca/getStatus': [Errno 111] Connection refused -Waiting until the CA is running -request POST http://ca-ldap03.us.domain.com:8080/ca/admin/ca/getStatus -request body '' -The CA status is: check interrupted due to error: cannot connect to 'http://ca-ld ap03.us.domain.com:8080/ca/admin/ca/getStatus': [Errno 111] Connection refused - Waiting for CA to start... - request POST http://ca-ldap03.us.domain.com:8080/ca/admin/ca/getStatus -request body '' - The CA status is: check interrupted due to error: cannot connect to 'http://ca-ld ap03.us.domain.com:8080/ca/admin/ca/getStatus': [Errno 111] Connection refused
[4] list of certificates
# getcert list | egrep -e status -e expire -e certificate Number of certificates and requests being tracked: 6. status: CA_WORKING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2018-08-14 20:49:38 UTC status: CA_WORKING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2018-08-14 20:49:35 UTC status: CA_WORKING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' expires: 2018-08-14 20:49:36 UTC status: MONITORING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2036-08-24 20:49:35 UTC status: CA_UNREACHABLE certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' expires: 2018-11-02 14:48:48 UTC status: CA_WORKING certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' expires: 2018-08-14 20:50:00 UTC
[5] journalctl -u certmonger --since yesterday
ca-ldap03.us.domain.com ipa-submit[1435]: GSSAPI client step 2 ca-ldap03.us.domain.com certmonger[963]: 2018-10-18 16:25:52 [963] Error 7 connecting to https://ca-ldap03.us.domain.com:8443/ca/agent/ca/pro ca-ldap03.us.domain.com python2[1409]: GSSAPI client step 1
[6] LDAP and HTTPD certs are issued by DigiCert, if that matters here.
So I guess something needs to be done to resolve [4] ?
thanks in advance. Zarko
On 10/20/18 5:40 AM, None via FreeIPA-users wrote:
Thanks Flo. [1] Service pki-tomcatd@pki-tomcat.service is active (running) [2] /var/log/pki/pki-tomcat/ca/debug reads among others:
- SSL handshake happened
- Could not connect to LDAP server host ca-ldap03.us.domain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
- Internal Database Error encountered: Could not connect to LDAP server host ca-ldap03.us.domain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)
I believe 48 is LDAP_INAPPROPRIATE_AUTH
[3] /var/log/ipa/ipactl.log reads:
-Failed to check CA status: cannot connect to 'http://ca-ldap03.us.domain.com:8080 /ca/admin/ca/getStatus': [Errno 111] Connection refused -Waiting until the CA is running -request POST http://ca-ldap03.us.domain.com:8080/ca/admin/ca/getStatus -request body '' -The CA status is: check interrupted due to error: cannot connect to 'http://ca-ld ap03.us.domain.com:8080/ca/admin/ca/getStatus': [Errno 111] Connection refused
- Waiting for CA to start...
- request POST http://ca-ldap03.us.domain.com:8080/ca/admin/ca/getStatus
-request body ''
- The CA status is: check interrupted due to error: cannot connect to 'http://ca-ld
ap03.us.domain.com:8080/ca/admin/ca/getStatus': [Errno 111] Connection refused
[4] list of certificates
# getcert list | egrep -e status -e expire -e certificate Number of certificates and requests being tracked: 6. status: CA_WORKING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2018-08-14 20:49:38 UTC status: CA_WORKING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2018-08-14 20:49:35 UTC status: CA_WORKING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' expires: 2018-08-14 20:49:36 UTC status: MONITORING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2036-08-24 20:49:35 UTC status: CA_UNREACHABLE certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' expires: 2018-11-02 14:48:48 UTC status: CA_WORKING certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' expires: 2018-08-14 20:50:00 UTC
Hi,
many certificates are expired and failed to renew. CA_WORKING means that the host is not the renewal master, and is trying to download the certificates from LDAP (PKI certs are renewed first on the renewal master, which in turns puts them in the LDAP store, and the replicas download them from LDAP).
You need first to identify who is the renewal master in your topology: # kinit admin # ipa config-show | grep 'IPA CA renewal master' IPA CA renewal master: hostx.domain.com (The command should return a single host, and this host must be active).
- If this node is still available, you will need to fix this one first (check that all certificates are still valid, or fix the renewal).
- If this node is not available anymore in your topology, you will need to select another master and move the renewal master role to this selected master, see [1]. The next step is to fix the renewal on this node.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
[5] journalctl -u certmonger --since yesterday
ca-ldap03.us.domain.com ipa-submit[1435]: GSSAPI client step 2 ca-ldap03.us.domain.com certmonger[963]: 2018-10-18 16:25:52 [963] Error 7 connecting to https://ca-ldap03.us.domain.com:8443/ca/agent/ca/pro ca-ldap03.us.domain.com python2[1409]: GSSAPI client step 1
[6] LDAP and HTTPD certs are issued by DigiCert, if that matters here.
So I guess something needs to be done to resolve [4] ?
thanks in advance. Zarko _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Flo, your feedback helps, thanks a lot !!!
Interestingly, 'ipa config-show' read that none of four (4) server is renewal master. I suspect it's the one that's installed first, indeed it has file /var/lib/ipa/pki-ca/publish/MasterCRL.bin
Finally I fixed that so ca-ldap01 now reads as "IPA CA renewal master". There is additional progress, but not complete. I reset clock to time before expiration, not before "notBefore", restart certmonger service, here is the status.
There are 3 expired certs, but status read "monitoring", actually on one it's switching between "monitoring" and "submitting"
status: MONITORING stuck: no CA: dogtag-ipa-ca-renew-agent subject: CN=CA Audit,O=US.ORACLE.COM expires: 2018-08-14 20:49:38 UTC track: yes auto-renew: yes
status: MONITORING stuck: no CA: dogtag-ipa-ca-renew-agent subject: CN=OCSP Subsystem,O=US.ORACLE.COM expires: 2020-10-11 20:15:53 UTC track: yes auto-renew: yes
status: switch between MONITORING and SUBMITTING stuck: no CA: dogtag-ipa-ca-renew-agent subject: CN=CA Subsystem,O=US.ORACLE.COM expires: 2018-08-14 20:49:36 UTC track: yes auto-renew: yes
status: MONITORING stuck: no CA: dogtag-ipa-ca-renew-agent subject: CN=Certificate Authority,O=US.ORACLE.COM expires: 2038-10-22 18:15:48 UTC track: yes auto-renew: yes
status: MONITORING stuck: no CA: dogtag-ipa-ca-renew-agent subject: CN=IPA RA,O=US.ORACLE.COM expires: 2018-08-14 20:50:00 UTC track: yes auto-renew: yes
status: MONITORING stuck: no CA: dogtag-ipa-renew-agent subject: CN=ca-ldap01.us.oracle.com,O=US.ORACLE.COM expires: 2020-07-07 01:47:45 UTC track: yes auto-renew: yes
I believe that track and auto-renew have been always "yes", but seems this didn't prevent problem. What would be next troubleshoot to finalize resolution, thanks in advance, Zarko
Hi Flo, the journalctl reports that request is rejected, error 2.
dogtag-ipa-ca-renew-agent-submit[29544]: Forwarding request to dogtag-ipa-renew-agent dogtag-ipa-renew-agent-submit[29558]: GET http://ca-ldap01.:8080/ca/ee/ca/profileSubmit?profil dogtag-ipa-renew-agent-submit[29558]: <html><head><title>Apache Tomcat/7.0.69 - Error report</title><style> dogtag-ipa-ca-renew-agent-submit[29544]: dogtag-ipa-renew-agent returned 2
I can't find a common date where all the certificates are valid, since ""ocspSigningCert cert-pki-ca" is not valid before today.
# certutil -L -d /etc/pki/pki-tomcat/alias -n "auditSigningCert cert-pki-ca" | egrep "Not Before|After" Not Before: Wed Aug 24 20:49:38 2016 Not After : Tue Aug 14 20:49:38 2018
# certutil -L -d /etc/pki/pki-tomcat/alias -n "ocspSigningCert cert-pki-ca" | egrep "Not Before|After" Not Before: Mon Oct 22 20:15:53 2018 Not After : Sun Oct 11 20:15:53 2020
# certutil -L -d /etc/pki/pki-tomcat/alias -n "subsystemCert cert-pki-ca" | egrep "Not Before|After" Not Before: Wed Aug 24 20:49:36 2016 Not After : Tue Aug 14 20:49:36 2018
# certutil -L -d /etc/pki/pki-tomcat/alias -n "caSigningCert cert-pki-ca" | egrep "Not Before|After" Not Before: Mon Oct 22 18:15:48 2018 Not After : Fri Oct 22 18:15:48 2038
# certutil -L -d /etc/httpd/alias -n "ipaCert" | egrep "Not Before|After" Not Before: Wed Aug 24 20:50:00 2016 Not After : Tue Aug 14 20:50:00 2018
# certutil -L -d /etc/pki/pki-tomcat/alias -n "Server-Cert cert-pki-ca" | egrep "Not Before|After" Not Before: Wed Jul 18 01:47:45 2018 Not After : Tue Jul 07 01:47:45 2020
On 10/23/18 5:24 AM, None via FreeIPA-users wrote:
Hi Flo, the journalctl reports that request is rejected, error 2.
dogtag-ipa-ca-renew-agent-submit[29544]: Forwarding request to dogtag-ipa-renew-agent dogtag-ipa-renew-agent-submit[29558]: GET http://ca-ldap01.:8080/ca/ee/ca/profileSubmit?profil dogtag-ipa-renew-agent-submit[29558]: <html><head><title>Apache Tomcat/7.0.69 - Error report</title><style> dogtag-ipa-ca-renew-agent-submit[29544]: dogtag-ipa-renew-agent returned 2
Hi,
PKI debug log may contain more information explaining why the request was rejected (/var/log/pki/pki-tomcat/ca/debug). You can also increase the debug level to get more information: edit (or create) /etc/ipa/server.conf and add the following: [global] debug=True
Then modify the certmonger helper that is used to renew the PKI certificates to increase verbosity: $ getcert modify-ca -c dogtag-ipa-ca-renew-agent -e '/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -vv'
(this simply adds -vv to the command executed by certmonger to renew the cert). The helper will log information in /var/log/ipa/renew.log.
HTH, flo
I can't find a common date where all the certificates are valid, since ""ocspSigningCert cert-pki-ca" is not valid before today.
# certutil -L -d /etc/pki/pki-tomcat/alias -n "auditSigningCert cert-pki-ca" | egrep "Not Before|After" Not Before: Wed Aug 24 20:49:38 2016 Not After : Tue Aug 14 20:49:38 2018
# certutil -L -d /etc/pki/pki-tomcat/alias -n "ocspSigningCert cert-pki-ca" | egrep "Not Before|After" Not Before: Mon Oct 22 20:15:53 2018 Not After : Sun Oct 11 20:15:53 2020
# certutil -L -d /etc/pki/pki-tomcat/alias -n "subsystemCert cert-pki-ca" | egrep "Not Before|After" Not Before: Wed Aug 24 20:49:36 2016 Not After : Tue Aug 14 20:49:36 2018
# certutil -L -d /etc/pki/pki-tomcat/alias -n "caSigningCert cert-pki-ca" | egrep "Not Before|After" Not Before: Mon Oct 22 18:15:48 2018 Not After : Fri Oct 22 18:15:48 2038
# certutil -L -d /etc/httpd/alias -n "ipaCert" | egrep "Not Before|After" Not Before: Wed Aug 24 20:50:00 2016 Not After : Tue Aug 14 20:50:00 2018
# certutil -L -d /etc/pki/pki-tomcat/alias -n "Server-Cert cert-pki-ca" | egrep "Not Before|After" Not Before: Wed Jul 18 01:47:45 2018 Not After : Tue Jul 07 01:47:45 2020
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Flo, I have debug enabled in both /etc/ipa/server.conf and /etc/ipa/default.conf and /var/log/pki/pki-tomcat/ca/debug reads:
[08/Aug/2018:10:12:02][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= java.lang.Exception: Certificate ocspSigningCert cert-pki-ca is invalid: Invalid certificate: (-8181) Peer's Certificate has expired. at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:844) at com.netscape.cmscore.cert.CertUtils.verifySystemCertByTag(CertUtils.java:936) at com.netscape.cmscore.cert.CertUtils.verifySystemCerts(CertUtils.java:1053) at com.netscape.cmscore.apps.CMSEngine.verifySystemCerts(CMSEngine.java:1803) at com.netscape.certsrv.apps.CMS.verifySystemCerts(CMS.java:1402) at com.netscape.cms.selftests.common.SystemCertsVerification.runSelfTest(SystemCertsVerification.java:193) at com.netscape.cmscore.selftests.SelfTestSubsystem.runSelfTestsAtStartup(SelfTestSubsystem.java:858) at com.netscape.cmscore.selftests.SelfTestSubsystem.startup(SelfTestSubsystem.java:1808) at com.netscape.cmscore.apps.CMSEngine.startupSubsystems(CMSEngine.java:1914) at com.netscape.cmscore.apps.CMSEngine.startup(CMSEngine.java:1355) at com.netscape.certsrv.apps.CMS.startup(CMS.java:200) at com.netscape.certsrv.apps.CMS.start(CMS.java:1617) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.security.cert.CertificateException: Invalid certificate: (-8181) Peer's Certificate has expired. at org.mozilla.jss.CryptoManager.verifyCertificateNowNative(Native Method) at org.mozilla.jss.CryptoManager.verifyCertificate(CryptoManager.java:1554) at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:842) ... 44 more Invalid class name repositorytop at com.netscape.cmscore.dbs.DBRegistry.createObject(DBRegistry.java:485) at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:167) at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:137) at com.netscape.cmscore.dbs.Repository.getSerialNumber(Repository.java:125) at com.netscape.cmscore.dbs.Repository.initCache(Repository.java:244) at com.netscape.cmscore.dbs.Repository.checkRanges(Repository.java:460) at com.netscape.cmscore.apps.CMSEngine.startup(CMSEngine.java:1371) at com.netscape.certsrv.apps.CMS.startup(CMS.java:200) at com.netscape.certsrv.apps.CMS.start(CMS.java:1617) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ############ end of debug ##############
## I worry now that I am not making progress with cert renewal. With stopped ntp and back in time /var/log/ipa/renew.log reads:
2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG Initializing principal host/ca-ldap01.domain.com@DOMAIN.COM using keytab /etc/krb5.keytab 2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG using ccache /var/run/certmonger/tmp-M09nld/ccache 2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG Attempt 1/1: success 2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2018-08-07T17:12:35Z 4375 MainThread ipa DEBUG Could not connect to the Directory Server on ca-ldap01.domain.com: Insufficient access: Invalid credentials
## OKAY, so need to enable NTPD and back in time again, now renew.log reads:
2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing all plugin modules in ipaserver.plugins... 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.baseldap is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.hbac is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.otp is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG Starting external process 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG args=klist -V 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG Process finished, return code=0 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG stdout=Kerberos 5 version 1.14.1 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG stderr= 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing plugin module ipaserver.plugins.rabase 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing plugin module ipaserver.plugins.sudo 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.sudo is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.virtual is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing plugin module ipaserver.plugins.xmlserver 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG Initializing principal host/ca-ldap01.domain.com@domain.com using keytab /etc/krb5.keytab 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG using ccache /var/run/certmonger/tmp-5bCOl7/ccache 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG Attempt 1/1: success 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2018-08-07T17:11:35Z 6773 MainThread ipa.ipapython.ipaldap.SchemaCache DEBUG flushing ldap://ca-ldap01.domain.com:389 from SchemaCache 2018-08-07T17:11:35Z 6773 MainThread ipa.ipapython.ipaldap.SchemaCache DEBUG retrieving schema for SchemaCache url=ldap://ca-ldap01.domain.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x5a69320> 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG Starting external process 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG args=/usr/libexec/certmonger/dogtag-ipa-renew-agent-submit -vv 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG Process finished, return code=2 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG stdout= 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG stderr=* About to connect() to ca-ldap01.domain.com port 8080 (#0) * Trying 10.211.9.58... * Connected to ca-ldap01.domain.com (10.211.9.58) port 8080 (#0)
GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true HTTP/1.1
Host: ca-ldap01.domain.com:8080 Accept: */*
< HTTP/1.1 404 Not Found < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 995 < Date: Thu, 25 Oct 2018 05:42:30 GMT < * Connection #0 to host ca-ldap01.domain.com left intact GET "http://ca-ldap01.domain.com:8080/ca/ee/ca/profileSubmit?profileId=caServerCe..." code = 0 code_text = "No error" results = "<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /ca/ee/ca/profileSubmit</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/ca/ee/ca/profileSubmit</u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body
</html>"
Entity: line 1: parser error : Opening and ending tag mismatch: HR line 1 and body able.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body> ^ Entity: line 1: parser error : Opening and ending tag mismatch: HR line 1 and html Entity: line 1: parser error : Premature end of data in tag body line 1 Entity: line 1: parser error : Premature end of data in tag html line 1 Entity: line 1: parser error : Opening and ending tag mismatch: HR line 1 and body able.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body> ^
## And status of certmonger service reads:
Aug 07 10:12:45 ca-ldap01.domain.com dogtag-ipa-renew-agent-submit[6998]: GET http://ca-ldap01.domain.com:8080/ca/ee/ca/profileSubmit?profileId=caServerCe...
Aug 07 10:12:45 ca-ldap01.domain.com dogtag-ipa-renew-agent-submit[6998]: <html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /ca/ee/ca/profileSubmit</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/ca/ee/ca/profileSubmit</u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></html>
Aug 07 10:12:45 ca-ldap01.domain.com dogtag-ipa-ca-renew-agent-submit[6884]: dogtag-ipa-renew-agent returned 2
Thanks in advance for any sugestion on next step.
On 10/25/18 8:11 AM, Z D via FreeIPA-users wrote:
Hi Flo, I have debug enabled in both /etc/ipa/server.conf and /etc/ipa/default.conf and /var/log/pki/pki-tomcat/ca/debug reads:
[08/Aug/2018:10:12:02][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= java.lang.Exception: Certificate ocspSigningCert cert-pki-ca is invalid: Invalid certificate: (-8181) Peer's Certificate has expired. at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:844) at com.netscape.cmscore.cert.CertUtils.verifySystemCertByTag(CertUtils.java:936) at com.netscape.cmscore.cert.CertUtils.verifySystemCerts(CertUtils.java:1053) at com.netscape.cmscore.apps.CMSEngine.verifySystemCerts(CMSEngine.java:1803) at com.netscape.certsrv.apps.CMS.verifySystemCerts(CMS.java:1402) at com.netscape.cms.selftests.common.SystemCertsVerification.runSelfTest(SystemCertsVerification.java:193) at com.netscape.cmscore.selftests.SelfTestSubsystem.runSelfTestsAtStartup(SelfTestSubsystem.java:858) at com.netscape.cmscore.selftests.SelfTestSubsystem.startup(SelfTestSubsystem.java:1808) at com.netscape.cmscore.apps.CMSEngine.startupSubsystems(CMSEngine.java:1914) at com.netscape.cmscore.apps.CMSEngine.startup(CMSEngine.java:1355) at com.netscape.certsrv.apps.CMS.startup(CMS.java:200) at com.netscape.certsrv.apps.CMS.start(CMS.java:1617) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.security.cert.CertificateException: Invalid certificate: (-8181) Peer's Certificate has expired. at org.mozilla.jss.CryptoManager.verifyCertificateNowNative(Native Method) at org.mozilla.jss.CryptoManager.verifyCertificate(CryptoManager.java:1554) at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:842) ... 44 more Invalid class name repositorytop at com.netscape.cmscore.dbs.DBRegistry.createObject(DBRegistry.java:485) at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:167) at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:137) at com.netscape.cmscore.dbs.Repository.getSerialNumber(Repository.java:125) at com.netscape.cmscore.dbs.Repository.initCache(Repository.java:244) at com.netscape.cmscore.dbs.Repository.checkRanges(Repository.java:460) at com.netscape.cmscore.apps.CMSEngine.startup(CMSEngine.java:1371) at com.netscape.certsrv.apps.CMS.startup(CMS.java:200) at com.netscape.certsrv.apps.CMS.start(CMS.java:1617) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ############ end of debug ##############
## I worry now that I am not making progress with cert renewal. With stopped ntp and back in time /var/log/ipa/renew.log reads:
2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG Initializing principal host/ca-ldap01.domain.com@DOMAIN.COM using keytab /etc/krb5.keytab 2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG using ccache /var/run/certmonger/tmp-M09nld/ccache 2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG Attempt 1/1: success 2018-08-07T17:12:34Z 4375 MainThread ipa DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2018-08-07T17:12:35Z 4375 MainThread ipa DEBUG Could not connect to the Directory Server on ca-ldap01.domain.com: Insufficient access: Invalid credentials
## OKAY, so need to enable NTPD and back in time again, now renew.log reads:
2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing all plugin modules in ipaserver.plugins... 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.baseldap is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.hbac is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.otp is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG Starting external process 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG args=klist -V 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG Process finished, return code=0 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG stdout=Kerberos 5 version 1.14.1 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG stderr= 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing plugin module ipaserver.plugins.rabase 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing plugin module ipaserver.plugins.sudo 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.sudo is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG ipaserver.plugins.virtual is not a valid plugin module 2018-08-07T17:11:34Z 6773 MainThread ipa DEBUG importing plugin module ipaserver.plugins.xmlserver 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG Initializing principal host/ca-ldap01.domain.com@domain.com using keytab /etc/krb5.keytab 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG using ccache /var/run/certmonger/tmp-5bCOl7/ccache 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG Attempt 1/1: success 2018-08-07T17:11:35Z 6773 MainThread ipa DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2018-08-07T17:11:35Z 6773 MainThread ipa.ipapython.ipaldap.SchemaCache DEBUG flushing ldap://ca-ldap01.domain.com:389 from SchemaCache 2018-08-07T17:11:35Z 6773 MainThread ipa.ipapython.ipaldap.SchemaCache DEBUG retrieving schema for SchemaCache url=ldap://ca-ldap01.domain.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x5a69320> 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG Starting external process 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG args=/usr/libexec/certmonger/dogtag-ipa-renew-agent-submit -vv 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG Process finished, return code=2 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG stdout= 2018-08-07T17:11:36Z 6773 MainThread ipa DEBUG stderr=* About to connect() to ca-ldap01.domain.com port 8080 (#0)
- Trying 10.211.9.58...
- Connected to ca-ldap01.domain.com (10.211.9.58) port 8080 (#0)
GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true HTTP/1.1
Host: ca-ldap01.domain.com:8080 Accept: */*
< HTTP/1.1 404 Not Found < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 995 < Date: Thu, 25 Oct 2018 05:42:30 GMT <
- Connection #0 to host ca-ldap01.domain.com left intact
GET "http://ca-ldap01.domain.com:8080/ca/ee/ca/profileSubmit?profileId=caServerCe..." code = 0 code_text = "No error" results = "<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /ca/ee/ca/profileSubmit</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/ca/ee/ca/profileSubmit</u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body
</html>"
Entity: line 1: parser error : Opening and ending tag mismatch: HR line 1 and body able.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body> ^ Entity: line 1: parser error : Opening and ending tag mismatch: HR line 1 and html Entity: line 1: parser error : Premature end of data in tag body line 1 Entity: line 1: parser error : Premature end of data in tag html line 1 Entity: line 1: parser error : Opening and ending tag mismatch: HR line 1 and body able.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body> ^
## And status of certmonger service reads:
Aug 07 10:12:45 ca-ldap01.domain.com dogtag-ipa-renew-agent-submit[6998]: GET http://ca-ldap01.domain.com:8080/ca/ee/ca/profileSubmit?profileId=caServerCe...
Aug 07 10:12:45 ca-ldap01.domain.com dogtag-ipa-renew-agent-submit[6998]: <html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /ca/ee/ca/profileSubmit</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>/ca/ee/ca/profileSubmit</u></p><p><b>description</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></html>
Aug 07 10:12:45 ca-ldap01.domain.com dogtag-ipa-ca-renew-agent-submit[6884]: dogtag-ipa-renew-agent returned 2
Thanks in advance for any sugestion on next step.
Hi, when the date is moved back to when the certs are valid, can you check if the CA component is running? curl http://%60hostname%60:8080/ca/ee/ca/getCertChain
If not, have a look at the logs from /var/log/pki/pki-tomcat to find the reason.
HTH, flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
No, CA component is not running, and seems not much activity under /var/log/pki/pki-tomcat. Maybe these can be of interest:
[1] selftests.log 0.localhost-startStop-1 - [08/Aug/2018:10:12:03 PDT] [20] [1] SystemCertsVerification: system certs verification failure: Certificate ocspSigningCert cert-pki-ca is invalid: Invalid certificate: (-8181) Peer's Certificate has expired. 0.localhost-startStop-1 - [08/Aug/2018:10:12:03 PDT] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED!
[2] catalina.log
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property
Flo, if I can suspect on this .... I recall before incident this one expires on 2036, now it's 2038
status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2038-10-22 18:15:48 UTC track: yes auto-renew: yes
And URI was hostname, not ipa-ca.
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'caSigningCert cert-pki-ca' | grep URI URI: "http://ipa-ca.domain.com/ca/ocsp"
Is there way to "manually" revert change or renew a cert?
Z D via FreeIPA-users wrote:
No, CA component is not running, and seems not much activity under /var/log/pki/pki-tomcat. Maybe these can be of interest:
[1] selftests.log 0.localhost-startStop-1 - [08/Aug/2018:10:12:03 PDT] [20] [1] SystemCertsVerification: system certs verification failure: Certificate ocspSigningCert cert-pki-ca is invalid: Invalid certificate: (-8181) Peer's Certificate has expired. 0.localhost-startStop-1 - [08/Aug/2018:10:12:03 PDT] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED!
Check the expiration dates of your certs. Perhaps you didn't go back far enough or you have some mix of old and renewed certs.
There is a way to disable the selftest but this is a sort of last resort.
[2] catalina.log
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property
Flo, if I can suspect on this .... I recall before incident this one expires on 2036, now it's 2038
status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' expires: 2038-10-22 18:15:48 UTC track: yes auto-renew: yesAnd URI was hostname, not ipa-ca.
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'caSigningCert cert-pki-ca' | grep URI URI: "http://ipa-ca.domain.com/ca/ocsp"
Did you run ipa cacert-manage renew at some point?
Is there way to "manually" revert change or renew a cert?
I don't understand the question, revert to what?
Manually renewing is possible theoretically but brings with it so many other problems.
rob
Hi Rob, thanks much.
Some of Flo's blogs about CA helps me to understand better now. Sure "ipa cacert-manage renew" and "ipa-certupdate" was run before, hopefully not harmful, "caSigningCert cert-pki-ca" was valid for 18 more years.
You're right, there is mix of old and renewed ones, three requres renewal: auditSigningCert, subsystemCert and ipaCert , all expired on 2018-08-14. Time I went back was 7 days earlier, 2018-08-07
Sorry, nothing to revert, please let me know what would you suggest now. The state of certs is:
status: MONITORING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' subject: CN=CA Audit,O=DOMAIN.COM expires: 2018-08-14 20:49:38 UTC
status: MONITORING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' subject: CN=OCSP Subsystem,O=DOMAIN.COM expires: 2020-10-11 20:15:53 UTC
status: MONITORING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' subject: CN=CA Subsystem,O=DOMAIN.COM expires: 2018-08-14 20:49:36 UTC
status: MONITORING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' subject: CN=Certificate Authority,O=DOMAIN.COM expires: 2038-10-22 18:15:48 UTC
status: MONITORING certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' subject: CN=IPA RA,O=DOMAIN.COM expires: 2018-08-14 20:50:00 UTC
status: MONITORING certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' subject: CN=ca-ldap01.DOMAIN.com,O=DOMAIN.COM expires: 2020-07-07 01:47:45 UTC
Hi Rob, I follow one of your suggestions in another post, it's : "certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens"
I did it, no success with messages:
- MainThread ipa DEBUG Could not connect to the Directory Server on ca-ldap01.domain.com: Insufficient access: Invalid credentials - ca-dap01 certmonger: 2018-08-07 10:40:39 [4831] Internal error - ca-ldap01 [sssd[ldap_child[5045]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection. - ca-ldap01 [sssd[ldap_child[5045]]]: Cannot contact any KDC for realm 'DOMAIN.COM'
And I notice that Kerberos somehow still shows current date, instead of 2018-08-07 (my back in time).
[root@ca-ldap01 ~]# klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@DOMAIN.COM Valid starting Expires Service principal 10/25/2018 20:55:02 10/26/2018 20:54:58 krbtgt/DOMAIN.COM@DOMAIN.COM
Is this reason for failure?
On 10/26/18 7:36 AM, Z D via FreeIPA-users wrote:
Hi Rob, I follow one of your suggestions in another post, it's : "certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens"
I did it, no success with messages:
- MainThread ipa DEBUG Could not connect to the Directory Server on ca-ldap01.domain.com: Insufficient access: Invalid credentials
- ca-dap01 certmonger: 2018-08-07 10:40:39 [4831] Internal error
- ca-ldap01 [sssd[ldap_child[5045]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
- ca-ldap01 [sssd[ldap_child[5045]]]: Cannot contact any KDC for realm 'DOMAIN.COM'
And I notice that Kerberos somehow still shows current date, instead of 2018-08-07 (my back in time).
[root@ca-ldap01 ~]# klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@DOMAIN.COM Valid starting Expires Service principal 10/25/2018 20:55:02 10/26/2018 20:54:58 krbtgt/DOMAIN.COM@DOMAIN.COM
Is this reason for failure?
Hi,
when you go back in time, avoid using ipactl start to restart all the services as it will restart ntpd (and set back the time to the current time). Use systemctl start instead.
flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Agree Flo, making sure that I am in the past, unfortunately still not resolution.
[root@ca-ldap01 ~]# systemctl restart krb5kdc [root@ca-ldap01 ~]# systemctl restart dirsrv@DOMAIN-COM.service [root@ca-ldap01 ~]# systemctl restart httpd [root@ca-ldap01 ~]# systemctl restart pki-tomcatd@pki-tomcat.service
Kerberos now understand we are in the past!
[root@ca-ldap01 ~]# klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@domain.COM Valid starting Expires Service principal 08/13/2018 11:21:32 08/14/2018 11:21:28 krbtgt/domain.COM@domain.COM
[root@ca-ldap01 ~]# date Mon Aug 13 11:22:20 PDT 2018
And these are renew.log errors when certmonger is restarted, or getcert resubmit, basically insufficient access to directory server !?
2018-08-13T18:46:49Z 13747 MainThread ipa DEBUG Initializing principal host/ca-ldap01.domain.com@DOMAIN.COM using keytab /etc/krb5.keytab 2018-08-13T18:46:49Z 13747 MainThread ipa DEBUG using ccache /var/run/certmonger/tmp-NGZEev/ccache 2018-08-13T18:46:49Z 13747 MainThread ipa DEBUG Attempt 1/1: success 2018-08-13T18:46:49Z 13747 MainThread ipa DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2018-08-13T18:46:49Z 13747 MainThread ipa DEBUG Could not connect to the Directory Server on ca-ldap01.domain.com: Insufficient access: Invalid credentials
Syslog message also reads "Insufficient access: Invalid credentials"
Aug 13 12:20:39 ca-ldap01 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012 File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 511, in <module>#012 sys.exit(main())#012 File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 497, in main#012 if ca.is_renewal_master():#012 File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1188, in is_renewal_master#012 self.ldap_connect()#012 File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 177, in ldap_connect#012 conn.do_bind(self.dm_password, autobind=self.autobind)#012 File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1690, in do_bind#012 self.do_sasl_gssapi_bind(timeout=timeout)#012 File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1668, in do_sasl_gssapi_bind#012 self.__bind_with_wait(self.gssapi_bind, timeout)#012 File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", lin e 1650, in __bind_with_wait#012 bind_func(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1108, in gssapi_bind#012 '', auth_tokens, server_controls, client_controls)#012 File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__#012 self.gen.throw(type, value, traceback)#012 File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 973, in error_handler#012 raise errors.ACIError(info="%s %s" % (info, desc))#012ACIError: Insufficient access: Invalid credentials Aug 13 12:20:39 ca-ldap01 certmonger: 2018-08-13 12:20:39 [16173] Internal error
I've been worrying now, it's being a week and this is prod environment. I've tried to add 5th server, but replica to it fails. Any hope for resolution?
Hi Flo and Rob, additional update. There is discrepancy in some of cert's expire time among 4 servers, I thought maybe another server can be candidate to be new renewal master. The command "ipa-csreplica-manage set-renewal-master ca-ldap02" worked well, hence "ipa config-show" on all 4 servers reads ca-ldap02 is IPA CA renewal master.
But it's still mixer of expired and valid certs, auditSigningCert, caSigningCert and ipaCert are expired. So on ca-ldap02 I repeated familiar process of "kill ntpd, going back a few days, restart krb5kdc, dirsrv, httpd, CA , then certmonger" and having error from previous update :
"Directory Server on ca-ldap02: Insufficient access: Invalid credentials"
Have a good weekend, hope to continue troubleshoot next week.
From what I experience, during " killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger", service ipa-dnskeysyncd.service is failing.
Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG Kerberos principal: ipa-dnskeysyncd/ca-ldap04.domain.com Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG Initializing principal ipa-dnskeysyncd/ca-ldap04.domain.com using keytab /etc/ipa/dnssec/ipa-dnskeysyncd.keytab Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG using ccache /tmp/ipa-dnskeysyncd.ccache Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG Attempt 1/5: success Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG LDAP URL: ldapi://%2Fvar%2Frun%2Fslapd-US-ORACLE-COM.socket/cn%3Ddns%2Cdc%3Dus%2Cdc%3Doracle%2Cdc%3Dcom??sub?%28%7C%28objectClass%3DidnsZone%29%28objectClass%3DidnsSecKey%29%28objectClass%3Dipk11PublicKey%29%29 Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : INFO LDAP bind... Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : ERROR Login to LDAP server failed: {'desc': 'Invalid credentials'} Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: Traceback (most recent call last): Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/libexec/ipa/ipa-dnskeysyncd", line 90, in <module> Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ldap_connection.sasl_interactive_bind_s("", ipaldap.SASL_GSSAPI) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 850, in sasl_interactive_bind_s Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: res = self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 818, in _apply_method_s Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: return func(self,*args,**kwargs) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in sasl_interactive_bind_s Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: return self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in _ldap_call Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: result = func(*args,**kwargs) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: INVALID_CREDENTIALS: {'desc': 'Invalid credentials'} Aug 10 10:19:18 ca-ldap04 systemd: ipa-dnskeysyncd.service: main process exited, code=exited, status=1/FAILURE Aug 10 10:19:18 ca-ldap04 systemd: Unit ipa-dnskeysyncd.service entered failed state. Aug 10 10:19:18 ca-ldap04 systemd: ipa-dnskeysyncd.service failed.
And other logs like :
Aug 10 10:17:38 ca-ldap04 ns-slapd: [10/Aug/2018:10:17:38.973963675 -0700] csngen_new_csn - Warning: too much time skew (-6959333 secs). Current seqnum=47 Aug 10 10:17:53 ca-ldap04 named-pkcs11[2514]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket not yet valid) Aug 10 10:17:53 ca-ldap04 named-pkcs11[2514]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Zarko D via FreeIPA-users wrote:
From what I experience, during " killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger", service ipa-dnskeysyncd.service is failing.
Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG Kerberos principal: ipa-dnskeysyncd/ca-ldap04.domain.com Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG Initializing principal ipa-dnskeysyncd/ca-ldap04.domain.com using keytab /etc/ipa/dnssec/ipa-dnskeysyncd.keytab Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG using ccache /tmp/ipa-dnskeysyncd.ccache Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG Attempt 1/5: success Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : DEBUG LDAP URL: ldapi://%2Fvar%2Frun%2Fslapd-US-ORACLE-COM.socket/cn%3Ddns%2Cdc%3Dus%2Cdc%3Doracle%2Cdc%3Dcom??sub?%28%7C%28objectClass%3DidnsZone%29%28objectClass%3DidnsSecKey%29%28objectClass%3Dipk11PublicKey%29%29 Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : INFO LDAP bind... Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ipa : ERROR Login to LDAP server failed: {'desc': 'Invalid credentials'} Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: Traceback (most recent call last): Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/libexec/ipa/ipa-dnskeysyncd", line 90, in <module> Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: ldap_connection.sasl_interactive_bind_s("", ipaldap.SASL_GSSAPI) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 850, in sasl_interactive_bind_s Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: res = self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 818, in _apply_method_s Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: return func(self,*args,**kwargs) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in sasl_interactive_bind_s Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: return self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in _ldap_call Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: result = func(*args,**kwargs) Aug 10 10:19:18 ca-ldap04 ipa-dnskeysyncd: INVALID_CREDENTIALS: {'desc': 'Invalid credentials'} Aug 10 10:19:18 ca-ldap04 systemd: ipa-dnskeysyncd.service: main process exited, code=exited, status=1/FAILURE Aug 10 10:19:18 ca-ldap04 systemd: Unit ipa-dnskeysyncd.service entered failed state. Aug 10 10:19:18 ca-ldap04 systemd: ipa-dnskeysyncd.service failed.
This doesn't matter. You are forcefull going back in time. As long as it doesn't prevent named from starting and at least limping along then it isn't worth pursuing until the certs are renewed.
And other logs like :
Aug 10 10:17:38 ca-ldap04 ns-slapd: [10/Aug/2018:10:17:38.973963675 -0700] csngen_new_csn - Warning: too much time skew (-6959333 secs). Current seqnum=47 Aug 10 10:17:53 ca-ldap04 named-pkcs11[2514]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket not yet valid) Aug 10 10:17:53 ca-ldap04 named-pkcs11[2514]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
This is expected.
rob
This doesn't . You are forcefull going back in time. As long as it doesn't prevent named from starting and at least limping along then it isn't worth pursuing until the certs are renewed.
I can confirm that going back in time prevents named running. It looks it's active but with errors. The returning to the present, service doesn't have any errors.
[root@ca-ldap04 ca]# systemctl status -l named-pkcs11.service
named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native PKCS#11 Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2018-10-30 21:41:43 PDT; 2 months 19 days left Process: 24525 ExecStop=/bin/sh -c /usr/sbin/rndc stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID (code=exited, status=0/SUCCESS) Process: 32575 ExecStart=/usr/sbin/named-pkcs11 -u named $OPTIONS (code=exited, status=0/SUCCESS) Process: 32571 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS) Main PID: 32576 (named-pkcs11) CGroup: /system.slice/named-pkcs11.service └─32576 /usr/sbin/named-pkcs11 -u named
Aug 11 20:24:32 ca-ldap04 named-pkcs11[32576]: GSSAPI client step 1 Aug 11 20:24:32 ca-ldap04 named-pkcs11[32576]: GSSAPI client step 1 Aug 11 20:24:32 ca-ldap04 named-pkcs11[32576]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket not yet valid) Aug 11 20:24:32 ca-ldap04.domain.com named-pkcs11[32576]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket not yet valid): bind to LDAP server failed Aug 11 20:24:32 ca-ldap04 named-pkcs11[32576]: ldap_syncrepl will reconnect in 60 seconds Aug 11 20:25:32 ca-ldap04 named-pkcs11[32576]: GSSAPI client step 1 Aug 11 20:25:32 ca-ldap04 named-pkcs11[32576]: GSSAPI client step 1 Aug 11 20:25:32 ca-ldap04 named-pkcs11[32576]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket not yet valid) Aug 11 20:25:32 ca-ldap04 named-pkcs11[32576]: LDAP error: Local error: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Ticket not yet valid): bind to LDAP server failed Aug 11 20:25:32 ca-ldap04 named-pkcs11[32576]: ldap_syncrepl will reconnect in 60 seconds
Hi Rob, any idea why going back in time prevents named running. It looks it's active but with errors. The returning to the present, service doesn't have any errors.
Zarko D via FreeIPA-users wrote:
There is a way to disable the selftest but this is a sort of last resort.
Hi Rob, I am afraid disabling SelfTest is maybe the way to resolve the issue. Are there any documentation on this, IPA 4.4.0 and pki-server 10.3.3
https://www.dogtagpki.org/wiki/SelfTest
I'm not convinced this will make much of a different if one or more of the certs is considered invalid by dogtag but I guess it's worth a shot.
rob
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Rob Crittenden -
Z D -
Zarko D
-
zarko＠etcfstab.com