Chris Mohler via FreeIPA-users wrote:

Thanks for looking at my issue!

There have been no recent updates on my system. Actually I was getting ready to update when I noticed things weren't good.

Here is the output from the log of the most recent update. Looks like it was completed successfully. The lines you asked about are in Bold/underlined.

...

2018-07-18T16:55:21Z INFO [Update certmonger certificate renewal configuration] 2018-07-18T16:55:21Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d /etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout= Certificate Nickname                                         Trust Attributes                                                              SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu subsystemCert cert-pki-ca                                    u,u,u ocspSigningCert cert-pki-ca                                  u,u,u auditSigningCert cert-pki-ca                                 u,u,Pu Server-Cert cert-pki-ca                                      u,u,u

2018-07-18T16:55:21Z DEBUG stderr= _*2018-07-18T16:55:21Z DEBUG Configuring certmonger to stop tracking system certificates for CA*_ 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start messagebus.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout= 2018-07-18T16:55:21Z DEBUG stderr= 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active messagebus.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout=active

2018-07-18T16:55:21Z DEBUG stderr= 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start certmonger.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout= 2018-07-18T16:55:21Z DEBUG stderr= 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active certmonger.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout=active

-snip- a few more lines like the section above.

...

2018-07-18T16:55:25Z DEBUG stderr= 2018-07-18T16:55:30Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2018-07-18T16:55:30Z DEBUG Starting external process 2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f /etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt 2018-07-18T16:55:30Z DEBUG Process finished, return code=0 2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN CERTIFICATE-----

-Snip- Cert and Key stuff goes here-

...

2018-07-18T16:55:34Z DEBUG stderr= _*2018-07-18T16:55:35Z INFO Certmonger certificate renewal configuration updated*_

Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster

You want to run the script on that master first to get the certs renewed.

I'd start by re-running ipa-server-upgrade. It is idempotent so there should be no risk. It may repair the tracking for you.

rob

On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote:

...

On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users wrote:

...

Hi Everyone,

I'm looking for some help. I'm having trouble with everything basically.

I think one of my CA's certs expired or something. I can't kinit admin, I can't login via the WebGui. If I "getcert list" it returns "Number of certificates and requests being tracked: 0."

This all started happening a few days ago and I am at a loss as to what happened. On a whim I set the system date and time back a few months to see if my certs were expired and like magic I can login to the Webgui but I'm still not tracking anything with "getcert list" I suspect the cert has expired but without tracking it I can't tell, or renew it.

Hi,

can you check if an upgrade happened recently (have a look at /var/log/ipaupgrade.log)? The upgrade stop tracking certs and re-configures certmonger, so if it failed in the middle you may be left without any tracking. You should be able to find lines like the following if the untracking/tracking went fine:

---

[Update certmonger certificate renewal configuration] Configuring certmonger to stop tracking system certificates for CA Certmonger certificate renewal configuration updated

---

HTH, flo

...

Please help

I'm running Centos 7, FreeIPA 4.5.4

Thanks,

-Chris

---

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

---

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

Well... That was a mess.

The ipa-server-upgrade didn't go so well. It failed and now my ca-replication master is broken. Here are the details. Any hope?

Upgrading IPA:. Estimated time: 1 minute 30 seconds   [1/11]: stopping directory server   [2/11]: saving configuration   [3/11]: disabling listeners   [4/11]: enabling DS global lock   [5/11]: disabling Schema Compat   [6/11]: starting directory server   [7/11]: updating schema   [8/11]: upgrading server   [9/11]: stopping directory server   [10/11]: restoring configuration   [11/11]: starting directory server Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. CA did not start in 300.0s The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Here is a wall of errors from my /var/log/ipaupgrade.log

Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504 -0500]

• ERR - set_krb5_creds - Could not get initial credentials for

principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522 -0500]

• ERR - slapi_ldap_bind - Error: could not send startTLS request:

error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514 -0500]

• ERR - set_krb5_creds - Could not get initial credentials for

principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb  4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_389)) Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432 -0500]

• ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication

Manager masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Feb  4 17:47:40 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:47:40 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:47:40 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Feb  4 17:47:40 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:47:50 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:47:50 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:47:50 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Feb  4 17:47:50 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:48:00 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:00 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:00 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Feb  4 17:48:00 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:48:10 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:10 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:10 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Feb  4 17:48:10 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb  4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb  4 17:48:20 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:20 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:20 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Feb  4 17:48:20 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:48:30 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:30 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:30 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Feb  4 17:48:30 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) ^C [root@ipa2 log]# less /var/log/ipaupgrade.log

<p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:13Z DEBUG Waiting for CA to start... 2019-02-04T22:46:14Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:14Z DEBUG request body '' 2019-02-04T22:46:14Z DEBUG response status 500 2019-02-04T22:46:14Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:14 GMT Connection: close

2019-02-04T22:46:14Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>

<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>

<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:14Z DEBUG Waiting for CA to start... 2019-02-04T22:46:15Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:15Z DEBUG request body '' 2019-02-04T22:46:15Z DEBUG response status 500 2019-02-04T22:46:15Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:15 GMT Connection: close

2019-02-04T22:46:15Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>

<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>

<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:15Z DEBUG Waiting for CA to start... 2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-02-04T22:46:16Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute     return_value = self.run()   File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 56, in run     raise admintool.ScriptError(str(e))

2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Thanks, -Chris

Rob,

I'll be honest. I think you are suggesting an ldapsearch with this

Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster

sorry I've not figured out how to successfully ldapsearch :-(

Instead I did this: ipa config-show |grep 'CA renewal master'

It came up blank. I suspect I didn't have a renewal master somehow.

Then I did This: ipa-csreplica-manage set-renewal-master ipa2 (hostname of working IPA server)

Next is a "yum update" to be safe, and lastly "ipa-server-upgrade" on ipa2. When that's all done I'll try "yum update" and "ipa-server-upgrade" on my broken IPA system ipa1

I'll report back here when finished.

Thanks,

-Chris

...

Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster

You want to run the script on that master first to get the certs renewed.

I'd start by re-running ipa-server-upgrade. It is idempotent so there should be no risk. It may repair the tracking for you.

rob

On 2/4/2019 3:30 PM, Rob Crittenden wrote:

...

Chris Mohler via FreeIPA-users wrote:

...

Thanks for looking at my issue!

There have been no recent updates on my system. Actually I was getting ready to update when I noticed things weren't good.

Here is the output from the log of the most recent update. Looks like it was completed successfully. The lines you asked about are in Bold/underlined.

...

2018-07-18T16:55:21Z INFO [Update certmonger certificate renewal configuration] 2018-07-18T16:55:21Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d /etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout= Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u ocspSigningCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu Server-Cert cert-pki-ca u,u,u

2018-07-18T16:55:21Z DEBUG stderr= _*2018-07-18T16:55:21Z DEBUG Configuring certmonger to stop tracking system certificates for CA*_ 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start messagebus.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout= 2018-07-18T16:55:21Z DEBUG stderr= 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active messagebus.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout=active

2018-07-18T16:55:21Z DEBUG stderr= 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start certmonger.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout= 2018-07-18T16:55:21Z DEBUG stderr= 2018-07-18T16:55:21Z DEBUG Starting external process 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active certmonger.service 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 2018-07-18T16:55:21Z DEBUG stdout=active

-snip- a few more lines like the section above.

...

2018-07-18T16:55:25Z DEBUG stderr= 2018-07-18T16:55:30Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2018-07-18T16:55:30Z DEBUG Starting external process 2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f /etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt 2018-07-18T16:55:30Z DEBUG Process finished, return code=0 2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN CERTIFICATE-----

-Snip- Cert and Key stuff goes here-

...

2018-07-18T16:55:34Z DEBUG stderr= _*2018-07-18T16:55:35Z INFO Certmonger certificate renewal configuration updated*_

Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster

You want to run the script on that master first to get the certs renewed.

I'd start by re-running ipa-server-upgrade. It is idempotent so there should be no risk. It may repair the tracking for you.

rob

...

On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote:

...

On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users wrote:

...

Hi Everyone,

I'm looking for some help. I'm having trouble with everything basically.

I think one of my CA's certs expired or something. I can't kinit admin, I can't login via the WebGui. If I "getcert list" it returns "Number of certificates and requests being tracked: 0."

This all started happening a few days ago and I am at a loss as to what happened. On a whim I set the system date and time back a few months to see if my certs were expired and like magic I can login to the Webgui but I'm still not tracking anything with "getcert list" I suspect the cert has expired but without tracking it I can't tell, or renew it.

Hi,

can you check if an upgrade happened recently (have a look at /var/log/ipaupgrade.log)? The upgrade stop tracking certs and re-configures certmonger, so if it failed in the middle you may be left without any tracking. You should be able to find lines like the following if the untracking/tracking went fine:

---

[Update certmonger certificate renewal configuration] Configuring certmonger to stop tracking system certificates for CA Certmonger certificate renewal configuration updated

---

HTH, flo

...

Please help

I'm running Centos 7, FreeIPA 4.5.4

Thanks,

-Chris

---

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

---

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

Followup summary:

Q: Seems like part of the problem is that the KDC was not running. Had you done ipactl stop prior to the upgrade?

A: I could not get the KDC to stay running. So yes it was off during the upgrade.

Q: Did it end up creating the tracking? Are there expired certs?

A: I was able to get the upgrade to finish successfully, after restoring the server from VM snapshot, rolling back the system date, and trying the update again. It did create the cert tracking!!! Yes there are expired certs.

Q: As an aside, I'd have suggest deferring the package upgrade until after the other things were sorted. It just adds another moving part. Water under the bridge now.

A: Yes sorry.

On 2/5/2019 11:18 AM, Rob Crittenden wrote:

...

Chris Mohler wrote:

...

Well... That was a mess.

The ipa-server-upgrade didn't go so well. It failed and now my ca-replication master is broken. Here are the details. Any hope?

...

Upgrading IPA:. Estimated time: 1 minute 30 seconds   [1/11]: stopping directory server   [2/11]: saving configuration   [3/11]: disabling listeners   [4/11]: enabling DS global lock   [5/11]: disabling Schema Compat   [6/11]: starting directory server   [7/11]: updating schema   [8/11]: upgrading server   [9/11]: stopping directory server   [10/11]: restoring configuration   [11/11]: starting directory server Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. CA did not start in 300.0s The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Seems like part of the problem is that the KDC was not running. Had you done ipactl stop prior to the upgrade?

Did it end up creating the tracking? Are there expired certs?

As an aside, I'd have suggest deferring the package upgrade until after the other things were sorted. It just adds another moving part. Water under the bridge now.

rob

...

Here is a wall of errors from my /var/log/ipaupgrade.log

...

Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504 -0500]

• ERR - set_krb5_creds - Could not get initial credentials for

principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522 -0500]

• ERR - slapi_ldap_bind - Error: could not send startTLS request:

error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514 -0500]

• ERR - set_krb5_creds - Could not get initial credentials for

principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb  4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_389)) Feb  4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432 -0500]

• ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication

Manager masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Feb  4 17:47:40 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:47:40 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:47:40 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Feb  4 17:47:40 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:47:50 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:47:50 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:47:50 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Feb  4 17:47:50 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:48:00 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:00 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:00 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Feb  4 17:48:00 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:48:10 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:10 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:10 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Feb  4 17:48:10 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb  4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb  4 17:48:20 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:20 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:20 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Feb  4 17:48:20 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb  4 17:48:30 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb  4 17:48:30 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb  4 17:48:30 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Feb  4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Feb  4 17:48:30 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb  4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) ^C [root@ipa2 log]# less /var/log/ipaupgrade.log

<p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:13Z DEBUG Waiting for CA to start... 2019-02-04T22:46:14Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:14Z DEBUG request body '' 2019-02-04T22:46:14Z DEBUG response status 500 2019-02-04T22:46:14Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:14 GMT Connection: close

2019-02-04T22:46:14Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>

<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>

<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:14Z DEBUG Waiting for CA to start... 2019-02-04T22:46:15Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:15Z DEBUG request body '' 2019-02-04T22:46:15Z DEBUG response status 500 2019-02-04T22:46:15Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:15 GMT Connection: close

2019-02-04T22:46:15Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>

<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>

<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:15Z DEBUG Waiting for CA to start... 2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-02-04T22:46:16Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute     return_value = self.run()   File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 56, in run     raise admintool.ScriptError(str(e))

2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Thanks, -Chris

...

Rob,

I'll be honest. I think you are suggesting an ldapsearch with this

Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster

sorry I've not figured out how to successfully ldapsearch :-(

Instead I did this: ipa config-show |grep 'CA renewal master'

It came up blank. I suspect I didn't have a renewal master somehow.

Then I did This: ipa-csreplica-manage set-renewal-master ipa2 (hostname of working IPA server)

Next is a "yum update" to be safe, and lastly "ipa-server-upgrade" on ipa2. When that's all done I'll try "yum update" and "ipa-server-upgrade" on my broken IPA system ipa1

I'll report back here when finished.

Thanks,

-Chris

...

Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster

You want to run the script on that master first to get the certs renewed.

I'd start by re-running ipa-server-upgrade. It is idempotent so there should be no risk. It may repair the tracking for you.

rob

On 2/4/2019 3:30 PM, Rob Crittenden wrote:

...

Chris Mohler via FreeIPA-users wrote:

...

Thanks for looking at my issue!

There have been no recent updates on my system. Actually I was getting ready to update when I noticed things weren't good.

Here is the output from the log of the most recent update. Looks like it was completed successfully. The lines you asked about are in Bold/underlined.

> 2018-07-18T16:55:21Z INFO [Update certmonger certificate renewal > configuration] > 2018-07-18T16:55:21Z DEBUG Loading Index file from > '/var/lib/ipa/sysrestore/sysrestore.index' > 2018-07-18T16:55:21Z DEBUG Starting external process > 2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d > /etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt > 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 > 2018-07-18T16:55:21Z DEBUG stdout= > Certificate Nickname Trust > Attributes > SSL,S/MIME,JAR/XPI > > caSigningCert cert-pki-ca CTu,Cu,Cu > subsystemCert cert-pki-ca u,u,u > ocspSigningCert cert-pki-ca u,u,u > auditSigningCert cert-pki-ca u,u,Pu > Server-Cert cert-pki-ca u,u,u > > 2018-07-18T16:55:21Z DEBUG stderr= > _*2018-07-18T16:55:21Z DEBUG Configuring certmonger to stop tracking > system certificates for CA*_ > 2018-07-18T16:55:21Z DEBUG Starting external process > 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start > messagebus.service > 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 > 2018-07-18T16:55:21Z DEBUG stdout= > 2018-07-18T16:55:21Z DEBUG stderr= > 2018-07-18T16:55:21Z DEBUG Starting external process > 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active > messagebus.service > 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 > 2018-07-18T16:55:21Z DEBUG stdout=active > > 2018-07-18T16:55:21Z DEBUG stderr= > 2018-07-18T16:55:21Z DEBUG Starting external process > 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start > certmonger.service > 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 > 2018-07-18T16:55:21Z DEBUG stdout= > 2018-07-18T16:55:21Z DEBUG stderr= > 2018-07-18T16:55:21Z DEBUG Starting external process > 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active > certmonger.service > 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 > 2018-07-18T16:55:21Z DEBUG stdout=active > -snip- a few more lines like the section above. > 2018-07-18T16:55:25Z DEBUG stderr= > 2018-07-18T16:55:30Z DEBUG Loading Index file from > '/var/lib/ipa/sysrestore/sysrestore.index' > 2018-07-18T16:55:30Z DEBUG Starting external process > 2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d > /etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f > /etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt > 2018-07-18T16:55:30Z DEBUG Process finished, return code=0 > 2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN CERTIFICATE----- -Snip- Cert and Key stuff goes here- > 2018-07-18T16:55:34Z DEBUG stderr= > _*2018-07-18T16:55:35Z INFO Certmonger certificate renewal > configuration updated*_

Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster

You want to run the script on that master first to get the certs renewed.

I'd start by re-running ipa-server-upgrade. It is idempotent so there should be no risk. It may repair the tracking for you.

rob

...

On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote: > On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users wrote: >> Hi Everyone, >> >> I'm looking for some help. I'm having trouble with everything >> basically. >> >> I think one of my CA's certs expired or something. I can't kinit >> admin, I can't login via the WebGui. If I "getcert list" it returns >> "Number of certificates and requests being tracked: 0." >> >> This all started happening a few days ago and I am at a loss as to >> what happened. On a whim I set the system date and time back a few >> months to see if my certs were expired and like magic I can login to >> the Webgui but I'm still not tracking anything with "getcert list" I >> suspect the cert has expired but without tracking it I can't tell, or >> renew it. >> > Hi, > > can you check if an upgrade happened recently (have a look at > /var/log/ipaupgrade.log)? The upgrade stop tracking certs and > re-configures certmonger, so if it failed in the middle you may be > left without any tracking. > You should be able to find lines like the following if the > untracking/tracking went fine: > --- > [Update certmonger certificate renewal configuration] > Configuring certmonger to stop tracking system certificates for CA > Certmonger certificate renewal configuration updated > --- > > HTH, > flo > >> Please help >> >> I'm running Centos 7, FreeIPA 4.5.4 >> >> Thanks, >> >> -Chris >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave@lists.fedorahosted.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >> >> _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

I have not been able the get the expired certs renewed. I would appreciate any help or advice that you have.

Thanks,

-Chris

Sorry for the delay and multiple posts. I'm having some trouble with my mail client.

thanks again for all the help

As requested Here is the output from getcert list on the CA renewal master:

Number of certificates and requests being tracked: 9. Request ID '20180131032610':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=CA Audit,O=domain.com         expires: 2018-12-31 13:28:03 UTC         key usage: digitalSignature,nonRepudiation         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032614':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=OCSP Subsystem,O=domain.com         expires: 2018-12-31 13:26:43 UTC         eku: id-kp-OCSPSigning         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032615':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=CA Subsystem,O=domain.com         expires: 2018-12-31 13:26:53 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032616':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=Certificate Authority,O=domain.com         expires: 2038-12-31 03:18:40 UTC         key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032623':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=IPA RA,O=domain.com         expires: 2018-12-31 13:27:15 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre         post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert         track: yes         auto-renew: yes Request ID '20180131032624':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2019-06-25 15:44:03 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032626':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-CS-OBERLIN-EDU/pwdfile.txt'         certificate: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2019-07-06 15:22:41 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv CS-OBERLIN-EDU         track: yes         auto-renew: yes Request ID '20180131032637':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'         certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2019-07-06 15:22:43 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_httpd         track: yes         auto-renew: yes Request ID '20180131032703':         status: MONITORING         stuck: no         key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'         CA: SelfSign         issuer: CN=ipa1.domain.com,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2020-02-05 02:11:51 UTC         principal name: krbtgt/domain.com@domain.com         certificate template/profile: KDCs_PKINIT_Certs         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert         track: yes         auto-renew: yes

Now the output from journalctl -u certmonger:

-- Logs begin at Sun 2018-12-30 22:18:38 EST, end at Fri 2019-02-08 11:34:16 EST. -- Dec 30 22:18:46 ipa1.domain.com systemd[1]: Starting Certificate monitoring and PKI enrollment... Dec 30 22:18:47 ipa1.domain.com systemd[1]: Started Certificate monitoring and PKI enrollment. Dec 30 22:19:06 ipa1.domain.com dogtag-ipa-ca-renew-agent-submit[4483]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): C Dec 30 22:19:06 ipa1.domain.com dogtag-ipa-ca-renew-agent-submit[4496]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): C Dec 30 22:19:06 ipa1.domain.com dogtag-ipa-ca-renew-agent-submit[4473]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): C Dec 30 22:19:06 ipa1.domain.com certmonger[3631]: 2018-12-30 22:19:06 [3631] Internal error Dec 30 22:19:06 ipa1.domain.com certmonger[3631]: 2018-12-30 22:19:06 [3631] Internal error Dec 30 22:19:06 ipa1.domain.com dogtag-ipa-ca-renew-agent-submit[4460]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): C Dec 30 22:19:06 ipa1.domain.com certmonger[3631]: 2018-12-30 22:19:06 [3631] Internal error Dec 30 22:19:06 ipa1.domain.com certmonger[3631]: 2018-12-30 22:19:06 [3631] Internal error Feb 05 02:50:09 ipa1.domain.com ipa-submit[16915]: GSSAPI client step 1

Lastly a few lines from /var/log/messages:

Feb  8 11:24:16 ipa1 dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Feb  8 11:24:16 ipa1 dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Feb  8 11:24:16 ipa1 certmonger: 2019-02-08 11:24:16 [3631] Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate. Feb  8 11:24:20 ipa1 dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Feb  8 11:24:20 ipa1 dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Feb  8 11:24:21 ipa1 certmonger: 2019-02-08 11:24:21 [3631] Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate. Feb  8 11:24:24 ipa1 dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Feb  8 11:24:25 ipa1 dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Feb  8 11:24:25 ipa1 certmonger: 2019-02-08 11:24:25 [3631] Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate. Feb  8 11:24:31 ipa1 dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Feb  8 11:24:31 ipa1 dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Feb  8 11:24:31 ipa1 certmonger: 2019-02-08 11:24:31 [3631] Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate. Feb  8 11:25:34 ipa1 ns-slapd: [08/Feb/2019:11:25:34.227979399 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected)

Sorry for the delay, It's been busy and I was not able to take the server down for even a few minutes to check.

Here is the output from the commands you asked about:

[root@ipa1 user]# date Thu Dec 20 11:27:02 EST 2018 [root@ipa1 user]# ipactl stop Stopping ipa-otpd Service Stopping pki-tomcatd Service Stopping ntpd Service Stopping ipa-custodia Service Stopping httpd Service Stopping kadmin Service Stopping krb5kdc Service Stopping Directory Service ipa: INFO: The ipactl command was successful [root@ipa1 user]# systemctl start dirsrv@domain.com httpd krb5kdc pki-tomcatd@pki-tomcat Job for dirsrv@domain.com.service failed because a configured resource limit was exceeded. See "systemctl status dirsrv@domain.com.service" and "journalctl -xe" for details. Job for krb5kdc.service failed because the control process exited with error code. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.

[root@ipa1 user]# systemctl status dirsrv@domain.com.service ● dirsrv@domain.com.service - 389 Directory Server domain.com.    Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)    Active: failed (Result: resources)

Dec 20 11:28:36 ipa1.domain.com systemd[1]: Failed to load environment files: No such file or directory Dec 20 11:28:36 ipa1.domain.com systemd[1]: dirsrv@domain.com.service failed to run 'start-pre' task: No such file or directory Dec 20 11:28:36 ipa1.domain.com systemd[1]: Failed to start 389 Directory Server domain.com.. Dec 20 11:28:36 ipa1.domain.com systemd[1]: Unit dirsrv@domain.com.service entered failed state. Dec 20 11:28:36 ipa1.domain.com systemd[1]: dirsrv@domain.com.service failed.

[root@ipa1 user]# journalctl -xe Dec 20 11:28:37 ipa1.domain.com server[13685]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java Dec 20 11:28:37 ipa1.domain.com server[13685]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar Dec 20 11:28:37 ipa1.domain.com server[13685]: main class used: org.apache.catalina.startup.Bootstrap Dec 20 11:28:37 ipa1.domain.com server[13685]: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni Dec 20 11:28:37 ipa1.domain.com server[13685]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-to Dec 20 11:28:37 ipa1.domain.com server[13685]: arguments used: start Dec 20 11:28:37 ipa1.domain.com systemd[1]: Started The Apache HTTP Server. -- Subject: Unit httpd.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit httpd.service has finished starting up. -- -- The start-up result is done. Dec 20 11:28:37 ipa1.domain.com polkitd[3328]: Unregistered Authentication Agent for unix-process:13541:73065620 (system bus name :1.540, object path /org/freedesktop/PolicyKit1/AuthenticationA Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://ipa1.domain.com:9080/ca/ocsp' did not Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not fi Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'true' did not find a matching property. Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=false,ssl3=false,tls=true' did not find a matchin Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_W Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_W Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeStream' to 'tls1_0:tls1_2' did not find a matching prop Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeDatagram' to 'tls1_1:tls1_2' did not find a matching pr Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslRangeCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_EC Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.co Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' d Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. Dec 20 11:28:38 ipa1.domain.com server[13685]: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. Dec 20 11:28:43 ipa1.domain.com server[13685]: CMSEngine.initializePasswordStore() begins Dec 20 11:28:43 ipa1.domain.com server[13685]: CMSEngine.initializePasswordStore(): tag=internaldb Dec 20 11:28:43 ipa1.domain.com server[13685]: CMSEngine.initializePasswordStore(): tag=replicationdb Dec 20 11:28:43 ipa1.domain.com server[13685]: Internal Database Error encountered: Could not connect to LDAP server host ipa1.domain.com port 636 Error netscape.ldap.LDAPException: Unable Dec 20 11:28:55 ipa1.domain.com server[13685]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1b352f background process Dec 20 11:28:55 ipa1.domain.com server[13685]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:55 ipa1.domain.com server[13685]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:55 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:55 ipa1.domain.com server[13685]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:55 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:55 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:55 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:55 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:55 ipa1.domain.com server[13685]: at java.lang.Thread.run(Thread.java:748) Dec 20 11:29:05 ipa1.domain.com server[13685]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1b352f background process Dec 20 11:29:05 ipa1.domain.com server[13685]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:29:05 ipa1.domain.com server[13685]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:29:05 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:29:05 ipa1.domain.com server[13685]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:29:05 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:29:05 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:05 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:05 ipa1.domain.com server[13685]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:29:05 ipa1.domain.com server[13685]: at java.lang.Thread.run(Thread.java:748) [root@ipa1 user]# systemctl status ipa1@domain.com.service Unit ipa1@domain.com.service could not be found.

[root@ipa1 user]# getcert list Number of certificates and requests being tracked: 9. Request ID '20180131032610':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=CA Audit,O=domain.com         expires: 2018-12-31 13:28:03 UTC         key usage: digitalSignature,nonRepudiation         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032614':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=OCSP Subsystem,O=domain.com         expires: 2018-12-31 13:26:43 UTC         eku: id-kp-OCSPSigning         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032615':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=CA Subsystem,O=domain.com         expires: 2018-12-31 13:26:53 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032616':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=Certificate Authority,O=domain.com         expires: 2038-12-31 03:18:40 UTC         key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032623':         status: CA_UNREACHABLE         ca-error: Error 58 connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.         stuck: no         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=IPA RA,O=domain.com         expires: 2018-12-31 13:27:15 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre         post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert         track: yes         auto-renew: yes Request ID '20180131032624':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2019-06-25 15:44:03 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032626':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-CS-OBERLIN-EDU/pwdfile.txt'         certificate: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2019-07-06 15:22:41 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv CS-OBERLIN-EDU         track: yes         auto-renew: yes Request ID '20180131032637':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'         certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2019-07-06 15:22:43 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_httpd         track: yes         auto-renew: yes Request ID '20180131032703':         status: MONITORING         stuck: no         key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'         CA: SelfSign         issuer: CN=ipa1.domain.com,O=domain.com         subject: CN=ipa1.domain.com,O=domain.com         expires: 2020-02-05 02:11:51 UTC         principal name: krbtgt/domain.com@domain.com         certificate template/profile: KDCs_PKINIT_Certs         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert         track: yes         auto-renew: yes

Dec 20 11:32:55 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:32:55 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:32:55 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:32:55 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:33:05 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1b352f background process Dec 20 11:33:05 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:33:05 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:33:05 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:33:05 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:33:05 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:33:05 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:05 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:05 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:33:05 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:33:10 ipa1 systemd: Starting Kerberos 5 KDC... Dec 20 11:33:10 ipa1 krb5kdc: krb5kdc: cannot initialize realm domain.com - see log file for details Dec 20 11:33:10 ipa1 systemd: krb5kdc.service: control process exited, code=exited status=1 Dec 20 11:33:10 ipa1 systemd: Failed to start Kerberos 5 KDC. Dec 20 11:33:10 ipa1 systemd: Unit krb5kdc.service entered failed state. Dec 20 11:33:10 ipa1 systemd: krb5kdc.service failed. Dec 20 11:33:15 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1b352f background process Dec 20 11:33:15 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:33:15 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:33:15 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:33:15 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:33:15 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:33:15 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:15 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:15 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:33:15 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:33:22 ipa1 systemd: Failed to load environment files: No such file or directory Dec 20 11:33:22 ipa1 systemd: dirsrv@domain.com.service failed to run 'start-pre' task: No such file or directory Dec 20 11:33:22 ipa1 systemd: Failed to start 389 Directory Server domain.com.. Dec 20 11:33:22 ipa1 systemd: dirsrv@domain.com.service failed. Dec 20 11:33:22 ipa1 systemd: Starting Kerberos 5 KDC... Dec 20 11:33:22 ipa1 krb5kdc: krb5kdc: cannot initialize realm domain.com - see log file for details Dec 20 11:33:22 ipa1 systemd: krb5kdc.service: control process exited, code=exited status=1 Dec 20 11:33:22 ipa1 systemd: Failed to start Kerberos 5 KDC. Dec 20 11:33:22 ipa1 systemd: Unit krb5kdc.service entered failed state. Dec 20 11:33:22 ipa1 systemd: krb5kdc.service failed. Dec 20 11:33:25 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1b352f background process Dec 20 11:33:25 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:33:25 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:33:25 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:33:25 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:33:25 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:33:25 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:25 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:25 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:33:25 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:33:35 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1b352f background process Dec 20 11:33:35 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:33:35 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:33:35 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:33:35 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:33:35 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:33:35 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:35 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:33:35 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:33:35 ipa1 server: at java.lang.Thread.run(Thread.java:748)

Try #2 Feb 13 08:54:23 ipa1 ns-slapd: [13/Feb/2019:08:54:23.983662956 -0500] - INFO - main - slapd stopped. Feb 13 08:54:24 ipa1 systemd: Stopped 389 Directory Server CS-OBERLIN-EDU.. Dec 20 11:27:00 ipa1 systemd: Time has been changed Dec 20 11:27:24 ipa1 systemd: Failed to load environment files: No such file or directory Dec 20 11:27:24 ipa1 systemd: dirsrv@domain.com.service failed to run 'start-pre' task: No such file or directory Dec 20 11:27:24 ipa1 systemd: Failed to start 389 Directory Server domain.com.. Dec 20 11:27:24 ipa1 systemd: Unit dirsrv@domain.com.service entered failed state. Dec 20 11:27:24 ipa1 systemd: dirsrv@domain.com.service failed. Dec 20 11:27:24 ipa1 systemd: Starting The Apache HTTP Server... Dec 20 11:27:24 ipa1 systemd: Starting Kerberos 5 KDC... Dec 20 11:27:24 ipa1 systemd: Starting PKI Tomcat Server pki-tomcat... Dec 20 11:27:24 ipa1 krb5kdc: krb5kdc: cannot initialize realm domain.com - see log file for details Dec 20 11:27:24 ipa1 systemd: krb5kdc.service: control process exited, code=exited status=1 Dec 20 11:27:24 ipa1 systemd: Failed to start Kerberos 5 KDC. Dec 20 11:27:24 ipa1 systemd: Unit krb5kdc.service entered failed state. Dec 20 11:27:24 ipa1 systemd: krb5kdc.service failed. Dec 20 11:27:25 ipa1 ipa-httpd-kdcproxy: ipa: WARNING: Unable to connect to dirsrv: cannot connect to 'ldapi://%2fvar%2frun%2fslapd-CS-OBERLIN-EDU.socket': Dec 20 11:27:25 ipa1 ipa-httpd-kdcproxy: ipa-httpd-kdcproxy: WARNING  Unable to connect to dirsrv: cannot connect to 'ldapi://%2fvar%2frun%2fslapd-CS-OBERLIN-EDU.socket': Dec 20 11:27:25 ipa1 ipa-httpd-kdcproxy: ipa: WARNING: Disabling KDC proxy Dec 20 11:27:25 ipa1 ipa-httpd-kdcproxy: ipa-httpd-kdcproxy: WARNING  Disabling KDC proxy Dec 20 11:27:25 ipa1 pkidaemon: ----------------------- Dec 20 11:27:25 ipa1 pkidaemon: Banner is not installed Dec 20 11:27:25 ipa1 pkidaemon: ----------------------- Dec 20 11:27:25 ipa1 pkidaemon: ---------------------- Dec 20 11:27:25 ipa1 pkidaemon: Enabled all subsystems Dec 20 11:27:25 ipa1 pkidaemon: ---------------------- Dec 20 11:27:25 ipa1 systemd: Started PKI Tomcat Server pki-tomcat. Dec 20 11:27:25 ipa1 server: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java Dec 20 11:27:25 ipa1 server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar Dec 20 11:27:25 ipa1 server: main class used: org.apache.catalina.startup.Bootstrap Dec 20 11:27:25 ipa1 server: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni Dec 20 11:27:25 ipa1 server: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util. logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager     -Djava.security.policy==/var/li b/pki/pki-tomcat/conf/catalina.policy Dec 20 11:27:25 ipa1 server: arguments used: start Dec 20 11:27:25 ipa1 systemd: Started The Apache HTTP Server. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://ipa1.domain.com:9080/ca/ocsp' did not find a matching prope rty. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'true' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=false,ssl3=false,tls=true' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD5,-SSL2_RC2_128_CBC_WITH_MD5,-SSL2_RC2_128_CBC_EXPORT40_WITH_MD5,-SSL2_DES_64_CBC_WITH_MD5,-SSL2_DES_192_EDE3_CBC_WITH_MD5' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_WITH_RC4_128_SHA,+SSL3_RSA_WITH_RC4_128_SHA,-SSL3_RSA_EXPORT_WITH_RC4_40_MD5,+SSL3_RSA_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_DES_CBC_SHA,-SSL3_RSA_EXPORT_WITH_RC2_CBC_40_MD5,-SSL3_FORTEZZA_DMS_WITH_FORTEZZA_CBC_SHA,-SSL_RSA_FIPS_WITH_DES_CBC_SHA,+SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_NULL_MD5,-TLS_RSA_EXPORT1024_WITH_RC4_56_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeStream' to 'tls1_0:tls1_2' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslVersionRangeDatagram' to 'tls1_1:tls1_2' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslRangeCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,-TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,-TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,-TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,-TLS_DHE_DSS_WITH_AES_128_CBC_SHA,-TLS_DHE_DSS_WITH_AES_256_CBC_SHA,-TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_DHE_RSA_WITH_AES_128_CBC_SHA,-TLS_DHE_RSA_WITH_AES_256_CBC_SHA,-TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,-TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,-TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,-TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,-TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,-TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,-TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,-TLS_RSA_WITH_AES_128_CBC_SHA256,-TLS_RSA_WITH_AES_256_CBC_SHA256,-TLS_RSA_WITH_AES_128_GCM_SHA256,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. Dec 20 11:27:26 ipa1 server: WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. Dec 20 11:27:31 ipa1 server: CMSEngine.initializePasswordStore() begins Dec 20 11:27:31 ipa1 server: CMSEngine.initializePasswordStore(): tag=internaldb Dec 20 11:27:31 ipa1 server: CMSEngine.initializePasswordStore(): tag=replicationdb Dec 20 11:27:31 ipa1 server: Internal Database Error encountered: Could not connect to LDAP server host ipa1.domain.com port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1) Dec 20 11:27:42 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:27:42 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:27:42 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:27:42 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:27:42 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:27:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:27:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:27:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:27:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:27:42 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:27:52 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:27:52 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:27:52 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:27:52 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:27:52 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:27:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:27:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:27:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:27:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:27:52 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:28:02 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:28:02 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:02 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:02 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:02 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:02 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:28:12 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:28:12 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:12 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:12 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:12 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:12 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:28:22 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:28:22 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:22 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:22 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:22 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:22 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:28:32 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:28:32 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:32 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:32 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:32 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:28:36 ipa1 systemd: Stopping Certificate monitoring and PKI enrollment... Dec 20 11:28:36 ipa1 systemd: Stopped Certificate monitoring and PKI enrollment. Dec 20 11:28:36 ipa1 systemd: Starting Certificate monitoring and PKI enrollment... Dec 20 11:28:37 ipa1 systemd: Started Certificate monitoring and PKI enrollment. Dec 20 11:28:42 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:28:42 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:42 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:42 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:42 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:42 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:42 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:28:47 ipa1 sssd[be[domain.com]]: Backend is offline Dec 20 11:28:52 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:28:52 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:52 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:52 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:52 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:52 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:52 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:28:54 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:28:54 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:28:54 ipa1 certmonger: 2018-12-20 11:28:54 [14640] Internal error Dec 20 11:28:55 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:28:55 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:28:55 ipa1 certmonger: 2018-12-20 11:28:55 [14640] Internal error Dec 20 11:28:55 ipa1 certmonger: 2018-12-20 11:28:55 [14640] Internal error Dec 20 11:28:55 ipa1 certmonger: 2018-12-20 11:28:55 [14640] Internal error Dec 20 11:29:02 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:29:02 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:29:02 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:29:02 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:29:02 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:29:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:29:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:02 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:29:02 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:29:12 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:29:12 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:29:12 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:29:12 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:29:12 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:29:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:29:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:12 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:29:12 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:29:22 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:29:22 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:29:22 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:29:22 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:29:22 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:29:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:29:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:22 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:29:22 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:29:32 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:29:32 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:29:32 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:29:32 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:29:32 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:29:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:29:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:32 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:29:32 ipa1 server: at java.lang.Thread.run(Thread.java:748)

less krb5kdc.log Feb 13 08:54:21 ipa1.domain.com krb5kdc[9604](info): IPA certauth plugin un-loaded. krb5kdc: Server error - while fetching master key K/M for realm domain.com

certmonger start related in log/messages Dec 20 11:35:00 ipa1 systemd: Stopped Certificate monitoring and PKI enrollment. Dec 20 11:35:00 ipa1 systemd: Starting Certificate monitoring and PKI enrollment... Dec 20 11:35:00 ipa1 systemd: Started Certificate monitoring and PKI enrollment. Dec 20 11:35:03 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:35:03 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:35:03 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:35:03 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:35:03 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:35:03 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:35:03 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:35:03 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:35:03 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:35:03 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:35:13 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:35:13 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:35:13 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:35:13 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:35:13 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:35:13 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:35:13 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:35:13 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:35:13 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:35:13 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:35:18 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:35:18 ipa1 certmonger: 2018-12-20 11:35:18 [14932] Internal error Dec 20 11:35:18 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:35:18 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:35:18 ipa1 certmonger: 2018-12-20 11:35:18 [14932] Internal error Dec 20 11:35:18 ipa1 certmonger: 2018-12-20 11:35:18 [14932] Internal error Dec 20 11:35:23 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@261f5a39 background process Dec 20 11:35:23 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:35:23 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:35:23 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:35:23 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:35:23 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:35:23 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:35:23 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:35:23 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:35:23 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:35:28 ipa1 dogtag-ipa-ca-renew-agent-submit: Traceback (most recent call last):#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module>#012    sys.exit(main())#012  File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main#012    kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)#012  File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in kinit_keytab#012    cred = gssapi.Credentials(name=name, store=store, usage='initiate')#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__#012    store=store)#012  File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire#012    usage)#012  File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)#012GSSError: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'domain.com' Dec 20 11:35:28 ipa1 certmonger: 2018-12-20 11:35:28 [14932] Internal error

Thanks again for looking.

-Chris

Here are some fresh errors. With the correct domain.com

[root@ipa1 user]# systemctl start dirsrv@CS-OBERLIN-EDU httpd krb5kdc pki-tomcatd@pki-tomcat Job for krb5kdc.service failed because the control process exited with error code. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.

[root@ipa1 user]# systemctl status krb5kdc.service ● krb5kdc.service - Kerberos 5 KDC    Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled)    Active: failed (Result: exit-code) since Thu 2018-12-20 11:29:39 EST; 20s ago   Process: 6328 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=1/FAILURE)

Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: Starting Kerberos 5 KDC... Dec 20 11:29:39 ipa1.cs.oberlin.edu krb5kdc[6328]: krb5kdc: cannot initialize realm CS.OBERLIN.EDU - see log file for details Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: krb5kdc.service: control process exited, code=exited status=1 Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: Failed to start Kerberos 5 KDC. Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: Unit krb5kdc.service entered failed state. Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: krb5kdc.service failed.

[root@ipa1 user]# journalctl -xe Dec 20 11:29:44 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:44.896674586 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=cs,dc=oberlin,dc=edu does not exist Dec 20 11:29:44 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:44.899381296 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=cs,dc=oberlin,dc=edu does not exist Dec 20 11:29:44 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:44.903264399 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=ad,cn=etc,dc=cs,dc=oberlin,dc=edu does not exist Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.040709963 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.048762794 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=cs,dc=ober Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.100326075 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.111839823 -0500] - INFO - slapd_daemon - slapd started.  Listening on All Interfaces port 389 for LDAP requests Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.117194087 -0500] - INFO - slapd_daemon - Listening on All Interfaces port 636 for LDAPS requests Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.119103939 -0500] - INFO - slapd_daemon - Listening on /var/run/slapd-CS-OBERLIN-EDU.socket for LDAPI requests Dec 20 11:29:45 ipa1.cs.oberlin.edu systemd[1]: Started 389 Directory Server CS-OBERLIN-EDU.. -- Subject: Unit dirsrv@CS-OBERLIN-EDU.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit dirsrv@CS-OBERLIN-EDU.service has finished starting up. -- -- The start-up result is done. Dec 20 11:29:45 ipa1.cs.oberlin.edu polkitd[3318]: Unregistered Authentication Agent for unix-process:6321:1000965 (system bus name :1.37, object path /org/freedesktop/PolicyKit1/AuthenticationAgen Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.143848383 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.146250152 -0500] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=masterAgreement1-ipa.cs.oberlin.edu-pki-tomc Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.221754444 -0500] - ERR - schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:48.233735556 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:48.240356284 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at java.lang.Thread.run(Thread.java:748) Dec 20 11:29:51 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:51.176732756 -0500] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=cs,dc=oberlin,d Dec 20 11:29:51 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:51.179418586 -0500] - ERR - schema-compat-plugin - Finished plugin initialization. Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:54.250696593 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:54.257694833 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at java.lang.Thread.run(Thread.java:748) Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:30:06.269784404 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:30:06.278382823 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 [root@ipa1 user]#

KRb Log Dec 20 11:31:18 ipa1 ns-slapd: [20/Dec/2018:11:31:18.366362542 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBERLIN.EDU] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Dec 20 11:31:18 ipa1 ns-slapd: [20/Dec/2018:11:31:18.373255123 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) Dec 20 11:31:18 ipa1 ns-slapd: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_389)) Dec 20 11:31:21 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:31:21 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:31:21 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:31:21 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:31:31 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:31:31 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:31:31 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:31:31 ipa1 server: at java.lang.Thread.run(Thread.java:748)

dirserv log Feb 13 09:11:57 ipa1.cs.oberlin.edu krb5kdc[4246](info): IPA certauth plugin un-loaded. krb5kdc: Server error - while fetching master key K/M for realm CS.OBERLIN.EDU krb5kdc: Server error - while fetching master key K/M for realm CS.OBERLIN.EDU krb5kdc: Server error - while fetching master key K/M for realm CS.OBERLIN.EDU (END)

systemctl restart certmonger Dec 20 11:28:11 ipa1 systemd: Stopping Certificate monitoring and PKI enrollment... Dec 20 11:28:11 ipa1 systemd: Stopped Certificate monitoring and PKI enrollment. Dec 20 11:28:11 ipa1 systemd: Starting Certificate monitoring and PKI enrollment... Dec 20 11:28:12 ipa1 systemd: Started Certificate monitoring and PKI enrollment. Dec 20 11:28:21 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@43df7c0 background process Dec 20 11:28:21 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:21 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Dec 20 11:28:21 ipa1 server: at java.lang.Thread.run(Thread.java:748)

getcert list: [root@ipa1 user]# getcert list Number of certificates and requests being tracked: 9. Request ID '20180131032610':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=CA Audit,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:28:03 UTC         key usage: digitalSignature,nonRepudiation         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032614':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=OCSP Subsystem,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:26:43 UTC         eku: id-kp-OCSPSigning         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032615':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=CA Subsystem,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:26:53 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032616':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=Certificate Authority,O=CS.OBERLIN.EDU         expires: 2038-12-31 03:18:40 UTC         key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032623':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=IPA RA,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:27:15 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre         post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert         track: yes         auto-renew: yes Request ID '20180131032624':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2019-06-25 15:44:03 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032626':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-CS-OBERLIN-EDU/pwdfile.txt'         certificate: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2019-07-06 15:22:41 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv CS-OBERLIN-EDU         track: yes         auto-renew: yes Request ID '20180131032637':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'         certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2019-07-06 15:22:43 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_httpd         track: yes         auto-renew: yes Request ID '20180131032703':         status: MONITORING         stuck: no         key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'         CA: SelfSign         issuer: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2020-02-05 02:11:51 UTC         principal name: krbtgt/CS.OBERLIN.EDU@CS.OBERLIN.EDU         certificate template/profile: KDCs_PKINIT_Certs         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert         track: yes         auto-renew: yes

On 2/13/2019 11:20 AM, Rob Crittenden wrote:

Chris Mohler via FreeIPA-users wrote:

...

Sorry for the delay, It's been busy and I was not able to take the server down for even a few minutes to check.

Here is the output from the commands you asked about:

[root@ipa1 user]# date Thu Dec 20 11:27:02 EST 2018 [root@ipa1 user]# ipactl stop Stopping ipa-otpd Service Stopping pki-tomcatd Service Stopping ntpd Service Stopping ipa-custodia Service Stopping httpd Service Stopping kadmin Service Stopping krb5kdc Service Stopping Directory Service ipa: INFO: The ipactl command was successful [root@ipa1 user]# systemctl start dirsrv@domain.com httpd krb5kdc pki-tomcatd@pki-tomcat Job for dirsrv@domain.com.service failed because a configured resource limit was exceeded. See "systemctl status dirsrv@domain.com.service" and "journalctl -xe" for details. Job for krb5kdc.service failed because the control process exited with error code. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.

[root@ipa1 user]# systemctl status dirsrv@domain.com.service ● dirsrv@domain.com.service - 389 Directory Server domain.com.    Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)    Active: failed (Result: resources)

Sorry, that's the wrong service.

systemctl -a |grep dirsrv

Start the one appropriate for your realm.

Without dirsrv nothing else will work so the rest of the errors are expected.

rob

Sorry,

It was a bit hidden in my last error dump but the /var/log/krb5kdc.log is here if that helps any. I'll redo the procedure and start one service at a time and post the results ASAP.

KRb Log Dec 20 11:31:18 ipa1 ns-slapd: [20/Dec/2018:11:31:18.366362542 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBERLIN.EDU] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Dec 20 11:31:18 ipa1 ns-slapd: [20/Dec/2018:11:31:18.373255123 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) Dec 20 11:31:18 ipa1 ns-slapd: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_389)) Dec 20 11:31:21 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:31:21 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:31:21 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Dec 20 11:31:21 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:31:31 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:31:31 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:31:31 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Dec 20 11:31:31 ipa1 server: at java.lang.Thread.run(Thread.java:748)

On 2/14/2019 4:24 PM, Rob Crittenden wrote:

Chris Mohler via FreeIPA-users wrote:

...

Here are some fresh errors. With the correct domain.com

[root@ipa1 user]# systemctl start dirsrv@CS-OBERLIN-EDU httpd krb5kdc pki-tomcatd@pki-tomcat Job for krb5kdc.service failed because the control process exited with error code. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.

[root@ipa1 user]# systemctl status krb5kdc.service ● krb5kdc.service - Kerberos 5 KDC    Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled)    Active: failed (Result: exit-code) since Thu 2018-12-20 11:29:39 EST; 20s ago   Process: 6328 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=1/FAILURE)

The KDC didn't start. The log is in /var/log/krb5kdc.log

Rather than starting everything on one line I suppose it would be better to start dirsrv, wait for it to settle down, then one by one start the other services.

rob

...

Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: Starting Kerberos 5 KDC... Dec 20 11:29:39 ipa1.cs.oberlin.edu krb5kdc[6328]: krb5kdc: cannot initialize realm CS.OBERLIN.EDU - see log file for details Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: krb5kdc.service: control process exited, code=exited status=1 Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: Failed to start Kerberos 5 KDC. Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: Unit krb5kdc.service entered failed state. Dec 20 11:29:39 ipa1.cs.oberlin.edu systemd[1]: krb5kdc.service failed.

[root@ipa1 user]# journalctl -xe Dec 20 11:29:44 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:44.896674586 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=cs,dc=oberlin,dc=edu does not exist Dec 20 11:29:44 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:44.899381296 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=cs,dc=oberlin,dc=edu does not exist Dec 20 11:29:44 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:44.903264399 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=ad,cn=etc,dc=cs,dc=oberlin,dc=edu does not exist Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.040709963 -0500] - ERR - NSACLPlugin - acl_parse - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.048762794 -0500] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=cs,dc=ober Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.100326075 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.111839823 -0500] - INFO - slapd_daemon - slapd started.  Listening on All Interfaces port 389 for LDAP requests Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.117194087 -0500] - INFO - slapd_daemon - Listening on All Interfaces port 636 for LDAPS requests Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.119103939 -0500] - INFO - slapd_daemon - Listening on /var/run/slapd-CS-OBERLIN-EDU.socket for LDAPI requests Dec 20 11:29:45 ipa1.cs.oberlin.edu systemd[1]: Started 389 Directory Server CS-OBERLIN-EDU.. -- Subject: Unit dirsrv@CS-OBERLIN-EDU.service has finished start-up -- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit dirsrv@CS-OBERLIN-EDU.service has finished starting up.

-- The start-up result is done. Dec 20 11:29:45 ipa1.cs.oberlin.edu polkitd[3318]: Unregistered Authentication Agent for unix-process:6321:1000965 (system bus name :1.37, object path /org/freedesktop/PolicyKit1/AuthenticationAgen Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.143848383 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.146250152 -0500] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=masterAgreement1-ipa.cs.oberlin.edu-pki-tomc Dec 20 11:29:45 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:45.221754444 -0500] - ERR - schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds! Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:48.233735556 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:48.240356284 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:48 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Dec 20 11:29:51 ipa1.cs.oberlin.edu server[6160]: at java.lang.Thread.run(Thread.java:748) Dec 20 11:29:51 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:51.176732756 -0500] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=cs,dc=oberlin,d Dec 20 11:29:51 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:51.179418586 -0500] - ERR - schema-compat-plugin - Finished plugin initialization. Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:54.250696593 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:29:54.257694833 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:29:54 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Dec 20 11:30:01 ipa1.cs.oberlin.edu server[6160]: at java.lang.Thread.run(Thread.java:748) Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:30:06.269784404 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBER Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: [20/Dec/2018:11:30:06.278382823 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) err Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI client step 1 Dec 20 11:30:06 ipa1.cs.oberlin.edu ns-slapd[6334]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_38 [root@ipa1 user]#

KRb Log Dec 20 11:31:18 ipa1 ns-slapd: [20/Dec/2018:11:31:18.366362542 -0500] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.cs.oberlin.edu@CS.OBERLIN.EDU] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Dec 20 11:31:18 ipa1 ns-slapd: [20/Dec/2018:11:31:18.373255123 -0500] - ERR - slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) Dec 20 11:31:18 ipa1 ns-slapd: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_389)) Dec 20 11:31:21 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:31:21 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:31:21 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Dec 20 11:31:21 ipa1 server: at java.lang.Thread.run(Thread.java:748) Dec 20 11:31:31 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@20492fd8 background process Dec 20 11:31:31 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:31:31 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:31:31 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Dec 20 11:31:31 ipa1 server: at java.lang.Thread.run(Thread.java:748)

dirserv log Feb 13 09:11:57 ipa1.cs.oberlin.edu krb5kdc[4246](info): IPA certauth plugin un-loaded. krb5kdc: Server error - while fetching master key K/M for realm CS.OBERLIN.EDU krb5kdc: Server error - while fetching master key K/M for realm CS.OBERLIN.EDU krb5kdc: Server error - while fetching master key K/M for realm CS.OBERLIN.EDU (END)

systemctl restart certmonger Dec 20 11:28:11 ipa1 systemd: Stopping Certificate monitoring and PKI enrollment... Dec 20 11:28:11 ipa1 systemd: Stopped Certificate monitoring and PKI enrollment. Dec 20 11:28:11 ipa1 systemd: Starting Certificate monitoring and PKI enrollment... Dec 20 11:28:12 ipa1 systemd: Started Certificate monitoring and PKI enrollment. Dec 20 11:28:21 ipa1 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@43df7c0 background process Dec 20 11:28:21 ipa1 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Dec 20 11:28:21 ipa1 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)

Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)

Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)

Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)

Dec 20 11:28:21 ipa1 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)

Dec 20 11:28:21 ipa1 server: at java.lang.Thread.run(Thread.java:748)

getcert list: [root@ipa1 user]# getcert list Number of certificates and requests being tracked: 9. Request ID '20180131032610':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=CA Audit,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:28:03 UTC         key usage: digitalSignature,nonRepudiation         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032614':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=OCSP Subsystem,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:26:43 UTC         eku: id-kp-OCSPSigning         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032615':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=CA Subsystem,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:26:53 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032616':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=Certificate Authority,O=CS.OBERLIN.EDU         expires: 2038-12-31 03:18:40 UTC         key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032623':         status: CA_UNREACHABLE         ca-error: Internal error         stuck: no         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=IPA RA,O=CS.OBERLIN.EDU         expires: 2018-12-31 13:27:15 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre         post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert         track: yes         auto-renew: yes Request ID '20180131032624':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'         CA: dogtag-ipa-ca-renew-agent         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2019-06-25 15:44:03 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"         track: yes         auto-renew: yes Request ID '20180131032626':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-CS-OBERLIN-EDU/pwdfile.txt'         certificate: type=NSSDB,location='/etc/dirsrv/slapd-CS-OBERLIN-EDU',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2019-07-06 15:22:41 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv CS-OBERLIN-EDU         track: yes         auto-renew: yes Request ID '20180131032637':         status: MONITORING         stuck: no         key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'         certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'         CA: IPA         issuer: CN=Certificate Authority,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2019-07-06 15:22:43 UTC         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment         eku: id-kp-serverAuth,id-kp-clientAuth         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/restart_httpd         track: yes         auto-renew: yes Request ID '20180131032703':         status: MONITORING         stuck: no         key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'         CA: SelfSign         issuer: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         subject: CN=ipa1.cs.oberlin.edu,O=CS.OBERLIN.EDU         expires: 2020-02-05 02:11:51 UTC         principal name: krbtgt/CS.OBERLIN.EDU@CS.OBERLIN.EDU         certificate template/profile: KDCs_PKINIT_Certs         pre-save command:         post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert         track: yes         auto-renew: yes

On 2/13/2019 11:20 AM, Rob Crittenden wrote:

...

Chris Mohler via FreeIPA-users wrote:

...

Sorry for the delay, It's been busy and I was not able to take the server down for even a few minutes to check.

Here is the output from the commands you asked about:

[root@ipa1 user]# date Thu Dec 20 11:27:02 EST 2018 [root@ipa1 user]# ipactl stop Stopping ipa-otpd Service Stopping pki-tomcatd Service Stopping ntpd Service Stopping ipa-custodia Service Stopping httpd Service Stopping kadmin Service Stopping krb5kdc Service Stopping Directory Service ipa: INFO: The ipactl command was successful [root@ipa1 user]# systemctl start dirsrv@domain.com httpd krb5kdc pki-tomcatd@pki-tomcat Job for dirsrv@domain.com.service failed because a configured resource limit was exceeded. See "systemctl status dirsrv@domain.com.service" and "journalctl -xe" for details. Job for krb5kdc.service failed because the control process exited with error code. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.

[root@ipa1 user]# systemctl status dirsrv@domain.com.service ● dirsrv@domain.com.service - 389 Directory Server domain.com.     Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)     Active: failed (Result: resources)

Sorry, that's the wrong service.

systemctl -a |grep dirsrv

Start the one appropriate for your realm.

Without dirsrv nothing else will work so the rest of the errors are expected.

rob

---

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...