I have not been able to renew the expired certificates yet. I would appreciate help if possible.
Followup summary:
Q: Seems like part of the problem is that the KDC was not running. Had you done ipactl stop prior to the upgrade?
A: I could not get the KDC to stay running. So yes it was off during the upgrade.
Q: Did it end up creating the tracking? Are there expired certs?
A: I was able to get the upgrade to finish successfully, after restoring the server from VM snapshot, rolling back the system date, and trying the update again. It did create the cert tracking!!! Yes there are expired certs.
Q: As an aside, I'd have suggest deferring the package upgrade until after the other things were sorted. It just adds another moving part. Water under the bridge now.
A: Yes sorry.
On 2/5/2019 11:18 AM, Rob Crittenden wrote:
Chris Mohler wrote:
Well... That was a mess.
The ipa-server-upgrade didn't go so well. It failed and now my ca-replication master is broken. Here are the details. Any hope?
Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/11]: stopping directory server [2/11]: saving configuration [3/11]: disabling listeners [4/11]: enabling DS global lock [5/11]: disabling Schema Compat [6/11]: starting directory server [7/11]: updating schema [8/11]: upgrading server [9/11]: stopping directory server [10/11]: restoring configuration [11/11]: starting directory server Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. CA did not start in 300.0s The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Seems like part of the problem is that the KDC was not running. Had you done ipactl stop prior to the upgrade?
Did it end up creating the tracking? Are there expired certs?
As an aside, I'd have suggest deferring the package upgrade until after the other things were sorted. It just adds another moving part. Water under the bridge now.
rob
Here is a wall of errors from my /var/log/ipaupgrade.log
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522 -0500]
- ERR - slapi_ldap_bind - Error: could not send startTLS request:
error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb 4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_389)) Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432 -0500]
- ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication
Manager masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Feb 4 17:47:40 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:47:40 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:47:40 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:40 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:47:50 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:47:50 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:47:50 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:50 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:48:00 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:00 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:00 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:00 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:48:10 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:10 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:10 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:10 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb 4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb 4 17:48:20 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:20 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:20 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:20 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:48:30 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:30 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:30 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:30 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) ^C [root@ipa2 log]# less /var/log/ipaupgrade.log
<p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:13Z DEBUG Waiting for CA to start... 2019-02-04T22:46:14Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:14Z DEBUG request body '' 2019-02-04T22:46:14Z DEBUG response status 500 2019-02-04T22:46:14Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:14 GMT Connection: close
2019-02-04T22:46:14Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:14Z DEBUG Waiting for CA to start... 2019-02-04T22:46:15Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:15Z DEBUG request body '' 2019-02-04T22:46:15Z DEBUG response status 500 2019-02-04T22:46:15Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:15 GMT Connection: close
2019-02-04T22:46:15Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:15Z DEBUG Waiting for CA to start... 2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-02-04T22:46:16Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 56, in run raise admintool.ScriptError(str(e))
2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Thanks, -Chris
Rob,
I'll be honest. I think you are suggesting an ldapsearch with this
Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster
sorry I've not figured out how to successfully ldapsearch
Instead I did this: ipa config-show |grep 'CA renewal master'
It came up blank. I suspect I didn't have a renewal master somehow.
Then I did This: ipa-csreplica-manage set-renewal-master ipa2 (hostname of working IPA server)
Next is a "yum update" to be safe, and lastly "ipa-server-upgrade" on ipa2. When that's all done I'll try "yum update" and "ipa-server-upgrade" on my broken IPA system ipa1
I'll report back here when finished.
Thanks,
-Chris
Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster
You want to run the script on that master first to get the certs renewed.
I'd start by re-running ipa-server-upgrade. It is idempotent so there should be no risk. It may repair the tracking for you.
rob
On 2/4/2019 3:30 PM, Rob Crittenden wrote:
Chris Mohler via FreeIPA-users wrote: > Thanks for looking at my issue! > > There have been no recent updates on my system. Actually I was > getting > ready to update when I noticed things weren't good. > > Here is the output from the log of the most recent update. Looks > like it > was completed successfully. The lines you asked about are in > Bold/underlined. > >> 2018-07-18T16:55:21Z INFO [Update certmonger certificate renewal >> configuration] >> 2018-07-18T16:55:21Z DEBUG Loading Index file from >> '/var/lib/ipa/sysrestore/sysrestore.index' >> 2018-07-18T16:55:21Z DEBUG Starting external process >> 2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d >> /etc/pki/pki-tomcat/alias -L -f >> /etc/pki/pki-tomcat/alias/pwdfile.txt >> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >> 2018-07-18T16:55:21Z DEBUG stdout= >> Certificate Nickname Trust >> Attributes >> SSL,S/MIME,JAR/XPI >> >> caSigningCert cert-pki-ca CTu,Cu,Cu >> subsystemCert cert-pki-ca u,u,u >> ocspSigningCert cert-pki-ca u,u,u >> auditSigningCert cert-pki-ca u,u,Pu >> Server-Cert cert-pki-ca u,u,u >> >> 2018-07-18T16:55:21Z DEBUG stderr= >> _*2018-07-18T16:55:21Z DEBUG Configuring certmonger to stop >> tracking >> system certificates for CA*_ >> 2018-07-18T16:55:21Z DEBUG Starting external process >> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start >> messagebus.service >> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >> 2018-07-18T16:55:21Z DEBUG stdout= >> 2018-07-18T16:55:21Z DEBUG stderr= >> 2018-07-18T16:55:21Z DEBUG Starting external process >> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active >> messagebus.service >> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >> 2018-07-18T16:55:21Z DEBUG stdout=active >> >> 2018-07-18T16:55:21Z DEBUG stderr= >> 2018-07-18T16:55:21Z DEBUG Starting external process >> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start >> certmonger.service >> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >> 2018-07-18T16:55:21Z DEBUG stdout= >> 2018-07-18T16:55:21Z DEBUG stderr= >> 2018-07-18T16:55:21Z DEBUG Starting external process >> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active >> certmonger.service >> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >> 2018-07-18T16:55:21Z DEBUG stdout=active >> > -snip- a few more lines like the section above. >> 2018-07-18T16:55:25Z DEBUG stderr= >> 2018-07-18T16:55:30Z DEBUG Loading Index file from >> '/var/lib/ipa/sysrestore/sysrestore.index' >> 2018-07-18T16:55:30Z DEBUG Starting external process >> 2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d >> /etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f >> /etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt >> 2018-07-18T16:55:30Z DEBUG Process finished, return code=0 >> 2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN CERTIFICATE----- > -Snip- Cert and Key stuff goes here- >> 2018-07-18T16:55:34Z DEBUG stderr= >> _*2018-07-18T16:55:35Z INFO Certmonger certificate renewal >> configuration updated*_ Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster
You want to run the script on that master first to get the certs renewed.
I'd start by re-running ipa-server-upgrade. It is idempotent so there should be no risk. It may repair the tracking for you.
rob
> On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote: >> On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users wrote: >>> Hi Everyone, >>> >>> I'm looking for some help. I'm having trouble with everything >>> basically. >>> >>> I think one of my CA's certs expired or something. I can't kinit >>> admin, I can't login via the WebGui. If I "getcert list" it >>> returns >>> "Number of certificates and requests being tracked: 0." >>> >>> This all started happening a few days ago and I am at a loss >>> as to >>> what happened. On a whim I set the system date and time back a >>> few >>> months to see if my certs were expired and like magic I can >>> login to >>> the Webgui but I'm still not tracking anything with "getcert >>> list" I >>> suspect the cert has expired but without tracking it I can't >>> tell, or >>> renew it. >>> >> Hi, >> >> can you check if an upgrade happened recently (have a look at >> /var/log/ipaupgrade.log)? The upgrade stop tracking certs and >> re-configures certmonger, so if it failed in the middle you may be >> left without any tracking. >> You should be able to find lines like the following if the >> untracking/tracking went fine: >> --- >> [Update certmonger certificate renewal configuration] >> Configuring certmonger to stop tracking system certificates for CA >> Certmonger certificate renewal configuration updated >> --- >> >> HTH, >> flo >> >>> Please help >>> >>> I'm running Centos 7, FreeIPA 4.5.4 >>> >>> Thanks, >>> >>> -Chris >>> >>> _______________________________________________ >>> FreeIPA-users mailing list -- >>> freeipa-users@lists.fedorahosted.org >>> To unsubscribe send an email to >>> freeipa-users-leave@lists.fedorahosted.org >>> Fedora Code of Conduct: >>> https://getfedora.org/code-of-conduct.html >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >>> >>> >>> > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > To unsubscribe send an email to > freeipa-users-leave@lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... > > >
On 2/7/19 8:22 PM, Chris Mohler via FreeIPA-users wrote:
I have not been able to renew the expired certificates yet. I would appreciate help if possible.
You will need to start repairing the CA renewal master. Can you provide the output of $ getcert list on that node?
Depending on the list of expired certs and their validity dates we will be able to suggest the next steps. It usually involves stopping ntpd, setting the date back to a time when all the certs are valid and letting certmonger do its job.
Any error related to certmonger in the journal may also help understand why the renewal failed.
HTH, flo
Followup summary:
Q: Seems like part of the problem is that the KDC was not running. Had you done ipactl stop prior to the upgrade?
A: I could not get the KDC to stay running. So yes it was off during the upgrade.
Q: Did it end up creating the tracking? Are there expired certs?
A: I was able to get the upgrade to finish successfully, after restoring the server from VM snapshot, rolling back the system date, and trying the update again. It did create the cert tracking!!! Yes there are expired certs.
Q: As an aside, I'd have suggest deferring the package upgrade until after the other things were sorted. It just adds another moving part. Water under the bridge now.
A: Yes sorry.
On 2/5/2019 11:18 AM, Rob Crittenden wrote:
Chris Mohler wrote:
Well... That was a mess.
The ipa-server-upgrade didn't go so well. It failed and now my ca-replication master is broken. Here are the details. Any hope?
Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/11]: stopping directory server [2/11]: saving configuration [3/11]: disabling listeners [4/11]: enabling DS global lock [5/11]: disabling Schema Compat [6/11]: starting directory server [7/11]: updating schema [8/11]: upgrading server [9/11]: stopping directory server [10/11]: restoring configuration [11/11]: starting directory server Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. CA did not start in 300.0s The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Seems like part of the problem is that the KDC was not running. Had you done ipactl stop prior to the upgrade?
Did it end up creating the tracking? Are there expired certs?
As an aside, I'd have suggest deferring the package upgrade until after the other things were sorted. It just adds another moving part. Water under the bridge now.
rob
Here is a wall of errors from my /var/log/ipaupgrade.log
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522 -0500]
- ERR - slapi_ldap_bind - Error: could not send startTLS request:
error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is not connected) Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/ipa2.domain.com@domain.com] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) Feb 4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: /tmp/krb5cc_389)) Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432 -0500]
- ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication
Manager masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Feb 4 17:47:40 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:47:40 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:47:40 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:40 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:47:50 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:47:50 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:47:50 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:50 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:48:00 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:00 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:00 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:00 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:48:10 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:10 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:10 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:10 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb 4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP connection. Feb 4 17:48:20 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:20 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:20 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:20 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) Feb 4 17:48:30 ipa2 server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@3badc78b background process Feb 4 17:48:30 ipa2 server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Feb 4 17:48:30 ipa2 server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:30 ipa2 server: at java.lang.Thread.run(Thread.java:748) Feb 4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to 132.162.1.131 port 67 (xid=0x27e7db13) ^C [root@ipa2 log]# less /var/log/ipaupgrade.log
<p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:13Z DEBUG Waiting for CA to start... 2019-02-04T22:46:14Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:14Z DEBUG request body '' 2019-02-04T22:46:14Z DEBUG response status 500 2019-02-04T22:46:14Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:14 GMT Connection: close
2019-02-04T22:46:14Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:14Z DEBUG Waiting for CA to start... 2019-02-04T22:46:15Z DEBUG request POST http://ipa2.domain.com:8080/ca/admin/ca/getStatus 2019-02-04T22:46:15Z DEBUG request body '' 2019-02-04T22:46:15Z DEBUG response status 500 2019-02-04T22:46:15Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Mon, 04 Feb 2019 22:46:15 GMT Connection: close
2019-02-04T22:46:15Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2019-02-04T22:46:15Z DEBUG Waiting for CA to start... 2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-02-04T22:46:16Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 56, in run raise admintool.ScriptError(str(e))
2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR CA did not start in 300.0s 2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Thanks, -Chris
Rob,
I'll be honest. I think you are suggesting an ldapsearch with this
Check to see which masteris the renewal master. Look in cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for ipaConfigString=caRenewalMaster
sorry I've not figured out how to successfully ldapsearch
Instead I did this: ipa config-show |grep 'CA renewal master'
It came up blank. I suspect I didn't have a renewal master somehow.
Then I did This: ipa-csreplica-manage set-renewal-master ipa2 (hostname of working IPA server)
Next is a "yum update" to be safe, and lastly "ipa-server-upgrade" on ipa2. When that's all done I'll try "yum update" and "ipa-server-upgrade" on my broken IPA system ipa1
I'll report back here when finished.
Thanks,
-Chris
> Check to see which masteris the renewal master. Look in > cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for > ipaConfigString=caRenewalMaster > > You want to run the script on that master first to get the certs > renewed. > > I'd start by re-running ipa-server-upgrade. It is idempotent so > there > should be no risk. It may repair the tracking for you. > > rob On 2/4/2019 3:30 PM, Rob Crittenden wrote: > Chris Mohler via FreeIPA-users wrote: >> Thanks for looking at my issue! >> >> There have been no recent updates on my system. Actually I was >> getting >> ready to update when I noticed things weren't good. >> >> Here is the output from the log of the most recent update. Looks >> like it >> was completed successfully. The lines you asked about are in >> Bold/underlined. >> >>> 2018-07-18T16:55:21Z INFO [Update certmonger certificate renewal >>> configuration] >>> 2018-07-18T16:55:21Z DEBUG Loading Index file from >>> '/var/lib/ipa/sysrestore/sysrestore.index' >>> 2018-07-18T16:55:21Z DEBUG Starting external process >>> 2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d >>> /etc/pki/pki-tomcat/alias -L -f >>> /etc/pki/pki-tomcat/alias/pwdfile.txt >>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >>> 2018-07-18T16:55:21Z DEBUG stdout= >>> Certificate Nickname Trust >>> Attributes >>> SSL,S/MIME,JAR/XPI >>> >>> caSigningCert cert-pki-ca CTu,Cu,Cu >>> subsystemCert cert-pki-ca u,u,u >>> ocspSigningCert cert-pki-ca u,u,u >>> auditSigningCert cert-pki-ca u,u,Pu >>> Server-Cert cert-pki-ca u,u,u >>> >>> 2018-07-18T16:55:21Z DEBUG stderr= >>> _*2018-07-18T16:55:21Z DEBUG Configuring certmonger to stop >>> tracking >>> system certificates for CA*_ >>> 2018-07-18T16:55:21Z DEBUG Starting external process >>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start >>> messagebus.service >>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >>> 2018-07-18T16:55:21Z DEBUG stdout= >>> 2018-07-18T16:55:21Z DEBUG stderr= >>> 2018-07-18T16:55:21Z DEBUG Starting external process >>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active >>> messagebus.service >>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >>> 2018-07-18T16:55:21Z DEBUG stdout=active >>> >>> 2018-07-18T16:55:21Z DEBUG stderr= >>> 2018-07-18T16:55:21Z DEBUG Starting external process >>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start >>> certmonger.service >>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >>> 2018-07-18T16:55:21Z DEBUG stdout= >>> 2018-07-18T16:55:21Z DEBUG stderr= >>> 2018-07-18T16:55:21Z DEBUG Starting external process >>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active >>> certmonger.service >>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0 >>> 2018-07-18T16:55:21Z DEBUG stdout=active >>> >> -snip- a few more lines like the section above. >>> 2018-07-18T16:55:25Z DEBUG stderr= >>> 2018-07-18T16:55:30Z DEBUG Loading Index file from >>> '/var/lib/ipa/sysrestore/sysrestore.index' >>> 2018-07-18T16:55:30Z DEBUG Starting external process >>> 2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d >>> /etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f >>> /etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt >>> 2018-07-18T16:55:30Z DEBUG Process finished, return code=0 >>> 2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN CERTIFICATE----- >> -Snip- Cert and Key stuff goes here- >>> 2018-07-18T16:55:34Z DEBUG stderr= >>> _*2018-07-18T16:55:35Z INFO Certmonger certificate renewal >>> configuration updated*_ > Check to see which masteris the renewal master. Look in > cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for > ipaConfigString=caRenewalMaster > > You want to run the script on that master first to get the certs > renewed. > > I'd start by re-running ipa-server-upgrade. It is idempotent so > there > should be no risk. It may repair the tracking for you. > > rob > >> On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote: >>> On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users wrote: >>>> Hi Everyone, >>>> >>>> I'm looking for some help. I'm having trouble with everything >>>> basically. >>>> >>>> I think one of my CA's certs expired or something. I can't kinit >>>> admin, I can't login via the WebGui. If I "getcert list" it >>>> returns >>>> "Number of certificates and requests being tracked: 0." >>>> >>>> This all started happening a few days ago and I am at a loss >>>> as to >>>> what happened. On a whim I set the system date and time back a >>>> few >>>> months to see if my certs were expired and like magic I can >>>> login to >>>> the Webgui but I'm still not tracking anything with "getcert >>>> list" I >>>> suspect the cert has expired but without tracking it I can't >>>> tell, or >>>> renew it. >>>> >>> Hi, >>> >>> can you check if an upgrade happened recently (have a look at >>> /var/log/ipaupgrade.log)? The upgrade stop tracking certs and >>> re-configures certmonger, so if it failed in the middle you may be >>> left without any tracking. >>> You should be able to find lines like the following if the >>> untracking/tracking went fine: >>> --- >>> [Update certmonger certificate renewal configuration] >>> Configuring certmonger to stop tracking system certificates for CA >>> Certmonger certificate renewal configuration updated >>> --- >>> >>> HTH, >>> flo >>> >>>> Please help >>>> >>>> I'm running Centos 7, FreeIPA 4.5.4 >>>> >>>> Thanks, >>>> >>>> -Chris >>>> >>>> _______________________________________________ >>>> FreeIPA-users mailing list -- >>>> freeipa-users@lists.fedorahosted.org >>>> To unsubscribe send an email to >>>> freeipa-users-leave@lists.fedorahosted.org >>>> Fedora Code of Conduct: >>>> https://getfedora.org/code-of-conduct.html >>>> List Guidelines: >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >>>> >>>> >>>> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave@lists.fedorahosted.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >> >> >>
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Chris Mohler -
Florence Blanc-Renaud