Hello all,
I'm running FreeIPA on two CentOS 7 servers, one, the master is on a physical server, the other (a replica with CA, DNS etc) is running on an Ovirt cluster.
I patched the boxes and upgraded IPA on the two servers a few days ago, and the master ran through the upgrade without any issue, however the replica fails when starting the CA, timing out after the 300 seconds. Increasing the timeout to 600 didn't help, and I rebuilt the replica from scratch which still gives the same error. If I try and restart the services after promoting it it tells me to run the upgrade, and if I do so I get the same error as the install:
2018-06-15T04:48:07Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2018-06-15T04:48:07Z DEBUG Waiting for CA to start... 2018-06-15T04:48:08Z DEBUG request POST <replica>:8080 2018-06-15T04:48:08Z DEBUG request body '' 2018-06-15T04:48:08Z DEBUG response status 500 2018-06-15T04:48:08Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8
2018-06-15T04:48:09Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2018-06-15T04:48:09Z ERROR CA did not start in 300.0s
Googling gets me similar problems people have had due to certificate expiry, but the dates look good as far as I can see and after a complete rebuild it should have issued new ones anyway I think.
Digging through the logs I see variations on the below error, but I'm not sure why this would be the case: Could not connect to LDAP server host <Replica> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Browsing to http://<Replica>:8080/ca/admin/ca/getStatus
Gets me this:
*type* Exception report
*message* *Subsystem unavailable*
*description* *The server encountered an internal error that prevented it from fulfilling this request.*
*exception*
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Thread.java:748)
*note* *The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.*
I'm a bit stuck as to how to proceed fixing this, I'm not overly familiar with what logs do what with IPA, and I'm not seeing anything obviously wrong with the configuration.
Has anyone seen this before, or can point me in the right direction to track this down?
Thanks,
Thomas
Hi Thomas,
you can have a look at https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
Usually the communication issue between PKI and LDAP is linked to an expired certificate, or a mismatch between the content of uid=pkidbuser,ou=people,o=ipaca and the actual certificate.
HTH, Flo
On 06/15/2018 06:52 AM, Thomas Letherby via FreeIPA-users wrote:
Hello all,
I'm running FreeIPA on two CentOS 7 servers, one, the master is on a physical server, the other (a replica with CA, DNS etc) is running on an Ovirt cluster.
I patched the boxes and upgraded IPA on the two servers a few days ago, and the master ran through the upgrade without any issue, however the replica fails when starting the CA, timing out after the 300 seconds. Increasing the timeout to 600 didn't help, and I rebuilt the replica from scratch which still gives the same error. If I try and restart the services after promoting it it tells me to run the upgrade, and if I do so I get the same error as the install:
2018-06-15T04:48:07Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2018-06-15T04:48:07Z DEBUG Waiting for CA to start... 2018-06-15T04:48:08Z DEBUG request POST <replica>:8080 2018-06-15T04:48:08Z DEBUG request body '' 2018-06-15T04:48:08Z DEBUG response status 500 2018-06-15T04:48:08Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8
2018-06-15T04:48:09Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2018-06-15T04:48:09Z ERROR CA did not start in 300.0s
Googling gets me similar problems people have had due to certificate expiry, but the dates look good as far as I can see and after a complete rebuild it should have issued new ones anyway I think.
Digging through the logs I see variations on the below error, but I'm not sure why this would be the case: Could not connect to LDAP server host <Replica> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Browsing to http://<Replica>:8080/ca/admin/ca/getStatus
Gets me this:
*type* Exception report
*message* _Subsystem unavailable_
*description* _The server encountered an internal error that prevented it from fulfilling this request._
*exception*
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Thread.java:748)
*note* _The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs._
I'm a bit stuck as to how to proceed fixing this, I'm not overly familiar with what logs do what with IPA, and I'm not seeing anything obviously wrong with the configuration.
Has anyone seen this before, or can point me in the right direction to track this down?
Thanks,
Thomas
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
You'd be right, I worked it out over the weekend. On the fifth time of checking, having convinced myself the certificates all looked good, I renewed the expried Kerberos certificate...
It didn't seem to take effect straight away for bringing up the replica though but I didn't have time to dig in until the next day, and that time it worked first time, so I suspect it was cached somewhere too.
Thanks for the help though! Much appreciated.
Thomas
On Mon, Jun 18, 2018, 11:41 PM Florence Blanc-Renaud flo@redhat.com wrote:
Hi Thomas,
you can have a look at
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
Usually the communication issue between PKI and LDAP is linked to an expired certificate, or a mismatch between the content of uid=pkidbuser,ou=people,o=ipaca and the actual certificate.
HTH, Flo
On 06/15/2018 06:52 AM, Thomas Letherby via FreeIPA-users wrote:
Hello all,
I'm running FreeIPA on two CentOS 7 servers, one, the master is on a physical server, the other (a replica with CA, DNS etc) is running on an Ovirt cluster.
I patched the boxes and upgraded IPA on the two servers a few days ago, and the master ran through the upgrade without any issue, however the replica fails when starting the CA, timing out after the 300 seconds. Increasing the timeout to 600 didn't help, and I rebuilt the replica from scratch which still gives the same error. If I try and restart the services after promoting it it tells me to run the upgrade, and if I do so I get the same error as the install:
2018-06-15T04:48:07Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2018-06-15T04:48:07Z DEBUG Waiting for CA to start... 2018-06-15T04:48:08Z DEBUG request POST <replica>:8080 2018-06-15T04:48:08Z DEBUG request body '' 2018-06-15T04:48:08Z DEBUG response status 500 2018-06-15T04:48:08Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8
2018-06-15T04:48:09Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2018-06-15T04:48:09Z ERROR CA did not start in 300.0s
Googling gets me similar problems people have had due to certificate expiry, but the dates look good as far as I can see and after a complete rebuild it should have issued new ones anyway I think.
Digging through the logs I see variations on the below error, but I'm not sure why this would be the case: Could not connect to LDAP server host <Replica> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Browsing to http://<Replica>:8080/ca/admin/ca/getStatus
Gets me this:
*type* Exception report
*message* _Subsystem unavailable_
*description* _The server encountered an internal error that prevented it from fulfilling this request._
*exception*
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
org.apache.tomcat.util.net
.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:748)
*note* _The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs._
I'm a bit stuck as to how to proceed fixing this, I'm not overly familiar with what logs do what with IPA, and I'm not seeing anything obviously wrong with the configuration.
Has anyone seen this before, or can point me in the right direction to track this down?
Thanks,
Thomas
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
freeipa-users@lists.fedorahosted.org