Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
Sina Owolabi via FreeIPA-users wrote:
Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob
Λ
On Mon, 4 Mar 2019, 19:11 Rob Crittenden via FreeIPA-users, < freeipa-users@lists.fedorahosted.org> wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi!
getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC
I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart".
On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob
Restarting ipa didnt create the logs. Please, what else can i do?
On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi!
getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC
I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart".
On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob
Hi!
I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got:
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
Could this be the root of my problems? And how can I convert them?
On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote:
Restarting ipa didnt create the logs. Please, what else can i do?
On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi!
getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC
I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart".
On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob
On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
Hi!
I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got:
Hi,
1. this note was written for RHEL 6 but you said in your first e-mail that your server is running CentOS 7 with ipa 4.5.4. Please don't follow those instructions as they are not adapted to your deployment. The instructions for RHEL 7 are available at https://access.redhat.com/solutions/3357261.
2. In a previous e-mail, the output of getcert list | grep -i expires did not show any expired certificates, so I would not rush into wrong conclusions. We need to understand first why pki did not start.
What is the output of: $ ipactl status $ systemctl status pki-tomcatd@pki-tomcat.service
flo
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
Could this be the root of my problems? And how can I convert them?
On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote:
Restarting ipa didnt create the logs. Please, what else can i do?
On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi!
getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC
I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart".
On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence
and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
systemctl status -l pki-tomcatd@pki-tomcat.service; cat ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
systemctl status pki-tomcatd@pki-tomcat.service:
Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748)
On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote:
On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
Hi!
I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got:
Hi,
- this note was written for RHEL 6 but you said in your first e-mail
that your server is running CentOS 7 with ipa 4.5.4. Please don't follow those instructions as they are not adapted to your deployment. The instructions for RHEL 7 are available at https://access.redhat.com/solutions/3357261.
- In a previous e-mail, the output of getcert list | grep -i expires
did not show any expired certificates, so I would not rush into wrong conclusions. We need to understand first why pki did not start.
What is the output of: $ ipactl status $ systemctl status pki-tomcatd@pki-tomcat.service
flo
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
Could this be the root of my problems? And how can I convert them?
On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote:
Restarting ipa didnt create the logs. Please, what else can i do?
On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi!
getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC
I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart".
On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi!
I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, api version 2.228), with one master, and two replicas, and I noticed that pki-tomcatd no longer works on the master, after attempting a reboot. pki-tomcatd works fine on the slaves. I noticed if I try to run IPA functions (dns record removal, hosts management, user passwords, etc), I receive responses like this:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Internal Server Error) But on the replicas, functions work fine. Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Sina Owolabi wrote:
Hi Florence
and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
systemctl status -l pki-tomcatd@pki-tomcat.service; cat ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
systemctl status pki-tomcatd@pki-tomcat.service:
Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748)
The logs will contain much more useful information. dogtag keeps changing the location of the logs and I forget exactly where it is in your version but it's somewhere in /var/log/pki*/pki*/ca/...
The log may be named debug or debug-<date>
Also look at the selftest log in the same directory.
There are a LOT of red herrings in the dogtag logs so proceed with caution.
You do not need to touch or create anything for this logging to take place. You should delete the directory you created.
rob
On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote:
On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
Hi!
I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got:
Hi,
- this note was written for RHEL 6 but you said in your first e-mail
that your server is running CentOS 7 with ipa 4.5.4. Please don't follow those instructions as they are not adapted to your deployment. The instructions for RHEL 7 are available at https://access.redhat.com/solutions/3357261.
- In a previous e-mail, the output of getcert list | grep -i expires
did not show any expired certificates, so I would not rush into wrong conclusions. We need to understand first why pki did not start.
What is the output of: $ ipactl status $ systemctl status pki-tomcatd@pki-tomcat.service
flo
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
Could this be the root of my problems? And how can I convert them?
On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote:
Restarting ipa didnt create the logs. Please, what else can i do?
On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi!
getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC
I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart".
On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote: > Hi! > > I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, > api version 2.228), with one master, and two replicas, and I noticed > that pki-tomcatd no longer works on the master, after attempting a > reboot. > pki-tomcatd works fine on the slaves. > I noticed if I try to run IPA functions (dns record removal, hosts > management, user passwords, etc), I receive responses like this: > > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Internal Server Error) > But on the replicas, functions work fine. > Please can someone guide me on how to fix this?
The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some pointers. I'd look at selftests.log first.
My guess is that some of the CA certificates have failed to renew.
getcert list | grep -i expires
rob
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Log directories on the server:
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs are both empty.
On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Hi Florence
and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
systemctl status -l pki-tomcatd@pki-tomcat.service; cat ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
systemctl status pki-tomcatd@pki-tomcat.service:
Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748)
The logs will contain much more useful information. dogtag keeps changing the location of the logs and I forget exactly where it is in your version but it's somewhere in /var/log/pki*/pki*/ca/...
The log may be named debug or debug-<date>
Also look at the selftest log in the same directory.
There are a LOT of red herrings in the dogtag logs so proceed with caution.
You do not need to touch or create anything for this logging to take place. You should delete the directory you created.
rob
On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote:
On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
Hi!
I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got:
Hi,
- this note was written for RHEL 6 but you said in your first e-mail
that your server is running CentOS 7 with ipa 4.5.4. Please don't follow those instructions as they are not adapted to your deployment. The instructions for RHEL 7 are available at https://access.redhat.com/solutions/3357261.
- In a previous e-mail, the output of getcert list | grep -i expires
did not show any expired certificates, so I would not rush into wrong conclusions. We need to understand first why pki did not start.
What is the output of: $ ipactl status $ systemctl status pki-tomcatd@pki-tomcat.service
flo
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
Could this be the root of my problems? And how can I convert them?
On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote:
Restarting ipa didnt create the logs. Please, what else can i do?
On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi!
getcert list | grep -i expires expires: 2019-04-13 12:08:20 UTC expires: 2019-04-13 12:08:06 UTC expires: 2019-04-13 12:07:50 UTC expires: 2035-06-01 08:33:01 UTC expires: 2019-04-13 12:07:41 UTC expires: 2019-04-13 12:06:55 UTC expires: 2019-05-05 12:06:41 UTC expires: 2019-05-05 12:06:56 UTC expires: 2020-01-17 19:56:03 UTC
I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am creating one and running "ipactl restart".
On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote: > > Sina Owolabi via FreeIPA-users wrote: >> Hi! >> >> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, >> api version 2.228), with one master, and two replicas, and I noticed >> that pki-tomcatd no longer works on the master, after attempting a >> reboot. >> pki-tomcatd works fine on the slaves. >> I noticed if I try to run IPA functions (dns record removal, hosts >> management, user passwords, etc), I receive responses like this: >> >> ipa: ERROR: Certificate operation cannot be completed: Unable to >> communicate with CMS (Internal Server Error) >> But on the replicas, functions work fine. >> Please can someone guide me on how to fix this? > > The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some > pointers. I'd look at selftests.log first. > > My guess is that some of the CA certificates have failed to renew. > > getcert list | grep -i expires > > rob
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Sina Owolabi wrote:
Log directories on the server:
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1
/var/log/pki/pki-tomcat/ca/debug
You stated you had created this directory yourself.
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
rob
/var/log/pki/pki-tomcat/ca/logs are both empty.
On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Hi Florence
and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
systemctl status -l pki-tomcatd@pki-tomcat.service; cat ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
systemctl status pki-tomcatd@pki-tomcat.service:
Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748)
The logs will contain much more useful information. dogtag keeps changing the location of the logs and I forget exactly where it is in your version but it's somewhere in /var/log/pki*/pki*/ca/...
The log may be named debug or debug-<date>
Also look at the selftest log in the same directory.
There are a LOT of red herrings in the dogtag logs so proceed with caution.
You do not need to touch or create anything for this logging to take place. You should delete the directory you created.
rob
On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote:
On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
Hi!
I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got:
Hi,
- this note was written for RHEL 6 but you said in your first e-mail
that your server is running CentOS 7 with ipa 4.5.4. Please don't follow those instructions as they are not adapted to your deployment. The instructions for RHEL 7 are available at https://access.redhat.com/solutions/3357261.
- In a previous e-mail, the output of getcert list | grep -i expires
did not show any expired certificates, so I would not rush into wrong conclusions. We need to understand first why pki did not start.
What is the output of: $ ipactl status $ systemctl status pki-tomcatd@pki-tomcat.service
flo
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
Could this be the root of my problems? And how can I convert them?
On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote:
Restarting ipa didnt create the logs. Please, what else can i do?
On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote: > > Hi! > > getcert list | grep -i expires > expires: 2019-04-13 12:08:20 UTC > expires: 2019-04-13 12:08:06 UTC > expires: 2019-04-13 12:07:50 UTC > expires: 2035-06-01 08:33:01 UTC > expires: 2019-04-13 12:07:41 UTC > expires: 2019-04-13 12:06:55 UTC > expires: 2019-05-05 12:06:41 UTC > expires: 2019-05-05 12:06:56 UTC > expires: 2020-01-17 19:56:03 UTC > > I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am > creating one and running "ipactl restart". > > On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote: >> >> Sina Owolabi via FreeIPA-users wrote: >>> Hi! >>> >>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, >>> api version 2.228), with one master, and two replicas, and I noticed >>> that pki-tomcatd no longer works on the master, after attempting a >>> reboot. >>> pki-tomcatd works fine on the slaves. >>> I noticed if I try to run IPA functions (dns record removal, hosts >>> management, user passwords, etc), I receive responses like this: >>> >>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>> communicate with CMS (Internal Server Error) >>> But on the replicas, functions work fine. >>> Please can someone guide me on how to fix this? >> >> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some >> pointers. I'd look at selftests.log first. >> >> My guess is that some of the CA certificates have failed to renew. >> >> getcert list | grep -i expires >> >> rob
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Rob
Today's catalina log file writes:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Log directories on the server:
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1
/var/log/pki/pki-tomcat/ca/debug
You stated you had created this directory yourself.
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
rob
/var/log/pki/pki-tomcat/ca/logs are both empty.
On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Hi Florence
and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
systemctl status -l pki-tomcatd@pki-tomcat.service; cat ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
systemctl status pki-tomcatd@pki-tomcat.service:
Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748)
The logs will contain much more useful information. dogtag keeps changing the location of the logs and I forget exactly where it is in your version but it's somewhere in /var/log/pki*/pki*/ca/...
The log may be named debug or debug-<date>
Also look at the selftest log in the same directory.
There are a LOT of red herrings in the dogtag logs so proceed with caution.
You do not need to touch or create anything for this logging to take place. You should delete the directory you created.
rob
On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote:
On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote:
Hi!
I tried to follow this solution for cert renewal for RHEL6: https://access.redhat.com/solutions/643753 (Sorry, desperation is setting in), but when I attempted Step 2, I got:
Hi,
- this note was written for RHEL 6 but you said in your first e-mail
that your server is running CentOS 7 with ipa 4.5.4. Please don't follow those instructions as they are not adapted to your deployment. The instructions for RHEL 7 are available at https://access.redhat.com/solutions/3357261.
- In a previous e-mail, the output of getcert list | grep -i expires
did not show any expired certificates, so I would not rush into wrong conclusions. We need to understand first why pki did not start.
What is the output of: $ ipactl status $ systemctl status pki-tomcatd@pki-tomcat.service
flo
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after; done auditSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. ocspSigningCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. subsystemCert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. Server-Cert cert-pki-ca certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
Could this be the root of my problems? And how can I convert them?
On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote: > > Restarting ipa didnt create the logs. > Please, what else can i do? > > On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote: >> >> Hi! >> >> getcert list | grep -i expires >> expires: 2019-04-13 12:08:20 UTC >> expires: 2019-04-13 12:08:06 UTC >> expires: 2019-04-13 12:07:50 UTC >> expires: 2035-06-01 08:33:01 UTC >> expires: 2019-04-13 12:07:41 UTC >> expires: 2019-04-13 12:06:55 UTC >> expires: 2019-05-05 12:06:41 UTC >> expires: 2019-05-05 12:06:56 UTC >> expires: 2020-01-17 19:56:03 UTC >> >> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am >> creating one and running "ipactl restart". >> >> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote: >>> >>> Sina Owolabi via FreeIPA-users wrote: >>>> Hi! >>>> >>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, >>>> api version 2.228), with one master, and two replicas, and I noticed >>>> that pki-tomcatd no longer works on the master, after attempting a >>>> reboot. >>>> pki-tomcatd works fine on the slaves. >>>> I noticed if I try to run IPA functions (dns record removal, hosts >>>> management, user passwords, etc), I receive responses like this: >>>> >>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>> communicate with CMS (Internal Server Error) >>>> But on the replicas, functions work fine. >>>> Please can someone guide me on how to fix this? >>> >>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some >>> pointers. I'd look at selftests.log first. >>> >>> My guess is that some of the CA certificates have failed to renew. >>> >>> getcert list | grep -i expires >>> >>> rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Rob
Sorry I missed the full question: What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
/var/log/pki/pki-tomcat/ca was empty until I created /var/log/pki/pki-tomcat/ca/logs and /var/log/pki/pki-tomcat/ca/debug directories. I dont think the ca would have trouble writing its logs, the structure is all owned by pkiuser: drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
Now that I think about it, I do remember some issues with runaway logs filling up /var/log, and I deleted some directories, and recreated them, but I dont think pki-tomcat suffered then.
On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi Rob
Today's catalina log file writes:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Log directories on the server:
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1
/var/log/pki/pki-tomcat/ca/debug
You stated you had created this directory yourself.
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
rob
/var/log/pki/pki-tomcat/ca/logs are both empty.
On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Hi Florence
and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
systemctl status -l pki-tomcatd@pki-tomcat.service; cat ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
systemctl status pki-tomcatd@pki-tomcat.service:
Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748)
The logs will contain much more useful information. dogtag keeps changing the location of the logs and I forget exactly where it is in your version but it's somewhere in /var/log/pki*/pki*/ca/...
The log may be named debug or debug-<date>
Also look at the selftest log in the same directory.
There are a LOT of red herrings in the dogtag logs so proceed with caution.
You do not need to touch or create anything for this logging to take place. You should delete the directory you created.
rob
On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote:
On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: > Hi! > > I tried to follow this solution for cert renewal for RHEL6: > https://access.redhat.com/solutions/643753 (Sorry, desperation is > setting in), but when I attempted Step 2, I got: > Hi,
- this note was written for RHEL 6 but you said in your first e-mail
that your server is running CentOS 7 with ipa 4.5.4. Please don't follow those instructions as they are not adapted to your deployment. The instructions for RHEL 7 are available at https://access.redhat.com/solutions/3357261.
- In a previous e-mail, the output of getcert list | grep -i expires
did not show any expired certificates, so I would not rush into wrong conclusions. We need to understand first why pki did not start.
What is the output of: $ ipactl status $ systemctl status pki-tomcatd@pki-tomcat.service
flo
> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert > cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do > echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" > | grep -i after; done > auditSigningCert cert-pki-ca > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > certificate/key database is in an old, unsupported format. > ocspSigningCert cert-pki-ca > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > certificate/key database is in an old, unsupported format. > subsystemCert cert-pki-ca > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > certificate/key database is in an old, unsupported format. > Server-Cert cert-pki-ca > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > certificate/key database is in an old, unsupported format. > > Could this be the root of my problems? > And how can I convert them? > > On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote: >> >> Restarting ipa didnt create the logs. >> Please, what else can i do? >> >> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote: >>> >>> Hi! >>> >>> getcert list | grep -i expires >>> expires: 2019-04-13 12:08:20 UTC >>> expires: 2019-04-13 12:08:06 UTC >>> expires: 2019-04-13 12:07:50 UTC >>> expires: 2035-06-01 08:33:01 UTC >>> expires: 2019-04-13 12:07:41 UTC >>> expires: 2019-04-13 12:06:55 UTC >>> expires: 2019-05-05 12:06:41 UTC >>> expires: 2019-05-05 12:06:56 UTC >>> expires: 2020-01-17 19:56:03 UTC >>> >>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am >>> creating one and running "ipactl restart". >>> >>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote: >>>> >>>> Sina Owolabi via FreeIPA-users wrote: >>>>> Hi! >>>>> >>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, >>>>> api version 2.228), with one master, and two replicas, and I noticed >>>>> that pki-tomcatd no longer works on the master, after attempting a >>>>> reboot. >>>>> pki-tomcatd works fine on the slaves. >>>>> I noticed if I try to run IPA functions (dns record removal, hosts >>>>> management, user passwords, etc), I receive responses like this: >>>>> >>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>> communicate with CMS (Internal Server Error) >>>>> But on the replicas, functions work fine. >>>>> Please can someone guide me on how to fix this? >>>> >>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some >>>> pointers. I'd look at selftests.log first. >>>> >>>> My guess is that some of the CA certificates have failed to renew. >>>> >>>> getcert list | grep -i expires >>>> >>>> rob > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >
Sina Owolabi via FreeIPA-users wrote:
Hi Rob
Sorry I missed the full question: What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
/var/log/pki/pki-tomcat/ca was empty until I created /var/log/pki/pki-tomcat/ca/logs and /var/log/pki/pki-tomcat/ca/debug directories. I dont think the ca would have trouble writing its logs, the structure is all owned by pkiuser: drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
Now that I think about it, I do remember some issues with runaway logs filling up /var/log, and I deleted some directories, and recreated them, but I dont think pki-tomcat suffered then.
Hard to know. If the process was already running at the time things may have appeared ok until it was restarted.
debug is a log file, not a directory.
My 4.4.4 install contains the following in /var/log/pki:
drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit
Be sure to run restorecon -R on /var/log/pki to ensure the SELinux contexts are correct.
rob
On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi Rob
Today's catalina log file writes:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Log directories on the server:
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1
/var/log/pki/pki-tomcat/ca/debug
You stated you had created this directory yourself.
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
rob
/var/log/pki/pki-tomcat/ca/logs are both empty.
On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Hi Florence
and thanks for the help. ipactl status: [root@services ~]# ipactl status --ignore-service-failure; cat Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
systemctl status -l pki-tomcatd@pki-tomcat.service; cat ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 1376 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
systemctl status pki-tomcatd@pki-tomcat.service:
Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process Mar 05 09:40:43 services.qrios.com server[1376]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Mar 05 09:40:43 services.qrios.com server[1376]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Mar 05 09:40:43 services.qrios.com server[1376]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Mar 05 09:40:43 services.qrios.com server[1376]: at java.lang.Thread.run(Thread.java:748)
The logs will contain much more useful information. dogtag keeps changing the location of the logs and I forget exactly where it is in your version but it's somewhere in /var/log/pki*/pki*/ca/...
The log may be named debug or debug-<date>
Also look at the selftest log in the same directory.
There are a LOT of red herrings in the dogtag logs so proceed with caution.
You do not need to touch or create anything for this logging to take place. You should delete the directory you created.
rob
On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote: > > On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: >> Hi! >> >> I tried to follow this solution for cert renewal for RHEL6: >> https://access.redhat.com/solutions/643753 (Sorry, desperation is >> setting in), but when I attempted Step 2, I got: >> > Hi, > > 1. this note was written for RHEL 6 but you said in your first e-mail > that your server is running CentOS 7 with ipa 4.5.4. Please don't follow > those instructions as they are not adapted to your deployment. > The instructions for RHEL 7 are available at > https://access.redhat.com/solutions/3357261. > > 2. In a previous e-mail, the output of getcert list | grep -i expires > did not show any expired certificates, so I would not rush into wrong > conclusions. We need to understand first why pki did not start. > > What is the output of: > $ ipactl status > $ systemctl status pki-tomcatd@pki-tomcat.service > > flo > >> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert >> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do >> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" >> | grep -i after; done >> auditSigningCert cert-pki-ca >> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >> certificate/key database is in an old, unsupported format. >> ocspSigningCert cert-pki-ca >> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >> certificate/key database is in an old, unsupported format. >> subsystemCert cert-pki-ca >> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >> certificate/key database is in an old, unsupported format. >> Server-Cert cert-pki-ca >> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >> certificate/key database is in an old, unsupported format. >> >> Could this be the root of my problems? >> And how can I convert them? >> >> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote: >>> >>> Restarting ipa didnt create the logs. >>> Please, what else can i do? >>> >>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote: >>>> >>>> Hi! >>>> >>>> getcert list | grep -i expires >>>> expires: 2019-04-13 12:08:20 UTC >>>> expires: 2019-04-13 12:08:06 UTC >>>> expires: 2019-04-13 12:07:50 UTC >>>> expires: 2035-06-01 08:33:01 UTC >>>> expires: 2019-04-13 12:07:41 UTC >>>> expires: 2019-04-13 12:06:55 UTC >>>> expires: 2019-05-05 12:06:41 UTC >>>> expires: 2019-05-05 12:06:56 UTC >>>> expires: 2020-01-17 19:56:03 UTC >>>> >>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am >>>> creating one and running "ipactl restart". >>>> >>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote: >>>>> >>>>> Sina Owolabi via FreeIPA-users wrote: >>>>>> Hi! >>>>>> >>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, >>>>>> api version 2.228), with one master, and two replicas, and I noticed >>>>>> that pki-tomcatd no longer works on the master, after attempting a >>>>>> reboot. >>>>>> pki-tomcatd works fine on the slaves. >>>>>> I noticed if I try to run IPA functions (dns record removal, hosts >>>>>> management, user passwords, etc), I receive responses like this: >>>>>> >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>> communicate with CMS (Internal Server Error) >>>>>> But on the replicas, functions work fine. >>>>>> Please can someone guide me on how to fix this? >>>>> >>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some >>>>> pointers. I'd look at selftests.log first. >>>>> >>>>> My guess is that some of the CA certificates have failed to renew. >>>>> >>>>> getcert list | grep -i expires >>>>> >>>>> rob >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >> >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Ah.. the server has SELinux disabled. Curious why its not recreating its log directories after a restart.
On Wed, Mar 6, 2019 at 4:33 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi Rob
Sorry I missed the full question: What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
/var/log/pki/pki-tomcat/ca was empty until I created /var/log/pki/pki-tomcat/ca/logs and /var/log/pki/pki-tomcat/ca/debug directories. I dont think the ca would have trouble writing its logs, the structure is all owned by pkiuser: drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
Now that I think about it, I do remember some issues with runaway logs filling up /var/log, and I deleted some directories, and recreated them, but I dont think pki-tomcat suffered then.
Hard to know. If the process was already running at the time things may have appeared ok until it was restarted.
debug is a log file, not a directory.
My 4.4.4 install contains the following in /var/log/pki:
drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit
Be sure to run restorecon -R on /var/log/pki to ensure the SELinux contexts are correct.
rob
On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi Rob
Today's catalina log file writes:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Log directories on the server:
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1
/var/log/pki/pki-tomcat/ca/debug
You stated you had created this directory yourself.
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
rob
/var/log/pki/pki-tomcat/ca/logs are both empty.
On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote: > Hi Florence > > and thanks for the help. > ipactl status: > [root@services ~]# ipactl status --ignore-service-failure; cat > Directory Service: RUNNING > krb5kdc Service: RUNNING > kadmin Service: RUNNING > named Service: RUNNING > httpd Service: RUNNING > ipa-custodia Service: RUNNING > ntpd Service: RUNNING > pki-tomcatd Service: STOPPED > ipa-otpd Service: RUNNING > ipa-dnskeysyncd Service: RUNNING > ipa: INFO: The ipactl command was successful > > > systemctl status -l pki-tomcatd@pki-tomcat.service; cat > ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat > Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; > vendor preset: disabled) > Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago > Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, > status=0/SUCCESS) > Main PID: 1376 (java) > CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service > └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath > /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > -Dcatalina.base=/var/lib/pki/pki-tomcat > -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= > -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > -Djava.security.manager > -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > org.apache.catalina.startup.Bootstrap start > > systemctl status pki-tomcatd@pki-tomcat.service: > > Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception > processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f > background process > Mar 05 09:40:43 services.qrios.com server[1376]: > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > Mar 05 09:40:43 services.qrios.com server[1376]: at > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > Mar 05 09:40:43 services.qrios.com server[1376]: at > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > Mar 05 09:40:43 services.qrios.com server[1376]: at > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > Mar 05 09:40:43 services.qrios.com server[1376]: at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > Mar 05 09:40:43 services.qrios.com server[1376]: at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > Mar 05 09:40:43 services.qrios.com server[1376]: at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > Mar 05 09:40:43 services.qrios.com server[1376]: at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > Mar 05 09:40:43 services.qrios.com server[1376]: at > java.lang.Thread.run(Thread.java:748)
The logs will contain much more useful information. dogtag keeps changing the location of the logs and I forget exactly where it is in your version but it's somewhere in /var/log/pki*/pki*/ca/...
The log may be named debug or debug-<date>
Also look at the selftest log in the same directory.
There are a LOT of red herrings in the dogtag logs so proceed with caution.
You do not need to touch or create anything for this logging to take place. You should delete the directory you created.
rob
> > On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote: >> >> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: >>> Hi! >>> >>> I tried to follow this solution for cert renewal for RHEL6: >>> https://access.redhat.com/solutions/643753 (Sorry, desperation is >>> setting in), but when I attempted Step 2, I got: >>> >> Hi, >> >> 1. this note was written for RHEL 6 but you said in your first e-mail >> that your server is running CentOS 7 with ipa 4.5.4. Please don't follow >> those instructions as they are not adapted to your deployment. >> The instructions for RHEL 7 are available at >> https://access.redhat.com/solutions/3357261. >> >> 2. In a previous e-mail, the output of getcert list | grep -i expires >> did not show any expired certificates, so I would not rush into wrong >> conclusions. We need to understand first why pki did not start. >> >> What is the output of: >> $ ipactl status >> $ systemctl status pki-tomcatd@pki-tomcat.service >> >> flo >> >>> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert >>> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do >>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" >>> | grep -i after; done >>> auditSigningCert cert-pki-ca >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>> certificate/key database is in an old, unsupported format. >>> ocspSigningCert cert-pki-ca >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>> certificate/key database is in an old, unsupported format. >>> subsystemCert cert-pki-ca >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>> certificate/key database is in an old, unsupported format. >>> Server-Cert cert-pki-ca >>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>> certificate/key database is in an old, unsupported format. >>> >>> Could this be the root of my problems? >>> And how can I convert them? >>> >>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote: >>>> >>>> Restarting ipa didnt create the logs. >>>> Please, what else can i do? >>>> >>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote: >>>>> >>>>> Hi! >>>>> >>>>> getcert list | grep -i expires >>>>> expires: 2019-04-13 12:08:20 UTC >>>>> expires: 2019-04-13 12:08:06 UTC >>>>> expires: 2019-04-13 12:07:50 UTC >>>>> expires: 2035-06-01 08:33:01 UTC >>>>> expires: 2019-04-13 12:07:41 UTC >>>>> expires: 2019-04-13 12:06:55 UTC >>>>> expires: 2019-05-05 12:06:41 UTC >>>>> expires: 2019-05-05 12:06:56 UTC >>>>> expires: 2020-01-17 19:56:03 UTC >>>>> >>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am >>>>> creating one and running "ipactl restart". >>>>> >>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote: >>>>>> >>>>>> Sina Owolabi via FreeIPA-users wrote: >>>>>>> Hi! >>>>>>> >>>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, >>>>>>> api version 2.228), with one master, and two replicas, and I noticed >>>>>>> that pki-tomcatd no longer works on the master, after attempting a >>>>>>> reboot. >>>>>>> pki-tomcatd works fine on the slaves. >>>>>>> I noticed if I try to run IPA functions (dns record removal, hosts >>>>>>> management, user passwords, etc), I receive responses like this: >>>>>>> >>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>> communicate with CMS (Internal Server Error) >>>>>>> But on the replicas, functions work fine. >>>>>>> Please can someone guide me on how to fix this? >>>>>> >>>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some >>>>>> pointers. I'd look at selftests.log first. >>>>>> >>>>>> My guess is that some of the CA certificates have failed to renew. >>>>>> >>>>>> getcert list | grep -i expires >>>>>> >>>>>> rob >>> _______________________________________________ >>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org >>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >>> >>
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Would reinstalling IPA software help?
On Wed, Mar 6, 2019 at 10:05 PM Sina Owolabi notify.sina@gmail.com wrote:
Ah.. the server has SELinux disabled. Curious why its not recreating its log directories after a restart.
On Wed, Mar 6, 2019 at 4:33 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi via FreeIPA-users wrote:
Hi Rob
Sorry I missed the full question: What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
/var/log/pki/pki-tomcat/ca was empty until I created /var/log/pki/pki-tomcat/ca/logs and /var/log/pki/pki-tomcat/ca/debug directories. I dont think the ca would have trouble writing its logs, the structure is all owned by pkiuser: drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca
Now that I think about it, I do remember some issues with runaway logs filling up /var/log, and I deleted some directories, and recreated them, but I dont think pki-tomcat suffered then.
Hard to know. If the process was already running at the time things may have appeared ok until it was restarted.
debug is a log file, not a directory.
My 4.4.4 install contains the following in /var/log/pki:
drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit
Be sure to run restorecon -R on /var/log/pki to ensure the SELinux contexts are correct.
rob
On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi notify.sina@gmail.com wrote:
Hi Rob
Today's catalina log file writes:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748)
On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden rcritten@redhat.com wrote:
Sina Owolabi wrote:
Log directories on the server:
/var/log/pki/pki-tomcat/ca/debug /var/log/pki/pki-tomcat/ca/logs /var/log/pki/server/upgrade/10.1.2 /var/log/pki/server/upgrade/10.1.99 /var/log/pki/server/upgrade/10.2.1 /var/log/pki/server/upgrade/10.2.2 /var/log/pki/server/upgrade/10.2.3 /var/log/pki/server/upgrade/10.2.4 /var/log/pki/server/upgrade/10.2.5 /var/log/pki/server/upgrade/10.2.6 /var/log/pki/server/upgrade/10.3.0 /var/log/pki/server/upgrade/10.3.3 /var/log/pki/server/upgrade/10.4.0 /var/log/pki/server/upgrade/10.4.1 /var/log/pki/server/upgrade/10.5.1
/var/log/pki/pki-tomcat/ca/debug
You stated you had created this directory yourself.
What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory?
rob
/var/log/pki/pki-tomcat/ca/logs are both empty.
On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden rcritten@redhat.com wrote: > > Sina Owolabi wrote: >> Hi Florence >> >> and thanks for the help. >> ipactl status: >> [root@services ~]# ipactl status --ignore-service-failure; cat >> Directory Service: RUNNING >> krb5kdc Service: RUNNING >> kadmin Service: RUNNING >> named Service: RUNNING >> httpd Service: RUNNING >> ipa-custodia Service: RUNNING >> ntpd Service: RUNNING >> pki-tomcatd Service: STOPPED >> ipa-otpd Service: RUNNING >> ipa-dnskeysyncd Service: RUNNING >> ipa: INFO: The ipactl command was successful >> >> >> systemctl status -l pki-tomcatd@pki-tomcat.service; cat >> ? pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat >> Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; >> vendor preset: disabled) >> Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago >> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, >> status=0/SUCCESS) >> Main PID: 1376 (java) >> CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service >> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java >> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath >> /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar >> -Dcatalina.base=/var/lib/pki/pki-tomcat >> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= >> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp >> -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >> -Djava.security.manager >> -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy >> org.apache.catalina.startup.Bootstrap start >> >> systemctl status pki-tomcatd@pki-tomcat.service: >> >> Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception >> processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f >> background process >> Mar 05 09:40:43 services.qrios.com server[1376]: >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> Mar 05 09:40:43 services.qrios.com server[1376]: at >> java.lang.Thread.run(Thread.java:748) > > The logs will contain much more useful information. dogtag keeps > changing the location of the logs and I forget exactly where it is in > your version but it's somewhere in /var/log/pki*/pki*/ca/... > > The log may be named debug or debug-<date> > > Also look at the selftest log in the same directory. > > There are a LOT of red herrings in the dogtag logs so proceed with caution. > > You do not need to touch or create anything for this logging to take > place. You should delete the directory you created. > > rob > > >> >> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud flo@redhat.com wrote: >>> >>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: >>>> Hi! >>>> >>>> I tried to follow this solution for cert renewal for RHEL6: >>>> https://access.redhat.com/solutions/643753 (Sorry, desperation is >>>> setting in), but when I attempted Step 2, I got: >>>> >>> Hi, >>> >>> 1. this note was written for RHEL 6 but you said in your first e-mail >>> that your server is running CentOS 7 with ipa 4.5.4. Please don't follow >>> those instructions as they are not adapted to your deployment. >>> The instructions for RHEL 7 are available at >>> https://access.redhat.com/solutions/3357261. >>> >>> 2. In a previous e-mail, the output of getcert list | grep -i expires >>> did not show any expired certificates, so I would not rush into wrong >>> conclusions. We need to understand first why pki did not start. >>> >>> What is the output of: >>> $ ipactl status >>> $ systemctl status pki-tomcatd@pki-tomcat.service >>> >>> flo >>> >>>> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert >>>> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do >>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" >>>> | grep -i after; done >>>> auditSigningCert cert-pki-ca >>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>> certificate/key database is in an old, unsupported format. >>>> ocspSigningCert cert-pki-ca >>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>> certificate/key database is in an old, unsupported format. >>>> subsystemCert cert-pki-ca >>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>> certificate/key database is in an old, unsupported format. >>>> Server-Cert cert-pki-ca >>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>> certificate/key database is in an old, unsupported format. >>>> >>>> Could this be the root of my problems? >>>> And how can I convert them? >>>> >>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi notify.sina@gmail.com wrote: >>>>> >>>>> Restarting ipa didnt create the logs. >>>>> Please, what else can i do? >>>>> >>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi notify.sina@gmail.com wrote: >>>>>> >>>>>> Hi! >>>>>> >>>>>> getcert list | grep -i expires >>>>>> expires: 2019-04-13 12:08:20 UTC >>>>>> expires: 2019-04-13 12:08:06 UTC >>>>>> expires: 2019-04-13 12:07:50 UTC >>>>>> expires: 2035-06-01 08:33:01 UTC >>>>>> expires: 2019-04-13 12:07:41 UTC >>>>>> expires: 2019-04-13 12:06:55 UTC >>>>>> expires: 2019-05-05 12:06:41 UTC >>>>>> expires: 2019-05-05 12:06:56 UTC >>>>>> expires: 2020-01-17 19:56:03 UTC >>>>>> >>>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am >>>>>> creating one and running "ipactl restart". >>>>>> >>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden rcritten@redhat.com wrote: >>>>>>> >>>>>>> Sina Owolabi via FreeIPA-users wrote: >>>>>>>> Hi! >>>>>>>> >>>>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version 4.5.4, >>>>>>>> api version 2.228), with one master, and two replicas, and I noticed >>>>>>>> that pki-tomcatd no longer works on the master, after attempting a >>>>>>>> reboot. >>>>>>>> pki-tomcatd works fine on the slaves. >>>>>>>> I noticed if I try to run IPA functions (dns record removal, hosts >>>>>>>> management, user passwords, etc), I receive responses like this: >>>>>>>> >>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>>> communicate with CMS (Internal Server Error) >>>>>>>> But on the replicas, functions work fine. >>>>>>>> Please can someone guide me on how to fix this? >>>>>>> >>>>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have some >>>>>>> pointers. I'd look at selftests.log first. >>>>>>> >>>>>>> My guess is that some of the CA certificates have failed to renew. >>>>>>> >>>>>>> getcert list | grep -i expires >>>>>>> >>>>>>> rob >>>> _______________________________________________ >>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >>>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org >>>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... >>>> >>> >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Rob Crittenden -
Simon Williams -
Sina Owolabi