Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hello all!
I have very similiar problem as this one: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
ipa-server-upgrade fails as below
-- Update complete Upgrading IPA services Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. CA did not start in 300.0s The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information --
And the log tells me that CA returns status 500
-- DEBUG Waiting for CA to start... DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatushttp://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus DEBUG request body '' DEBUG response status 500 DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 2208 Date: Fri, 15 Jun 2018 10:05:29 GMT Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 DEBUG Waiting for CA to start... ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s ERROR CA did not start in 300.0s --
With command "ipactl start --ignore-service-failures" I can start all the services but pki-tomcatd.
-- Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING pki-tomcatd Service: STOPPED smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful --
Suggested resolution to above problem doesn't help me since the LDAP and NSS DB seem to have same certificates (some difference in wrapping but the string is same if I take out the line breaks) and even the serial number matches.
-- certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDjD... ...Prh2G -----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep Serial Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDjD... ...Prh2 G description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem, O=<<REALM>> seeAlso: CN=CA Subsystem,O=<<REALM>> --
And here's where my actual knowledge of things end. I've been trying to figure out all kind of logs (tomcat, Kerberos, directory server, ...) but haven't found a solid reason for it. I'm starting to believe this is a certificate issue, because although "getcert list" tells me that the certificate status is "Monitoring" on all certificates the expiry date is already in the past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certificates and it won't update even if I resubmit it or delete certificate and manually redo it (it got the same date as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", "Server-Cert cert-pki-ca" and two directory server certs) are valid for years (until 2020+).
-- Request ID '20160331084234': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=OCSP Subsystem,O=<<REALM>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331085008': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>> expires: 2020-03-04 09:58:23 UTC principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes --
Has anyone else bumped into same kind of issues? Any ideas where I should continue looking? I'm starting to run out of ideas...
Eemeli Jokinen
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
ERROR CA did not start in 300.0s
--
With command “ipactl start --ignore-service-failures” I can start all the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn’t help me since the LDAP and NSS DB seem to have same certificates (some difference in wrapping but the string is same if I take out the line breaks) and even the serial number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD…
…Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD…
…Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here’s where my actual knowledge of things end. I’ve been trying to figure out all kind of logs (tomcat, Kerberos, directory server, …) but haven’t found a solid reason for it. I’m starting to believe this is a certificate issue, because although “getcert list” tells me that the certificate status is “Monitoring” on all certificates the expiry date is already in the past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certificates and it won’t update even if I resubmit it or delete certificate and manually redo it (it got the same date as the “old ones”). The “main certs” (“caSigningCert cert-pki-ca”, “Server-Cert cert-pki-ca” and two directory server certs) are valid for years (until 2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I should continue looking? I’m starting to run out of ideas…
Eemeli Jokinen
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi,
does your topology include multiple CA instances? You need first to find which master is the CA renewal master: ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent with the content of the LDAP entries. If it is not the case, you need to repair the CA renewal master first.
When the CA renewal master is OK, check if the replication is working with the other CA instances, and repair the other masters. HTH, Flo
Hi!
We do have 2 IPA nodes configured but the second node has been down for some time. Tried to update it to same version as node1: - Won't start tells me to use ipa-server-upgrade - Ipa-server-upgrade fails at start, doesn't start directory server - /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse errors but last row tells me -- EMERG - main - Fatal Error---No ports specified. Exiting now. --
On node1 ipa config-show results only -- ipa: ERROR: did not receive Kerberos credentials --
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: keskiviikko 20. kesäkuuta 2018 19.49 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedor ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whit e;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font -size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstrai nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authenti catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.va lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache. catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg. apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445 )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstract Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abstr actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache. tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\ n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Thr eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre ad.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade .py", line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
ERROR CA did not start in 300.0s
--
With command "ipactl start --ignore-service-failures" I can start all the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn't help me since the LDAP and NSS DB seem to have same certificates (some difference in wrapping but the string is same if I take out the line breaks) and even the serial number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD.
.Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD.
.Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here's where my actual knowledge of things end. I've been trying to figure out all kind of logs (tomcat, Kerberos, directory server, .) but haven't found a solid reason for it. I'm starting to believe this is a certificate issue, because although "getcert list" tells me that the certificate status is "Monitoring" on all certificates the expiry date is already in the past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certificates and it won't update even if I resubmit it or delete certificate and manually redo it (it got the same date as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", "Server-Cert cert-pki-ca" and two directory server certs) are valid for years (until 2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC ert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC ert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N SS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I should continue looking? I'm starting to run out of ideas.
Eemeli Jokinen
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
Hi,
does your topology include multiple CA instances? You need first to find which master is the CA renewal master: ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent with the content of the LDAP entries. If it is not the case, you need to repair the CA renewal master first.
When the CA renewal master is OK, check if the replication is working with the other CA instances, and repair the other masters. HTH, Flo
On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
We do have 2 IPA nodes configured but the second node has been down for some time. Tried to update it to same version as node1:
- Won't start tells me to use ipa-server-upgrade
- Ipa-server-upgrade fails at start, doesn't start directory server
- /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse errors but last row tells me
-- EMERG - main - Fatal Error---No ports specified. Exiting now. --
Hi,
in this case let's concentrate on node1.
On node1 ipa config-show results only
ipa: ERROR: did not receive Kerberos credentials
You need to run kinit to get kerberos credentials before any ipa * command. If the ipa stack is stopped because pki-tomcat refused to start, you can force the start up, ignoring failed services: $ sudo ipactl start --ignore-service-failures
Then to check who is renewal master: $ sudo kinit admin $ sudo ipa config-show
If node1 is not the renewal master, make it the renewal master from now on (follow instructions from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...). After this step, check the certificates in the NSS databases vs the certificates stored in LDAP.
HTH, Flo
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: keskiviikko 20. kesäkuuta 2018 19.49 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedor ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whit e;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525 D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font -size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstrai nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authenti catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.va lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache. catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg. apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445 )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstract Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abstr actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache. tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\ n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Thr eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre ad.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade .py", line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
ERROR CA did not start in 300.0s
--
With command "ipactl start --ignore-service-failures" I can start all the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn't help me since the LDAP and NSS DB seem to have same certificates (some difference in wrapping but the string is same if I take out the line breaks) and even the serial number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD.
.Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD.
.Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here's where my actual knowledge of things end. I've been trying to figure out all kind of logs (tomcat, Kerberos, directory server, .) but haven't found a solid reason for it. I'm starting to believe this is a certificate issue, because although "getcert list" tells me that the certificate status is "Monitoring" on all certificates the expiry date is already in the past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certificates and it won't update even if I resubmit it or delete certificate and manually redo it (it got the same date as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", "Server-Cert cert-pki-ca" and two directory server certs) are valid for years (until 2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC ert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC ert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N SS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I should continue looking? I'm starting to run out of ideas.
Eemeli Jokinen
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
Hi,
does your topology include multiple CA instances? You need first to find which master is the CA renewal master: ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent with the content of the LDAP entries. If it is not the case, you need to repair the CA renewal master first.
When the CA renewal master is OK, check if the replication is working with the other CA instances, and repair the other masters. HTH, Flo _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi!
Forgot kinit:
-- kinit admin Password for admin@<<REALM>>: klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@<<REALM>>
Valid starting Expires Service principal 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<<ipa1.fqdn>>@<<REALM>> 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<<REALM>>@<<REALM>> ipa config-show ipa: ERROR: No valid Negotiate header in server response --
Still no luck getting ipa config
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: torstai 21. kesäkuuta 2018 9.43 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
We do have 2 IPA nodes configured but the second node has been down for some time. Tried to update it to same version as node1:
- Won't start tells me to use ipa-server-upgrade
- Ipa-server-upgrade fails at start, doesn't start directory server
- /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse
errors but last row tells me
EMERG - main - Fatal Error---No ports specified. Exiting now.
Hi,
in this case let's concentrate on node1.
On node1 ipa config-show results only
ipa: ERROR: did not receive Kerberos credentials
You need to run kinit to get kerberos credentials before any ipa * command. If the ipa stack is stopped because pki-tomcat refused to start, you can force the start up, ignoring failed services: $ sudo ipactl start --ignore-service-failures
Then to check who is renewal master: $ sudo kinit admin $ sudo ipa config-show
If node1 is not the renewal master, make it the renewal master from now on (follow instructions from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...). After this step, check the certificates in the NSS databases vs the certificates stored in LDAP.
HTH, Flo
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: keskiviikko 20. kesäkuuta 2018 19.49 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedo r ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whi t e;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;fon t -size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstra i nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authent i catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.v a lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache. catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg. apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:44 5 )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstrac t Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abst r actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache. tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) \ n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut o r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Th r eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea d $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thr e ad.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrad e .py", line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
ERROR CA did not start in 300.0s
--
With command "ipactl start --ignore-service-failures" I can start all the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn't help me since the LDAP and NSS DB seem to have same certificates (some difference in wrapping but the string is same if I take out the line breaks) and even the serial number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD.
.Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD.
.Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here's where my actual knowledge of things end. I've been trying to figure out all kind of logs (tomcat, Kerberos, directory server, .) but haven't found a solid reason for it. I'm starting to believe this is a certificate issue, because although "getcert list" tells me that the certificate status is "Monitoring" on all certificates the expiry date is already in the past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certificates and it won't update even if I resubmit it or delete certificate and manually redo it (it got the same date as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", "Server-Cert cert-pki-ca" and two directory server certs) are valid for years (until 2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning C ert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning C ert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token=' N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token=' N SS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I should continue looking? I'm starting to run out of ideas.
Eemeli Jokinen
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fed o rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
Hi,
does your topology include multiple CA instances? You need first to find which master is the CA renewal master: ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent with the content of the LDAP entries. If it is not the case, you need to repair the CA renewal master first.
When the CA renewal master is OK, check if the replication is working with the other CA instances, and repair the other masters. HTH, Flo _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/
On 06/21/2018 08:57 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Forgot kinit:
-- kinit admin Password for admin@<<REALM>>: klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@<<REALM>>
Valid starting Expires Service principal 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<<ipa1.fqdn>>@<<REALM>> 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<<REALM>>@<<REALM>> ipa config-show ipa: ERROR: No valid Negotiate header in server response --
Hi, can you check if the service gssproxy is running: # systemctl status gssproxy and if not, restart it.
Flo
Still no luck getting ipa config
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: torstai 21. kesäkuuta 2018 9.43 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
We do have 2 IPA nodes configured but the second node has been down for some time. Tried to update it to same version as node1:
- Won't start tells me to use ipa-server-upgrade
- Ipa-server-upgrade fails at start, doesn't start directory server
- /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse
errors but last row tells me
EMERG - main - Fatal Error---No ports specified. Exiting now.
Hi,
in this case let's concentrate on node1.
On node1 ipa config-show results only
ipa: ERROR: did not receive Kerberos credentials
You need to run kinit to get kerberos credentials before any ipa * command. If the ipa stack is stopped because pki-tomcat refused to start, you can force the start up, ignoring failed services: $ sudo ipactl start --ignore-service-failures
Then to check who is renewal master: $ sudo kinit admin $ sudo ipa config-show
If node1 is not the renewal master, make it the renewal master from now on (follow instructions from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...). After this step, check the certificates in the NSS databases vs the certificates stored in LDAP.
HTH, Flo
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: keskiviikko 20. kesäkuuta 2018 19.49 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedo r ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whi t e;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 5 D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;fon t -size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstra i nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authent i catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.v a lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache. catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg. apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:44 5 )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstrac t Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abst r actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache. tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) \ n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut o r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Th r eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea d $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thr e ad.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrad e .py", line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
ERROR CA did not start in 300.0s
--
With command "ipactl start --ignore-service-failures" I can start all the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn't help me since the LDAP and NSS DB seem to have same certificates (some difference in wrapping but the string is same if I take out the line breaks) and even the serial number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD.
.Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD.
.Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here's where my actual knowledge of things end. I've been trying to figure out all kind of logs (tomcat, Kerberos, directory server, .) but haven't found a solid reason for it. I'm starting to believe this is a certificate issue, because although "getcert list" tells me that the certificate status is "Monitoring" on all certificates the expiry date is already in the past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certificates and it won't update even if I resubmit it or delete certificate and manually redo it (it got the same date as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", "Server-Cert cert-pki-ca" and two directory server certs) are valid for years (until 2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning C ert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning C ert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token=' N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token=' N SS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I should continue looking? I'm starting to run out of ideas.
Eemeli Jokinen
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fed o rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
Hi,
does your topology include multiple CA instances? You need first to find which master is the CA renewal master: ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent with the content of the LDAP entries. If it is not the case, you need to repair the CA renewal master first.
When the CA renewal master is OK, check if the replication is working with the other CA instances, and repair the other masters. HTH, Flo _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi!
gssproxy up and running
-- systemctl status gssproxy ● gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 days ago Process: 3807 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS) --
Also seems like there's some default configuration of gssproxy, no ipa.conf (googling said that there should probably be also ipa.conf?).
-- ls /etc/gssproxy/ 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf --
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: perjantai 22. kesäkuuta 2018 15.39 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/21/2018 08:57 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Forgot kinit:
-- kinit admin Password for admin@<<REALM>>: klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@<<REALM>>
Valid starting Expires Service principal 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<<ipa1.fqdn>>@<<REALM>> 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<<REALM>>@<<REALM>> ipa config-show ipa: ERROR: No valid Negotiate header in server response --
Hi, can you check if the service gssproxy is running: # systemctl status gssproxy and if not, restart it.
Flo
Still no luck getting ipa config
Eemeli
On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
gssproxy up and running
-- systemctl status gssproxy ● gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 days ago Process: 3807 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS) --
Also seems like there's some default configuration of gssproxy, no ipa.conf (googling said that there should probably be also ipa.conf?).
-- ls /etc/gssproxy/ 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf --
Hi, you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file should be created during ipa-server-upgrade, but after the step restarting pki-tomcat.
So let's go back to our initial goal: finding which master is the renewal master. You can use a ldapsearch query to find out the renewal master: # ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,$BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=myrenewalmaster.domain.com,cn=masters,cn=ipa,cn=etc,$BASEDN
(replace BASEDN with your own setting that can be found in /etc/ipa/default.conf)
Flo
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: perjantai 22. kesäkuuta 2018 15.39 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/21/2018 08:57 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Forgot kinit:
-- kinit admin Password for admin@<<REALM>>: klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@<<REALM>>
Valid starting Expires Service principal 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<<ipa1.fqdn>>@<<REALM>> 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<<REALM>>@<<REALM>> ipa config-show ipa: ERROR: No valid Negotiate header in server response --
Hi, can you check if the service gssproxy is running: # systemctl status gssproxy and if not, restart it.
Flo
Still no luck getting ipa config
Eemeli
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi!
The node 1 is the Renewal Master -- ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=<<ipa1.fqdn>>,cn=masters,cn=ipa,cn=etc,BASEDN --
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: maanantai 25. kesäkuuta 2018 12.53 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
gssproxy up and running
-- systemctl status gssproxy ● gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 days ago Process: 3807 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS) --
Also seems like there's some default configuration of gssproxy, no ipa.conf (googling said that there should probably be also ipa.conf?).
-- ls /etc/gssproxy/ 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf --
Hi, you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file should be created during ipa-server-upgrade, but after the step restarting pki-tomcat.
So let's go back to our initial goal: finding which master is the renewal master. You can use a ldapsearch query to find out the renewal master: # ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,$BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=myrenewalmaster.domain.com,cn=masters,cn=ipa,cn=etc,$BASEDN
(replace BASEDN with your own setting that can be found in /etc/ipa/default.conf)
Flo
On 06/25/2018 01:59 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
The node 1 is the Renewal Master
ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=<<ipa1.fqdn>>,cn=masters,cn=ipa,cn=etc,BASEDN --
OK, so we know that your host node1 is the renewal master and it has 4 expired certificates. What is the full output of getcert list?
The journal will show why it was not able to renew them: # journalctl -u certmonger
Can you also provide the version of FreeIPA you are using, and the one you had before the upgrade? (can be found in /var/log/ipaupgrade.log with the string "IPA version 4.xx", this file keeps the whole upgrade history).
Flo
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: maanantai 25. kesäkuuta 2018 12.53 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
gssproxy up and running
-- systemctl status gssproxy ● gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 days ago Process: 3807 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS) --
Also seems like there's some default configuration of gssproxy, no ipa.conf (googling said that there should probably be also ipa.conf?).
-- ls /etc/gssproxy/ 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf --
Hi, you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file should be created during ipa-server-upgrade, but after the step restarting pki-tomcat.
So let's go back to our initial goal: finding which master is the renewal master. You can use a ldapsearch query to find out the renewal master: # ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,$BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=myrenewalmaster.domain.com,cn=masters,cn=ipa,cn=etc,$BASEDN
(replace BASEDN with your own setting that can be found in /etc/ipa/default.conf)
Flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hello!
Thank you for your answers by the way, seems like we're getting closer and closer every step although haven't had a breakthrough yet... At least I feel like I understand the structure of IPA better alredy! A bit long message incoming... :)
First getcert list. Some sites say that there should be 9 certificates listed as of ipa-server 4.5
-- getcert list Number of certificates and requests being tracked: 8. Request ID '20160331084233': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=CA Audit,O=<<DOMAIN>> expires: 2018-03-21 09:42:06 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084234': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=OCSP Subsystem,O=<<DOMAIN>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084236': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=Certificate Authority,O=<<DOMAIN>> expires: 2036-03-31 08:42:02 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084238': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>> expires: 2020-02-11 09:58:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca” track: yes auto-renew: yes Request ID '20160331084308': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<<REALM>>',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<<REALM>>/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-<<REALM>>',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>> expires: 2020-03-04 09:58:32 UTC principal name: ldap/<<ipa1.fqdn>>@<<DOMAIN>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <<REALM>> track: yes auto-renew: yes Request ID '20160331085008': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>> expires: 2020-03-04 09:58:23 UTC principal name: HTTP/<<ipa1.fqdn>>@<<DOMAIN>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20180611071929': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=IPA RA,O=<<DOMAIN>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20180615083528': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=CA Subsystem,O=<<DOMAIN>> expires: 2018-03-21 09:42:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes --
Next journalctl... I've tried changing the date of the server back to older days to get certmonger automatically renew them. Should I try this one again?
-- journalctl -u certmonger -- Logs begin at Mon 2018-06-25 17:46:25 EEST, end at Tue 2018-06-26 10:43:30 EEST. -- Jun 25 17:46:27 <<ipa1.fqdn>> certmonger[16802]: Certificate named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 25 17:46:29 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16804]: Forwarding request to dogtag-ipa-renew-agent Jun 25 17:46:29 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16804]: dogtag-ipa-renew-agent returned 2 Jun 25 17:46:36 <<ipa1.fqdn>> certmonger[16822]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 25 17:46:39 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16824]: Forwarding request to dogtag-ipa-renew-agent Jun 25 17:46:39 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16824]: dogtag-ipa-renew-agent returned 2 Jun 25 17:46:41 <<ipa1.fqdn>> certmonger[16839]: Certificate in file "/var/lib/ipa/ra-agent.pem" is no longer valid. Jun 25 17:46:43 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16841]: Forwarding request to dogtag-ipa-renew-agent Jun 25 17:46:43 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16841]: dogtag-ipa-renew-agent returned 2 ... Jun 26 10:40:47 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2530]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:40:47 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2530]: dogtag-ipa-renew-agent returned 2 Jun 26 10:40:48 <<ipa1.fqdn>> certmonger[2546]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:40:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2548]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:40:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2548]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:15 <<ipa1.fqdn>> certmonger[2580]: Certificate named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:17 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2582]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:17 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2582]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:18 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2594]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:18 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2594]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:20 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2608]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:20 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2608]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:21 <<ipa1.fqdn>> certmonger[2624]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:24 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2626]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:24 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2626]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:48 <<ipa1.fqdn>> certmonger[2667]: Certificate named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:50 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2669]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2669]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2682]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2682]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:53 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2697]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:53 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2697]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:54 <<ipa1.fqdn>> certmonger[2713]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:57 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2715]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:57 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2715]: dogtag-ipa-renew-agent returned 2 --
About versions: OS CentOS 7.5.1804 Current IPA version 4.5.4-10.el7.centos.1 (from ipaupgrade.log) Previous IPA version 4.2.0-15.0.1.el7.centos.6 (from ipaserver-install.log) The date of the ipaserver-install.log is 2016.03.31 so exactly 720 days before the expire date of those 4 certificates...
I tought I had upgraded it once before but probably I just remember it wrong (we have a test environment also and it might be that I updated that one as part of troubleshooting process of another problem) because can't find any mark of it.
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: tiistai 26. kesäkuuta 2018 10.27 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/25/2018 01:59 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
The node 1 is the Renewal Master
ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=<<ipa1.fqdn>>,cn=masters,cn=ipa,cn=etc,BASEDN --
OK, so we know that your host node1 is the renewal master and it has 4 expired certificates. What is the full output of getcert list?
The journal will show why it was not able to renew them: # journalctl -u certmonger
Can you also provide the version of FreeIPA you are using, and the one you had before the upgrade? (can be found in /var/log/ipaupgrade.log with the string "IPA version 4.xx", this file keeps the whole upgrade history).
Flo
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: maanantai 25. kesäkuuta 2018 12.53 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
gssproxy up and running
-- systemctl status gssproxy ● gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 days ago Process: 3807 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS) --
Also seems like there's some default configuration of gssproxy, no ipa.conf (googling said that there should probably be also ipa.conf?).
-- ls /etc/gssproxy/ 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf --
Hi, you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file should be created during ipa-server-upgrade, but after the step restarting pki-tomcat.
So let's go back to our initial goal: finding which master is the renewal master. You can use a ldapsearch query to find out the renewal master: # ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,$BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=myrenewalmaster.domain.com,cn=masters,cn=ipa,cn=etc,$BASEDN
(replace BASEDN with your own setting that can be found in /etc/ipa/default.conf)
Flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/VMQPV3EF4XN2QYAFQEG63KU5YNQW64TX/
On 06/26/2018 09:58 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hello!
Thank you for your answers by the way, seems like we're getting closer and closer every step although haven't had a breakthrough yet... At least I feel like I understand the structure of IPA better alredy! A bit long message incoming... :)
First getcert list. Some sites say that there should be 9 certificates listed as of ipa-server 4.5
-- getcert list Number of certificates and requests being tracked: 8. Request ID '20160331084233': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=CA Audit,O=<<DOMAIN>> expires: 2018-03-21 09:42:06 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084234': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=OCSP Subsystem,O=<<DOMAIN>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084236': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=Certificate Authority,O=<<DOMAIN>> expires: 2036-03-31 08:42:02 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084238': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>> expires: 2020-02-11 09:58:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca” track: yes auto-renew: yes Request ID '20160331084308': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<<REALM>>',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<<REALM>>/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-<<REALM>>',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>> expires: 2020-03-04 09:58:32 UTC principal name: ldap/<<ipa1.fqdn>>@<<DOMAIN>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <<REALM>> track: yes auto-renew: yes Request ID '20160331085008': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>> expires: 2020-03-04 09:58:23 UTC principal name: HTTP/<<ipa1.fqdn>>@<<DOMAIN>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20180611071929': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=IPA RA,O=<<DOMAIN>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20180615083528': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<DOMAIN>> subject: CN=CA Subsystem,O=<<DOMAIN>> expires: 2018-03-21 09:42:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes --
Hi,
the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejected" (see [1] for the return codes). This probably happens because the cert for IPA RA is no longer valid (this cert is used to authenticate to Dogtag, and without proper authentication any renewal op is refused).
The expired certificates all expire on 2018-03-21. On the other hand, ServerCert cert-pki-ca, slapd and httpd certificates were properly renewed. You need to find at which date they were renewed: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' | grep "Not Before") # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Before" # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before"
You need then to find a common date where all the certificates are valid (ie before 2018-03-21 so that the expired certs are not expired yet, and after the 'Not Before' date so that the renewed certs are already valid). Then stop ntpd, change the date to this common date, restart certmonger and look in the journal if the renewal goes smoothly or if there are errors that could point you in the right direction.
You can also find instructions on this blog post [2] to increase the log level for the renewal.
HTH, Flo
[1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 [2] https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-...
Next journalctl... I've tried changing the date of the server back to older days to get certmonger automatically renew them. Should I try this one again?
-- journalctl -u certmonger -- Logs begin at Mon 2018-06-25 17:46:25 EEST, end at Tue 2018-06-26 10:43:30 EEST. -- Jun 25 17:46:27 <<ipa1.fqdn>> certmonger[16802]: Certificate named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 25 17:46:29 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16804]: Forwarding request to dogtag-ipa-renew-agent Jun 25 17:46:29 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16804]: dogtag-ipa-renew-agent returned 2 Jun 25 17:46:36 <<ipa1.fqdn>> certmonger[16822]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 25 17:46:39 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16824]: Forwarding request to dogtag-ipa-renew-agent Jun 25 17:46:39 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16824]: dogtag-ipa-renew-agent returned 2 Jun 25 17:46:41 <<ipa1.fqdn>> certmonger[16839]: Certificate in file "/var/lib/ipa/ra-agent.pem" is no longer valid. Jun 25 17:46:43 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16841]: Forwarding request to dogtag-ipa-renew-agent Jun 25 17:46:43 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16841]: dogtag-ipa-renew-agent returned 2 ... Jun 26 10:40:47 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2530]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:40:47 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2530]: dogtag-ipa-renew-agent returned 2 Jun 26 10:40:48 <<ipa1.fqdn>> certmonger[2546]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:40:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2548]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:40:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2548]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:15 <<ipa1.fqdn>> certmonger[2580]: Certificate named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:17 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2582]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:17 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2582]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:18 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2594]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:18 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2594]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:20 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2608]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:20 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2608]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:21 <<ipa1.fqdn>> certmonger[2624]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:24 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2626]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:24 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2626]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:48 <<ipa1.fqdn>> certmonger[2667]: Certificate named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:50 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2669]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2669]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2682]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2682]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:53 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2697]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:53 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2697]: dogtag-ipa-renew-agent returned 2 Jun 26 10:41:54 <<ipa1.fqdn>> certmonger[2713]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid. Jun 26 10:41:57 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2715]: Forwarding request to dogtag-ipa-renew-agent Jun 26 10:41:57 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2715]: dogtag-ipa-renew-agent returned 2 --
About versions: OS CentOS 7.5.1804 Current IPA version 4.5.4-10.el7.centos.1 (from ipaupgrade.log) Previous IPA version 4.2.0-15.0.1.el7.centos.6 (from ipaserver-install.log) The date of the ipaserver-install.log is 2016.03.31 so exactly 720 days before the expire date of those 4 certificates...
I tought I had upgraded it once before but probably I just remember it wrong (we have a test environment also and it might be that I updated that one as part of troubleshooting process of another problem) because can't find any mark of it.
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: tiistai 26. kesäkuuta 2018 10.27 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/25/2018 01:59 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
The node 1 is the Renewal Master
ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=<<ipa1.fqdn>>,cn=masters,cn=ipa,cn=etc,BASEDN --
OK, so we know that your host node1 is the renewal master and it has 4 expired certificates. What is the full output of getcert list?
The journal will show why it was not able to renew them: # journalctl -u certmonger
Can you also provide the version of FreeIPA you are using, and the one you had before the upgrade? (can be found in /var/log/ipaupgrade.log with the string "IPA version 4.xx", this file keeps the whole upgrade history).
Flo
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: maanantai 25. kesäkuuta 2018 12.53 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
gssproxy up and running
-- systemctl status gssproxy ● gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 days ago Process: 3807 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS) --
Also seems like there's some default configuration of gssproxy, no ipa.conf (googling said that there should probably be also ipa.conf?).
-- ls /etc/gssproxy/ 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf --
Hi, you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file should be created during ipa-server-upgrade, but after the step restarting pki-tomcat.
So let's go back to our initial goal: finding which master is the renewal master. You can use a ldapsearch query to find out the renewal master: # ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,$BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password: dn: cn=CA,cn=myrenewalmaster.domain.com,cn=masters,cn=ipa,cn=etc,$BASEDN
(replace BASEDN with your own setting that can be found in /etc/ipa/default.conf)
Flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/VMQPV3EF4XN2QYAFQEG63KU5YNQW64TX/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi!
-- certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |grep "Not Before" Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d /etc/dirsrv/slapd-<<REALM>> -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using https://access.redhat.com/solutions/3357261 as a guideline.
-- systemctl stop ntpd date 031603162018 Fri Mar 16 03:16:00 EET 2018 systemctl restart certmonger certutil -d /var/lib/pki/pki-tomcat/alias/ -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 "expires: 2018-03" | grep ID Request ID '20160331084233': Request ID '20160331084234': Request ID '20180611071929': Request ID '20180615083528': ipa-getcert resubmit -i 20160331084233 -v Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20160331084234 -v Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180611071929 -v Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180615083528 -v Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". journalctl -n 20 -u certmonger -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certificate monitoring and PKI enrollment. Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agent returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET 2018 --
I waited for some time to be sure, no luck on my opinion:
-- date Fri Mar 16 03:52:24 EET 2018 getcert list |grep expires expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
Also did steps 6 & 8 on the guideline page, certificates match. However step 7 fails to error 500.
Still wondering if I'm missing some kind of cert from certmonger since the site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certificates on "getcert list", I only have 8. However if I try to do the tracking requests again as suggested by RHEL, I get no new certificates for my list.
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: tiistai 26. kesäkuuta 2018 21.28 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi,
the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejected" (see [1] for the return codes). This probably happens because the cert for IPA RA is no longer valid (this cert is used to authenticate to Dogtag, and without proper authentication any renewal op is refused).
The expired certificates all expire on 2018-03-21. On the other hand, ServerCert cert-pki-ca, slapd and httpd certificates were properly renewed. You need to find at which date they were renewed: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' | grep "Not Before") # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Before" # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before"
You need then to find a common date where all the certificates are valid (ie before 2018-03-21 so that the expired certs are not expired yet, and after the 'Not Before' date so that the renewed certs are already valid). Then stop ntpd, change the date to this common date, restart certmonger and look in the journal if the renewal goes smoothly or if there are errors that could point you in the right direction.
You can also find instructions on this blog post [2] to increase the log level for the renewal.
HTH, Flo
[1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 [2] https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-...
On 06/27/2018 08:56 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
-- certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |grep "Not Before" Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d /etc/dirsrv/slapd-<<REALM>> -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using https://access.redhat.com/solutions/3357261 as a guideline.
-- systemctl stop ntpd date 031603162018 Fri Mar 16 03:16:00 EET 2018 systemctl restart certmonger certutil -d /var/lib/pki/pki-tomcat/alias/ -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 "expires: 2018-03" | grep ID Request ID '20160331084233': Request ID '20160331084234': Request ID '20180611071929': Request ID '20180615083528': ipa-getcert resubmit -i 20160331084233 -v Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20160331084234 -v Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180611071929 -v Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180615083528 -v Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". journalctl -n 20 -u certmonger -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certificate monitoring and PKI enrollment. Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agent returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET 2018 --
I waited for some time to be sure, no luck on my opinion:
-- date Fri Mar 16 03:52:24 EET 2018 getcert list |grep expires expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
Also did steps 6 & 8 on the guideline page, certificates match. However step 7 fails to error 500.
Error 500 is internal error. Can you check the content of Dogtag log? /var/log/pki/pki-tomcat/localhost_access_log_$date.txt must show the command getCertChain has been received: 10.37.171.235 - - [date] "GET /ca/ee/ca/getCertChain HTTP/1.1" 200 1534
and /var/log/pki/pki-tomcat/ca/debug may show more information. On a working system: [date][http-bio-8443-exec-13]: SignedAuditLogger: event ACCESS_SESSION_ESTABLISH [date][http-bio-8443-exec-13]: according to ccMode, authorization for servlet: caGetCertChain is LDAP based, not XML {1}, use default authz mgr: {2}. [date][http-bio-8443-exec-13]: CMSServlet:service() uri = /ca/ee/ca/getCertChain [date][http-bio-8443-exec-13]: CMSServlet: caGetCertChain start to service. [date][http-bio-8443-exec-13]: GetCertChain: certificate chain: [date][http-bio-8443-exec-13]: GetCertChain: - CN=Certificate Authority,O=DOMAIN.COM [date][http-bio-8443-exec-13]: CMSServlet: curDate=Wed Jun 27 09:33:23 CEST 2018 id=caGetCertChain time=22 [date][http-bio-8443-exec-13]: SignedAuditLogger: event ACCESS_SESSION_TERMINATED
Flo
Still wondering if I'm missing some kind of cert from certmonger since the site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certificates on "getcert list", I only have 8. However if I try to do the tracking requests again as suggested by RHEL, I get no new certificates for my list.
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: tiistai 26. kesäkuuta 2018 21.28 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi,
the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejected" (see [1] for the return codes). This probably happens because the cert for IPA RA is no longer valid (this cert is used to authenticate to Dogtag, and without proper authentication any renewal op is refused).
The expired certificates all expire on 2018-03-21. On the other hand, ServerCert cert-pki-ca, slapd and httpd certificates were properly renewed. You need to find at which date they were renewed: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' | grep "Not Before") # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Before" # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before"
You need then to find a common date where all the certificates are valid (ie before 2018-03-21 so that the expired certs are not expired yet, and after the 'Not Before' date so that the renewed certs are already valid). Then stop ntpd, change the date to this common date, restart certmonger and look in the journal if the renewal goes smoothly or if there are errors that could point you in the right direction.
You can also find instructions on this blog post [2] to increase the log level for the renewal.
HTH, Flo
[1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 [2] https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi!
Checked access log for today date:
-- <<IP>> - - [27/Jun/2018:10:57:38 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=4&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:57:41 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=7&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:57:51 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:58:00 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=2&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:58:11 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=4&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:58:14 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=7&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:58:24 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:58:33 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=2&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>>- - [27/Jun/2018:10:58:44 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=4&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 <<IP>> - - [27/Jun/2018:10:58:47 +0300] "GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=7&renewal=true&xml=true&requestor_name=IPA HTTP/1.1" 500 2208 --
No other kind of responses, only timestamps vary.
There's no access_log-file with date 2018-03-16 but there is a Catalina.out-file with that date
-- Mar 16, 2018 3:16:06 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@4a53d31b background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748) --
This seems to have gotten date of which I used on my "time travel". The error matches 100% with Catalina.out with timestamp matching today.
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: keskiviikko 27. kesäkuuta 2018 10.40 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 06/27/2018 08:56 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
-- certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |grep "Not Before" Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d /etc/dirsrv/slapd-<<REALM>> -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using https://access.redhat.com/solutions/3357261 as a guideline.
-- systemctl stop ntpd date 031603162018 Fri Mar 16 03:16:00 EET 2018 systemctl restart certmonger certutil -d /var/lib/pki/pki-tomcat/alias/ -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 "expires: 2018-03" | grep ID Request ID '20160331084233': Request ID '20160331084234': Request ID '20180611071929': Request ID '20180615083528': ipa-getcert resubmit -i 20160331084233 -v Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20160331084234 -v Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180611071929 -v Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180615083528 -v Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". journalctl -n 20 -u certmonger -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certificate monitoring and PKI enrollment. Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agent returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET 2018 --
I waited for some time to be sure, no luck on my opinion:
-- date Fri Mar 16 03:52:24 EET 2018 getcert list |grep expires expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
Also did steps 6 & 8 on the guideline page, certificates match. However step 7 fails to error 500.
Error 500 is internal error. Can you check the content of Dogtag log? /var/log/pki/pki-tomcat/localhost_access_log_$date.txt must show the command getCertChain has been received: 10.37.171.235 - - [date] "GET /ca/ee/ca/getCertChain HTTP/1.1" 200 1534
and /var/log/pki/pki-tomcat/ca/debug may show more information. On a working system: [date][http-bio-8443-exec-13]: SignedAuditLogger: event ACCESS_SESSION_ESTABLISH [date][http-bio-8443-exec-13]: according to ccMode, authorization for servlet: caGetCertChain is LDAP based, not XML {1}, use default authz mgr: {2}. [date][http-bio-8443-exec-13]: CMSServlet:service() uri = /ca/ee/ca/getCertChain [date][http-bio-8443-exec-13]: CMSServlet: caGetCertChain start to service. [date][http-bio-8443-exec-13]: GetCertChain: certificate chain: [date][http-bio-8443-exec-13]: GetCertChain: - CN=Certificate Authority,O=DOMAIN.COM [date][http-bio-8443-exec-13]: CMSServlet: curDate=Wed Jun 27 09:33:23 CEST 2018 id=caGetCertChain time=22 [date][http-bio-8443-exec-13]: SignedAuditLogger: event ACCESS_SESSION_TERMINATED
Flo
Still wondering if I'm missing some kind of cert from certmonger since the site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certificates on "getcert list", I only have 8. However if I try to do the tracking requests again as suggested by RHEL, I get no new certificates for my list.
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: tiistai 26. kesäkuuta 2018 21.28 To: FreeIPA users list freeipa-users@lists.fedorahosted.org Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi,
the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejected" (see [1] for the return codes). This probably happens because the cert for IPA RA is no longer valid (this cert is used to authenticate to Dogtag, and without proper authentication any renewal op is refused).
The expired certificates all expire on 2018-03-21. On the other hand, ServerCert cert-pki-ca, slapd and httpd certificates were properly renewed. You need to find at which date they were renewed: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' | grep "Not Before") # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Before" # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before"
You need then to find a common date where all the certificates are valid (ie before 2018-03-21 so that the expired certs are not expired yet, and after the 'Not Before' date so that the renewed certs are already valid). Then stop ntpd, change the date to this common date, restart certmonger and look in the journal if the renewal goes smoothly or if there are errors that could point you in the right direction.
You can also find instructions on this blog post [2] to increase the log level for the renewal.
HTH, Flo
[1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 [2] https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-i ssues-with-freeipa/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/X6XG7L2WYYIHHT72V2OCRVSKINVRCPMU/
Jokinen Eemeli via FreeIPA-users wrote:
Hi!
-- certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |grep "Not Before" Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d /etc/dirsrv/slapd-<<REALM>> -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using https://access.redhat.com/solutions/3357261 as a guideline.
-- systemctl stop ntpd date 031603162018 Fri Mar 16 03:16:00 EET 2018 systemctl restart certmonger certutil -d /var/lib/pki/pki-tomcat/alias/ -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 "expires: 2018-03" | grep ID Request ID '20160331084233': Request ID '20160331084234': Request ID '20180611071929': Request ID '20180615083528': ipa-getcert resubmit -i 20160331084233 -v Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20160331084234 -v Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180611071929 -v Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180615083528 -v Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". journalctl -n 20 -u certmonger -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certificate monitoring and PKI enrollment. Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agent returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET 2018 --
I waited for some time to be sure, no luck on my opinion:
-- date Fri Mar 16 03:52:24 EET 2018 getcert list |grep expires expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
Also did steps 6 & 8 on the guideline page, certificates match. However step 7 fails to error 500.
Still wondering if I'm missing some kind of cert from certmonger since the site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certificates on "getcert list", I only have 8. However if I try to do the tracking requests again as suggested by RHEL, I get no new certificates for my list.
Hard to know without seeing the list of certs.
Are you restarting dogtag, Apache and 389-ds when setting the date back? That is necessary as well.
rob
Hi!
No I haven't since my guide line didn't tell me to.
I tried to set the date back, restart certmonger and then I did "ipactl restart" and then it got 2 certs renewed! One of the remaining two certificates was on "CA_UNREACHABLE" state, so I ran another certmonger restart and it did get updated. The last one didn't seem to go anywhere so I resubmitted the cert request and then that one also got renewed. I time jumped back to today and did another ipactl restart and All this mess got started with failed "ipa-server-upgrade" so I ran it afterwards and it completed successfully with no errors. That also increased the number of certmonger tracked certificates to 9 from 8 so I believe that one is fixed too.
Thank you a lot! It's a bit complicated mess to understand every aspect of it (for example I was trying to hunt missing certificate that certmonger didn't track even though it wasn't the issue but the outcome of failed server upgrade) but after this I believe very that I understand it a way better!
Eemeli
-----Original Message----- From: Rob Crittenden [mailto:rcritten@redhat.com] Sent: keskiviikko 27. kesäkuuta 2018 16.26 To: FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Jokinen Eemeli via FreeIPA-users wrote:
Hi!
-- certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |grep "Not Before" Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d /etc/dirsrv/slapd-<<REALM>> -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using https://access.redhat.com/solutions/3357261 as a guideline.
-- systemctl stop ntpd date 031603162018 Fri Mar 16 03:16:00 EET 2018 systemctl restart certmonger certutil -d /var/lib/pki/pki-tomcat/alias/ -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 "expires: 2018-03" | grep ID Request ID '20160331084233': Request ID '20160331084234': Request ID '20180611071929': Request ID '20180615083528': ipa-getcert resubmit -i 20160331084233 -v Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20160331084234 -v Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180611071929 -v Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180615083528 -v Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". journalctl -n 20 -u certmonger -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certificate monitoring and PKI enrollment. Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agent returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET 2018 --
I waited for some time to be sure, no luck on my opinion:
-- date Fri Mar 16 03:52:24 EET 2018 getcert list |grep expires expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
Also did steps 6 & 8 on the guideline page, certificates match. However step 7 fails to error 500.
Still wondering if I'm missing some kind of cert from certmonger since the site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certificates on "getcert list", I only have 8. However if I try to do the tracking requests again as suggested by RHEL, I get no new certificates for my list.
Hard to know without seeing the list of certs.
Are you restarting dogtag, Apache and 389-ds when setting the date back? That is necessary as well.
rob
Jokinen Eemeli wrote:
Hi!
No I haven't since my guide line didn't tell me to.
I tried to set the date back, restart certmonger and then I did "ipactl restart" and then it got 2 certs renewed! One of the remaining two certificates was on "CA_UNREACHABLE" state, so I ran another certmonger restart and it did get updated. The last one didn't seem to go anywhere so I resubmitted the cert request and then that one also got renewed. I time jumped back to today and did another ipactl restart and All this mess got started with failed "ipa-server-upgrade" so I ran it afterwards and it completed successfully with no errors. That also increased the number of certmonger tracked certificates to 9 from 8 so I believe that one is fixed too.
There are layers of dependencies on the certs so sometimes multiple rounds of renewal are needed to sort things out. This normally happens gracefully as expiration approaches but in some cases that we haven't been able to identify this doesn't happen.
Thank you a lot! It's a bit complicated mess to understand every aspect of it (for example I was trying to hunt missing certificate that certmonger didn't track even though it wasn't the issue but the outcome of failed server upgrade) but after this I believe very that I understand it a way better!
Cool, glad you are back up and running.
Note that the cert issues weren't caused by the upgrade, the upgrade just made it more apparent.
In order to be sure the upgrade is complete you should run: # ipa-server-upgrade
The upgrade will also check all of the certs tracked by certmonger and ensure they are set up correctly.
rob
Eemeli
-----Original Message----- From: Rob Crittenden [mailto:rcritten@redhat.com] Sent: keskiviikko 27. kesäkuuta 2018 16.26 To: FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Jokinen Eemeli via FreeIPA-users wrote:
Hi!
-- certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |grep "Not Before" Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d /etc/dirsrv/slapd-<<REALM>> -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using https://access.redhat.com/solutions/3357261 as a guideline.
-- systemctl stop ntpd date 031603162018 Fri Mar 16 03:16:00 EET 2018 systemctl restart certmonger certutil -d /var/lib/pki/pki-tomcat/alias/ -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 "expires: 2018-03" | grep ID Request ID '20160331084233': Request ID '20160331084234': Request ID '20180611071929': Request ID '20180615083528': ipa-getcert resubmit -i 20160331084233 -v Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20160331084234 -v Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180611071929 -v Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180615083528 -v Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". journalctl -n 20 -u certmonger -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certificate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certificate monitoring and PKI enrollment. Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agent returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET 2018 --
I waited for some time to be sure, no luck on my opinion:
-- date Fri Mar 16 03:52:24 EET 2018 getcert list |grep expires expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC --
Also did steps 6 & 8 on the guideline page, certificates match. However step 7 fails to error 500.
Still wondering if I'm missing some kind of cert from certmonger since the site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certificates on "getcert list", I only have 8. However if I try to do the tracking requests again as suggested by RHEL, I get no new certificates for my list.
Hard to know without seeing the list of certs.
Are you restarting dogtag, Apache and 389-ds when setting the date back? That is necessary as well.
rob
Hi!
I reply to this since there's some data in this message queue already related to my problem:
I had 2 ipa node cluster, where the second node had been offline for some time and at some point we had an error while trying to reboot node1 which was a Renewal Master. The issue was that some certs had expired and after a bit of special work we got the node1 back on track. I can spot three problems and I can't (again) figure out which one is the cause and which one I should repair first.
Now I got assigned the case to get the node2 back on track also. It had some certificates expired (obviously) so I did a small time jump and some of the certs were renewed. However not all of them were upgraded. "getcert list" reports 3 certs "CA Unreachable", other 3 certs seem fine.
-- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=OCSP Subsystem,O=<<REALM>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=IPA RA,O=<<REALM>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=<<ipa2.fqdn>>,O=<<REALM>> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection --
Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) but shouldn't node1 be the renewal master? Restarting httpd, certmonger and pki-tomcat don't seem to help, time traveling helped on other certs but not on these.
Directory service seems to work if I start it manually but ipa-server-upgrade fails on directory server not starting with "No ports specified" so something wrong with it or is it the certificates? -- ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server
-- <<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EMERG - main - Fatal Error---No ports specified. Exiting now. --
Also certmonger has issues: -- dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 43, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm '<<REALM>>' --
but KDCs should be able to be resolved even from ipa node2 -- nslookup -type=srv _kerberos._tcp.<<REALM>> Server: <<ipa1.ip>> Address: <<ipa1.ip>>#53
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>. _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>. --
For testing purposes I turned off firewall on ipa node1
Eemeli
-----Original Message----- From: Rob Crittenden [mailto:rcritten@redhat.com] Sent: torstai 28. kesäkuuta 2018 16.05 To: Jokinen Eemeli Eemeli.Jokinen@cinia.fi; FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Jokinen Eemeli wrote:
Hi!
No I haven't since my guide line didn't tell me to.
I tried to set the date back, restart certmonger and then I did "ipactl restart" and then it got 2 certs renewed! One of the remaining two certificates was on "CA_UNREACHABLE" state, so I ran another certmonger restart and it did get updated. The last one didn't seem to go anywhere so I resubmitted the cert request and then that one also got renewed. I time jumped back to today and did another ipactl restart and All this mess got started with failed "ipa-server-upgrade" so I ran it afterwards and it completed successfully with no errors. That also increased the number of certmonger tracked certificates to 9 from 8 so I believe that one is fixed too.
There are layers of dependencies on the certs so sometimes multiple rounds of renewal are needed to sort things out. This normally happens gracefully as expiration approaches but in some cases that we haven't been able to identify this doesn't happen.
Thank you a lot! It's a bit complicated mess to understand every aspect of it (for example I was trying to hunt missing certificate that certmonger didn't track even though it wasn't the issue but the outcome of failed server upgrade) but after this I believe very that I understand it a way better!
Cool, glad you are back up and running.
Note that the cert issues weren't caused by the upgrade, the upgrade just made it more apparent.
In order to be sure the upgrade is complete you should run: # ipa-server-upgrade
The upgrade will also check all of the certs tracked by certmonger and ensure they are set up correctly.
rob
Eemeli
-----Original Message----- From: Rob Crittenden [mailto:rcritten@redhat.com] Sent: keskiviikko 27. kesäkuuta 2018 16.26 To: FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hard to know without seeing the list of certs.
Are you restarting dogtag, Apache and 389-ds when setting the date back? That is necessary as well.
rob
Hi!
Anybody can help me with this one?
Summary:
2 node freeipa server cluster, node 2 was initially down for other reasons and node 1 (renewal master) had forgot to update own certificates which resulted faulty cluster. With help from mailing list we got the node 1 back online and it's working great! Now I'm trying to get node2 back to working order in cluster but it won't update the certificates even when trying the timejump. Seems like it tries to renew certificates locally although somehow I tought that it should renew the certificates from node 1...?
Eemeli
-----Original Message----- From: Jokinen Eemeli Sent: keskiviikko 4. heinäkuuta 2018 16.08 To: 'Rob Crittenden' rcritten@redhat.com; FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi!
I reply to this since there's some data in this message queue already related to my problem:
I had 2 ipa node cluster, where the second node had been offline for some time and at some point we had an error while trying to reboot node1 which was a Renewal Master. The issue was that some certs had expired and after a bit of special work we got the node1 back on track. I can spot three problems and I can't (again) figure out which one is the cause and which one I should repair first.
Now I got assigned the case to get the node2 back on track also. It had some certificates expired (obviously) so I did a small time jump and some of the certs were renewed. However not all of them were upgraded. "getcert list" reports 3 certs "CA Unreachable", other 3 certs seem fine.
-- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=OCSP Subsystem,O=<<REALM>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=IPA RA,O=<<REALM>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=<<ipa2.fqdn>>,O=<<REALM>> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection --
Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) but shouldn't node1 be the renewal master? Restarting httpd, certmonger and pki-tomcat don't seem to help, time traveling helped on other certs but not on these.
Directory service seems to work if I start it manually but ipa-server-upgrade fails on directory server not starting with "No ports specified" so something wrong with it or is it the certificates? -- ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server
-- <<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EMERG - main - Fatal Error---No ports specified. Exiting now. --
Also certmonger has issues: -- dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 43, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm '<<REALM>>' --
but KDCs should be able to be resolved even from ipa node2 -- nslookup -type=srv _kerberos._tcp.<<REALM>> Server: <<ipa1.ip>> Address: <<ipa1.ip>>#53
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>. _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>. --
For testing purposes I turned off firewall on ipa node1
Eemeli
On 08/15/2018 01:20 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Anybody can help me with this one?
Summary:
2 node freeipa server cluster, node 2 was initially down for other reasons and node 1 (renewal master) had forgot to update own certificates which resulted faulty cluster. With help from mailing list we got the node 1 back online and it's working great! Now I'm trying to get node2 back to working order in cluster but it won't update the certificates even when trying the timejump. Seems like it tries to renew certificates locally although somehow I tought that it should renew the certificates from node 1...?
Hi,
you probably have a combination of multiple issues on your second node.
The ipa-server-upgrade failure may leave your instance in a wrong state, where dse.ldif has disabled the ports for 389-ds (see BZ https://bugzilla.redhat.com/show_bug.cgi?id=1569011 or pagure ticket https://pagure.io/freeipa/issue/7534). During the upgrade, dse.ldif is edited in order to temporarily disable the LDAP ports (to prevent ldap modifications during the upgrade). Sometimes, if the upgrade fails, dse.ldif is not restored and the ports remain disabled. You will have to stop the ldap server, manually edit dse.ldif (in /etc/dirsrv/slapd-DOMxxx) and set: nsslapd-port: 389 nsslapd-security: on
then restart the LDAP server.
For the cert renewal, your procedure is the valid one. The kerberos error is probably linked to 389-ds not being accessible.
HTH, flo
Eemeli
-----Original Message----- From: Jokinen Eemeli Sent: keskiviikko 4. heinäkuuta 2018 16.08 To: 'Rob Crittenden' rcritten@redhat.com; FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi!
I reply to this since there's some data in this message queue already related to my problem:
I had 2 ipa node cluster, where the second node had been offline for some time and at some point we had an error while trying to reboot node1 which was a Renewal Master. The issue was that some certs had expired and after a bit of special work we got the node1 back on track. I can spot three problems and I can't (again) figure out which one is the cause and which one I should repair first.
Now I got assigned the case to get the node2 back on track also. It had some certificates expired (obviously) so I did a small time jump and some of the certs were renewed. However not all of them were upgraded. "getcert list" reports 3 certs "CA Unreachable", other 3 certs seem fine.
-- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=OCSP Subsystem,O=<<REALM>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=IPA RA,O=<<REALM>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=<<ipa2.fqdn>>,O=<<REALM>> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection --
Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) but shouldn't node1 be the renewal master? Restarting httpd, certmonger and pki-tomcat don't seem to help, time traveling helped on other certs but not on these.
Directory service seems to work if I start it manually but ipa-server-upgrade fails on directory server not starting with "No ports specified" so something wrong with it or is it the certificates?
ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server
--
<<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EMERG - main - Fatal Error---No ports specified. Exiting now.
Also certmonger has issues:
dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 43, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm '<<REALM>>' --
but KDCs should be able to be resolved even from ipa node2
nslookup -type=srv _kerberos._tcp.<<REALM>> Server: <<ipa1.ip>> Address: <<ipa1.ip>>#53
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>. _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>. --
For testing purposes I turned off firewall on ipa node1
Eemeli
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi!
Yes, seems like there was "security: off" but that doesn't seem to do it, I think I have ended up in the situation that I need to recreate some certificates, because:
I check the renewal dates.
-- getcert list |grep expires: expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-06-27 07:01:38 UTC expires: 2020-08-17 10:17:32 UTC expires: 2020-06-28 05:49:50 UTC --
I timejump to "before oldest expired" = 2018-03-20. Dirsrv seems to start ok. Certmonger restarts ok.
Httpd does not start. Error from /etc/httpd/logs/error_log:
-- [Tue Mar 20 07:44:39.500363 2018] [:warn] [pid 11961] NSSSessionCacheTimeout is deprecated. Ignoring. [Tue Mar 20 07:44:39.688595 2018] [:error] [pid 11961] SSL Library Error: -8181 Certificate has expired [Tue Mar 20 07:44:39.688637 2018] [:error] [pid 11961] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved. --
Seems like httpd has managed to renew some certificates at some point:
-- certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep Not Not Before: Thu Jun 28 05:49:50 2018 Not After : Sun Jun 28 05:49:50 2020 --
Should I remove httpd certificate to be able to start httpd in "before 21-03-2018? I can't seem to be able to renew these 2 (ipaCert, ocspSigningCert) without httpd because if I try to resubmit them I get "CA_Unreachable"?
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: torstai 16. elokuuta 2018 21.54 To: FreeIPA users list freeipa-users@lists.fedorahosted.org; Rob Crittenden rcritten@redhat.com Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 08/15/2018 01:20 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Anybody can help me with this one?
Summary:
2 node freeipa server cluster, node 2 was initially down for other reasons and node 1 (renewal master) had forgot to update own certificates which resulted faulty cluster. With help from mailing list we got the node 1 back online and it's working great! Now I'm trying to get node2 back to working order in cluster but it won't update the certificates even when trying the timejump. Seems like it tries to renew certificates locally although somehow I tought that it should renew the certificates from node 1...?
Hi,
you probably have a combination of multiple issues on your second node.
The ipa-server-upgrade failure may leave your instance in a wrong state, where dse.ldif has disabled the ports for 389-ds (see BZ https://bugzilla.redhat.com/show_bug.cgi?id=1569011 or pagure ticket https://pagure.io/freeipa/issue/7534). During the upgrade, dse.ldif is edited in order to temporarily disable the LDAP ports (to prevent ldap modifications during the upgrade). Sometimes, if the upgrade fails, dse.ldif is not restored and the ports remain disabled. You will have to stop the ldap server, manually edit dse.ldif (in /etc/dirsrv/slapd-DOMxxx) and set: nsslapd-port: 389 nsslapd-security: on
then restart the LDAP server.
For the cert renewal, your procedure is the valid one. The kerberos error is probably linked to 389-ds not being accessible.
HTH, flo
Eemeli
-----Original Message----- From: Jokinen Eemeli Sent: keskiviikko 4. heinäkuuta 2018 16.08 To: 'Rob Crittenden' rcritten@redhat.com; FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi!
I reply to this since there's some data in this message queue already related to my problem:
I had 2 ipa node cluster, where the second node had been offline for some time and at some point we had an error while trying to reboot node1 which was a Renewal Master. The issue was that some certs had expired and after a bit of special work we got the node1 back on track. I can spot three problems and I can't (again) figure out which one is the cause and which one I should repair first.
Now I got assigned the case to get the node2 back on track also. It had some certificates expired (obviously) so I did a small time jump and some of the certs were renewed. However not all of them were upgraded. "getcert list" reports 3 certs "CA Unreachable", other 3 certs seem fine.
-- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=OCSP Subsystem,O=<<REALM>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=IPA RA,O=<<REALM>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=<<ipa2.fqdn>>,O=<<REALM>> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection --
Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) but shouldn't node1 be the renewal master? Restarting httpd, certmonger and pki-tomcat don't seem to help, time traveling helped on other certs but not on these.
Directory service seems to work if I start it manually but ipa-server-upgrade fails on directory server not starting with "No ports specified" so something wrong with it or is it the certificates?
ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server
--
<<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EMERG - main - Fatal Error---No ports specified. Exiting now.
Also certmonger has issues:
dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 43, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm '<<REALM>>' --
but KDCs should be able to be resolved even from ipa node2
nslookup -type=srv _kerberos._tcp.<<REALM>> Server: <<ipa1.ip>> Address: <<ipa1.ip>>#53
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>. _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>. --
For testing purposes I turned off firewall on ipa node1
Eemeli
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/XOUL2VQ26BKQHNY2XB3CDSJRKYQCHJ3X/
On 08/17/2018 12:59 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Yes, seems like there was "security: off" but that doesn't seem to do it, I think I have ended up in the situation that I need to recreate some certificates, because:
I check the renewal dates.
-- getcert list |grep expires: expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-06-27 07:01:38 UTC expires: 2020-08-17 10:17:32 UTC expires: 2020-06-28 05:49:50 UTC --
I timejump to "before oldest expired" = 2018-03-20. Dirsrv seems to start ok. Certmonger restarts ok.
Httpd does not start. Error from /etc/httpd/logs/error_log:
-- [Tue Mar 20 07:44:39.500363 2018] [:warn] [pid 11961] NSSSessionCacheTimeout is deprecated. Ignoring. [Tue Mar 20 07:44:39.688595 2018] [:error] [pid 11961] SSL Library Error: -8181 Certificate has expired [Tue Mar 20 07:44:39.688637 2018] [:error] [pid 11961] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved. --
Seems like httpd has managed to renew some certificates at some point:
-- certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep Not Not Before: Thu Jun 28 05:49:50 2018 Not After : Sun Jun 28 05:49:50 2020 --
Should I remove httpd certificate to be able to start httpd in "before 21-03-2018? I can't seem to be able to renew these 2 (ipaCert, ocspSigningCert) without httpd because if I try to resubmit them I get "CA_Unreachable"?
You can temporarily allow httpd to start even with expired/not yet valid certificates: edit /etc/httpd/conf.d/nss.conf and set the NSSEnforceValidCerts parameter to off, then restart httpd. (Do not forget to revert the setting when you have fixed everything). See [1] for more information.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: torstai 16. elokuuta 2018 21.54 To: FreeIPA users list freeipa-users@lists.fedorahosted.org; Rob Crittenden rcritten@redhat.com Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 08/15/2018 01:20 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Anybody can help me with this one?
Summary:
2 node freeipa server cluster, node 2 was initially down for other reasons and node 1 (renewal master) had forgot to update own certificates which resulted faulty cluster. With help from mailing list we got the node 1 back online and it's working great! Now I'm trying to get node2 back to working order in cluster but it won't update the certificates even when trying the timejump. Seems like it tries to renew certificates locally although somehow I tought that it should renew the certificates from node 1...?
Hi,
you probably have a combination of multiple issues on your second node.
The ipa-server-upgrade failure may leave your instance in a wrong state, where dse.ldif has disabled the ports for 389-ds (see BZ https://bugzilla.redhat.com/show_bug.cgi?id=1569011 or pagure ticket https://pagure.io/freeipa/issue/7534). During the upgrade, dse.ldif is edited in order to temporarily disable the LDAP ports (to prevent ldap modifications during the upgrade). Sometimes, if the upgrade fails, dse.ldif is not restored and the ports remain disabled. You will have to stop the ldap server, manually edit dse.ldif (in /etc/dirsrv/slapd-DOMxxx) and set: nsslapd-port: 389 nsslapd-security: on
then restart the LDAP server.
For the cert renewal, your procedure is the valid one. The kerberos error is probably linked to 389-ds not being accessible.
HTH, flo
Eemeli
-----Original Message----- From: Jokinen Eemeli Sent: keskiviikko 4. heinäkuuta 2018 16.08 To: 'Rob Crittenden' rcritten@redhat.com; FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi!
I reply to this since there's some data in this message queue already related to my problem:
I had 2 ipa node cluster, where the second node had been offline for some time and at some point we had an error while trying to reboot node1 which was a Renewal Master. The issue was that some certs had expired and after a bit of special work we got the node1 back on track. I can spot three problems and I can't (again) figure out which one is the cause and which one I should repair first.
Now I got assigned the case to get the node2 back on track also. It had some certificates expired (obviously) so I did a small time jump and some of the certs were renewed. However not all of them were upgraded. "getcert list" reports 3 certs "CA Unreachable", other 3 certs seem fine.
-- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=OCSP Subsystem,O=<<REALM>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=IPA RA,O=<<REALM>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=<<ipa2.fqdn>>,O=<<REALM>> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection --
Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) but shouldn't node1 be the renewal master? Restarting httpd, certmonger and pki-tomcat don't seem to help, time traveling helped on other certs but not on these.
Directory service seems to work if I start it manually but ipa-server-upgrade fails on directory server not starting with "No ports specified" so something wrong with it or is it the certificates?
ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server
--
<<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EMERG - main - Fatal Error---No ports specified. Exiting now.
Also certmonger has issues:
dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 43, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm '<<REALM>>' --
but KDCs should be able to be resolved even from ipa node2
nslookup -type=srv _kerberos._tcp.<<REALM>> Server: <<ipa1.ip>> Address: <<ipa1.ip>>#53
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>. _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>. --
For testing purposes I turned off firewall on ipa node1
Eemeli
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/XOUL2VQ26BKQHNY2XB3CDSJRKYQCHJ3X/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
Hi!
Date: 20-03-2018 Services running (certmonger, dirsrv, httpd, pki-tomcatd)
-- ipa-getcert resubmit -i 20170425122557 Resubmitting "20170425122557" to "dogtag-ipa-ca-renew-agent".
getcert list |grep -A 1 20170425122557 Request ID '20170425122557': status: CA_UNREACHABLE --
Certmonger tells me:
-- Mar 20 03:26:02 <<ipa2.fqdn>> dogtag-ipa-ca-renew-agent-submit[14559]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 517, in main api.Backend.ldap2.connect() File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66, in connect conn = self.create_connection(*args, **kw) File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py", line 190, in create_connection client_controls=clientctrls) File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1111, in external_bind '', auth_tokens, server_controls, client_controls) File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1005, in error_handler error=info) NetworkError: cannot connect to 'ldapi://%2fvar%2frun%2fslapd-<<REALM>>.socket': Mar 20 03:26:02 <<ipa2.fqdn>> certmonger[14121]: 2018-03-20 03:26:02 [14121] Internal error --
Next?
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: perjantai 17. elokuuta 2018 14.43 To: FreeIPA users list freeipa-users@lists.fedorahosted.org; Rob Crittenden rcritten@redhat.com Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 08/17/2018 12:59 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Yes, seems like there was "security: off" but that doesn't seem to do it, I think I have ended up in the situation that I need to recreate some certificates, because:
I check the renewal dates.
-- getcert list |grep expires: expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-06-27 07:01:38 UTC expires: 2020-08-17 10:17:32 UTC expires: 2020-06-28 05:49:50 UTC --
I timejump to "before oldest expired" = 2018-03-20. Dirsrv seems to start ok. Certmonger restarts ok.
Httpd does not start. Error from /etc/httpd/logs/error_log:
-- [Tue Mar 20 07:44:39.500363 2018] [:warn] [pid 11961] NSSSessionCacheTimeout is deprecated. Ignoring. [Tue Mar 20 07:44:39.688595 2018] [:error] [pid 11961] SSL Library Error: -8181 Certificate has expired [Tue Mar 20 07:44:39.688637 2018] [:error] [pid 11961] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved. --
Seems like httpd has managed to renew some certificates at some point:
-- certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep Not Not Before: Thu Jun 28 05:49:50 2018 Not After : Sun Jun 28 05:49:50 2020 --
Should I remove httpd certificate to be able to start httpd in "before 21-03-2018? I can't seem to be able to renew these 2 (ipaCert, ocspSigningCert) without httpd because if I try to resubmit them I get "CA_Unreachable"?
You can temporarily allow httpd to start even with expired/not yet valid certificates: edit /etc/httpd/conf.d/nss.conf and set the NSSEnforceValidCerts parameter to off, then restart httpd. (Do not forget to revert the setting when you have fixed everything). See [1] for more information.
HTH, flo
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
Eemeli
-----Original Message----- From: Florence Blanc-Renaud [mailto:flo@redhat.com] Sent: torstai 16. elokuuta 2018 21.54 To: FreeIPA users list freeipa-users@lists.fedorahosted.org; Rob Crittenden rcritten@redhat.com Cc: Jokinen Eemeli Eemeli.Jokinen@cinia.fi Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
On 08/15/2018 01:20 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!
Anybody can help me with this one?
Summary:
2 node freeipa server cluster, node 2 was initially down for other reasons and node 1 (renewal master) had forgot to update own certificates which resulted faulty cluster. With help from mailing list we got the node 1 back online and it's working great! Now I'm trying to get node2 back to working order in cluster but it won't update the certificates even when trying the timejump. Seems like it tries to renew certificates locally although somehow I tought that it should renew the certificates from node 1...?
Hi,
you probably have a combination of multiple issues on your second node.
The ipa-server-upgrade failure may leave your instance in a wrong state, where dse.ldif has disabled the ports for 389-ds (see BZ https://bugzilla.redhat.com/show_bug.cgi?id=1569011 or pagure ticket https://pagure.io/freeipa/issue/7534). During the upgrade, dse.ldif is edited in order to temporarily disable the LDAP ports (to prevent ldap modifications during the upgrade). Sometimes, if the upgrade fails, dse.ldif is not restored and the ports remain disabled. You will have to stop the ldap server, manually edit dse.ldif (in /etc/dirsrv/slapd-DOMxxx) and set: nsslapd-port: 389 nsslapd-security: on
then restart the LDAP server.
For the cert renewal, your procedure is the valid one. The kerberos error is probably linked to 389-ds not being accessible.
HTH, flo
Eemeli
-----Original Message----- From: Jokinen Eemeli Sent: keskiviikko 4. heinäkuuta 2018 16.08 To: 'Rob Crittenden' rcritten@redhat.com; FreeIPA users list freeipa-users@lists.fedorahosted.org; Florence Blanc-Renaud flo@redhat.com Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start
Hi!
I reply to this since there's some data in this message queue already related to my problem:
I had 2 ipa node cluster, where the second node had been offline for some time and at some point we had an error while trying to reboot node1 which was a Renewal Master. The issue was that some certs had expired and after a bit of special work we got the node1 back on track. I can spot three problems and I can't (again) figure out which one is the cause and which one I should repair first.
Now I got assigned the case to get the node2 back on track also. It had some certificates expired (obviously) so I did a small time jump and some of the certs were renewed. However not all of them were upgraded. "getcert list" reports 3 certs "CA Unreachable", other 3 certs seem fine.
-- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=OCSP Subsystem,O=<<REALM>> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=IPA RA,O=<<REALM>> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=<<REALM>> subject: CN=<<ipa2.fqdn>>,O=<<REALM>> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection --
Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) but shouldn't node1 be the renewal master? Restarting httpd, certmonger and pki-tomcat don't seem to help, time traveling helped on other certs but not on these.
Directory service seems to work if I start it manually but ipa-server-upgrade fails on directory server not starting with "No ports specified" so something wrong with it or is it the certificates?
ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server
--
<<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EMERG - main - Fatal Error---No ports specified. Exiting now.
Also certmonger has issues:
dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in <module> sys.exit(main()) File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 43, in kinit_keytab cred = gssapi.Credentials(name=name, store=store, usage='initiate') File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ store=store) File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire usage) File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm '<<REALM>>' --
but KDCs should be able to be resolved even from ipa node2
nslookup -type=srv _kerberos._tcp.<<REALM>> Server: <<ipa1.ip>> Address: <<ipa1.ip>>#53
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>. _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>. --
For testing purposes I turned off firewall on ipa node1
Eemeli
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fed o rahosted.org/message/XOUL2VQ26BKQHNY2XB3CDSJRKYQCHJ3X/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo rahosted.org/message/XP7I6WTBMX2PYDBSI2OBCRGQA3HCRNWE/
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Jokinen Eemeli -
Rob Crittenden