New subject: Cert expired for pki-tomcat and process would not start

18 Jun 2019

      On 6/17/19 11:05 PM, Sayfiddin, Farhad via FreeIPA-users wrote:
...
Thanks for your response Rob, really appreciate it.
I have stopped the IPA and went back in time of Jan 7 of 2019 since Server-Cert cert-pki-ca would expire on: 2019-01-08 20:16:52 UTC
Started dirsrv, krb5kdc and pki-tomcatd@pki-tomcat.service manually.
[root@sl1mmgplidm0002 ~]# date
Mon Jan  7 20:23:50 CST 2019
[root@sl1mmgplidm0002 ~]#
[root@sl1mmgplidm0002 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: STOPPED
named Service: STOPPED
ipa_memcached Service: STOPPED
httpd Service: STOPPED
ipa-custodia Service: STOPPED
pki-tomcatd Service: STOPPED
smb Service: STOPPED
winbind Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful
[root@sl1mmgplidm0002 ~]# systemctl status pki-tomcatd@pki-tomcat.service
● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat
    Loaded: loaded (/usr/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled)
    Active: active (running) since Mon 2019-01-07 20:17:53 CST; 4min 59s ago
   Process: 58524 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS)
  Main PID: 58637 (java)
    CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
            └─58637 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tom...
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: INFO: Starting ProtocolHandler ["http-bio-8443"]
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: Jan 07, 2019 8:17:57 PM org.apache.coyote.AbstractProtocol start
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: INFO: Starting ProtocolHandler ["ajp-bio-0:0:0:0:0:0:0:1-8009"]
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: PKIListener: org.apache.catalina.core.StandardServer[after_start]
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: PKIListener: Subsystem CA is disabled.
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: PKIListener: Check /var/log/pki/pki-tomcat/ca/selftests.log for possible errors.
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: PKIListener: To enable the subsystem:
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: PKIListener:   pki-server subsystem-enable -i pki-tomcat ca
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: Jan 07, 2019 8:17:57 PM org.apache.catalina.startup.Catalina start
Jan 07 20:17:57 sl1mmgplidm0002.ipa.gen.zone server[58637]: INFO: Server startup in 2477 ms
[root@sl1mmgplidm0002 ~]#
Ran " certmonger resubmit -i 20170214143200" but cert is still showing to expires on same date, it is not forcing for it to update.
Status is changed to Monitoring now, but it is only because I went back in time.
Request ID '20170214143200':
         status: MONITORING
         stuck: no
         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-renew-agent
         issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
         subject: CN=sl1mmgplidm0002.ipa.gen.zone,O=IPA.GEN.ZONE
         expires: 2019-01-08 20:16:52 UTC
         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
         eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
         track: yes
         auto-renew: yes
I have tried to restart certmonger with no luck. Please advise.
Hi,
when you run certmonger resubmit, please have a look at the logs 
generated in the journal. When everything goes smoothly, you should be 
able to see the following steps in the journal (may be separated by 
other unrelated logs):
dogtag-ipa-ca-renew-agent-submit[20831]: Forwarding request to 
dogtag-ipa-renew-agent
dogtag-ipa-ca-renew-agent-submit[20831]: dogtag-ipa-renew-agent returned 5
The above 2 lines may appear multiple times and show that the CA helper 
is using another helper. This other command is directly contacting PKI 
and authenticates with the RA cert (the 'ipaCert' stored in 
/etc/http/alias). It is calling the profileSubmit API, then the 
profileReview API.
Then at around the same time in /var/log/pki/pki-tomcat/ca/debug, check 
if there is a line with "uri = /ca/ee/ca/profileSubmit" and another one 
with "uri = /ca/ee/ca/profileReview". This shows that the PKI server 
received a renewal request. The following lines may help diagnose any 
issue (for instance the authentication failed).
flo
...
-----Original Message-----
From: Rob Crittenden rcritten@redhat.com
Sent: Monday, June 17, 2019 2:17 PM
To: Sayfiddin, Farhad fsayfiddin@tkcholdings.com; FreeIPA users list freeipa-users@lists.fedorahosted.org
Subject: Re: [Freeipa-users] Cert expired for pki-tomcat and process would not start
Sayfiddin, Farhad wrote:
...
Here is the output of getcert list
I think if you stop IPA, go back in time to when this server cert is valid (it is the TLS cert for the CA server) and manually start dirsrv, dogtag and krb5 then run certmonger resubmit -i 20170214143200
You want to be sure ntpd (or chronyc) isn't running to force time back to now.
rob
...
[root@sl1mmgplidm0002 ~]# getcert list Number of certificates and
requests being tracked: 8.
Request ID '20170214143155':
          status: MONITORING
          stuck: no
          key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
          certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
          CA: dogtag-ipa-ca-renew-agent
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=CA Audit,O=IPA.GEN.ZONE
          expires: 2020-12-01 18:52:55 UTC
          key usage: digitalSignature,nonRepudiation
          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
          track: yes
          auto-renew: yes
Request ID '20170214143156':
          status: MONITORING
          stuck: no
          key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
          certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
          CA: dogtag-ipa-ca-renew-agent
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=OCSP Subsystem,O=IPA.GEN.ZONE
          expires: 2020-12-01 18:52:54 UTC
          eku: id-kp-OCSPSigning
          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
          track: yes
          auto-renew: yes
Request ID '20170214143157':
          status: MONITORING
          stuck: no
          key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
          certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
          CA: dogtag-ipa-ca-renew-agent
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=CA Subsystem,O=IPA.GEN.ZONE
          expires: 2020-12-01 18:53:15 UTC
          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
          eku: id-kp-serverAuth,id-kp-clientAuth
          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
          track: yes
          auto-renew: yes
Request ID '20170214143158':
          status: MONITORING
          stuck: no
          key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
          certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
          CA: dogtag-ipa-ca-renew-agent
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=Certificate Authority,O=IPA.GEN.ZONE
          expires: 2037-01-18 20:02:36 UTC
          key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
          track: yes
          auto-renew: yes
Request ID '20170214143159':
          status: MONITORING
          stuck: no
          key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
          certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
          CA: dogtag-ipa-ca-renew-agent
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=IPA RA,O=IPA.GEN.ZONE
          expires: 2020-12-01 18:52:44 UTC
          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
          eku: id-kp-serverAuth,id-kp-clientAuth
          pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
          post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
          track: yes
          auto-renew: yes
Request ID '20170214143200':
          status: CA_UNREACHABLE
          ca-error: Error 60 connecting to https://sl1mmgplidm0002.ipa.gen.zone:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
          stuck: no
          key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
          certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
          CA: dogtag-ipa-renew-agent
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=sl1mmgplidm0002.ipa.gen.zone,O=IPA.GEN.ZONE
          expires: 2019-01-08 20:16:52 UTC
          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
          eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
          track: yes
          auto-renew: yes
Request ID '20170214143201':
          status: MONITORING
          stuck: no
          key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-GEN-ZONE',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-GEN-ZONE/pwdfile.txt'
          certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-GEN-ZONE',nickname='Server-Cert',token='NSS Certificate DB'
          CA: IPA
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=sl1mmgplidm0002.ipa.gen.zone,O=IPA.GEN.ZONE
          expires: 2020-12-23 03:40:21 UTC
          principal name: ldap/sl1mmgplidm0002.ipa.gen.zone@IPA.GEN.ZONE
          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
          eku: id-kp-serverAuth,id-kp-clientAuth
          pre-save command:
          post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-GEN-ZONE
          track: yes
          auto-renew: yes
Request ID '20170214143202':
          status: MONITORING
          stuck: no
          key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
          certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
          CA: IPA
          issuer: CN=Certificate Authority,O=IPA.GEN.ZONE
          subject: CN=sl1mmgplidm0002.ipa.gen.zone,O=IPA.GEN.ZONE
          expires: 2020-12-23 03:40:31 UTC
          principal name: HTTP/sl1mmgplidm0002.ipa.gen.zone@IPA.GEN.ZONE
          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
          eku: id-kp-serverAuth,id-kp-clientAuth
          pre-save command:
          post-save command: /usr/libexec/ipa/certmonger/restart_httpd
          track: yes
          auto-renew: yes
Already tried this solution with no luck:
https://urldefense.proofpoint.com/v2/url?u=https-3A__rcritten.wordpres
s.com_2017_09_20_peer-2Dcertificate-2Dcannot-2Dbe-2Dauthenticated-2Dwi
th-2Dgiven-2Dca-2Dcertificates_&d=DwIDAw&c=YQjZbjrpZrGDVqAPwjXLR6FCrpS
yubErKtFCyGSfD8I&r=d-TYcZJsaxSN2fvTay_nSbRETC6Fq1LvfisROgToD30&m=vYnOq
UeSIamQw5SC2J9Rs9eMlJ1Jd7WemUOfBlK_wz4&s=hu2FmrcSxYTX9VEY0j-d7kejsKMn3
204Kkt_3BRIc80&e=
[root@sl1mmgplidm0002 ~]# certutil -d /etc/httpd/alias -L
Certificate Nickname                                         Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert                                                  u,u,u
ipaCert                                                      u,u,u
IPA.GEN.ZONE IPA CA                                          CT,C,C
[root@sl1mmgplidm0002 ~]# certutil -d /etc/httpd/alias -M -n 'IPA.GEN.ZONE IPA CA' -t ',,'
[root@sl1mmgplidm0002 ~]# certutil -d /etc/httpd/alias -M -n 'IPA.GEN.ZONE IPA CA' -t 'CT,C,C'
Curl command still fails
[root@sl1mmgplidm0002 ~]# SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://urldefense.proofpoint.com/v2/url?u=https-3A__-2560hostname-2560-3A84...
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to sl1mmgplidm0002.ipa.gen.zone port 8443 (#0)

Trying 172.20.0.36...
Connected to sl1mmgplidm0002.ipa.gen.zone (172.20.0.36) port 8443

(#0)

Initializing NSS with certpath: sql:/etc/httpd/alias/
CAfile: /etc/ipa/ca.crt
CApath: none
Server certificate:
  subject: CN=sl1mmgplidm0002.ipa.gen.zone,O=IPA.GEN.ZONE

  start date: Jan 18 20:16:52 2017 GMT

  expire date: Jan 08 20:16:52 2019 GMT

  common name: sl1mmgplidm0002.ipa.gen.zone

  issuer: CN=Certificate Authority,O=IPA.GEN.ZONE

NSS error -8181 (SEC_ERROR_EXPIRED_CERTIFICATE)
Peer's Certificate has expired.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Closing connection 0

curl: (60) Peer's Certificate has expired.
More details here:
https://urldefense.proofpoint.com/v2/url?u=http-3A__curl.haxx.se_docs_
sslcerts.html&d=DwIDaQ&c=YQjZbjrpZrGDVqAPwjXLR6FCrpSyubErKtFCyGSfD8I&r
=d-TYcZJsaxSN2fvTay_nSbRETC6Fq1LvfisROgToD30&m=Z8zd7LpACPgATImRFhdrk52
3IIIKpfTP44sN22Z5k5k&s=PkVO7ngwiWZqwUzfzDqJ6HiWaal9XEglmhYc4u_gkps&e=
curl performs SSL certificate verification by default, using a "bundle"
   of Certificate Authority (CA) public keys (CA certs). If the default
   bundle file isn't adequate, you can specify an alternate file
   using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
   the bundle, the certificate verification probably failed due to a
   problem with the certificate (it might be expired, or the name might
   not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
   the -k (or --insecure) option.
-----Original Message-----
From: Rob Crittenden rcritten@redhat.com
Sent: Thursday, June 13, 2019 4:08 PM
To: FreeIPA users list freeipa-users@lists.fedorahosted.org
Cc: Sayfiddin, Farhad fsayfiddin@tkcholdings.com
Subject: Re: [Freeipa-users] Cert expired for pki-tomcat and process
would not start
Sayfiddin, Farhad via FreeIPA-users wrote:
...
We have two replica servers sl1mmgplidm0001/2.
sl1mmgplidm0001 is functioning as CRL master and has no issues.
[root@sl1mmgplidm0001 ~]# ipa config-show | grep 'CA renewal master'
IPA CA renewal master: sl1mmgplidm0001
[root@sl1mmgplidm0001 ~]#
[root@sl1mmgplidm0001 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@sl1mmgplidm0001 ~]#
sl1mmgplidm0002 is having an issue where pki-tomcat process would not
start due to expired cert. It has CA_UNREACHABLE error
[root@sl1mmgplidm0002 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@sl1mmgplidm0002 ~]#
[root@sl1mmgplidm0002 ~]# getcert list | grep -A 10 20170214143200
Request ID '20170214143200':
status: CA_UNREACHABLE
ca-error: Error 60 connecting to
https://urldefense.proofpoint.com/v2/url?u=https-3A__sl1mmgplidm0002-
3
A8443_ca_agent_ca_profileReview&d=DwIDAw&c=YQjZbjrpZrGDVqAPwjXLR6FCrpSyubErKtFCyGSfD8I&r=d-TYcZJsaxSN2fvTay_nSbRETC6Fq1LvfisROgToD30&m=vYnOqUeSIamQw5SC2J9Rs9eMlJ1Jd7WemUOfBlK_wz4&s=EvNOXdLcm_vL9kIJfZltxwLVIojayf1wau_ByrzA_m0&e= : Peer certificate cannot be authenticated with given CA certificates.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-renew-agent
issuer: CN=Certificate Authority,O=IPA
subject: CN=sl1mmgplidm0002,O=IPA
expires: 2019-01-08 20:16:52 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
[root@sl1mmgplidm0002 ~]#
Tried running renew_ca_cert command and "getcert resubmit -i" with no luck.
Don't run ipa-cacert-manage renew. It renews only the root CA cert which won't help.
We need to see the full output of getcert list to see what status all the certs are in.
You might also try this:
https://urldefense.proofpoint.com/v2/url?u=https-3A__rcritten.wordpres
s.com_2017_09_20_peer-2Dcertificate-2Dcannot-2Dbe-2Dauthenticated-2Dwi
th-2Dgiven-2Dca-2Dcertificates_&d=DwIDAw&c=YQjZbjrpZrGDVqAPwjXLR6FCrpS
yubErKtFCyGSfD8I&r=d-TYcZJsaxSN2fvTay_nSbRETC6Fq1LvfisROgToD30&m=vYnOq
UeSIamQw5SC2J9Rs9eMlJ1Jd7WemUOfBlK_wz4&s=hu2FmrcSxYTX9VEY0j-d7kejsKMn3
204Kkt_3BRIc80&e=
rob

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...

Re: Cert expired for pki-tomcat and process would not start