Upgraded from CentOS 7.5 to 7.6 which includes IPA upgrade.from 4.5.4-10 to 4.6.4-10 upgrade was done via yum upgrade
Upgrade went fine. I see no alarming errors in the logs. It stopped and started all the servers did the ipa upgrade. All was fine once completed. Reboot and now pki-tomcatd CA will not start. Tomcat starts, gets all the way to were it should start the CA and doesn't. No errors, Debug doesn't show any blatant errors. It does have "Repository: Server not completely started. Returning .." which is the closest thing I see to an error.
All the certs are in monitoring state. None are expired. Domain is not quite a year old. PKI is communicating to LDAP without issues. Validated that. Also checked for and replication errors. There are none.
This is happening on all 4 systems. In the exact same way. DNS is up, we can authenticate, kerbrose is working. Can search LDAP via SSL and non-SSL Rebooted into the older kernel just to make sure. Reverted back to an old CS.cfg also, no different. I'm at a complete loss. Most other posts and pages about this all deal with expired certs. And the one that wasn't (from Redhat) was about replication conflicts. Nothing is panning out.
Fully patched CentOS Linux release 7.6.1810 (Core)
ipa-client-4.6.4-10.el7.centos.x86_64 ipa-client-common-4.6.4-10.el7.centos.noarch ipa-common-4.6.4-10.el7.centos.noarch ipa-server-4.6.4-10.el7.centos.x86_64 ipa-server-common-4.6.4-10.el7.centos.noarch ipa-server-dns-4.6.4-10.el7.centos.noarch libipa_hbac-1.16.2-13.el7.x86_64 python2-ipaclient-4.6.4-10.el7.centos.noarch python2-ipalib-4.6.4-10.el7.centos.noarch python2-ipaserver-4.6.4-10.el7.centos.noarch python-iniparse-0.4-9.el7.noarch python-libipa_hbac-1.16.2-13.el7.x86_64 sssd-ipa-1.16.2-13.el7.x86_64
krb5-pkinit-1.15.1-34.el7.x86_64 pki-base-10.5.9-6.el7.noarch pki-base-java-10.5.9-6.el7.noarch pki-ca-10.5.9-6.el7.noarch pki-kra-10.5.9-6.el7.noarch pki-server-10.5.9-6.el7.noarch pki-tools-10.5.9-6.el7.x86_64
Jason,
On Sat, 22 Dec 2018, Jason Wood via FreeIPA-users wrote:
Upgraded from CentOS 7.5 to 7.6 which includes IPA upgrade.from 4.5.4-10 to 4.6.4-10 upgrade was done via yum upgrade
Upgrade went fine. I see no alarming errors in the logs. It stopped and started all the servers did the ipa upgrade. All was fine once completed. Reboot and now pki-tomcatd CA will not start. Tomcat starts, gets all the way to were it should start the CA and doesn't. No errors, Debug doesn't show any blatant errors. It does have "Repository: Server not completely started. Returning .." which is the closest thing I see to an error. [...]
i have no idea what went wrong exactly on your system, but you may want to have a look at [1]. I had similar trouble recently and exchanging the invalid certificate by hand (ldapmodify) fixed it. At least the article helps to get an idea what is wrong.
[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
Mit freundlichen Gruessen/With best regards,
--Daniel.
Already went through that page several times, All checks passed. All certs are good. none are expired. The cert in NSS is the same in LDAP. No errors communicating/logging in.
It is the lack of errors that is the most troubling.
A little more information.
pki-tomcatd is starting. ports 8080, 8443 and 8009 are open and responding. gssproxy is up and working Still no errors in any logs. PKI is able to make SSL connections to LDAP, the certificates are all valid and it is using the correct certificates.
In the tomcat logs this is where it sits Dec 26 11:06:51 IPA.SERVER server[17191]: CMSEngine.initializePasswordStore() begins Dec 26 11:06:51 IPA.SERVER server[17191]: CMSEngine.initializePasswordStore(): tag=internaldb Dec 26 11:06:51 IPA.SERVER server[17191]: CMSEngine.initializePasswordStore(): tag=replicationdb
I've left it there over the weekend, with no change. Port 8005 (shutdown port) never opens. Tomcat never deploys the ca.jar Selinux is disabled, along with the firewall. Still think this is environment related as the issue only started after the reboot from patching. IPA upgrade did complete successfully on all 4 systems after update. Booting older kernel has no effect. I did have Oracle JRE installed along with OpenJDK. removed Oracle, same thing.
Once again, there are no errors in any logs. So I have very little to go on. Any help will be appreciated.
I know what is not the issue. No certs are expired expires: 2020-01-13 13:27:04 UTC expires: 2020-01-02 13:25:21 UTC expires: 2020-01-02 13:25:20 UTC expires: 2020-01-02 13:25:20 UTC expires: 2038-01-12 13:25:20 UTC expires: 2020-01-02 13:25:38 UTC expires: 2020-01-02 13:25:20 UTC expires: 2020-01-13 13:26:31 UTC expires: 2020-01-13 13:26:54 UTC
trusts are good Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u
and Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
SERVER IPA CA CT,C,C Server-Cert u,u,u
I have moved system time back to a day prior to failure, restarted certmonger, then tried to start everything again. No go.
Removed custom ldap Default group objectclasses Reverted CS.cfg to an archive Re-installed all the PKI rpms Checked file permissions Changed from SSL + Cert auth in CS.cfg to basic auth - spoiler, no difference.
I do not want to rebuild the domain. Its looking more and more like I may have to. I may try the recover/replace CA steps I found in my searches.
I recently performed this on my servers. what does “ipa —version” show ? after the yum update, did you run “ipa-server-upgrade” ?
- grant This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.
This is on all 4 systems having the issue ipa --version VERSION: 4.6.4, API_VERSION: 2.229
When system was updated ipa-server-upgrade was ran, and it did complete successful 2018-12-19T23:34:26Z INFO The ipa-server-upgrade command was successful
Running the command fails now, as the CA won't start, which is expected.
Jason,
Could you please attach the latest PKI debug log from /var/log/pki/pki-tomcat/ca/ - everything from the beginning of startup to where it hangs?
Thanks, Fraser
On Sat, Dec 29, 2018 at 11:07:07PM -0000, Jason Wood via FreeIPA-users wrote:
This is on all 4 systems having the issue ipa --version VERSION: 4.6.4, API_VERSION: 2.229
When system was updated ipa-server-upgrade was ran, and it did complete successful 2018-12-19T23:34:26Z INFO The ipa-server-upgrade command was successful
Running the command fails now, as the CA won't start, which is expected. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Attached is the logs
On Tue, Jan 1, 2019 at 7:36 PM Fraser Tweedale ftweedal@redhat.com wrote:
Jason,
Could you please attach the latest PKI debug log from /var/log/pki/pki-tomcat/ca/ - everything from the beginning of startup to where it hangs?
Thanks, Fraser
On Sat, Dec 29, 2018 at 11:07:07PM -0000, Jason Wood via FreeIPA-users wrote:
This is on all 4 systems having the issue ipa --version VERSION: 4.6.4, API_VERSION: 2.229
When system was updated ipa-server-upgrade was ran, and it did complete successful 2018-12-19T23:34:26Z INFO The ipa-server-upgrade command was successful
Running the command fails now, as the CA won't start, which is expected. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Was wondering if anyone had a chance to look through the logs posted for anything useful?
Jason,
I've seen this caused by a replication conflict that prevents the CA certificate profiles from loading completely during start-up. Can you double-check that there are really no conflicts in ou=certificateprofiles,ou=ca,o=ipaca, just to confirm?
Try this:
1. Check for replication conflicts in ou=certificateprofiles,ou=ca,o=ipaca:
# ldapsearch -xLLL -D "cn=Directory Manager" -W -b ou=certificateprofiles,ou=ca,o=ipaca '(|(objectclass=*)(&(objectclass=ldapsubentry)(nsds5ReplConflict=*))'
2. If any objects are found, remove them and then try starting IPA:
# ldapdelete -x -D "cn=Directory Manager" -W <dn of replication conflict object>
running your LDAP search returns invalid search, missing a ) on the end, (I think). Adding ) to the end returns a lot of data but nothing with nsds5ReplConflict. this is the end statement
# search result search: 2 result: 0 Success
# numResponses: 105 # numEntries: 104
Running this, search for conflicts, from the Redhat guides ldapsearch -x -D "cn=directory manager" -W -b "dc=IPA,dc=EXAMPLE,dc=net" "nsds5ReplConflict=*" * nsds5ReplConflict # search result search: 2 result: 0 Success
# numResponses: 1
Doesn't look like there are conflicts, it is one of the first things I suspected myself.
On Fri, Jan 11, 2019 at 2:01 PM Marco Rhodes via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
Jason,
I've seen this caused by a replication conflict that prevents the CA certificate profiles from loading completely during start-up. Can you double-check that there are really no conflicts in ou=certificateprofiles,ou=ca,o=ipaca, just to confirm?
Try this:
Check for replication conflicts in ou=certificateprofiles,ou=ca,o=ipaca:
# ldapsearch -xLLL -D "cn=Directory Manager" -W -b ou=certificateprofiles,ou=ca,o=ipaca '(|(objectclass=*)(&(objectclass=ldapsubentry)(nsds5ReplConflict=*))'
If any objects are found, remove them and then try starting IPA:
# ldapdelete -x -D "cn=Directory Manager" -W <dn of replication conflict object>
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Jason,
Yes, bad search filter there - apologies.
This one is better:
# ldapsearch -xLLL -D "cn=Directory Manager" -W -b ou=certificateprofiles,ou=ca,o=ipaca '(&(nsds5ReplConflict=*)(objectclass=ldapsubentry))'
The base DN you want to specify is 'ou=certificateprofiles,ou=ca,o=ipaca'. This is the area from which the profile monitor thread obtains data.
Jason,
Yes, bad search filter there - apologies.
This one is better:
# ldapsearch -xLLL -D "cn=Directory Manager" -W -b ou=certificateprofiles,ou=ca,o=ipaca '(&(nsds5ReplConflict=*)(objectclass=ldapsubentry))'
The base DN you want to specify is 'ou=certificateprofiles,ou=ca,o=ipaca'. This is the area from which the profile monitor thread obtains data.
I totally missed your reply, sorry. Here's the output of that ldapsearch, I do not see nsds5ReplConflict in the output. Does this output mean all of these are conflicts?
dn: cn=caCMCECserverCert+nsuniqueid=191ae601-03e811e9-a000b184-ffa545f3,ou=cer tificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..c2UK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caCMCECserverCert classId: caEnrollImpl
dn: cn=caCMCECsubsystemCert+nsuniqueid=191ae602-03e811e9-a000b184-ffa545f3,ou= certificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..Cg== objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caCMCECsubsystemCert classId: caEnrollImpl
dn: cn=ECAdminCert+nsuniqueid=191ae603-03e811e9-a000b184-ffa545f3,ou=certifica teProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..dWUK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: ECAdminCert classId: caEnrollImpl
dn: cn=caECServerCert+nsuniqueid=191ae604-03e811e9-a000b184-ffa545f3,ou=certif icateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..dWUK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECServerCert classId: caEnrollImpl
dn: cn=caECSubsystemCert+nsuniqueid=191ae605-03e811e9-a000b184-ffa545f3,ou=cer tificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..ZQo= objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECSubsystemCert classId: caEnrollImpl
dn: cn=caECDirPinUserCert+nsuniqueid=191ae606-03e811e9-a000b184-ffa545f3,ou=ce rtificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..dWUK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECDirPinUserCert classId: caEnrollImpl
dn: cn=caECAgentServerCert+nsuniqueid=191ae607-03e811e9-a000b184-ffa545f3,ou=c ertificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..dWUK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECAgentServerCert classId: caEnrollImpl
dn: cn=caCMCECUserCert+nsuniqueid=191ae608-03e811e9-a000b184-ffa545f3,ou=certi ficateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..ZQo= objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caCMCECUserCert classId: caEnrollImpl
dn: cn=caECFullCMCUserCert+nsuniqueid=191ae609-03e811e9-a000b184-ffa545f3,ou=c ertificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..Cg== objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECFullCMCUserCert classId: caEnrollImpl
dn: cn=caECFullCMCUserSignedCert+nsuniqueid=191ae60a-03e811e9-a000b184-ffa545f 3,ou=certificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..c2UK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECFullCMCUserSignedCert classId: caEnrollImpl
dn: cn=caECFullCMCSelfSignedCert+nsuniqueid=191ae60b-03e811e9-a000b184-ffa545f 3,ou=certificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..c2UK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECFullCMCSelfSignedCert classId: caEnrollImpl
dn: cn=caECSimpleCMCUserCert+nsuniqueid=191ae60c-03e811e9-a000b184-ffa545f3,ou =certificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..c2UK objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECSimpleCMCUserCert classId: caEnrollImpl
dn: cn=caECAdminCert+nsuniqueid=191ae60d-03e811e9-a000b184-ffa545f3,ou=certifi cateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..ZQo= objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECAdminCert classId: caEnrollImpl
dn: cn=caECInternalAuthServerCert+nsuniqueid=191ae60e-03e811e9-a000b184-ffa545 f3,ou=certificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..Cg== objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECInternalAuthServerCert classId: caEnrollImpl
dn: cn=caECInternalAuthSubsystemCert+nsuniqueid=191ae60f-03e811e9-a000b184-ffa 545f3,ou=certificateProfiles,ou=ca,o=ipaca certProfileConfig:: YXV0..Cg== objectClass: top objectClass: certProfile objectClass: ldapsubentry cn: caECInternalAuthSubsystemCert classId: caEnrollImpl
Jason -
Yes, all of these entries with +nsuniqueid=... embedded in the DN are conflict entries.
Try removing each with:
# ldapdelete -D "cn=directory manager" -W <DN of conflict entry>
Once they are all removed, try restarting IPA.
This worked! Deleted all the conflicts on one system, stopped pki-tomcatd and started it again. Came up. Worked on 3 of the 4 systems. The last one still has conflicts that I will clear out, and is of course the master.
Very much appreciative with helping with this issue.
Jason Wood
On 2/5/2019 3:10 PM, Marco Rhodes via FreeIPA-users wrote:
Jason -
Yes, all of these entries with +nsuniqueid=... embedded in the DN are conflict entries.
Try removing each with:
# ldapdelete -D "cn=directory manager" -W <DN of conflict entry>
Once they are all removed, try restarting IPA. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On Wed, Feb 06, 2019 at 10:09:00AM -0500, Jason L Wood via FreeIPA-users wrote:
This worked! Deleted all the conflicts on one system, stopped pki-tomcatd and started it again. Came up. Worked on 3 of the 4 systems. The last one still has conflicts that I will clear out, and is of course the master.
There is a ticket for this issue: https://pagure.io/dogtagpki/issue/3078
I have a work-in-progress patch for the issue but it needs a bit more work and testing.
Cheers, Fraser
Very much appreciative with helping with this issue.
Jason Wood
On 2/5/2019 3:10 PM, Marco Rhodes via FreeIPA-users wrote:
Jason -
Yes, all of these entries with +nsuniqueid=... embedded in the DN are conflict entries.
Try removing each with:
# ldapdelete -D "cn=directory manager" -W <DN of conflict entry>
Once they are all removed, try restarting IPA. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
I recently performed this on my servers. what does “ipa —version” show ? after the yum update, did you run “ipa-server-upgrade” ?
- grant This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.
So I have an expired cert somewhere. Or something really weird. Setting system time to 10/01/2018 PKI-Tomcat starts. Restarted certmonger and dirsrv. Moved date to 11/01/2018, restarted certmonger, dirsrv and pki-tomcat. pki-tomcat started. Moved date to 12/01/2018, restarted services, tomcat fails to start. This is on a replica, not the master. Hmmmm?
OK. Moved date to 10/17/2018, restarted services. Starting with certmonger. Did work. did certlist and there were some certs on that systems that were different start/end dates. I thought, if pki-tomcatd started then I should make this my master, which I did. tomcat may start, but it gives back 500 errors. Nothing I could do would get it to work. Moved the time back to current and right back to the same.
The master, didn't work no matter what I did. I keep coming back, that I have ZERO errors. Nothing. No where. Since the OS update and the ipa-server-update ran without issue during the OS update, and the issue only happened after the system was rebooted, then some configuration some where was changed to the point that it effects this. What that is, I do not know. I have not found any added or removed settings in any system files comparing it to a system that upgraded fine. Maybe something in the LDAP db that I'm not seeing. I am very LDAP challenged, so there is a good possibility I missed something.
At this point I've spent way too much time on this without resolution. Are there any docs/howto's on replacing a domain. I do not want to lose the config or anything. Just basically spin up new servers, copy the data, and connect the clients to it all on the same domain. I would rather fix whatever broke, but that isn't looking like it will happen.
Jason Wood via FreeIPA-users wrote:
So I have an expired cert somewhere. Or something really weird. Setting system time to 10/01/2018 PKI-Tomcat starts. Restarted certmonger and dirsrv. Moved date to 11/01/2018, restarted certmonger, dirsrv and pki-tomcat. pki-tomcat started. Moved date to 12/01/2018, restarted services, tomcat fails to start. This is on a replica, not the master. Hmmmm?
OK. Moved date to 10/17/2018, restarted services. Starting with certmonger. Did work. did certlist and there were some certs on that systems that were different start/end dates. I thought, if pki-tomcatd started then I should make this my master, which I did. tomcat may start, but it gives back 500 errors. Nothing I could do would get it to work. Moved the time back to current and right back to the same.
The master, didn't work no matter what I did. I keep coming back, that I have ZERO errors. Nothing. No where. Since the OS update and the ipa-server-update ran without issue during the OS update, and the issue only happened after the system was rebooted, then some configuration some where was changed to the point that it effects this. What that is, I do not know. I have not found any added or removed settings in any system files comparing it to a system that upgraded fine. Maybe something in the LDAP db that I'm not seeing. I am very LDAP challenged, so there is a good possibility I missed something.
At this point I've spent way too much time on this without resolution. Are there any docs/howto's on replacing a domain. I do not want to lose the config or anything. Just basically spin up new servers, copy the data, and connect the clients to it all on the same domain. I would rather fix whatever broke, but that isn't looking like it will happen.
The debug log looks cut off. Right after that IIRC it should do the selftest. There should be a selftest log in the same directory as debug, what does that say?
You might try using certutil to look at each cert in /etc/pki/pki-tomcat/alias/ to double-check its validity.
rob
Log isn't cut off, that is where it stops. It never reaches selftests.
From the debug logs, it shows that it is connecting with the cert and there are no errors there. This leads me to believe the certs are fine, certmonger is doing what it is supposed to, I think the issue could be in the LDAP db. We did disable anonymous binds. I've changed that to allow rootdse, that did not effect anything.
Also replaced the server.xml and CA.cfg with backups, same outcome.
All the certs look good, valid dates, don't expire for another year
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Validity: Not Before: Fri Jan 12 13:25:20 2018 Not After : Tue Jan 12 13:25:20 2038 Subject: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: c0:...:5d Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: f1:...:2f
Name: Certificate Basic Constraints Critical: True Data: Is a CA with no maximum path length.
Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Certificate Signing CRL Signing
Name: Certificate Subject Key ID Data: f1:...:2f
Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.IPA.EXAMPLE.NET/ca/ocsp"
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: a8:...:4e Fingerprint (SHA-256): 6E:...:32 Fingerprint (SHA1): 98:...:0F
Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: Valid CA Trusted CA User Trusted Client CA Email Flags: Valid CA Trusted CA User Object Signing Flags: Valid CA Trusted CA User
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'ocspSigningCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 2 (0x2) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Validity: Not Before: Fri Jan 12 13:25:20 2018 Not After : Thu Jan 02 13:25:20 2020 Subject: "CN=OCSP Subsystem,O=IPA.EXAMPLE.NET" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: 9f:...:5b Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: f1:...:2f
Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Certificate Signing CRL Signing
Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.IPA.EXAMPLE.NET/ca/ocsp"
Name: Extended Key Usage OCSP Responder Certificate
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: 2f:...:32 Fingerprint (SHA-256): 85:...:3C Fingerprint (SHA1): 73:...:9F
Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: User Email Flags: User Object Signing Flags: User
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'subsystemCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 4 (0x4) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=IPA.EXAMPLE.NET" Validity: Not Before: Fri Jan 12 13:25:20 2018 Not After : Thu Jan 02 13:25:20 2020 Subject: "CN=CA Subsystem,O=IPA.EXAMPLE.NET" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: cc:...:89 Exponent: 65537 (0x10001) Signed Extensions: Name: Certificate Authority Key Identifier Key ID: f1:...:2f
Name: Authority Information Access Method: PKIX Online Certificate Status Protocol Location: URI: "http://ipa-ca.IPA.EXAMPLE.NET/ca/ocsp"
Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Key Encipherment Data Encipherment
Name: Extended Key Usage TLS Web Server Authentication Certificate TLS Web Client Authentication Certificate
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature: 50:...:49 Fingerprint (SHA-256): CB:...:E5 Fingerprint (SHA1): 7A:...:CB
Mozilla-CA-Policy: false (attribute missing) Certificate Trust Flags: SSL Flags: User Email Flags: User Object Signing Flags: User
certutil -L -d /etc/pki/pki-tomcat/alias/
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u
On Thu, Jan 10, 2019 at 6:00 PM Rob Crittenden rcritten@redhat.com wrote:
Jason Wood via FreeIPA-users wrote:
So I have an expired cert somewhere. Or something really weird. Setting system time to 10/01/2018 PKI-Tomcat starts. Restarted certmonger and dirsrv. Moved date to 11/01/2018, restarted certmonger, dirsrv and pki-tomcat. pki-tomcat started. Moved date to 12/01/2018, restarted services, tomcat fails to start. This is on a replica, not the master. Hmmmm?
OK. Moved date to 10/17/2018, restarted services. Starting with certmonger. Did work. did certlist and there were some certs on that systems that were different start/end dates. I thought, if pki-tomcatd started then I should make this my master, which I did. tomcat may start, but it gives back 500 errors. Nothing I could do would get it to work. Moved the time back to current and right back to the same.
The master, didn't work no matter what I did. I keep coming back, that I have ZERO errors. Nothing. No where. Since the OS update and the ipa-server-update ran without issue during the OS update, and the issue only happened after the system was rebooted, then some configuration some where was changed to the point that it effects this. What that is, I do not know. I have not found any added or removed settings in any system files comparing it to a system that upgraded fine. Maybe something in the LDAP db that I'm not seeing. I am very LDAP challenged, so there is a good possibility I missed something.
At this point I've spent way too much time on this without resolution. Are there any docs/howto's on replacing a domain. I do not want to lose the config or anything. Just basically spin up new servers, copy the data, and connect the clients to it all on the same domain. I would rather fix whatever broke, but that isn't looking like it will happen.
The debug log looks cut off. Right after that IIRC it should do the selftest. There should be a selftest log in the same directory as debug, what does that say?
You might try using certutil to look at each cert in /etc/pki/pki-tomcat/alias/ to double-check its validity.
rob
do you have any news on this issue... i havea similar trouble...after yum update from centos 7.4 to 7.6 the pki-tomcatd services failed to start...it worked for a while then failed and now it won't start anymore... looked at the certificate and all seems ok... after a quick look at the different log file, here's what i found:
In Ldap (bound) connection pool to host ipa port 636, Cannot connect to LDAP server. Error: netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
also, when i run ipa-server-upgrade, it hang waiting for CA to come up ...
The CA status is: check interrupted due to error: cannot connect to 'http://cgi-mgmt01d.mgmt.internal:8080/ca/admin/ca/getStatus': timed out
something broke with the update... and i can't put my finger on it...
unfortunately no I have not. I did get the socket error at one point trying to troubleshoot the issue. Was caused by something I did, so wasn't related.
I have no usable errors anywhere. I do know it is not getting far enough in the sequence to do self tests. The upgrade did work when the packages were installed, the failure started only after a reboot. I'm hoping its a simple fix, if we can ever discover the cause.
Jason W
On 1/28/2019 10:19 PM, Patrice Gamache via FreeIPA-users wrote:
do you have any news on this issue... i havea similar trouble...after yum update from centos 7.4 to 7.6 the pki-tomcatd services failed to start...it worked for a while then failed and now it won't start anymore... looked at the certificate and all seems ok... after a quick look at the different log file, here's what i found:
In Ldap (bound) connection pool to host ipa port 636, Cannot connect to LDAP server. Error: netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
also, when i run ipa-server-upgrade, it hang waiting for CA to come up ...
The CA status is: check interrupted due to error: cannot connect to 'http://cgi-mgmt01d.mgmt.internal:8080/ca/admin/ca/getStatus': timed out
something broke with the update... and i can't put my finger on it... _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
dbischof＠hrz.uni-kassel.de -
Fraser Tweedale -
Grant Janssen -
Jason L Wood -
Jason Wood
-
Marco Rhodes -
Patrice Gamache -
Rob Crittenden