I am having problems with the server that currently is my main CA and was considering trying to switch that function to a different server. I have tried some of the stuff I found online but the CA role can't be enabled on another server because it is broken on the one that has it right now. Hence the operation fails. Any other ideas on how to resolve this? It is OK if I have to abandon my old certificates and generate entirely new one on the new CA server.
On 10/26/2017 04:58 PM, Kristian Petersen via FreeIPA-users wrote:
I am having problems with the server that currently is my main CA and was considering trying to switch that function to a different server. I have tried some of the stuff I found online but the CA role can't be enabled on another server because it is broken on the one that has it right now. Hence the operation fails. Any other ideas on how to resolve this? It is OK if I have to abandon my old certificates and generate entirely new one on the new CA server.
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Hi,
which issues do you currently have with the CA? Maybe we can help fix the CA first.
Flo
When I recently updated one of my IPA servers (it reports 4.5.0-21.el7_4.1.2 in yum), the result was that it could not start back up because pki-tomcatd kept failing. I was able to get it running for now by ignoring the failure of that one service, but I haven't been able to to determine the cause. The logs are pretty quiet on this one. They show the failure itself, but not information that helps me fix the problem. It also appears to be causing some weird UI issues. Without the certificate stuff working I can't add any new replicas as CAs because it can't send the needed info to the new server.
I have talked a little bit with Rob Crittenden about this but always run into an impasse hen trying to find the debug logs.
On Thu, Oct 26, 2017 at 10:25 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 10/26/2017 04:58 PM, Kristian Petersen via FreeIPA-users wrote:
I am having problems with the server that currently is my main CA and was considering trying to switch that function to a different server. I have tried some of the stuff I found online but the CA role can't be enabled on another server because it is broken on the one that has it right now. Hence the operation fails. Any other ideas on how to resolve this? It is OK if I have to abandon my old certificates and generate entirely new one on the new CA server.
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
Hi,
which issues do you currently have with the CA? Maybe we can help fix the CA first.
Flo
Kristian Petersen via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
When I recently updated one of my IPA servers (it reports 4.5.0-21.el7_4.1.2 in yum), the result was that it could not start back up because pki-tomcatd kept failing. I was able to get it running for now by ignoring the failure of that one service, but I haven't been able to to determine the cause. The logs are pretty quiet on this one. They show the failure itself, but not information that helps me fix the problem.
Can you show the relevant logs? Is there something in the dirsrv logs at that time? CA logs aren't easy to read, but should give at least a hint where to look further.
Jochen
The dirsrv log just shows a bunch of the following: [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object)
That makes sense though since pki-tomcat won't start. Rob was asking what was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path doesn't exist on any of my IPA servers. He said that would normally be the first place to look. Hence, I am looking for other solutions.
On Thu, Oct 26, 2017 at 12:37 PM, Jochen Hein jochen@jochen.org wrote:
Kristian Petersen via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
When I recently updated one of my IPA servers (it reports 4.5.0-21.el7_4.1.2 in yum), the result was that it could not start back
up
because pki-tomcatd kept failing. I was able to get it running for now
by
ignoring the failure of that one service, but I haven't been able to to determine the cause. The logs are pretty quiet on this one. They show
the
failure itself, but not information that helps me fix the problem.
Can you show the relevant logs? Is there something in the dirsrv logs at that time? CA logs aren't easy to read, but should give at least a hint where to look further.
Jochen
-- This space is intentionally left blank.
Kristian Petersen via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
The dirsrv log just shows a bunch of the following: [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object)
That makes sense though since pki-tomcat won't start. Rob was asking what was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path doesn't exist on any of my IPA servers. He said that would normally be the first place to look. Hence, I am looking for other solutions.
Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help.
Jochen
I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat:
from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein jochen@jochen.org wrote:
Kristian Petersen via FreeIPA-users freeipa-users@lists.fedorahosted.org writes:
The dirsrv log just shows a bunch of the following: [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object)
That makes sense though since pki-tomcat won't start. Rob was asking
what
was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that
path
doesn't exist on any of my IPA servers. He said that would normally be
the
first place to look. Hence, I am looking for other solutions.
Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help.
Jochen
-- This space is intentionally left blank.
On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote:
I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat:
from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
Hi,
just a wild guess, but we saw issues during update related either to certificates or IPv6. - Is IPv6 enabled on your server? The server doesn't need an IPv6 address but IPv6 should not be disabled. - If selinux is in enforcing mode, there were known issues during certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent.
Flo
[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org mailto:jochen@jochen.org> wrote:
Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions. Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help. Jochen -- This space is intentionally left blank.-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
So it looks like things aren't associated properly anymore. Not sure what my next steps would be though.
On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote:
I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat:
from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
Hi,
just a wild guess, but we saw issues during update related either to certificates or IPv6.
- Is IPv6 enabled on your server? The server doesn't need an IPv6 address
but IPv6 should not be disabled.
- If selinux is in enforcing mode, there were known issues during
certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent.
Flo
[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting- freeipa-pki-tomcatd-fails-to-start/
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org mailto: jochen@jochen.org> wrote:
Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind -Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions.
Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help. Jochen -- This space is intentionally left blank.-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
I also found that the certs don't match! LDAP and certutil return different certs when you query them. The blog post didn't suggest a method for fixing this and I don't want to make the problem worse by doing it the wrong way. Suggestions?
On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen nesretep@chem.byu.edu wrote:
I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
So it looks like things aren't associated properly anymore. Not sure what my next steps would be though.
On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote:
I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat:
from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
Hi,
just a wild guess, but we saw issues during update related either to certificates or IPv6.
- Is IPv6 enabled on your server? The server doesn't need an IPv6 address
but IPv6 should not be disabled.
- If selinux is in enforcing mode, there were known issues during
certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent.
Flo
[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-fr eeipa-pki-tomcatd-fails-to-start/
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org mailto: jochen@jochen.org> wrote:
Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind -Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions.
Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help. Jochen -- This space is intentionally left blank.-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
I forgot to include the results of the commands in case it is helpful:
-bash-4.2$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDdTCCAl2gAwIBAgIBBDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxD
SEVNLkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNTEwMTMyMDUwM
jhaFw0xNzEwMDIyMDUwMjhaMC4xFTATBgNVBAoMDENIRU0uQllVLkVEVTEVMBMGA1UEAwwMQ0EgU3
Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+
z8z82YinNVC9YzOejrRqRHST4ZiJIq2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0
sBM9Aplx/wBu0IIyA4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HX
Npiocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ETitHkbCCxn+
8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o+NuoAYaWi/hjpgq0ZXa2zM8
zIy33h+A+UQIDAQABo4GUMIGRMB8GA1UdIwQYMBaAFB0PNWo+emloojFyMjHrItpaAfVCMD8GCCsG
AQUFBwEBBDMwMTAvBggrBgEFBQcwAYYjaHR0cDovL2lwYTEuY2hlbS5ieXUuZWR1OjgwL2NhL29jc
3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w
0BAQsFAAOCAQEAnsZeWq5e0UWJwaJqTiJdm+1jvQJrzOPWRYPfu9MTpfFjyhlNEwMX0azVzTrFbn2
7+JjQpcxH60zNurhjfavdx3S+/Dmz0dZPgX6AKBeZMfKyyfLeXaoCz3AW9uIbiQZZFdQloGGB82Ek
M78W6rJVxb5x9Juck4D4GaeqOuHgNPYVnpNkWR4shCnbGdGjrG4kQRO4I91DxYBrKnY8Fmucxq2y1
4Xi29RT9Plx6p4g4E+LjqdZVAPlK/x3IQDxL2Shp/ycQxGEjfmPX8t3gbyi9e4QvHv5EdmrGpHlIQ
bicsPmJ3gmDLn+EcIyoxpT7BLmJKPrn0FjF+FTyE/OrzHBkg== description: 2;4;CN=Certificate Authority,O=CHEM.BYU.EDU;CN=CA Subsystem,O=CHE M.BYU.EDU seeAlso: CN=CA Subsystem,O=CHEM.BYU.EDU
-bash-4.2$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIBMDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxDSEVN LkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzA5 MDQyMDUwNThaFw0xOTA4MjUyMDUwNThaMC4xFTATBgNVBAoMDENIRU0uQllVLkVE VTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+z8z82YinNVC9YzOejrRqRHST4ZiJI q2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0sBM9Aplx/wBu0IIy A4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HXNpi ocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ET itHkbCCxn+8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o +NuoAYaWi/hjpgq0ZXa2zM8zIy33h+A+UQIDAQABo4GTMIGQMB8GA1UdIwQYMBaA FB0PNWo+emloojFyMjHrItpaAfVCMD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcw AYYiaHR0cDovL2lwYS1jYS5jaGVtLmJ5dS5lZHUvY2Evb2NzcDAOBgNVHQ8BAf8E BAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEB CwUAA4IBAQC3eGtIqHewdEtW7EagaUGkc4LoCulmhhmTC7lxOYYT+ADBrve6RSOA UpXSNCoetQU0QmXQkEXDtaZpjYFV2DaniwoAB6HuyG7do/BYdJoX8vKP/vCoJJCJ V64BuCE/uipYclGXbKZPkElbfASIAiNa6X+pSvhIqdTHS0dE7DpHK+m7sIlb1AO0 yVmCZBIh1OT/sKajOaLA7epksAA1c9M0BSkdgjrIxAKaeHTtadnLPDEGVQor357Z yPyQ+vSM6GNI/Z02z+paX7WxuI/uZRHzD2MoprmUCfv03isv66EUu0EVox3wSEBT zXGp0EVo/JHfrENJKzszJ4qWGhXJfyII -----END CERTIFICATE----- -bash-4.2$
On Fri, Oct 27, 2017 at 5:08 PM, Kristian Petersen nesretep@chem.byu.edu wrote:
I also found that the certs don't match! LDAP and certutil return different certs when you query them. The blog post didn't suggest a method for fixing this and I don't want to make the problem worse by doing it the wrong way. Suggestions?
On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen nesretep@chem.byu.edu wrote:
I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
So it looks like things aren't associated properly anymore. Not sure what my next steps would be though.
On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote:
I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat:
from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600]
- ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager
cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
Hi,
just a wild guess, but we saw issues during update related either to certificates or IPv6.
- Is IPv6 enabled on your server? The server doesn't need an IPv6
address but IPv6 should not be disabled.
- If selinux is in enforcing mode, there were known issues during
certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent.
Flo
[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-fr eeipa-pki-tomcatd-fails-to-start/
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org mailto:jochen@jochen.org> wrote:
Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind -Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions.
Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help. Jochen -- This space is intentionally left blank.-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
On 10/28/2017 01:15 AM, Kristian Petersen via FreeIPA-users wrote:
I forgot to include the results of the commands in case it is helpful:
-bash-4.2$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDdTCCAl2gAwIBAgIBBDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxD SEVNLkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNTEwMTMyMDUwM
jhaFw0xNzEwMDIyMDUwMjhaMC4xFTATBgNVBAoMDENIRU0uQllVLkVEVTEVMBMGA1UEAwwMQ0EgU3
Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+
z8z82YinNVC9YzOejrRqRHST4ZiJIq2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0
sBM9Aplx/wBu0IIyA4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HX
Npiocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ETitHkbCCxn+
8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o+NuoAYaWi/hjpgq0ZXa2zM8
zIy33h+A+UQIDAQABo4GUMIGRMB8GA1UdIwQYMBaAFB0PNWo+emloojFyMjHrItpaAfVCMD8GCCsG
AQUFBwEBBDMwMTAvBggrBgEFBQcwAYYjaHR0cDovL2lwYTEuY2hlbS5ieXUuZWR1OjgwL2NhL29jc
3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w
0BAQsFAAOCAQEAnsZeWq5e0UWJwaJqTiJdm+1jvQJrzOPWRYPfu9MTpfFjyhlNEwMX0azVzTrFbn2
7+JjQpcxH60zNurhjfavdx3S+/Dmz0dZPgX6AKBeZMfKyyfLeXaoCz3AW9uIbiQZZFdQloGGB82Ek
M78W6rJVxb5x9Juck4D4GaeqOuHgNPYVnpNkWR4shCnbGdGjrG4kQRO4I91DxYBrKnY8Fmucxq2y1
4Xi29RT9Plx6p4g4E+LjqdZVAPlK/x3IQDxL2Shp/ycQxGEjfmPX8t3gbyi9e4QvHv5EdmrGpHlIQ
bicsPmJ3gmDLn+EcIyoxpT7BLmJKPrn0FjF+FTyE/OrzHBkg== description: 2;4;CN=Certificate Authority,O=CHEM.BYU.EDU http://CHEM.BYU.EDU;CN=CA Subsystem,O=CHE M.BYU.EDU http://M.BYU.EDU seeAlso: CN=CA Subsystem,O=CHEM.BYU.EDU http://CHEM.BYU.EDU
-bash-4.2$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIBMDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxDSEVN LkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzA5 MDQyMDUwNThaFw0xOTA4MjUyMDUwNThaMC4xFTATBgNVBAoMDENIRU0uQllVLkVE VTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+z8z82YinNVC9YzOejrRqRHST4ZiJI q2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0sBM9Aplx/wBu0IIy A4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HXNpi ocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ET itHkbCCxn+8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o +NuoAYaWi/hjpgq0ZXa2zM8zIy33h+A+UQIDAQABo4GTMIGQMB8GA1UdIwQYMBaA FB0PNWo+emloojFyMjHrItpaAfVCMD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcw AYYiaHR0cDovL2lwYS1jYS5jaGVtLmJ5dS5lZHUvY2Evb2NzcDAOBgNVHQ8BAf8E BAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEB CwUAA4IBAQC3eGtIqHewdEtW7EagaUGkc4LoCulmhhmTC7lxOYYT+ADBrve6RSOA UpXSNCoetQU0QmXQkEXDtaZpjYFV2DaniwoAB6HuyG7do/BYdJoX8vKP/vCoJJCJ V64BuCE/uipYclGXbKZPkElbfASIAiNa6X+pSvhIqdTHS0dE7DpHK+m7sIlb1AO0 yVmCZBIh1OT/sKajOaLA7epksAA1c9M0BSkdgjrIxAKaeHTtadnLPDEGVQor357Z yPyQ+vSM6GNI/Z02z+paX7WxuI/uZRHzD2MoprmUCfv03isv66EUu0EVox3wSEBT zXGp0EVo/JHfrENJKzszJ4qWGhXJfyII -----END CERTIFICATE----- -bash-4.2$
Hi,
so it looks like the certificate 'subsystemCert cert-pki-ca' has been renewed, stored in /etc/pki/pki-tomcat/alias but not copied into the LDAP server.
The most recent version is the one in /etc/pki/pki-tomcat/alias (we can see that by comparing the serial numbers) and needs to be put into the LDAP entry. You can perform this using ldapmodify tool or a graphical LDAP browser.
With ldapmodify: 1/ extract the certificate from /etc/pki/pki-tomcat/alias into a single line, without the -----BEGIN CERTIFICATE---- and -----END CERTIFICATE----- delimiters: $ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a | tail -n +2 | head -n -1 | tr -d '\r\n' MIIDdDCC...WGhXJfyII
(tail -n +2 removes the -----BEGIN CERTIFICATE----- and head -n -1 removes the -----END CERTIFICATE-----, while tr -d '\r\n' deletes new line and return characters).
2/ Find the certificate serial number sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 48 (0x30)
2/ perform ldapmodify with the value obtained above: ldapmodify -x -D 'cn=directory manager' -W dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: usercertificate usercertificate:: <PASTE output from above step 1 here> - replace: description description: 2;48;CN=Certificate Authority,O=CHEM.BYU.EDU,;CN=CA Subsystem,O=CHEM.BYU.EDU
(do not forget to type return twice to send the modify command). In my example, the description field contains "48" as it is the serial number of the new subsystemCert cert-pki-ca obtained in the step 2.
After that, you should be able to restart pki-tomcatd. Please tell me if you still encounter issues,
Flo.
On Fri, Oct 27, 2017 at 5:08 PM, Kristian Petersen <nesretep@chem.byu.edu mailto:nesretep@chem.byu.edu> wrote:
I also found that the certs don't match! LDAP and certutil return different certs when you query them. The blog post didn't suggest a method for fixing this and I don't want to make the problem worse by doing it the wrong way. Suggestions? On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>> wrote: I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca' RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. So it looks like things aren't associated properly anymore. Not sure what my next steps would be though. On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote: I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat: from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY) Hi, just a wild guess, but we saw issues during update related either to certificates or IPv6. - Is IPv6 enabled on your server? The server doesn't need an IPv6 address but IPv6 should not be disabled. - If selinux is in enforcing mode, there were known issues during certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent. Flo [1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ <https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/> On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>>> wrote: Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions. Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help. Jochen -- This space is intentionally left blank. -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
OK I think I got the ldapmodify to work. I reran the commands to check the two certs and they appear to match now. However, when I run an ipactl restart the system still fails on pki-tomcatd.
On Mon, Oct 30, 2017 at 3:42 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 10/28/2017 01:15 AM, Kristian Petersen via FreeIPA-users wrote:
I forgot to include the results of the commands in case it is helpful:
-bash-4.2$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDdTCCAl2gAwIBAgIBBDANBgkqhk iG9w0BAQsFADA3MRUwEwYDVQQKDAxD SEVNLkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNTEwMTMyMDUwM
jhaFw0xNzEwMDIyMDUwMjhaMC4xFTATBgNVBAoMDENIRU0uQllVLkVEVTEVMBMGA1UEAwwMQ0EgU3
Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+
z8z82YinNVC9YzOejrRqRHST4ZiJIq2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0
sBM9Aplx/wBu0IIyA4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HX
Npiocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ETitHkbCCxn+
8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o+NuoAYaWi/hjpgq0ZXa2zM8
zIy33h+A+UQIDAQABo4GUMIGRMB8GA1UdIwQYMBaAFB0PNWo+emloojFyMjHrItpaAfVCMD8GCCsG
AQUFBwEBBDMwMTAvBggrBgEFBQcwAYYjaHR0cDovL2lwYTEuY2hlbS5ieXUuZWR1OjgwL2NhL29jc
3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w
0BAQsFAAOCAQEAnsZeWq5e0UWJwaJqTiJdm+1jvQJrzOPWRYPfu9MTpfFjyhlNEwMX0azVzTrFbn2
7+JjQpcxH60zNurhjfavdx3S+/Dmz0dZPgX6AKBeZMfKyyfLeXaoCz3AW9uIbiQZZFdQloGGB82Ek
M78W6rJVxb5x9Juck4D4GaeqOuHgNPYVnpNkWR4shCnbGdGjrG4kQRO4I91DxYBrKnY8Fmucxq2y1
4Xi29RT9Plx6p4g4E+LjqdZVAPlK/x3IQDxL2Shp/ycQxGEjfmPX8t3gbyi9e4QvHv5EdmrGpHlIQ
bicsPmJ3gmDLn+EcIyoxpT7BLmJKPrn0FjF+FTyE/OrzHBkg== description: 2;4;CN=Certificate Authority,O=CHEM.BYU.EDU < http://CHEM.BYU.EDU%3E;CN=CA Subsystem,O=CHE M.BYU.EDU http://M.BYU.EDU seeAlso: CN=CA Subsystem,O=CHEM.BYU.EDU http://CHEM.BYU.EDU
-bash-4.2$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIBMDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxDSEVN LkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzA5 MDQyMDUwNThaFw0xOTA4MjUyMDUwNThaMC4xFTATBgNVBAoMDENIRU0uQllVLkVE VTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+z8z82YinNVC9YzOejrRqRHST4ZiJI q2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0sBM9Aplx/wBu0IIy A4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HXNpi ocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ET itHkbCCxn+8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o +NuoAYaWi/hjpgq0ZXa2zM8zIy33h+A+UQIDAQABo4GTMIGQMB8GA1UdIwQYMBaA FB0PNWo+emloojFyMjHrItpaAfVCMD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcw AYYiaHR0cDovL2lwYS1jYS5jaGVtLmJ5dS5lZHUvY2Evb2NzcDAOBgNVHQ8BAf8E BAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEB CwUAA4IBAQC3eGtIqHewdEtW7EagaUGkc4LoCulmhhmTC7lxOYYT+ADBrve6RSOA UpXSNCoetQU0QmXQkEXDtaZpjYFV2DaniwoAB6HuyG7do/BYdJoX8vKP/vCoJJCJ V64BuCE/uipYclGXbKZPkElbfASIAiNa6X+pSvhIqdTHS0dE7DpHK+m7sIlb1AO0 yVmCZBIh1OT/sKajOaLA7epksAA1c9M0BSkdgjrIxAKaeHTtadnLPDEGVQor357Z yPyQ+vSM6GNI/Z02z+paX7WxuI/uZRHzD2MoprmUCfv03isv66EUu0EVox3wSEBT zXGp0EVo/JHfrENJKzszJ4qWGhXJfyII -----END CERTIFICATE----- -bash-4.2$
Hi,
so it looks like the certificate 'subsystemCert cert-pki-ca' has been renewed, stored in /etc/pki/pki-tomcat/alias but not copied into the LDAP server.
The most recent version is the one in /etc/pki/pki-tomcat/alias (we can see that by comparing the serial numbers) and needs to be put into the LDAP entry. You can perform this using ldapmodify tool or a graphical LDAP browser.
With ldapmodify: 1/ extract the certificate from /etc/pki/pki-tomcat/alias into a single line, without the -----BEGIN CERTIFICATE---- and -----END CERTIFICATE----- delimiters: $ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a | tail -n +2 | head -n -1 | tr -d '\r\n' MIIDdDCC...WGhXJfyII
(tail -n +2 removes the -----BEGIN CERTIFICATE----- and head -n -1 removes the -----END CERTIFICATE-----, while tr -d '\r\n' deletes new line and return characters).
2/ Find the certificate serial number sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 48 (0x30)
2/ perform ldapmodify with the value obtained above: ldapmodify -x -D 'cn=directory manager' -W dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: usercertificate usercertificate:: <PASTE output from above step 1 here>
replace: description description: 2;48;CN=Certificate Authority,O=CHEM.BYU.EDU,;CN=CA Subsystem,O=CHEM.BYU.EDU
(do not forget to type return twice to send the modify command). In my example, the description field contains "48" as it is the serial number of the new subsystemCert cert-pki-ca obtained in the step 2.
After that, you should be able to restart pki-tomcatd. Please tell me if you still encounter issues,
Flo.
On Fri, Oct 27, 2017 at 5:08 PM, Kristian Petersen <nesretep@chem.byu.edu
mailto:nesretep@chem.byu.edu> wrote:
I also found that the certs don't match! LDAP and certutil return different certs when you query them. The blog post didn't suggest a method for fixing this and I don't want to make the problem worse by doing it the wrong way. Suggestions? On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>> wrote: I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca' RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. So it looks like things aren't associated properly anymore. Not sure what my next steps would be though. On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote: I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat: from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
Hi, just a wild guess, but we saw issues during update related either to certificates or IPv6. - Is IPv6 enabled on your server? The server doesn't need an IPv6 address but IPv6 should not be disabled. - If selinux is in enforcing mode, there were known issues during certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent. Flo [1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ https://floblanc.wordpress.com/2017/09/11/troubleshooting- freeipa-pki-tomcatd-fails-to-start/
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>>> wrote: Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions. Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help. Jochen -- This space is intentionally left blank. -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
On 10/30/2017 05:23 PM, Kristian Petersen via FreeIPA-users wrote:
OK I think I got the ldapmodify to work. I reran the commands to check the two certs and they appear to match now. However, when I run an ipactl restart the system still fails on pki-tomcatd.
Hi, In this case I think that the next item to investigate is why the key cannot be listed using sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
In a previous mail, you wrote that the output of this command was certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
This tells that 1/ the password is OK (otherwise certutil would display an error message) 2/ the key for 'subsystemCert cert-pki-ca' is missing from the nssdb.
Do you have a backup of the NSS DB /etc/pki/pki-tomcat/alias or was the CA installed on another master, so that we can get the private key? Can you also list which keys are present in /etc/pki/pki-tomcat/alias with sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
Flo
On Mon, Oct 30, 2017 at 3:42 AM, Florence Blanc-Renaud <flo@redhat.com mailto:flo@redhat.com> wrote:
On 10/28/2017 01:15 AM, Kristian Petersen via FreeIPA-users wrote: I forgot to include the results of the commands in case it is helpful: -bash-4.2$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDdTCCAl2gAwIBAgIBBDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxD SEVNLkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNTEwMTMyMDUwM jhaFw0xNzEwMDIyMDUwMjhaMC4xFTATBgNVBAoMDENIRU0uQllVLkVEVTEVMBMGA1UEAwwMQ0EgU3 Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+ z8z82YinNVC9YzOejrRqRHST4ZiJIq2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0 sBM9Aplx/wBu0IIyA4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HX Npiocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ETitHkbCCxn+ 8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o+NuoAYaWi/hjpgq0ZXa2zM8 zIy33h+A+UQIDAQABo4GUMIGRMB8GA1UdIwQYMBaAFB0PNWo+emloojFyMjHrItpaAfVCMD8GCCsG AQUFBwEBBDMwMTAvBggrBgEFBQcwAYYjaHR0cDovL2lwYTEuY2hlbS5ieXUuZWR1OjgwL2NhL29jc 3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w 0BAQsFAAOCAQEAnsZeWq5e0UWJwaJqTiJdm+1jvQJrzOPWRYPfu9MTpfFjyhlNEwMX0azVzTrFbn2 7+JjQpcxH60zNurhjfavdx3S+/Dmz0dZPgX6AKBeZMfKyyfLeXaoCz3AW9uIbiQZZFdQloGGB82Ek M78W6rJVxb5x9Juck4D4GaeqOuHgNPYVnpNkWR4shCnbGdGjrG4kQRO4I91DxYBrKnY8Fmucxq2y1 4Xi29RT9Plx6p4g4E+LjqdZVAPlK/x3IQDxL2Shp/ycQxGEjfmPX8t3gbyi9e4QvHv5EdmrGpHlIQ bicsPmJ3gmDLn+EcIyoxpT7BLmJKPrn0FjF+FTyE/OrzHBkg== description: 2;4;CN=Certificate Authority,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> <http://CHEM.BYU.EDU>;CN=CA Subsystem,O=CHE M.BYU.EDU <http://M.BYU.EDU> <http://M.BYU.EDU> seeAlso: CN=CA Subsystem,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> <http://CHEM.BYU.EDU> -bash-4.2$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIBMDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxDSEVN LkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzA5 MDQyMDUwNThaFw0xOTA4MjUyMDUwNThaMC4xFTATBgNVBAoMDENIRU0uQllVLkVE VTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+z8z82YinNVC9YzOejrRqRHST4ZiJI q2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0sBM9Aplx/wBu0IIy A4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HXNpi ocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ET itHkbCCxn+8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o +NuoAYaWi/hjpgq0ZXa2zM8zIy33h+A+UQIDAQABo4GTMIGQMB8GA1UdIwQYMBaA FB0PNWo+emloojFyMjHrItpaAfVCMD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcw AYYiaHR0cDovL2lwYS1jYS5jaGVtLmJ5dS5lZHUvY2Evb2NzcDAOBgNVHQ8BAf8E BAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEB CwUAA4IBAQC3eGtIqHewdEtW7EagaUGkc4LoCulmhhmTC7lxOYYT+ADBrve6RSOA UpXSNCoetQU0QmXQkEXDtaZpjYFV2DaniwoAB6HuyG7do/BYdJoX8vKP/vCoJJCJ V64BuCE/uipYclGXbKZPkElbfASIAiNa6X+pSvhIqdTHS0dE7DpHK+m7sIlb1AO0 yVmCZBIh1OT/sKajOaLA7epksAA1c9M0BSkdgjrIxAKaeHTtadnLPDEGVQor357Z yPyQ+vSM6GNI/Z02z+paX7WxuI/uZRHzD2MoprmUCfv03isv66EUu0EVox3wSEBT zXGp0EVo/JHfrENJKzszJ4qWGhXJfyII -----END CERTIFICATE----- -bash-4.2$ Hi, so it looks like the certificate 'subsystemCert cert-pki-ca' has been renewed, stored in /etc/pki/pki-tomcat/alias but not copied into the LDAP server. The most recent version is the one in /etc/pki/pki-tomcat/alias (we can see that by comparing the serial numbers) and needs to be put into the LDAP entry. You can perform this using ldapmodify tool or a graphical LDAP browser. With ldapmodify: 1/ extract the certificate from /etc/pki/pki-tomcat/alias into a single line, without the -----BEGIN CERTIFICATE---- and -----END CERTIFICATE----- delimiters: $ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a | tail -n +2 | head -n -1 | tr -d '\r\n' MIIDdDCC...WGhXJfyII (tail -n +2 removes the -----BEGIN CERTIFICATE----- and head -n -1 removes the -----END CERTIFICATE-----, while tr -d '\r\n' deletes new line and return characters). 2/ Find the certificate serial number sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 48 (0x30) 2/ perform ldapmodify with the value obtained above: ldapmodify -x -D 'cn=directory manager' -W dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: usercertificate usercertificate:: <PASTE output from above step 1 here> - replace: description description: 2;48;CN=Certificate Authority,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU>,;CN=CA Subsystem,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> (do not forget to type return twice to send the modify command). In my example, the description field contains "48" as it is the serial number of the new subsystemCert cert-pki-ca obtained in the step 2. After that, you should be able to restart pki-tomcatd. Please tell me if you still encounter issues, Flo. On Fri, Oct 27, 2017 at 5:08 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu> <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>> wrote: I also found that the certs don't match! LDAP and certutil return different certs when you query them. The blog post didn't suggest a method for fixing this and I don't want to make the problem worse by doing it the wrong way. Suggestions? On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu> <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>> wrote: I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca' RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. So it looks like things aren't associated properly anymore. Not sure what my next steps would be though. On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com> <mailto:flo@redhat.com <mailto:flo@redhat.com>>> wrote: On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote: I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki or pki-tomcat: from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY) Hi, just a wild guess, but we saw issues during update related either to certificates or IPv6. - Is IPv6 enabled on your server? The server doesn't need an IPv6 address but IPv6 should not be disabled. - If selinux is in enforcing mode, there were known issues during certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent. Flo [1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ <https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/> <https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ <https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/>> On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>> <mailto:jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>>>> wrote: Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions. Brute force: reproduce the error and run "find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help. Jochen -- This space is intentionally left blank. -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Unfortunately, this machine is the only CA. I tried making one of my replicas a CA but because the pki-tomcat stuff was broken, of course that didn't work. Super bad, I know. Here is the result of that last command: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa 53ace5456cb0c07b79d061b7aada366063799089 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa e9ff606015f8c6a032ee88c51459e1952ba7f901 (orphan) < 2> rsa f40b78512366c34f88fa2da4900978a778048d4a NSS Certificate DB:ocspSigningCert cert-pki-ca < 3> rsa 8caa824ccc68966582b02dbc14aa422c3d08dee6 NSS Certificate DB:Server-Cert cert-pki-ca < 4> rsa 6410804f149a562865b616fa3054640b45305ea2 caSigningCert cert-pki-ca < 5> rsa 13cd3399d4c0734796fee85eca65a2ee05281146 NSS Certificate DB:auditSigningCert cert-pki-ca
On Tue, Oct 31, 2017 at 2:57 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 10/30/2017 05:23 PM, Kristian Petersen via FreeIPA-users wrote:
OK I think I got the ldapmodify to work. I reran the commands to check the two certs and they appear to match now. However, when I run an ipactl restart the system still fails on pki-tomcatd.
Hi,
In this case I think that the next item to investigate is why the key cannot be listed using sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
In a previous mail, you wrote that the output of this command was certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
This tells that 1/ the password is OK (otherwise certutil would display an error message) 2/ the key for 'subsystemCert cert-pki-ca' is missing from the nssdb.
Do you have a backup of the NSS DB /etc/pki/pki-tomcat/alias or was the CA installed on another master, so that we can get the private key? Can you also list which keys are present in /etc/pki/pki-tomcat/alias with sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
Flo
On Mon, Oct 30, 2017 at 3:42 AM, Florence Blanc-Renaud <flo@redhat.com
mailto:flo@redhat.com> wrote:
On 10/28/2017 01:15 AM, Kristian Petersen via FreeIPA-users wrote: I forgot to include the results of the commands in case it is helpful: -bash-4.2$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate descriptionseeAlso Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDdTCCAl2gAwIBAgIBBDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxD SEVNLkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAe Fw0xNTEwMTMyMDUwM
jhaFw0xNzEwMDIyMDUwMjhaMC4xFTATBgNVBAoMDENIRU0uQllVLkVEVTEVMBMGA1UEAwwMQ0EgU3
Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+
z8z82YinNVC9YzOejrRqRHST4ZiJIq2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0
sBM9Aplx/wBu0IIyA4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HX
Npiocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ETitHkbCCxn+
8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o+NuoAYaWi/hjpgq0ZXa2zM8
zIy33h+A+UQIDAQABo4GUMIGRMB8GA1UdIwQYMBaAFB0PNWo+emloojFyMjHrItpaAfVCMD8GCCsG
AQUFBwEBBDMwMTAvBggrBgEFBQcwAYYjaHR0cDovL2lwYTEuY2hlbS5ieXUuZWR1OjgwL2NhL29jc
3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w
0BAQsFAAOCAQEAnsZeWq5e0UWJwaJqTiJdm+1jvQJrzOPWRYPfu9MTpfFjyhlNEwMX0azVzTrFbn2
7+JjQpcxH60zNurhjfavdx3S+/Dmz0dZPgX6AKBeZMfKyyfLeXaoCz3AW9uIbiQZZFdQloGGB82Ek
M78W6rJVxb5x9Juck4D4GaeqOuHgNPYVnpNkWR4shCnbGdGjrG4kQRO4I91DxYBrKnY8Fmucxq2y1
4Xi29RT9Plx6p4g4E+LjqdZVAPlK/x3IQDxL2Shp/ycQxGEjfmPX8t3gbyi9e4QvHv5EdmrGpHlIQ
bicsPmJ3gmDLn+EcIyoxpT7BLmJKPrn0FjF+FTyE/OrzHBkg== description: 2;4;CN=Certificate Authority,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> <http://CHEM.BYU.EDU>;CN=CA Subsystem,O=CHE M.BYU.EDU <http://M.BYU.EDU> <http://M.BYU.EDU> seeAlso: CN=CA Subsystem,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> <http://CHEM.BYU.EDU> -bash-4.2$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIBMDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxDSEVN LkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzA5 MDQyMDUwNThaFw0xOTA4MjUyMDUwNThaMC4xFTATBgNVBAoMDENIRU0uQllVLkVE VTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+z8z82YinNVC9YzOejrRqRHST4ZiJI q2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0sBM9Aplx/wBu0IIy A4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HXNpi ocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ET itHkbCCxn+8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o +NuoAYaWi/hjpgq0ZXa2zM8zIy33h+A+UQIDAQABo4GTMIGQMB8GA1UdIwQYMBaA FB0PNWo+emloojFyMjHrItpaAfVCMD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcw AYYiaHR0cDovL2lwYS1jYS5jaGVtLmJ5dS5lZHUvY2Evb2NzcDAOBgNVHQ8BAf8E BAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEB CwUAA4IBAQC3eGtIqHewdEtW7EagaUGkc4LoCulmhhmTC7lxOYYT+ADBrve6RSOA UpXSNCoetQU0QmXQkEXDtaZpjYFV2DaniwoAB6HuyG7do/BYdJoX8vKP/vCoJJCJ V64BuCE/uipYclGXbKZPkElbfASIAiNa6X+pSvhIqdTHS0dE7DpHK+m7sIlb1AO0 yVmCZBIh1OT/sKajOaLA7epksAA1c9M0BSkdgjrIxAKaeHTtadnLPDEGVQor357Z yPyQ+vSM6GNI/Z02z+paX7WxuI/uZRHzD2MoprmUCfv03isv66EUu0EVox3wSEBT zXGp0EVo/JHfrENJKzszJ4qWGhXJfyII -----END CERTIFICATE----- -bash-4.2$ Hi, so it looks like the certificate 'subsystemCert cert-pki-ca' has been renewed, stored in /etc/pki/pki-tomcat/alias but not copied into the LDAP server. The most recent version is the one in /etc/pki/pki-tomcat/alias (we can see that by comparing the serial numbers) and needs to be put into the LDAP entry. You can perform this using ldapmodify tool or a graphical LDAP browser. With ldapmodify: 1/ extract the certificate from /etc/pki/pki-tomcat/alias into a single line, without the -----BEGIN CERTIFICATE---- and -----END CERTIFICATE----- delimiters: $ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a | tail -n +2 | head -n -1 | tr -d '\r\n' MIIDdDCC...WGhXJfyII (tail -n +2 removes the -----BEGIN CERTIFICATE----- and head -n -1 removes the -----END CERTIFICATE-----, while tr -d '\r\n' deletes new line and return characters). 2/ Find the certificate serial number sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 48 (0x30) 2/ perform ldapmodify with the value obtained above: ldapmodify -x -D 'cn=directory manager' -W dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: usercertificate usercertificate:: <PASTE output from above step 1 here> - replace: description description: 2;48;CN=Certificate Authority,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU>,;CN=CA Subsystem,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> (do not forget to type return twice to send the modify command). In my example, the description field contains "48" as it is the serial number of the new subsystemCert cert-pki-ca obtained in the step 2. After that, you should be able to restart pki-tomcatd. Please tell me if you still encounter issues, Flo. On Fri, Oct 27, 2017 at 5:08 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu> <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>> wrote: I also found that the certs don't match! LDAP and certutil return different certs when you query them. The blog post didn't suggest a method for fixing this and I don't want to make the problem worse by doing it the wrong way. Suggestions? On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu> <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>> wrote: I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca' RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys:SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
So it looks like things aren't associated properly anymore. Not sure what my next steps would be though. On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com> <mailto:flo@redhat.com <mailto:flo@redhat.com>>> wrote: On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote: I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki orpki-tomcat:
from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
Hi, just a wild guess, but we saw issues during update related either to certificates or IPv6. - Is IPv6 enabled on your server? The server doesn't need an IPv6 address but IPv6 should not be disabled. - If selinux is in enforcing mode, there were known issues during certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and must be consistent. Flo [1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ https://floblanc.wordpress.com/2017/09/11/troubleshooting- freeipa-pki-tomcatd-fails-to-start/ <https://floblanc.wordpress.co m/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ https://floblanc.wordpress.com/2017/09/11/troubleshooting- freeipa-pki-tomcatd-fails-to-start/>
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>> <mailto:jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>>>> wrote: Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> <mailto:freeipa-users@lists.fedorahosted.org mailto:freeipa-users@lists.fedorahosted.org
<mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions. Brute force: reproduce the error and run"find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help.
Jochen -- This space is intentionally left blank. -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> <mailto:freeipa-users-leave@lists.fedorahosted.org mailto:freeipa-users-leave@lists.fedorahosted.org>
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
I did some checking of some of the same stuff on my other IPA server (ipa2; not a CA), and in LDAP it still has the old certificate that we replaced on ipa1. Could the mismatch between these two servers be what is causing pki-tomcat to fail? journalctl shows an error that seems to indicate some kind of communication error between the two LDAPs when trying to replicate. I'm not sure though if that is a symptom of the problem we are trying to fix or part of the cause.
On Tue, Oct 31, 2017 at 8:06 AM, Kristian Petersen nesretep@chem.byu.edu wrote:
Unfortunately, this machine is the only CA. I tried making one of my replicas a CA but because the pki-tomcat stuff was broken, of course that didn't work. Super bad, I know. Here is the result of that last command: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa 53ace5456cb0c07b79d061b7aada366063799089 NSS Certificate DB:subsystemCert cert-pki-ca < 1> rsa e9ff606015f8c6a032ee88c51459e1952ba7f901 (orphan) < 2> rsa f40b78512366c34f88fa2da4900978a778048d4a NSS Certificate DB:ocspSigningCert cert-pki-ca < 3> rsa 8caa824ccc68966582b02dbc14aa422c3d08dee6 NSS Certificate DB:Server-Cert cert-pki-ca < 4> rsa 6410804f149a562865b616fa3054640b45305ea2 caSigningCert cert-pki-ca < 5> rsa 13cd3399d4c0734796fee85eca65a2ee05281146 NSS Certificate DB:auditSigningCert cert-pki-ca
On Tue, Oct 31, 2017 at 2:57 AM, Florence Blanc-Renaud flo@redhat.com wrote:
On 10/30/2017 05:23 PM, Kristian Petersen via FreeIPA-users wrote:
OK I think I got the ldapmodify to work. I reran the commands to check the two certs and they appear to match now. However, when I run an ipactl restart the system still fails on pki-tomcatd.
Hi,
In this case I think that the next item to investigate is why the key cannot be listed using sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
In a previous mail, you wrote that the output of this command was certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
This tells that 1/ the password is OK (otherwise certutil would display an error message) 2/ the key for 'subsystemCert cert-pki-ca' is missing from the nssdb.
Do you have a backup of the NSS DB /etc/pki/pki-tomcat/alias or was the CA installed on another master, so that we can get the private key? Can you also list which keys are present in /etc/pki/pki-tomcat/alias with sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
Flo
On Mon, Oct 30, 2017 at 3:42 AM, Florence Blanc-Renaud <flo@redhat.com
mailto:flo@redhat.com> wrote:
On 10/28/2017 01:15 AM, Kristian Petersen via FreeIPA-users wrote: I forgot to include the results of the commands in case it is helpful: -bash-4.2$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate descriptionseeAlso Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDdTCCAl2gAwIBAgIBBDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxD SEVNLkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAe Fw0xNTEwMTMyMDUwM
jhaFw0xNzEwMDIyMDUwMjhaMC4xFTATBgNVBAoMDENIRU0uQllVLkVEVTEVMBMGA1UEAwwMQ0EgU3
Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+
z8z82YinNVC9YzOejrRqRHST4ZiJIq2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0
sBM9Aplx/wBu0IIyA4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HX
Npiocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ETitHkbCCxn+
8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o+NuoAYaWi/hjpgq0ZXa2zM8
zIy33h+A+UQIDAQABo4GUMIGRMB8GA1UdIwQYMBaAFB0PNWo+emloojFyMjHrItpaAfVCMD8GCCsG
AQUFBwEBBDMwMTAvBggrBgEFBQcwAYYjaHR0cDovL2lwYTEuY2hlbS5ieXUuZWR1OjgwL2NhL29jc
3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w
0BAQsFAAOCAQEAnsZeWq5e0UWJwaJqTiJdm+1jvQJrzOPWRYPfu9MTpfFjyhlNEwMX0azVzTrFbn2
7+JjQpcxH60zNurhjfavdx3S+/Dmz0dZPgX6AKBeZMfKyyfLeXaoCz3AW9uIbiQZZFdQloGGB82Ek
M78W6rJVxb5x9Juck4D4GaeqOuHgNPYVnpNkWR4shCnbGdGjrG4kQRO4I91DxYBrKnY8Fmucxq2y1
4Xi29RT9Plx6p4g4E+LjqdZVAPlK/x3IQDxL2Shp/ycQxGEjfmPX8t3gbyi9e4QvHv5EdmrGpHlIQ
bicsPmJ3gmDLn+EcIyoxpT7BLmJKPrn0FjF+FTyE/OrzHBkg== description: 2;4;CN=Certificate Authority,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> <http://CHEM.BYU.EDU>;CN=CASubsystem,O=CHE M.BYU.EDU http://M.BYU.EDU http://M.BYU.EDU
seeAlso: CN=CA Subsystem,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> <http://CHEM.BYU.EDU> -bash-4.2$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDdDCCAlygAwIBAgIBMDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxDSEVN LkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzA5 MDQyMDUwNThaFw0xOTA4MjUyMDUwNThaMC4xFTATBgNVBAoMDENIRU0uQllVLkVE VTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+z8z82YinNVC9YzOejrRqRHST4ZiJI q2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0sBM9Aplx/wBu0IIy A4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HXNpi ocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ET itHkbCCxn+8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o +NuoAYaWi/hjpgq0ZXa2zM8zIy33h+A+UQIDAQABo4GTMIGQMB8GA1UdIwQYMBaA FB0PNWo+emloojFyMjHrItpaAfVCMD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcw AYYiaHR0cDovL2lwYS1jYS5jaGVtLmJ5dS5lZHUvY2Evb2NzcDAOBgNVHQ8BAf8E BAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEB CwUAA4IBAQC3eGtIqHewdEtW7EagaUGkc4LoCulmhhmTC7lxOYYT+ADBrve6RSOA UpXSNCoetQU0QmXQkEXDtaZpjYFV2DaniwoAB6HuyG7do/BYdJoX8vKP/vCoJJCJ V64BuCE/uipYclGXbKZPkElbfASIAiNa6X+pSvhIqdTHS0dE7DpHK+m7sIlb1AO0 yVmCZBIh1OT/sKajOaLA7epksAA1c9M0BSkdgjrIxAKaeHTtadnLPDEGVQor357Z yPyQ+vSM6GNI/Z02z+paX7WxuI/uZRHzD2MoprmUCfv03isv66EUu0EVox3wSEBT zXGp0EVo/JHfrENJKzszJ4qWGhXJfyII -----END CERTIFICATE----- -bash-4.2$ Hi, so it looks like the certificate 'subsystemCert cert-pki-ca' has been renewed, stored in /etc/pki/pki-tomcat/alias but not copied into the LDAP server. The most recent version is the one in /etc/pki/pki-tomcat/alias (we can see that by comparing the serial numbers) and needs to be put into the LDAP entry. You can perform this using ldapmodify tool or a graphical LDAP browser. With ldapmodify: 1/ extract the certificate from /etc/pki/pki-tomcat/alias into a single line, without the -----BEGIN CERTIFICATE---- and -----END CERTIFICATE----- delimiters: $ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a | tail -n +2 | head -n -1 | tr -d '\r\n' MIIDdDCC...WGhXJfyII (tail -n +2 removes the -----BEGIN CERTIFICATE----- and head -n -1 removes the -----END CERTIFICATE-----, while tr -d '\r\n' deletes new line and return characters). 2/ Find the certificate serial number sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 48 (0x30) 2/ perform ldapmodify with the value obtained above: ldapmodify -x -D 'cn=directory manager' -W dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: usercertificate usercertificate:: <PASTE output from above step 1 here> - replace: description description: 2;48;CN=Certificate Authority,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU>,;CN=CA Subsystem,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU> (do not forget to type return twice to send the modify command). In my example, the description field contains "48" as it is the serial number of the new subsystemCert cert-pki-ca obtained in the step 2. After that, you should be able to restart pki-tomcatd. Please tell me if you still encounter issues, Flo. On Fri, Oct 27, 2017 at 5:08 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu> <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>> wrote: I also found that the certs don't match! LDAP and certutil return different certs when you query them. The blog post didn't suggest a method for fixing this and I don't want to make the problem worse by doing it the wrong way. Suggestions? On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu> <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>> wrote: I followed some of the steps outlined in the blog post you liked to and when I got to the part where make sure that the private key can be read using the password found in /var/lib/pki/pki-tomcat/conf/password.conf using: sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca' RESULT: certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys:SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
So it looks like things aren't associated properly anymore. Not sure what my next steps would be though. On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com> <mailto:flo@redhat.com <mailto:flo@redhat.com>>> wrote: On 10/27/2017 12:55 AM, Kristian Petersen via FreeIPA-users wrote: I checked the logs that turned up after running the find command suggested by Jochen and only a couple of them turned up anything that mention pki orpki-tomcat:
from /var/log/audit/audit.log: type=SERVICE_START msg=audit(1508873851.623:163448): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' from /var/log/messages: Oct 26 16:01:58 ipa1 ns-slapd: [26/Oct/2017:16:01:58.077129423 -0600] - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication Manager cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client 192.168.105.11#37937: request has invalid signature: TSIG DHCP_UPDATER: tsig verify failure (BADKEY)
Hi, just a wild guess, but we saw issues during update related either to certificates or IPv6. - Is IPv6 enabled on your server? The server doesn't need an IPv6 address but IPv6 should not be disabled. - If selinux is in enforcing mode, there were known issues during certificate renewals that could lead to pki-tomcat not able to start any more. You can refer to this blog post [1] to check that the certificate 'subsystemCert cert-pki-ca' is properly associated to the user uid=pkidbuser,ou=people,o=ipaca. The certificate is stored in multiple places (ldap server, nss dbs) and mustbe consistent.
Flo [1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ https://floblanc.wordpress.com/2017/09/11/troubleshooting-f reeipa-pki-tomcatd-fails-to-start/ <https://floblanc.wordpress.co m/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ https://floblanc.wordpress.com/2017/09/11/troubleshooting-f reeipa-pki-tomcatd-fails-to-start/>
On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein <jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>> <mailto:jochen@jochen.org <mailto:jochen@jochen.org> <mailto:jochen@jochen.org <mailto:jochen@jochen.org>>>> wrote: Kristian Petersen via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> <mailto:freeipa-users@lists.fedorahosted.org mailto:freeipa-users@lists.fedorahosted.org
<mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>>> writes: > The dirsrv log just shows a bunch of the following: > [13/Oct/2017:14:32:07.132312021 -0600] - ERR - slapi_ldap_bind - Error: > could not bind id [cn=Replication Manager cloneAgreement1-ipa > 2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config] authentication mechanism > [SIMPLE]: error 32 (No such object) > > That makes sense though since pki-tomcat won't start. Rob was asking what > was in the logs located at /var/log/pki/pki-tomcat/ca/debug, but that path > doesn't exist on any of my IPA servers. He said that would normally be the > first place to look. Hence, I am looking for other solutions. Brute force: reproduce the error and run"find /var/log -mmin -1 -type f -ls". This finds the files changed in the last minute - one of these might help.
Jochen -- This space is intentionally left blank. -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> <mailto:freeipa-users-leave@lists.fedorahosted.org mailto:freeipa-users-leave@lists.fedorahosted.org>
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedo rahosted.org
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Jochen Hein -
Kristian Petersen