Hi,
I seem to be facing a similar issue with one of my KRAs. My KRA certificates were, for some reason, not automatically renewed when they expired last month. Using `ipa-cert-fix` correctly fixed them on _one_ host. On the other, they seem to be stuck in the renewal state and `ipa-cert-fix` claims there's nothing to do:
``` Request ID '20191031183458': status: MONITORING ca-error: Server at " http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG subject: CN=KRA Audit,O=MYDOMAIN.ORG expires: 2020-06-27 01:54:34 EDT key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183459': status: MONITORING ca-error: Server at " http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG expires: 2020-06-27 01:54:30 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183500': status: MONITORING ca-error: Server at " http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG expires: 2020-06-27 01:54:32 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes ```
Here are the sequence of events that seem to have led to this:
1. Install FreeIPA Master many years ago and continue to upgrade it from time to time. 2. Install FreeIPA Replica a few years after and continue to upgrade it from time to time. 3. Allow the certificates to expire on both nodes. 4. Attempt to patch the replica via `yum upgrade` on the second node. 5. Notice after reboot that `pki-tomcatd` is having trouble and discover certificate issues. 5. Issue `ipa-cert-fix`, reboot again, and notice that things are working. Try and create a key in the vault. 6. Attempt to patch the master via `yum upgrade` on the first node. 7. Notice after reboot that everything seems to be ok. Try and create a key in the vault. 8. Notice a few days later that renewal seems to be broken on the first node.
At this point `ipa-cert-fix` just shows that everything is fine. If I run it with -v, and then check the "storageCert cert-pki-kra" certificate with `openssl x509 -text -in`, I'm shown:
Validity Not Before: Jun 29 00:52:33 2020 GMT Not After : Jun 19 00:52:33 2022 GMT
On the second known, `getcert list` shows correct expirations for those certificates:
Request ID '20191206005909': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG expires: 2022-06-18 20:52:33 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes
It seems like _something_, perhaps `ipa-cert-fix` somehow renewed these certificates but...outside of certmonger? Is this some other version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The certificates are not in CA_WORKING though, they're in MONITORING.
What can I do to get myself out of this state as it seems like I'm in a "this could explode at any moment" situation?
This is on Fedora 30 with IP version:
Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 07:59:16 PM EDT. Installed Packages Name : certmonger Version : 0.79.9 Release : 1.fc30 Architecture : x86_64 Size : 3.4 M Source : certmonger-0.79.9-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Name : freeipa-server Version : 4.8.3 Release : 1.fc30 Architecture : x86_64 Size : 1.3 M Source : freeipa-4.8.3-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Thanks!
Ilya Kogan w: github.com/ikogan e: ikogan@mythicnet.org http://twitter.com/ilkogan https://www.linkedin.com/in/ilyakogan/
On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote:
Hi,
I seem to be facing a similar issue with one of my KRAs. My KRA certificates were, for some reason, not automatically renewed when they expired last month. Using `ipa-cert-fix` correctly fixed them on _one_ host. On the other, they seem to be stuck in the renewal state and `ipa-cert-fix` claims there's nothing to do:
Request ID '20191031183458': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> expires: 2020-06-27 01:54:34 EDT key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183459': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> expires: 2020-06-27 01:54:30 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183500': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> expires: 2020-06-27 01:54:32 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yesHere are the sequence of events that seem to have led to this:
- Install FreeIPA Master many years ago and continue to upgrade it from
time to time. 2. Install FreeIPA Replica a few years after and continue to upgrade it from time to time. 3. Allow the certificates to expire on both nodes. 4. Attempt to patch the replica via `yum upgrade` on the second node. 5. Notice after reboot that `pki-tomcatd` is having trouble and discover certificate issues. 5. Issue `ipa-cert-fix`, reboot again, and notice that things are working. Try and create a key in the vault. 6. Attempt to patch the master via `yum upgrade` on the first node. 7. Notice after reboot that everything seems to be ok. Try and create a key in the vault. 8. Notice a few days later that renewal seems to be broken on the first node.
At this point `ipa-cert-fix` just shows that everything is fine. If I run it with -v, and then check the "storageCert cert-pki-kra" certificate with `openssl x509 -text -in`, I'm shown:
Hi, just double-checking, but did you run ipa-cert-fix on the replica that was repaired in step 5? If that's the case, it's normal that ipa-cert-fix does not see any issue as it's running only locally and does not attempt to repair remote nodes.
You will need to login to the node with expired certs and run ipa-cert-fix there.
HTH, flo
Validity Not Before: Jun 29 00:52:33 2020 GMT Not After : Jun 19 00:52:33 2022 GMT
On the second known, `getcert list` shows correct expirations for those certificates:
Request ID '20191206005909': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG http://MYDOMAIN.ORG subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG http://MYDOMAIN.ORG expires: 2022-06-18 20:52:33 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes
It seems like _something_, perhaps `ipa-cert-fix` somehow renewed these certificates but...outside of certmonger? Is this some other version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The certificates are not in CA_WORKING though, they're in MONITORING.
What can I do to get myself out of this state as it seems like I'm in a "this could explode at any moment" situation?
This is on Fedora 30 with IP version:
Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 07:59:16 PM EDT. Installed Packages Name : certmonger Version : 0.79.9 Release : 1.fc30 Architecture : x86_64 Size : 3.4 M Source : certmonger-0.79.9-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Name : freeipa-server Version : 4.8.3 Release : 1.fc30 Architecture : x86_64 Size : 1.3 M Source : freeipa-4.8.3-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Thanks!
Ilya Kogan w: github.com/ikogan http://github.com/ikogan e: ikogan@mythicnet.org mailto:ikogan@mythicnet.org http://twitter.com/ilkogan https://www.linkedin.com/in/ilyakogan/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Florence Blanc-Renaud via FreeIPA-users wrote:
On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote:
Hi,
I seem to be facing a similar issue with one of my KRAs. My KRA certificates were, for some reason, not automatically renewed when they expired last month. Using `ipa-cert-fix` correctly fixed them on _one_ host. On the other, they seem to be stuck in the renewal state and `ipa-cert-fix` claims there's nothing to do:
Request ID '20191031183458': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> expires: 2020-06-27 01:54:34 EDT key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183459': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> expires: 2020-06-27 01:54:30 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183500': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> expires: 2020-06-27 01:54:32 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yesHere are the sequence of events that seem to have led to this:
- Install FreeIPA Master many years ago and continue to upgrade it
from time to time. 2. Install FreeIPA Replica a few years after and continue to upgrade it from time to time. 3. Allow the certificates to expire on both nodes. 4. Attempt to patch the replica via `yum upgrade` on the second node. 5. Notice after reboot that `pki-tomcatd` is having trouble and discover certificate issues. 5. Issue `ipa-cert-fix`, reboot again, and notice that things are working. Try and create a key in the vault. 6. Attempt to patch the master via `yum upgrade` on the first node. 7. Notice after reboot that everything seems to be ok. Try and create a key in the vault. 8. Notice a few days later that renewal seems to be broken on the first node.
At this point `ipa-cert-fix` just shows that everything is fine. If I run it with -v, and then check the "storageCert cert-pki-kra" certificate with `openssl x509 -text -in`, I'm shown:
Hi, just double-checking, but did you run ipa-cert-fix on the replica that was repaired in step 5? If that's the case, it's normal that ipa-cert-fix does not see any issue as it's running only locally and does not attempt to repair remote nodes.
You will need to login to the node with expired certs and run ipa-cert-fix there.
I'd also look to see which one is the renewal master. That is the one that should renew the cert. I'm too curious why the renewal raised an error (as if it actually tried to renew) rather than either go into CA_WORKING or pick up the updated cert.
I'd also make sure that replication is working. On each master:
# ipa-csreplica-manage list -v `hostname`
rob
HTH, flo
Validity Not Before: Jun 29 00:52:33 2020 GMT Not After : Jun 19 00:52:33 2022 GMT
On the second known, `getcert list` shows correct expirations for those certificates:
Request ID '20191206005909': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG http://MYDOMAIN.ORG subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG http://MYDOMAIN.ORG expires: 2022-06-18 20:52:33 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes
It seems like _something_, perhaps `ipa-cert-fix` somehow renewed these certificates but...outside of certmonger? Is this some other version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The certificates are not in CA_WORKING though, they're in MONITORING.
What can I do to get myself out of this state as it seems like I'm in a "this could explode at any moment" situation?
This is on Fedora 30 with IP version:
Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 07:59:16 PM EDT. Installed Packages Name : certmonger Version : 0.79.9 Release : 1.fc30 Architecture : x86_64 Size : 3.4 M Source : certmonger-0.79.9-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Name : freeipa-server Version : 4.8.3 Release : 1.fc30 Architecture : x86_64 Size : 1.3 M Source : freeipa-4.8.3-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Thanks!
Ilya Kogan w: github.com/ikogan http://github.com/ikogan e: ikogan@mythicnet.org mailto:ikogan@mythicnet.org http://twitter.com/ilkogan https://www.linkedin.com/in/ilyakogan/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi,
Thanks for the help so far! I've actually run `ipa-cert-fix` on both nodes, it says everything is ok on both nodes. When I run it with verbose mode, it spits out the command it's running and the certificate it got, for example:
``` ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] ```
If I then take that cert and ask what `openssl x509 -text -noout` thinks about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. Strangely, though, when I ask `getcert list`, it shows that the certificate:
``` certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' ```
expires on 2020-06-27. It's almost as if this node's certificate has _already_ been renewed but certmonger (I think) doesn't know about it, which might be why it's having trouble renewing it.
Here's what the two nodes say about replication:
From node one:
``` ipa-two.mydomain.org last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ```
From node two:
``` ipa-one.gaea.mythicnet.org last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ```
I suppose this might be a good time to mention that this is a simple two node multi-master setup. Finally, I'm not sure if I'm doing this correctly, but to make absolutely sure about which node is the renewal master, I ran this on both nodes:
``` ldapsearch -H ldap://ipa-one.gaea.mythicnet.org -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ldapsearch -H ldap://ipa-two.gaea.mythicnet.org -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ```
The result for both is:
``` dn: cn=CA,cn=ipa-one.gaea.mythicnet.org ,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org ```
So it looks like the renewal master is the one having this problem.
Ilya Kogan w: github.com/ikogan e: ikogan@mythicnet.org http://twitter.com/ilkogan https://www.linkedin.com/in/ilyakogan/
On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden rcritten@redhat.com wrote:
Florence Blanc-Renaud via FreeIPA-users wrote:
On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote:
Hi,
I seem to be facing a similar issue with one of my KRAs. My KRA certificates were, for some reason, not automatically renewed when they expired last month. Using `ipa-cert-fix` correctly fixed them on _one_ host. On the other, they seem to be stuck in the renewal state and `ipa-cert-fix` claims there's nothing to do:
Request ID '20191031183458': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-kra',token='NSS Certificate DB',pin set certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG http://MYDOMAIN.ORG subject: CN=KRA Audit,O=MYDOMAIN.ORG http://MYDOMAIN.ORG expires: 2020-06-27 01:54:34 EDT key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183459': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
cert-pki-kra',token='NSS
Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
cert-pki-kra',token='NSS
Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG http://MYDOMAIN.ORG subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG http://MYDOMAIN.ORG expires: 2020-06-27 01:54:30 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra" track: yes auto-renew: yes Request ID '20191031183500': status: MONITORING ca-error: Server at "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: Missing credential: sessionID stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG http://MYDOMAIN.ORG subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG http://MYDOMAIN.ORG expires: 2020-06-27 01:54:32 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes
Here are the sequence of events that seem to have led to this: 1. Install FreeIPA Master many years ago and continue to upgrade it from time to time. 2. Install FreeIPA Replica a few years after and continue to upgrade it from time to time. 3. Allow the certificates to expire on both nodes. 4. Attempt to patch the replica via `yum upgrade` on the second node. 5. Notice after reboot that `pki-tomcatd` is having trouble and discover certificate issues. 5. Issue `ipa-cert-fix`, reboot again, and notice that things are working. Try and create a key in the vault. 6. Attempt to patch the master via `yum upgrade` on the first node. 7. Notice after reboot that everything seems to be ok. Try and create a key in the vault. 8. Notice a few days later that renewal seems to be broken on the first node. At this point `ipa-cert-fix` just shows that everything is fine. If I run it with -v, and then check the "storageCert cert-pki-kra" certificate with `openssl x509 -text -in`, I'm shown:Hi, just double-checking, but did you run ipa-cert-fix on the replica that was repaired in step 5? If that's the case, it's normal that ipa-cert-fix does not see any issue as it's running only locally and does not attempt to repair remote nodes.
You will need to login to the node with expired certs and run ipa-cert-fix there.
I'd also look to see which one is the renewal master. That is the one that should renew the cert. I'm too curious why the renewal raised an error (as if it actually tried to renew) rather than either go into CA_WORKING or pick up the updated cert.
I'd also make sure that replication is working. On each master:
# ipa-csreplica-manage list -v `hostname`
rob
HTH, flo
Validity Not Before: Jun 29 00:52:33 2020 GMT Not After : Jun 19 00:52:33 2022 GMTOn the second known, `getcert list` shows correct expirations for those certificates:
Request ID '20191206005909': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN.ORG http://MYDOMAIN.ORG subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG http://MYDOMAIN.ORG expires: 2022-06-18 20:52:33 EDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes
It seems like _something_, perhaps `ipa-cert-fix` somehow renewed these certificates but...outside of certmonger? Is this some other version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The certificates are not in CA_WORKING though, they're in MONITORING.
What can I do to get myself out of this state as it seems like I'm in a "this could explode at any moment" situation?
This is on Fedora 30 with IP version:
Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 07:59:16 PM EDT. Installed Packages Name : certmonger Version : 0.79.9 Release : 1.fc30 Architecture : x86_64 Size : 3.4 M Source : certmonger-0.79.9-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Name : freeipa-server Version : 4.8.3 Release : 1.fc30 Architecture : x86_64 Size : 1.3 M Source : freeipa-4.8.3-1.fc30.src.rpm Repository : @System From repo : updates
.. snip ..
Thanks!
Ilya Kogan w: github.com/ikogan http://github.com/ikogan e: ikogan@mythicnet.org mailto:ikogan@mythicnet.org http://twitter.com/ilkogan https://www.linkedin.com/in/ilyakogan/
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote:
Hi,
Thanks for the help so far! I've actually run `ipa-cert-fix` on both nodes, it says everything is ok on both nodes. When I run it with verbose mode, it spits out the command it's running and the certificate it got, for example:
``` ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] ```If I then take that cert and ask what `openssl x509 -text -noout` thinks about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. Strangely, though, when I ask `getcert list`, it shows that the certificate:
``` certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' ```expires on 2020-06-27. It's almost as if this node's certificate has _already_ been renewed but certmonger (I think) doesn't know about it, which might be why it's having trouble renewing it.
Hi,
you may want to restart certmonger to force it re-reading the certificate information: # sudo systemctl restart certmonger
flo
Here's what the two nodes say about replication:
From node one:
``` ipa-two.mydomain.org <http://ipa-two.mydomain.org> last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ```From node two:
``` ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ```I suppose this might be a good time to mention that this is a simple two node multi-master setup. Finally, I'm not sure if I'm doing this correctly, but to make absolutely sure about which node is the renewal master, I ran this on both nodes:
``` ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ```The result for both is:
``` dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org ```So it looks like the renewal master is the one having this problem.
Ilya Kogan w: github.com/ikogan http://github.com/ikogan e: ikogan@mythicnet.org mailto:ikogan@mythicnet.org http://twitter.com/ilkogan https://www.linkedin.com/in/ilyakogan/
On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
Florence Blanc-Renaud via FreeIPA-users wrote: > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: >> Hi, >> >> I seem to be facing a similar issue with one of my KRAs. My KRA >> certificates were, for some reason, not automatically renewed when >> they expired last month. Using `ipa-cert-fix` correctly fixed them on >> _one_ host. On the other, they seem to be stuck in the renewal state >> and `ipa-cert-fix` claims there's nothing to do: >> >> ``` >> Request ID '20191031183458': >> status: MONITORING >> ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >> Missing credential: sessionID >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-kra',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-kra',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >> expires: 2020-06-27 01:54:34 EDT >> key usage: digitalSignature,nonRepudiation >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-kra" >> track: yes >> auto-renew: yes >> Request ID '20191031183459': >> status: MONITORING >> ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >> Missing credential: sessionID >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS >> Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS >> Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> expires: 2020-06-27 01:54:30 EDT >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-clientAuth >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "transportCert cert-pki-kra" >> track: yes >> auto-renew: yes >> Request ID '20191031183500': >> status: MONITORING >> ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: >> Missing credential: sessionID >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> expires: 2020-06-27 01:54:32 EDT >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-clientAuth >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "storageCert cert-pki-kra" >> track: yes >> auto-renew: yes >> ``` >> >> Here are the sequence of events that seem to have led to this: >> >> 1. Install FreeIPA Master many years ago and continue to upgrade it >> from time to time. >> 2. Install FreeIPA Replica a few years after and continue to upgrade >> it from time to time. >> 3. Allow the certificates to expire on both nodes. >> 4. Attempt to patch the replica via `yum upgrade` on the second node. >> 5. Notice after reboot that `pki-tomcatd` is having trouble and >> discover certificate issues. >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are >> working. Try and create a key in the vault. >> 6. Attempt to patch the master via `yum upgrade` on the first node. >> 7. Notice after reboot that everything seems to be ok. Try and create >> a key in the vault. >> 8. Notice a few days later that renewal seems to be broken on the >> first node. >> >> At this point `ipa-cert-fix` just shows that everything is fine. If I >> run it with -v, and then check the "storageCert cert-pki-kra" >> certificate with `openssl x509 -text -in`, I'm shown: > > Hi, > just double-checking, but did you run ipa-cert-fix on the replica that > was repaired in step 5? If that's the case, it's normal that > ipa-cert-fix does not see any issue as it's running only locally and > does not attempt to repair remote nodes. > > You will need to login to the node with expired certs and run > ipa-cert-fix there. I'd also look to see which one is the renewal master. That is the one that should renew the cert. I'm too curious why the renewal raised an error (as if it actually tried to renew) rather than either go into CA_WORKING or pick up the updated cert. I'd also make sure that replication is working. On each master: # ipa-csreplica-manage list -v `hostname` rob > > HTH, > flo > >> >> Validity >> Not Before: Jun 29 00:52:33 2020 GMT >> Not After : Jun 19 00:52:33 2022 GMT >> >> On the second known, `getcert list` shows correct expirations for >> those certificates: >> >> Request ID '20191206005909': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> expires: 2022-06-18 20:52:33 EDT >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-clientAuth >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "storageCert cert-pki-kra" >> track: yes >> auto-renew: yes >> >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed >> these certificates but...outside of certmonger? Is this some other >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The >> certificates are not in CA_WORKING though, they're in MONITORING. >> >> What can I do to get myself out of this state as it seems like I'm in >> a "this could explode at any moment" situation? >> >> This is on Fedora 30 with IP version: >> >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 >> 07:59:16 PM EDT. >> Installed Packages >> Name : certmonger >> Version : 0.79.9 >> Release : 1.fc30 >> Architecture : x86_64 >> Size : 3.4 M >> Source : certmonger-0.79.9-1.fc30.src.rpm >> Repository : @System >> From repo : updates >> >> .. snip .. >> >> Name : freeipa-server >> Version : 4.8.3 >> Release : 1.fc30 >> Architecture : x86_64 >> Size : 1.3 M >> Source : freeipa-4.8.3-1.fc30.src.rpm >> Repository : @System >> From repo : updates >> >> .. snip .. >> >> Thanks! >> >> >> Ilya Kogan >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> e: >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> >> <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >> To unsubscribe send an email to >> freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >> >> > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Wow ok, that was easy. `getcert list` now reports correct expiration dates for those certificates and they're all in MONITORING. It still has that ca-error field although it's no longer trying to renew. Is that going to be an issue or is it just going to try again when it's time to renew and succeed?
On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud flo@redhat.com wrote:
On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote:
Hi,
Thanks for the help so far! I've actually run `ipa-cert-fix` on both nodes, it says everything is ok on both nodes. When I run it with verbose mode, it spits out the command it's running and the certificate it got, for example:
``` ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] ```If I then take that cert and ask what `openssl x509 -text -noout` thinks about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. Strangely, though, when I ask `getcert list`, it shows that the
certificate:
``` certificate:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS
Certificate DB' ```expires on 2020-06-27. It's almost as if this node's certificate has _already_ been renewed but certmonger (I think) doesn't know about it, which might be why it's having trouble renewing it.
Hi,
you may want to restart certmonger to force it re-reading the certificate information: # sudo systemctl restart certmonger
flo
Here's what the two nodes say about replication:
From node one:
``` ipa-two.mydomain.org <http://ipa-two.mydomain.org> last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ```From node two:
``` ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ```I suppose this might be a good time to mention that this is a simple two node multi-master setup. Finally, I'm not sure if I'm doing this correctly, but to make absolutely sure about which node is the renewal master, I ran this on both nodes:
``` ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ```The result for both is:
``` dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org ```
So it looks like the renewal master is the one having this problem.
Ilya Kogan w: github.com/ikogan http://github.com/ikogan e:
ikogan@mythicnet.org
mailto:ikogan@mythicnet.org http://twitter.com/ilkogan https://www.linkedin.com/in/ilyakogan/
On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
Florence Blanc-Renaud via FreeIPA-users wrote: > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: >> Hi, >> >> I seem to be facing a similar issue with one of my KRAs. My KRA >> certificates were, for some reason, not automatically renewedwhen
>> they expired last month. Using `ipa-cert-fix` correctly fixed them on >> _one_ host. On the other, they seem to be stuck in therenewal state
>> and `ipa-cert-fix` claims there's nothing to do: >> >> ``` >> Request ID '20191031183458': >> status: MONITORING >> ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"replied:
>> Missing credential: sessionID >> stuck: no >> key pair storage: >>type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>> cert-pki-kra',token='NSS Certificate DB',pin set >> certificate: >>type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>> cert-pki-kra',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> >> expires: 2020-06-27 01:54:34 EDT >> key usage: digitalSignature,nonRepudiation >> pre-save command:/usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-kra" >> track: yes >> auto-renew: yes >> Request ID '20191031183459': >> status: MONITORING >> ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"replied:
>> Missing credential: sessionID >> stuck: no >> key pair storage: >>type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
cert-pki-kra',token='NSS >> Certificate DB',pin set >> certificate: >>type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
cert-pki-kra',token='NSS >> Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> expires: 2020-06-27 01:54:30 EDT >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-clientAuth >> pre-save command:/usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "transportCert cert-pki-kra" >> track: yes >> auto-renew: yes >> Request ID '20191031183500': >> status: MONITORING >> ca-error: Server at >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"replied:
>> Missing credential: sessionID >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> expires: 2020-06-27 01:54:32 EDT >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-clientAuth >> pre-save command:/usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "storageCert cert-pki-kra" >> track: yes >> auto-renew: yes >> ``` >> >> Here are the sequence of events that seem to have led to this: >> >> 1. Install FreeIPA Master many years ago and continue to upgradeit
>> from time to time. >> 2. Install FreeIPA Replica a few years after and continue toupgrade
>> it from time to time. >> 3. Allow the certificates to expire on both nodes. >> 4. Attempt to patch the replica via `yum upgrade` on the second node. >> 5. Notice after reboot that `pki-tomcatd` is having trouble and >> discover certificate issues. >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are >> working. Try and create a key in the vault. >> 6. Attempt to patch the master via `yum upgrade` on the firstnode.
>> 7. Notice after reboot that everything seems to be ok. Try and create >> a key in the vault. >> 8. Notice a few days later that renewal seems to be broken on the >> first node. >> >> At this point `ipa-cert-fix` just shows that everything is fine. If I >> run it with -v, and then check the "storageCert cert-pki-kra" >> certificate with `openssl x509 -text -in`, I'm shown: > > Hi, > just double-checking, but did you run ipa-cert-fix on the replica that > was repaired in step 5? If that's the case, it's normal that > ipa-cert-fix does not see any issue as it's running only locallyand
> does not attempt to repair remote nodes. > > You will need to login to the node with expired certs and run > ipa-cert-fix there. I'd also look to see which one is the renewal master. That is the one that should renew the cert. I'm too curious why the renewal raised an error (as if it actually tried to renew) rather than either go into CA_WORKING or pick up the updated cert. I'd also make sure that replication is working. On each master: # ipa-csreplica-manage list -v `hostname` rob > > HTH, > flo > >> >> Validity >> Not Before: Jun 29 00:52:33 2020 GMT >> Not After : Jun 19 00:52:33 2022 GMT >> >> On the second known, `getcert list` shows correct expirations for >> those certificates: >> >> Request ID '20191206005909': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert >> cert-pki-kra',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> >> <http://MYDOMAIN.ORG> >> expires: 2022-06-18 20:52:33 EDT >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-clientAuth >> pre-save command:/usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "storageCert cert-pki-kra" >> track: yes >> auto-renew: yes >> >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed >> these certificates but...outside of certmonger? Is this someother
>> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907?The
>> certificates are not in CA_WORKING though, they're in MONITORING. >> >> What can I do to get myself out of this state as it seems like I'm in >> a "this could explode at any moment" situation? >> >> This is on Fedora 30 with IP version: >> >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 >> 07:59:16 PM EDT. >> Installed Packages >> Name : certmonger >> Version : 0.79.9 >> Release : 1.fc30 >> Architecture : x86_64 >> Size : 3.4 M >> Source : certmonger-0.79.9-1.fc30.src.rpm >> Repository : @System >> From repo : updates >> >> .. snip .. >> >> Name : freeipa-server >> Version : 4.8.3 >> Release : 1.fc30 >> Architecture : x86_64 >> Size : 1.3 M >> Source : freeipa-4.8.3-1.fc30.src.rpm >> Repository : @System >> From repo : updates >> >> .. snip .. >> >> Thanks! >> >> >> Ilya Kogan >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> e: >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> >> <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> >> To unsubscribe send an email to >> freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >>https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
>> >> > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: >https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
>
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Ilya Kogan wrote:
Wow ok, that was easy. `getcert list` now reports correct expiration dates for those certificates and they're all in MONITORING. It still has that ca-error field although it's no longer trying to renew. Is that going to be an issue or is it just going to try again when it's time to renew and succeed?
I don't know. I'd check the journal to see if it logged any errors post-restart. I don't believe that the ca-error is stored between restarts. You could grep in /var/lib/certmonger/requests to see I suppose.
rob
On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <flo@redhat.com mailto:flo@redhat.com> wrote:
On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote: > Hi, > > Thanks for the help so far! I've actually run `ipa-cert-fix` on both > nodes, it says everything is ok on both nodes. When I run it with > verbose mode, it spits out the command it's running and the certificate > it got, for example: > > ``` > ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', > 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert > cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] > ``` > > > If I then take that cert and ask what `openssl x509 -text -noout` thinks > about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. > Strangely, though, when I ask `getcert list`, it shows that the certificate: > > ``` > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS > Certificate DB' > ``` > > > expires on 2020-06-27. It's almost as if this node's certificate has > _already_ been renewed but certmonger (I think) doesn't know about it, > which might be why it's having trouble renewing it. > Hi, you may want to restart certmonger to force it re-reading the certificate information: # sudo systemctl restart certmonger flo > Here's what the two nodes say about replication: > > From node one: > > ``` > ipa-two.mydomain.org <http://ipa-two.mydomain.org> <http://ipa-two.mydomain.org> > last update status: Error (0) Replica acquired successfully: > Incremental update succeeded > last update ended: 2020-07-06 17:46:17+00:00 > ``` > > > From node two: > > ``` > ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> <http://ipa-one.gaea.mythicnet.org> > last update status: Error (0) Replica acquired successfully: > Incremental update succeeded > last update ended: 2020-07-06 17:46:17+00:00 > ``` > > > I suppose this might be a good time to mention that this is a simple two > node multi-master setup. Finally, I'm not sure if I'm doing this > correctly, but to make absolutely sure about which node is the renewal > master, I ran this on both nodes: > > ``` > ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> > <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > ``` > > > The result for both is: > > ``` > dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org > ``` > > > So it looks like the renewal master is the one having this problem. > > > Ilya Kogan > w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> e: ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> > <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> > > > > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote: > > Florence Blanc-Renaud via FreeIPA-users wrote: > > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: > >> Hi, > >> > >> I seem to be facing a similar issue with one of my KRAs. My KRA > >> certificates were, for some reason, not automatically renewed when > >> they expired last month. Using `ipa-cert-fix` correctly fixed > them on > >> _one_ host. On the other, they seem to be stuck in the renewal state > >> and `ipa-cert-fix` claims there's nothing to do: > >> > >> ``` > >> Request ID '20191031183458': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:34 EDT > >> key usage: digitalSignature,nonRepudiation > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "auditSigningCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> Request ID '20191031183459': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > cert-pki-kra',token='NSS > >> Certificate DB',pin set > >> certificate: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > cert-pki-kra',token='NSS > >> Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:30 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "transportCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> Request ID '20191031183500': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:32 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "storageCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> ``` > >> > >> Here are the sequence of events that seem to have led to this: > >> > >> 1. Install FreeIPA Master many years ago and continue to upgrade it > >> from time to time. > >> 2. Install FreeIPA Replica a few years after and continue to upgrade > >> it from time to time. > >> 3. Allow the certificates to expire on both nodes. > >> 4. Attempt to patch the replica via `yum upgrade` on the second > node. > >> 5. Notice after reboot that `pki-tomcatd` is having trouble and > >> discover certificate issues. > >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are > >> working. Try and create a key in the vault. > >> 6. Attempt to patch the master via `yum upgrade` on the first node. > >> 7. Notice after reboot that everything seems to be ok. Try and > create > >> a key in the vault. > >> 8. Notice a few days later that renewal seems to be broken on the > >> first node. > >> > >> At this point `ipa-cert-fix` just shows that everything is fine. > If I > >> run it with -v, and then check the "storageCert cert-pki-kra" > >> certificate with `openssl x509 -text -in`, I'm shown: > > > > Hi, > > just double-checking, but did you run ipa-cert-fix on the replica > that > > was repaired in step 5? If that's the case, it's normal that > > ipa-cert-fix does not see any issue as it's running only locally and > > does not attempt to repair remote nodes. > > > > You will need to login to the node with expired certs and run > > ipa-cert-fix there. > > I'd also look to see which one is the renewal master. That is the one > that should renew the cert. I'm too curious why the renewal raised an > error (as if it actually tried to renew) rather than either go into > CA_WORKING or pick up the updated cert. > > I'd also make sure that replication is working. On each master: > > # ipa-csreplica-manage list -v `hostname` > > rob > > > > > HTH, > > flo > > > >> > >> Validity > >> Not Before: Jun 29 00:52:33 2020 GMT > >> Not After : Jun 19 00:52:33 2022 GMT > >> > >> On the second known, `getcert list` shows correct expirations for > >> those certificates: > >> > >> Request ID '20191206005909': > >> status: MONITORING > >> stuck: no > >> key pair storage: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> expires: 2022-06-18 20:52:33 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "storageCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> > >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed > >> these certificates but...outside of certmonger? Is this some other > >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The > >> certificates are not in CA_WORKING though, they're in MONITORING. > >> > >> What can I do to get myself out of this state as it seems like > I'm in > >> a "this could explode at any moment" situation? > >> > >> This is on Fedora 30 with IP version: > >> > >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 > >> 07:59:16 PM EDT. > >> Installed Packages > >> Name : certmonger > >> Version : 0.79.9 > >> Release : 1.fc30 > >> Architecture : x86_64 > >> Size : 3.4 M > >> Source : certmonger-0.79.9-1.fc30.src.rpm > >> Repository : @System > >> From repo : updates > >> > >> .. snip .. > >> > >> Name : freeipa-server > >> Version : 4.8.3 > >> Release : 1.fc30 > >> Architecture : x86_64 > >> Size : 1.3 M > >> Source : freeipa-4.8.3-1.fc30.src.rpm > >> Repository : @System > >> From repo : updates > >> > >> .. snip .. > >> > >> Thanks! > >> > >> > >> Ilya Kogan > >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> > <http://github.com/ikogan> e: > >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>> > >> <http://twitter.com/ilkogan> > <https://www.linkedin.com/in/ilyakogan/> > >> > >> > >> _______________________________________________ > >> FreeIPA-users mailing list -- > freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > >> To unsubscribe send an email to > >> freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > >> Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > >> > >> > > _______________________________________________ > > FreeIPA-users mailing list -- > freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > > To unsubscribe send an email to > freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org >
Thanks for that info, I don't see any suspicious errors in startup that I haven't seen before. Just the following:
- Token named "NSS Generic Crypto Services", not "NSS Certificate DB", skipping. - Error opening "/etc/httpd/alias/pwdfile.txt": No such file or directory.
I don't think either of these are really an issue but I could be wrong.
Grepping the request files does indeed show those ca-error values though. They don't really bother me if they won't cause issues. It seems like it's just the last error it got from the CA, which just won't be updated until it tries to request something next time.
On Wed, Jul 8, 2020, 2:41 PM Rob Crittenden rcritten@redhat.com wrote:
Ilya Kogan wrote:
Wow ok, that was easy. `getcert list` now reports correct expiration dates for those certificates and they're all in MONITORING. It still has that ca-error field although it's no longer trying to renew. Is that going to be an issue or is it just going to try again when it's time to renew and succeed?
I don't know. I'd check the journal to see if it logged any errors post-restart. I don't believe that the ca-error is stored between restarts. You could grep in /var/lib/certmonger/requests to see I suppose.
rob
On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <flo@redhat.com mailto:flo@redhat.com> wrote:
On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote: > Hi, > > Thanks for the help so far! I've actually run `ipa-cert-fix` onboth
> nodes, it says everything is ok on both nodes. When I run it with > verbose mode, it spits out the command it's running and the certificate > it got, for example: > > ``` > ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', > 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert > cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] > ``` > > > If I then take that cert and ask what `openssl x509 -text -noout` thinks > about it, it tells me that it's valid from 2020-06-29 to2022-06-29.
> Strangely, though, when I ask `getcert list`, it shows that the certificate: > > ``` > certificate: >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
cert-pki-kra',token='NSS > Certificate DB' > ``` > > > expires on 2020-06-27. It's almost as if this node's certificatehas
> _already_ been renewed but certmonger (I think) doesn't know about it, > which might be why it's having trouble renewing it. > Hi, you may want to restart certmonger to force it re-reading the certificate information: # sudo systemctl restart certmonger flo > Here's what the two nodes say about replication: > > From node one: > > ``` > ipa-two.mydomain.org <http://ipa-two.mydomain.org> <http://ipa-two.mydomain.org> > last update status: Error (0) Replica acquired successfully: > Incremental update succeeded > last update ended: 2020-07-06 17:46:17+00:00 > ``` > > > From node two: > > ``` > ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> <http://ipa-one.gaea.mythicnet.org> > last update status: Error (0) Replica acquired successfully: > Incremental update succeeded > last update ended: 2020-07-06 17:46:17+00:00 > ``` > > > I suppose this might be a good time to mention that this is a simple two > node multi-master setup. Finally, I'm not sure if I'm doing this > correctly, but to make absolutely sure about which node is the renewal > master, I ran this on both nodes: > > ``` > ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> > <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > ``` > > > The result for both is: > > ``` > dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org > ``` > > > So it looks like the renewal master is the one having this problem. > > > Ilya Kogan > w: github.com/ikogan http://github.com/ikogan http://github.com/ikogan e: ikogan@mythicnet.org mailto:ikogan@mythicnet.org > <mailto:ikogan@mythicnet.org mailto:ikogan@mythicnet.org> > http://twitter.com/ilkogan <
https://www.linkedin.com/in/ilyakogan/%3E
> > > > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote: > > Florence Blanc-Renaud via FreeIPA-users wrote: > > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: > >> Hi, > >> > >> I seem to be facing a similar issue with one of my KRAs. My KRA > >> certificates were, for some reason, not automatically renewed when > >> they expired last month. Using `ipa-cert-fix` correctlyfixed
> them on > >> _one_ host. On the other, they seem to be stuck in the renewal state > >> and `ipa-cert-fix` claims there's nothing to do: > >> > >> ``` > >> Request ID '20191031183458': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:34 EDT > >> key usage: digitalSignature,nonRepudiation > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "auditSigningCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> Request ID '20191031183459': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
> cert-pki-kra',token='NSS > >> Certificate DB',pin set > >> certificate: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
> cert-pki-kra',token='NSS > >> Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:30 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "transportCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> Request ID '20191031183500': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:32 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "storageCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> ``` > >> > >> Here are the sequence of events that seem to have led tothis:
> >> > >> 1. Install FreeIPA Master many years ago and continue to upgrade it > >> from time to time. > >> 2. Install FreeIPA Replica a few years after and continue to upgrade > >> it from time to time. > >> 3. Allow the certificates to expire on both nodes. > >> 4. Attempt to patch the replica via `yum upgrade` on the second > node. > >> 5. Notice after reboot that `pki-tomcatd` is having trouble and > >> discover certificate issues. > >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are > >> working. Try and create a key in the vault. > >> 6. Attempt to patch the master via `yum upgrade` on the first node. > >> 7. Notice after reboot that everything seems to be ok. Tryand
> create > >> a key in the vault. > >> 8. Notice a few days later that renewal seems to be broken on the > >> first node. > >> > >> At this point `ipa-cert-fix` just shows that everything is fine. > If I > >> run it with -v, and then check the "storageCertcert-pki-kra"
> >> certificate with `openssl x509 -text -in`, I'm shown: > > > > Hi, > > just double-checking, but did you run ipa-cert-fix on the replica > that > > was repaired in step 5? If that's the case, it's normal that > > ipa-cert-fix does not see any issue as it's running only locally and > > does not attempt to repair remote nodes. > > > > You will need to login to the node with expired certs andrun
> > ipa-cert-fix there. > > I'd also look to see which one is the renewal master. That is the one > that should renew the cert. I'm too curious why the renewal raised an > error (as if it actually tried to renew) rather than either go into > CA_WORKING or pick up the updated cert. > > I'd also make sure that replication is working. On each master: > > # ipa-csreplica-manage list -v `hostname` > > rob > > > > > HTH, > > flo > > > >> > >> Validity > >> Not Before: Jun 29 00:52:33 2020 GMT > >> Not After : Jun 19 00:52:33 2022 GMT > >> > >> On the second known, `getcert list` shows correct expirations for > >> those certificates: > >> > >> Request ID '20191206005909': > >> status: MONITORING > >> stuck: no > >> key pair storage: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> >type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > >> <http://MYDOMAIN.ORG> > >> expires: 2022-06-18 20:52:33 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: > /usr/libexec/ipa/certmonger/renew_ca_cert > >> "storageCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> > >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed > >> these certificates but...outside of certmonger? Is this some other > >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The > >> certificates are not in CA_WORKING though, they're in MONITORING. > >> > >> What can I do to get myself out of this state as it seemslike
> I'm in > >> a "this could explode at any moment" situation? > >> > >> This is on Fedora 30 with IP version: > >> > >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul2020
> >> 07:59:16 PM EDT. > >> Installed Packages > >> Name : certmonger > >> Version : 0.79.9 > >> Release : 1.fc30 > >> Architecture : x86_64 > >> Size : 3.4 M > >> Source : certmonger-0.79.9-1.fc30.src.rpm > >> Repository : @System > >> From repo : updates > >> > >> .. snip .. > >> > >> Name : freeipa-server > >> Version : 4.8.3 > >> Release : 1.fc30 > >> Architecture : x86_64 > >> Size : 1.3 M > >> Source : freeipa-4.8.3-1.fc30.src.rpm > >> Repository : @System > >> From repo : updates > >> > >> .. snip .. > >> > >> Thanks! > >> > >> > >> Ilya Kogan > >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> > <http://github.com/ikogan> e: > >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>> > >> <http://twitter.com/ilkogan> > <https://www.linkedin.com/in/ilyakogan/> > >> > >> > >> _______________________________________________ > >> FreeIPA-users mailing list -- > freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > >> To unsubscribe send an email to > >> freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > >> Fedora Code of Conduct: > >>https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> >https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
> >> > >> > > _______________________________________________ > > FreeIPA-users mailing list -- > freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > > To unsubscribe send an email to > freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > > Fedora Code of Conduct: > >https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > >https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
> > > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives:https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
>
Ilya Kogan via FreeIPA-users wrote:
Thanks for that info, I don't see any suspicious errors in startup that I haven't seen before. Just the following:
- Token named "NSS Generic Crypto Services", not "NSS Certificate DB",
skipping.
- Error opening "/etc/httpd/alias/pwdfile.txt": No such file or directory.
I don't think either of these are really an issue but I could be wrong.
You're right.
Grepping the request files does indeed show those ca-error values though. They don't really bother me if they won't cause issues. It seems like it's just the last error it got from the CA, which just won't be updated until it tries to request something next time.
That's probably true as well. The error won't clear until certmonger tries the request again.
rob
On Wed, Jul 8, 2020, 2:41 PM Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
Ilya Kogan wrote: > Wow ok, that was easy. `getcert list` now reports correct expiration > dates for those certificates and they're all in MONITORING. It still has > that ca-error field although it's no longer trying to renew. Is that > going to be an issue or is it just going to try again when it's time to > renew and succeed? I don't know. I'd check the journal to see if it logged any errors post-restart. I don't believe that the ca-error is stored between restarts. You could grep in /var/lib/certmonger/requests to see I suppose. rob > > On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com> > <mailto:flo@redhat.com <mailto:flo@redhat.com>>> wrote: > > On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote: > > Hi, > > > > Thanks for the help so far! I've actually run `ipa-cert-fix` on both > > nodes, it says everything is ok on both nodes. When I run it with > > verbose mode, it spits out the command it's running and the > certificate > > it got, for example: > > > > ``` > > ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', > > 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert > > cert-pki-kra', '-a', '-f', > '/etc/pki/pki-tomcat/alias/pwdfile.txt'] > > ``` > > > > > > If I then take that cert and ask what `openssl x509 -text -noout` > thinks > > about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. > > Strangely, though, when I ask `getcert list`, it shows that the > certificate: > > > > ``` > > certificate: > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > cert-pki-kra',token='NSS > > Certificate DB' > > ``` > > > > > > expires on 2020-06-27. It's almost as if this node's certificate has > > _already_ been renewed but certmonger (I think) doesn't know about > it, > > which might be why it's having trouble renewing it. > > > Hi, > > you may want to restart certmonger to force it re-reading the > certificate information: > # sudo systemctl restart certmonger > > flo > > > Here's what the two nodes say about replication: > > > > From node one: > > > > ``` > > ipa-two.mydomain.org <http://ipa-two.mydomain.org> <http://ipa-two.mydomain.org> > <http://ipa-two.mydomain.org> > > last update status: Error (0) Replica acquired successfully: > > Incremental update succeeded > > last update ended: 2020-07-06 17:46:17+00:00 > > ``` > > > > > > From node two: > > > > ``` > > ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org> > > last update status: Error (0) Replica acquired successfully: > > Incremental update succeeded > > last update ended: 2020-07-06 17:46:17+00:00 > > ``` > > > > > > I suppose this might be a good time to mention that this is a > simple two > > node multi-master setup. Finally, I'm not sure if I'm doing this > > correctly, but to make absolutely sure about which node is the > renewal > > master, I ran this on both nodes: > > > > ``` > > ldapsearch -H ldap://ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org> > > <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' > -W -b > > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > > ldapsearch -H ldap://ipa-two.gaea.mythicnet.org <http://ipa-two.gaea.mythicnet.org> > <http://ipa-two.gaea.mythicnet.org> > > <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' > -W -b > > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' > > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > > ``` > > > > > > The result for both is: > > > > ``` > > dn: cn=CA,cn=ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org> > <http://ipa-one.gaea.mythicnet.org> > > > <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org > > ``` > > > > > > So it looks like the renewal master is the one having this problem. > > > > > > Ilya Kogan > > w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> > <http://github.com/ikogan> e: ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> > > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>> > > <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> > > > > > > > > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>> wrote: > > > > Florence Blanc-Renaud via FreeIPA-users wrote: > > > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: > > >> Hi, > > >> > > >> I seem to be facing a similar issue with one of my KRAs. > My KRA > > >> certificates were, for some reason, not automatically > renewed when > > >> they expired last month. Using `ipa-cert-fix` correctly fixed > > them on > > >> _one_ host. On the other, they seem to be stuck in the > renewal state > > >> and `ipa-cert-fix` claims there's nothing to do: > > >> > > >> ``` > > >> Request ID '20191031183458': > > >> status: MONITORING > > >> ca-error: Server at > > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" > replied: > > >> Missing credential: sessionID > > >> stuck: no > > >> key pair storage: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > >> certificate: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > > >> cert-pki-kra',token='NSS Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > > >> expires: 2020-06-27 01:54:34 EDT > > >> key usage: digitalSignature,nonRepudiation > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "auditSigningCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> Request ID '20191031183459': > > >> status: MONITORING > > >> ca-error: Server at > > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" > replied: > > >> Missing credential: sessionID > > >> stuck: no > > >> key pair storage: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > > cert-pki-kra',token='NSS > > >> Certificate DB',pin set > > >> certificate: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > > cert-pki-kra',token='NSS > > >> Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Transport > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> expires: 2020-06-27 01:54:30 EDT > > >> key usage: > > >> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >> eku: id-kp-clientAuth > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "transportCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> Request ID '20191031183500': > > >> status: MONITORING > > >> ca-error: Server at > > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" > replied: > > >> Missing credential: sessionID > > >> stuck: no > > >> key pair storage: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > >> certificate: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Storage > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> expires: 2020-06-27 01:54:32 EDT > > >> key usage: > > >> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >> eku: id-kp-clientAuth > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "storageCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> ``` > > >> > > >> Here are the sequence of events that seem to have led to this: > > >> > > >> 1. Install FreeIPA Master many years ago and continue to > upgrade it > > >> from time to time. > > >> 2. Install FreeIPA Replica a few years after and continue > to upgrade > > >> it from time to time. > > >> 3. Allow the certificates to expire on both nodes. > > >> 4. Attempt to patch the replica via `yum upgrade` on the > second > > node. > > >> 5. Notice after reboot that `pki-tomcatd` is having > trouble and > > >> discover certificate issues. > > >> 5. Issue `ipa-cert-fix`, reboot again, and notice that > things are > > >> working. Try and create a key in the vault. > > >> 6. Attempt to patch the master via `yum upgrade` on the > first node. > > >> 7. Notice after reboot that everything seems to be ok. Try and > > create > > >> a key in the vault. > > >> 8. Notice a few days later that renewal seems to be broken > on the > > >> first node. > > >> > > >> At this point `ipa-cert-fix` just shows that everything is > fine. > > If I > > >> run it with -v, and then check the "storageCert cert-pki-kra" > > >> certificate with `openssl x509 -text -in`, I'm shown: > > > > > > Hi, > > > just double-checking, but did you run ipa-cert-fix on the > replica > > that > > > was repaired in step 5? If that's the case, it's normal that > > > ipa-cert-fix does not see any issue as it's running only > locally and > > > does not attempt to repair remote nodes. > > > > > > You will need to login to the node with expired certs and run > > > ipa-cert-fix there. > > > > I'd also look to see which one is the renewal master. That is > the one > > that should renew the cert. I'm too curious why the renewal > raised an > > error (as if it actually tried to renew) rather than either go > into > > CA_WORKING or pick up the updated cert. > > > > I'd also make sure that replication is working. On each master: > > > > # ipa-csreplica-manage list -v `hostname` > > > > rob > > > > > > > > HTH, > > > flo > > > > > >> > > >> Validity > > >> Not Before: Jun 29 00:52:33 2020 GMT > > >> Not After : Jun 19 00:52:33 2022 GMT > > >> > > >> On the second known, `getcert list` shows correct > expirations for > > >> those certificates: > > >> > > >> Request ID '20191206005909': > > >> status: MONITORING > > >> stuck: no > > >> key pair storage: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB',pin set > > >> certificate: > > >> > > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > > >> cert-pki-kra',token='NSS Certificate DB' > > >> CA: dogtag-ipa-ca-renew-agent > > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> subject: CN=KRA Storage > Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG> > > <http://MYDOMAIN.ORG> > > >> <http://MYDOMAIN.ORG> > > >> expires: 2022-06-18 20:52:33 EDT > > >> key usage: > > >> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > >> eku: id-kp-clientAuth > > >> pre-save command: > /usr/libexec/ipa/certmonger/stop_pkicad > > >> post-save command: > > /usr/libexec/ipa/certmonger/renew_ca_cert > > >> "storageCert cert-pki-kra" > > >> track: yes > > >> auto-renew: yes > > >> > > >> It seems like _something_, perhaps `ipa-cert-fix` somehow > renewed > > >> these certificates but...outside of certmonger? Is this > some other > > >> version of > https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The > > >> certificates are not in CA_WORKING though, they're in > MONITORING. > > >> > > >> What can I do to get myself out of this state as it seems like > > I'm in > > >> a "this could explode at any moment" situation? > > >> > > >> This is on Fedora 30 with IP version: > > >> > > >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 > > >> 07:59:16 PM EDT. > > >> Installed Packages > > >> Name : certmonger > > >> Version : 0.79.9 > > >> Release : 1.fc30 > > >> Architecture : x86_64 > > >> Size : 3.4 M > > >> Source : certmonger-0.79.9-1.fc30.src.rpm > > >> Repository : @System > > >> From repo : updates > > >> > > >> .. snip .. > > >> > > >> Name : freeipa-server > > >> Version : 4.8.3 > > >> Release : 1.fc30 > > >> Architecture : x86_64 > > >> Size : 1.3 M > > >> Source : freeipa-4.8.3-1.fc30.src.rpm > > >> Repository : @System > > >> From repo : updates > > >> > > >> .. snip .. > > >> > > >> Thanks! > > >> > > >> > > >> Ilya Kogan > > >> w: github.com/ikogan <http://github.com/ikogan> <http://github.com/ikogan> > <http://github.com/ikogan> > > <http://github.com/ikogan> e: > > >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>> > > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>>> > > >> <http://twitter.com/ilkogan> > > <https://www.linkedin.com/in/ilyakogan/> > > >> > > >> > > >> _______________________________________________ > > >> FreeIPA-users mailing list -- > > freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>> > > >> To unsubscribe send an email to > > >> freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>>> > > >> Fedora Code of Conduct: > > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > >> List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > >> List Archives: > > >> > > > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > >> > > >> > > > _______________________________________________ > > > FreeIPA-users mailing list -- > > freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>>> > > > To unsubscribe send an email to > > freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>>> > > > Fedora Code of Conduct: > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > > > > > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > > > > > > > > _______________________________________________ > > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > <mailto:freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> > > To unsubscribe send an email to > freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > <mailto:freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org>> > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Ilya Kogan -
Rob Crittenden