Hi,
We have FreeIPA running on Ubuntu 16.04 since about two years now. For the last few day we see these messages in the log
Oct 22 17:32:14 ipasrv certmonger[1813]: 2018-10-22 17:32:14 [1813] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 22 17:32:20 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 22 17:32:20 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
Where should I start looking to recover from this?
Could it be that this error already existed since we started? Notice the Request ID of 2016..., and the expires: 2018-10-24.
# getcert list -n ipaCert | sed blabla Number of certificates and requests being tracked: 8. Request ID '20161103094546': status: CA_UNREACHABLE ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN subject: CN=IPA RA,O=MYDOMAIN expires: 2018-10-24 08:45:40 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ?
On 25-10-18 09:56, Kees Bakker via FreeIPA-users wrote:
Hi,
We have FreeIPA running on Ubuntu 16.04 since about two years now. For the last few day we see these messages in the log
Oct 22 17:32:14 ipasrv certmonger[1813]: 2018-10-22 17:32:14 [1813] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 22 17:32:20 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 22 17:32:20 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
Where should I start looking to recover from this?
Kees Bakker via FreeIPA-users wrote:
Could it be that this error already existed since we started? Notice the Request ID of 2016..., and the expires: 2018-10-24.
# getcert list -n ipaCert | sed blabla Number of certificates and requests being tracked: 8. Request ID '20161103094546': status: CA_UNREACHABLE ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN subject: CN=IPA RA,O=MYDOMAIN expires: 2018-10-24 08:45:40 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ?
The problem is your certs expired yesterday so connections won't work (the code and message don't come from within certmonger).
certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens.
rob
On 25-10-18 09:56, Kees Bakker via FreeIPA-users wrote:
Hi,
We have FreeIPA running on Ubuntu 16.04 since about two years now. For the last few day we see these messages in the log
Oct 22 17:32:14 ipasrv certmonger[1813]: 2018-10-22 17:32:14 [1813] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 22 17:32:20 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 22 17:32:20 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
Where should I start looking to recover from this?
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
On 25-10-18 14:18, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Could it be that this error already existed since we started? Notice the Request ID of 2016..., and the expires: 2018-10-24.
# getcert list -n ipaCert | sed blabla Number of certificates and requests being tracked: 8. Request ID '20161103094546': status: CA_UNREACHABLE ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN subject: CN=IPA RA,O=MYDOMAIN expires: 2018-10-24 08:45:40 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ?
The problem is your certs expired yesterday so connections won't work (the code and message don't come from within certmonger).
certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens.
Easy for you to say. You know what you're doing :-) For me it's all magic.
Anyway, I'll try it. I'm just scared to set the clock back, because there may be clients in the network that use this server as a NTP server.
Another thing I want to mention is that the error started showing up two days ago, on Oct 22, while the expiration is today, Oct 24.
Kees Bakker via FreeIPA-users wrote:
On 25-10-18 14:18, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Could it be that this error already existed since we started? Notice the Request ID of 2016..., and the expires: 2018-10-24.
# getcert list -n ipaCert | sed blabla Number of certificates and requests being tracked: 8. Request ID '20161103094546': status: CA_UNREACHABLE ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN subject: CN=IPA RA,O=MYDOMAIN expires: 2018-10-24 08:45:40 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ?
The problem is your certs expired yesterday so connections won't work (the code and message don't come from within certmonger).
certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens.
Easy for you to say. You know what you're doing :-) For me it's all magic.
Anyway, I'll try it. I'm just scared to set the clock back, because there may be clients in the network that use this server as a NTP server.
Another thing I want to mention is that the error started showing up two days ago, on Oct 22, while the expiration is today, Oct 24.
It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
rob
On 25-10-18 16:11, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 25-10-18 14:18, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Could it be that this error already existed since we started? Notice the Request ID of 2016..., and the expires: 2018-10-24.
# getcert list -n ipaCert | sed blabla Number of certificates and requests being tracked: 8. Request ID '20161103094546': status: CA_UNREACHABLE ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN subject: CN=IPA RA,O=MYDOMAIN expires: 2018-10-24 08:45:40 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ?
The problem is your certs expired yesterday so connections won't work (the code and message don't come from within certmonger).
certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens.
Easy for you to say. You know what you're doing :-) For me it's all magic.
Anyway, I'll try it. I'm just scared to set the clock back, because there may be clients in the network that use this server as a NTP server.
Another thing I want to mention is that the error started showing up two days ago, on Oct 22, while the expiration is today, Oct 24.
It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 25-10-18 14:18, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Could it be that this error already existed since we started? Notice the Request ID of 2016..., and the expires: 2018-10-24.
# getcert list -n ipaCert | sed blabla Number of certificates and requests being tracked: 8. Request ID '20161103094546': status: CA_UNREACHABLE ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN subject: CN=IPA RA,O=MYDOMAIN expires: 2018-10-24 08:45:40 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ?
The problem is your certs expired yesterday so connections won't work (the code and message don't come from within certmonger).
certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens.
Easy for you to say. You know what you're doing :-) For me it's all magic.
Anyway, I'll try it. I'm just scared to set the clock back, because there may be clients in the network that use this server as a NTP server.
Another thing I want to mention is that the error started showing up two days ago, on Oct 22, while the expiration is today, Oct 24.
It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
rob
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 25-10-18 14:18, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Could it be that this error already existed since we started? Notice the Request ID of 2016..., and the expires: 2018-10-24.
# getcert list -n ipaCert | sed blabla Number of certificates and requests being tracked: 8. Request ID '20161103094546': status: CA_UNREACHABLE ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=MYDOMAIN subject: CN=IPA RA,O=MYDOMAIN expires: 2018-10-24 08:45:40 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes
In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ?
The problem is your certs expired yesterday so connections won't work (the code and message don't come from within certmonger).
certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens.
Easy for you to say. You know what you're doing :-) For me it's all magic.
Anyway, I'll try it. I'm just scared to set the clock back, because there may be clients in the network that use this server as a NTP server.
Another thing I want to mention is that the error started showing up two days ago, on Oct 22, while the expiration is today, Oct 24.
It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 25-10-18 14:18, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote: > Could it be that this error already existed since we started? Notice > the Request ID of 2016..., and the expires: 2018-10-24. > > # getcert list -n ipaCert | sed blabla > Number of certificates and requests being tracked: 8. > Request ID '20161103094546': > status: CA_UNREACHABLE > ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). > stuck: no > key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' > certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=MYDOMAIN > subject: CN=IPA RA,O=MYDOMAIN > expires: 2018-10-24 08:45:40 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre > post-save command: /usr/lib/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > > In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? The problem is your certs expired yesterday so connections won't work (the code and message don't come from within certmonger).
certmonger _should_ have renewed them. Try killing ntpd, going back a few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and see what happens.
Easy for you to say. You know what you're doing :-) For me it's all magic.
Anyway, I'll try it. I'm just scared to set the clock back, because there may be clients in the network that use this server as a NTP server.
Another thing I want to mention is that the error started showing up two days ago, on Oct 22, while the expiration is today, Oct 24.
It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 25-10-18 14:18, Rob Crittenden wrote: > Kees Bakker via FreeIPA-users wrote: >> Could it be that this error already existed since we started? Notice >> the Request ID of 2016..., and the expires: 2018-10-24. >> >> # getcert list -n ipaCert | sed blabla >> Number of certificates and requests being tracked: 8. >> Request ID '20161103094546': >> status: CA_UNREACHABLE >> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >> stuck: no >> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=MYDOMAIN >> subject: CN=IPA RA,O=MYDOMAIN >> expires: 2018-10-24 08:45:40 UTC >> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >> track: yes >> auto-renew: yes >> >> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? > The problem is your certs expired yesterday so connections won't work > (the code and message don't come from within certmonger). > > certmonger _should_ have renewed them. Try killing ntpd, going back a > few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and > see what happens. > Easy for you to say. You know what you're doing :-) For me it's all magic.
Anyway, I'll try it. I'm just scared to set the clock back, because there may be clients in the network that use this server as a NTP server.
Another thing I want to mention is that the error started showing up two days ago, on Oct 22, while the expiration is today, Oct 24.
It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote: > On 25-10-18 14:18, Rob Crittenden wrote: >> Kees Bakker via FreeIPA-users wrote: >>> Could it be that this error already existed since we started? Notice >>> the Request ID of 2016..., and the expires: 2018-10-24. >>> >>> # getcert list -n ipaCert | sed blabla >>> Number of certificates and requests being tracked: 8. >>> Request ID '20161103094546': >>> status: CA_UNREACHABLE >>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>> stuck: no >>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>> CA: dogtag-ipa-ca-renew-agent >>> issuer: CN=Certificate Authority,O=MYDOMAIN >>> subject: CN=IPA RA,O=MYDOMAIN >>> expires: 2018-10-24 08:45:40 UTC >>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>> track: yes >>> auto-renew: yes >>> >>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >> The problem is your certs expired yesterday so connections won't work >> (the code and message don't come from within certmonger). >> >> certmonger _should_ have renewed them. Try killing ntpd, going back a >> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >> see what happens. >> > Easy for you to say. You know what you're doing :-) > For me it's all magic. > > Anyway, I'll try it. I'm just scared to set the clock back, because there may > be clients in the network that use this server as a NTP server. > > Another thing I want to mention is that the error started showing up two days > ago, on Oct 22, while the expiration is today, Oct 24. > It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I'm on Ubuntu 16.04. The PPA has no packages for xenial. Err:5 http://ppa.launchpad.net/freeipa/staging/ubuntu xenial Release 404 Not Found
On 26.10.2018 18.30, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote: > Kees Bakker via FreeIPA-users wrote: >> On 25-10-18 14:18, Rob Crittenden wrote: >>> Kees Bakker via FreeIPA-users wrote: >>>> Could it be that this error already existed since we started? Notice >>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>> >>>> # getcert list -n ipaCert | sed blabla >>>> Number of certificates and requests being tracked: 8. >>>> Request ID '20161103094546': >>>> status: CA_UNREACHABLE >>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>> CA: dogtag-ipa-ca-renew-agent >>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>> subject: CN=IPA RA,O=MYDOMAIN >>>> expires: 2018-10-24 08:45:40 UTC >>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>> track: yes >>>> auto-renew: yes >>>> >>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>> The problem is your certs expired yesterday so connections won't work >>> (the code and message don't come from within certmonger). >>> >>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>> see what happens. >>> >> Easy for you to say. You know what you're doing :-) >> For me it's all magic. >> >> Anyway, I'll try it. I'm just scared to set the clock back, because there may >> be clients in the network that use this server as a NTP server. >> >> Another thing I want to mention is that the error started showing up two days >> ago, on Oct 22, while the expiration is today, Oct 24. >> > It shouldn't take more than a few minutes to roll back time, restart > services and see what happens. I think your NTP clients will be able to > recover ok if the server is not available for a few minutes. > > certmonger logs to syslog so you probably want to look at that to see if > you can find a reason the certs weren't renewed automatically. > No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I'm on Ubuntu 16.04. The PPA has no packages for xenial. Err:5 http://ppa.launchpad.net/freeipa/staging/ubuntu xenial Release 404 Not Found
That's why I said "from 18.04"
On 26-10-18 17:33, Timo Aaltonen wrote:
On 26.10.2018 18.30, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote: > On 25-10-18 16:11, Rob Crittenden wrote: >> Kees Bakker via FreeIPA-users wrote: >>> On 25-10-18 14:18, Rob Crittenden wrote: >>>> Kees Bakker via FreeIPA-users wrote: >>>>> Could it be that this error already existed since we started? Notice >>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>> >>>>> # getcert list -n ipaCert | sed blabla >>>>> Number of certificates and requests being tracked: 8. >>>>> Request ID '20161103094546': >>>>> status: CA_UNREACHABLE >>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>> CA: dogtag-ipa-ca-renew-agent >>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>> expires: 2018-10-24 08:45:40 UTC >>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>> track: yes >>>>> auto-renew: yes >>>>> >>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>> The problem is your certs expired yesterday so connections won't work >>>> (the code and message don't come from within certmonger). >>>> >>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>> see what happens. >>>> >>> Easy for you to say. You know what you're doing :-) >>> For me it's all magic. >>> >>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>> be clients in the network that use this server as a NTP server. >>> >>> Another thing I want to mention is that the error started showing up two days >>> ago, on Oct 22, while the expiration is today, Oct 24. >>> >> It shouldn't take more than a few minutes to roll back time, restart >> services and see what happens. I think your NTP clients will be able to >> recover ok if the server is not available for a few minutes. >> >> certmonger logs to syslog so you probably want to look at that to see if >> you can find a reason the certs weren't renewed automatically. >> > No, that didn't help. > And in the syslog there was nothing more than this. (I had to stop the > nameserver because it was spitting out lots of messages.) > > Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed > Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed > Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... > Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. > Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... > Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. > Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile > Review: Problem with the SSL CA cert (path? access rights?). > Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent > Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 > Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). > Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent > Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 > Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). > Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I'm on Ubuntu 16.04. The PPA has no packages for xenial. Err:5 http://ppa.launchpad.net/freeipa/staging/ubuntu xenial Release 404 Not Found
That's why I said "from 18.04"
Ah, OK
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote: > On 25-10-18 14:18, Rob Crittenden wrote: >> Kees Bakker via FreeIPA-users wrote: >>> Could it be that this error already existed since we started? Notice >>> the Request ID of 2016..., and the expires: 2018-10-24. >>> >>> # getcert list -n ipaCert | sed blabla >>> Number of certificates and requests being tracked: 8. >>> Request ID '20161103094546': >>> status: CA_UNREACHABLE >>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>> stuck: no >>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>> CA: dogtag-ipa-ca-renew-agent >>> issuer: CN=Certificate Authority,O=MYDOMAIN >>> subject: CN=IPA RA,O=MYDOMAIN >>> expires: 2018-10-24 08:45:40 UTC >>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>> eku: id-kp-serverAuth,id-kp-clientAuth >>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>> track: yes >>> auto-renew: yes >>> >>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >> The problem is your certs expired yesterday so connections won't work >> (the code and message don't come from within certmonger). >> >> certmonger _should_ have renewed them. Try killing ntpd, going back a >> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >> see what happens. >> > Easy for you to say. You know what you're doing :-) > For me it's all magic. > > Anyway, I'll try it. I'm just scared to set the clock back, because there may > be clients in the network that use this server as a NTP server. > > Another thing I want to mention is that the error started showing up two days > ago, on Oct 22, while the expiration is today, Oct 24. > It shouldn't take more than a few minutes to roll back time, restart services and see what happens. I think your NTP clients will be able to recover ok if the server is not available for a few minutes.
certmonger logs to syslog so you probably want to look at that to see if you can find a reason the certs weren't renewed automatically.
No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
On 26.10.2018 18.59, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote:
On 25-10-18 16:11, Rob Crittenden wrote: > Kees Bakker via FreeIPA-users wrote: >> On 25-10-18 14:18, Rob Crittenden wrote: >>> Kees Bakker via FreeIPA-users wrote: >>>> Could it be that this error already existed since we started? Notice >>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>> >>>> # getcert list -n ipaCert | sed blabla >>>> Number of certificates and requests being tracked: 8. >>>> Request ID '20161103094546': >>>> status: CA_UNREACHABLE >>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>> stuck: no >>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>> CA: dogtag-ipa-ca-renew-agent >>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>> subject: CN=IPA RA,O=MYDOMAIN >>>> expires: 2018-10-24 08:45:40 UTC >>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>> track: yes >>>> auto-renew: yes >>>> >>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>> The problem is your certs expired yesterday so connections won't work >>> (the code and message don't come from within certmonger). >>> >>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>> see what happens. >>> >> Easy for you to say. You know what you're doing :-) >> For me it's all magic. >> >> Anyway, I'll try it. I'm just scared to set the clock back, because there may >> be clients in the network that use this server as a NTP server. >> >> Another thing I want to mention is that the error started showing up two days >> ago, on Oct 22, while the expiration is today, Oct 24. >> > It shouldn't take more than a few minutes to roll back time, restart > services and see what happens. I think your NTP clients will be able to > recover ok if the server is not available for a few minutes. > > certmonger logs to syslog so you probably want to look at that to see if > you can find a reason the certs weren't renewed automatically. > No, that didn't help. And in the syslog there was nothing more than this. (I had to stop the nameserver because it was spitting out lots of messages.)
Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile Review: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?).
Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
systemctl restart ipa?
On 26-10-18 18:00, Timo Aaltonen wrote:
On 26.10.2018 18.59, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote:
Kees Bakker wrote: > On 25-10-18 16:11, Rob Crittenden wrote: >> Kees Bakker via FreeIPA-users wrote: >>> On 25-10-18 14:18, Rob Crittenden wrote: >>>> Kees Bakker via FreeIPA-users wrote: >>>>> Could it be that this error already existed since we started? Notice >>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>> >>>>> # getcert list -n ipaCert | sed blabla >>>>> Number of certificates and requests being tracked: 8. >>>>> Request ID '20161103094546': >>>>> status: CA_UNREACHABLE >>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>> stuck: no >>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>> CA: dogtag-ipa-ca-renew-agent >>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>> expires: 2018-10-24 08:45:40 UTC >>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>> track: yes >>>>> auto-renew: yes >>>>> >>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>> The problem is your certs expired yesterday so connections won't work >>>> (the code and message don't come from within certmonger). >>>> >>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>> see what happens. >>>> >>> Easy for you to say. You know what you're doing :-) >>> For me it's all magic. >>> >>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>> be clients in the network that use this server as a NTP server. >>> >>> Another thing I want to mention is that the error started showing up two days >>> ago, on Oct 22, while the expiration is today, Oct 24. >>> >> It shouldn't take more than a few minutes to roll back time, restart >> services and see what happens. I think your NTP clients will be able to >> recover ok if the server is not available for a few minutes. >> >> certmonger logs to syslog so you probably want to look at that to see if >> you can find a reason the certs weren't renewed automatically. >> > No, that didn't help. > And in the syslog there was nothing more than this. (I had to stop the > nameserver because it was spitting out lots of messages.) > > Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed > Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed > Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... > Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. > Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... > Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. > Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile > Review: Problem with the SSL CA cert (path? access rights?). > Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent > Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 > Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). > Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent > Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 > Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). > Ok, I think I know what is going on. This is Ubuntu which AFAIK still lacks nss-pem. That is probably why it can't connect to renew the certs.
I don't know if there is a workaround. Timo, do you know?
Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
systemctl restart ipa?
I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting. It's getting too late now, time for weekend. I'll give it another try on Monday. Meanwhile I want to point at the changed message. In case that rings a bell for someone.
Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
On 26.10.2018 19.09, Kees Bakker wrote:
On 26-10-18 18:00, Timo Aaltonen wrote:
On 26.10.2018 18.59, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote: > Kees Bakker wrote: >> On 25-10-18 16:11, Rob Crittenden wrote: >>> Kees Bakker via FreeIPA-users wrote: >>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>> Kees Bakker via FreeIPA-users wrote: >>>>>> Could it be that this error already existed since we started? Notice >>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>> >>>>>> # getcert list -n ipaCert | sed blabla >>>>>> Number of certificates and requests being tracked: 8. >>>>>> Request ID '20161103094546': >>>>>> status: CA_UNREACHABLE >>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>> stuck: no >>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>> track: yes >>>>>> auto-renew: yes >>>>>> >>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>> The problem is your certs expired yesterday so connections won't work >>>>> (the code and message don't come from within certmonger). >>>>> >>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>> see what happens. >>>>> >>>> Easy for you to say. You know what you're doing :-) >>>> For me it's all magic. >>>> >>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>> be clients in the network that use this server as a NTP server. >>>> >>>> Another thing I want to mention is that the error started showing up two days >>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>> >>> It shouldn't take more than a few minutes to roll back time, restart >>> services and see what happens. I think your NTP clients will be able to >>> recover ok if the server is not available for a few minutes. >>> >>> certmonger logs to syslog so you probably want to look at that to see if >>> you can find a reason the certs weren't renewed automatically. >>> >> No, that didn't help. >> And in the syslog there was nothing more than this. (I had to stop the >> nameserver because it was spitting out lots of messages.) >> >> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >> Review: Problem with the SSL CA cert (path? access rights?). >> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >> > Ok, I think I know what is going on. This is Ubuntu which AFAIK still > lacks nss-pem. That is probably why it can't connect to renew the certs. > > I don't know if there is a workaround. Timo, do you know? Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
systemctl restart ipa?
I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting.
CA is 'pki-tomcatd', dirsrv is 'dirsrv@REALM' if you want to avoid restarting the whole thing
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:00, Timo Aaltonen wrote:
On 26.10.2018 18.59, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote:
On 25.10.2018 21.44, Rob Crittenden wrote: > Kees Bakker wrote: >> On 25-10-18 16:11, Rob Crittenden wrote: >>> Kees Bakker via FreeIPA-users wrote: >>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>> Kees Bakker via FreeIPA-users wrote: >>>>>> Could it be that this error already existed since we started? Notice >>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>> >>>>>> # getcert list -n ipaCert | sed blabla >>>>>> Number of certificates and requests being tracked: 8. >>>>>> Request ID '20161103094546': >>>>>> status: CA_UNREACHABLE >>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>> stuck: no >>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>> track: yes >>>>>> auto-renew: yes >>>>>> >>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>> The problem is your certs expired yesterday so connections won't work >>>>> (the code and message don't come from within certmonger). >>>>> >>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>> see what happens. >>>>> >>>> Easy for you to say. You know what you're doing :-) >>>> For me it's all magic. >>>> >>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>> be clients in the network that use this server as a NTP server. >>>> >>>> Another thing I want to mention is that the error started showing up two days >>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>> >>> It shouldn't take more than a few minutes to roll back time, restart >>> services and see what happens. I think your NTP clients will be able to >>> recover ok if the server is not available for a few minutes. >>> >>> certmonger logs to syslog so you probably want to look at that to see if >>> you can find a reason the certs weren't renewed automatically. >>> >> No, that didn't help. >> And in the syslog there was nothing more than this. (I had to stop the >> nameserver because it was spitting out lots of messages.) >> >> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >> Review: Problem with the SSL CA cert (path? access rights?). >> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >> > Ok, I think I know what is going on. This is Ubuntu which AFAIK still > lacks nss-pem. That is probably why it can't connect to renew the certs. > > I don't know if there is a workaround. Timo, do you know? Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've never tested cert renewal though.
Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
systemctl restart ipa?
I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting.
And you're right! The CA is pki-tomcatd, so you already restarted it.
It's getting too late now, time for weekend. I'll give it another try on Monday. Meanwhile I want to point at the changed message. In case that rings a bell for someone.
Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
HTH, flo
On 26-10-18 18:20, Florence Blanc-Renaud wrote:
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:00, Timo Aaltonen wrote:
On 26.10.2018 18.59, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
On 25-10-18 20:46, Timo Aaltonen wrote: > On 25.10.2018 21.44, Rob Crittenden wrote: >> Kees Bakker wrote: >>> On 25-10-18 16:11, Rob Crittenden wrote: >>>> Kees Bakker via FreeIPA-users wrote: >>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>> Could it be that this error already existed since we started? Notice >>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>> >>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>> Number of certificates and requests being tracked: 8. >>>>>>> Request ID '20161103094546': >>>>>>> status: CA_UNREACHABLE >>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>> stuck: no >>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>> track: yes >>>>>>> auto-renew: yes >>>>>>> >>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>> The problem is your certs expired yesterday so connections won't work >>>>>> (the code and message don't come from within certmonger). >>>>>> >>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>> see what happens. >>>>>> >>>>> Easy for you to say. You know what you're doing :-) >>>>> For me it's all magic. >>>>> >>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>> be clients in the network that use this server as a NTP server. >>>>> >>>>> Another thing I want to mention is that the error started showing up two days >>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>> >>>> It shouldn't take more than a few minutes to roll back time, restart >>>> services and see what happens. I think your NTP clients will be able to >>>> recover ok if the server is not available for a few minutes. >>>> >>>> certmonger logs to syslog so you probably want to look at that to see if >>>> you can find a reason the certs weren't renewed automatically. >>>> >>> No, that didn't help. >>> And in the syslog there was nothing more than this. (I had to stop the >>> nameserver because it was spitting out lots of messages.) >>> >>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>> Review: Problem with the SSL CA cert (path? access rights?). >>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>> >> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >> lacks nss-pem. That is probably why it can't connect to renew the certs. >> >> I don't know if there is a workaround. Timo, do you know? > Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've > never tested cert renewal though. > Does that mean, I'm screwed? What options do I have? Live with it? Migrate to, say Centos? Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? Something else?
Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
systemctl restart ipa?
I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting.
And you're right! The CA is pki-tomcatd, so you already restarted it.
It's getting too late now, time for weekend. I'll give it another try on Monday. Meanwhile I want to point at the changed message. In case that rings a bell for someone.
Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:20, Florence Blanc-Renaud wrote:
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:00, Timo Aaltonen wrote:
On 26.10.2018 18.59, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote:
On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: > On 25-10-18 20:46, Timo Aaltonen wrote: >> On 25.10.2018 21.44, Rob Crittenden wrote: >>> Kees Bakker wrote: >>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>> Kees Bakker via FreeIPA-users wrote: >>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>> Could it be that this error already existed since we started? Notice >>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>> >>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>> Request ID '20161103094546': >>>>>>>> status: CA_UNREACHABLE >>>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>> stuck: no >>>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>> track: yes >>>>>>>> auto-renew: yes >>>>>>>> >>>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>>> The problem is your certs expired yesterday so connections won't work >>>>>>> (the code and message don't come from within certmonger). >>>>>>> >>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>>> see what happens. >>>>>>> >>>>>> Easy for you to say. You know what you're doing :-) >>>>>> For me it's all magic. >>>>>> >>>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>>> be clients in the network that use this server as a NTP server. >>>>>> >>>>>> Another thing I want to mention is that the error started showing up two days >>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>> >>>>> It shouldn't take more than a few minutes to roll back time, restart >>>>> services and see what happens. I think your NTP clients will be able to >>>>> recover ok if the server is not available for a few minutes. >>>>> >>>>> certmonger logs to syslog so you probably want to look at that to see if >>>>> you can find a reason the certs weren't renewed automatically. >>>>> >>>> No, that didn't help. >>>> And in the syslog there was nothing more than this. (I had to stop the >>>> nameserver because it was spitting out lots of messages.) >>>> >>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>> Review: Problem with the SSL CA cert (path? access rights?). >>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>> >>> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >>> lacks nss-pem. That is probably why it can't connect to renew the certs. >>> >>> I don't know if there is a workaround. Timo, do you know? >> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've >> never tested cert renewal though. >> > Does that mean, I'm screwed? What options do I have? > Live with it? > Migrate to, say Centos? > Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? > Something else? Stock 18.04 has other issues, there's an updated version on ppa:freeipa/staging which is backported from 18.10 and should be fine and hopefully provided as a stable update on 18.04 later on.
But you could try pulling libnsspem from 18.04, and *then* roll back time?
I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
systemctl restart ipa?
I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting.
And you're right! The CA is pki-tomcatd, so you already restarted it.
It's getting too late now, time for weekend. I'll give it another try on Monday. Meanwhile I want to point at the changed message. In case that rings a bell for someone.
Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
I tried it once more to set the date back and to restart the renewal process. I keep getting Error 60 with Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with "... the fix was to reset the NSS trust flags in the Apache NSS database" The curl command (with the date set back) seems to connect but it gives a gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if " You should get client certificate not found." So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it gave a lot of messages in my /var/log/syslog.
Hi Kees, I've been also looking to Rob's blog as part of working on my problem ("ipa.service "fails" to start"). In my case, when running the curl command (with -v), I do see
* About to connect() to ca-ldap03 port 8443 (#0) * Trying x.x.x..x ... * Connected to ca-ldap03 port 8443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/ipa/ca.crt CApath: none * NSS: client certificate not found (nickname not specified)
... but then I see :
< HTTP/1.1 500 Internal Server Error
.. finally command's exit status is 0, hence I understand no need to modify trust flag.
Z D via FreeIPA-users wrote:
Hi Kees, I've been also looking to Rob's blog as part of working on my problem ("ipa.service "fails" to start"). In my case, when running the curl command (with -v), I do see
- About to connect() to ca-ldap03 port 8443 (#0)
- Trying x.x.x..x ...
- Connected to ca-ldap03 port 8443 (#0)
- Initializing NSS with certpath: sql:/etc/pki/nssdb
- CAfile: /etc/ipa/ca.crt CApath: none
- NSS: client certificate not found (nickname not specified)
... but then I see :
< HTTP/1.1 500 Internal Server Error
.. finally command's exit status is 0, hence I understand no need to modify trust flag.
You need to see why the CA is failing to respond to the request with a non-500 response.
rob
Rob, what kind of response means success, one server return 404 ?
GET /ca/agent/ca/profileReview HTTP/1.1 User-Agent: curl/7.29.0 Host: ca-ldap01:8443 Accept: */*
< HTTP/1.1 404 Not Found < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 1001 < Date: Tue, 30 Oct 2018 05:31:38 GMT <
Zarko D via FreeIPA-users wrote:
Rob, what kind of response means success, one server return 404 ?
GET /ca/agent/ca/profileReview HTTP/1.1 User-Agent: curl/7.29.0 Host: ca-ldap01:8443 Accept: */*
< HTTP/1.1 404 Not Found < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 1001 < Date: Tue, 30 Oct 2018 05:31:38 GMT
I need to see the full command you executed. It very well could mean that the CA is not running (the debug and/or selftest log will tell you).
rob
Kees Bakker via FreeIPA-users wrote:
On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:20, Florence Blanc-Renaud wrote:
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:00, Timo Aaltonen wrote:
On 26.10.2018 18.59, Kees Bakker wrote:
On 26-10-18 14:55, Timo Aaltonen wrote: > On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: >> On 25-10-18 20:46, Timo Aaltonen wrote: >>> On 25.10.2018 21.44, Rob Crittenden wrote: >>>> Kees Bakker wrote: >>>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>> Could it be that this error already existed since we started? Notice >>>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>>> >>>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>>> Request ID '20161103094546': >>>>>>>>> status: CA_UNREACHABLE >>>>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>> stuck: no >>>>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>>> track: yes >>>>>>>>> auto-renew: yes >>>>>>>>> >>>>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>>>> The problem is your certs expired yesterday so connections won't work >>>>>>>> (the code and message don't come from within certmonger). >>>>>>>> >>>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>>>> see what happens. >>>>>>>> >>>>>>> Easy for you to say. You know what you're doing :-) >>>>>>> For me it's all magic. >>>>>>> >>>>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>>>> be clients in the network that use this server as a NTP server. >>>>>>> >>>>>>> Another thing I want to mention is that the error started showing up two days >>>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>>> >>>>>> It shouldn't take more than a few minutes to roll back time, restart >>>>>> services and see what happens. I think your NTP clients will be able to >>>>>> recover ok if the server is not available for a few minutes. >>>>>> >>>>>> certmonger logs to syslog so you probably want to look at that to see if >>>>>> you can find a reason the certs weren't renewed automatically. >>>>>> >>>>> No, that didn't help. >>>>> And in the syslog there was nothing more than this. (I had to stop the >>>>> nameserver because it was spitting out lots of messages.) >>>>> >>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>>> Review: Problem with the SSL CA cert (path? access rights?). >>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>> >>>> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >>>> lacks nss-pem. That is probably why it can't connect to renew the certs. >>>> >>>> I don't know if there is a workaround. Timo, do you know? >>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've >>> never tested cert renewal though. >>> >> Does that mean, I'm screwed? What options do I have? >> Live with it? >> Migrate to, say Centos? >> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? >> Something else? > Stock 18.04 has other issues, there's an updated version on > ppa:freeipa/staging which is backported from 18.10 and should be fine > and hopefully provided as a stable update on 18.04 later on. > > But you could try pulling libnsspem from 18.04, and *then* roll back time? > I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
Then I stopped ntp (and bind). Set the time back to Oct 11 Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger (in that order).
Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
:-(
Rob said also to restart CA. "restart krb5kdc, dirsrv, httpd and the CA then certmonger" I don't know which service that is. Does that matter?
systemctl restart ipa?
I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting.
And you're right! The CA is pki-tomcatd, so you already restarted it.
It's getting too late now, time for weekend. I'll give it another try on Monday. Meanwhile I want to point at the changed message. In case that rings a bell for someone.
Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
I tried it once more to set the date back and to restart the renewal process. I keep getting Error 60 with Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with "... the fix was to reset the NSS trust flags in the Apache NSS database" The curl command (with the date set back) seems to connect but it gives a gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if " You should get client certificate not found." So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it gave a lot of messages in my /var/log/syslog.
Right, this is for a previous version of IPA on RHEL/Fedora. The RA agent cert used to be stored in the Apache NSS database.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid back in time?
rob
On 29-10-18 19:30, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:20, Florence Blanc-Renaud wrote:
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:00, Timo Aaltonen wrote:
On 26.10.2018 18.59, Kees Bakker wrote: > On 26-10-18 14:55, Timo Aaltonen wrote: >> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: >>> On 25-10-18 20:46, Timo Aaltonen wrote: >>>> On 25.10.2018 21.44, Rob Crittenden wrote: >>>>> Kees Bakker wrote: >>>>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>> Could it be that this error already existed since we started? Notice >>>>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>>>> >>>>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>>>> Request ID '20161103094546': >>>>>>>>>> status: CA_UNREACHABLE >>>>>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>>> stuck: no >>>>>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>>>> track: yes >>>>>>>>>> auto-renew: yes >>>>>>>>>> >>>>>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>>>>> The problem is your certs expired yesterday so connections won't work >>>>>>>>> (the code and message don't come from within certmonger). >>>>>>>>> >>>>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>>>>> see what happens. >>>>>>>>> >>>>>>>> Easy for you to say. You know what you're doing :-) >>>>>>>> For me it's all magic. >>>>>>>> >>>>>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>>>>> be clients in the network that use this server as a NTP server. >>>>>>>> >>>>>>>> Another thing I want to mention is that the error started showing up two days >>>>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>>>> >>>>>>> It shouldn't take more than a few minutes to roll back time, restart >>>>>>> services and see what happens. I think your NTP clients will be able to >>>>>>> recover ok if the server is not available for a few minutes. >>>>>>> >>>>>>> certmonger logs to syslog so you probably want to look at that to see if >>>>>>> you can find a reason the certs weren't renewed automatically. >>>>>>> >>>>>> No, that didn't help. >>>>>> And in the syslog there was nothing more than this. (I had to stop the >>>>>> nameserver because it was spitting out lots of messages.) >>>>>> >>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>>>> Review: Problem with the SSL CA cert (path? access rights?). >>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>> >>>>> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >>>>> lacks nss-pem. That is probably why it can't connect to renew the certs. >>>>> >>>>> I don't know if there is a workaround. Timo, do you know? >>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've >>>> never tested cert renewal though. >>>> >>> Does that mean, I'm screwed? What options do I have? >>> Live with it? >>> Migrate to, say Centos? >>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? >>> Something else? >> Stock 18.04 has other issues, there's an updated version on >> ppa:freeipa/staging which is backported from 18.10 and should be fine >> and hopefully provided as a stable update on 18.04 later on. >> >> But you could try pulling libnsspem from 18.04, and *then* roll back time? >> > I installed libnsspem_1.0.3-0ubuntu2_amd64.deb > > Then I stopped ntp (and bind). > Set the time back to Oct 11 > Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger > (in that order). > > Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent > Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 > Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. > Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. > > :-( > > Rob said also to restart CA. > "restart krb5kdc, dirsrv, httpd and the CA then certmonger" > I don't know which service that is. Does that matter? systemctl restart ipa?
I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting.
And you're right! The CA is pki-tomcatd, so you already restarted it.
It's getting too late now, time for weekend. I'll give it another try on Monday. Meanwhile I want to point at the changed message. In case that rings a bell for someone.
Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
I tried it once more to set the date back and to restart the renewal process. I keep getting Error 60 with Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with "... the fix was to reset the NSS trust flags in the Apache NSS database" The curl command (with the date set back) seems to connect but it gives a gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if " You should get client certificate not found." So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it gave a lot of messages in my /var/log/syslog.
Right, this is for a previous version of IPA on RHEL/Fedora. The RA agent cert used to be stored in the Apache NSS database.
That's a problem in general with notes and blogs. When you write things you don't know what to write about possible validness in the future.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid back in time?
Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA' certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
Validity: Not Before: Thu Nov 03 09:45:18 2016 Not After : Mon Nov 03 09:45:18 2036
Kees Bakker wrote:
On 29-10-18 19:30, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:20, Florence Blanc-Renaud wrote:
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:00, Timo Aaltonen wrote: > On 26.10.2018 18.59, Kees Bakker wrote: >> On 26-10-18 14:55, Timo Aaltonen wrote: >>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: >>>> On 25-10-18 20:46, Timo Aaltonen wrote: >>>>> On 25.10.2018 21.44, Rob Crittenden wrote: >>>>>> Kees Bakker wrote: >>>>>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>>> Could it be that this error already existed since we started? Notice >>>>>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>>>>> >>>>>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>>>>> Request ID '20161103094546': >>>>>>>>>>> status: CA_UNREACHABLE >>>>>>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>>>> stuck: no >>>>>>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>>>>> track: yes >>>>>>>>>>> auto-renew: yes >>>>>>>>>>> >>>>>>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>>>>>> The problem is your certs expired yesterday so connections won't work >>>>>>>>>> (the code and message don't come from within certmonger). >>>>>>>>>> >>>>>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>>>>>> see what happens. >>>>>>>>>> >>>>>>>>> Easy for you to say. You know what you're doing :-) >>>>>>>>> For me it's all magic. >>>>>>>>> >>>>>>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>>>>>> be clients in the network that use this server as a NTP server. >>>>>>>>> >>>>>>>>> Another thing I want to mention is that the error started showing up two days >>>>>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>>>>> >>>>>>>> It shouldn't take more than a few minutes to roll back time, restart >>>>>>>> services and see what happens. I think your NTP clients will be able to >>>>>>>> recover ok if the server is not available for a few minutes. >>>>>>>> >>>>>>>> certmonger logs to syslog so you probably want to look at that to see if >>>>>>>> you can find a reason the certs weren't renewed automatically. >>>>>>>> >>>>>>> No, that didn't help. >>>>>>> And in the syslog there was nothing more than this. (I had to stop the >>>>>>> nameserver because it was spitting out lots of messages.) >>>>>>> >>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>>>>> Review: Problem with the SSL CA cert (path? access rights?). >>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>> >>>>>> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >>>>>> lacks nss-pem. That is probably why it can't connect to renew the certs. >>>>>> >>>>>> I don't know if there is a workaround. Timo, do you know? >>>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've >>>>> never tested cert renewal though. >>>>> >>>> Does that mean, I'm screwed? What options do I have? >>>> Live with it? >>>> Migrate to, say Centos? >>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? >>>> Something else? >>> Stock 18.04 has other issues, there's an updated version on >>> ppa:freeipa/staging which is backported from 18.10 and should be fine >>> and hopefully provided as a stable update on 18.04 later on. >>> >>> But you could try pulling libnsspem from 18.04, and *then* roll back time? >>> >> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb >> >> Then I stopped ntp (and bind). >> Set the time back to Oct 11 >> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger >> (in that order). >> >> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >> >> :-( >> >> Rob said also to restart CA. >> "restart krb5kdc, dirsrv, httpd and the CA then certmonger" >> I don't know which service that is. Does that matter? > systemctl restart ipa? > > I'm a bit scared to restart service ipa, because it also restarts several other services, link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control of not starting.
And you're right! The CA is pki-tomcatd, so you already restarted it.
It's getting too late now, time for weekend. I'll give it another try on Monday. Meanwhile I want to point at the changed message. In case that rings a bell for someone.
Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
I tried it once more to set the date back and to restart the renewal process. I keep getting Error 60 with Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with "... the fix was to reset the NSS trust flags in the Apache NSS database" The curl command (with the date set back) seems to connect but it gives a gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if " You should get client certificate not found." So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it gave a lot of messages in my /var/log/syslog.
Right, this is for a previous version of IPA on RHEL/Fedora. The RA agent cert used to be stored in the Apache NSS database.
That's a problem in general with notes and blogs. When you write things you don't know what to write about possible validness in the future.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid back in time?
Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA' certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
Validity: Not Before: Thu Nov 03 09:45:18 2016 Not After : Mon Nov 03 09:45:18 2036
Ok I'm just flying blind right now, reading curl code, but in the gnutls code it throws this error if the server cert validation fails.
I don't know where gnutls reads its CA trust from or whether it honors the system-wide trust, or even if your IPA CA is in the global trust properly.
When back in time do IPA operations work? ipa user-show admin, ipa cert-show 1, etc?
rob
On 30-10-18 19:41, Rob Crittenden wrote:
Kees Bakker wrote:
On 29-10-18 19:30, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:20, Florence Blanc-Renaud wrote:
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote: > On 26-10-18 18:00, Timo Aaltonen wrote: >> On 26.10.2018 18.59, Kees Bakker wrote: >>> On 26-10-18 14:55, Timo Aaltonen wrote: >>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: >>>>> On 25-10-18 20:46, Timo Aaltonen wrote: >>>>>> On 25.10.2018 21.44, Rob Crittenden wrote: >>>>>>> Kees Bakker wrote: >>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>>>> Could it be that this error already existed since we started? Notice >>>>>>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>>>>>> >>>>>>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>>>>>> Request ID '20161103094546': >>>>>>>>>>>> status: CA_UNREACHABLE >>>>>>>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>>>>> stuck: no >>>>>>>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>>>>>> track: yes >>>>>>>>>>>> auto-renew: yes >>>>>>>>>>>> >>>>>>>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>>>>>>> The problem is your certs expired yesterday so connections won't work >>>>>>>>>>> (the code and message don't come from within certmonger). >>>>>>>>>>> >>>>>>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>>>>>>> see what happens. >>>>>>>>>>> >>>>>>>>>> Easy for you to say. You know what you're doing :-) >>>>>>>>>> For me it's all magic. >>>>>>>>>> >>>>>>>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>>>>>>> be clients in the network that use this server as a NTP server. >>>>>>>>>> >>>>>>>>>> Another thing I want to mention is that the error started showing up two days >>>>>>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>>>>>> >>>>>>>>> It shouldn't take more than a few minutes to roll back time, restart >>>>>>>>> services and see what happens. I think your NTP clients will be able to >>>>>>>>> recover ok if the server is not available for a few minutes. >>>>>>>>> >>>>>>>>> certmonger logs to syslog so you probably want to look at that to see if >>>>>>>>> you can find a reason the certs weren't renewed automatically. >>>>>>>>> >>>>>>>> No, that didn't help. >>>>>>>> And in the syslog there was nothing more than this. (I had to stop the >>>>>>>> nameserver because it was spitting out lots of messages.) >>>>>>>> >>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>>>>>> Review: Problem with the SSL CA cert (path? access rights?). >>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>> >>>>>>> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >>>>>>> lacks nss-pem. That is probably why it can't connect to renew the certs. >>>>>>> >>>>>>> I don't know if there is a workaround. Timo, do you know? >>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've >>>>>> never tested cert renewal though. >>>>>> >>>>> Does that mean, I'm screwed? What options do I have? >>>>> Live with it? >>>>> Migrate to, say Centos? >>>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? >>>>> Something else? >>>> Stock 18.04 has other issues, there's an updated version on >>>> ppa:freeipa/staging which is backported from 18.10 and should be fine >>>> and hopefully provided as a stable update on 18.04 later on. >>>> >>>> But you could try pulling libnsspem from 18.04, and *then* roll back time? >>>> >>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb >>> >>> Then I stopped ntp (and bind). >>> Set the time back to Oct 11 >>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger >>> (in that order). >>> >>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >>> >>> :-( >>> >>> Rob said also to restart CA. >>> "restart krb5kdc, dirsrv, httpd and the CA then certmonger" >>> I don't know which service that is. Does that matter? >> systemctl restart ipa? >> >> > I'm a bit scared to restart service ipa, because it also restarts several other services, > link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control > of not starting. And you're right! The CA is pki-tomcatd, so you already restarted it.
> It's getting too late now, time for weekend. I'll give it another try on Monday. > Meanwhile I want to point at the changed message. In case that rings a bell for > someone. > > Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. > You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
I tried it once more to set the date back and to restart the renewal process. I keep getting Error 60 with Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with "... the fix was to reset the NSS trust flags in the Apache NSS database" The curl command (with the date set back) seems to connect but it gives a gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if " You should get client certificate not found." So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it gave a lot of messages in my /var/log/syslog.
Right, this is for a previous version of IPA on RHEL/Fedora. The RA agent cert used to be stored in the Apache NSS database.
That's a problem in general with notes and blogs. When you write things you don't know what to write about possible validness in the future.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid back in time?
Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA' certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
Validity: Not Before: Thu Nov 03 09:45:18 2016 Not After : Mon Nov 03 09:45:18 2036
Ok I'm just flying blind right now, reading curl code, but in the gnutls code it throws this error if the server cert validation fails.
I don't know where gnutls reads its CA trust from or whether it honors the system-wide trust, or even if your IPA CA is in the global trust properly.
When back in time do IPA operations work? ipa user-show admin, ipa cert-show 1, etc?
I'll do that maybe later this week. I don't want to mess with the IPA server while it is still working (sort of).
Meanwhile, here is a new question, a different approach. Can I manually generate a new certificate? For example the "Server-Cert cert-pki-ca" of the pki-tomcat server. And if so, do you have a suggestion about the commands to use?
On 31-10-18 14:27, Kees Bakker via FreeIPA-users wrote:
On 30-10-18 19:41, Rob Crittenden wrote:
Kees Bakker wrote:
On 29-10-18 19:30, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:20, Florence Blanc-Renaud wrote: > On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote: >> On 26-10-18 18:00, Timo Aaltonen wrote: >>> On 26.10.2018 18.59, Kees Bakker wrote: >>>> On 26-10-18 14:55, Timo Aaltonen wrote: >>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: >>>>>> On 25-10-18 20:46, Timo Aaltonen wrote: >>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote: >>>>>>>> Kees Bakker wrote: >>>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>>>>> Could it be that this error already existed since we started? Notice >>>>>>>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>>>>>>> >>>>>>>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>>>>>>> Request ID '20161103094546': >>>>>>>>>>>>> status: CA_UNREACHABLE >>>>>>>>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>>>>>> stuck: no >>>>>>>>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>>>>>>> track: yes >>>>>>>>>>>>> auto-renew: yes >>>>>>>>>>>>> >>>>>>>>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>>>>>>>> The problem is your certs expired yesterday so connections won't work >>>>>>>>>>>> (the code and message don't come from within certmonger). >>>>>>>>>>>> >>>>>>>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>>>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>>>>>>>> see what happens. >>>>>>>>>>>> >>>>>>>>>>> Easy for you to say. You know what you're doing :-) >>>>>>>>>>> For me it's all magic. >>>>>>>>>>> >>>>>>>>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>>>>>>>> be clients in the network that use this server as a NTP server. >>>>>>>>>>> >>>>>>>>>>> Another thing I want to mention is that the error started showing up two days >>>>>>>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>>>>>>> >>>>>>>>>> It shouldn't take more than a few minutes to roll back time, restart >>>>>>>>>> services and see what happens. I think your NTP clients will be able to >>>>>>>>>> recover ok if the server is not available for a few minutes. >>>>>>>>>> >>>>>>>>>> certmonger logs to syslog so you probably want to look at that to see if >>>>>>>>>> you can find a reason the certs weren't renewed automatically. >>>>>>>>>> >>>>>>>>> No, that didn't help. >>>>>>>>> And in the syslog there was nothing more than this. (I had to stop the >>>>>>>>> nameserver because it was spitting out lots of messages.) >>>>>>>>> >>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>>>>>>> Review: Problem with the SSL CA cert (path? access rights?). >>>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>> >>>>>>>> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >>>>>>>> lacks nss-pem. That is probably why it can't connect to renew the certs. >>>>>>>> >>>>>>>> I don't know if there is a workaround. Timo, do you know? >>>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've >>>>>>> never tested cert renewal though. >>>>>>> >>>>>> Does that mean, I'm screwed? What options do I have? >>>>>> Live with it? >>>>>> Migrate to, say Centos? >>>>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? >>>>>> Something else? >>>>> Stock 18.04 has other issues, there's an updated version on >>>>> ppa:freeipa/staging which is backported from 18.10 and should be fine >>>>> and hopefully provided as a stable update on 18.04 later on. >>>>> >>>>> But you could try pulling libnsspem from 18.04, and *then* roll back time? >>>>> >>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb >>>> >>>> Then I stopped ntp (and bind). >>>> Set the time back to Oct 11 >>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger >>>> (in that order). >>>> >>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >>>> >>>> :-( >>>> >>>> Rob said also to restart CA. >>>> "restart krb5kdc, dirsrv, httpd and the CA then certmonger" >>>> I don't know which service that is. Does that matter? >>> systemctl restart ipa? >>> >>> >> I'm a bit scared to restart service ipa, because it also restarts several other services, >> link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control >> of not starting. > And you're right! The CA is pki-tomcatd, so you already restarted it. > >> It's getting too late now, time for weekend. I'll give it another try on Monday. >> Meanwhile I want to point at the changed message. In case that rings a bell for >> someone. >> >> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >> > You can have a look at Rob's blog for additional items to check: > https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent... Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
I tried it once more to set the date back and to restart the renewal process. I keep getting Error 60 with Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with "... the fix was to reset the NSS trust flags in the Apache NSS database" The curl command (with the date set back) seems to connect but it gives a gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if " You should get client certificate not found." So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it gave a lot of messages in my /var/log/syslog.
Right, this is for a previous version of IPA on RHEL/Fedora. The RA agent cert used to be stored in the Apache NSS database.
That's a problem in general with notes and blogs. When you write things you don't know what to write about possible validness in the future.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid back in time?
Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA' certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
Validity: Not Before: Thu Nov 03 09:45:18 2016 Not After : Mon Nov 03 09:45:18 2036
Ok I'm just flying blind right now, reading curl code, but in the gnutls code it throws this error if the server cert validation fails.
I don't know where gnutls reads its CA trust from or whether it honors the system-wide trust, or even if your IPA CA is in the global trust properly.
When back in time do IPA operations work? ipa user-show admin, ipa cert-show 1, etc?
I'll do that maybe later this week. I don't want to mess with the IPA server while it is still working (sort of).
Meanwhile, here is a new question, a different approach. Can I manually generate a new certificate? For example the "Server-Cert cert-pki-ca" of the pki-tomcat server. And if so, do you have a suggestion about the commands to use?
Since nobody told me it was a bad idea, I went ahead and created a new Cert for the pki-tomcat. It was not easy, but now I have a new cert that is valid. If I check it with openssl.
# openssl s_client -showcerts -connect `hostname`:8443 < /dev/null CONNECTED(00000003) depth=1 O = GHS.NL, CN = Certificate Authority verify return:1 depth=0 O = GHS.NL, CN = rotte.ghs.nl verify return:1 ... Verify return code: 0 (ok)
The curl test command still fails
# SSL_DIR=/etc/apache2/nssdb/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://%60hostname%60:8443/ca/agent/ca/profileReview * Connected to rotte.ghs.nl (172.16.16.75) port 8443 (#0) * found 1 certificates in /etc/ipa/ca.crt * found 616 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / RSA_AES_128_CBC_SHA1 * server certificate verification OK * server certificate status verification SKIPPED * common name: rotte.ghs.nl (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: O=GHS.NL,CN=rotte.ghs.nl * start date: Thu, 01 Nov 2018 10:48:12 GMT * expire date: Mon, 01 Nov 2021 10:48:12 GMT * issuer: O=GHS.NL,CN=Certificate Authority * compression: NULL * ALPN, server did not agree to a protocol
GET /ca/agent/ca/profileReview HTTP/1.1 Host: rotte.ghs.nl:8443 User-Agent: curl/7.47.0 Accept: */*
< HTTP/1.1 500 Internal Server Error < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 2161 < Date: Thu, 01 Nov 2018 11:08:53 GMT < Connection: close < { [2161 bytes data] 100 2161 100 2161 0 0 18051 0 --:--:-- --:--:-- --:--:-- 18159 * Closing connection 0
Certmonger still gives that Error 60 message
Nov 1 13:28:23 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent Nov 1 13:28:23 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 Nov 1 13:28:23 ipasrv certmonger[41919]: 2018-11-01 13:28:23 [41919] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates.
Are there log files to be looked at to see what that "Internal Server Error" is?
On 30-10-18 19:41, Rob Crittenden wrote:
Kees Bakker wrote:
On 29-10-18 19:30, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
On 26-10-18 18:20, Florence Blanc-Renaud wrote:
On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote: > On 26-10-18 18:00, Timo Aaltonen wrote: >> On 26.10.2018 18.59, Kees Bakker wrote: >>> On 26-10-18 14:55, Timo Aaltonen wrote: >>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote: >>>>> On 25-10-18 20:46, Timo Aaltonen wrote: >>>>>> On 25.10.2018 21.44, Rob Crittenden wrote: >>>>>>> Kees Bakker wrote: >>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote: >>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote: >>>>>>>>>>> Kees Bakker via FreeIPA-users wrote: >>>>>>>>>>>> Could it be that this error already existed since we started? Notice >>>>>>>>>>>> the Request ID of 2016..., and the expires: 2018-10-24. >>>>>>>>>>>> >>>>>>>>>>>> # getcert list -n ipaCert | sed blabla >>>>>>>>>>>> Number of certificates and requests being tracked: 8. >>>>>>>>>>>> Request ID '20161103094546': >>>>>>>>>>>> status: CA_UNREACHABLE >>>>>>>>>>>> ca-error: Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>>>>>> stuck: no >>>>>>>>>>>> key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' >>>>>>>>>>>> certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' >>>>>>>>>>>> CA: dogtag-ipa-ca-renew-agent >>>>>>>>>>>> issuer: CN=Certificate Authority,O=MYDOMAIN >>>>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN >>>>>>>>>>>> expires: 2018-10-24 08:45:40 UTC >>>>>>>>>>>> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>>>>>>>>>>> eku: id-kp-serverAuth,id-kp-clientAuth >>>>>>>>>>>> pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre >>>>>>>>>>>> post-save command: /usr/lib/ipa/certmonger/renew_ra_cert >>>>>>>>>>>> track: yes >>>>>>>>>>>> auto-renew: yes >>>>>>>>>>>> >>>>>>>>>>>> In other words, is this the same issue as https://pagure.io/freeipa/issue/7422 ? >>>>>>>>>>> The problem is your certs expired yesterday so connections won't work >>>>>>>>>>> (the code and message don't come from within certmonger). >>>>>>>>>>> >>>>>>>>>>> certmonger _should_ have renewed them. Try killing ntpd, going back a >>>>>>>>>>> few days, restart krb5kdc, dirsrv, httpd and the CA then certmonger and >>>>>>>>>>> see what happens. >>>>>>>>>>> >>>>>>>>>> Easy for you to say. You know what you're doing :-) >>>>>>>>>> For me it's all magic. >>>>>>>>>> >>>>>>>>>> Anyway, I'll try it. I'm just scared to set the clock back, because there may >>>>>>>>>> be clients in the network that use this server as a NTP server. >>>>>>>>>> >>>>>>>>>> Another thing I want to mention is that the error started showing up two days >>>>>>>>>> ago, on Oct 22, while the expiration is today, Oct 24. >>>>>>>>>> >>>>>>>>> It shouldn't take more than a few minutes to roll back time, restart >>>>>>>>> services and see what happens. I think your NTP clients will be able to >>>>>>>>> recover ok if the server is not available for a few minutes. >>>>>>>>> >>>>>>>>> certmonger logs to syslog so you probably want to look at that to see if >>>>>>>>> you can find a reason the certs weren't renewed automatically. >>>>>>>>> >>>>>>>> No, that didn't help. >>>>>>>> And in the syslog there was nothing more than this. (I had to stop the >>>>>>>> nameserver because it was spitting out lots of messages.) >>>>>>>> >>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed >>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been changed >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate monitoring and PKI enrollment... >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate monitoring and PKI enrollment. >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate monitoring and PKI enrollment... >>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate monitoring and PKI enrollment. >>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profile >>>>>>>> Review: Problem with the SSL CA cert (path? access rights?). >>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access rights?). >>>>>>>> >>>>>>> Ok, I think I know what is going on. This is Ubuntu which AFAIK still >>>>>>> lacks nss-pem. That is probably why it can't connect to renew the certs. >>>>>>> >>>>>>> I don't know if there is a workaround. Timo, do you know? >>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it. I've >>>>>> never tested cert renewal though. >>>>>> >>>>> Does that mean, I'm screwed? What options do I have? >>>>> Live with it? >>>>> Migrate to, say Centos? >>>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether it will work)? >>>>> Something else? >>>> Stock 18.04 has other issues, there's an updated version on >>>> ppa:freeipa/staging which is backported from 18.10 and should be fine >>>> and hopefully provided as a stable update on 18.04 later on. >>>> >>>> But you could try pulling libnsspem from 18.04, and *then* roll back time? >>>> >>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb >>> >>> Then I stopped ntp (and bind). >>> Set the time back to Oct 11 >>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger >>> (in that order). >>> >>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent >>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3 >>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. >>> >>> :-( >>> >>> Rob said also to restart CA. >>> "restart krb5kdc, dirsrv, httpd and the CA then certmonger" >>> I don't know which service that is. Does that matter? >> systemctl restart ipa? >> >> > I'm a bit scared to restart service ipa, because it also restarts several other services, > link bind, and perhaps ntp. The latter is the one that I want to be absolutely in control > of not starting. And you're right! The CA is pki-tomcatd, so you already restarted it.
> It's getting too late now, time for weekend. I'll give it another try on Monday. > Meanwhile I want to point at the changed message. In case that rings a bell for > someone. > > Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60 connecting to https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. > You can have a look at Rob's blog for additional items to check: https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-authent...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite understand all details.
I really need some guidance what to do next. I tried the date trick, I installed libnsspem (from Ubunu 18.04). The certmonger error message changed from Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background process java.lang.NullPointerException at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last time?
I tried it once more to set the date back and to restart the renewal process. I keep getting Error 60 with Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with "... the fix was to reset the NSS trust flags in the Apache NSS database" The curl command (with the date set back) seems to connect but it gives a gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if " You should get client certificate not found." So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it gave a lot of messages in my /var/log/syslog.
Right, this is for a previous version of IPA on RHEL/Fedora. The RA agent cert used to be stored in the Apache NSS database.
That's a problem in general with notes and blogs. When you write things you don't know what to write about possible validness in the future.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid back in time?
Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca' certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA' certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
Validity: Not Before: Thu Nov 03 09:45:18 2016 Not After : Mon Nov 03 09:45:18 2036
Ok I'm just flying blind right now, reading curl code, but in the gnutls code it throws this error if the server cert validation fails.
I don't know where gnutls reads its CA trust from or whether it honors the system-wide trust, or even if your IPA CA is in the global trust properly.
When back in time do IPA operations work? ipa user-show admin, ipa cert-show 1, etc?
rob
ipa user-show admin works OK, even with today's date.
ipa cert-show 1 works OK when the date is set back. With today's date it gives:
ipa: ERROR: cannot connect to 'https://ipasrv.mydomain:443/ca/agent/ca/displayBySerial': (SSL_ERROR_EXPIRED_CERT_ALERT) SSL peer rejected your certificate as expired.
Meanwhile I have setup a test system with a copy of the master. Disconnected from the network, of course. This gives me more freedom to tryout things (e.g. setting the date back).
If nobody has anymore suggestions then I have no other choice then to start all over. Sad, but that's the way it is.
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Kees Bakker -
Rob Crittenden -
Timo Aaltonen -
Z D -
Zarko D