Hi,
We recently renewed our IPA CA cert using the "ipa-cacert-manage renew” command. The renewal was successful, and our CA cert no longer expires in 2020, but in 2040.
Running “ipa-certupdate” on existing IPA clients and ipa-client-install on new IPA clients also works, however both the new and the old CA cert is pulled down to the IPA client and stored in /etc/ipa/ca.crt.
This creates some issues as most applications reading /etc/ipa/ca.crt only reads the first entry, which happens to be the old CA cert.
For the moment everything works OK as the old CA cert is still valid, however this will become a major issue in a few months time.
Is this expected? To continue to service both the old and the new CA certificate to old and new IPA clients? Will the old certificate be automatically removed at some point? If not, what is the safe steps to remove the old CA certificate from the IPA servers?
Thanks.
Regards, Siggi
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi,
We recently renewed our IPA CA cert using the "ipa-cacert-manage renew” command. The renewal was successful, and our CA cert no longer expires in 2020, but in 2040.
Running “ipa-certupdate” on existing IPA clients and ipa-client-install on new IPA clients also works, however both the new and the old CA cert is pulled down to the IPA client and stored in /etc/ipa/ca.crt.
This creates some issues as most applications reading /etc/ipa/ca.crt only reads the first entry, which happens to be the old CA cert.
For the moment everything works OK as the old CA cert is still valid, however this will become a major issue in a few months time.
Is this expected? To continue to service both the old and the new CA certificate to old and new IPA clients? Will the old certificate be automatically removed at some point? If not, what is the safe steps to remove the old CA certificate from the IPA servers?
Could you please detail what systems are not able to process multi-cert ca.crt? Is that any of Debian/Ubuntu systems?
What applications you are encountering the problem with?
Hi Alex,
Thanks for your prompt response.
There are no Debian/Ubuntu systems in our environment.
From your response, is the dual CA cert to be expected / by design?
I have not verified what certificate every application in our environment ends up utilizing yet, as serving both the old and the new CA certificates seem to me to be a bug, and I would rather fix the bug than make workarounds.
Back to my original question, what is the reason for keep serving the old certificate? Would it not be sufficient to serve only the new certificate to new clients being enrolled and clients using the ipa-certupdate command?
Regards, Siggi
On 4 Mar 2020, at 13:51, Alexander Bokovoy <abokovoy@redhat.com mailto:abokovoy@redhat.com> wrote:
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi,
We recently renewed our IPA CA cert using the "ipa-cacert-manage renew” command. The renewal was successful, and our CA cert no longer expires in 2020, but in 2040.
Running “ipa-certupdate” on existing IPA clients and ipa-client-install on new IPA clients also works, however both the new and the old CA cert is pulled down to the IPA client and stored in /etc/ipa/ca.crt.
This creates some issues as most applications reading /etc/ipa/ca.crt only reads the first entry, which happens to be the old CA cert.
For the moment everything works OK as the old CA cert is still valid, however this will become a major issue in a few months time.
Is this expected? To continue to service both the old and the new CA certificate to old and new IPA clients? Will the old certificate be automatically removed at some point? If not, what is the safe steps to remove the old CA certificate from the IPA servers?
Could you please detail what systems are not able to process multi-cert ca.crt? Is that any of Debian/Ubuntu systems?
What applications you are encountering the problem with?
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi Alex,
Thanks for your prompt response.
There are no Debian/Ubuntu systems in our environment.
From your response, is the dual CA cert to be expected / by design?
Yes, actually, it is to be expected for any setup with external CA root.
I have not verified what certificate every application in our environment ends up utilizing yet, as serving both the old and the new CA certificates seem to me to be a bug, and I would rather fix the bug than make workarounds.
No it is not a bug. It is normal and common to have multiple CA roots available in a certificate store. The checks are done against a valid CA root for the specific certificate and if you have one issued with the use of older CA root certificate, you need to verify against that.
What I'd like to get clear is why are you pointing the applications to /etc/ipa/ca.crt? Supposedly, the content of this file is already a part of the system-wide certificate store. On RHEL/CentOS/Fedora systems the way how system-wide store works, there are multiple representations that are supported by all crypto libraries and frameworks. So you don't need to put a direct reference to /etc/ipa/ca.crt.
Back to my original question, what is the reason for keep serving the old certificate? Would it not be sufficient to serve only the new certificate to new clients being enrolled and clients using the ipa-certupdate command?
It is to allow clients to verify certificates issued with the previous CA root certificate. Until you have renewed all certificates issued with the old CA root, you need to keep that in place or clients/servers using that wouldn't be able to trust the certificate.
On 4 Mar 2020, at 14:27, Alexander Bokovoy via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi Alex,
Thanks for your prompt response.
There are no Debian/Ubuntu systems in our environment.
From your response, is the dual CA cert to be expected / by design?
Yes, actually, it is to be expected for any setup with external CA root.
This is not an external CA root. I presume both internal and external CA root is treated the same then.
I have not verified what certificate every application in our environment ends up utilizing yet, as serving both the old and the new CA certificates seem to me to be a bug, and I would rather fix the bug than make workarounds.
No it is not a bug. It is normal and common to have multiple CA roots available in a certificate store. The checks are done against a valid CA root for the specific certificate and if you have one issued with the use of older CA root certificate, you need to verify against that.
This does not seem to be correct for IPA. As far as I recall there was a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Verifying a certificate that was issues by the old CA against the new CA returns OK, and there are no issues connecting to the website.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
What I'd like to get clear is why are you pointing the applications to /etc/ipa/ca.crt? Supposedly, the content of this file is already a part of the system-wide certificate store. On RHEL/CentOS/Fedora systems the way how system-wide store works, there are multiple representations that are supported by all crypto libraries and frameworks. So you don't need to put a direct reference to /etc/ipa/ca.crt.
We have been using IPA in production since 2012. In testing even a couple of years earlier. Back then the only place the ca cert was written to the client was /etc/ipa/ca.crt, and so this is what has been used in our Puppet setup ever since the beginning. The fact that the ipa-client installs the CA certificate in the system-wide certificate store is a more recent development. (https://pagure.io/freeipa/issue/3504)
Back to my original question, what is the reason for keep serving the old certificate? Would it not be sufficient to serve only the new certificate to new clients being enrolled and clients using the ipa-certupdate command?
It is to allow clients to verify certificates issued with the previous CA root certificate. Until you have renewed all certificates issued with the old CA root, you need to keep that in place or clients/servers using that wouldn't be able to trust the certificate.
This is perhaps true for most PKI setups, however as mentioned, I seem to recall that a a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Again, openssl returns OK when verifying existing certificates with the new CA, and there are no issues connecting to the website where this is hosted.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
As this duplicated CA cert is a feature, what will happen when we move pass the expiry date of the old CA? Will it be automatically removed from IPA or is there any manual cleanup required?
Regards, Siggi
On pe, 06 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
On 4 Mar 2020, at 14:27, Alexander Bokovoy via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi Alex,
Thanks for your prompt response.
There are no Debian/Ubuntu systems in our environment.
From your response, is the dual CA cert to be expected / by design?
Yes, actually, it is to be expected for any setup with external CA root.
This is not an external CA root. I presume both internal and external CA root is treated the same then.
Yes, there is no difference in this sense. In both cases Dogtag owns the key -- the difference would only be where a self-signed root is located in a CA path.
I have not verified what certificate every application in our environment ends up utilizing yet, as serving both the old and the new CA certificates seem to me to be a bug, and I would rather fix the bug than make workarounds.
No it is not a bug. It is normal and common to have multiple CA roots available in a certificate store. The checks are done against a valid CA root for the specific certificate and if you have one issued with the use of older CA root certificate, you need to verify against that.
This does not seem to be correct for IPA. As far as I recall there was a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Verifying a certificate that was issues by the old CA against the new CA returns OK, and there are no issues connecting to the website.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
openssl verification is done down to a self-signed trust anchor. If your new CA root is using the same key (no re-keying happened on CA root renewal), the same key is in place, and IPA CA is self-signed, that's why it works. My understanding is that if you re-keyed CA root certificate on renewal, this wouldn't be true and you would need the old CA certificate to validate these server certificates.
I might be wrong here, though. See man page for openssl-verify, section 'VERIFY OPERATION' for some logic description.
What I'd like to get clear is why are you pointing the applications to /etc/ipa/ca.crt? Supposedly, the content of this file is already a part of the system-wide certificate store. On RHEL/CentOS/Fedora systems the way how system-wide store works, there are multiple representations that are supported by all crypto libraries and frameworks. So you don't need to put a direct reference to /etc/ipa/ca.crt.
We have been using IPA in production since 2012. In testing even a couple of years earlier. Back then the only place the ca cert was written to the client was /etc/ipa/ca.crt, and so this is what has been used in our Puppet setup ever since the beginning. The fact that the ipa-client installs the CA certificate in the system-wide certificate store is a more recent development. (https://pagure.io/freeipa/issue/3504)
Understood. The ticket mentioned was closed in 2014, so we are talking about all RHEL 7+/Fedora 19+ systems.
Back to my original question, what is the reason for keep serving the old certificate? Would it not be sufficient to serve only the new certificate to new clients being enrolled and clients using the ipa-certupdate command?
It is to allow clients to verify certificates issued with the previous CA root certificate. Until you have renewed all certificates issued with the old CA root, you need to keep that in place or clients/servers using that wouldn't be able to trust the certificate.
This is perhaps true for most PKI setups, however as mentioned, I seem to recall that a a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Again, openssl returns OK when verifying existing certificates with the new CA, and there are no issues connecting to the website where this is hosted.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
As this duplicated CA cert is a feature, what will happen when we move pass the expiry date of the old CA? Will it be automatically removed from IPA or is there any manual cleanup required?
There is no automatic cleanup right now. I thought we had a ticket for the clean up tool but I cannot find it right now. Please open one?
On Fri, Mar 06, 2020 at 12:48:50PM +0200, Alexander Bokovoy via FreeIPA-users wrote:
On pe, 06 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
On 4 Mar 2020, at 14:27, Alexander Bokovoy via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi Alex,
Thanks for your prompt response.
There are no Debian/Ubuntu systems in our environment.
From your response, is the dual CA cert to be expected / by design?
Yes, actually, it is to be expected for any setup with external CA root.
This is not an external CA root. I presume both internal and external CA root is treated the same then.
Yes, there is no difference in this sense. In both cases Dogtag owns the key -- the difference would only be where a self-signed root is located in a CA path.
I have not verified what certificate every application in our environment ends up utilizing yet, as serving both the old and the new CA certificates seem to me to be a bug, and I would rather fix the bug than make workarounds.
No it is not a bug. It is normal and common to have multiple CA roots available in a certificate store. The checks are done against a valid CA root for the specific certificate and if you have one issued with the use of older CA root certificate, you need to verify against that.
This does not seem to be correct for IPA. As far as I recall there was a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Verifying a certificate that was issues by the old CA against the new CA returns OK, and there are no issues connecting to the website.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
openssl verification is done down to a self-signed trust anchor. If your new CA root is using the same key (no re-keying happened on CA root renewal), the same key is in place, and IPA CA is self-signed, that's why it works. My understanding is that if you re-keyed CA root certificate on renewal, this wouldn't be true and you would need the old CA certificate to validate these server certificates.
I might be wrong here, though. See man page for openssl-verify, section 'VERIFY OPERATION' for some logic description.
What I'd like to get clear is why are you pointing the applications to /etc/ipa/ca.crt? Supposedly, the content of this file is already a part of the system-wide certificate store. On RHEL/CentOS/Fedora systems the way how system-wide store works, there are multiple representations that are supported by all crypto libraries and frameworks. So you don't need to put a direct reference to /etc/ipa/ca.crt.
We have been using IPA in production since 2012. In testing even a couple of years earlier. Back then the only place the ca cert was written to the client was /etc/ipa/ca.crt, and so this is what has been used in our Puppet setup ever since the beginning. The fact that the ipa-client installs the CA certificate in the system-wide certificate store is a more recent development. (https://pagure.io/freeipa/issue/3504)
Understood. The ticket mentioned was closed in 2014, so we are talking about all RHEL 7+/Fedora 19+ systems.
Back to my original question, what is the reason for keep serving the old certificate? Would it not be sufficient to serve only the new certificate to new clients being enrolled and clients using the ipa-certupdate command?
It is to allow clients to verify certificates issued with the previous CA root certificate. Until you have renewed all certificates issued with the old CA root, you need to keep that in place or clients/servers using that wouldn't be able to trust the certificate.
This is perhaps true for most PKI setups, however as mentioned, I seem to recall that a a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Again, openssl returns OK when verifying existing certificates with the new CA, and there are no issues connecting to the website where this is hosted.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
As this duplicated CA cert is a feature, what will happen when we move pass the expiry date of the old CA? Will it be automatically removed from IPA or is there any manual cleanup required?
There is no automatic cleanup right now. I thought we had a ticket for the clean up tool but I cannot find it right now. Please open one?
Rob recently implemented `ipa-cacert-manage delete` subcommand, on master and ipa-4-8 branch (there hasn't been a release containing it yet, though). It can be used to remove a specified certificate from the IPA trust store. But it is not automatic.
If expired CA certs are present in trust stores, clients will (or should) ignore them.
Cheers, Fraser
Fraser Tweedale via FreeIPA-users wrote:
On Fri, Mar 06, 2020 at 12:48:50PM +0200, Alexander Bokovoy via FreeIPA-users wrote:
On pe, 06 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
On 4 Mar 2020, at 14:27, Alexander Bokovoy via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi Alex,
Thanks for your prompt response.
There are no Debian/Ubuntu systems in our environment.
From your response, is the dual CA cert to be expected / by design?
Yes, actually, it is to be expected for any setup with external CA root.
This is not an external CA root. I presume both internal and external CA root is treated the same then.
Yes, there is no difference in this sense. In both cases Dogtag owns the key -- the difference would only be where a self-signed root is located in a CA path.
I have not verified what certificate every application in our environment ends up utilizing yet, as serving both the old and the new CA certificates seem to me to be a bug, and I would rather fix the bug than make workarounds.
No it is not a bug. It is normal and common to have multiple CA roots available in a certificate store. The checks are done against a valid CA root for the specific certificate and if you have one issued with the use of older CA root certificate, you need to verify against that.
This does not seem to be correct for IPA. As far as I recall there was a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Verifying a certificate that was issues by the old CA against the new CA returns OK, and there are no issues connecting to the website.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
openssl verification is done down to a self-signed trust anchor. If your new CA root is using the same key (no re-keying happened on CA root renewal), the same key is in place, and IPA CA is self-signed, that's why it works. My understanding is that if you re-keyed CA root certificate on renewal, this wouldn't be true and you would need the old CA certificate to validate these server certificates.
I might be wrong here, though. See man page for openssl-verify, section 'VERIFY OPERATION' for some logic description.
What I'd like to get clear is why are you pointing the applications to /etc/ipa/ca.crt? Supposedly, the content of this file is already a part of the system-wide certificate store. On RHEL/CentOS/Fedora systems the way how system-wide store works, there are multiple representations that are supported by all crypto libraries and frameworks. So you don't need to put a direct reference to /etc/ipa/ca.crt.
We have been using IPA in production since 2012. In testing even a couple of years earlier. Back then the only place the ca cert was written to the client was /etc/ipa/ca.crt, and so this is what has been used in our Puppet setup ever since the beginning. The fact that the ipa-client installs the CA certificate in the system-wide certificate store is a more recent development. (https://pagure.io/freeipa/issue/3504)
Understood. The ticket mentioned was closed in 2014, so we are talking about all RHEL 7+/Fedora 19+ systems.
Back to my original question, what is the reason for keep serving the old certificate? Would it not be sufficient to serve only the new certificate to new clients being enrolled and clients using the ipa-certupdate command?
It is to allow clients to verify certificates issued with the previous CA root certificate. Until you have renewed all certificates issued with the old CA root, you need to keep that in place or clients/servers using that wouldn't be able to trust the certificate.
This is perhaps true for most PKI setups, however as mentioned, I seem to recall that a a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Again, openssl returns OK when verifying existing certificates with the new CA, and there are no issues connecting to the website where this is hosted.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
As this duplicated CA cert is a feature, what will happen when we move pass the expiry date of the old CA? Will it be automatically removed from IPA or is there any manual cleanup required?
There is no automatic cleanup right now. I thought we had a ticket for the clean up tool but I cannot find it right now. Please open one?
Rob recently implemented `ipa-cacert-manage delete` subcommand, on master and ipa-4-8 branch (there hasn't been a release containing it yet, though). It can be used to remove a specified certificate from the IPA trust store. But it is not automatic.
If expired CA certs are present in trust stores, clients will (or should) ignore them.
I should point out that the delete command deletes ALL certs for a nickname so it wouldn't help in this particular case.
rob
On Tue, Mar 10, 2020 at 10:25:01AM -0400, Rob Crittenden wrote:
Fraser Tweedale via FreeIPA-users wrote:
On Fri, Mar 06, 2020 at 12:48:50PM +0200, Alexander Bokovoy via FreeIPA-users wrote:
On pe, 06 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
On 4 Mar 2020, at 14:27, Alexander Bokovoy via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
Hi Alex,
Thanks for your prompt response.
There are no Debian/Ubuntu systems in our environment.
From your response, is the dual CA cert to be expected / by design?
Yes, actually, it is to be expected for any setup with external CA root.
This is not an external CA root. I presume both internal and external CA root is treated the same then.
Yes, there is no difference in this sense. In both cases Dogtag owns the key -- the difference would only be where a self-signed root is located in a CA path.
I have not verified what certificate every application in our environment ends up utilizing yet, as serving both the old and the new CA certificates seem to me to be a bug, and I would rather fix the bug than make workarounds.
No it is not a bug. It is normal and common to have multiple CA roots available in a certificate store. The checks are done against a valid CA root for the specific certificate and if you have one issued with the use of older CA root certificate, you need to verify against that.
This does not seem to be correct for IPA. As far as I recall there was a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Verifying a certificate that was issues by the old CA against the new CA returns OK, and there are no issues connecting to the website.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
openssl verification is done down to a self-signed trust anchor. If your new CA root is using the same key (no re-keying happened on CA root renewal), the same key is in place, and IPA CA is self-signed, that's why it works. My understanding is that if you re-keyed CA root certificate on renewal, this wouldn't be true and you would need the old CA certificate to validate these server certificates.
I might be wrong here, though. See man page for openssl-verify, section 'VERIFY OPERATION' for some logic description.
What I'd like to get clear is why are you pointing the applications to /etc/ipa/ca.crt? Supposedly, the content of this file is already a part of the system-wide certificate store. On RHEL/CentOS/Fedora systems the way how system-wide store works, there are multiple representations that are supported by all crypto libraries and frameworks. So you don't need to put a direct reference to /etc/ipa/ca.crt.
We have been using IPA in production since 2012. In testing even a couple of years earlier. Back then the only place the ca cert was written to the client was /etc/ipa/ca.crt, and so this is what has been used in our Puppet setup ever since the beginning. The fact that the ipa-client installs the CA certificate in the system-wide certificate store is a more recent development. (https://pagure.io/freeipa/issue/3504)
Understood. The ticket mentioned was closed in 2014, so we are talking about all RHEL 7+/Fedora 19+ systems.
Back to my original question, what is the reason for keep serving the old certificate? Would it not be sufficient to serve only the new certificate to new clients being enrolled and clients using the ipa-certupdate command?
It is to allow clients to verify certificates issued with the previous CA root certificate. Until you have renewed all certificates issued with the old CA root, you need to keep that in place or clients/servers using that wouldn't be able to trust the certificate.
This is perhaps true for most PKI setups, however as mentioned, I seem to recall that a a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Again, openssl returns OK when verifying existing certificates with the new CA, and there are no issues connecting to the website where this is hosted.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
As this duplicated CA cert is a feature, what will happen when we move pass the expiry date of the old CA? Will it be automatically removed from IPA or is there any manual cleanup required?
There is no automatic cleanup right now. I thought we had a ticket for the clean up tool but I cannot find it right now. Please open one?
Rob recently implemented `ipa-cacert-manage delete` subcommand, on master and ipa-4-8 branch (there hasn't been a release containing it yet, though). It can be used to remove a specified certificate from the IPA trust store. But it is not automatic.
If expired CA certs are present in trust stores, clients will (or should) ignore them.
I should point out that the delete command deletes ALL certs for a nickname so it wouldn't help in this particular case.
Thanks for the clarification, Rob.
If you need to remove just a single cert for the same subject (e.g. an older expired one), you can delete that particular userCertificate attribute value from its LDAP entry under cn=certificates,cn=ipa,cn=etc,{basedn}.
I also want to clarify that it is expected behaviour for IPA will put all trusted CA certs, including possibly expired variants, in the /etc/ipa/ca.crt and other system trust stores. If it is causing an issue for some other program, the problem is with that program, not with FreeIPA.
Thanks, Fraser
Fraser Tweedale wrote:
On Tue, Mar 10, 2020 at 10:25:01AM -0400, Rob Crittenden wrote:
Fraser Tweedale via FreeIPA-users wrote:
On Fri, Mar 06, 2020 at 12:48:50PM +0200, Alexander Bokovoy via FreeIPA-users wrote:
On pe, 06 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
On 4 Mar 2020, at 14:27, Alexander Bokovoy via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote: > Hi Alex, > > Thanks for your prompt response. > > There are no Debian/Ubuntu systems in our environment. > > From your response, is the dual CA cert to be expected / by design?
Yes, actually, it is to be expected for any setup with external CA root.
This is not an external CA root. I presume both internal and external CA root is treated the same then.
Yes, there is no difference in this sense. In both cases Dogtag owns the key -- the difference would only be where a self-signed root is located in a CA path.
> I have not verified what certificate every application in our > environment ends up utilizing yet, as serving both the old and the new > CA certificates seem to me to be a bug, and I would rather fix the bug > than make workarounds.
No it is not a bug. It is normal and common to have multiple CA roots available in a certificate store. The checks are done against a valid CA root for the specific certificate and if you have one issued with the use of older CA root certificate, you need to verify against that.
This does not seem to be correct for IPA. As far as I recall there was a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Verifying a certificate that was issues by the old CA against the new CA returns OK, and there are no issues connecting to the website.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
openssl verification is done down to a self-signed trust anchor. If your new CA root is using the same key (no re-keying happened on CA root renewal), the same key is in place, and IPA CA is self-signed, that's why it works. My understanding is that if you re-keyed CA root certificate on renewal, this wouldn't be true and you would need the old CA certificate to validate these server certificates.
I might be wrong here, though. See man page for openssl-verify, section 'VERIFY OPERATION' for some logic description.
What I'd like to get clear is why are you pointing the applications to /etc/ipa/ca.crt? Supposedly, the content of this file is already a part of the system-wide certificate store. On RHEL/CentOS/Fedora systems the way how system-wide store works, there are multiple representations that are supported by all crypto libraries and frameworks. So you don't need to put a direct reference to /etc/ipa/ca.crt.
We have been using IPA in production since 2012. In testing even a couple of years earlier. Back then the only place the ca cert was written to the client was /etc/ipa/ca.crt, and so this is what has been used in our Puppet setup ever since the beginning. The fact that the ipa-client installs the CA certificate in the system-wide certificate store is a more recent development. (https://pagure.io/freeipa/issue/3504)
Understood. The ticket mentioned was closed in 2014, so we are talking about all RHEL 7+/Fedora 19+ systems.
> Back to my original question, what is the reason for keep serving the > old certificate? Would it not be sufficient to serve only the new > certificate to new clients being enrolled and clients using the > ipa-certupdate command?
It is to allow clients to verify certificates issued with the previous CA root certificate. Until you have renewed all certificates issued with the old CA root, you need to keep that in place or clients/servers using that wouldn't be able to trust the certificate.
This is perhaps true for most PKI setups, however as mentioned, I seem to recall that a a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Again, openssl returns OK when verifying existing certificates with the new CA, and there are no issues connecting to the website where this is hosted.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
As this duplicated CA cert is a feature, what will happen when we move pass the expiry date of the old CA? Will it be automatically removed from IPA or is there any manual cleanup required?
There is no automatic cleanup right now. I thought we had a ticket for the clean up tool but I cannot find it right now. Please open one?
Rob recently implemented `ipa-cacert-manage delete` subcommand, on master and ipa-4-8 branch (there hasn't been a release containing it yet, though). It can be used to remove a specified certificate from the IPA trust store. But it is not automatic.
If expired CA certs are present in trust stores, clients will (or should) ignore them.
I should point out that the delete command deletes ALL certs for a nickname so it wouldn't help in this particular case.
Thanks for the clarification, Rob.
If you need to remove just a single cert for the same subject (e.g. an older expired one), you can delete that particular userCertificate attribute value from its LDAP entry under cn=certificates,cn=ipa,cn=etc,{basedn}.
I also want to clarify that it is expected behaviour for IPA will put all trusted CA certs, including possibly expired variants, in the /etc/ipa/ca.crt and other system trust stores. If it is causing an issue for some other program, the problem is with that program, not with FreeIPA.
I completely agree but practically speaking an expired CA isn't all that useful is it?
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
On Tue, Mar 10, 2020 at 08:39:39PM -0400, Rob Crittenden wrote:
Fraser Tweedale wrote:
On Tue, Mar 10, 2020 at 10:25:01AM -0400, Rob Crittenden wrote:
Fraser Tweedale via FreeIPA-users wrote:
On Fri, Mar 06, 2020 at 12:48:50PM +0200, Alexander Bokovoy via FreeIPA-users wrote:
On pe, 06 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote:
> On 4 Mar 2020, at 14:27, Alexander Bokovoy via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote: > > On ke, 04 maalis 2020, Sigbjorn Lie via FreeIPA-users wrote: >> Hi Alex, >> >> Thanks for your prompt response. >> >> There are no Debian/Ubuntu systems in our environment. >> >> From your response, is the dual CA cert to be expected / by design? > > Yes, actually, it is to be expected for any setup with external CA root.
This is not an external CA root. I presume both internal and external CA root is treated the same then.
Yes, there is no difference in this sense. In both cases Dogtag owns the key -- the difference would only be where a self-signed root is located in a CA path.
>> I have not verified what certificate every application in our >> environment ends up utilizing yet, as serving both the old and the new >> CA certificates seem to me to be a bug, and I would rather fix the bug >> than make workarounds. > > No it is not a bug. It is normal and common to have multiple CA roots > available in a certificate store. The checks are done against a valid > CA root for the specific certificate and if you have one issued with the > use of older CA root certificate, you need to verify against that.
This does not seem to be correct for IPA. As far as I recall there was a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Verifying a certificate that was issues by the old CA against the new CA returns OK, and there are no issues connecting to the website.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
openssl verification is done down to a self-signed trust anchor. If your new CA root is using the same key (no re-keying happened on CA root renewal), the same key is in place, and IPA CA is self-signed, that's why it works. My understanding is that if you re-keyed CA root certificate on renewal, this wouldn't be true and you would need the old CA certificate to validate these server certificates.
I might be wrong here, though. See man page for openssl-verify, section 'VERIFY OPERATION' for some logic description.
> What I'd like to get clear is why are you pointing the applications to > /etc/ipa/ca.crt? Supposedly, the content of this file is already a part > of the system-wide certificate store. On RHEL/CentOS/Fedora systems the > way how system-wide store works, there are multiple representations that > are supported by all crypto libraries and frameworks. So you don't need > to put a direct reference to /etc/ipa/ca.crt.
We have been using IPA in production since 2012. In testing even a couple of years earlier. Back then the only place the ca cert was written to the client was /etc/ipa/ca.crt, and so this is what has been used in our Puppet setup ever since the beginning. The fact that the ipa-client installs the CA certificate in the system-wide certificate store is a more recent development. (https://pagure.io/freeipa/issue/3504)
Understood. The ticket mentioned was closed in 2014, so we are talking about all RHEL 7+/Fedora 19+ systems.
>> Back to my original question, what is the reason for keep serving the >> old certificate? Would it not be sufficient to serve only the new >> certificate to new clients being enrolled and clients using the >> ipa-certupdate command? > > It is to allow clients to verify certificates issued with the previous > CA root certificate. Until you have renewed all certificates issued with > the old CA root, you need to keep that in place or clients/servers using > that wouldn't be able to trust the certificate.
This is perhaps true for most PKI setups, however as mentioned, I seem to recall that a a feature for making sure at that the renewed IPA CA certificate (when using self-signed CA cert) continue to work for the existing issued certificates. Again, openssl returns OK when verifying existing certificates with the new CA, and there are no issues connecting to the website where this is hosted.
sudo openssl verify -verbose -CAfile /etc/ipa/ca-new.crt /etc/pki/httpd/website1.crt /etc/pki/httpd/website1.crt: OK
As this duplicated CA cert is a feature, what will happen when we move pass the expiry date of the old CA? Will it be automatically removed from IPA or is there any manual cleanup required?
There is no automatic cleanup right now. I thought we had a ticket for the clean up tool but I cannot find it right now. Please open one?
Rob recently implemented `ipa-cacert-manage delete` subcommand, on master and ipa-4-8 branch (there hasn't been a release containing it yet, though). It can be used to remove a specified certificate from the IPA trust store. But it is not automatic.
If expired CA certs are present in trust stores, clients will (or should) ignore them.
I should point out that the delete command deletes ALL certs for a nickname so it wouldn't help in this particular case.
Thanks for the clarification, Rob.
If you need to remove just a single cert for the same subject (e.g. an older expired one), you can delete that particular userCertificate attribute value from its LDAP entry under cn=certificates,cn=ipa,cn=etc,{basedn}.
I also want to clarify that it is expected behaviour for IPA will put all trusted CA certs, including possibly expired variants, in the /etc/ipa/ca.crt and other system trust stores. If it is causing an issue for some other program, the problem is with that program, not with FreeIPA.
I completely agree but practically speaking an expired CA isn't all that useful is it?
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
Cheers, Fraser
On ke, 11 maalis 2020, Fraser Tweedale via FreeIPA-users wrote:
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
I agree. So, we still need a ticket for ipa-certupdate to gain an explicit option to ignore expired certs.
On Wed, Mar 11, 2020 at 09:26:54AM +0200, Alexander Bokovoy wrote:
On ke, 11 maalis 2020, Fraser Tweedale via FreeIPA-users wrote:
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
I agree. So, we still need a ticket for ipa-certupdate to gain an explicit option to ignore expired certs.
I think we can ignore (i.e. not install) expired certs by default. And maybe have option to install all certs even if expired.
What would customers expect? It is not the first time a customer was surprised to see expired certs there and asked about it.
Cheers, Fraser
On Wed, Mar 11, 2020 at 9:12 AM Fraser Tweedale via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
On Wed, Mar 11, 2020 at 09:26:54AM +0200, Alexander Bokovoy wrote:
On ke, 11 maalis 2020, Fraser Tweedale via FreeIPA-users wrote:
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
I agree. So, we still need a ticket for ipa-certupdate to gain an explicit option to ignore expired certs.
I think we can ignore (i.e. not install) expired certs by default. And maybe have option to install all certs even if expired.
Yes. While the current behavior does not lead to any malfunctioning service, various useful tools cease to function when the first cert in ca.crt is expired.
What would customers expect? It is not the first time a customer was surprised to see expired certs there and asked about it.
My guess: not having the tools above fail in the first place.
Cheers François
Cheers, Fraser _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Alexander Bokovoy wrote:
On ke, 11 maalis 2020, Fraser Tweedale via FreeIPA-users wrote:
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
I agree. So, we still need a ticket for ipa-certupdate to gain an explicit option to ignore expired certs.
IMHO it should be the default for certstore.get_ca_certs(). I opened https://pagure.io/freeipa/issue/8223
I don't know of a case where we would want to fetch non-valid CA certificates, please update the ticket if you know of any.
rob
On ke, 11 maalis 2020, Rob Crittenden wrote:
Alexander Bokovoy wrote:
On ke, 11 maalis 2020, Fraser Tweedale via FreeIPA-users wrote:
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
I agree. So, we still need a ticket for ipa-certupdate to gain an explicit option to ignore expired certs.
IMHO it should be the default for certstore.get_ca_certs(). I opened https://pagure.io/freeipa/issue/8223
I don't know of a case where we would want to fetch non-valid CA certificates, please update the ticket if you know of any.
Valid from which point of view? A system we run on? E.g. based on the local time setup?
Alexander Bokovoy via FreeIPA-users wrote:
On ke, 11 maalis 2020, Rob Crittenden wrote:
Alexander Bokovoy wrote:
On ke, 11 maalis 2020, Fraser Tweedale via FreeIPA-users wrote:
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
I agree. So, we still need a ticket for ipa-certupdate to gain an explicit option to ignore expired certs.
IMHO it should be the default for certstore.get_ca_certs(). I opened https://pagure.io/freeipa/issue/8223
I don't know of a case where we would want to fetch non-valid CA certificates, please update the ticket if you know of any.
Valid from which point of view? A system we run on? E.g. based on the local time setup?
Correct, local time.
Francois updated the issue to indicate that the expired CA first causes issues. I wonder if we should test sorting by expiration date instead.
rob
On 11 Mar 2020, at 14:29, Rob Crittenden via FreeIPA-users <freeipa-users@lists.fedorahosted.org mailto:freeipa-users@lists.fedorahosted.org> wrote:
Alexander Bokovoy via FreeIPA-users wrote:
On ke, 11 maalis 2020, Rob Crittenden wrote:
Alexander Bokovoy wrote:
On ke, 11 maalis 2020, Fraser Tweedale via FreeIPA-users wrote:
Makes me look at this a different way. Perhaps change the certstore to only return valid CA certs. That way they are stored if anyone ever wants them but they won't get pulled down for ipa-certupdate or ipaclilent-install.
Or to try the ipa-cacert-manage route, it was mostly the UI part for why I didn't do it. I wasn't sure if the best way would be to interactively show each cert and do a delete Y/N or what. Perhaps a delete with --expired-only to do the cleanup. I'm open to suggestions.
rob
I think it's fine to change ipa-certupdate so it skips expired / not-yet-valid certs.
IMO we should never automatically prune expired certs from the LDAP trust store, so that if customer needs to do time travel to fix an issue, the old CA certs will still be there and an ipa-certupdate will "restore" them to the various certificate DBs.
And for the same reason, I'd be hesitant to offer a UI to prune expired certs from the trust store.
I agree. So, we still need a ticket for ipa-certupdate to gain an explicit option to ignore expired certs.
IMHO it should be the default for certstore.get_ca_certs(). I opened https://pagure.io/freeipa/issue/8223 https://pagure.io/freeipa/issue/8223
I don't know of a case where we would want to fetch non-valid CA certificates, please update the ticket if you know of any.
Valid from which point of view? A system we run on? E.g. based on the local time setup?
Correct, local time.
Francois updated the issue to indicate that the expired CA first causes issues. I wonder if we should test sorting by expiration date instead.
rob _______________________________________________
Hi,
A quick update on this issue. The CA cert I mentioned initially has now expired.
The first thing that stopped working was Satellite, which could no longer connect to IPA.
Using “openssl s_client” to connect to the LDAP server in IPA also returned "verify error:num=10:certificate has expired”.
Both the old and the new certificate was present in /etc/pki/tls/cert.pem (symlinked to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem) on the Satellite server.
Removing the expired certificate and restarting the httpd and foreman-proxy services allowed Satellite to resume operations. And “openssl s_client” started returning "verify return:1”.
Further running ipa-certupdate on any client removed the expired cert from /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, however SSSD stopped working. When I ran ipa-certupdate shortly after the CA cert was renewed in February, both the new and the old certificate was kept.
When users logged in the following was logged in /var/log/secure: pam_sss(sshd:account): Access denied for user username: 6 (Permission denied)
Running "sssctl user-checks username” on the server returned "pam_acct_mgmt: Permission denied”.
The "sss_cache -E” command was not sufficient to fix the issue.The SSSD service and had to be stopped, then manually flush the cache (rm -f /var/lib/sss/db/*), and then start the SSSD service again.
Now "sssctl user-checks username” returned "pam_acct_mgmt: Success”, and the users we’re able to log on again.
As I mentioned ipa-certupdate was run in February as well, however the SSSD issues did not occur at this time.
Regards, Siggi
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
François Cami -
Fraser Tweedale -
Rob Crittenden -
Sigbjorn Lie