https://fedoraproject.org/wiki/Changes/krb5_crypto_modernization
krb5 will be removing support for DES, 3DES, crc-32, and MD4 entirely; they will not be allowed in session keys or long-term keys. Additionally, RC4 and MD5 will be marked deprecated and dangerous.
== Owner == * Name: [[User:rharwood|Robbie Harwood]] * Email: rharwood at fp dot o
== Detailed Description ==
For formal deprecation announcements, there have been two RFCs. First was [https://tools.ietf.org/html/rfc6649 rfc6649], which deprecated single-DES (and export strength RC4). More recently, there has also been [https://tools.ietf.org/html/rfc8429 rfc8429] which deprecated the rest of RC4 as well as 3DES. (Note that 3DES was paired with SHA1, RC4 with MD5, and DES with MD4, MD5, or crc32, and that these were the only usages of these digests in krb5's Kerberos implementation.)
Of particular note, single-DES has been known-bad since the mid 90s. It's now so bad that it can be entirely broken in 13 hours for $300: [https://crack.sh/kerb5/ https://crack.sh/kerb5/] But even though other encryption types in this list are less bad, none are what we'd call good - especially since krb5 supports much stronger encryption, such as AES256/SHA2.
Finally, field data suggests that complete removal is preferable to disabling by default. Admins have unfortunately been using `allow_weak_crypto` (which restored disabled-by-default single-DES functionality to krb5) to this day. A mere config setting doesn't suffice to convey how bad this is, nor does it indicate the appropriate level of urgency of migration. For that reason, and in anticipation that our crypto backends might one day also remove these algorithms, we are going straight to removal as well, rather than repeating `allow_weak_crypto` or similar.
Unfortunately, it's not possible for RC4 to be removed entirely at this time because it is still in demand for Active Directory integration (which also affects freeIPA and samba). For this reason, it's moved to `allow_weak_crypto`.
== Benefit to Fedora ==
Improved crpytography is good for security. Fedora will be leading on this change; no other distributions or OS releases have limited the bad ciphers that Kerberos can use.
== Scope == * Proposal owners: rharwood * Other developers: N/A (not a System Wide Change) * Release engineering: [https://pagure.io/releng/issue/8012 #8012] * Policies and guidelines: N/A (not a System Wide Change) * Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact == Users of RC4 who need to continue doing so will need to add `allow_weak_crypto = True` to /etc/krb5.conf.
== How To Test ==
If you'd like to test either this change or your site's readiness for more modern cryptography, that's great! `krb5.conf(5)` has several encryption type (enctype) -related settings - of particular note is permitted_enctypes. Set to only removed enctyptes to verify they don't work anymore; set to only modern enctypes to verify that you're ready. (Then just perform your normal tasks with krb5, such as `kinit`).
C-friendly users might wish to take a look at t_enctypes.c in the krb5 source tree. (How this will be affected depends on how the removal work internally, which I haven't worked out yet.)
== User Experience ==
Ideally no change! Worst case some users will see krb5 produce error messages about bad enctypes not being able to be used (has no enctype, could not fullfill enctype, etc.). These pains are the feeling of the world grinding forward security-wise.
== Dependencies == N/A (not a System Wide Change)
== Contingency Plan == * Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change) * Contingency deadline: N/A (not a System Wide Change) * Blocks release? N/A (not a System Wide Change) * Blocks product? N/A
== Documentation == N/A (not a System Wide Change)
== Release Notes ==
krb5 removes support for several known-bad encryption types. Hopefully users will see no changes.
Hi,
personally, I don't like the advertisment for that commercial service, see the given price and link. It does not comply with FLOSS policies, therefore I commented in the releng ticket.
Just my 5ct, Raphael
Raphael, I'm confused how this doesn't comply. Their source code lives here: https://github.com/h1kari/des_kpt
Also, let's keep comments on this thread *and CC me* - then there's only one place to look for replies (and I'm not subscribed to this list). In any case, as I stated there, if you have an alternate phrasing that doesn't come across to you as "advertising", I'm open to suggestion.
Thanks, --Robbie
On Fri, 2018-12-21 at 15:35 -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/krb5_crypto_modernization
krb5 will be removing support for DES, 3DES, crc-32, and MD4 entirely; they will not be allowed in session keys or long-term keys. Additionally, RC4 and MD5 will be marked deprecated and dangerous.
== Owner ==
- Name: [[|Robbie Harwood]]
- Email: rharwood at fp dot o
== Detailed Description ==
For formal deprecation announcements, there have been two RFCs. First was [https://tools.ietf.org/html/rfc6649 rfc6649], which deprecated single-DES (and export strength RC4). More recently, there has also been [https://tools.ietf.org/html/rfc8429 rfc8429] which deprecated the rest of RC4 as well as 3DES. (Note that 3DES was paired with SHA1, RC4 with MD5, and DES with MD4, MD5, or crc32, and that these were the only usages of these digests in krb5's Kerberos implementation.)
Of particular note, single-DES has been known-bad since the mid 90s. It's now so bad that it can be entirely broken in 13 hours for $300: [https://crack.sh/kerb5/ https://crack.sh/kerb5/] But even though other encryption types in this list are less bad, none are what we'd call good - especially since krb5 supports much stronger encryption, such as AES256/SHA2.
Finally, field data suggests that complete removal is preferable to disabling by default. Admins have unfortunately been using `allow_weak_crypto` (which restored disabled-by-default single-DES functionality to krb5) to this day. A mere config setting doesn't suffice to convey how bad this is, nor does it indicate the appropriate level of urgency of migration. For that reason, and in anticipation that our crypto backends might one day also remove these algorithms, we are going straight to removal as well, rather than repeating `allow_weak_crypto` or similar.
Unfortunately, it's not possible for RC4 to be removed entirely at this time because it is still in demand for Active Directory integration (which also affects freeIPA and samba). For this reason, it's moved to `allow_weak_crypto`.
== Benefit to Fedora ==
Improved crpytography is good for security. Fedora will be leading on this change; no other distributions or OS releases have limited the bad ciphers that Kerberos can use.
== Scope ==
- Proposal owners: rharwood
- Other developers: N/A (not a System Wide Change)
- Release engineering: [https://pagure.io/releng/issue/8012 #8012]
- Policies and guidelines: N/A (not a System Wide Change)
- Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact == Users of RC4 who need to continue doing so will need to add `allow_weak_crypto = True` to /etc/krb5.conf.
== How To Test ==
If you'd like to test either this change or your site's readiness for more modern cryptography, that's great! `krb5.conf(5)` has several encryption type (enctype) -related settings - of particular note is permitted_enctypes. Set to only removed enctyptes to verify they don't work anymore; set to only modern enctypes to verify that you're ready. (Then just perform your normal tasks with krb5, such as `kinit`).
C-friendly users might wish to take a look at t_enctypes.c in the krb5 source tree. (How this will be affected depends on how the removal work internally, which I haven't worked out yet.)
Hi, How does this ties with crypto policies? libkrb5 is already under crypto policies and has these ciphers disabled by default. Is this change about removing them from the code or removing them from the capabilities of the KDC which is not covered by crypto policies?
== User Experience ==
Ideally no change! Worst case some users will see krb5 produce error messages about bad enctypes not being able to be used (has no enctype, could not fullfill enctype, etc.). These pains are the feeling of the world grinding forward security-wise.
I guess there will be different experience in client side where these are already disabled by default in fedora, and in server side which they are not.
regards, Nikos
Nikos Mavrogiannopoulos nmav@redhat.com writes:
How does this ties with crypto policies? libkrb5 is already under crypto policies and has these ciphers disabled by default. Is this change about removing them from the code or removing them from the capabilities of the KDC which is not covered by crypto policies?
== User Experience ==
Ideally no change! Worst case some users will see krb5 produce error messages about bad enctypes not being able to be used (has no enctype, could not fullfill enctype, etc.). These pains are the feeling of the world grinding forward security-wise.
I guess there will be different experience in client side where these are already disabled by default in fedora, and in server side which they are not.
To address I think both your questions: the defaults we set matter a lot less than we think they do. I observe plenty of machines running single-DES in the wild - despite this being permitted at no level of crypto-policies. This is obviously not the fault of crypto-policies in any way; users have explicitly chosen to route around the system's notion of "permitted" - and not without reason, in many cases.
The two biggest culprits for this (beyond general legacy infrastructure) are Active Directory and AFS. Active Directory uses RC4 (crypto-policies LEGACY) as an interop cipher; this is why it's not actually removed in this change. AFS has been using single-DES (for interop) for quite some time. AFS does support more modern cryptography, but its adoption is far from universal.
But I think the biggest takeaway here is that requiring users to perform additional configuration in order to hurt themselves won't actually stop them all from hurting themselves - especially not in the age of automation. Maybe it would have been more effective if we'd followed the 2009 `allow_weak_crypto` designation with a 2010 removal or some threat thereof. Hard for me to know; in 2009 I was busy being an obnoxious teenager.
Thanks, --Robbie
"BC" == Ben Cotton bcotton@redhat.com writes:
BC> == Detailed Description ==
[elided]
Is it just me or does this not actually say clearly what is changing? The first paragraph talks about two RFCs. The second paragraph talks about how easy it is to break single DES. The third paragraph talks about how disabled by default is undesirable. The fourth paragraph talks about it not being possible to remove RC4.
But nothing says what is changing. The release notes section says:
BC> krb5 removes support for several known-bad encryption types. BC> Hopefully users will see no changes.
but it doesn't really say what "removes support" means, or exactly which encryption types will no longer be supported.
BC> == User Experience ==
BC> Ideally no change! Worst case some users will see krb5 produce BC> error messages about bad enctypes not being able to be used (has no BC> enctype, could not fullfill enctype, etc.). These pains are the BC> feeling of the world grinding forward security-wise.
I think this is an extremely optimistic description of the worst case behavior.
There is a good document for migrating from old encryption types at https://web.mit.edu/kerberos/krb5-1.16/doc/admin/advanced/retiring-des.html Note that the last paragraph there mentions that single-DES is still basically OK for the K/M principal and it's not required to use allow_weak_crypto to access a database encrypted with single-DES. I can't tell if this change actually intends to remove all support for the single-DES algorithm, but if so then it is highly important to communicate the additional need to roll over the master key and stash files.
It would also be nice to document a way to find principals which do not have a sufficiently strong key.
I personally have a kerberos database which has existed for a very long time now and am still dealing with the 3DES deprecation (and the fact that the F29 crypto policies changed to remove it from supported_enctypes).
- J<
"JLT" == Jason L Tibbitts tibbs@math.uh.edu writes:
JLT> Is it just me or does this not actually say clearly what is JLT> changing?
Seems it's just me; somehow that's in the summary but not in the detailed description. Seems odd for the details to have less information than the short version, but I guess that's how it goes.
My questions about whether K/M will need to be rotated (even though the existing kerberos docs say that it doesn't) and how a site can easily tell if it has old principals still stand.
- J<
BC> == Detailed Description ==
Is it just me or does this not actually say clearly what is changing? The first paragraph talks about two RFCs. The second paragraph talks about how easy it is to break single DES. The third paragraph talks about how disabled by default is undesirable. The fourth paragraph talks about it not being possible to remove RC4.
But nothing says what is changing. The release notes section says:
BC> krb5 removes support for several known-bad encryption types. BC> Hopefully users will see no changes.
but it doesn't really say what "removes support" means, or exactly which encryption types will no longer be supported.
Per your follow-up email, I'm not clear on whether you want changes here. If you do, speak up, especially if you have suggestions.
BC> == User Experience ==
BC> Ideally no change! Worst case some users will see krb5 produce BC> error messages about bad enctypes not being able to be used (has no BC> enctype, could not fullfill enctype, etc.). These pains are the BC> feeling of the world grinding forward security-wise.
I think this is an extremely optimistic description of the worst case behavior.
I really don't think that "it won't work and there'll be error messages" is an "extremely optimistic description". If you have langauge changes, I'm happy to hear them out.
There is a good document for migrating from old encryption types at https://web.mit.edu/kerberos/krb5-1.16/doc/admin/advanced/retiring-des.html Note that the last paragraph there mentions that single-DES is still basically OK for the K/M principal and it's not required to use allow_weak_crypto to access a database encrypted with single-DES.
That reading only applies to sites with a stash file nearby (though sites set up otherwise remain relatively uncommon in my impression, or are freeipa and will have AES keys).
I can't tell if this change actually intends to remove all support for the single-DES algorithm, but if so then it is highly important to communicate the additional need to roll over the master key and stash files.
Session keys for sure, but ideally all of it. I tried to state this in the Summary; if I was insufficiently clear I'm happy to make changes! (Added rollover langauge to "Upgrade/Compatability Impact" section.)
It would also be nice to document a way to find principals which do not have a sufficiently strong key.
This is good feedback. I will explore and see if I can make something.
I personally have a kerberos database which has existed for a very long time now and am still dealing with the 3DES deprecation (and the fact that the F29 crypto policies changed to remove it from supported_enctypes).
I'm sure you're aware, then, that there's only so much I can do as a packager here. In particular, the KDC can't push keytab updates (because servers with keytabs are not required to be able to communicate with the KDC). The UX on this is *going* to be bad for anyone with weak crypto in use - which is part of why I want to pull it all out at once, so we don't have to do this again for a while.
Please CC me on all future mails; I'm not subscribed to this list.
Thanks, --Robbie
On Thu, 2019-01-03 at 23:07 +0000, Robbie Harwood wrote:
BC> == Detailed Description ==
Is it just me or does this not actually say clearly what is changing? The first paragraph talks about two RFCs. The second paragraph talks about how easy it is to break single DES. The third paragraph talks about how disabled by default is undesirable. The fourth paragraph talks about it not being possible to remove RC4.
But nothing says what is changing. The release notes section says:
BC> krb5 removes support for several known-bad encryption types. BC> Hopefully users will see no changes.
but it doesn't really say what "removes support" means, or exactly which encryption types will no longer be supported.
Per your follow-up email, I'm not clear on whether you want changes here. If you do, speak up, especially if you have suggestions.
BC> == User Experience ==
BC> Ideally no change! Worst case some users will see krb5 produce BC> error messages about bad enctypes not being able to be used (has no BC> enctype, could not fullfill enctype, etc.). These pains are the BC> feeling of the world grinding forward security-wise.
I think this is an extremely optimistic description of the worst case behavior.
I really don't think that "it won't work and there'll be error messages" is an "extremely optimistic description". If you have langauge changes, I'm happy to hear them out.
Well, the wording doesn't actually say "it won't work", does it? It just says "users will see krb5 produce error messages". You could at least plausibly read that as "it will produce error messages...but work anyway". It would certainly be *clearer* if it explicitly said "stuff may not work without at least some kind of configuration change".
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> Per your follow-up email, I'm not clear on whether you want changes RH> here. If you do, speak up, especially if you have suggestions.
Well, it was just odd that the summary had information not contained at all within the detailed description. Since this was a change that particularly interests me, I skipped straight to the details and then was left with the question about exactly what was changing.
RH> I really don't think that "it won't work and there'll be error RH> messages" is an "extremely optimistic description".
But to be fair, MIT krb5 is not known for having great error output. Not being able to start at all because the K/M has an enctype which is acceptable and not at all deprecated according to the documentation that exists today _and_ failing in the way the software tends to fail (with obscure and sometimes numeric messages) would be... tough. Not that I think anyone would just do dnf system-upgrade on their master KDC.
RH> If you have langauge changes, I'm happy to hear them out.
I'm just not optimistic that it would be much better than "may fail in a completely inexplicable fashion". Having a couple of principals that can't get tickets because they lack a supported enctype is one thing. (I've seen that and the error messages you get are OK, I guess.) But having your entire database not load because it doesn't the enctype on K/M is a different thing entirely. Note that currently things do work fine if the enctype on K/M isn't in supported_enctypes. It doesn't even warn. Presumably because supported_enctypes is related to issuing tickets, not decrypting the database. But if the algorithms are disabled completely, then... I don't know what would happen. That's why I was asking whether the algorithms will still exist.
There is a good document for migrating from old encryption types at https://web.mit.edu/kerberos/krb5-1.16/doc/admin/advanced/retiring-des.html Note that the last paragraph there mentions that single-DES is still basically OK for the K/M principal and it's not required to use allow_weak_crypto to access a database encrypted with single-DES.
RH> That reading only applies to sites with a stash file nearby (though RH> sites set up otherwise remain relatively uncommon in my impression, RH> or are freeipa and will have AES keys).
I guess I can see that if I read it a few times. The idea is that if the stash file is there on the disk, then having raw access to the database probably means you can also just read the stash and so the encryption type doesn't really matter much. If you're not using a stash file then it then becomes more important. Still, rolling over K/M seems relatively simple (though I haven't actually done it).
RH> (Added rollover langauge to "Upgrade/Compatability Impact" section.)
Thanks.
RH> I'm sure you're aware, then, that there's only so much I can do as a RH> packager here.
Well certainly there isn't much you can do to fix old principals on existing systems. But the current versions should be complaining loudly when it has to issue a ticket for a principal that lacks a modern enctype today, if not six months ago.
Also, I'm curious, because I read all of this as "Fedora is going to patch upstream to disable certain algorithms because we don't think the software should support them at all even if upstream does." If that's a correct interpretation, then it's a bit disingenuous to say there's nothing a packager can do when it's the packager who is introducing the change. For example, if we're deviating from upsteam, then it's not a stretch to patch in additional deprecation warnings for existing releases (though that should have been done some time ago if the deprecation is happening in F30). The package could provide additional utilities to check for problematic principals and have them run when the server starts. Certainly people have to check their logs, but the server certainly has enough information to warn.
Of course, if I'm reading that wrong and this is merely an effect of moving to version 1.18 then.... the current devel documentation doesn't seem to say much about deprecation of additional algorithms. Which I know doesn't mean much, but given how kerberos development progresses, I would have expected the complete removal of an encryption type to be telegraphed years in advance.
But after all that, the number of sites running a KDC on Fedora with old principals has to be vanishingly small so I may be the only person who would actually care.
- J<
On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
But to be fair, MIT krb5 is not known for having great error output. Not being able to start at all because the K/M has an enctype which is acceptable and not at all deprecated according to the documentation that exists today _and_ failing in the way the software tends to fail (with obscure and sometimes numeric messages) would be... tough. Not that I think anyone would just do dnf system-upgrade on their master KDC.
Anyone using FreeIPA and upgrading it is doing this, I guess. (Like me...)
Adam Williamson adamwill@fedoraproject.org writes:
On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
But to be fair, MIT krb5 is not known for having great error output. Not being able to start at all because the K/M has an enctype which is acceptable and not at all deprecated according to the documentation that exists today _and_ failing in the way the software tends to fail (with obscure and sometimes numeric messages) would be... tough. Not that I think anyone would just do dnf system-upgrade on their master KDC.
Anyone using FreeIPA and upgrading it is doing this, I guess. (Like me...)
We appreciate that you *do* do this because it means others will hit fewer bugs that they need to roll back! But while krb5 version upgrades are safe, distro upgrades aren't something that can be tested except in a distro context.
Snapshots are always a good idea.
Thanks, --Robbie
On Mon, Jan 07, 2019 at 04:12:47PM -0500, Robbie Harwood wrote:
Adam Williamson adamwill@fedoraproject.org writes:
On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
But to be fair, MIT krb5 is not known for having great error output. Not being able to start at all because the K/M has an enctype which is acceptable and not at all deprecated according to the documentation that exists today _and_ failing in the way the software tends to fail (with obscure and sometimes numeric messages) would be... tough. Not that I think anyone would just do dnf system-upgrade on their master KDC.
Anyone using FreeIPA and upgrading it is doing this, I guess. (Like me...)
We appreciate that you *do* do this because it means others will hit fewer bugs that they need to roll back! But while krb5 version upgrades are safe, distro upgrades aren't something that can be tested except in a distro context.
Distro context? But we are talking about Fedora here, this couldn't be less “distro context”. Distro upgrade is a suggested way to keep one's installation at supportable release (as opposed to reinstall), so distro-upgrade path MUST be tested and working.
Tomasz Torcz tomek@pipebreaker.pl writes:
On Mon, Jan 07, 2019 at 04:12:47PM -0500, Robbie Harwood wrote:
Adam Williamson adamwill@fedoraproject.org writes:
On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
But to be fair, MIT krb5 is not known for having great error output. Not being able to start at all because the K/M has an enctype which is acceptable and not at all deprecated according to the documentation that exists today _and_ failing in the way the software tends to fail (with obscure and sometimes numeric messages) would be... tough. Not that I think anyone would just do dnf system-upgrade on their master KDC.
Anyone using FreeIPA and upgrading it is doing this, I guess. (Like me...)
We appreciate that you *do* do this because it means others will hit fewer bugs that they need to roll back! But while krb5 version upgrades are safe, distro upgrades aren't something that can be tested except in a distro context.
Distro context? But we are talking about Fedora here, this couldn't be less “distro context”. Distro upgrade is a suggested way to keep one's installation at supportable release (as opposed to reinstall), so distro-upgrade path MUST be tested and working.
I don't disagree with any of that. What I mean to say is that if you hold all other package versions the same, krb5 upgrades are always safe - but we can't, as krb5, make guarantees about what any other packages do, especially our dependencies. The testing needs to happen when the distro is assembled - i.e., in the context of the distro, not beforehand.
Thanks, --Robbie
On ma, 14 tammi 2019, Robbie Harwood wrote:
Tomasz Torcz tomek@pipebreaker.pl writes:
On Mon, Jan 07, 2019 at 04:12:47PM -0500, Robbie Harwood wrote:
Adam Williamson adamwill@fedoraproject.org writes:
On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
But to be fair, MIT krb5 is not known for having great error output. Not being able to start at all because the K/M has an enctype which is acceptable and not at all deprecated according to the documentation that exists today _and_ failing in the way the software tends to fail (with obscure and sometimes numeric messages) would be... tough. Not that I think anyone would just do dnf system-upgrade on their master KDC.
Anyone using FreeIPA and upgrading it is doing this, I guess. (Like me...)
We appreciate that you *do* do this because it means others will hit fewer bugs that they need to roll back! But while krb5 version upgrades are safe, distro upgrades aren't something that can be tested except in a distro context.
Distro context? But we are talking about Fedora here, this couldn't be less “distro context”. Distro upgrade is a suggested way to keep one's installation at supportable release (as opposed to reinstall), so distro-upgrade path MUST be tested and working.
I don't disagree with any of that. What I mean to say is that if you hold all other package versions the same, krb5 upgrades are always safe
- but we can't, as krb5, make guarantees about what any other packages
do, especially our dependencies. The testing needs to happen when the distro is assembled - i.e., in the context of the distro, not beforehand.
With FreeIPA we attempt to do that in Fedora and it helped to find quite a few issues with FreeIPA components already. If there are specific additional tests that can be exercised in deploying IPA server and a client and then upgrading both of them in a sequence, using IPA Web UI and other tools, feel free to suggest those scenarios.
For existing tests see https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/tests, tests with freeipa and domain_controller in the names
Jason L Tibbitts III tibbs@math.uh.edu writes:
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> I really don't think that "it won't work and there'll be error RH> messages" is an "extremely optimistic description".
But to be fair, MIT krb5 is not known for having great error output.
I've spent a nontrivial amount of time working on improving that, but am always willing to process more bugs in the documentation/errors area. krb5 only has five bugzillas in Fedora, so don't sweat giving me more to do :)
RH> If you have langauge changes, I'm happy to hear them out.
I'm just not optimistic that it would be much better than "may fail in a completely inexplicable fashion". Having a couple of principals that can't get tickets because they lack a supported enctype is one thing. (I've seen that and the error messages you get are OK, I guess.) But having your entire database not load because it doesn't the enctype on K/M is a different thing entirely. Note that currently things do work fine if the enctype on K/M isn't in supported_enctypes. It doesn't even warn. Presumably because supported_enctypes is related to issuing tickets, not decrypting the database.
Correct.
But if the algorithms are disabled completely, then... I don't know what would happen.
My memory is that it complains about a bad enctype on K/M, but it's been a while. If you want, as part of this change I can make sure it generates a useful error message.
RH> I'm sure you're aware, then, that there's only so much I can do as RH> a packager here.
Well certainly there isn't much you can do to fix old principals on existing systems. But the current versions should be complaining loudly when it has to issue a ticket for a principal that lacks a modern enctype today, if not six months ago.
Yes but also no. The problem is where to complain.
For most users, Kerberos functionally exists in library form (with some CLI for various functions). Libraries can't log anywhere - which is why krb5 has KRB5_TRACE. But unless something isn't working, there's no reason to go look at the KRB5_TRACE output.
For folks running KDCs, we do have existing logging. But as an admin, you know that I'm already logging every issuance - including their etypes. (Now, the raw numbers aren't very helpful unless you're used to looking at them - but that's why I've got https://bugzilla.redhat.com/show_bug.cgi?id=1664156 .) This is pretty chatty! I don't think that putting more messages in here would be useful, but I could be persuaded otherwise.
Also, I'm curious, because I read all of this as "Fedora is going to patch upstream to disable certain algorithms because we don't think the software should support them at all even if upstream does." If that's a correct interpretation, then it's a bit disingenuous to say there's nothing a packager can do when it's the packager who is introducing the change. For example, if we're deviating from upsteam, then it's not a stretch to patch in additional deprecation warnings for existing releases (though that should have been done some time ago if the deprecation is happening in F30).
The comment you're quoting is intended to refer to pushing changes to non-KDC machines in the network (such as keytabs). My point was that I can't make everything seamless. See above for a discussion of "deprecation warnings".
The package could provide additional utilities to check for problematic principals and have them run when the server starts. Certainly people have to check their logs, but the server certainly has enough information to warn.
As previously stated, I'm investigating providing tooling.
Of course, if I'm reading that wrong and this is merely an effect of moving to version 1.18 then.... the current devel documentation doesn't seem to say much about deprecation of additional algorithms. Which I know doesn't mean much, but given how kerberos development progresses, I would have expected the complete removal of an encryption type to be telegraphed years in advance.
Arguably, we telegraphed single-DES removal in 2009 (and there have been two RFCs about this that I cited), but you're right that Kerberos has never been a "move fast and deprecate things" kind of situation.
This has worked to our detriment and we shouldn't let tradition be the enemy of improvement.
I don't want to go on too long about this, but the perception that Kerberos is somehow "maintenance mode" isn't quite right. A lot of the changes we've made recently haven't really been end-user visible (like SPAKE), but things in the pipe will be more so (this crypto change, FIDO/U2F support, etc.).
But after all that, the number of sites running a KDC on Fedora with old principals has to be vanishingly small so I may be the only person who would actually care.
Maybe in Fedora, but we're leading here. The concerns you're raising are important for other distros that make these changes down the line. I may not be happy about them (because they make this harder), but we still need to talk them through :)
For Fedora specifically, I do think we have a lot of end users (even just from kinit followed by fedpkg), a smaller-but-not-insignificant number of service users (people running Kerberized services), and a pretty small number of folks running full-on KDCs (whose testing is very helpful) in comparison to other distros.
Thanks, --Robbie
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> I've spent a nontrivial amount of time working on improving that, RH> but am always willing to process more bugs in the RH> documentation/errors area.
I know, and I don't mean to denigrate any work that's been done in making the MIT KRB stack better. It's certainly a whole lot better than it used to be. I've been using it since, I don't know, sometime in the previous century.
RH> My memory is that it complains about a bad enctype on K/M, but it's RH> been a while. If you want, as part of this change I can make sure RH> it generates a useful error message.
Certainly don't add a message on my account, but on the general principle that if krb5kdc can't start at all then it would try to say why.
Well certainly there isn't much you can do to fix old principals on existing systems. But the current versions should be complaining loudly when it has to issue a ticket for a principal that lacks a modern enctype today, if not six months ago.
RH> Yes but also no. The problem is where to complain.
Maybe I wasn't clear, but I was saying that maybe the KDC should complain when it has to issue a ticket with a key type that's deprecated.
RH> For folks running KDCs, we do have existing logging.
Then I have to say that I'm not seeing it. For some background, here's what I was recently working on.
I set up a test slave KDC with Fedora 29 and did some testing against it. Trying to get a ticket using an old principal generated this bit of log:
Dec 07 19:08:30 XXX krb5kdc[846](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) YYY: NEEDED_PREAUTH: ZZZ for krbtgt/QQQ, Additional pre-authentication required Dec 07 19:08:30 XXX krb5kdc[846](info): DISPATCH: repeated (retransmitted?) request from YYY, resending previous response
The last line is repeated a number of times. There are no other logs related to this principal.
The problem is that the principal in question hasn't been touched in twelve years (which I guess you can see from the list of etypes.) That coupled with the change (which I don't think was your doing) to /etc/crypto-policies/back-ends/krb5.config to remove des3-cbc-sha1 caused the KDC to fail to issue those tickets.
So obviously I figured out what the problem was, but the logged info wasn't remotely helpful in figuring that out.
Now, krb5kdc couldn't really know that F29 was going to drop something from supported_enctypes, and there's no setting like "deprecated_enctypes" which would have told it what to complain about (though that doesn't sound like a terrible idea). But if the server had said what the problem was, or there was something which bugged me about principals having only deprecated etypes, then I wouldn't have had any confusion at all.
RH> But as an admin, you know that I'm already logging every issuance - RH> including their etypes.
Yes, but of course that doesn't help you if it doesn't issue anything. I guess if you know the list of etypes by heart you could see the problem from the logged info. And if it does issue something today but will fail in F30 due to removal of some etypes, there's no complaint at all.
RH> I don't think that putting more messages in here would be useful, RH> but I could be persuaded otherwise.
Doesn't technically have to be more messages, even adding more information to the existing message would help. Adding ", Only deprecated etypes issued" to the end of the ISSUE lines would sort of be in keeping with the existing format.
But really, whether it's in something that gets logged or some separate tool I can run or something, it would just be nice to have some established procedure to know that that you're going to be hit by expiring crypto before you actually get hit by it.
Of course, if I'm reading that wrong and this is merely an effect of moving to version 1.18 then.... the current devel documentation doesn't seem to say much about deprecation of additional algorithms. Which I know doesn't mean much, but given how kerberos development progresses, I would have expected the complete removal of an encryption type to be telegraphed years in advance.
RH> Arguably, we telegraphed single-DES removal in 2009 (and there have RH> been two RFCs about this that I cited), but you're right that RH> Kerberos has never been a "move fast and deprecate things" kind of RH> situation.
Yes, single-DES removal has been yelled about loudly for a long time. There's a section in every release announcement about it. But isn't the current proposal about removing more than just 1DES?
RH> I don't want to go on too long about this, but the perception that RH> Kerberos is somehow "maintenance mode" isn't quite right.
I certainly don't harbor the belief that MIT Kerberos is just in maintenance mode. I was trying to say that it moves carefully and with significant effort put into backwards compatibility.
- J<
Jason L Tibbitts III tibbs@math.uh.edu writes:
"RH" == Robbie Harwood rharwood@redhat.com writes:
Well certainly there isn't much you can do to fix old principals on existing systems. But the current versions should be complaining loudly when it has to issue a ticket for a principal that lacks a modern enctype today, if not six months ago.
RH> For folks running KDCs, we do have existing logging.
The problem is that the principal in question hasn't been touched in twelve years (which I guess you can see from the list of etypes.) That coupled with the change (which I don't think was your doing) to /etc/crypto-policies/back-ends/krb5.config to remove des3-cbc-sha1 caused the KDC to fail to issue those tickets.
Ah, I see, you're talking about the case when the enctype is already not permitted. That all makes sense then.
RH> I don't think that putting more messages in here would be useful, RH> but I could be persuaded otherwise.
Doesn't technically have to be more messages, even adding more information to the existing message would help. Adding ", Only deprecated etypes issued" to the end of the ISSUE lines would sort of be in keeping with the existing format.
But really, whether it's in something that gets logged or some separate tool I can run or something, it would just be nice to have some established procedure to know that that you're going to be hit by expiring crypto before you actually get hit by it.
This is good feedback, thanks. My plan is to try for an external "realm check" type tool - but I need to make it first :)
Of course, if I'm reading that wrong and this is merely an effect of moving to version 1.18 then.... the current devel documentation doesn't seem to say much about deprecation of additional algorithms. Which I know doesn't mean much, but given how kerberos development progresses, I would have expected the complete removal of an encryption type to be telegraphed years in advance.
RH> Arguably, we telegraphed single-DES removal in 2009 (and there have RH> been two RFCs about this that I cited), but you're right that RH> Kerberos has never been a "move fast and deprecate things" kind of RH> situation.
Yes, single-DES removal has been yelled about loudly for a long time. There's a section in every release announcement about it. But isn't the current proposal about removing more than just 1DES?
It may help understanding my motivations here to know that upstream won't consider removal until some distro has done so and they have a sense of what it entails. Until then, it's a bit Catch-22: no one wants to divert from upstream and upstream doesn't want to break downstreams... We've yet to remove anything of import from any Kerberos implementation; MIT still includes Kerberos4 support, for instance.
The MIT-specific telegraphing is single-DES specific. The RFCs (which also include RC4 and 3DES) are applicable to the entire ecosystem. But ignoring all that, fundamentally I don't want to do this same process three times - and I can't imagine admins do, either. Which is to say: I think that the only telegraphing I can meaningfully do is administrative (RFCs, release notes, change proposals, etc.), and then try to smooth the roadbumps.
Thanks, --Robbie
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> Ah, I see, you're talking about the case when the enctype is already RH> not permitted. That all makes sense then.
Right. Basically, if any one of these:
* Warnings in previous versions about principals without modern etypes * Logging in the new version to say why tickets wouldn't issue for principals with old etypes * A checkup tool in either the old or current versions to tell me what's gone wrong
had existed then there would have been no confusion. Certainly I was able to figure it out but... if someone had just done an OS update without proper testing then they could be in a pretty bad position.i
So basically the big issue as I see it is that there's simply nothing to tell you that things are going to break, and after the update there's nothing that tells you why things are broken. And I was concerned that if some encryption routines go away completely then it would be possible to be in a state where you can't even decrypt the database.
- J<
On Tue, Jan 08, 2019 at 04:45:53PM -0600, Jason L Tibbitts III wrote:
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> Ah, I see, you're talking about the case when the enctype is already RH> not permitted. That all makes sense then.
Hi,
after re-reading this thread, I'm still unclear on some issues. Please correct me if I'm wrong.
- The plan is to patch the Fedora package to remove support for some algorithms above and beyond what upstream is removing right now. - Current implementation in F29 does not warn that those algorithms will become unimplemented. - Because of the combination of two previous points, users who simply upgrade to F30 without paying attention will have to temporarily downgrade to the F29 version, perform key roll-over, and only then upgrade.
If this understanding is correct, this seems like a trap for the unwary.
What about turning the removal into a big warning in the logs each time kdc is started and needs to use those deprecated algorithms in F30, and only doing the removal in F31?
Zbyszek
Right. Basically, if any one of these:
- Warnings in previous versions about principals without modern etypes
- Logging in the new version to say why tickets wouldn't issue for principals with old etypes
- A checkup tool in either the old or current versions to tell me what's gone wrong
had existed then there would have been no confusion. Certainly I was able to figure it out but... if someone had just done an OS update without proper testing then they could be in a pretty bad position.i
So basically the big issue as I see it is that there's simply nothing to tell you that things are going to break, and after the update there's nothing that tells you why things are broken. And I was concerned that if some encryption routines go away completely then it would be possible to be in a state where you can't even decrypt the database.
- J<
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
On Tue, Jan 08, 2019 at 04:45:53PM -0600, Jason L Tibbitts III wrote:
> "RH" == Robbie Harwood rharwood@redhat.com writes:
RH> Ah, I see, you're talking about the case when the enctype is already RH> not permitted. That all makes sense then.
Hi,
after re-reading this thread, I'm still unclear on some issues. Please correct me if I'm wrong.
- The plan is to patch the Fedora package to remove support for some algorithms above and beyond what upstream is removing right now.
Upstream has never removed an algorithm. Hopefully the Fedora changes will allow us to do so, both by providing the code, and by showing that the fallout isn't catastrophic.
Current implementation in F29 does not warn that those algorithms will become unimplemented.
Because of the combination of two previous points, users who simply upgrade to F30 without paying attention will have to temporarily downgrade to the F29 version, perform key roll-over, and only then upgrade.
That's not a consequence of the previous two points but is nontheless true.
If this understanding is correct, this seems like a trap for the unwary.
So is keeping any of these algorithms around.
What about turning the removal into a big warning in the logs each time kdc is started and needs to use those deprecated algorithms in F30,
Still in progress. It may prove too invasive to be feasible.
If I can't, then I'll be providing external tooling that will check for some common-ish legacy cruft in setups that might break.
Also, I don't know if you've ever looked at krb5kdc logs, but they're really chatty - I doubt anyone reads regularly them unless they're trying to debug a problem.
and only doing the removal in F31?
It may come to that. If it does, then I want to be sure everything is lined up for it, and we won't slip to fc32 or later.
Finally, I do want to call out here that announcing crypto deprecations in the previous release is not the norm for Fedora. Since fc20, there have been seven crypto Changes (mostly to do with adding crypto-policies support), none of which performed any notification other than what was in the release notes:
https://fedoraproject.org/wiki/Changes/RemoveSSL3andRc4 https://fedoraproject.org/wiki/Changes/CryptoPolicyKrb5 https://fedoraproject.org/wiki/Changes/NSSCryptoPolicies https://fedoraproject.org/wiki/Changes/OpenSSH_Crypto_Policy https://fedoraproject.org/wiki/Changes/Remove_SSH-1_from_OpenSSH https://fedoraproject.org/wiki/Changes/OpenSSH_Server_Crypto_Policy https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
Furthermore, at no point have there even been Change/Feature pages for all the TLS changes relating to vulnerable ciphers.
Thanks, --Robbie
after re-reading this thread, I'm still unclear on some issues. Please correct me if I'm wrong.
- The plan is to patch the Fedora package to remove support for some algorithms above and beyond what upstream is removing right now.
Upstream has never removed an algorithm. Hopefully the Fedora changes will allow us to do so, both by providing the code, and by showing that the fallout isn't catastrophic.
Current implementation in F29 does not warn that those algorithms will become unimplemented.
Because of the combination of two previous points, users who simply upgrade to F30 without paying attention will have to temporarily downgrade to the F29 version, perform key roll-over, and only then upgrade.
That's not a consequence of the previous two points but is nontheless true.
If this understanding is correct, this seems like a trap for the unwary.
So is keeping any of these algorithms around.
Maybe a middle ground can be found here:
- remove support for legacy algorithms - let it blow in production in a helpful manner - provide legacy algorithms support for key roll-over only
For the last point, that could either be a version of the library outside of the default library path, or a static program dedicated to that task. This way you pull the rug under the users' feet but you don't break their legs in the process. And maybe this is something that should be discussed upstream before making such a change in Fedora so they finally have a means to remove legacy algorithms and still offer an upgrade path towards proper crypto for existing installations.
Dridi
Jason L Tibbitts III tibbs@math.uh.edu writes:
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> Ah, I see, you're talking about the case when the enctype is already RH> not permitted. That all makes sense then.
Right. Basically, if any one of these:
- Warnings in previous versions about principals without modern etypes
- Logging in the new version to say why tickets wouldn't issue for principals with old etypes
- A checkup tool in either the old or current versions to tell me what's gone wrong
had existed then there would have been no confusion. Certainly I was able to figure it out but... if someone had just done an OS update without proper testing then they could be in a pretty bad position.i
So basically the big issue as I see it is that there's simply nothing to tell you that things are going to break, and after the update there's nothing that tells you why things are broken. And I was concerned that if some encryption routines go away completely then it would be possible to be in a state where you can't even decrypt the database.
I have added warnings for:
- deprecated enctype issuance on KDC - deprecated enctypes in klist output - deprecated enctype in stash / K/M
to the latest rawhide builds (krb5-1.17-3.fc30).
If I backport this to fc29, will that assuage people's concerns? Or do I need to defer the change until fc31?
Thanks, --Robbie
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> If I backport this to fc29, will that assuage people's concerns?
I think it would certainly help and I wouldn't complain. In fact, I'd love to start running that as soon as I can. However, it wouldn't help anyone who does a (supposedly supported) F28->F30 update. I do not know if that is a significant concern.
- J<
Jason L Tibbitts III tibbs@math.uh.edu writes:
"RH" == Robbie Harwood rharwood@redhat.com writes:
RH> If I backport this to fc29, will that assuage people's concerns?
I think it would certainly help and I wouldn't complain. In fact, I'd love to start running that as soon as I can. However, it wouldn't help anyone who does a (supposedly supported) F28->F30 update. I do not know if that is a significant concern.
Nothing would break, so it's technically correct to leave out right now I think. Since my f28 builds match f29 right now, I just did both:
fc28: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0956d60ffd fc29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d226c3ecf1
I've also updated the change page to mention that these warnings are present (and the versions they start appearing in).
Thanks, --Robbie
On Thu, Jan 17, 2019 at 05:01:45PM -0500, Robbie Harwood wrote:
Jason L Tibbitts III tibbs@math.uh.edu writes:
> "RH" == Robbie Harwood rharwood@redhat.com writes:
RH> If I backport this to fc29, will that assuage people's concerns?
I think it would certainly help and I wouldn't complain. In fact, I'd love to start running that as soon as I can. However, it wouldn't help anyone who does a (supposedly supported) F28->F30 update. I do not know if that is a significant concern.
Nothing would break, so it's technically correct to leave out right now I think. Since my f28 builds match f29 right now, I just did both:
fc28: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0956d60ffd fc29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d226c3ecf1
I've also updated the change page to mention that these warnings are present (and the versions they start appearing in).
So that gives 4+ months of warnings (assuming the F28 and F29 updates go to stable soon), which should be enough for people to take notice. It seems it's fine to remove in F30 as originally planned.
Zbyszek
devel@lists.stg.fedoraproject.org
-
Adam Williamson -
Alexander Bokovoy -
Ben Cotton -
Dridi Boukelmoune -
Jason L Tibbitts III -
Nikos Mavrogiannopoulos -
Raphael Groner -
Robbie Harwood -
Tomasz Torcz -
Zbigniew Jędrzejewski-Szmek