Sumit, thank you very much for this. Very helpful, but I am still not seeing the problem So at first I will try with the following in krb5.confkdc=server1 <--shut off on the network#kdc=server2 <--shut off on the network and commented out in krb5.confkdc=server3 <--up and running KRB5_TRACE=/dev/stdout kinit user@test.domain [12583] 1501113245.556036: Getting initial credentials for user@test.domain [12583] 1501113245.556244: Sending request (181 bytes) to test.domain [12583] 1501113245.556282: Resolving hostname server1 [12583] 1501113245.557235: Sending initial UDP request to dgram ip_addr_server1:88 [12583] 1501113246.558328: Resolving hostname server3 [12583] 1501113246.558974: Sending initial UDP request to dgram ip_addr_server3:88 [12583] 1501113246.729059: Received answer (275 bytes) from dgram ip_addr_server3:88 [12583] 1501113246.729111: Response was from master KDC [12583] 1501113246.729155: Received error from KDC: -1765328359/Additional pre-authentication required [12583] 1501113246.729219: Processing preauth types: 136, 19, 2, 133 [12583] 1501113246.729245: Selected etype info: etype aes256-cts, salt "pY;=XB+5_*EjJC%S", params "" [12583] 1501113246.729254: Received cookie: MIT Password for user@test.domain <--get prompted for password Now with all three kdc uncommentedkdc=server1 <-shut off and uncommentedkdc=server2 <--shut off and uncommentedkdc=server3 <--up and running KRB5_TRACE=/dev/stdout kinit user@test.domain [12536] 1501112935.251721: Getting initial credentials for user@test.domain [12536] 1501112935.251917: Sending request (181 bytes) to test.domain [12536] 1501112935.251956: Resolving hostname server1 [12536] 1501112935.252875: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112936.253962: Resolving hostname server2 [12536] 1501112936.255680: Retrying AS request with master KDC [12536] 1501112936.255699: Getting initial credentials for user@test.domain [12536] 1501112936.255763: Sending request (181 bytes) to test.domain (master) [12536] 1501112936.255779: Resolving hostname server1 [12536] 1501112936.256379: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112937.257451: Resolving hostname server2 kinit: Invalid argument while getting initial credentials
So as you can see server3 is never even tried for authentication. One of my theories is that there might be maximum number of kdc's to tryor maximum total authentication timeout?! Just a wild guess as I'm reaching for straws. -------------------------------My other question with regards to how sssd and krb work together was prompted by sssd.confipa_server= _srv_ option which supposed to find available IPA servers from DNS records. We do indeed have this option set in sssd.confand are able to resolve server1,server2 server3 when querying for following records _ldap._tcp _kerberos._tcp _kerberos._udp _kerberos-master._tcp _kerberos-master._udp _ntp._udp
If the _srv_ is enabled then am i correct in assuming that we wouldn't even need kdc= records in krb5.conf ??I tried removing kdc= linesand was unable to authenticate.
If the _srv_ is enabled then am i correct in assuming that we wouldn't even need kdc= records in krb5.conf ??>I tried removing kdc= linesand was unable to authenticate.
In my experience, sssd relies upon the local kerberos stack. Maybe others have different experiences. mpapet
On Thu, Jul 27, 2017 at 02:15:33AM +0000, Michael Papet via FreeIPA-users wrote:
If the _srv_ is enabled then am i correct in assuming that we wouldn't even need kdc= records in krb5.conf ??>I tried removing kdc= linesand was unable to authenticate.
In my experience, sssd relies upon the local kerberos stack. Maybe others have different experiences. mpapet
This really depends on what domain the user is authenticating from.
If the user comes from the joined domain, then currently sssd resolves the KDC on its own and puts the address of the KDC server into the list of KDC addresses known by libkrb5 via a locator plugin: https://jhrozek.wordpress.com/2014/11/04/how-does-sssd-interact-with-tools-l...
But for users from trusted domains (typically when talking about IPA-AD trusts), this is currently not done and sssd just calls a kinit equivalent and pretty much relies on what is already configured in krb5.conf.
Jacub, yes we do have a one way trust between AD->FreeIPA. That explainswhy krb5.conf is used instead of the sssd.conf _srv_ to retrieve DNS records. Can you also please comment on why I'm only getting lookups on the first two kdc's listed in krb5.conf thank you so much and I'm bookmarking your blog.
Date: Thu, 27 Jul 2017 10:01:11 +0200 From: Jakub Hrozek jhrozek@redhat.com Subject: [Freeipa-users] Re: Krb5.conf only sees first two kdc servers To: freeipa-users@lists.fedorahosted.org Message-ID: 20170727080111.ekj3mqbuilkrlxpa@hendrix Content-Type: text/plain; charset=iso-8859-1
On Thu, Jul 27, 2017 at 02:15:33AM +0000, Michael Papet via FreeIPA-users wrote:
If the _srv_ is enabled then am i correct in assuming that we wouldn't even need kdc= records in krb5.conf ??>I tried removing kdc= linesand was unable to authenticate.
In my experience, sssd relies upon the local kerberos stack. Maybe others have different experiences. mpapet
This really depends on what domain the user is authenticating from.
If the user comes from the joined domain, then currently sssd resolves the KDC on its own and puts the address of the KDC server into the list of KDC addresses known by libkrb5 via a locator plugin: https://jhrozek.wordpress.com/2014/11/04/how-does-sssd-interact-with-tools-l...
But for users from trusted domains (typically when talking about IPA-AD trusts), this is currently not done and sssd just calls a kinit equivalent and pretty much relies on what is already configured in krb5.conf.
On Thu, Jul 27, 2017 at 02:19:38PM +0000, pgb205 via FreeIPA-users wrote:
Jacub, yes we do have a one way trust between AD->FreeIPA. That explainswhy krb5.conf is used instead of the sssd.conf _srv_ to retrieve DNS records. Can you also please comment on why I'm only getting lookups on the first two kdc's listed in krb5.conf
I'm really not sure. I would say the same what Sumit did in his reply (and he actually tested his setup) and same as Sumit, I'm not aware of any limits.
It would be nice to illustrate the problems you are seeing with logs..
small update as I've managed to narrow things down. the following will NOT work [realms]kdc=server1kdc=server2kdc=server3kdc=server4master_kdc=server1master_kdc=server2master_kdc=server3master_kdc=server4 when server1 and server2 are down on the network. server3 and server4 are up and functional but the following WILL work [realms]kdc=server1kdc=server2kdc=server3kdc=server4master_kdc=server1# master_kdc=server2master_kdc=server3master_kdc=server4 So the problem seems to be with having more than one non-functioning master_kdc ??! One seems to be ok but more than one and kinit chokes.
Jakub, After doing some more troubleshooting I agree that there is no problem with having multiple kdc servers.However, having more than one non-functional master_kdc is what's causing the failure. server1 and server2 are down. server3 is up this works.
kdc=server1kdc=server2kdc=server3master_kdc=server1#master_kdc=server2master_kdc=server3 this will fail: kdc=server1kdc=server2kdc=server3master_kdc=server1master_kdc=server2master_kdc=server3
I've provided the log KRB5_TRACE output and it doesn't seem like kinit is even attempting to reach server3 if it sees that the first two master_kdc are down.
Here is the log that I sent in yesterday. With server1 and server2 down, but server3 up.
kdc=server1 kdc=server2 kdc=server3 kdc_master=server1 kdc_master=server2 kdc_master=server3
kinit tries server1 and server2 but never even attempts server3 KRB5_TRACE=/dev/stdout kinit user(a)test.domain [12536] 1501112935.251721: Getting initial credentials for user(a)test.domain [12536] 1501112935.251917: Sending request (181 bytes) to test.domain [12536] 1501112935.251956: Resolving hostname server1 [12536] 1501112935.252875: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112936.253962: Resolving hostname server2 [12536] 1501112936.255680: Retrying AS request with master KDC [12536] 1501112936.255699: Getting initial credentials for user(a)test.domain [12536] 1501112936.255763: Sending request (181 bytes) to test.domain (master) [12536] 1501112936.255779: Resolving hostname server1 [12536] 1501112936.256379: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112937.257451: Resolving hostname server2 kinit: Invalid argument while getting initial credentials
kinit with following configuration will work, however. kdc=server1 kdc=server2 kdc=server3 kdc_master=server1 # kdc_master=server2 kdc_master=server3
On Fri, Jul 28, 2017 at 02:05:05AM -0000, pgb 205 via FreeIPA-users wrote:
Here is the log that I sent in yesterday. With server1 and server2 down, but server3 up.
kdc=server1 kdc=server2 kdc=server3 kdc_master=server1 kdc_master=server2 kdc_master=server3
kinit tries server1 and server2 but never even attempts server3 KRB5_TRACE=/dev/stdout kinit user(a)test.domain [12536] 1501112935.251721: Getting initial credentials for user(a)test.domain [12536] 1501112935.251917: Sending request (181 bytes) to test.domain [12536] 1501112935.251956: Resolving hostname server1 [12536] 1501112935.252875: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112936.253962: Resolving hostname server2 [12536] 1501112936.255680: Retrying AS request with master KDC [12536] 1501112936.255699: Getting initial credentials for user(a)test.domain [12536] 1501112936.255763: Sending request (181 bytes) to test.domain (master) [12536] 1501112936.255779: Resolving hostname server1 [12536] 1501112936.256379: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112937.257451: Resolving hostname server2 kinit: Invalid argument while getting initial credentials
kinit with following configuration will work, however. kdc=server1 kdc=server2 kdc=server3 kdc_master=server1 # kdc_master=server2 kdc_master=server3
Interesting, but I admit I'm getting out of my depth here..
Perhaps some of the kerberos maintainers would like to chime in here?
I'd appreciate any advise on how to even troubleshoot this further. KRB_TRACE is the only utility that I'm aware to assist with this.
pgb 205 via FreeIPA-users writes:
Here is the log that I sent in yesterday. With server1 and server2 down, but server3 up.
kdc=server1 kdc=server2 kdc=server3 kdc_master=server1 kdc_master=server2 kdc_master=server3
kdc_master isn't a valid directive for krb5.conf (we call it master_kdc). Can you show your entire krb5.conf, including [realms] and [libdefaults] sections?
kinit tries server1 and server2 but never even attempts server3 KRB5_TRACE=/dev/stdout kinit user(a)test.domain
I assume "(a)" is standing in for '@'?
[12536] 1501112935.251721: Getting initial credentials for user(a)test.domain [12536] 1501112935.251917: Sending request (181 bytes) to test.domain [12536] 1501112935.251956: Resolving hostname server1 [12536] 1501112935.252875: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112936.253962: Resolving hostname server2 [12536] 1501112936.255680: Retrying AS request with master KDC
Alright, so something spooks krb5 here, it looks like. I need to see the whole krb5.conf to have a better idea, but:
- is udp_preference_limit set? - is one of these configured for KKDCP? - is the DNS for server2 weird in some way? - same question but for server3?
Can you tell me what the OS/Kerberos versions are for server1, server2, and server3? Also the OS/krb5 version/sssd version for the client you're using.
[12536] 1501112936.255699: Getting initial credentials for user(a)test.domain [12536] 1501112936.255763: Sending request (181 bytes) to test.domain (master) [12536] 1501112936.255779: Resolving hostname server1 [12536] 1501112936.256379: Sending initial UDP request to dgram server1_ip:88 [12536] 1501112937.257451: Resolving hostname server2 kinit: Invalid argument while getting initial credentials
Yeah, I suspect getaddrinfo() returns something weird for server2. If you can, I'd suggest getting the return values from it; if you're not comfortable doing that, I can bake you a shim that'll print out that information.
kinit with following configuration will work, however. kdc=server1 kdc=server2 kdc=server3 kdc_master=server1 # kdc_master=server2 kdc_master=server3
See above; as written this isn't different from the configuration above (krb5 will ignore lines it doesn't recognize). Assuming you meant "master_kdc" there: this presumably because is because it never retries server2 after switching to querying masters, and instead goes on to server3.
Thanks, --Robbie
freeipa-users@lists.fedorahosted.org
-
Jakub Hrozek -
Michael Papet -
pgb 205 -
pgb205
-
Robbie Harwood