Dear IPA Gurus
I have a client that's incapable of joining the FreeIPA realm, it's in a different DNS sub-zone but is in the same realm. I get the feeling that there's a kerberos principal missing somewhere to get this all to work, but I can't quite see where it might be. Simple authentication ldapsearch using cn=Directory Manager functions perfectly well to the ipa host in question, however anonymous binds are disabled. I'm not clear why this wouldn't be working.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote:
Dear IPA Gurus
I have a client that's incapable of joining the FreeIPA realm, it's in a different DNS sub-zone but is in the same realm. I get the feeling that there's a kerberos principal missing somewhere to get this all to work, but I can't quite see where it might be. Simple authentication ldapsearch using cn=Directory Manager functions perfectly well to the ipa host in question, however anonymous binds are disabled. I'm not clear why this wouldn't be working.
From the above it is unclear what your problem is.
Can you show what exactly is failing? ipa-client-install is failing? Show logs from /var/log/ipaclient-install.log. You aren't using FreeIPA enrollment? How exactly did you try to enroll that client? Show sequence of commands you ran.
It is not easy to help with no logs and exact steps tried.
Dear Alexander,
Sorry, yes indeed using ipa-client-install. The ipaclient-install.log should be attached, I can upload to dropbox if needed. Discovery happens succesfully, but LDAP GSSAPI authentication is failing for some reason.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 14:27, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear IPA Gurus
I have a client that's incapable of joining the FreeIPA realm, it's in a different DNS sub-zone but is in the same realm. I get the feeling that there's a kerberos principal missing somewhere to get this all to work, but I can't quite see where it might be. Simple authentication ldapsearch using cn=Directory Manager functions perfectly well to the ipa host in question, however anonymous binds are disabled. I'm not clear why this wouldn't be working. From the above it is unclear what your problem is.
Can you show what exactly is failing? ipa-client-install is failing? Show logs from /var/log/ipaclient-install.log. You aren't using FreeIPA enrollment? How exactly did you try to enroll that client? Show sequence of commands you ran.
It is not easy to help with no logs and exact steps tried.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote:
Dear Alexander,
Sorry, yes indeed using ipa-client-install. The ipaclient-install.log should be attached, I can upload to dropbox if needed. Discovery happens succesfully, but LDAP GSSAPI authentication is failing for some reason.
Sorry! I didn't check the attachments, this was my fault!
I'll look later tonight.
Dear Alexander,
Thank you, there was a significant time discrepancy between the server and everything else, but having fixed that the problem is presenting identically. Just in case that comes up as a first point-of-call.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 14:39, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear Alexander,
Sorry, yes indeed using ipa-client-install. The ipaclient-install.log should be attached, I can upload to dropbox if needed. Discovery happens succesfully, but LDAP GSSAPI authentication is failing for some reason. Sorry! I didn't check the attachments, this was my fault!
I'll look later tonight.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ma, 11 maalis 2019, Alexander Bokovoy via FreeIPA-users wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote:
Dear Alexander,
Sorry, yes indeed using ipa-client-install. The ipaclient-install.log should be attached, I can upload to dropbox if needed. Discovery happens succesfully, but LDAP GSSAPI authentication is failing for some reason.
Sorry! I didn't check the attachments, this was my fault!
I'll look later tonight.
.. I think the issue is that your configuration is definitely broken.
in ipaclient-install.log we can see DNS SRV record that has weird name ipa-a.virt.in.bmrc.ox.ac.uk.virt.in.bmrc.ox.ac.uk.
2019-03-11T12:30:58Z DEBUG Search DNS for SRV record of _kerberos._udp.virt.in.bmrc.ox.ac.uk 2019-03-11T12:30:58Z DEBUG DNS record found: 0 100 88 ipa-a.virt.in.bmrc.ox.ac.uk.virt.in.bmrc.ox.ac.uk. 2019-03-11T12:30:58Z DEBUG DNS record found: 0 100 88 ipa-b.virt.in.bmrc.ox.ac.uk.
For LDAP discovery then we are OK:
2019-03-11T12:30:58Z DEBUG Start searching for LDAP SRV record in "virt.in.bmrc.ox.ac.uk" (Validating DNS Discovery) and its sub-domains 2019-03-11T12:30:58Z DEBUG Search DNS for SRV record of _ldap._tcp.virt.in.bmrc.ox.ac.uk 2019-03-11T12:30:58Z DEBUG DNS record found: 0 100 389 ipa-a.virt.in.bmrc.ox.ac.uk. 2019-03-11T12:30:58Z DEBUG DNS record found: 0 100 389 ipa-b.virt.in.bmrc.ox.ac.uk. 2019-03-11T12:30:58Z DEBUG DNS validated, enabling discovery
However, there seem to be some issue with DNS setup foor ipa-b.virt.$domain machine -- is this a CNAME?
In the ipaclient-install.log we see that admin user can get an initial ticket granting ticket just fine:
2019-03-11T12:31:04Z DEBUG Initializing principal admin@IN.BMRC.OX.AC.UK using password 2019-03-11T12:31:04Z DEBUG Starting external process 2019-03-11T12:31:04Z DEBUG args=/usr/bin/kinit admin@IN.BMRC.OX.AC.UK -c /tmp/krbccEqCmTM/ccache 2019-03-11T12:31:04Z DEBUG Process finished, return code=0 2019-03-11T12:31:04Z DEBUG stdout=Password for admin@IN.BMRC.OX.AC.UK:
2019-03-11T12:31:04Z DEBUG stderr=
But when trying to authenticate to LDAP with SASL GSSAPI we fail:
2019-03-11T12:31:04Z DEBUG trying to retrieve CA cert via LDAP from ipa-b.virt.in.bmrc.ox.ac.uk 2019-03-11T12:31:04Z DEBUG get_ca_certs_from_ldap() error: Insufficient access: Invalid credentials 2019-03-11T12:31:04Z DEBUG Insufficient access: Invalid credentials
In KDC logs we see that we requested a service ticket for ldap/ipa-b.in.bmrc.ox.ac.uk rather than for ldap/ipa-b.virt.in.bmrc.ox.ac.uk:
Mar 11 12:31:06 ipa-b.in.bmrc.ox.ac.uk krb5kdc[5701](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.248.2: ISSUE: authtime 1552307464, etypes {rep=18 tkt=18 ses=18}, +admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK
Note that ldap/ipa-b.$domain looks like a correct Kerberos service principal because KDC knows about it. However, this is definitely not the same principal as used by the LDAP server itself as LDAP server cannot use own key to decode the service ticket sent by the client, thus resulting in 'Invalid credentials'.
So, you need to look at what you have define as a service principal ldap/* and what you have defined in DNS for that LDAP server.
Can you also look at /etc/dirsrv/ds.keytab on ipa-b server? Use 'klist -kt /etc/dirsrv/ds.keytab'.
Dear Alexander,
klist -kt /etc/dirsrv/ds.keytab Keytab name: FILE:/etc/dirsrv/ds.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK
This is a bit non-standard i understand, but so far this configuration is working ok. I guess the issue is that the ticket is being issued for the wrong domain. [cid:F8DF5B93-5D52-46D5-88AC-E9BEA54760FD@in.bmrc.ox.ac.uk]
I've attached a screenshot of the DNS configuration for the sub-zone.
Our intention here is to ensure that the DNS entry and host for the IPA server within a different sub-zone and subnet resolves to a single IP for speed. So a "host" has been created for each of the interfaces, all of the respective kerberos principals for the host services (ldap in this case) and then a new certificate issued with the alt names on it to allow for LDAPS. This works well, right up until the point of GSSAPI getting involved. There must be a piece of the puzzle we're missing here!
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 14:58, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
klist -kt /etc/dirsrv/ds.keytab
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote:
Dear Alexander,
klist -kt /etc/dirsrv/ds.keytab Keytab name: FILE:/etc/dirsrv/ds.keytab KVNO Timestamp Principal
1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK
This is a bit non-standard i understand, but so far this configuration is working ok. I guess the issue is that the ticket is being issued for the wrong domain. [cid:F8DF5B93-5D52-46D5-88AC-E9BEA54760FD@in.bmrc.ox.ac.uk]
I've attached a screenshot of the DNS configuration for the sub-zone.
Our intention here is to ensure that the DNS entry and host for the IPA server within a different sub-zone and subnet resolves to a single IP for speed. So a "host" has been created for each of the interfaces, all of the respective kerberos principals for the host services (ldap in this case) and then a new certificate issued with the alt names on it to allow for LDAPS. This works well, right up until the point of GSSAPI getting involved. There must be a piece of the puzzle we're missing here!
Can you check in cn=config which value is set for nsslapd-localhost attribute? This is the hostname value used by the LDAP server when it initializes own TGT from the keytab.
It should be ipa-b.$domain to make sure that both the client and the server are utilizing the same service principal. I suspect it is set to ipa-b.virt.$domain and thus the issue.
From dse.ldiff nsslapd-localhost: ipa-b.in.bmrc.ox.ac.uk
Fairly sure this is representative of the current running configuration, as the node was rebooted only hours ago.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 15:58, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear Alexander,
klist -kt /etc/dirsrv/ds.keytab Keytab name: FILE:/etc/dirsrv/ds.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK
This is a bit non-standard i understand, but so far this configuration is working ok. I guess the issue is that the ticket is being issued for the wrong domain. [cid:F8DF5B93-5D52-46D5-88AC-E9BEA54760FD@in.bmrc.ox.ac.uk]
I've attached a screenshot of the DNS configuration for the sub-zone.
Our intention here is to ensure that the DNS entry and host for the IPA server within a different sub-zone and subnet resolves to a single IP for speed. So a "host" has been created for each of the interfaces, all of the respective kerberos principals for the host services (ldap in this case) and then a new certificate issued with the alt names on it to allow for LDAPS. This works well, right up until the point of GSSAPI getting involved. There must be a piece of the puzzle we're missing here! Can you check in cn=config which value is set for nsslapd-localhost attribute? This is the hostname value used by the LDAP server when it initializes own TGT from the keytab.
It should be ipa-b.$domain to make sure that both the client and the server are utilizing the same service principal. I suspect it is set to ipa-b.virt.$domain and thus the issue.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
Locally on the IPA server I note that doing an ldapsearch using GSSAPI works, if i use the ldap host: ldaps://ipa-b.in.bmrc.ox.ac.uk/ but not: ldaps://ipa-b.virt.in.bmrc.ox.ac.uk/
Since the client can only access the network that is ipa-b.virt.in.bmrc.ox.ac.uk it needs to be able to communicate to LDAP via that hostname. Is this actually possible, since the TGT is _always_ going to be on ipa-b.$domain because of the nsslapd-localhost entry?
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 15:58, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear Alexander,
klist -kt /etc/dirsrv/ds.keytab Keytab name: FILE:/etc/dirsrv/ds.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 02/11/18 12:09:17 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 3 08/03/19 16:11:12 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:11:44 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 4 08/03/19 16:25:20 ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 1 11/03/19 10:50:01 ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:17 ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.cloud.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK 2 11/03/19 10:50:22 ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.hpc.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK
This is a bit non-standard i understand, but so far this configuration is working ok. I guess the issue is that the ticket is being issued for the wrong domain. [cid:F8DF5B93-5D52-46D5-88AC-E9BEA54760FD@in.bmrc.ox.ac.uk]
I've attached a screenshot of the DNS configuration for the sub-zone.
Our intention here is to ensure that the DNS entry and host for the IPA server within a different sub-zone and subnet resolves to a single IP for speed. So a "host" has been created for each of the interfaces, all of the respective kerberos principals for the host services (ldap in this case) and then a new certificate issued with the alt names on it to allow for LDAPS. This works well, right up until the point of GSSAPI getting involved. There must be a piece of the puzzle we're missing here! Can you check in cn=config which value is set for nsslapd-localhost attribute? This is the hostname value used by the LDAP server when it initializes own TGT from the keytab.
It should be ipa-b.$domain to make sure that both the client and the server are utilizing the same service principal. I suspect it is set to ipa-b.virt.$domain and thus the issue.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ma, 11 maalis 2019, Callum Smith wrote:
Locally on the IPA server I note that doing an ldapsearch using GSSAPI works, if i use the ldap host: ldaps://ipa-b.in.bmrc.ox.ac.uk/ but not: ldaps://ipa-b.virt.in.bmrc.ox.ac.uk/
Since the client can only access the network that is ipa-b.virt.in.bmrc.ox.ac.uk it needs to be able to communicate to LDAP via that hostname. Is this actually possible, since the TGT is _always_ going to be on ipa-b.$domain because of the nsslapd-localhost entry?
Question I have is why the client actually chooses ldap/ipa-b.$domain itself? This is probably the easiest place to change since it is driven by the DNS discovery so you can influence by whatever is put in the DNS SRV records.
Dear Alexander,
We're wondering that too, there's obviously a disparity between the domain that either end is issuing the LDAP ticket for, and the SRV records for the `virt.in.bmrc.ox.ac.uk` domain all point to the LDAP endpoint. Do i need specific SRV records for ldaps and not ldap? I earlier attached a screenshot of our domain setup for the VIRT subdomain.
I fear the opposite may be the case and the client is requesting the correct one but the ldap server is defaulting to the root domain not the subdomain.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 16:19, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith wrote: Locally on the IPA server I note that doing an ldapsearch using GSSAPI works, if i use the ldap host: ldaps://ipa-b.in.bmrc.ox.ac.uk/ but not: ldaps://ipa-b.virt.in.bmrc.ox.ac.uk/
Since the client can only access the network that is ipa-b.virt.in.bmrc.ox.ac.uk it needs to be able to communicate to LDAP via that hostname. Is this actually possible, since the TGT is _always_ going to be on ipa-b.$domain because of the nsslapd-localhost entry? Question I have is why the client actually chooses ldap/ipa-b.$domain itself? This is probably the easiest place to change since it is driven by the DNS discovery so you can influence by whatever is put in the DNS SRV records.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ma, 11 maalis 2019, Callum Smith wrote:
Dear Alexander,
We're wondering that too, there's obviously a disparity between the domain that either end is issuing the LDAP ticket for, and the SRV records for the `virt.in.bmrc.ox.ac.uk` domain all point to the LDAP endpoint. Do i need specific SRV records for ldaps and not ldap? I earlier attached a screenshot of our domain setup for the VIRT subdomain.
I fear the opposite may be the case and the client is requesting the correct one but the ldap server is defaulting to the root domain not the subdomain.
Well, the server is doing the right thing as it doesn't know anything about the subdomain's hostname. Kernel has only a single hostname.
Can you do a check like this from the client:
export KRB5_TRACE=/dev/stderr kinit admin ldapsearch -Y GSSAPI -h ipa-b.virt.in.bmrc.ox.ac.uk -b dc=in,dc=bmrc,dc=ox,dc=ac,dc=uk -s base
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 16:19, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith wrote: Locally on the IPA server I note that doing an ldapsearch using GSSAPI works, if i use the ldap host: ldaps://ipa-b.in.bmrc.ox.ac.uk/ but not: ldaps://ipa-b.virt.in.bmrc.ox.ac.uk/
Since the client can only access the network that is ipa-b.virt.in.bmrc.ox.ac.uk it needs to be able to communicate to LDAP via that hostname. Is this actually possible, since the TGT is _always_ going to be on ipa-b.$domain because of the nsslapd-localhost entry? Question I have is why the client actually chooses ldap/ipa-b.$domain itself? This is probably the easiest place to change since it is driven by the DNS discovery so you can influence by whatever is put in the DNS SRV records.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
Dear Alexander,
Some more (hopefully) helpful information with a KRB5_TRACE on while running ipa-client install:
ipa-client-install WARNING: ntpd time&date synchronization service will not be configured as conflicting service (chronyd) is enabled Use --force-ntpd option to disable it and force configuration of ntpd
Discovery was successful! Client hostname: virt-test.virt.in.bmrc.ox.ac.uk Realm: IN.BMRC.OX.AC.UK DNS Domain: virt.in.bmrc.ox.ac.uk IPA Server: ipa-b.virt.in.bmrc.ox.ac.uk BaseDN: dc=in,dc=bmrc,dc=ox,dc=ac,dc=uk
Continue to configure the system with these values? [no]: yes Skipping synchronizing time with NTP server. User authorized to enroll computers: admin Password for admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK: [7792] 1552322394.293495: ccselect module realm chose cache FILE:/tmp/krbccQ6OHiN/ccache with client principal admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for server principal ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK [7792] 1552322394.293496: Getting credentials admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK using ccache FILE:/tmp/krbccQ6OHiN/ccache [7792] 1552322394.293497: Retrieving admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK from FILE:/tmp/krbccQ6OHiN/ccache with result: -1765328243/Matching credential not found (filename: /tmp/krbccQ6OHiN/ccache) [7792] 1552322394.293498: Retrieving admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK from FILE:/tmp/krbccQ6OHiN/ccache with result: 0/Success [7792] 1552322394.293499: Starting with TGT for client realm: admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK [7792] 1552322394.293500: Requesting tickets for ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK, referrals on [7792] 1552322394.293501: Generated subkey for TGS request: aes256-cts/6474 [7792] 1552322394.293502: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts [7792] 1552322394.293504: Encoding request body and padata into FAST request [7792] 1552322394.293505: Sending request (985 bytes) to IN.BMRC.OX.AC.UK [7792] 1552322394.293506: Resolving hostname ipa-b.virt.in.bmrc.ox.ac.uk [7792] 1552322394.293507: Initiating TCP connection to stream 10.141.31.252:88 [7792] 1552322394.293508: Sending TCP request to stream 10.141.31.252:88 [7792] 1552322394.293509: Received answer (883 bytes) from stream 10.141.31.252:88 [7792] 1552322394.293510: Terminating TCP connection to stream 10.141.31.252:88 [7792] 1552322394.293511: Response was from master KDC [7792] 1552322394.293512: Decoding FAST response [7792] 1552322394.293513: FAST reply key: aes256-cts/7B54 [7792] 1552322394.293514: TGS reply is for admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK with session key aes256-cts/0013 [7792] 1552322394.293515: TGS request result: 0/Success [7792] 1552322394.293516: Received creds for desired service ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK [7792] 1552322394.293517: Storing admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK in FILE:/tmp/krbccQ6OHiN/ccache [7792] 1552322394.293519: Creating authenticator for admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK, seqnum 27249405, subkey aes256-cts/2328, session key aes256-cts/0013 Unable to download CA cert from LDAP.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 16:19, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith wrote: Locally on the IPA server I note that doing an ldapsearch using GSSAPI works, if i use the ldap host: ldaps://ipa-b.in.bmrc.ox.ac.uk/ but not: ldaps://ipa-b.virt.in.bmrc.ox.ac.uk/
Since the client can only access the network that is ipa-b.virt.in.bmrc.ox.ac.uk it needs to be able to communicate to LDAP via that hostname. Is this actually possible, since the TGT is _always_ going to be on ipa-b.$domain because of the nsslapd-localhost entry? Question I have is why the client actually chooses ldap/ipa-b.$domain itself? This is probably the easiest place to change since it is driven by the DNS discovery so you can influence by whatever is put in the DNS SRV records.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ma, 11 maalis 2019, Callum Smith wrote:
Dear Alexander,
Some more (hopefully) helpful information with a KRB5_TRACE on while running ipa-client install:
Thanks, I just sent a request for basically the same. ;)
ipa-client-install WARNING: ntpd time&date synchronization service will not be configured as conflicting service (chronyd) is enabled Use --force-ntpd option to disable it and force configuration of ntpd
Discovery was successful! Client hostname: virt-test.virt.in.bmrc.ox.ac.uk Realm: IN.BMRC.OX.AC.UK DNS Domain: virt.in.bmrc.ox.ac.uk IPA Server: ipa-b.virt.in.bmrc.ox.ac.uk BaseDN: dc=in,dc=bmrc,dc=ox,dc=ac,dc=uk
Continue to configure the system with these values? [no]: yes Skipping synchronizing time with NTP server. User authorized to enroll computers: admin Password for admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK: [7792] 1552322394.293495: ccselect module realm chose cache FILE:/tmp/krbccQ6OHiN/ccache with client principal admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for server principal ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK [7792] 1552322394.293496: Getting credentials admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK using ccache FILE:/tmp/krbccQ6OHiN/ccache [7792] 1552322394.293497: Retrieving admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK from FILE:/tmp/krbccQ6OHiN/ccache with result: -1765328243/Matching credential not found (filename: /tmp/krbccQ6OHiN/ccache) [7792] 1552322394.293498: Retrieving admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK from FILE:/tmp/krbccQ6OHiN/ccache with result: 0/Success [7792] 1552322394.293499: Starting with TGT for client realm: admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK [7792] 1552322394.293500: Requesting tickets for ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK, referrals on [7792] 1552322394.293501: Generated subkey for TGS request: aes256-cts/6474 [7792] 1552322394.293502: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts [7792] 1552322394.293504: Encoding request body and padata into FAST request [7792] 1552322394.293505: Sending request (985 bytes) to IN.BMRC.OX.AC.UK [7792] 1552322394.293506: Resolving hostname ipa-b.virt.in.bmrc.ox.ac.uk [7792] 1552322394.293507: Initiating TCP connection to stream 10.141.31.252:88 [7792] 1552322394.293508: Sending TCP request to stream 10.141.31.252:88 [7792] 1552322394.293509: Received answer (883 bytes) from stream 10.141.31.252:88 [7792] 1552322394.293510: Terminating TCP connection to stream 10.141.31.252:88 [7792] 1552322394.293511: Response was from master KDC [7792] 1552322394.293512: Decoding FAST response [7792] 1552322394.293513: FAST reply key: aes256-cts/7B54 [7792] 1552322394.293514: TGS reply is for admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK with session key aes256-cts/0013 [7792] 1552322394.293515: TGS request result: 0/Success [7792] 1552322394.293516: Received creds for desired service ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK [7792] 1552322394.293517: Storing admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK in FILE:/tmp/krbccQ6OHiN/ccache [7792] 1552322394.293519: Creating authenticator for admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK -> ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK, seqnum 27249405, subkey aes256-cts/2328, session key aes256-cts/0013 Unable to download CA cert from LDAP.
Ok, so the client actually asks for the ldap/ipa-b.virt.$domain already, good. It means the server is only knowing about the key for ldap/ipa-b.$domain.
An option would be to turn ldap/ipa-b.virt.$domain into a service principal alias of ldap/ipa-b.$domain.
You would need to delete ldap/ipa-b.virt.$domain principal first.
ipa service-del ldap/ipa-b.virt.$domain
and then add it as an alias for ldap/ipa-b.$domain:
ipa service-add-principal ldap/ipa-b.$domain ldap/ipa-b.virt.$domain
Dear Alexander,
It seems setting up the principal alias has gotten us to a further point down the line, but we're seeing other issues now.
We've moved both ldap/ and HTTP/ principals to aliases of the main principal (the downside being we can't do an altname-based automated certificate request - but manually issuing certificates is an ok workaround). We're still seeing issues potentially relating to the HTTP principal authentication though, despite it now being an alias. Any ideas here?
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 14:27, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear IPA Gurus
I have a client that's incapable of joining the FreeIPA realm, it's in a different DNS sub-zone but is in the same realm. I get the feeling that there's a kerberos principal missing somewhere to get this all to work, but I can't quite see where it might be. Simple authentication ldapsearch using cn=Directory Manager functions perfectly well to the ipa host in question, however anonymous binds are disabled. I'm not clear why this wouldn't be working. From the above it is unclear what your problem is.
Can you show what exactly is failing? ipa-client-install is failing? Show logs from /var/log/ipaclient-install.log. You aren't using FreeIPA enrollment? How exactly did you try to enroll that client? Show sequence of commands you ran.
It is not easy to help with no logs and exact steps tried.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ti, 12 maalis 2019, Callum Smith wrote:
Dear Alexander,
It seems setting up the principal alias has gotten us to a further point down the line, but we're seeing other issues now.
We've moved both ldap/ and HTTP/ principals to aliases of the main principal (the downside being we can't do an altname-based automated certificate request - but manually issuing certificates is an ok workaround). We're still seeing issues potentially relating to the HTTP principal authentication though, despite it now being an alias. Any ideas here?
From your sentence above I'm not sure whether you made HTTP/ipa-b.virt.$domain an alias of ldap/... or HTTP/ipa-b.$domain?
If the former, then it is not correct.
If the latter, then please show krb5 traces.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 14:27, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear IPA Gurus
I have a client that's incapable of joining the FreeIPA realm, it's in a different DNS sub-zone but is in the same realm. I get the feeling that there's a kerberos principal missing somewhere to get this all to work, but I can't quite see where it might be. Simple authentication ldapsearch using cn=Directory Manager functions perfectly well to the ipa host in question, however anonymous binds are disabled. I'm not clear why this wouldn't be working. From the above it is unclear what your problem is.
Can you show what exactly is failing? ipa-client-install is failing? Show logs from /var/log/ipaclient-install.log. You aren't using FreeIPA enrollment? How exactly did you try to enroll that client? Show sequence of commands you ran.
It is not easy to help with no logs and exact steps tried.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
ldap/ipa-b.virt.$domain > ldap/ipa-b.$domain HTTP/ipa-b.virt.$domain > HTTP/ipa-b.$domain
both aliases as above - krb5trace should be in attachments on previous message.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 11:09, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith wrote: Dear Alexander,
It seems setting up the principal alias has gotten us to a further point down the line, but we're seeing other issues now.
We've moved both ldap/ and HTTP/ principals to aliases of the main principal (the downside being we can't do an altname-based automated certificate request - but manually issuing certificates is an ok workaround). We're still seeing issues potentially relating to the HTTP principal authentication though, despite it now being an alias. Any ideas here? From your sentence above I'm not sure whether you made HTTP/ipa-b.virt.$domain an alias of ldap/... or HTTP/ipa-b.$domain?
If the former, then it is not correct.
If the latter, then please show krb5 traces.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 11 Mar 2019, at 14:27, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ma, 11 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear IPA Gurus
I have a client that's incapable of joining the FreeIPA realm, it's in a different DNS sub-zone but is in the same realm. I get the feeling that there's a kerberos principal missing somewhere to get this all to work, but I can't quite see where it might be. Simple authentication ldapsearch using cn=Directory Manager functions perfectly well to the ipa host in question, however anonymous binds are disabled. I'm not clear why this wouldn't be working. From the above it is unclear what your problem is.
Can you show what exactly is failing? ipa-client-install is failing? Show logs from /var/log/ipaclient-install.log. You aren't using FreeIPA enrollment? How exactly did you try to enroll that client? Show sequence of commands you ran.
It is not easy to help with no logs and exact steps tried.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ti, 12 maalis 2019, Callum Smith via FreeIPA-users wrote:
ldap/ipa-b.virt.$domain > ldap/ipa-b.$domain HTTP/ipa-b.virt.$domain > HTTP/ipa-b.$domain
both aliases as above - krb5trace should be in attachments on previous message.
My bad. Thanks, can you also give krb5kdc.log output from the KDC server the client talked to?
It looks like KDC is not finding something and returning PROCESS_TGS. I have no time to look into details right now.
Dear Alexander,
No worries - here's the krb5kdc.log relevant area when you get a moment. I understand that service aliases are relatively new to FreeIPA so debugging them is proving to be a bit tricky.
Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552388071, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552388071, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552388071, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (1 etypes {18}) 10.141.17.1: ISSUE: authtime 1552388071, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.248.2: ISSUE: authtime 1552388071, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552388071, etypes {rep=18 tkt=18 ses=18}, host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 10:54:31 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11
We're very grateful for your time - particularly when it may be taking you away from things like implementing the Global Catalogue we're eager for :D.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 11:52, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith via FreeIPA-users wrote: ldap/ipa-b.virt.$domain > ldap/ipa-b.$domain HTTP/ipa-b.virt.$domain > HTTP/ipa-b.$domain
both aliases as above - krb5trace should be in attachments on previous message. My bad. Thanks, can you also give krb5kdc.log output from the KDC server the client talked to?
It looks like KDC is not finding something and returning PROCESS_TGS. I have no time to look into details right now.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ti, 12 maalis 2019, Callum Smith wrote:
Dear Alexander,
No worries - here's the krb5kdc.log relevant area when you get a moment. I understand that service aliases are relatively new to FreeIPA so debugging them is proving to be a bit tricky.
Hm.. the log you provided does not include a line where host/virt-test... client asks for a service ticket (TGS_REQ) to HTTP/virt-b... that results in PROCESS_TGS response.
The log entries around that one are needed.
We're very grateful for your time - particularly when it may be taking you away from things like implementing the Global Catalogue we're eager for :D.
:) I wish I had time for that already. I'm trying to fix https://pagure.io/freeipa/issue/7181 right now.
So I've just re-run the client install to avoid the noise of krb5kdc.log (just as to why the timestamps don't match) and this is the entire block:
Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (1 etypes {18}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.248.2: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 12:04, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith wrote: Dear Alexander,
No worries - here's the krb5kdc.log relevant area when you get a moment. I understand that service aliases are relatively new to FreeIPA so debugging them is proving to be a bit tricky. Hm.. the log you provided does not include a line where host/virt-test... client asks for a service ticket (TGS_REQ) to HTTP/virt-b... that results in PROCESS_TGS response.
The log entries around that one are needed.
We're very grateful for your time - particularly when it may be taking you away from things like implementing the Global Catalogue we're eager for :D. :) I wish I had time for that already. I'm trying to fix https://pagure.io/freeipa/issue/7181 right now.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ti, 12 maalis 2019, Callum Smith wrote:
So I've just re-run the client install to avoid the noise of krb5kdc.log (just as to why the timestamps don't match) and this is the entire block:
In the client krb5 trace I can see it talks to four different KDCs, not to ipa-b alone, because the krb5.conf generated during install does not pin you to ipa-b anymore. So I guess you need to look at other KDCs logs too.
Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (1 etypes {18}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.248.2: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 12:04, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith wrote: Dear Alexander,
No worries - here's the krb5kdc.log relevant area when you get a moment. I understand that service aliases are relatively new to FreeIPA so debugging them is proving to be a bit tricky. Hm.. the log you provided does not include a line where host/virt-test... client asks for a service ticket (TGS_REQ) to HTTP/virt-b... that results in PROCESS_TGS response.
The log entries around that one are needed.
We're very grateful for your time - particularly when it may be taking you away from things like implementing the Global Catalogue we're eager for :D. :) I wish I had time for that already. I'm trying to fix https://pagure.io/freeipa/issue/7181 right now.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
Yep you're not wrong, one of our IPA replica was being evil and spitting errors. That replica is destined for the bin anyway so i've not worried about it. All of the kerberos issues have now gone away - except one which is more of a question than anything. Is it intentional that the sub-zone _kerberos._tcp SRV records are ignored and only the top level SRV records are used. We were hoping that defining _kerberos._tcp in .virt.in.bmrc.ox.ac.uk would work and over-ride the _kerberos._tcp SRV records in .in.bmrc.ox.ac.ukhttp://in.bmrc.ox.ac.uk
I have a feeling this behaviour is only in the installer however.
Another (smaller) issue is that the DNS record creation as part of `ipa-client-install` isn't working. I'm having trouble finding where to look for the error:
2019-03-12T14:43:39Z DEBUG The DNS query name does not exist: virt-test.virt.in.bmrc.ox.ac.uk. 2019-03-12T14:43:39Z WARNING Hostname (virt-test.virt.in.bmrc.ox.ac.uk) does not have A/AAAA record. 2019-03-12T14:43:39Z DEBUG IP check failed: cannot use loopback IP address 127.0.0.1 2019-03-12T14:43:39Z DEBUG IP check failed: cannot use loopback IP address ::1 2019-03-12T14:43:39Z DEBUG IP check successful: 10.141.17.1 2019-03-12T14:43:39Z DEBUG IP check failed: cannot use link-local IP address fe80::546f:67ff:fe51:1c%eth0 2019-03-12T14:43:39Z DEBUG IP check successful: 10.141.17.1 2019-03-12T14:43:39Z DEBUG IP check failed: cannot use link-local IP address fe80::546f:67ff:fe51:1c%eth0 2019-03-12T14:43:39Z DEBUG Searching for an interface of IP address: 10.141.17.1 2019-03-12T14:43:39Z DEBUG Testing local IP address: 127.0.0.1/255.0.0.0 (interface: lo) 2019-03-12T14:43:39Z DEBUG Testing local IP address: 10.141.17.1/255.255.240.0 (interface: eth0) 2019-03-12T14:43:39Z DEBUG Writing nsupdate commands to /etc/ipa/.dns_update.txt: 2019-03-12T14:43:39Z DEBUG debug
update delete virt-test.virt.in.bmrc.ox.ac.uk. IN A show send
update delete virt-test.virt.in.bmrc.ox.ac.uk. IN AAAA show send
update add virt-test.virt.in.bmrc.ox.ac.uk. 1200 IN A 10.141.17.1 show send
2019-03-12T14:43:39Z DEBUG Starting external process 2019-03-12T14:43:39Z DEBUG args=/usr/bin/nsupdate -g /etc/ipa/.dns_update.txt 2019-03-12T14:45:23Z DEBUG Process finished, return code=1 2019-03-12T14:45:23Z DEBUG stdout=Outgoing update query: ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0 ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0 ;; UPDATE SECTION: virt-test.virt.in.bmrc.ox.ac.uk. 0 ANY A
Outgoing update query: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55036 ;; flags:; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;3005929322.sig-ipa-a.in.bmrc.ox.ac.uk. ANY TKEY
;; ADDITIONAL SECTION: 3005929322.sig-ipa-a.in.bmrc.ox.ac.uk. 0 ANY TKEY gss-tsig. 1552401823 1552401823 3 NOERROR 688 YIICrAYJKoZIhvcSAQICAQBuggKbMIICl6ADAgEFoQMCAQ6iBwMFACAA AACjggGKYYIBhjCCAYKgAwIBBaE SGxBJTi5CTVJDLk9YLkFDLlVLoigw JqADAgEDoR8wHRsDRE5TGxZpcGEtYS5pbi5ibXJjLm94LmFjLnVro4IB OzCCATegAwIBEqEDAgECooIBKQSCASX4L4yJ9gPwWyHU5szTktPPJP+G Hjf/Bzworzuk1ODfJ5k/rG35UYurnk1KB0FI RYaeblQ8CPyYZ9eAmo1l WiPHFT+GwVtiUN6nhiPno5cQway4I5BCBOAQBEuxJd96GGqMhZYZLzWZ EomtIyl3JGL7GcuXFV62S9Dwg3FXsME3XYkBGrCQXHgXX35Yq0sh5sWI JM/XDPfbTxDHonLc+l/FSCyXB1KlOBc0v9KGX02V3aPlc NssV2xvk8y/ Nt/nyCI8VtzIa/6fSy/ZDpdwCkLqF2TbXY3ans6x1YbtS6GXIQtB3SFr n5PLZ+D/s6iHDHw7x4+q2on9+zlytLJahdoJLUO6/Zbr0MQrJPTjGmEb /RMySXyzEFz/evVVwlApnGlYY8ToIKSB8zCB8KADAgESooHoBIHl/v gZ 5/9qdzXOnRNBsmlgXU4viWXwbncZgQJ3E14rZOybp3/V9CVon0TjA4W4 +DsvWTeFiW9TO8ItLEsy/Am5phN3JemwPbSzYlZjUUovAKcCUg19Bn9o T6U2uopI38PxIIW7hieiQbcwu2thzjmVZCTLzl/ecxzHPhfWYbgJAz3T WLsYS+ 7TvVBU7UwYrbYb6Pbs3jF6VZCkEGRUz6DrQ8ukoL/hjBNcJ7uP MtNz9IVk61Monet/6fAT/EqIgvBYTGXySclw4/x8q2VxShtZ9NwT104+ eMijav0t8wsxeoL0HIq67w== 0
2019-03-12T14:45:23Z DEBUG stderr=Reply from SOA query: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21780 ;; flags: qr aa rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;virt-test.virt.in.bmrc.ox.ac.uk. IN SOA
;; AUTHORITY SECTION: virt.in.bmrc.ox.ac.uk. 0 IN SOA ipa-a.in.bmrc.ox.ac.uk. hostmaster.virt.in.bmrc.ox.ac.uk. 1552319704 3600 900 1209600 3600
Found zone name: virt.in.bmrc.ox.ac.uk The master is: ipa-a.in.bmrc.ox.ac.uk start_gssrequest send_gssrequest ; Communication with 10.141.247.129#53 failed: timed out Reply from SOA query: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26740 ;; flags: qr ra; QUESTION: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;3005929322.sig-ipa-a.in.bmrc.ox.ac.uk. ANY TKEY
;; ANSWER SECTION: 3005929322.sig-ipa-a.in.bmrc.ox.ac.uk. 0 ANY TKEY gss-tsig. 0 0 3 BADKEY 0 0
Reply from SOA query: ;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 22380 ;; flags: qr ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;sig-ipa-a.in.bmrc.ox.ac.uk. ANY TKEY
response to SOA query was unsuccessful
2019-03-12T14:45:23Z DEBUG nsupdate failed: Command '/usr/bin/nsupdate -g /etc/ipa/.dns_update.txt' returned non-zero exit status 1 2019-03-12T14:45:23Z ERROR Failed to update DNS records. 2019-03-12T14:45:23Z DEBUG DNS resolver: Query: virt-test.virt.in.bmrc.ox.ac.uk IN A 2019-03-12T14:45:23Z DEBUG DNS resolver: No record. 2019-03-12T14:45:23Z DEBUG DNS resolver: Query: virt-test.virt.in.bmrc.ox.ac.uk IN AAAA 2019-03-12T14:45:23Z DEBUG DNS resolver: No record. 2019-03-12T14:45:23Z DEBUG DNS resolver: Query: 1.17.141.10.in-addr.arpa. IN PTR 2019-03-12T14:45:23Z DEBUG DNS resolver: No record. 2019-03-12T14:45:23Z WARNING Missing A/AAAA record(s) for host virt-test.virt.in.bmrc.ox.ac.uk: 10.141.17.1. 2019-03-12T14:45:23Z WARNING Missing reverse record(s) for address(es): 10.141.17.1.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 12:37, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith wrote: So I've just re-run the client install to avoid the noise of krb5kdc.log (just as to why the timestamps don't match) and this is the entire block: In the client krb5 trace I can see it talks to four different KDCs, not to ipa-b alone, because the krb5.conf generated during install does not pin you to ipa-b anymore. So I guess you need to look at other KDCs logs too.
Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:HTTP/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (1 etypes {18}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.248.2: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UKmailto:admin@IN.BMRC.OX.AC.UK for ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:ldap/ipa-b.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: NEEDED_PREAUTH: host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK, Additional pre-authentication required Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11 Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.141.17.1: ISSUE: authtime 1552392528, etypes {rep=18 tkt=18 ses=18}, host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UKmailto:host/virt-test.virt.in.bmrc.ox.ac.uk@IN.BMRC.OX.AC.UK for krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UKmailto:krbtgt/IN.BMRC.OX.AC.UK@IN.BMRC.OX.AC.UK Mar 12 12:08:48 ipa-b.in.bmrc.ox.ac.uk krb5kdc[1967](info): closing down fd 11
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 12:04, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith wrote: Dear Alexander,
No worries - here's the krb5kdc.log relevant area when you get a moment. I understand that service aliases are relatively new to FreeIPA so debugging them is proving to be a bit tricky. Hm.. the log you provided does not include a line where host/virt-test... client asks for a service ticket (TGS_REQ) to HTTP/virt-b... that results in PROCESS_TGS response.
The log entries around that one are needed.
We're very grateful for your time - particularly when it may be taking you away from things like implementing the Global Catalogue we're eager for :D. :) I wish I had time for that already. I'm trying to fix https://pagure.io/freeipa/issue/7181 right now.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ti, 12 maalis 2019, Callum Smith wrote:
Yep you're not wrong, one of our IPA replica was being evil and spitting errors. That replica is destined for the bin anyway so i've not worried about it. All of the kerberos issues have now gone away - except one which is more of a question than anything. Is it intentional that the sub-zone _kerberos._tcp SRV records are ignored and only the top level SRV records are used. We were hoping that defining _kerberos._tcp in .virt.in.bmrc.ox.ac.uk would work and over-ride the _kerberos._tcp SRV records in .in.bmrc.ox.ac.uk
I have a feeling this behaviour is only in the installer however.
Installer uses _ldap SRV records to discover IPA masters. This is documented in the manual page for ipa-client-install.
You can also pass --domain virt.in.bmrc.ox.ac.uk and then clients will use this one to discover servers and use them. I guess you'll need to add _kerberos TXT record there with 'IN.BMRC.OX.AC.UK' to properly glue Kerberos clients but LDAP SRV records should just work.
Another (smaller) issue is that the DNS record creation as part of `ipa-client-install` isn't working. I'm having trouble finding where to look for the error:
Found zone name: virt.in.bmrc.ox.ac.uk The master is: ipa-a.in.bmrc.ox.ac.uk start_gssrequest send_gssrequest ; Communication with 10.141.247.129#53 failed: timed out
Authoritative server is not available, thus failing to communicate with it. I don't think it will be possible to fix this as you are running the very same servers as dual network ones and we don't have DNS views that would rewrite IP addresses of the authoritative servers.
Dear Alexander,
We already have the correct _ldap._tcp.virt.$domain in place, and the discovery at the start of ipa-client-install is working correctly, it discovers the correct information and installs based on that: Discovery was successful! Client hostname: virt-test.virt.in.bmrc.ox.ac.uk Realm: IN.BMRC.OX.AC.UK DNS Domain: virt.in.bmrc.ox.ac.uk IPA Server: ipa-b.virt.in.bmrc.ox.ac.uk BaseDN: dc=in,dc=bmrc,dc=ox,dc=ac,dc=uk
But it is further into the process where it goes a bit wrong. I've attached two files krb5trace and ipaclient-installer.log so that we're not confusing the previous woes.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 16:03, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith wrote: Yep you're not wrong, one of our IPA replica was being evil and spitting errors. That replica is destined for the bin anyway so i've not worried about it. All of the kerberos issues have now gone away - except one which is more of a question than anything. Is it intentional that the sub-zone _kerberos._tcp SRV records are ignored and only the top level SRV records are used. We were hoping that defining _kerberos._tcp in .virt.in.bmrc.ox.ac.uk would work and over-ride the _kerberos._tcp SRV records in .in.bmrc.ox.ac.ukhttp://in.bmrc.ox.ac.uk
I have a feeling this behaviour is only in the installer however. Installer uses _ldap SRV records to discover IPA masters. This is documented in the manual page for ipa-client-install.
You can also pass --domain virt.in.bmrc.ox.ac.uk and then clients will use this one to discover servers and use them. I guess you'll need to add _kerberos TXT record there with 'IN.BMRC.OX.AC.UK' to properly glue Kerberos clients but LDAP SRV records should just work.
Another (smaller) issue is that the DNS record creation as part of `ipa-client-install` isn't working. I'm having trouble finding where to look for the error:
Found zone name: virt.in.bmrc.ox.ac.uk The master is: ipa-a.in.bmrc.ox.ac.uk start_gssrequest send_gssrequest ; Communication with 10.141.247.129#53 failed: timed out Authoritative server is not available, thus failing to communicate with it. I don't think it will be possible to fix this as you are running the very same servers as dual network ones and we don't have DNS views that would rewrite IP addresses of the authoritative servers.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ti, 12 maalis 2019, Callum Smith via FreeIPA-users wrote:
Dear Alexander,
We already have the correct _ldap._tcp.virt.$domain in place, and the discovery at the start of ipa-client-install is working correctly, it discovers the correct information and installs based on that: Discovery was successful! Client hostname: virt-test.virt.in.bmrc.ox.ac.uk Realm: IN.BMRC.OX.AC.UK DNS Domain: virt.in.bmrc.ox.ac.uk IPA Server: ipa-b.virt.in.bmrc.ox.ac.uk BaseDN: dc=in,dc=bmrc,dc=ox,dc=ac,dc=uk
But it is further into the process where it goes a bit wrong. I've attached two files krb5trace and ipaclient-installer.log so that we're not confusing the previous woes.
The difference is that during install the temporary krb5.conf written pins you down to a specific IPA master. This is done for the purpose to avoid replication issues if a different master was chosen at a different stage of the install process.
Later, the actual krb5.conf written to /etc/krb5.conf does not include that master because installation options weren't forcing us to stick to a specific master. At this point selection of the KDCs is left to krb5 library. Actual order of service locator tries is this:
- try locator plugins first - try krb5.conf profile - try DNS resolution as a callback
We have nothing in krb5.conf. We also have nothing in sssd.conf so SSSD locator plugin would give us whatever IPA master it chose. But at the point of completing ipa-client-installer job SSSD is not yet running so we end up with DNS resolution.
The only way of solving this is by forcing use of specific servers during install. E.g. specifying
ipa-client-install --server ipa-a.virt.$domain --server ipa-b.virt.$domain ...
would make sure both servers are added to krb5.conf and to sssd.conf.
Perhaps, this what would work for you?
Dear Alexander,
The last small wrinkle, setting the server options is fine and works well, but the DNS record creation still doesn't work. I see it queries the SOA record and then appears to use that as the server to send the changes to.
I tried to set the SOA records for the virt.$domain realm, but it doesnt seem to overwrite the top-level SOA record: ipa dnszone-mod virt.in.bmrc.ox.ac.uk. --name-server ipa-a --admin-email ipa-a I note that admin-email appears to be the option that actually changes the record returned here, which was unexpected for me.
Trying to understand as much as possible from the documentation where possible, but still not quite there. IS there a way of forcing only the virt.$domain SOA record to be returned, or specifically remove the top level ipa-a.$domain record from the virt.$domain sub-zone SOA record somehow?
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0 ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0 ;; UPDATE SECTION: virt-test.virt.in.bmrc.ox.ac.uk. 0 ANY A
Outgoing update query: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61088 ;; flags:; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;859519045.sig-ipa-a.in.bmrc.ox.ac.uk. ANY TKEY
;; ADDITIONAL SECTION: 859519045.sig-ipa-a.in.bmrc.ox.ac.uk. 0 ANY TKEY gss-tsig. 1552471501 1552471501 3 NOERROR 688 YIICrAYJKoZIhvcSAQICAQBuggKbMIICl6ADAgEFoQMCAQ6iBwMFACAA AACjggGKYYIBhjCCAYKgAwIBBaES GxBJTi5CTVJDLk9YLkFDLlVLoigw JqADAgEDoR8wHRsDRE5TGxZpcGEtYS5pbi5ibXJjLm94LmFjLnVro4IB OzCCATegAwIBEqEDAgECooIBKQSCASWh1n7sjwfpXDidKWGk8HSALBBW OwjtcqBJAGcS7yB5YGKzb4t3LUQFPXhzmZAxhZGTrkg+YLRJ3Ysty4AI HY1Tu465eJ0yPIOAxwVrhlQXBrs6T7K8OqyjN/rOO9KLhLMjTLz76x3S m5u8FE/L0FuTM3uF53qg2l00y4hjsztaDAsKFjL4vZALLDY796tGBDS0 C8RybVcdVGeoe5L7IrHG14hTd1ppMXaGuFcIOLlEuJHW0m+YjZHlQWBT HYAPVKJqgBOrRiqKIVkeTBSyvUMhAG5YNMKHOtmtfBbr3hyh3xb0yRlT NakBI9TRSdulBkfP4ONGjnhg48huUgsaiuNl/WzdDNvzz3qepbJU8zVE d/B/NM5mNDmaUzYVhAnPdfb2ht6YaaSB8zCB8KADAgESooHoBIHlXbse XPn5DwGyQy8HWW4lwny7PrJTLmnDczg7OjSkWLsgsA9c2Ok7IBO1pRZB Q1DK48TZ09vEpU9nTULdKmciqikdKV7Zi50afJXVc79wGaDOhHdGByzo KhnZy8kDgciN9BYTJ6se7Sd+f6agZ9Fh5t5cDb4kk2LUE9bVKERqrE1D CgASPFqxYm60GmOOSJDlVevYAycHfmy1DFcsCJOGYAiXNWDYSxP13bhe DwTlhvXPOjxXhwhQxWwz+E8aNHCHEuniT1+iTHVi5xgsU98qi489Deta SocvV0sI1eKMoalIe0TXIw== 0
2019-03-13T10:06:41Z DEBUG stderr=Reply from SOA query: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 26845 ;; flags: qr aa rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;virt-test.virt.in.bmrc.ox.ac.uk. IN SOA
;; AUTHORITY SECTION: virt.in.bmrc.ox.ac.uk. 0 IN SOA ipa-a.in.bmrc.ox.ac.uk. ipa-a.virt.in.bmrc.ox.ac.uk. 1552471476 3600 900 1209600 3600
Found zone name: virt.in.bmrc.ox.ac.uk The master is: ipa-a.in.bmrc.ox.ac.uk start_gssrequest send_gssrequest ; Communication with 10.141.247.129#53 failed: timed out Reply from SOA query: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62319 ;; flags: qr ra; QUESTION: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;859519045.sig-ipa-a.in.bmrc.ox.ac.uk. ANY TKEY
;; ANSWER SECTION: 859519045.sig-ipa-a.in.bmrc.ox.ac.uk. 0 ANY TKEY gss-tsig. 0 0 3 BADKEY 0 0
Reply from SOA query: ;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 1248 ;; flags: qr ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;sig-ipa-a.in.bmrc.ox.ac.uk. ANY TKEY
response to SOA query was unsuccessful
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 12 Mar 2019, at 17:08, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ti, 12 maalis 2019, Callum Smith via FreeIPA-users wrote: Dear Alexander,
We already have the correct _ldap._tcp.virt.$domain in place, and the discovery at the start of ipa-client-install is working correctly, it discovers the correct information and installs based on that: Discovery was successful! Client hostname: virt-test.virt.in.bmrc.ox.ac.uk Realm: IN.BMRC.OX.AC.UK DNS Domain: virt.in.bmrc.ox.ac.uk IPA Server: ipa-b.virt.in.bmrc.ox.ac.uk BaseDN: dc=in,dc=bmrc,dc=ox,dc=ac,dc=uk
But it is further into the process where it goes a bit wrong. I've attached two files krb5trace and ipaclient-installer.log so that we're not confusing the previous woes. The difference is that during install the temporary krb5.conf written pins you down to a specific IPA master. This is done for the purpose to avoid replication issues if a different master was chosen at a different stage of the install process.
Later, the actual krb5.conf written to /etc/krb5.conf does not include that master because installation options weren't forcing us to stick to a specific master. At this point selection of the KDCs is left to krb5 library. Actual order of service locator tries is this:
- try locator plugins first - try krb5.conf profile - try DNS resolution as a callback
We have nothing in krb5.conf. We also have nothing in sssd.conf so SSSD locator plugin would give us whatever IPA master it chose. But at the point of completing ipa-client-installer job SSSD is not yet running so we end up with DNS resolution.
The only way of solving this is by forcing use of specific servers during install. E.g. specifying ipa-client-install --server ipa-a.virt.$domain --server ipa-b.virt.$domain ...
would make sure both servers are added to krb5.conf and to sssd.conf.
Perhaps, this what would work for you?
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ke, 13 maalis 2019, Callum Smith wrote:
Dear Alexander,
The last small wrinkle, setting the server options is fine and works well, but the DNS record creation still doesn't work. I see it queries the SOA record and then appears to use that as the server to send the changes to.
I tried to set the SOA records for the virt.$domain realm, but it doesnt seem to overwrite the top-level SOA record: ipa dnszone-mod virt.in.bmrc.ox.ac.uk. --name-server ipa-a --admin-email ipa-a I note that admin-email appears to be the option that actually changes the record returned here, which was unexpected for me.
There are three levels of overrides here:
- /etc/named.conf can have 'fake_mname' defined - 'ipa dnsserver-*' commands allow to define per-server override with ipa dnsserver-mod <server> --soa-mname-override <some-server> - DNS zone SOA mname value
If you have SOA mname overridden in the 'ipa dnsserver-show', it will override whatever is set in the zone. This is to allow DNS location specific updates to be localized to that location's DNS server.
If you want to control it fully from the DNS zone settings, remove fake_mname from the /etc/named.conf and from the dnsserver's record:
ipa dnsserver-mod <server> --soa-mname-override=
(--soa-mname-override= sets it to empty value, meaning removal)
--admin-email in the zone should not be affecting SOA mname at all. I suspect you saw it act conflated with the first two overrides.
Dear Alexander,
Golden! We are in business - all puzzle pieces are in place so thank you very much for ongoing stamina with this. I'll write this all up so that someone else might take some value from it in the future.
Thank you again.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 13 Mar 2019, at 11:02, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ke, 13 maalis 2019, Callum Smith wrote: Dear Alexander,
The last small wrinkle, setting the server options is fine and works well, but the DNS record creation still doesn't work. I see it queries the SOA record and then appears to use that as the server to send the changes to.
I tried to set the SOA records for the virt.$domain realm, but it doesnt seem to overwrite the top-level SOA record: ipa dnszone-mod virt.in.bmrc.ox.ac.uk. --name-server ipa-a --admin-email ipa-a I note that admin-email appears to be the option that actually changes the record returned here, which was unexpected for me. There are three levels of overrides here:
- /etc/named.conf can have 'fake_mname' defined - 'ipa dnsserver-*' commands allow to define per-server override with ipa dnsserver-mod <server> --soa-mname-override <some-server> - DNS zone SOA mname value
If you have SOA mname overridden in the 'ipa dnsserver-show', it will override whatever is set in the zone. This is to allow DNS location specific updates to be localized to that location's DNS server.
If you want to control it fully from the DNS zone settings, remove fake_mname from the /etc/named.conf and from the dnsserver's record:
ipa dnsserver-mod <server> --soa-mname-override=
(--soa-mname-override= sets it to empty value, meaning removal)
--admin-email in the zone should not be affecting SOA mname at all. I suspect you saw it act conflated with the first two overrides.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
On ke, 13 maalis 2019, Callum Smith wrote:
Dear Alexander,
Golden! We are in business - all puzzle pieces are in place so thank you very much for ongoing stamina with this. I'll write this all up so that someone else might take some value from it in the future.
Great.
Yes, please do a write up!
Thank you again.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.ukmailto:callum@well.ox.ac.uk
On 13 Mar 2019, at 11:02, Alexander Bokovoy <abokovoy@redhat.commailto:abokovoy@redhat.com> wrote:
On ke, 13 maalis 2019, Callum Smith wrote: Dear Alexander,
The last small wrinkle, setting the server options is fine and works well, but the DNS record creation still doesn't work. I see it queries the SOA record and then appears to use that as the server to send the changes to.
I tried to set the SOA records for the virt.$domain realm, but it doesnt seem to overwrite the top-level SOA record: ipa dnszone-mod virt.in.bmrc.ox.ac.uk. --name-server ipa-a --admin-email ipa-a I note that admin-email appears to be the option that actually changes the record returned here, which was unexpected for me. There are three levels of overrides here:
- /etc/named.conf can have 'fake_mname' defined
- 'ipa dnsserver-*' commands allow to define per-server override with
ipa dnsserver-mod <server> --soa-mname-override <some-server>
- DNS zone SOA mname value
If you have SOA mname overridden in the 'ipa dnsserver-show', it will override whatever is set in the zone. This is to allow DNS location specific updates to be localized to that location's DNS server.
If you want to control it fully from the DNS zone settings, remove fake_mname from the /etc/named.conf and from the dnsserver's record:
ipa dnsserver-mod <server> --soa-mname-override=
(--soa-mname-override= sets it to empty value, meaning removal)
--admin-email in the zone should not be affecting SOA mname at all. I suspect you saw it act conflated with the first two overrides.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
Callum Smith