Hi,
I'd like to ask for your advise for the following topology... On a given site, IPA server has two legs (two NICs), let's call them "inside NIC" and "outside NIC". The inside NIC subnet is local to the site. The outside NIC subnet is interconnecting sites.
All the local clients talk to IPA via the inside NIC. But to setup a replica on another site, we must reach IPA via outside NIC (the inside subnet is not routable beyond the local site boundaries).
So the question arises: how to configure proper DNS resolution for the hostname of the IPA server itself? DNS is handled by IPA itself, fully in our control. So we have two options:
1. We create two A records for the same IPA hostname, let's say "ipa.site1.example.com". But then not sure if it will work fine... Technically, two IPs for the same name means load-balancing, right? So will I have intermittent connectivity issues, because it will return inside and outside IP interchangebly?
2. We create a new DNS name, e.g. "ipa-outside-site1.example.com", for the outside IP, and manually add it to the @ entry of "example.com", so that wannabe-replica on the remote site can use that FQDN as its master IPA. Will this work fine..? Will it not cause issues to the local clients on site1, who must keep using IPA with inside IP? Will it not cause issues on IPA server itself for some reason?
Please share your experience on this! Thanks.
The progress so far...
- We create two A records for the same IPA hostname, let's say
"ipa.site1.example.com". But then not sure if it will work fine... Technically, two IPs for the same name means load-balancing, right? So will I have intermittent connectivity issues, because it will return inside and outside IP interchangebly?
Bad idea... Indeed, clients get load-balanced responses with the two IPA IPs, and hence, half of the time they can't communicate with it...
- We create a new DNS name, e.g. "ipa-outside-site1.example.com", for the
outside IP, and manually add it to the @ entry of "example.com", so that wannabe-replica on the remote site can use that FQDN as its master IPA. Will this work fine..? Will it not cause issues to the local clients on site1, who must keep using IPA with inside IP? Will it not cause issues on IPA server itself for some reason?
OK, so with this one I managed to get a bit further... I am now just trying to enroll remote host with the IPA server over the "outside" interface (once I am good with this, I could try promoting it to replica...)
So, I created a new DNS zone for the remote site (site2.example.com). I created there SRV entries for "_kerberos._udp" and "_ldap._tcp", pointing to the outside DNS name of my IPA server: "ipa-outside-site1.example.com). Then I set the nameserver to this outside IP and run "ipa-client-install". Since it looks for the above SRV records first, discovery is successful. So I can continue with client installation.
Next problem was: ipa-client-install failed to authenticate to site1 IPA over LDAP, because it was trying to access it using ldap://ipa-outside-site1.example.com (and not via ldap://ipa.site1.example.com). I solved it by adding principal alias to the ldap/ipa.site1.example.com service.
Next problem was: libcurl was complaing about mismatch between certificate subject and the URL. Again, that's because it was browsing to ipa-outside-site1.example.com, but IPA cert has "ipa.site1.example.com" in subject. OK, so I fixed it by adding principal alias to http/ipa.site1.example.com and then reissuing certificate for IPA HTTP, adding a new DNS name to subjectAltName. This solved the warning.
But then my next problem was (and is):
RPC failed at server. Missing or invalid HTTP Referer, https://ipa-outside-site1.example.com/ipa/xml
I understand why - because ipa-client-install puts the above referer, but IPA server expects only https://ipa.site1.example.com... How can I solve this? How can I add another "alias referer" to be accepted..?
Basically, I need to convince all the components of the IPA server that they have two perfectly valid DNS names, with different IPs...
--- Regards, Dmitry Perets
On ti, 30 heinä 2019, Dmitry Perets via FreeIPA-users wrote:
The progress so far...
- We create two A records for the same IPA hostname, let's say
"ipa.site1.example.com". But then not sure if it will work fine... Technically, two IPs for the same name means load-balancing, right? So will I have intermittent connectivity issues, because it will return inside and outside IP interchangebly?
Bad idea... Indeed, clients get load-balanced responses with the two IPA IPs, and hence, half of the time they can't communicate with it...
- We create a new DNS name, e.g. "ipa-outside-site1.example.com", for the
outside IP, and manually add it to the @ entry of "example.com", so that wannabe-replica on the remote site can use that FQDN as its master IPA. Will this work fine..? Will it not cause issues to the local clients on site1, who must keep using IPA with inside IP? Will it not cause issues on IPA server itself for some reason?
OK, so with this one I managed to get a bit further... I am now just trying to enroll remote host with the IPA server over the "outside" interface (once I am good with this, I could try promoting it to replica...)
So, I created a new DNS zone for the remote site (site2.example.com). I created there SRV entries for "_kerberos._udp" and "_ldap._tcp", pointing to the outside DNS name of my IPA server: "ipa-outside-site1.example.com). Then I set the nameserver to this outside IP and run "ipa-client-install". Since it looks for the above SRV records first, discovery is successful. So I can continue with client installation.
Next problem was: ipa-client-install failed to authenticate to site1 IPA over LDAP, because it was trying to access it using ldap://ipa-outside-site1.example.com (and not via ldap://ipa.site1.example.com). I solved it by adding principal alias to the ldap/ipa.site1.example.com service.
Next problem was: libcurl was complaing about mismatch between certificate subject and the URL. Again, that's because it was browsing to ipa-outside-site1.example.com, but IPA cert has "ipa.site1.example.com" in subject. OK, so I fixed it by adding principal alias to http/ipa.site1.example.com and then reissuing certificate for IPA HTTP, adding a new DNS name to subjectAltName. This solved the warning.
But then my next problem was (and is):
RPC failed at server. Missing or invalid HTTP Referer, https://ipa-outside-site1.example.com/ipa/xml
I understand why - because ipa-client-install puts the above referer, but IPA server expects only https://ipa.site1.example.com... How can I solve this? How can I add another "alias referer" to be accepted..?
Basically, I need to convince all the components of the IPA server that they have two perfectly valid DNS names, with different IPs...
As you noticed, we do not support this scenario.
In order to support it, we would need: - decide what is the 'primary' hostname and maintain other names as aliases for all Kerberos principals associated with the host:
host/name1.example.com would need to have aliases host/name2.example.com, host/name3.example.com...
ldap/name1.example.com would need to have aliases ldap/name2.example.com, ldap/name3.example.com...
HTTP/name1.example.com would need to have aliases HTTP/name2.example.com, HTTP/name3.example.com...
- allow adding name2.example.com, name3.example.com... as dNS SAN entries into certificates issued for services on name1.example.com.
- keep the list of alternative hostnames to validate access to IPA API.
This is not possible at installation time right now. I'm not sure whether the second part will be possible without having nameX.example.com all created as separate host objects and setting managedBy attribute for them to name1.example.com
The latter will then break the first part: it is possible to add host/nameX.example.com as a Kerberos principal alias to existing host or service principal (ipa host-add-principal/service-add-principal), but adding nameX.example.com as a separate host object would interfere with that because you cannot then add an alias to host/name1.example.com named as host/nameX.example.com.
We don't really want to have separate Kerberos principals for */nameX.example.com because that would mean using separate keys for them. We definitely want to have them as aliases (a.k.a. SPNs in Active Directory) that share the same key. The latter is only possible when they aren't separate objects.
And all of it then needed to be added to the installer as a special logic.
The third part is a minor thing in terms of fixing it: https://github.com/abbra/freeipa/pull/9/files. But I don't want to submit it without fixing all the other pieces.
Hi Alexander,
Yes, indeed, I got the feeling that it is not supported =) Problem is that it cought me by surprise in the middle of a huge project, as I wouldn't expect that - out of all things - this would not have a solution. First hit was when I learned that "DNS views" were not supported, but I still had hope that I could find some trick with separate DNS names...
Having this done automatically during installation time is too much luxary for me, I will be happy also with manual tweaks at this point =)
To your items:
In order to support it, we would need:
decide what is the 'primary' hostname and maintain other names as aliases for all Kerberos principals associated with the host:
host/name1.example.com would need to have aliases host/name2.example.com, host/name3.example.com...
ldap/name1.example.com would need to have aliases ldap/name2.example.com, ldap/name3.example.com...
HTTP/name1.example.com would need to have aliases HTTP/name2.example.com, HTTP/name3.example.com...
Yes, this is what I did (manually).
- allow adding name2.example.com, name3.example.com... as dNS SAN entries into certificates issued for services on name1.example.com.
Yes, this actually works today. I managed to reissue the certificate for HTTP endpoint on IPA server, simply with "ipa-getcert resubmit", adding two times "-D" flag. I got a certificate with two DNS:SAN entries, and this solved my issue with libcurl validating certificates. Of course, to do that, I had to add the new DNS name as principal alias to the existing IPA object (was enough to add it to service object - http/name1.example.com). So there was no need to create separate hosts and/or services.
- keep the list of alternative hostnames to validate access to IPA API.
The third part is a minor thing in terms of fixing it: https://github.com/abbra/freeipa/pull/9/files. But I don't want to submit it without fixing all the other pieces.
Well, looks like the rest can be done at least manually today.
Problem is that even if you push the fix today, it will take a long time until we will see it in RHEL. Do you see any manual way to make it work today, as a workaround? (only the third point I mean, because the first two are solvable).
--- Regards, Dmitry Perets
On ti, 30 heinä 2019, Dmitry Perets via FreeIPA-users wrote:
Hi Alexander,
Yes, indeed, I got the feeling that it is not supported =) Problem is that it cought me by surprise in the middle of a huge project, as I wouldn't expect that - out of all things - this would not have a solution. First hit was when I learned that "DNS views" were not supported, but I still had hope that I could find some trick with separate DNS names...
Having this done automatically during installation time is too much luxary for me, I will be happy also with manual tweaks at this point =)
To your items:
In order to support it, we would need:
decide what is the 'primary' hostname and maintain other names as aliases for all Kerberos principals associated with the host:
host/name1.example.com would need to have aliases host/name2.example.com, host/name3.example.com...
ldap/name1.example.com would need to have aliases ldap/name2.example.com, ldap/name3.example.com...
HTTP/name1.example.com would need to have aliases HTTP/name2.example.com, HTTP/name3.example.com...
Yes, this is what I did (manually).
- allow adding name2.example.com, name3.example.com... as dNS SAN entries into certificates issued for services on name1.example.com.
Yes, this actually works today. I managed to reissue the certificate for HTTP endpoint on IPA server, simply with "ipa-getcert resubmit", adding two times "-D" flag. I got a certificate with two DNS:SAN entries, and this solved my issue with libcurl validating certificates. Of course, to do that, I had to add the new DNS name as principal alias to the existing IPA object (was enough to add it to service object - http/name1.example.com). So there was no need to create separate hosts and/or services.
Good to hear. I was not sure it was allowed as we have quite a number of rules around certificate issuance.
- keep the list of alternative hostnames to validate access to IPA API.
The third part is a minor thing in terms of fixing it: https://github.com/abbra/freeipa/pull/9/files. But I don't want to submit it without fixing all the other pieces.
Well, looks like the rest can be done at least manually today.
Problem is that even if you push the fix today, it will take a long time until we will see it in RHEL.
In RHEL 6 and 7 -- unlikely, given that both are already beyond a phase where RFEs like this could be added.
Do you see any manual way to make it work today, as a workaround? (only the third point I mean, because the first two are solvable).
You need to change the code any way.
Hi Alexander,
Thanks again for your help on this...
Just one question regarding your fix here: https://github.com/abbra/freeipa/pull/9/files
When you say "host aliases", what exactly do you mean? DNS CNAME records? Because then, I am afraid, this would not solve my problem. I can't create CNAME record name2.example.com for my IPA server, because the whole problem was that name2 and name1 must resolve to different IPs. Which CNAME cannot do for me.
I was thinking that HTTP Referers check should accept any of the configured _principal aliases_ (added to the service HTTP/name1.example.com). This would be a true "IPA-way", I would say... But at least from a manual test of your code fix - this doesn't seem to be the case (or maybe I did it wrong).
Well... the (kinda) good news is that when I commented out the referers check I could finally enroll without further errors.
--- Regards, Dmitry Perets
On ti, 30 heinä 2019, Dmitry Perets via FreeIPA-users wrote:
Hi Alexander,
Thanks again for your help on this...
Just one question regarding your fix here: https://github.com/abbra/freeipa/pull/9/files
When you say "host aliases", what exactly do you mean? DNS CNAME records?
Any other host names this host is known about so that IPA CLI requests could come with the HTTP referrer for it.
Because then, I am afraid, this would not solve my problem. I can't create CNAME record name2.example.com for my IPA server, because the whole problem was that name2 and name1 must resolve to different IPs. Which CNAME cannot do for me.
You can add A/AAAA records.
I was thinking that HTTP Referers check should accept any of the configured _principal aliases_ (added to the service HTTP/name1.example.com). This would be a true "IPA-way", I would say... But at least from a manual test of your code fix - this doesn't seem to be the case (or maybe I did it wrong).
We cannot -- at the point of this check we aren't talking to LDAP to see a list of aliases. We only rely on what is set in the config files, so you should set this in /etc/ipa/server.conf
[global] host_aliases = name2.example.com name3.example.com ...
Well... the (kinda) good news is that when I commented out the referers check I could finally enroll without further errors.
You know that you are basically opening your environment for CSRF and phishing attack?
You know that you are basically opening your environment for CSRF and phishing attack?
Yes, absolutely! That was just to ensure that it was the last visible problem (still in lab environment).
Now returned the check, added the host_aliases, as you shown above - and it works. Again, thanks a lot!
--- Regards, Dmitry Perets
Hi Alexander,
I am going to submit an RFE via Red Hat support case, to support multihomed IPA server setup. I list two proposals in the RFE:
(1) Support Split DNS (views) which, I believe, is already supported by the underlying BIND. This would allow us to define one view for IPA servers and another view for all the rest. The IPA servers view would resolve IPA servers to their external IPs, so that they can communicate to each other. And the other view would resolve the same IPA servers to their internal IPs, for the local clients. I think this would be a simple solution for us, that would avoid the need for all the tweaks described here. No need for host aliases then.
(2) Support host aliases - based on all the discussion in this thread (DNS:SAN in all IPA certs + proper support for aliases in HTTP Referer). Basically, make everything that we discussed officially qualified and supported.
For us - any of the above solutions would do the job, I think. I suppose, that's the right thing to ask for the long term.
In the shorter term - I was just wondering if it would be possible just to commit that pending code change related to the HTTP Referers that you shared, considering that it is anyway open for a while now? Because as you saw, all the rest can be solved by configuration, and I am happy to automate it for our project. But of course, manually changing the code in production is a "no go", due to supportability. Having this small fix in place would greatly help us to continue, otherwise we have to consider difficult design changes...
--- Regards, Dmitry Perets
On ti, 06 elo 2019, Dmitry Perets via FreeIPA-users wrote:
Hi Alexander,
I am going to submit an RFE via Red Hat support case, to support multihomed IPA server setup. I list two proposals in the RFE:
(1) Support Split DNS (views) which, I believe, is already supported by the underlying BIND. This would allow us to define one view for IPA servers and another view for all the rest. The IPA servers view would resolve IPA servers to their external IPs, so that they can communicate to each other. And the other view would resolve the same IPA servers to their internal IPs, for the local clients. I think this would be a simple solution for us, that would avoid the need for all the tweaks described here. No need for host aliases then.
It is not going to be supported, sorry. bind-dyndb-ldap is too invasive and has to re-implement bunch of places in Bind already. It relies on some of the views code to do so but we cannot support full split horizon there. It also breaks badly with DNSSEC.
(2) Support host aliases - based on all the discussion in this thread (DNS:SAN in all IPA certs + proper support for aliases in HTTP Referer). Basically, make everything that we discussed officially qualified and supported.
I think what is not implemented is automatic discovery of this situation and configuring the aliases. My opinion is that we need an input from admin, not automatic discovery here.
This could, perhaps, be done with an addition to ansible-freeipa for post-install configuration once we fix aliases in HTTP referer. AFAIU, that would be a better place because it allows us to not extend IPA installers' options and still automate the configuration. It would also allow admins to have a repeatable playbook that describes what exactly they want.
For us - any of the above solutions would do the job, I think. I suppose, that's the right thing to ask for the long term.
In the shorter term - I was just wondering if it would be possible just to commit that pending code change related to the HTTP Referers that you shared, considering that it is anyway open for a while now? Because as you saw, all the rest can be solved by configuration, and I am happy to automate it for our project. But of course, manually changing the code in production is a "no go", due to supportability. Having this small fix in place would greatly help us to continue, otherwise we have to consider difficult design changes...
HTTP referers update could be done, yes. However, most likely this will happen in FreeIPA 4.8+ only which means RHEL 8.x. Next RHEL 7 update that we could develop for is going to be past the timeline when features (RFEs) could be added.
Hi Alexander,
Posting here, as it might be useful to others.
I've got a suggestion via Red Hat support ticket that looks promising... So, earlier I tried to configure all IPA servers to resolve to internal IP and then add external IP to their /etc/hosts when creating replica. And that failed badly, because replica install relies on DNS heavily and does all forward/reverse lookups... But now I was asked to do the other way around: resolve IPA servers to external IPs, but add their internal IP to /etc/hosts on all internal clients.
And so far it seems to do the job actually... ipa-client-install is fine with that. I did have an issue to reach KDC, but I solved it by adding to /etc/hosts not only the IPA FQDN, but also the FQDN ending with dot (.) - because that's what _kerberos.* entries return to the client.
So, as long as none of the internal clients needs to become a replica (and it's indeed so in our case), this solution might actually work. Of course, it's not ideal to add entries to /etc/hosts like this, but we can add it to our automated deployment, so we can live with that.
Any issues that you predict with this solution...?
--- Regards, Dmitry Perets
On Tue, Jul 30, 2019 at 3:28 PM Dmitry Perets via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
The progress so far...
- We create two A records for the same IPA hostname, let's say
"ipa.site1.example.com". But then not sure if it will work fine... Technically, two IPs for the same name means load-balancing, right? So will I have intermittent connectivity issues, because it will return inside and outside IP interchangebly?
Bad idea... Indeed, clients get load-balanced responses with the two IPA IPs, and hence, half of the time they can't communicate with it...
- We create a new DNS name, e.g. "ipa-outside-site1.example.com", for the
outside IP, and manually add it to the @ entry of "example.com", so that wannabe-replica on the remote site can use that FQDN as its master IPA. Will this work fine..? Will it not cause issues to the local clients on site1, who must keep using IPA with inside IP? Will it not cause issues on IPA server itself for some reason?
OK, so with this one I managed to get a bit further... I am now just trying to enroll remote host with the IPA server over the "outside" interface (once I am good with this, I could try promoting it to replica...)
So, I created a new DNS zone for the remote site (site2.example.com). I created there SRV entries for "_kerberos._udp" and "_ldap._tcp", pointing to the outside DNS name of my IPA server: "ipa-outside-site1.example.com). Then I set the nameserver to this outside IP and run "ipa-client-install". Since it looks for the above SRV records first, discovery is successful. So I can continue with client installation.
You do not need to add SRV records, this is difficult to manage in the long run.
ipa-client-install(1) covers this scenario with the --domain=DOMAIN parameter.
See also https://bugzilla.redhat.com/show_bug.cgi?id=1385515#c14 for a detailed explanation.
Next problem was: ipa-client-install failed to authenticate to site1 IPA over LDAP, because it was trying to access it using ldap://ipa-outside-site1.example.com (and not via ldap://ipa.site1.example.com). I solved it by adding principal alias to the ldap/ipa.site1.example.com service.
Next problem was: libcurl was complaing about mismatch between certificate subject and the URL. Again, that's because it was browsing to ipa-outside-site1.example.com, but IPA cert has "ipa.site1.example.com" in subject. OK, so I fixed it by adding principal alias to http/ipa.site1.example.com and then reissuing certificate for IPA HTTP, adding a new DNS name to subjectAltName. This solved the warning.
But then my next problem was (and is):
RPC failed at server. Missing or invalid HTTP Referer, https://ipa-outside-site1.example.com/ipa/xml
I understand why - because ipa-client-install puts the above referer, but IPA server expects only https://ipa.site1.example.com... How can I solve this? How can I add another "alias referer" to be accepted..?
Basically, I need to convince all the components of the IPA server that they have two perfectly valid DNS names, with different IPs...
Regards, Dmitry Perets _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Francois,
You do not need to add SRV records, this is difficult to manage in the long run.
ipa-client-install(1) covers this scenario with the --domain=DOMAIN parameter.
See also https://bugzilla.redhat.com/show_bug.cgi?id=1385515#c14 for a detailed explanation.
Well, that is not enough in my case... It is not a problem to detect the domain. The problem is that existing SRV records in the main domain contain IPA server hostname that is resolvable to site1-internal IP address. And I must keep it this way, for the clients inside site1. But I can't reach those IP addresses from site2. So in order to create a replica in site2, I have to "trick" it with the new SRV records, pointing to the new DNS name that resolves to the external IP on the external NIC of the site1-IPA server... Except, of course, that even that trick didn't work, due to HTTP Referal mismatch...
--- Regards, Dmitry Perets
On 30/07/2019 09:34, Dmitry Perets via FreeIPA-users wrote:
Hi,
I'd like to ask for your advise for the following topology... On a given site, IPA server has two legs (two NICs), let's call them "inside NIC" and "outside NIC". The inside NIC subnet is local to the site. The outside NIC subnet is interconnecting sites.
All the local clients talk to IPA via the inside NIC. But to setup a replica on another site, we must reach IPA via outside NIC (the inside subnet is not routable beyond the local site boundaries).
So the question arises: how to configure proper DNS resolution for the hostname of the IPA server itself? DNS is handled by IPA itself, fully in our control. So we have two options:
We create two A records for the same IPA hostname, let's say "ipa.site1.example.com". But then not sure if it will work fine... Technically, two IPs for the same name means load-balancing, right? So will I have intermittent connectivity issues, because it will return inside and outside IP interchangebly?
We create a new DNS name, e.g. "ipa-outside-site1.example.com", for the outside IP, and manually add it to the @ entry of "example.com", so that wannabe-replica on the remote site can use that FQDN as its master IPA. Will this work fine..? Will it not cause issues to the local clients on site1, who must keep using IPA with inside IP? Will it not cause issues on IPA server itself for some reason?
normally it would be... if only IPA's dns supported "views" :) maybe some day?
Please share your experience on this! Thanks. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org