Hi all,
This is a newbie question with respect to FreeIPA, and I haven't seen this elsewhere, so I thought I'd ask.
I've just cleaned up an issue with trying to implement a new replica on our domain, and I've realized that there are a couple of areas I don't understand that are causing more stress than I need at this point, for I am maximally paranoid and overworked, and I need some advice on how not to make that worse when implementing FreeIPA.
I've been reading this list on and off for most of the last year, and what's struck me is how complicated this project is, especially of course areas where I personally have less expertise (e.g., LDAP, Kerberos). Implementing and managing this is just one of my jobs, and our IdM becomes more critical as time goes on, so I need to understand a few things.
Question 1: As I just had, for the first time, to manually modify LDAP to remove data, I'd like to understand how taking that approach can backfire. In other words, it's clear this isn't habitually a good idea, because mistakes will replicate, for example. But, where are the real danger points: for example, I've seen stories of having to recover a server in an environment where the time was intentionally set back to allow an operation on an expired cert. As with a database update that triggers other (unexpected) changes, are there LDAP operations that can't practically be "un-done"? I know that deleting records is permanent (obvious), but are there gotchas like changing a particular object fires some large number of events that there's no way to revert? Or, in colloquial terms, are there places that are really, really bad to try to outwit the management interfaces? I'm not talking everyday updates, but instances where we get stuck (as I was yesterday and today)?
Question 2: How to stay safe. Our installation on a small network as a pair of masters replicating with each other, CA and DNS installed on both. They are VMs allocated on separate physical hosts. Before updates, I snapshot both for recovery purposes. I update one at a time, now, and ensure that they're functional before doing the other one (I know that schema changes will propagate before service is applied elsewhere, but I can't do anything else about that). I haven't had to deal with the question of how to make sure my (self-signed) CA certificate doesn't expire, but I know when it will and am leaving myself ample time to understand that. I'm about to start pushing out this functionality to multiple geos, again at small-ish scale, and I'm reading the topology references to be sure I understand what I'm doing. But: what am I missing? I get the impression that trying to (conventionally) "back up" data isn't useful. I've tried to design the network to make it as simple as possible to meet our needs. Does anyone else have anything to add in the sense of "best practices", or to echo my first question "there be dragons <here>"?
I've learned a lot from this list, and I'd also like to add a thank you to everyone here who have helped with that! I do see this getting better and better, and it's appreciated.
K. M. Peterson via FreeIPA-users wrote:
Hi all,
This is a newbie question with respect to FreeIPA, and I haven't seen this elsewhere, so I thought I'd ask.
I've just cleaned up an issue with trying to implement a new replica on our domain, and I've realized that there are a couple of areas I don't understand that are causing more stress than I need at this point, for I am maximally paranoid and overworked, and I need some advice on how not to make that worse when implementing FreeIPA.
I've been reading this list on and off for most of the last year, and what's struck me is how complicated this project is, especially of course areas where I personally have less expertise (e.g., LDAP, Kerberos). Implementing and managing this is just one of my jobs, and our IdM becomes more critical as time goes on, so I need to understand a few things.
Question 1: As I just had, for the first time, to manually modify LDAP to remove data, I'd like to understand how taking that approach can backfire. In other words, it's clear this isn't habitually a good idea, because mistakes will replicate, for example. But, where are the real danger points: for example, I've seen stories of having to recover a server in an environment where the time was intentionally set back to allow an operation on an expired cert. As with a database update that triggers other (unexpected) changes, are there LDAP operations that can't practically be "un-done"? I know that deleting records is permanent (obvious), but are there gotchas like changing a particular object fires some large number of events that there's no way to revert? Or, in colloquial terms, are there places that are really, really bad to try to outwit the management interfaces? I'm not talking everyday updates, but instances where we get stuck (as I was yesterday and today)?
Question 2: How to stay safe. Our installation on a small network as a pair of masters replicating with each other, CA and DNS installed on both. They are VMs allocated on separate physical hosts. Before updates, I snapshot both for recovery purposes. I update one at a time, now, and ensure that they're functional before doing the other one (I know that schema changes will propagate before service is applied elsewhere, but I can't do anything else about that). I haven't had to deal with the question of how to make sure my (self-signed) CA certificate doesn't expire, but I know when it will and am leaving myself ample time to understand that. I'm about to start pushing out this functionality to multiple geos, again at small-ish scale, and I'm reading the topology references to be sure I understand what I'm doing. But: what am I missing? I get the impression that trying to (conventionally) "back up" data isn't useful. I've tried to design the network to make it as simple as possible to meet our needs. Does anyone else have anything to add in the sense of "best practices", or to echo my first question "there be dragons <here>"?
I've learned a lot from this list, and I'd also like to add a thank you to everyone here who have helped with that! I do see this getting better and better, and it's appreciated.
Sort of reverse order...
So yeah, there are a lot of moving parts in IPA. We try to minimize where there are looser connections but there are still some things that can crop up, for all kinds of reasons, some of which are still baffling to us (like all the reasons that certs don't renew automatically).
In an effort to shine some light on this we are planning a monitoring tool that will check various things that can wrong and provide advanced notification of them (certs, file permissions, replication, etc).
It is still very much in the planning stage and will take some time to complete but it's the kind of thing that will grow over time to hopefully let you sleep better are night, or at least minimize the impact of some failures.
As for backups yes, they are non-trivial. The IPA backups should generally work but these aren't really made for the "ooops, I deleted this one record" case. They are for the "oh crap I just lost my disk" case. Snapshots work as well or better (IMHO an offline full system backup is golden).
If you are using the IPA self-signed CA it is good for 20 years. It is renewable (not automatic). If you are using an external CA or certs then it's up to you to manage renewal now (the healthcheck will notice too eventually).
Mistakes on one entry are hard to deal with. Theoretically you can resurrect a deleted entry in 389-ds. It's in the docs (I've never done it myself).
A data-only backup can be useful for this type of case. It should be pretty straigthforward to pull the value(s) out of the ldif to try to get things reset, depending on the relationships of the thing you deleted. Recovering a leaf node deletion is easy. Deleting a group with 1000 users is a bit more complex as it affects a lot more entries. I can't think of any gotchas, like I said, we _try_ to keep things safe for you.
rob
Rob,
Thanks for your response, it was very helpful!
A monitoring tool would be great, I should say that I am sleeping better because I'm getting to the point where I can at least back-out things when there's that kind of failure.
It sounds like the level of caution that I've applied to this is appropriate. I was reading a blog post https://fy.blackhats.net.au/blog/html/2018/12/21/identity_ideas.html yesterday from William Brown that to some extent confirms what you'd said - and my experience.
It's really helpful to hear, and I appreciate the assist. I hope I can get to the point where I have something to contribute to this project.
One more thing: I think I've figured out the series (or, perhaps "a" series) of events that got me into this situation. It appears that initially on client installation for some reason, I would receive errors indicating that there was a problem with the OpenLDAP configuration and it failed because it could not update the /etc/openldap/ldap.conf file. My base installation doesn't have such a file, and previous installations of clients may or may not have hit the same issue but it was never a problem for us. I created /etc/openldap/ldap.conf with a single "#" line, re-spun the server and installed the client and ipa-replica-install - and everything is working as expected now.
Again, many thanks!
On Tue, Jan 8, 2019 at 5:25 PM Rob Crittenden rcritten@redhat.com wrote:
K. M. Peterson via FreeIPA-users wrote:
Hi all,
This is a newbie question with respect to FreeIPA, and I haven't seen this elsewhere, so I thought I'd ask.
I've just cleaned up an issue with trying to implement a new replica on our domain, and I've realized that there are a couple of areas I don't understand that are causing more stress than I need at this point, for I am maximally paranoid and overworked, and I need some advice on how not to make that worse when implementing FreeIPA.
I've been reading this list on and off for most of the last year, and what's struck me is how complicated this project is, especially of course areas where I personally have less expertise (e.g., LDAP, Kerberos). Implementing and managing this is just one of my jobs, and our IdM becomes more critical as time goes on, so I need to understand a few things.
Question 1: As I just had, for the first time, to manually modify LDAP to remove data, I'd like to understand how taking that approach can backfire. In other words, it's clear this isn't habitually a good idea, because mistakes will replicate, for example. But, where are the real danger points: for example, I've seen stories of having to recover a server in an environment where the time was intentionally set back to allow an operation on an expired cert. As with a database update that triggers other (unexpected) changes, are there LDAP operations that can't practically be "un-done"? I know that deleting records is permanent (obvious), but are there gotchas like changing a particular object fires some large number of events that there's no way to revert? Or, in colloquial terms, are there places that are really, really bad to try to outwit the management interfaces? I'm not talking everyday updates, but instances where we get stuck (as I was yesterday and today)?
Question 2: How to stay safe. Our installation on a small network as a pair of masters replicating with each other, CA and DNS installed on both. They are VMs allocated on separate physical hosts. Before updates, I snapshot both for recovery purposes. I update one at a time, now, and ensure that they're functional before doing the other one (I know that schema changes will propagate before service is applied elsewhere, but I can't do anything else about that). I haven't had to deal with the question of how to make sure my (self-signed) CA certificate doesn't expire, but I know when it will and am leaving myself ample time to understand that. I'm about to start pushing out this functionality to multiple geos, again at small-ish scale, and I'm reading the topology references to be sure I understand what I'm doing. But: what am I missing? I get the impression that trying to (conventionally) "back up" data isn't useful. I've tried to design the network to make it as simple as possible to meet our needs. Does anyone else have anything to add in the sense of "best practices", or to echo my first question "there be dragons <here>"?
I've learned a lot from this list, and I'd also like to add a thank you to everyone here who have helped with that! I do see this getting better and better, and it's appreciated.
Sort of reverse order...
So yeah, there are a lot of moving parts in IPA. We try to minimize where there are looser connections but there are still some things that can crop up, for all kinds of reasons, some of which are still baffling to us (like all the reasons that certs don't renew automatically).
In an effort to shine some light on this we are planning a monitoring tool that will check various things that can wrong and provide advanced notification of them (certs, file permissions, replication, etc).
It is still very much in the planning stage and will take some time to complete but it's the kind of thing that will grow over time to hopefully let you sleep better are night, or at least minimize the impact of some failures.
As for backups yes, they are non-trivial. The IPA backups should generally work but these aren't really made for the "ooops, I deleted this one record" case. They are for the "oh crap I just lost my disk" case. Snapshots work as well or better (IMHO an offline full system backup is golden).
If you are using the IPA self-signed CA it is good for 20 years. It is renewable (not automatic). If you are using an external CA or certs then it's up to you to manage renewal now (the healthcheck will notice too eventually).
Mistakes on one entry are hard to deal with. Theoretically you can resurrect a deleted entry in 389-ds. It's in the docs (I've never done it myself).
A data-only backup can be useful for this type of case. It should be pretty straigthforward to pull the value(s) out of the ldif to try to get things reset, depending on the relationships of the thing you deleted. Recovering a leaf node deletion is easy. Deleting a group with 1000 users is a bit more complex as it affects a lot more entries. I can't think of any gotchas, like I said, we _try_ to keep things safe for you.
rob
Rob mentioned issues with restoring data for one entry. We run on VMs, and periodically take snapshots. We can copy a snapshot to a new VM. Since the hostname is critical, edit /etc/hosts and add an entry for the new IP address giving it the original hostname. That way the system will think it’s the original host. Then the system works. At that point you can get entries. ldapsearch will let you get all the attributes, so you could put an old entry back. We actually did this one time.
I generally feel pretty safe with ongoing operation. The last few updates have been mostly OK, though we often have to do some minor intervention. To me the scary thing is adding replicas. Last time I tried, the first attempt failed, and I also had to go through a text dump to find all references and get rid of them. We actually ended up using a new name when we did the second try, just to avoid any chance of issues.
We use commercial certs, so I don’t have to worry about ipa’s cert management. From postings here it looks like a weak point.
On Jan 9, 2019, at 1:05 PM, K. M. Peterson via FreeIPA-users <freeipa-users@lists.fedorahosted.orgmailto:freeipa-users@lists.fedorahosted.org> wrote:
Rob,
Thanks for your response, it was very helpful!
A monitoring tool would be great, I should say that I am sleeping better because I'm getting to the point where I can at least back-out things when there's that kind of failure.
It sounds like the level of caution that I've applied to this is appropriate. I was reading a blog posthttps://fy.blackhats.net.au/blog/html/2018/12/21/identity_ideas.html yesterday from William Brown that to some extent confirms what you'd said - and my experience.
It's really helpful to hear, and I appreciate the assist. I hope I can get to the point where I have something to contribute to this project.
One more thing: I think I've figured out the series (or, perhaps "a" series) of events that got me into this situation. It appears that initially on client installation for some reason, I would receive errors indicating that there was a problem with the OpenLDAP configuration and it failed because it could not update the /etc/openldap/ldap.conf file. My base installation doesn't have such a file, and previous installations of clients may or may not have hit the same issue but it was never a problem for us. I created /etc/openldap/ldap.conf with a single "#" line, re-spun the server and installed the client and ipa-replica-install - and everything is working as expected now.
Again, many thanks!
On Tue, Jan 8, 2019 at 5:25 PM Rob Crittenden <rcritten@redhat.commailto:rcritten@redhat.com> wrote: K. M. Peterson via FreeIPA-users wrote:
Hi all,
This is a newbie question with respect to FreeIPA, and I haven't seen this elsewhere, so I thought I'd ask.
I've just cleaned up an issue with trying to implement a new replica on our domain, and I've realized that there are a couple of areas I don't understand that are causing more stress than I need at this point, for I am maximally paranoid and overworked, and I need some advice on how not to make that worse when implementing FreeIPA.
I've been reading this list on and off for most of the last year, and what's struck me is how complicated this project is, especially of course areas where I personally have less expertise (e.g., LDAP, Kerberos). Implementing and managing this is just one of my jobs, and our IdM becomes more critical as time goes on, so I need to understand a few things.
Question 1: As I just had, for the first time, to manually modify LDAP to remove data, I'd like to understand how taking that approach can backfire. In other words, it's clear this isn't habitually a good idea, because mistakes will replicate, for example. But, where are the real danger points: for example, I've seen stories of having to recover a server in an environment where the time was intentionally set back to allow an operation on an expired cert. As with a database update that triggers other (unexpected) changes, are there LDAP operations that can't practically be "un-done"? I know that deleting records is permanent (obvious), but are there gotchas like changing a particular object fires some large number of events that there's no way to revert? Or, in colloquial terms, are there places that are really, really bad to try to outwit the management interfaces? I'm not talking everyday updates, but instances where we get stuck (as I was yesterday and today)?
Question 2: How to stay safe. Our installation on a small network as a pair of masters replicating with each other, CA and DNS installed on both. They are VMs allocated on separate physical hosts. Before updates, I snapshot both for recovery purposes. I update one at a time, now, and ensure that they're functional before doing the other one (I know that schema changes will propagate before service is applied elsewhere, but I can't do anything else about that). I haven't had to deal with the question of how to make sure my (self-signed) CA certificate doesn't expire, but I know when it will and am leaving myself ample time to understand that. I'm about to start pushing out this functionality to multiple geos, again at small-ish scale, and I'm reading the topology references to be sure I understand what I'm doing. But: what am I missing? I get the impression that trying to (conventionally) "back up" data isn't useful. I've tried to design the network to make it as simple as possible to meet our needs. Does anyone else have anything to add in the sense of "best practices", or to echo my first question "there be dragons <here>"?
I've learned a lot from this list, and I'd also like to add a thank you to everyone here who have helped with that! I do see this getting better and better, and it's appreciated.
Sort of reverse order...
So yeah, there are a lot of moving parts in IPA. We try to minimize where there are looser connections but there are still some things that can crop up, for all kinds of reasons, some of which are still baffling to us (like all the reasons that certs don't renew automatically).
In an effort to shine some light on this we are planning a monitoring tool that will check various things that can wrong and provide advanced notification of them (certs, file permissions, replication, etc).
It is still very much in the planning stage and will take some time to complete but it's the kind of thing that will grow over time to hopefully let you sleep better are night, or at least minimize the impact of some failures.
As for backups yes, they are non-trivial. The IPA backups should generally work but these aren't really made for the "ooops, I deleted this one record" case. They are for the "oh crap I just lost my disk" case. Snapshots work as well or better (IMHO an offline full system backup is golden).
If you are using the IPA self-signed CA it is good for 20 years. It is renewable (not automatic). If you are using an external CA or certs then it's up to you to manage renewal now (the healthcheck will notice too eventually).
Mistakes on one entry are hard to deal with. Theoretically you can resurrect a deleted entry in 389-ds. It's in the docs (I've never done it myself).
A data-only backup can be useful for this type of case. It should be pretty straigthforward to pull the value(s) out of the ldif to try to get things reset, depending on the relationships of the thing you deleted. Recovering a leaf node deletion is easy. Deleting a group with 1000 users is a bit more complex as it affects a lot more entries. I can't think of any gotchas, like I said, we _try_ to keep things safe for you.
rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.orgmailto:freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.orgmailto:freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Charles,
Helpful fo know. The snapshot methodology is what we’ve done as well, though we haven’t yet put it fully into production; I’ll still hold my breath if we need it, but it’s good to hear it has worked for you.
Thanks!
On Wed, Jan 9, 2019 at 13:28 Charles Hedrick hedrick@rutgers.edu wrote:
Rob mentioned issues with restoring data for one entry. We run on VMs, and periodically take snapshots. ...
freeipa-users@lists.fedorahosted.org
-
Charles Hedrick -
K. M. Peterson -
Rob Crittenden