As part of some ongoing work we are doing [1] to unify ostree repos I'd like to inquire about backup policy and data recovery for the storage that holds ostree repos within Fedora infra. The following directory is the subject:
https://kojipkgs.fedoraproject.org/atomic/
Can someone tell me what we do to back those files up, how often we snapshot, how long the backups are kept and how we would go about restoring the content?
Thanks, Dusty
On 07/07/2017 05:12 PM, Dusty Mabe wrote:
As part of some ongoing work we are doing [1] to unify ostree repos I'd like to inquire about backup policy and data recovery for the storage that holds ostree repos within Fedora infra. The following directory is the subject:
https://kojipkgs.fedoraproject.org/atomic/
Can someone tell me what we do to back those files up, how often we snapshot, how long the backups are kept and how we would go about restoring the content?
forgot the [1]
On 07/07/2017 03:12 PM, Dusty Mabe wrote:
As part of some ongoing work we are doing [1] to unify ostree repos I'd like to inquire about backup policy and data recovery for the storage that holds ostree repos within Fedora infra. The following directory is the subject:
https://kojipkgs.fedoraproject.org/atomic/
Can someone tell me what we do to back those files up, how often we snapshot, how long the backups are kept and how we would go about restoring the content?
This space is on our fedora_koji netapp volume.
There's daily snapshots for 5 days currently.
There's also a snapmirror from phx2 to offsite space in rdu2.
Restores from snapshot would be via infra ticket and someone with access to the netapp can get them.
If we need a different backup caidence here, we may need to move this to different storage.
kevin
On 07/10/2017 11:24 AM, Kevin Fenzi wrote:
On 07/07/2017 03:12 PM, Dusty Mabe wrote:
As part of some ongoing work we are doing [1] to unify ostree repos I'd like to inquire about backup policy and data recovery for the storage that holds ostree repos within Fedora infra. The following directory is the subject:
https://kojipkgs.fedoraproject.org/atomic/
Can someone tell me what we do to back those files up, how often we snapshot, how long the backups are kept and how we would go about restoring the content?
This space is on our fedora_koji netapp volume.
There's daily snapshots for 5 days currently.
so backups are taken daily. cool. Is it possible to view what the snapshots are? i.e. readonly mountpoints on infra machines where content could be browsed. That would make requesting a restoration super easy since the requester could investigate by inspecting readonly directories before hand. This also may make it so that a restoration isn't needed at all for some cases where only a few files need to be restored.
There's also a snapmirror from phx2 to offsite space in rdu2.
What exactly does this mean?
Restores from snapshot would be via infra ticket and someone with access to the netapp can get them.
If we need a different backup caidence here, we may need to move this to different storage.
kevin
Thanks Kevin! If this info is documented somewhere already then please point me to it, or maybe we should add it somewhere.
Dusty
On 10 July 2017 at 12:54, Dusty Mabe dusty@dustymabe.com wrote:
On 07/10/2017 11:24 AM, Kevin Fenzi wrote:
On 07/07/2017 03:12 PM, Dusty Mabe wrote:
As part of some ongoing work we are doing [1] to unify ostree repos I'd like to inquire about backup policy and data recovery for the storage that holds ostree repos within Fedora infra. The following directory is the subject:
https://kojipkgs.fedoraproject.org/atomic/
Can someone tell me what we do to back those files up, how often we snapshot, how long the backups are kept and how we would go about restoring the content?
This space is on our fedora_koji netapp volume.
There's daily snapshots for 5 days currently.
so backups are taken daily. cool. Is it possible to view what the snapshots are? i.e. readonly mountpoints on infra machines where content could be browsed. That would make requesting a restoration super easy since the requester could investigate by inspecting readonly directories before hand. This also may make it so that a restoration isn't needed at all for some cases where only a few files need to be restored.
OK let us be careful about terminology here. Backups are usually considered data which is taken of a system and held somewhere else to be restored later. (EG a tape backup, a rdiff backup, etc).
Netapp snapshots are not backups in that sense. What you get is the ability to go into a .snapshot directory and look at what was present at a certain point to certain limits (netapp snapshots are similar (but not the same) to a hardlink list of data on disk at a certain time. You can only change so much or keep so many before you run out of snapshot space and those go away).
Snapshots can be seen on certain machines by going into the .snapshot directory.
There's also a snapmirror from phx2 to offsite space in rdu2.
What exactly does this mean?
The Red Hat Netapps regularly copy the data from RDU2 to PHX2. As long as we don't change too much over on PHX2, then the RDU2 side is within sync with PHX2 within an hour. If we have a large delta or other network traffic is full it can take many hours or certain snapshots may get 'dropped' because they are no longer possible to reconstruct.
On 07/10/2017 10:15 AM, Stephen John Smoogen wrote:
On 10 July 2017 at 12:54, Dusty Mabe dusty@dustymabe.com wrote:
On 07/10/2017 11:24 AM, Kevin Fenzi wrote:
On 07/07/2017 03:12 PM, Dusty Mabe wrote:
As part of some ongoing work we are doing [1] to unify ostree repos I'd like to inquire about backup policy and data recovery for the storage that holds ostree repos within Fedora infra. The following directory is the subject:
https://kojipkgs.fedoraproject.org/atomic/
Can someone tell me what we do to back those files up, how often we snapshot, how long the backups are kept and how we would go about restoring the content?
This space is on our fedora_koji netapp volume.
There's daily snapshots for 5 days currently.
so backups are taken daily. cool. Is it possible to view what the snapshots are? i.e. readonly mountpoints on infra machines where content could be browsed. That would make requesting a restoration super easy since the requester could investigate by inspecting readonly directories before hand. This also may make it so that a restoration isn't needed at all for some cases where only a few files need to be restored.
OK let us be careful about terminology here. Backups are usually considered data which is taken of a system and held somewhere else to be restored later. (EG a tape backup, a rdiff backup, etc).
Netapp snapshots are not backups in that sense. What you get is the ability to go into a .snapshot directory and look at what was present at a certain point to certain limits (netapp snapshots are similar (but not the same) to a hardlink list of data on disk at a certain time. You can only change so much or keep so many before you run out of snapshot space and those go away).
Snapshots can be seen on certain machines by going into the .snapshot directory.
I'll add that we disabled this on most machines. It caused various scripts to look at the snapshot data when it shouldn't have. We could re-enable it on some specific machines if needed, but right now it's off I think.
There's also a snapmirror from phx2 to offsite space in rdu2.
What exactly does this mean?
The Red Hat Netapps regularly copy the data from RDU2 to PHX2. As long as we don't change too much over on PHX2, then the RDU2 side is within sync with PHX2 within an hour. If we have a large delta or other network traffic is full it can take many hours or certain snapshots may get 'dropped' because they are no longer possible to reconstruct.
Yeah, basically it means we have a copy (as up to date as sync time) in RDU2 of our PHX2 koji data.
kevin
infrastructure@lists.fedoraproject.org