--- configs/db/backup-dbs | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/configs/db/backup-dbs b/configs/db/backup-dbs index 83ed6c8..7af66fb 100755 --- a/configs/db/backup-dbs +++ b/configs/db/backup-dbs @@ -20,6 +20,8 @@ mv $DEST/$HOSTNAME.new $DEST/$HOSTNAME
# Sync out for host in db01 db02 db03; do + # Sleep if any other rsyncs are already + while ssh $host "pgrep rsync" | grep -q [0-9]; do sleep 10; done if [ "$host" != $HOSTNAME ]; then su - dbbackup -c "ssh $host mkdir -p $DEST/$HOSTNAME" su - dbbackup -c "rsync -azr --bwlimit=5000 -e ssh $DEST/$HOSTNAME/* $host:$DEST/$HOSTNAME/"
On Wed, 2010-03-24 at 10:54 -0500, Mike McGrath wrote:
# Sync out for host in db01 db02 db03; do
- # Sleep if any other rsyncs are already
- while ssh $host "pgrep rsync" | grep -q [0-9]; do sleep 10; done
Seems like a pretty wide net, what if somebody is rsyncing something else to/from that box? Would it not be better to make use of a lockfile?
On Wed, 24 Mar 2010, Jesse Keating wrote:
On Wed, 2010-03-24 at 10:54 -0500, Mike McGrath wrote:
# Sync out for host in db01 db02 db03; do
- # Sleep if any other rsyncs are already
- while ssh $host "pgrep rsync" | grep -q [0-9]; do sleep 10; done
Seems like a pretty wide net, what if somebody is rsyncing something else to/from that box? Would it not be better to make use of a lockfile?
I'd think in that scenario we'd want to wait until they're done too.
-Mike
On Wed, Mar 24, 2010 at 10:54:07AM -0500, Mike McGrath wrote:
configs/db/backup-dbs | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/configs/db/backup-dbs b/configs/db/backup-dbs index 83ed6c8..7af66fb 100755 --- a/configs/db/backup-dbs +++ b/configs/db/backup-dbs @@ -20,6 +20,8 @@ mv $DEST/$HOSTNAME.new $DEST/$HOSTNAME
# Sync out for host in db01 db02 db03; do
- # Sleep if any other rsyncs are already
- while ssh $host "pgrep rsync" | grep -q [0-9]; do sleep 10; done if [ "$host" != $HOSTNAME ]; then su - dbbackup -c "ssh $host mkdir -p $DEST/$HOSTNAME" su - dbbackup -c "rsync -azr --bwlimit=5000 -e ssh $DEST/$HOSTNAME/* $host:$DEST/$HOSTNAME/"
So this means that among the three db hosts, db01, db02, and db03 we'll have at most one rsync running at any one time. That's going to increase the time to sync some more.
I think that this code wonn't quite function properly when more than one backup-dbs script runs on a box at a time... what gets transferred to the remote host will be a mixture of what was in $DEST/$HOSTNAME when the rsync starts and what's in there when the rsync ends.. We would get errors if filenames were removed (ie: a database is removed while the rsync is still procesing). Do to copying the directory prior to rsyncing, I don't think we'll get corruption of the actual dump files -- we'll just get dump files from two separate runs intermingled on the host being backed up to.
It doesn't look like the scripts are currently stacking on a single host. They run at every six hours on db01 and db02 (every 12 hours on db03).
So the risk of problems at this time doesn't seem too bad.
+1
-Toshio
On Wed, 24 Mar 2010, Toshio Kuratomi wrote:
On Wed, Mar 24, 2010 at 10:54:07AM -0500, Mike McGrath wrote:
configs/db/backup-dbs | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/configs/db/backup-dbs b/configs/db/backup-dbs index 83ed6c8..7af66fb 100755 --- a/configs/db/backup-dbs +++ b/configs/db/backup-dbs @@ -20,6 +20,8 @@ mv $DEST/$HOSTNAME.new $DEST/$HOSTNAME
# Sync out for host in db01 db02 db03; do
- # Sleep if any other rsyncs are already
- while ssh $host "pgrep rsync" | grep -q [0-9]; do sleep 10; done if [ "$host" != $HOSTNAME ]; then su - dbbackup -c "ssh $host mkdir -p $DEST/$HOSTNAME" su - dbbackup -c "rsync -azr --bwlimit=5000 -e ssh $DEST/$HOSTNAME/* $host:$DEST/$HOSTNAME/"
So this means that among the three db hosts, db01, db02, and db03 we'll have at most one rsync running at any one time. That's going to increase the time to sync some more.
I think that this code wonn't quite function properly when more than one backup-dbs script runs on a box at a time... what gets transferred to the remote host will be a mixture of what was in $DEST/$HOSTNAME when the rsync starts and what's in there when the rsync ends.. We would get errors if filenames were removed (ie: a database is removed while the rsync is still procesing). Do to copying the directory prior to rsyncing, I don't think we'll get corruption of the actual dump files -- we'll just get dump files from two separate runs intermingled on the host being backed up to.
It doesn't look like the scripts are currently stacking on a single host. They run at every six hours on db01 and db02 (every 12 hours on db03).
So the risk of problems at this time doesn't seem too bad.
+1
In the above scenarios we'd at least get an error. The sync times only take a few minutes right now. If one hangs long enough to hit another backup we'd at least get notified about it. We'd be notified now and so far that hasn't happened but it's something to think about. perhaps we want to re-think this whole script and just store the data somewhere else completely.
The initial idea was that if say, db1 died. Db2 and db3 could take it's place, and since the dumps are already there, downtime is kept to a minimum. But if it's causing us performance issues getting the data there perhaps it's not worth it. I may also want to tone the rsync throttle down a bit more. Something to think about after the freeze.
-Mike
On Wednesday 24 March 2010 12:08:00 pm Toshio Kuratomi wrote:
On Wed, Mar 24, 2010 at 10:54:07AM -0500, Mike McGrath wrote:
configs/db/backup-dbs | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/configs/db/backup-dbs b/configs/db/backup-dbs index 83ed6c8..7af66fb 100755 --- a/configs/db/backup-dbs +++ b/configs/db/backup-dbs @@ -20,6 +20,8 @@ mv $DEST/$HOSTNAME.new $DEST/$HOSTNAME
# Sync out for host in db01 db02 db03; do
# Sleep if any other rsyncs are already
while ssh $host "pgrep rsync" | grep -q [0-9]; do sleep 10; done
if [ "$host" != $HOSTNAME ]; then
su - dbbackup -c "ssh $host mkdir -p $DEST/$HOSTNAME" su - dbbackup -c "rsync -azr --bwlimit=5000 -e ssh $DEST/$HOSTNAME/* $host:$DEST/$HOSTNAME/"So this means that among the three db hosts, db01, db02, and db03 we'll have at most one rsync running at any one time. That's going to increase the time to sync some more.
I think that this code wonn't quite function properly when more than one backup-dbs script runs on a box at a time... what gets transferred to the remote host will be a mixture of what was in $DEST/$HOSTNAME when the rsync starts and what's in there when the rsync ends.. We would get errors if filenames were removed (ie: a database is removed while the rsync is still procesing). Do to copying the directory prior to rsyncing, I don't think we'll get corruption of the actual dump files -- we'll just get dump files from two separate runs intermingled on the host being backed up to.
It doesn't look like the scripts are currently stacking on a single host. They run at every six hours on db01 and db02 (every 12 hours on db03).
So the risk of problems at this time doesn't seem too bad.
+1
-Toshio
+1 here
Dennis
infrastructure@lists.fedoraproject.org
-
Dennis Gilmore -
Jesse Keating -
Mike McGrath -
Toshio Kuratomi