First, I noticed we are running the full sync twice right now, at the same time:
[root@mm-backend01 cron.d][PROD]# cat /etc/cron.d/s3.sh #Ansible: s3sync 0 0,11 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2> &1 #Ansible: s3sync-main 0 0 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync-main /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2>&1
Second, the attached patch changes the sync scripts to:
* do one sync with no --delete and excluding repodata * do another one with --delete and including repodata * invalidate the repodata
I adjusted the cron jobs to handle the repodata invalidate (I think).
TODO: only sync when things have changed.
+1s?
kevin --
+1 ok so the s3sync does allow for delete. I thought from previous conversations it didn't work that way so my view of what was needed was wrong.
On Thu, 5 Mar 2020 at 16:06, Kevin Fenzi kevin@scrye.com wrote:
First, I noticed we are running the full sync twice right now, at the same time:
[root@mm-backend01 cron.d][PROD]# cat /etc/cron.d/s3.sh #Ansible: s3sync 0 0,11 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2> &1 #Ansible: s3sync-main 0 0 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync-main /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2>&1
Second, the attached patch changes the sync scripts to:
- do one sync with no --delete and excluding repodata
- do another one with --delete and including repodata
- invalidate the repodata
I adjusted the cron jobs to handle the repodata invalidate (I think).
TODO: only sync when things have changed.
+1s?
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
The only thing which is not clear in the documentation is when the delete is happening. With rsync there is an option '--delete-after' to make sure first all files are transferred and deletion happens afterwards. If 'aws s3 sync' does the deletion during or before all files are transferred you might again have a small time window where repodata has not been updated but files have already been deleted.
To be on the safe side the repodata probably needs to be transferred before files are being deleted.
Deletion should probably only happen after doing the invalidate to be sure that no files are being referenced by the caches which are currently being deleted.
Adrian
On Thu, Mar 05, 2020 at 01:05:20PM -0800, Kevin Fenzi wrote:
First, I noticed we are running the full sync twice right now, at the same time:
[root@mm-backend01 cron.d][PROD]# cat /etc/cron.d/s3.sh #Ansible: s3sync 0 0,11 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2> &1 #Ansible: s3sync-main 0 0 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync-main /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2>&1
Second, the attached patch changes the sync scripts to:
- do one sync with no --delete and excluding repodata
- do another one with --delete and including repodata
- invalidate the repodata
I adjusted the cron jobs to handle the repodata invalidate (I think).
TODO: only sync when things have changed.
+1s?
kevin
From 72a86a9f4d80344b96f8752ead2cc2877de292ef Mon Sep 17 00:00:00 2001 From: Kevin Fenzi kevin@scrye.com Date: Thu, 5 Mar 2020 20:46:53 +0000 Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.
In order to make sure the s3 mirror always is consistent, split out the commands to make it sync without repodata and delete, then do another run with those, then finally invalidate all the repodata/* files.
Some of the cron jobs are adjusted to allow the repodata invalidation.
Signed-off-by: Kevin Fenzi kevin@scrye.com
roles/s3-mirror/files/s3-sync-path.sh | 38 ++++++++---- roles/s3-mirror/files/s3.sh | 110 +++++++++++++++++++++++++++++++++- roles/s3-mirror/tasks/main.yml | 4 +- 3 files changed, 136 insertions(+), 16 deletions(-)
diff --git a/roles/s3-mirror/files/s3-sync-path.sh b/roles/s3-mirror/files/s3-sync-path.sh index e6ac994..40a0f90 100644 --- a/roles/s3-mirror/files/s3-sync-path.sh +++ b/roles/s3-mirror/files/s3-sync-path.sh @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */ ]]; then exit 1 fi
-CMD="aws s3 sync \ +# first run do not delete anything or copy the repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude *.snapshot/* \
- --exclude *source/* \
- --exclude *SRPMS/* \
- --exclude *debug/* \
- --exclude *beta/* \
- --exclude *ppc/* \
- --exclude *ppc64/* \
- --exclude *repoview/* \
- --exclude *Fedora/* \
- --exclude *EFI/* \
- --exclude *core/* \
- --exclude *extras/* \
- --exclude *LiveOS/* \
- --exclude *development/rawhide/* \
- --no-follow-symlinks \
- --only-show-errors \
- "
- #--dryrun \
+# second we delete old content and also copy the repodata +CMD2="aws s3 sync \ --delete \ --exclude *.snapshot/* \ --exclude *source/* \ @@ -32,19 +55,12 @@ CMD="aws s3 sync \
#echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1" echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
# Always do the invalidations because they are quick and prevent issues # depending on which path is synced. -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do +for file in $(echo $1/repodata/* ); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null done diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh index 55c1940..e36744e 100644 --- a/roles/s3-mirror/files/s3.sh +++ b/roles/s3-mirror/files/s3.sh @@ -3,7 +3,96 @@ # LGPL # Author: Rick Elrod relrod@redhat.com
-CMD="aws s3 sync \ +# first run this command that syncs, but does not delete. +# It also excludes repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude */.snapshot/* \
- --exclude */source/* \
- --exclude */SRPMS/* \
- --exclude */debug/* \
- --exclude */beta/* \
- --exclude */ppc/* \
- --exclude */ppc64/* \
- --exclude */repoview/* \
- --exclude */Fedora/* \
- --exclude */EFI/* \
- --exclude */core/* \
- --exclude */extras/* \
- --exclude */LiveOS/* \
- --exclude */development/rawhide/* \
- --exclude */releases/8/* \
- --exclude */releases/9/* \
- --exclude */releases/10/* \
- --exclude */releases/11/* \
- --exclude */releases/12/* \
- --exclude */releases/13/* \
- --exclude */releases/14/* \
- --exclude */releases/15/* \
- --exclude */releases/16/* \
- --exclude */releases/17/* \
- --exclude */releases/18/* \
- --exclude */releases/19/* \
- --exclude */releases/20/* \
- --exclude */releases/21/* \
- --exclude */releases/22/* \
- --exclude */releases/23/* \
- --exclude */releases/24/* \
- --exclude */releases/25/* \
- --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \
- --exclude */updates/8/* \
- --exclude */updates/9/* \
- --exclude */updates/10/* \
- --exclude */updates/11/* \
- --exclude */updates/12/* \
- --exclude */updates/13/* \
- --exclude */updates/14/* \
- --exclude */updates/15/* \
- --exclude */updates/16/* \
- --exclude */updates/17/* \
- --exclude */updates/18/* \
- --exclude */updates/19/* \
- --exclude */updates/20/* \
- --exclude */updates/21/* \
- --exclude */updates/22/* \
- --exclude */updates/23/* \
- --exclude */updates/24/* \
- --exclude */updates/25/* \
- --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \
- --exclude */updates/testing/8/* \
- --exclude */updates/testing/9/* \
- --exclude */updates/testing/10/* \
- --exclude */updates/testing/11/* \
- --exclude */updates/testing/12/* \
- --exclude */updates/testing/13/* \
- --exclude */updates/testing/14/* \
- --exclude */updates/testing/15/* \
- --exclude */updates/testing/16/* \
- --exclude */updates/testing/17/* \
- --exclude */updates/testing/18/* \
- --exclude */updates/testing/19/* \
- --exclude */updates/testing/20/* \
- --exclude */updates/testing/21/* \
- --exclude */updates/testing/22/* \
- --exclude */updates/testing/23/* \
- --exclude */updates/testing/24/* \
- --exclude */updates/testing/25/* \
- --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \
- --no-follow-symlinks \
- "
- #--dryrun \
+# Next we run this command which deletes old content and also includes repodata. +CMD2="aws s3 sync \ --delete \ --exclude */.snapshot/* \ --exclude */source/* \ @@ -38,6 +127,9 @@ CMD="aws s3 sync \ --exclude */releases/24/* \ --exclude */releases/25/* \ --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \ --exclude */updates/8/* \ --exclude */updates/9/* \ --exclude */updates/10/* \
@@ -57,6 +149,9 @@ CMD="aws s3 sync \ --exclude */updates/24/* \ --exclude */updates/25/* \ --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \ --exclude */updates/testing/8/* \ --exclude */updates/testing/9/* \ --exclude */updates/testing/10/* \
@@ -76,6 +171,9 @@ CMD="aws s3 sync \ --exclude */updates/testing/24/* \ --exclude */updates/testing/25/* \ --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \ --no-follow-symlinks \ " #--dryrun \
@@ -83,7 +181,8 @@ CMD="aws s3 sync \ # Sync EPEL #echo $CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD1 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD2 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do @@ -94,10 +193,15 @@ for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file"
+done
# Sync Fedora #echo $CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD1 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD2 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do diff --git a/roles/s3-mirror/tasks/main.yml b/roles/s3-mirror/tasks/main.yml index 12351cb..5da7a02 100644 --- a/roles/s3-mirror/tasks/main.yml +++ b/roles/s3-mirror/tasks/main.yml @@ -69,7 +69,7 @@
- name: s3sync cron - updates for current cron: name="s3sync-updates-current" minute="0" hour="3,9,15,21" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/Everything/x86_64/os" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1' cron_file=s3-updates-current.sh
@@ -95,7 +95,7 @@
- name: s3sync cron - updates for current-1 cron: name="s3sync-updates-previous" minute="30" hour="0,6,12,18" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/Everything/x86_64/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1' cron_file=s3-updates-previous.sh
-- 1.8.3.1
On Thu, Mar 05, 2020 at 10:31:22PM +0100, Adrian Reber wrote:
The only thing which is not clear in the documentation is when the delete is happening. With rsync there is an option '--delete-after' to make sure first all files are transferred and deletion happens afterwards. If 'aws s3 sync' does the deletion during or before all files are transferred you might again have a small time window where repodata has not been updated but files have already been deleted.
Yeah, it's not well documented, but it sounds like it does delete first. :(
https://github.com/aws/aws-cli/issues/1417
To be on the safe side the repodata probably needs to be transferred before files are being deleted.
ok. Let me add in a repodata sync step
Deletion should probably only happen after doing the invalidate to be sure that no files are being referenced by the caches which are currently being deleted.
sure.
kevin --
Adrian
On Thu, Mar 05, 2020 at 01:05:20PM -0800, Kevin Fenzi wrote:
First, I noticed we are running the full sync twice right now, at the same time:
[root@mm-backend01 cron.d][PROD]# cat /etc/cron.d/s3.sh #Ansible: s3sync 0 0,11 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2> &1 #Ansible: s3sync-main 0 0 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync-main /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2>&1
Second, the attached patch changes the sync scripts to:
- do one sync with no --delete and excluding repodata
- do another one with --delete and including repodata
- invalidate the repodata
I adjusted the cron jobs to handle the repodata invalidate (I think).
TODO: only sync when things have changed.
+1s?
kevin
From 72a86a9f4d80344b96f8752ead2cc2877de292ef Mon Sep 17 00:00:00 2001 From: Kevin Fenzi kevin@scrye.com Date: Thu, 5 Mar 2020 20:46:53 +0000 Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.
In order to make sure the s3 mirror always is consistent, split out the commands to make it sync without repodata and delete, then do another run with those, then finally invalidate all the repodata/* files.
Some of the cron jobs are adjusted to allow the repodata invalidation.
Signed-off-by: Kevin Fenzi kevin@scrye.com
roles/s3-mirror/files/s3-sync-path.sh | 38 ++++++++---- roles/s3-mirror/files/s3.sh | 110 +++++++++++++++++++++++++++++++++- roles/s3-mirror/tasks/main.yml | 4 +- 3 files changed, 136 insertions(+), 16 deletions(-)
diff --git a/roles/s3-mirror/files/s3-sync-path.sh b/roles/s3-mirror/files/s3-sync-path.sh index e6ac994..40a0f90 100644 --- a/roles/s3-mirror/files/s3-sync-path.sh +++ b/roles/s3-mirror/files/s3-sync-path.sh @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */ ]]; then exit 1 fi
-CMD="aws s3 sync \ +# first run do not delete anything or copy the repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude *.snapshot/* \
- --exclude *source/* \
- --exclude *SRPMS/* \
- --exclude *debug/* \
- --exclude *beta/* \
- --exclude *ppc/* \
- --exclude *ppc64/* \
- --exclude *repoview/* \
- --exclude *Fedora/* \
- --exclude *EFI/* \
- --exclude *core/* \
- --exclude *extras/* \
- --exclude *LiveOS/* \
- --exclude *development/rawhide/* \
- --no-follow-symlinks \
- --only-show-errors \
- "
- #--dryrun \
+# second we delete old content and also copy the repodata +CMD2="aws s3 sync \ --delete \ --exclude *.snapshot/* \ --exclude *source/* \ @@ -32,19 +55,12 @@ CMD="aws s3 sync \
#echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1" echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
# Always do the invalidations because they are quick and prevent issues # depending on which path is synced. -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do +for file in $(echo $1/repodata/* ); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null done diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh index 55c1940..e36744e 100644 --- a/roles/s3-mirror/files/s3.sh +++ b/roles/s3-mirror/files/s3.sh @@ -3,7 +3,96 @@ # LGPL # Author: Rick Elrod relrod@redhat.com
-CMD="aws s3 sync \ +# first run this command that syncs, but does not delete. +# It also excludes repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude */.snapshot/* \
- --exclude */source/* \
- --exclude */SRPMS/* \
- --exclude */debug/* \
- --exclude */beta/* \
- --exclude */ppc/* \
- --exclude */ppc64/* \
- --exclude */repoview/* \
- --exclude */Fedora/* \
- --exclude */EFI/* \
- --exclude */core/* \
- --exclude */extras/* \
- --exclude */LiveOS/* \
- --exclude */development/rawhide/* \
- --exclude */releases/8/* \
- --exclude */releases/9/* \
- --exclude */releases/10/* \
- --exclude */releases/11/* \
- --exclude */releases/12/* \
- --exclude */releases/13/* \
- --exclude */releases/14/* \
- --exclude */releases/15/* \
- --exclude */releases/16/* \
- --exclude */releases/17/* \
- --exclude */releases/18/* \
- --exclude */releases/19/* \
- --exclude */releases/20/* \
- --exclude */releases/21/* \
- --exclude */releases/22/* \
- --exclude */releases/23/* \
- --exclude */releases/24/* \
- --exclude */releases/25/* \
- --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \
- --exclude */updates/8/* \
- --exclude */updates/9/* \
- --exclude */updates/10/* \
- --exclude */updates/11/* \
- --exclude */updates/12/* \
- --exclude */updates/13/* \
- --exclude */updates/14/* \
- --exclude */updates/15/* \
- --exclude */updates/16/* \
- --exclude */updates/17/* \
- --exclude */updates/18/* \
- --exclude */updates/19/* \
- --exclude */updates/20/* \
- --exclude */updates/21/* \
- --exclude */updates/22/* \
- --exclude */updates/23/* \
- --exclude */updates/24/* \
- --exclude */updates/25/* \
- --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \
- --exclude */updates/testing/8/* \
- --exclude */updates/testing/9/* \
- --exclude */updates/testing/10/* \
- --exclude */updates/testing/11/* \
- --exclude */updates/testing/12/* \
- --exclude */updates/testing/13/* \
- --exclude */updates/testing/14/* \
- --exclude */updates/testing/15/* \
- --exclude */updates/testing/16/* \
- --exclude */updates/testing/17/* \
- --exclude */updates/testing/18/* \
- --exclude */updates/testing/19/* \
- --exclude */updates/testing/20/* \
- --exclude */updates/testing/21/* \
- --exclude */updates/testing/22/* \
- --exclude */updates/testing/23/* \
- --exclude */updates/testing/24/* \
- --exclude */updates/testing/25/* \
- --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \
- --no-follow-symlinks \
- "
- #--dryrun \
+# Next we run this command which deletes old content and also includes repodata. +CMD2="aws s3 sync \ --delete \ --exclude */.snapshot/* \ --exclude */source/* \ @@ -38,6 +127,9 @@ CMD="aws s3 sync \ --exclude */releases/24/* \ --exclude */releases/25/* \ --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \ --exclude */updates/8/* \ --exclude */updates/9/* \ --exclude */updates/10/* \
@@ -57,6 +149,9 @@ CMD="aws s3 sync \ --exclude */updates/24/* \ --exclude */updates/25/* \ --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \ --exclude */updates/testing/8/* \ --exclude */updates/testing/9/* \ --exclude */updates/testing/10/* \
@@ -76,6 +171,9 @@ CMD="aws s3 sync \ --exclude */updates/testing/24/* \ --exclude */updates/testing/25/* \ --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \ --no-follow-symlinks \ " #--dryrun \
@@ -83,7 +181,8 @@ CMD="aws s3 sync \ # Sync EPEL #echo $CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD1 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD2 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do @@ -94,10 +193,15 @@ for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file"
+done
# Sync Fedora #echo $CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD1 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD2 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do diff --git a/roles/s3-mirror/tasks/main.yml b/roles/s3-mirror/tasks/main.yml index 12351cb..5da7a02 100644 --- a/roles/s3-mirror/tasks/main.yml +++ b/roles/s3-mirror/tasks/main.yml @@ -69,7 +69,7 @@
- name: s3sync cron - updates for current cron: name="s3sync-updates-current" minute="0" hour="3,9,15,21" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/Everything/x86_64/os" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1' cron_file=s3-updates-current.sh
@@ -95,7 +95,7 @@
- name: s3sync cron - updates for current-1 cron: name="s3sync-updates-previous" minute="30" hour="0,6,12,18" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/Everything/x86_64/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1' cron_file=s3-updates-previous.sh
-- 1.8.3.1
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Thu, Mar 05, 2020 at 02:39:36PM -0800, Kevin Fenzi wrote:
-CMD="aws s3 sync \ +# first run do not delete anything or copy the repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude *.snapshot/* \
- --exclude *source/* \
- --exclude *SRPMS/* \
- --exclude *debug/* \
- --exclude *beta/* \
- --exclude *ppc/* \
- --exclude *ppc64/* \
- --exclude *repoview/* \
- --exclude *Fedora/* \
- --exclude *EFI/* \
- --exclude *core/* \
- --exclude *extras/* \
- --exclude *LiveOS/* \
- --exclude *development/rawhide/* \
- --no-follow-symlinks \
- --only-show-errors \
- "
- #--dryrun \
+# second we delete old content and also copy the repodata +CMD2="aws s3 sync \ --delete \ --exclude *.snapshot/* \ --exclude *source/* \ @@ -32,19 +55,12 @@ CMD="aws s3 sync \
#echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1" echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 -echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
+$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD1 /srv$1/repodata/ s3://s3-mirror-us-west-1-02.fedoraproject.org$1/repodata/ # Always do the invalidations because they are quick and prevent issues # depending on which path is synced. -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do +for file in $(echo $1/repodata/* ); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null done +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh index 55c1940..97abc59 100644 --- a/roles/s3-mirror/files/s3.sh +++ b/roles/s3-mirror/files/s3.sh @@ -3,8 +3,10 @@ # LGPL # Author: Rick Elrod relrod@redhat.com
-CMD="aws s3 sync \
- --delete \
+# first run this command that syncs, but does not delete. +# It also excludes repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \ --exclude */.snapshot/* \ --exclude */source/* \ --exclude */SRPMS/* \
@@ -38,6 +40,9 @@ CMD="aws s3 sync \ --exclude */releases/24/* \ --exclude */releases/25/* \ --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \ --exclude */updates/8/* \ --exclude */updates/9/* \ --exclude */updates/10/* \
@@ -57,6 +62,9 @@ CMD="aws s3 sync \ --exclude */updates/24/* \ --exclude */updates/25/* \ --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \ --exclude */updates/testing/8/* \ --exclude */updates/testing/9/* \ --exclude */updates/testing/10/* \
@@ -76,6 +84,95 @@ CMD="aws s3 sync \ --exclude */updates/testing/24/* \ --exclude */updates/testing/25/* \ --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \
- --no-follow-symlinks \
- "
- #--dryrun \
+# Next we run this command which also includes repodata. +CMD2="aws s3 sync \
- --exclude */.snapshot/* \
- --exclude */source/* \
- --exclude */SRPMS/* \
- --exclude */debug/* \
- --exclude */beta/* \
- --exclude */ppc/* \
- --exclude */ppc64/* \
- --exclude */repoview/* \
- --exclude */Fedora/* \
- --exclude */EFI/* \
- --exclude */core/* \
- --exclude */extras/* \
- --exclude */LiveOS/* \
- --exclude */development/rawhide/* \
- --exclude */releases/8/* \
- --exclude */releases/9/* \
- --exclude */releases/10/* \
- --exclude */releases/11/* \
- --exclude */releases/12/* \
- --exclude */releases/13/* \
- --exclude */releases/14/* \
- --exclude */releases/15/* \
- --exclude */releases/16/* \
- --exclude */releases/17/* \
- --exclude */releases/18/* \
- --exclude */releases/19/* \
- --exclude */releases/20/* \
- --exclude */releases/21/* \
- --exclude */releases/22/* \
- --exclude */releases/23/* \
- --exclude */releases/24/* \
- --exclude */releases/25/* \
- --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \
- --exclude */updates/8/* \
- --exclude */updates/9/* \
- --exclude */updates/10/* \
- --exclude */updates/11/* \
- --exclude */updates/12/* \
- --exclude */updates/13/* \
- --exclude */updates/14/* \
- --exclude */updates/15/* \
- --exclude */updates/16/* \
- --exclude */updates/17/* \
- --exclude */updates/18/* \
- --exclude */updates/19/* \
- --exclude */updates/20/* \
- --exclude */updates/21/* \
- --exclude */updates/22/* \
- --exclude */updates/23/* \
- --exclude */updates/24/* \
- --exclude */updates/25/* \
- --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \
- --exclude */updates/testing/8/* \
- --exclude */updates/testing/9/* \
- --exclude */updates/testing/10/* \
- --exclude */updates/testing/11/* \
- --exclude */updates/testing/12/* \
- --exclude */updates/testing/13/* \
- --exclude */updates/testing/14/* \
- --exclude */updates/testing/15/* \
- --exclude */updates/testing/16/* \
- --exclude */updates/testing/17/* \
- --exclude */updates/testing/18/* \
- --exclude */updates/testing/19/* \
- --exclude */updates/testing/20/* \
- --exclude */updates/testing/21/* \
- --exclude */updates/testing/22/* \
- --exclude */updates/testing/23/* \
- --exclude */updates/testing/24/* \
- --exclude */updates/testing/25/* \
- --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \ --no-follow-symlinks \ " #--dryrun \
@@ -83,7 +180,9 @@ CMD="aws s3 sync \ # Sync EPEL #echo $CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD1 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD2 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD2 --delete /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
You are deleting files here in a third step which looks like the correct order to sync it.
echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do @@ -94,10 +193,16 @@ for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file"
+done
But the invalidation happens here, right? So between delete and invalidate there is a short time where the cache might point to files which have been deleted a bit earlier. For me it seems that '$CMD2 --delete ...' should be after invalidation. Same for the Fedora sync a bit later.
In the s3-sync-path.sh script the delete is done after the invalidation.
Any reason to handle it differently in these two scripts?
# Sync Fedora #echo $CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD1 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD2 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD2 --delete /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do diff --git a/roles/s3-mirror/tasks/main.yml b/roles/s3-mirror/tasks/main.yml index 12351cb..5da7a02 100644 --- a/roles/s3-mirror/tasks/main.yml +++ b/roles/s3-mirror/tasks/main.yml @@ -69,7 +69,7 @@
- name: s3sync cron - updates for current cron: name="s3sync-updates-current" minute="0" hour="3,9,15,21" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/Everything/x86_64/os" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1' cron_file=s3-updates-current.sh
@@ -95,7 +95,7 @@
- name: s3sync cron - updates for current-1 cron: name="s3sync-updates-previous" minute="30" hour="0,6,12,18" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/Everything/x86_64/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1' cron_file=s3-updates-previous.sh
-- 1.8.3.1
Adrian
On Fri, Mar 06, 2020 at 07:53:05AM +0100, Adrian Reber wrote: ...snip...
You are deleting files here in a third step which looks like the correct order to sync it.
echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do @@ -94,10 +193,16 @@ for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file"
+done
But the invalidation happens here, right? So between delete and invalidate there is a short time where the cache might point to files which have been deleted a bit earlier. For me it seems that '$CMD2 --delete ...' should be after invalidation. Same for the Fedora sync a bit later.
In the s3-sync-path.sh script the delete is done after the invalidation.
Any reason to handle it differently in these two scripts?
Nope. :( Let me do another commit here. ;)
kevin
Looks correct from my point of view, for both scripts we are now doing:
1. sync everything besides repodata 2. sync everything including (or only) repodata 3. invalidate repodata 4. delete files
The only problem I can see now is that the files on the master are changing between those steps. Not sure how often the master is updated, but probably not more often than once or twice a day, right?
In theory this should help with the problems we have been seeing in COPR.
+1
Adrian
On Fri, Mar 06, 2020 at 02:04:21PM -0800, Kevin Fenzi wrote:
Final version? You be the judge. :)
kevin
From bd3c100e3a5fdd453ebbd4b88cc3bba365d260c3 Mon Sep 17 00:00:00 2001 From: Kevin Fenzi kevin@scrye.com Date: Thu, 5 Mar 2020 20:46:53 +0000 Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.
In order to make sure the s3 mirror always is consistent, split out the commands to make it sync without repodata and delete, then do another run with those, then finally invalidate all the repodata/* files.
Some of the cron jobs are adjusted to allow the repodata invalidation.
Signed-off-by: Kevin Fenzi kevin@scrye.com
roles/s3-mirror/files/s3-sync-path.sh | 42 +++++++++---- roles/s3-mirror/files/s3.sh | 114 ++++++++++++++++++++++++++++++++-- roles/s3-mirror/tasks/main.yml | 4 +- 3 files changed, 140 insertions(+), 20 deletions(-)
diff --git a/roles/s3-mirror/files/s3-sync-path.sh b/roles/s3-mirror/files/s3-sync-path.sh index e6ac994..79b4d63 100644 --- a/roles/s3-mirror/files/s3-sync-path.sh +++ b/roles/s3-mirror/files/s3-sync-path.sh @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */ ]]; then exit 1 fi
-CMD="aws s3 sync \ +# first run do not delete anything or copy the repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude *.snapshot/* \
- --exclude *source/* \
- --exclude *SRPMS/* \
- --exclude *debug/* \
- --exclude *beta/* \
- --exclude *ppc/* \
- --exclude *ppc64/* \
- --exclude *repoview/* \
- --exclude *Fedora/* \
- --exclude *EFI/* \
- --exclude *core/* \
- --exclude *extras/* \
- --exclude *LiveOS/* \
- --exclude *development/rawhide/* \
- --no-follow-symlinks \
- --only-show-errors \
- "
- #--dryrun \
+# second we delete old content and also copy the repodata +CMD2="aws s3 sync \ --delete \ --exclude *.snapshot/* \ --exclude *source/* \ @@ -32,19 +55,12 @@ CMD="aws s3 sync \
#echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1" echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 -echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
+$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD1 /srv$1/repodata/ s3://s3-mirror-us-west-1-02.fedoraproject.org$1/repodata/ # Always do the invalidations because they are quick and prevent issues # depending on which path is synced. -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do +for file in $(echo $1/repodata/* ); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null done +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh index 55c1940..c70defb 100644 --- a/roles/s3-mirror/files/s3.sh +++ b/roles/s3-mirror/files/s3.sh @@ -3,8 +3,10 @@ # LGPL # Author: Rick Elrod relrod@redhat.com
-CMD="aws s3 sync \
- --delete \
+# first run this command that syncs, but does not delete. +# It also excludes repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \ --exclude */.snapshot/* \ --exclude */source/* \ --exclude */SRPMS/* \
@@ -38,6 +40,9 @@ CMD="aws s3 sync \ --exclude */releases/24/* \ --exclude */releases/25/* \ --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \ --exclude */updates/8/* \ --exclude */updates/9/* \ --exclude */updates/10/* \
@@ -57,6 +62,9 @@ CMD="aws s3 sync \ --exclude */updates/24/* \ --exclude */updates/25/* \ --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \ --exclude */updates/testing/8/* \ --exclude */updates/testing/9/* \ --exclude */updates/testing/10/* \
@@ -76,6 +84,95 @@ CMD="aws s3 sync \ --exclude */updates/testing/24/* \ --exclude */updates/testing/25/* \ --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \
- --no-follow-symlinks \
- "
- #--dryrun \
+# Next we run this command which also includes repodata. +CMD2="aws s3 sync \
- --exclude */.snapshot/* \
- --exclude */source/* \
- --exclude */SRPMS/* \
- --exclude */debug/* \
- --exclude */beta/* \
- --exclude */ppc/* \
- --exclude */ppc64/* \
- --exclude */repoview/* \
- --exclude */Fedora/* \
- --exclude */EFI/* \
- --exclude */core/* \
- --exclude */extras/* \
- --exclude */LiveOS/* \
- --exclude */development/rawhide/* \
- --exclude */releases/8/* \
- --exclude */releases/9/* \
- --exclude */releases/10/* \
- --exclude */releases/11/* \
- --exclude */releases/12/* \
- --exclude */releases/13/* \
- --exclude */releases/14/* \
- --exclude */releases/15/* \
- --exclude */releases/16/* \
- --exclude */releases/17/* \
- --exclude */releases/18/* \
- --exclude */releases/19/* \
- --exclude */releases/20/* \
- --exclude */releases/21/* \
- --exclude */releases/22/* \
- --exclude */releases/23/* \
- --exclude */releases/24/* \
- --exclude */releases/25/* \
- --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \
- --exclude */updates/8/* \
- --exclude */updates/9/* \
- --exclude */updates/10/* \
- --exclude */updates/11/* \
- --exclude */updates/12/* \
- --exclude */updates/13/* \
- --exclude */updates/14/* \
- --exclude */updates/15/* \
- --exclude */updates/16/* \
- --exclude */updates/17/* \
- --exclude */updates/18/* \
- --exclude */updates/19/* \
- --exclude */updates/20/* \
- --exclude */updates/21/* \
- --exclude */updates/22/* \
- --exclude */updates/23/* \
- --exclude */updates/24/* \
- --exclude */updates/25/* \
- --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \
- --exclude */updates/testing/8/* \
- --exclude */updates/testing/9/* \
- --exclude */updates/testing/10/* \
- --exclude */updates/testing/11/* \
- --exclude */updates/testing/12/* \
- --exclude */updates/testing/13/* \
- --exclude */updates/testing/14/* \
- --exclude */updates/testing/15/* \
- --exclude */updates/testing/16/* \
- --exclude */updates/testing/17/* \
- --exclude */updates/testing/18/* \
- --exclude */updates/testing/19/* \
- --exclude */updates/testing/20/* \
- --exclude */updates/testing/21/* \
- --exclude */updates/testing/22/* \
- --exclude */updates/testing/23/* \
- --exclude */updates/testing/24/* \
- --exclude */updates/testing/25/* \
- --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \ --no-follow-symlinks \ " #--dryrun \
@@ -83,9 +180,9 @@ CMD="aws s3 sync \ # Sync EPEL #echo $CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD1 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD2 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done @@ -94,12 +191,19 @@ for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file"
+done +$CMD2 --delete /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
# Sync Fedora #echo $CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD1 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD2 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done +$CMD2 --delete /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ diff --git a/roles/s3-mirror/tasks/main.yml b/roles/s3-mirror/tasks/main.yml index 12351cb..5da7a02 100644 --- a/roles/s3-mirror/tasks/main.yml +++ b/roles/s3-mirror/tasks/main.yml @@ -69,7 +69,7 @@
- name: s3sync cron - updates for current cron: name="s3sync-updates-current" minute="0" hour="3,9,15,21" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/Everything/x86_64/os" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1' cron_file=s3-updates-current.sh
@@ -95,7 +95,7 @@
- name: s3sync cron - updates for current-1 cron: name="s3sync-updates-previous" minute="30" hour="0,6,12,18" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/Everything/x86_64/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1' cron_file=s3-updates-previous.sh
-- 1.8.3.1
On Sat, Mar 07, 2020 at 11:16:48AM +0100, Adrian Reber wrote:
Looks correct from my point of view, for both scripts we are now doing:
- sync everything besides repodata
- sync everything including (or only) repodata
- invalidate repodata
- delete files
The only problem I can see now is that the files on the master are changing between those steps. Not sure how often the master is updated, but probably not more often than once or twice a day, right?
Yeah, it's once a day for most everything.
We definitely could be smarter on times, but for now this should be fine.
In theory this should help with the problems we have been seeing in COPR.
Yeah. I hope so.
kevin --
+1
Adrian
On Fri, Mar 06, 2020 at 02:04:21PM -0800, Kevin Fenzi wrote:
Final version? You be the judge. :)
kevin
From bd3c100e3a5fdd453ebbd4b88cc3bba365d260c3 Mon Sep 17 00:00:00 2001 From: Kevin Fenzi kevin@scrye.com Date: Thu, 5 Mar 2020 20:46:53 +0000 Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.
In order to make sure the s3 mirror always is consistent, split out the commands to make it sync without repodata and delete, then do another run with those, then finally invalidate all the repodata/* files.
Some of the cron jobs are adjusted to allow the repodata invalidation.
Signed-off-by: Kevin Fenzi kevin@scrye.com
roles/s3-mirror/files/s3-sync-path.sh | 42 +++++++++---- roles/s3-mirror/files/s3.sh | 114 ++++++++++++++++++++++++++++++++-- roles/s3-mirror/tasks/main.yml | 4 +- 3 files changed, 140 insertions(+), 20 deletions(-)
diff --git a/roles/s3-mirror/files/s3-sync-path.sh b/roles/s3-mirror/files/s3-sync-path.sh index e6ac994..79b4d63 100644 --- a/roles/s3-mirror/files/s3-sync-path.sh +++ b/roles/s3-mirror/files/s3-sync-path.sh @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */ ]]; then exit 1 fi
-CMD="aws s3 sync \ +# first run do not delete anything or copy the repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude *.snapshot/* \
- --exclude *source/* \
- --exclude *SRPMS/* \
- --exclude *debug/* \
- --exclude *beta/* \
- --exclude *ppc/* \
- --exclude *ppc64/* \
- --exclude *repoview/* \
- --exclude *Fedora/* \
- --exclude *EFI/* \
- --exclude *core/* \
- --exclude *extras/* \
- --exclude *LiveOS/* \
- --exclude *development/rawhide/* \
- --no-follow-symlinks \
- --only-show-errors \
- "
- #--dryrun \
+# second we delete old content and also copy the repodata +CMD2="aws s3 sync \ --delete \ --exclude *.snapshot/* \ --exclude *source/* \ @@ -32,19 +55,12 @@ CMD="aws s3 sync \
#echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1" echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 -echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
+$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD1 /srv$1/repodata/ s3://s3-mirror-us-west-1-02.fedoraproject.org$1/repodata/ # Always do the invalidations because they are quick and prevent issues # depending on which path is synced. -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do +for file in $(echo $1/repodata/* ); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" > /dev/null done +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh index 55c1940..c70defb 100644 --- a/roles/s3-mirror/files/s3.sh +++ b/roles/s3-mirror/files/s3.sh @@ -3,8 +3,10 @@ # LGPL # Author: Rick Elrod relrod@redhat.com
-CMD="aws s3 sync \
- --delete \
+# first run this command that syncs, but does not delete. +# It also excludes repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \ --exclude */.snapshot/* \ --exclude */source/* \ --exclude */SRPMS/* \
@@ -38,6 +40,9 @@ CMD="aws s3 sync \ --exclude */releases/24/* \ --exclude */releases/25/* \ --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \ --exclude */updates/8/* \ --exclude */updates/9/* \ --exclude */updates/10/* \
@@ -57,6 +62,9 @@ CMD="aws s3 sync \ --exclude */updates/24/* \ --exclude */updates/25/* \ --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \ --exclude */updates/testing/8/* \ --exclude */updates/testing/9/* \ --exclude */updates/testing/10/* \
@@ -76,6 +84,95 @@ CMD="aws s3 sync \ --exclude */updates/testing/24/* \ --exclude */updates/testing/25/* \ --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \
- --no-follow-symlinks \
- "
- #--dryrun \
+# Next we run this command which also includes repodata. +CMD2="aws s3 sync \
- --exclude */.snapshot/* \
- --exclude */source/* \
- --exclude */SRPMS/* \
- --exclude */debug/* \
- --exclude */beta/* \
- --exclude */ppc/* \
- --exclude */ppc64/* \
- --exclude */repoview/* \
- --exclude */Fedora/* \
- --exclude */EFI/* \
- --exclude */core/* \
- --exclude */extras/* \
- --exclude */LiveOS/* \
- --exclude */development/rawhide/* \
- --exclude */releases/8/* \
- --exclude */releases/9/* \
- --exclude */releases/10/* \
- --exclude */releases/11/* \
- --exclude */releases/12/* \
- --exclude */releases/13/* \
- --exclude */releases/14/* \
- --exclude */releases/15/* \
- --exclude */releases/16/* \
- --exclude */releases/17/* \
- --exclude */releases/18/* \
- --exclude */releases/19/* \
- --exclude */releases/20/* \
- --exclude */releases/21/* \
- --exclude */releases/22/* \
- --exclude */releases/23/* \
- --exclude */releases/24/* \
- --exclude */releases/25/* \
- --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \
- --exclude */updates/8/* \
- --exclude */updates/9/* \
- --exclude */updates/10/* \
- --exclude */updates/11/* \
- --exclude */updates/12/* \
- --exclude */updates/13/* \
- --exclude */updates/14/* \
- --exclude */updates/15/* \
- --exclude */updates/16/* \
- --exclude */updates/17/* \
- --exclude */updates/18/* \
- --exclude */updates/19/* \
- --exclude */updates/20/* \
- --exclude */updates/21/* \
- --exclude */updates/22/* \
- --exclude */updates/23/* \
- --exclude */updates/24/* \
- --exclude */updates/25/* \
- --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \
- --exclude */updates/testing/8/* \
- --exclude */updates/testing/9/* \
- --exclude */updates/testing/10/* \
- --exclude */updates/testing/11/* \
- --exclude */updates/testing/12/* \
- --exclude */updates/testing/13/* \
- --exclude */updates/testing/14/* \
- --exclude */updates/testing/15/* \
- --exclude */updates/testing/16/* \
- --exclude */updates/testing/17/* \
- --exclude */updates/testing/18/* \
- --exclude */updates/testing/19/* \
- --exclude */updates/testing/20/* \
- --exclude */updates/testing/21/* \
- --exclude */updates/testing/22/* \
- --exclude */updates/testing/23/* \
- --exclude */updates/testing/24/* \
- --exclude */updates/testing/25/* \
- --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \ --no-follow-symlinks \ " #--dryrun \
@@ -83,9 +180,9 @@ CMD="aws s3 sync \ # Sync EPEL #echo $CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD1 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ +$CMD2 /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/ echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done @@ -94,12 +191,19 @@ for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed 's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file"
+done +$CMD2 --delete /srv/pub/epel/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
# Sync Fedora #echo $CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD1 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ +$CMD2 /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU --paths "$file" done +$CMD2 --delete /srv/pub/fedora/ s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/ diff --git a/roles/s3-mirror/tasks/main.yml b/roles/s3-mirror/tasks/main.yml index 12351cb..5da7a02 100644 --- a/roles/s3-mirror/tasks/main.yml +++ b/roles/s3-mirror/tasks/main.yml @@ -69,7 +69,7 @@
- name: s3sync cron - updates for current cron: name="s3sync-updates-current" minute="0" hour="3,9,15,21" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-current "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/Everything/x86_64/os" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1' cron_file=s3-updates-current.sh
@@ -95,7 +95,7 @@
- name: s3sync cron - updates for current-1 cron: name="s3sync-updates-previous" minute="30" hour="0,6,12,18" user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
when: env != 'staging' and inventory_hostname.startswith('mm-backend01.') tags:job='/usr/local/bin/lock-wrapper s3sync-updates-previous "/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/Everything/x86_64/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1' cron_file=s3-updates-previous.sh
-- 1.8.3.1
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
+1
On Sat, 7 Mar 2020 at 05:17, Adrian Reber adrian@lisas.de wrote:
Looks correct from my point of view, for both scripts we are now doing:
- sync everything besides repodata
- sync everything including (or only) repodata
- invalidate repodata
- delete files
The only problem I can see now is that the files on the master are changing between those steps. Not sure how often the master is updated, but probably not more often than once or twice a day, right?
In theory this should help with the problems we have been seeing in COPR.
+1
Adrian
On Fri, Mar 06, 2020 at 02:04:21PM -0800, Kevin Fenzi wrote:
Final version? You be the judge. :)
kevin
From bd3c100e3a5fdd453ebbd4b88cc3bba365d260c3 Mon Sep 17 00:00:00 2001 From: Kevin Fenzi kevin@scrye.com Date: Thu, 5 Mar 2020 20:46:53 +0000 Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.
In order to make sure the s3 mirror always is consistent, split out the
commands
to make it sync without repodata and delete, then do another run with
those, then finally
invalidate all the repodata/* files.
Some of the cron jobs are adjusted to allow the repodata invalidation.
Signed-off-by: Kevin Fenzi kevin@scrye.com
roles/s3-mirror/files/s3-sync-path.sh | 42 +++++++++---- roles/s3-mirror/files/s3.sh | 114
++++++++++++++++++++++++++++++++--
roles/s3-mirror/tasks/main.yml | 4 +- 3 files changed, 140 insertions(+), 20 deletions(-)
diff --git a/roles/s3-mirror/files/s3-sync-path.sh
b/roles/s3-mirror/files/s3-sync-path.sh
index e6ac994..79b4d63 100644 --- a/roles/s3-mirror/files/s3-sync-path.sh +++ b/roles/s3-mirror/files/s3-sync-path.sh @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */
]]; then
exit 1 fi
-CMD="aws s3 sync \ +# first run do not delete anything or copy the repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude *.snapshot/* \
- --exclude *source/* \
- --exclude *SRPMS/* \
- --exclude *debug/* \
- --exclude *beta/* \
- --exclude *ppc/* \
- --exclude *ppc64/* \
- --exclude *repoview/* \
- --exclude *Fedora/* \
- --exclude *EFI/* \
- --exclude *core/* \
- --exclude *extras/* \
- --exclude *LiveOS/* \
- --exclude *development/rawhide/* \
- --no-follow-symlinks \
- --only-show-errors \
- "
- #--dryrun \
+# second we delete old content and also copy the repodata +CMD2="aws s3 sync \ --delete \ --exclude *.snapshot/* \ --exclude *source/* \ @@ -32,19 +55,12 @@ CMD="aws s3 sync \
#echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1" echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 -echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
+$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD1 /srv$1/repodata/ s3://s3-mirror-us-west-1-02.fedoraproject.org
$1/repodata/
# Always do the invalidations because they are quick and prevent issues # depending on which path is synced. -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed
's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed
's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file" > /dev/null
-done
-for file in $(echo
/srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do
+for file in $(echo $1/repodata/* ); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file" > /dev/null
done +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh index 55c1940..c70defb 100644 --- a/roles/s3-mirror/files/s3.sh +++ b/roles/s3-mirror/files/s3.sh @@ -3,8 +3,10 @@ # LGPL # Author: Rick Elrod relrod@redhat.com
-CMD="aws s3 sync \
- --delete \
+# first run this command that syncs, but does not delete. +# It also excludes repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \ --exclude */.snapshot/* \ --exclude */source/* \ --exclude */SRPMS/* \
@@ -38,6 +40,9 @@ CMD="aws s3 sync \ --exclude */releases/24/* \ --exclude */releases/25/* \ --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \ --exclude */updates/8/* \ --exclude */updates/9/* \ --exclude */updates/10/* \
@@ -57,6 +62,9 @@ CMD="aws s3 sync \ --exclude */updates/24/* \ --exclude */updates/25/* \ --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \ --exclude */updates/testing/8/* \ --exclude */updates/testing/9/* \ --exclude */updates/testing/10/* \
@@ -76,6 +84,95 @@ CMD="aws s3 sync \ --exclude */updates/testing/24/* \ --exclude */updates/testing/25/* \ --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \
- --no-follow-symlinks \
- "
- #--dryrun \
+# Next we run this command which also includes repodata. +CMD2="aws s3 sync \
- --exclude */.snapshot/* \
- --exclude */source/* \
- --exclude */SRPMS/* \
- --exclude */debug/* \
- --exclude */beta/* \
- --exclude */ppc/* \
- --exclude */ppc64/* \
- --exclude */repoview/* \
- --exclude */Fedora/* \
- --exclude */EFI/* \
- --exclude */core/* \
- --exclude */extras/* \
- --exclude */LiveOS/* \
- --exclude */development/rawhide/* \
- --exclude */releases/8/* \
- --exclude */releases/9/* \
- --exclude */releases/10/* \
- --exclude */releases/11/* \
- --exclude */releases/12/* \
- --exclude */releases/13/* \
- --exclude */releases/14/* \
- --exclude */releases/15/* \
- --exclude */releases/16/* \
- --exclude */releases/17/* \
- --exclude */releases/18/* \
- --exclude */releases/19/* \
- --exclude */releases/20/* \
- --exclude */releases/21/* \
- --exclude */releases/22/* \
- --exclude */releases/23/* \
- --exclude */releases/24/* \
- --exclude */releases/25/* \
- --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \
- --exclude */updates/8/* \
- --exclude */updates/9/* \
- --exclude */updates/10/* \
- --exclude */updates/11/* \
- --exclude */updates/12/* \
- --exclude */updates/13/* \
- --exclude */updates/14/* \
- --exclude */updates/15/* \
- --exclude */updates/16/* \
- --exclude */updates/17/* \
- --exclude */updates/18/* \
- --exclude */updates/19/* \
- --exclude */updates/20/* \
- --exclude */updates/21/* \
- --exclude */updates/22/* \
- --exclude */updates/23/* \
- --exclude */updates/24/* \
- --exclude */updates/25/* \
- --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \
- --exclude */updates/testing/8/* \
- --exclude */updates/testing/9/* \
- --exclude */updates/testing/10/* \
- --exclude */updates/testing/11/* \
- --exclude */updates/testing/12/* \
- --exclude */updates/testing/13/* \
- --exclude */updates/testing/14/* \
- --exclude */updates/testing/15/* \
- --exclude */updates/testing/16/* \
- --exclude */updates/testing/17/* \
- --exclude */updates/testing/18/* \
- --exclude */updates/testing/19/* \
- --exclude */updates/testing/20/* \
- --exclude */updates/testing/21/* \
- --exclude */updates/testing/22/* \
- --exclude */updates/testing/23/* \
- --exclude */updates/testing/24/* \
- --exclude */updates/testing/25/* \
- --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \ --no-follow-symlinks \ " #--dryrun \
@@ -83,9 +180,9 @@ CMD="aws s3 sync \ # Sync EPEL #echo $CMD /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
+$CMD1 /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
+$CMD2 /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed
's#/srv##g'); do
aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
done @@ -94,12 +191,19 @@ for file in $(echo
/srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do
aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed
's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
+done +$CMD2 --delete /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
# Sync Fedora #echo $CMD /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
+$CMD1 /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
+$CMD2 /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo
/srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do
aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
done +$CMD2 --delete /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
diff --git a/roles/s3-mirror/tasks/main.yml
b/roles/s3-mirror/tasks/main.yml
index 12351cb..5da7a02 100644 --- a/roles/s3-mirror/tasks/main.yml +++ b/roles/s3-mirror/tasks/main.yml @@ -69,7 +69,7 @@
- name: s3sync cron - updates for current cron: name="s3sync-updates-current" minute="0" hour="3,9,15,21"
user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-current
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
job='/usr/local/bin/lock-wrapper s3sync-updates-current
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/Everything/x86_64/os" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
cron_file=s3-updates-current.sh
when: env != 'staging' and
inventory_hostname.startswith('mm-backend01.')
tags: @@ -95,7 +95,7 @@
- name: s3sync cron - updates for current-1 cron: name="s3sync-updates-previous" minute="30" hour="0,6,12,18"
user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-previous
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
job='/usr/local/bin/lock-wrapper s3sync-updates-previous
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/Everything/x86_64/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
cron_file=s3-updates-previous.sh
when: env != 'staging' and
inventory_hostname.startswith('mm-backend01.')
tags:
1.8.3.1
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
+1 too
On Mon, 9 Mar 2020 at 17:00, Stephen John Smoogen smooge@gmail.com wrote:
+1
On Sat, 7 Mar 2020 at 05:17, Adrian Reber adrian@lisas.de wrote:
Looks correct from my point of view, for both scripts we are now doing:
- sync everything besides repodata
- sync everything including (or only) repodata
- invalidate repodata
- delete files
The only problem I can see now is that the files on the master are changing between those steps. Not sure how often the master is updated, but probably not more often than once or twice a day, right?
In theory this should help with the problems we have been seeing in COPR.
+1
Adrian
On Fri, Mar 06, 2020 at 02:04:21PM -0800, Kevin Fenzi wrote:
Final version? You be the judge. :)
kevin
From bd3c100e3a5fdd453ebbd4b88cc3bba365d260c3 Mon Sep 17 00:00:00 2001 From: Kevin Fenzi kevin@scrye.com Date: Thu, 5 Mar 2020 20:46:53 +0000 Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without repodata and delete, the other with both.
In order to make sure the s3 mirror always is consistent, split out the
commands
to make it sync without repodata and delete, then do another run with
those, then finally
invalidate all the repodata/* files.
Some of the cron jobs are adjusted to allow the repodata invalidation.
Signed-off-by: Kevin Fenzi kevin@scrye.com
roles/s3-mirror/files/s3-sync-path.sh | 42 +++++++++---- roles/s3-mirror/files/s3.sh | 114
++++++++++++++++++++++++++++++++--
roles/s3-mirror/tasks/main.yml | 4 +- 3 files changed, 140 insertions(+), 20 deletions(-)
diff --git a/roles/s3-mirror/files/s3-sync-path.sh
b/roles/s3-mirror/files/s3-sync-path.sh
index e6ac994..79b4d63 100644 --- a/roles/s3-mirror/files/s3-sync-path.sh +++ b/roles/s3-mirror/files/s3-sync-path.sh @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 !=
*/ ]]; then
exit 1 fi
-CMD="aws s3 sync \ +# first run do not delete anything or copy the repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \
- --exclude *.snapshot/* \
- --exclude *source/* \
- --exclude *SRPMS/* \
- --exclude *debug/* \
- --exclude *beta/* \
- --exclude *ppc/* \
- --exclude *ppc64/* \
- --exclude *repoview/* \
- --exclude *Fedora/* \
- --exclude *EFI/* \
- --exclude *core/* \
- --exclude *extras/* \
- --exclude *LiveOS/* \
- --exclude *development/rawhide/* \
- --no-follow-symlinks \
- --only-show-errors \
- "
- #--dryrun \
+# second we delete old content and also copy the repodata +CMD2="aws s3 sync \ --delete \ --exclude *.snapshot/* \ --exclude *source/* \ @@ -32,19 +55,12 @@ CMD="aws s3 sync \
#echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1" echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 -echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
+$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +$CMD1 /srv$1/repodata/ s3://s3-mirror-us-west-1-02.fedoraproject.org
$1/repodata/
# Always do the invalidations because they are quick and prevent issues # depending on which path is synced. -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed
's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file" > /dev/null
-done
-for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed
's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file" > /dev/null
-done
-for file in $(echo
/srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do
+for file in $(echo $1/repodata/* ); do aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file" > /dev/null
done +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1 +echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh index 55c1940..c70defb 100644 --- a/roles/s3-mirror/files/s3.sh +++ b/roles/s3-mirror/files/s3.sh @@ -3,8 +3,10 @@ # LGPL # Author: Rick Elrod relrod@redhat.com
-CMD="aws s3 sync \
- --delete \
+# first run this command that syncs, but does not delete. +# It also excludes repodata. +CMD1="aws s3 sync \
- --exclude */repodata/* \ --exclude */.snapshot/* \ --exclude */source/* \ --exclude */SRPMS/* \
@@ -38,6 +40,9 @@ CMD="aws s3 sync \ --exclude */releases/24/* \ --exclude */releases/25/* \ --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \ --exclude */updates/8/* \ --exclude */updates/9/* \ --exclude */updates/10/* \
@@ -57,6 +62,9 @@ CMD="aws s3 sync \ --exclude */updates/24/* \ --exclude */updates/25/* \ --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \ --exclude */updates/testing/8/* \ --exclude */updates/testing/9/* \ --exclude */updates/testing/10/* \
@@ -76,6 +84,95 @@ CMD="aws s3 sync \ --exclude */updates/testing/24/* \ --exclude */updates/testing/25/* \ --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \
- --no-follow-symlinks \
- "
- #--dryrun \
+# Next we run this command which also includes repodata. +CMD2="aws s3 sync \
- --exclude */.snapshot/* \
- --exclude */source/* \
- --exclude */SRPMS/* \
- --exclude */debug/* \
- --exclude */beta/* \
- --exclude */ppc/* \
- --exclude */ppc64/* \
- --exclude */repoview/* \
- --exclude */Fedora/* \
- --exclude */EFI/* \
- --exclude */core/* \
- --exclude */extras/* \
- --exclude */LiveOS/* \
- --exclude */development/rawhide/* \
- --exclude */releases/8/* \
- --exclude */releases/9/* \
- --exclude */releases/10/* \
- --exclude */releases/11/* \
- --exclude */releases/12/* \
- --exclude */releases/13/* \
- --exclude */releases/14/* \
- --exclude */releases/15/* \
- --exclude */releases/16/* \
- --exclude */releases/17/* \
- --exclude */releases/18/* \
- --exclude */releases/19/* \
- --exclude */releases/20/* \
- --exclude */releases/21/* \
- --exclude */releases/22/* \
- --exclude */releases/23/* \
- --exclude */releases/24/* \
- --exclude */releases/25/* \
- --exclude */releases/26/* \
- --exclude */releases/27/* \
- --exclude */releases/28/* \
- --exclude */releases/29/* \
- --exclude */updates/8/* \
- --exclude */updates/9/* \
- --exclude */updates/10/* \
- --exclude */updates/11/* \
- --exclude */updates/12/* \
- --exclude */updates/13/* \
- --exclude */updates/14/* \
- --exclude */updates/15/* \
- --exclude */updates/16/* \
- --exclude */updates/17/* \
- --exclude */updates/18/* \
- --exclude */updates/19/* \
- --exclude */updates/20/* \
- --exclude */updates/21/* \
- --exclude */updates/22/* \
- --exclude */updates/23/* \
- --exclude */updates/24/* \
- --exclude */updates/25/* \
- --exclude */updates/26/* \
- --exclude */updates/27/* \
- --exclude */updates/28/* \
- --exclude */updates/29/* \
- --exclude */updates/testing/8/* \
- --exclude */updates/testing/9/* \
- --exclude */updates/testing/10/* \
- --exclude */updates/testing/11/* \
- --exclude */updates/testing/12/* \
- --exclude */updates/testing/13/* \
- --exclude */updates/testing/14/* \
- --exclude */updates/testing/15/* \
- --exclude */updates/testing/16/* \
- --exclude */updates/testing/17/* \
- --exclude */updates/testing/18/* \
- --exclude */updates/testing/19/* \
- --exclude */updates/testing/20/* \
- --exclude */updates/testing/21/* \
- --exclude */updates/testing/22/* \
- --exclude */updates/testing/23/* \
- --exclude */updates/testing/24/* \
- --exclude */updates/testing/25/* \
- --exclude */updates/testing/26/* \
- --exclude */updates/testing/27/* \
- --exclude */updates/testing/28/* \
- --exclude */updates/testing/29/* \ --no-follow-symlinks \ " #--dryrun \
@@ -83,9 +180,9 @@ CMD="aws s3 sync \ # Sync EPEL #echo $CMD /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
+$CMD1 /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
+$CMD2 /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed
's#/srv##g'); do
aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
done @@ -94,12 +191,19 @@ for file in $(echo
/srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); do
aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
done
+for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed
's#/srv##g'); do
- aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
+done +$CMD2 --delete /srv/pub/epel/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
# Sync Fedora #echo $CMD /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps -$CMD /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
+$CMD1 /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
+$CMD2 /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
for file in $(echo
/srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed 's#/srv##g'); do
aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU
--paths "$file"
done +$CMD2 --delete /srv/pub/fedora/ s3://
s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
diff --git a/roles/s3-mirror/tasks/main.yml
b/roles/s3-mirror/tasks/main.yml
index 12351cb..5da7a02 100644 --- a/roles/s3-mirror/tasks/main.yml +++ b/roles/s3-mirror/tasks/main.yml @@ -69,7 +69,7 @@
- name: s3sync cron - updates for current cron: name="s3sync-updates-current" minute="0" hour="3,9,15,21"
user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-current
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
job='/usr/local/bin/lock-wrapper s3sync-updates-current
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int }}/Everything/x86_64/os" 2>&1 | /usr/local/bin/nag-once s3-updates-current.sh 1d 2>&1'
cron_file=s3-updates-current.sh
when: env != 'staging' and
inventory_hostname.startswith('mm-backend01.')
tags: @@ -95,7 +95,7 @@
- name: s3sync cron - updates for current-1 cron: name="s3sync-updates-previous" minute="30" hour="0,6,12,18"
user="s3-mirror"
job='/usr/local/bin/lock-wrapper s3sync-updates-previous
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
job='/usr/local/bin/lock-wrapper s3sync-updates-previous
"/usr/local/bin/s3-sync-path.sh /pub/fedora/linux/updates/{{ FedoraCycleNumber|int - 1 }}/Everything/x86_64/" 2>&1 | /usr/local/bin/nag-once s3-updates-previous.sh 1d 2>&1'
cron_file=s3-updates-previous.sh
when: env != 'staging' and
inventory_hostname.startswith('mm-backend01.')
tags:
1.8.3.1
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
-- Stephen J Smoogen.
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure@lists.fedoraproject.org