Good Morning Everyone,
Kevin and I had a discussion on IRC the other day about mirroring the ansible repo. The options we considered are: - mirroring to pagure - mirroring from pagure - using pagure's mirroring feature - using a different approach
- Mirroring to pagure This meant, the canonical place for ansible remains in the batcave and pagure only periodically pulls from there to update itself (or a hook updates pagure on push). This would work, but it basically make the pull-request workflow for reviewers a little more complex, as you would have to download the patch of the PR, apply it (potentially adding the: Merging <id> so it marks the PR as merged) and then push to the batcave. If you merged via the UI in pagure, you would then have to pull from pagure and then push to the batcave, with the risk of a race condition if someone or something updates pagure right after your merge and suddenly the commit(s) of the PR disapear (since the content in pagure would be overriden by what is coming from the batcave).
So, not an ideal workflow.
- Mirroring from pagure This means, the canonical place for ansible is in pagure.io. However, we need to still be able to run ansible when pagure.io goes down (keeping in mind batcave and pagure.io are in two different data-center). So we want to keep a mirror of the ansible repo in the batcave for emergencies and that repo should be as up to date as possible.
With this approach, all the work happens on pagure.io, the PR workflow is the usual, straight forward one and we have a mirror in the batcave that is mostly up to date.
- Using pagure's mirroring feature Pagure can mirror in and out, out is done right after a push, however, batcave is not accessible from pagure.io, so we just can't use this.
- Using a different approach We have two ways we could do this: a simple cron job, a message-based service.
For a first step I went with a third approach: a small python service that runs every 3 minutes (configurable): git fetch && git fsck (to ensure the git is in a correct state). It actually doesn't run every 3 minutes, it calls git, then wait for 3 minutes then calls git again and so on. So if git was ever to takes 4 minutes to run, there is no risk of the program stepping on its own toes.
I would like to propose that install and configure this for a test. I propose that we keep /git/ansible.git as is and just have a /git/mirrors/ansible.git which will be the mirror from pagure. We can migrate the hook from /git/ansible.git to /git/mirrors/ansible.git so /srv/web/infra/ansible (ie: the exploded tree) is coming from the mirror.
I guess the lag time to get /git/mirrors updated will quickly be annoying (up to 3 minutes between when the PR is merged and when the playbook is updated), so if we like the workflow we will likely want to switch pretty quickly to a message-based service and then we could keep the current cron-like service to run hourly and update /git/ansible.git which would then become some kind of backup.
What do you think? FBR worthy? (ie: if F32 goes out next week, I think we can wait, if it doesn't do we want to try this sooner?)
If we like the idea, I'll start looking into the fedora-messaging based consumer, using the approach from the current service, it should be pretty straight forward (I'll re-use the same project for it).
Oh, and if you want to see the code of the current service: https://pagure.io/Fedora-Infra/mirror_from_pagure
Thanks, Pierre
On Thu, Apr 23, 2020 at 5:47 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
Kevin and I had a discussion on IRC the other day about mirroring the ansible repo. The options we considered are:
mirroring to pagure
mirroring from pagure
- using pagure's mirroring feature
- using a different approach
Mirroring to pagure This meant, the canonical place for ansible remains in the batcave and pagure only periodically pulls from there to update itself (or a hook updates pagure on push). This would work, but it basically make the pull-request workflow for reviewers a little more complex, as you would have to download the patch of the PR, apply it (potentially adding the: Merging <id> so it marks the PR as merged) and then push to the batcave. If you merged via the UI in pagure, you would then have to pull from pagure and then push to the batcave, with the risk of a race condition if someone or something updates pagure right after your merge and suddenly the commit(s) of the PR disapear (since the content in pagure would be overriden by what is coming from the batcave).
So, not an ideal workflow.
Mirroring from pagure This means, the canonical place for ansible is in pagure.io. However, we need to still be able to run ansible when pagure.io goes down (keeping in mind batcave and pagure.io are in two different data-center). So we want to keep a mirror of the ansible repo in the batcave for emergencies and that repo should be as up to date as possible.
With this approach, all the work happens on pagure.io, the PR workflow is the usual, straight forward one and we have a mirror in the batcave that is mostly up to date.
- Using pagure's mirroring feature
Pagure can mirror in and out, out is done right after a push, however, batcave is not accessible from pagure.io, so we just can't use this.
- Using a different approach
We have two ways we could do this: a simple cron job, a message-based service.
For a first step I went with a third approach: a small python service that runs every 3 minutes (configurable): git fetch && git fsck (to ensure the git is in a correct state). It actually doesn't run every 3 minutes, it calls git, then wait for 3 minutes then calls git again and so on. So if git was ever to takes 4 minutes to run, there is no risk of the program stepping on its own toes.
I would like to propose that install and configure this for a test. I propose that we keep /git/ansible.git as is and just have a /git/mirrors/ansible.git which will be the mirror from pagure. We can migrate the hook from /git/ansible.git to /git/mirrors/ansible.git so /srv/web/infra/ansible (ie: the exploded tree) is coming from the mirror.
I guess the lag time to get /git/mirrors updated will quickly be annoying (up to 3 minutes between when the PR is merged and when the playbook is updated), so if we like the workflow we will likely want to switch pretty quickly to a message-based service and then we could keep the current cron-like service to run hourly and update /git/ansible.git which would then become some kind of backup.
What do you think? FBR worthy? (ie: if F32 goes out next week, I think we can wait, if it doesn't do we want to try this sooner?)
If we like the idea, I'll start looking into the fedora-messaging based consumer, using the approach from the current service, it should be pretty straight forward (I'll re-use the same project for it).
Oh, and if you want to see the code of the current service: https://pagure.io/Fedora-Infra/mirror_from_pagure
I think this is FBR worthy. Heck, I think it's fine to do now. It's really not that invasive, and it makes contributions to the ansible repo a lot more straightforward. We don't _have_ to accept any PRs without an FBR approval, but I think switching to the PR workflow now will make life better.
Hi,
Pierre-Yves Chibon wrote:
For a first step I went with a third approach: a small python service that runs every 3 minutes (configurable): git fetch && git fsck (to ensure the git is in a correct state).
You could likely set transfer.fsckObjects¹ and skip the secondary git fsck call.
The transfer.fsckObjects option will check objects as they are pulled in via fetch (or git-receive-pack). The option is available with git-1.8.3.1 in RHEL 7 that is currently installed on batcave.
That could be set in the repo config or via git -c for just the invocation in your script.
Here's the docs from the current git release:
https://git-scm.com/docs/git-config#Documentation/git-config.txt-transferfsc...
I don't know whether all of the later improvements to catch malicious objects are backported to the RHEL 7 version or not. Some aren't relevant due to the features which allow for the malicious behaviors not being available in that version of git. But the core of the check is still present and should handle the "fsck on fetch" portion. Details are in git-config(1).
¹ or fetch.transferObjects
On Thu, Apr 23, 2020 at 11:18:15AM -0400, Todd Zullinger wrote:
Hi,
Pierre-Yves Chibon wrote:
For a first step I went with a third approach: a small python service that runs every 3 minutes (configurable): git fetch && git fsck (to ensure the git is in a correct state).
You could likely set transfer.fsckObjects¹ and skip the secondary git fsck call.
The transfer.fsckObjects option will check objects as they are pulled in via fetch (or git-receive-pack). The option is available with git-1.8.3.1 in RHEL 7 that is currently installed on batcave.
That could be set in the repo config or via git -c for just the invocation in your script.
Here's the docs from the current git release:
https://git-scm.com/docs/git-config#Documentation/git-config.txt-transferfsc...
I don't know whether all of the later improvements to catch malicious objects are backported to the RHEL 7 version or not. Some aren't relevant due to the features which allow for the malicious behaviors not being available in that version of git. But the core of the check is still present and should handle the "fsck on fetch" portion. Details are in git-config(1).
¹ or fetch.transferObjects
Nice! Thanks, I'll look into this.
Pierre
On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote:
Good Morning Everyone,
Kevin and I had a discussion on IRC the other day about mirroring the ansible repo. The options we considered are:
...snip...
For a first step I went with a third approach: a small python service that runs every 3 minutes (configurable): git fetch && git fsck (to ensure the git is in a correct state). It actually doesn't run every 3 minutes, it calls git, then wait for 3 minutes then calls git again and so on. So if git was ever to takes 4 minutes to run, there is no risk of the program stepping on its own toes.
ok.
I'm wondering if that might be overkill.
Really if pagure.io blows up, we want to be able to look at the repo we have on batcave01 and be able to still operate while we recover pagure.io. If someone pushed or merged some commits in between the last time we synced and blowup, someone should have them in their local repo right? so we could just push them to the repo on batcave and be back completly in sync, no?
but I guess it doesn't hurt to do every 3minutes, just some BW.
Another option I mentioned the other day when we were talking, but I didnt really explain well was that we could wrap ansible-playbook into a wrapper that runs a git pull then ansible-playbook... so it would always be in sync at first you run it. That could be messy tho if there's another process doing pulls, etc. So I guess scrap that idea.
I would like to propose that install and configure this for a test. I propose that we keep /git/ansible.git as is and just have a /git/mirrors/ansible.git which will be the mirror from pagure. We can migrate the hook from /git/ansible.git to /git/mirrors/ansible.git so /srv/web/infra/ansible (ie: the exploded tree) is coming from the mirror.
I think that could be confusing. If we switch we should switch and make all the old checkouts no longer work for pushing, or we are going to cause a lot of confusion. I'd say move the old one to some .back name and chmod 000 it.
Also, there is the question about the existing git repo for fedora-infrastructure on pagure. It has stuff in it now. Do we completely replace it with the current ansible repo? Or do we stream all the ansible commits on top of it so current checkouts would just be able to pull? Or do we put it on another project. I'd really prefer to have it on the same project our tickets are to avoid confusion. I kinda think it might be nice to stream it on top of the existing repo so it's not broken, but not sure how long or messy that would be.
I guess the lag time to get /git/mirrors updated will quickly be annoying (up to 3 minutes between when the PR is merged and when the playbook is updated), so if we like the workflow we will likely want to switch pretty quickly to a message-based service and then we could keep the current cron-like service to run hourly and update /git/ansible.git which would then become some kind of backup.
So, we already have fedmsg-hub listening on batcave01 for fas messages (which no longer happen). We could just reuse that? Or a fedora-messaging version of it I guess would be better.
What do you think? FBR worthy? (ie: if F32 goes out next week, I think we can wait, if it doesn't do we want to try this sooner?)
Wait until after freeze. We have waited this long and we still have release things to do in ansible. I don't want us to mess up release for this. But we can get all our birds in a line to do it right after freeze.
If we like the idea, I'll start looking into the fedora-messaging based consumer, using the approach from the current service, it should be pretty straight forward (I'll re-use the same project for it).
So this would just listen for commit / pr merge and do a git pull / sync?
kevin
On Thu, 23 Apr 2020 at 16:37, Kevin Fenzi kevin@scrye.com wrote:
On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote:
Good Morning Everyone,
Kevin and I had a discussion on IRC the other day about mirroring the
ansible
repo. The options we considered are:
...snip...
For a first step I went with a third approach: a small python service
that
runs every 3 minutes (configurable): git fetch && git fsck (to ensure
the git
is in a correct state). It actually doesn't run every 3 minutes, it
calls git,
then wait for 3 minutes then calls git again and so on. So if git was
ever to
takes 4 minutes to run, there is no risk of the program stepping on
its own
toes.
ok.
I'm wondering if that might be overkill.
Really if pagure.io blows up, we want to be able to look at the repo we have on batcave01 and be able to still operate while we recover pagure.io. If someone pushed or merged some commits in between the last time we synced and blowup, someone should have them in their local repo right? so we could just push them to the repo on batcave and be back completly in sync, no?
but I guess it doesn't hurt to do every 3minutes, just some BW.
Another option I mentioned the other day when we were talking, but I didnt really explain well was that we could wrap ansible-playbook into a wrapper that runs a git pull then ansible-playbook... so it would always be in sync at first you run it. That could be messy tho if there's another process doing pulls, etc. So I guess scrap that idea.
If we were to do that I would put in a lock logic like we do with our rsync's and such.
New workflow, everyone uses rbac-playbook and we add some lock code to check, make a lockfile, do the git pull, run the playbook, remove the lockfile. Check to see if a lockfile exists exit unless a --dontcare was put in place at which point it could do somehting like: if lockfile is younger than 10 minutes, assume that the plays are fresh enough and use that. if lockfile is older than 10 minutes, exit saying "we seem to have a dead git pull going, please manually check what the problem is"
And I think that wont' work well for our super long ansible runs... I am beginning to think we have an unsolvable problem here and everything will suck somewhere.
On Thu, Apr 23, 2020 at 01:36:48PM -0700, Kevin Fenzi wrote:
On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote:
Good Morning Everyone,
Kevin and I had a discussion on IRC the other day about mirroring the ansible repo. The options we considered are:
...snip...
For a first step I went with a third approach: a small python service that runs every 3 minutes (configurable): git fetch && git fsck (to ensure the git is in a correct state). It actually doesn't run every 3 minutes, it calls git, then wait for 3 minutes then calls git again and so on. So if git was ever to takes 4 minutes to run, there is no risk of the program stepping on its own toes.
ok.
I'm wondering if that might be overkill.
Really if pagure.io blows up, we want to be able to look at the repo we have on batcave01 and be able to still operate while we recover pagure.io. If someone pushed or merged some commits in between the last time we synced and blowup, someone should have them in their local repo right? so we could just push them to the repo on batcave and be back completly in sync, no?
Someone will have the commits sure, but since it's a PR, you could be merging a couple of commits from me while I am asleep. If pagure goes down right after the merge, you will not have the commits on your fork and batcave may not have synced them, leading to these commits not being accessible. This isn't a situation we couldn't recover from, but one of the two repos will need to get a force-push once they are both available again.
but I guess it doesn't hurt to do every 3minutes, just some BW.
I think the fedora-messaging client would reduce the BW usage, and I think I have it ready, so we may be able to go with it from the start.
Another option I mentioned the other day when we were talking, but I didnt really explain well was that we could wrap ansible-playbook into a wrapper that runs a git pull then ansible-playbook... so it would always be in sync at first you run it. That could be messy tho if there's another process doing pulls, etc. So I guess scrap that idea.
I would like to propose that install and configure this for a test. I propose that we keep /git/ansible.git as is and just have a /git/mirrors/ansible.git which will be the mirror from pagure. We can migrate the hook from /git/ansible.git to /git/mirrors/ansible.git so /srv/web/infra/ansible (ie: the exploded tree) is coming from the mirror.
I think that could be confusing. If we switch we should switch and make all the old checkouts no longer work for pushing, or we are going to cause a lot of confusion. I'd say move the old one to some .back name and chmod 000 it.
Works for me, I was thinking to keep it as some sort of backup that would be updated hourly or so and thus would give us a little breathing room in case of disaster (quickly diagnosed). I'm ok with your approach that the breathing room gained may be overkilled there (the chances that we diagnose an issue and think about stopping the service while we're trying to debug the issue and that the service hasn't synced in the mean time seems... low).
Also, there is the question about the existing git repo for fedora-infrastructure on pagure. It has stuff in it now. Do we completely replace it with the current ansible repo? Or do we stream all the ansible commits on top of it so current checkouts would just be able to pull? Or do we put it on another project. I'd really prefer to have it on the same project our tickets are to avoid confusion. I kinda think it might be nice to stream it on top of the existing repo so it's not broken, but not sure how long or messy that would be.
I was going for: https://pagure.io/Fedora-Infra/ansible/ which is marked as being the future location of our ansible repo and already has a copy of it. I had not realized we have content in https://pagure.io/fedora-infrastructure/ 's git. If we really want to have the ansible repo in that repo, the easiest may be to rebase these commits on the top of ansible and force push to that repo (so current ansible clone would pull fine once they've updated their urls=).
I guess the lag time to get /git/mirrors updated will quickly be annoying (up to 3 minutes between when the PR is merged and when the playbook is updated), so if we like the workflow we will likely want to switch pretty quickly to a message-based service and then we could keep the current cron-like service to run hourly and update /git/ansible.git which would then become some kind of backup.
So, we already have fedmsg-hub listening on batcave01 for fas messages (which no longer happen). We could just reuse that? Or a fedora-messaging version of it I guess would be better.
Almost there :)
What do you think? FBR worthy? (ie: if F32 goes out next week, I think we can wait, if it doesn't do we want to try this sooner?)
Wait until after freeze. We have waited this long and we still have release things to do in ansible. I don't want us to mess up release for this. But we can get all our birds in a line to do it right after freeze.
If we like the idea, I'll start looking into the fedora-messaging based consumer, using the approach from the current service, it should be pretty straight forward (I'll re-use the same project for it).
So this would just listen for commit / pr merge and do a git pull / sync?
Listens to commits in pagure.io and pull when it sees some in the ansible repo.
Pierre
On Fri, Apr 24, 2020 at 02:21:14PM +0200, Pierre-Yves Chibon wrote:
...snip...
I was going for: https://pagure.io/Fedora-Infra/ansible/ which is marked as being the future location of our ansible repo and already has a copy of it.
Yeah, I am not even sure who made that. It was not me. ;)
I had not realized we have content in https://pagure.io/fedora-infrastructure/ 's git.
Yes, mostly stuff that doesn't matter too much anymore, but it used to have 'public' things we needed to share out from puppet repo.
If we really want to have the ansible repo in that repo, the easiest may be to rebase these commits on the top of ansible and force push to that repo (so current ansible clone would pull fine once they've updated their urls=).
well, I guess it depends on what way you want to preserve things:
1. Rebase the fedora-infrastructure.git on top of ansible.git and force push that. That would mean everyone with a existing fedora-infrastructure repo would have to scrap it and repull. But everyone with fedora-infra/ansible.git could just pull.
2. Push ansible.git on top of fedora-infrastructure. Then the opposite. People who have an existing fedora-infrastructure repo could just pull, people with fedora-infra/ansible would need to recheckout.
3. Just scrap the fedora-infrastructure repo entirely and replace it with ansible.git. This would mean both groups would have to reclone.
I don't think there's really enough people with checkouts that couldn't just reclone, so I think we could do anything here and I don't much care. I think we should probibly move the old fedora-infrastructre files under some directory to keep them cluttering the ansible repo tho.
Most if not all the fedora-infrastructure.git is old stuff we could do without too.
kevin
On Friday, April 24, 2020 5:47:22 PM CEST Kevin Fenzi wrote:
On Fri, Apr 24, 2020 at 02:21:14PM +0200, Pierre-Yves Chibon wrote:
...snip...
I was going for: https://pagure.io/Fedora-Infra/ansible/ which is marked as being the future location of our ansible repo and already has a copy of it.
Yeah, I am not even sure who made that. It was not me. ;)
I had not realized we have content in https://pagure.io/fedora-infrastructure/ 's git.
Yes, mostly stuff that doesn't matter too much anymore, but it used to have 'public' things we needed to share out from puppet repo.
If we really want to have the ansible repo in that repo, the easiest may be to rebase these commits on the top of ansible and force push to that repo (so current ansible clone would pull fine once they've updated their urls=).
well, I guess it depends on what way you want to preserve things:
- Rebase the fedora-infrastructure.git on top of ansible.git and force
push that. That would mean everyone with a existing fedora-infrastructure repo would have to scrap it and repull. But everyone with fedora-infra/ansible.git could just pull.
- Push ansible.git on top of fedora-infrastructure. Then the opposite.
People who have an existing fedora-infrastructure repo could just pull, people with fedora-infra/ansible would need to recheckout.
- Just scrap the fedora-infrastructure repo entirely and replace it
with ansible.git. This would mean both groups would have to reclone.
I don't think there's really enough people with checkouts that couldn't just reclone, so I think we could do anything here and I don't much care. I think we should probibly move the old fedora-infrastructre files under some directory to keep them cluttering the ansible repo tho.
Most if not all the fedora-infrastructure.git is old stuff we could do without too.
Use the force Luke, .... no! Never force push :-) did this really happen?
It means that all the links from outside world pointing to specific ansible.git hashes are just pointing to nowhere now...
If anything, the repositories should be merged together with --allow-unrelated histories or something (so all the objects stay available). I never seen that in practice, but I never seen force push in such important git repo so far, either. :-(
Any chance we could go back, do a proper merge (with unrelated histories), rebase the latest commits we did since the forcepush on top of that, and force-push once more? I mean ... it is ***definitely*** worth it ... I mean it still is, so I'd do it ASAP.
Pavel
On Thu, Apr 30, 2020 at 02:06:15PM +0200, Pavel Raiskup wrote:
On Friday, April 24, 2020 5:47:22 PM CEST Kevin Fenzi wrote:
On Fri, Apr 24, 2020 at 02:21:14PM +0200, Pierre-Yves Chibon wrote:
...snip...
I was going for: https://pagure.io/Fedora-Infra/ansible/ which is marked as being the future location of our ansible repo and already has a copy of it.
Yeah, I am not even sure who made that. It was not me. ;)
I had not realized we have content in https://pagure.io/fedora-infrastructure/ 's git.
Yes, mostly stuff that doesn't matter too much anymore, but it used to have 'public' things we needed to share out from puppet repo.
If we really want to have the ansible repo in that repo, the easiest may be to rebase these commits on the top of ansible and force push to that repo (so current ansible clone would pull fine once they've updated their urls=).
well, I guess it depends on what way you want to preserve things:
- Rebase the fedora-infrastructure.git on top of ansible.git and force
push that. That would mean everyone with a existing fedora-infrastructure repo would have to scrap it and repull. But everyone with fedora-infra/ansible.git could just pull.
- Push ansible.git on top of fedora-infrastructure. Then the opposite.
People who have an existing fedora-infrastructure repo could just pull, people with fedora-infra/ansible would need to recheckout.
- Just scrap the fedora-infrastructure repo entirely and replace it
with ansible.git. This would mean both groups would have to reclone.
I don't think there's really enough people with checkouts that couldn't just reclone, so I think we could do anything here and I don't much care. I think we should probibly move the old fedora-infrastructre files under some directory to keep them cluttering the ansible repo tho.
Most if not all the fedora-infrastructure.git is old stuff we could do without too.
Use the force Luke, .... no! Never force push :-) did this really happen?
It did not, you're all good. We're still figuring this out and haven't started on merging the two repos yet.
It means that all the links from outside world pointing to specific ansible.git hashes are just pointing to nowhere now...
If anything, the repositories should be merged together with --allow-unrelated histories or something (so all the objects stay available). I never seen that in practice, but I never seen force push in such important git repo so far, either. :-(
This sounds like an interesting idea, I'll look into it
Thanks!
Pierre
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Neal Gompa -
Pavel Raiskup -
Pierre-Yves Chibon -
Stephen John Smoogen -
Todd Zullinger