Good Morning Everyone,
I figured it has been a while since I reported progress on making pagure a front-end for dist-git.
So here is a small status update.
What works: - Well currently pagure is working as a front-end for dist-git in stg: https://src.stg.fedoraproject.org/pagure/ - Hosting repos, browsing them, creating PR works all fine (w/ one , see below) - Features we want to exclude have been turned off (no user management on the pagure side)
What does not work: - Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod, in stg, the updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure) runs in ~30/35 minutes (if it doesn't crash, which my last run just did) This is a blocker since it means it takes 30 to 35 minutes for someone to get access to their git repo or (worse) their fork - This pagure instance has the same issue as the main one, including a heisenbug we're trying to track but have had no luck reproducing so far :(
What needs to be done: - Fix the sync script - Make it *way* faster than it is - Make it creates the project on pagure using the releng user rather than relying on the first contributor it finds in the list of maintainers - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we re-generate the gitolite configuration file when someone forks a project (and thus give them access to their fork) - Once above is done: call for more testers
In the future: - I think we will want to deprecate pkgdb entirely, so while the work above is important, ideally it won't be there for too long. - With pkgdb out of the loop, we'll need to figure some things out: - Where/How to store the contact info for bugzilla - Not sure relying on pagure's ACLs there is the way to go since we would loose a level of granularity in the ACLs that I know people like and ask for (having commit w/o being on the CC list in bugzilla or being on the CC list w/o being a packager) - How/when to require people be part of the packager group in FAS? - Since one of the idea of pagure is to make it easier for "drive-by" contribution to spec files, requiring to be a packager should only be there for maintainers, but pagure doesn't have this level of information/requirement, so we would need to find something or some place to add this requirement or see if that requirement still stands
This is all I can think of for now, I'll update this thread if I come up with more ideas/challenges.
Have a nice day, Pierre
Can you please report this to releng also?
Dennis
El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
Good Morning Everyone,
I figured it has been a while since I reported progress on making pagure a front-end for dist-git.
So here is a small status update.
What works:
- Well currently pagure is working as a front-end for dist-git in
stg: https://src.stg.fedoraproject.org/pagure/
- Hosting repos, browsing them, creating PR works all fine (w/ one ,
see below)
- Features we want to exclude have been turned off (no user
management on the pagure side)
What does not work:
- Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod,
in stg, the updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure) runs in ~30/35 minutes (if it doesn't crash, which my last run just did) This is a blocker since it means it takes 30 to 35 minutes for someone to get access to their git repo or (worse) their fork
- This pagure instance has the same issue as the main one, including
a heisenbug we're trying to track but have had no luck reproducing so far :(
What needs to be done:
- Fix the sync script
- Make it *way* faster than it is - Make it creates the project on pagure using the releng user rather than relying on the first contributor it finds in the list of maintainers - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we re-generate the gitolite configuration file when someone forks a project (and thus give them access to their fork)
- Once above is done: call for more testers
In the future:
- I think we will want to deprecate pkgdb entirely, so while the work
above is important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
- Where/How to store the contact info for bugzilla - Not sure relying on pagure's ACLs there is the way to go since we would loose a level of granularity in the ACLs that I know people like and ask for (having commit w/o being on the CC list in bugzilla or being on the CC list w/o being a packager) - How/when to require people be part of the packager group in FAS? - Since one of the idea of pagure is to make it easier for "drive-by" contribution to spec files, requiring to be a packager should only be there for maintainers, but pagure doesn't have this level of information/requirement, so we would need to find something or some place to add this requirement or see if that requirement still stands
This is all I can think of for now, I'll update this thread if I come up with more ideas/challenges.
Have a nice day, Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproj ect.org
Reply-to is set to infra@ so we can keep discussion in one place.
On Thu, Apr 20, 2017 at 12:08:35PM -0500, Dennis Gilmore wrote:
Can you please report this to releng also?
Dennis
El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
Good Morning Everyone,
I figured it has been a while since I reported progress on making pagure a front-end for dist-git.
So here is a small status update.
What works:
- Well currently pagure is working as a front-end for dist-git in
stg: https://src.stg.fedoraproject.org/pagure/
- Hosting repos, browsing them, creating PR works all fine (w/ one ,
see below)
- Features we want to exclude have been turned off (no user
management on the pagure side)
What does not work:
- Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod,
in stg, the updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure) runs in ~30/35 minutes (if it doesn't crash, which my last run just did) This is a blocker since it means it takes 30 to 35 minutes for someone to get access to their git repo or (worse) their fork
- This pagure instance has the same issue as the main one, including
a heisenbug we're trying to track but have had no luck reproducing so far :(
What needs to be done:
- Fix the sync script
- Make it *way* faster than it is - Make it creates the project on pagure using the releng user rather than relying on the first contributor it finds in the list of maintainers - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we re-generate the gitolite configuration file when someone forks a project (and thus give them access to their fork)
- Once above is done: call for more testers
In the future:
- I think we will want to deprecate pkgdb entirely, so while the work
above is important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
- Where/How to store the contact info for bugzilla - Not sure relying on pagure's ACLs there is the way to go since we would loose a level of granularity in the ACLs that I know people like and ask for (having commit w/o being on the CC list in bugzilla or being on the CC list w/o being a packager) - How/when to require people be part of the packager group in FAS? - Since one of the idea of pagure is to make it easier for "drive-by" contribution to spec files, requiring to be a packager should only be there for maintainers, but pagure doesn't have this level of information/requirement, so we would need to find something or some place to add this requirement or see if that requirement still stands
This is all I can think of for now, I'll update this thread if I come up with more ideas/challenges.
Have a nice day, Pierre _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproj ect.org
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
Just making sure the whole original message made it to rel-eng list. Reply-to is set to infra@ so we can keep discussion in one place.
On Thu, Apr 20, 2017 at 12:08:35PM -0500, Dennis Gilmore wrote:
Can you please report this to releng also?
Dennis
El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
Good Morning Everyone,
I figured it has been a while since I reported progress on making pagure a front-end for dist-git.
So here is a small status update.
What works:
- Well currently pagure is working as a front-end for dist-git in stg:
https://src.stg.fedoraproject.org/pagure/
- Hosting repos, browsing them, creating PR works all fine (w/ one , see
below)
- Features we want to exclude have been turned off (no user management on the
pagure side)
What does not work:
- Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod, in stg, the
updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure) runs in ~30/35 minutes (if it doesn't crash, which my last run just did) This is a blocker since it means it takes 30 to 35 minutes for someone to get access to their git repo or (worse) their fork
- This pagure instance has the same issue as the main one, including a heisenbug
we're trying to track but have had no luck reproducing so far :(
What needs to be done:
- Fix the sync script
- Make it *way* faster than it is - Make it creates the project on pagure using the releng user rather than relying on the first contributor it finds in the list of maintainers - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we re-generate the gitolite configuration file when someone forks a project (and thus give them access to their fork)
- Once above is done: call for more testers
In the future:
- I think we will want to deprecate pkgdb entirely, so while the work above is
important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
- Where/How to store the contact info for bugzilla - Not sure relying on pagure's ACLs there is the way to go since we would loose a level of granularity in the ACLs that I know people like and ask for (having commit w/o being on the CC list in bugzilla or being on the CC list w/o being a packager) - How/when to require people be part of the packager group in FAS? - Since one of the idea of pagure is to make it easier for "drive-by" contribution to spec files, requiring to be a packager should only be there for maintainers, but pagure doesn't have this level of information/requirement, so we would need to find something or some place to add this requirement or see if that requirement still stands
This is all I can think of for now, I'll update this thread if I come up with more ideas/challenges.
Have a nice day, Pierre
Hi,
On Thu, Apr 20, 2017 at 03:45:16PM +0200, Pierre-Yves Chibon wrote:
- With pkgdb out of the loop, we'll need to figure some things out:
- Where/How to store the contact info for bugzilla
- Not sure relying on pagure's ACLs there is the way to go since we would loose a level of granularity in the ACLs that I know people like and ask for (having commit w/o being on the CC list in bugzilla or being on the CC list w/o being a packager)
- How/when to require people be part of the packager group in FAS?
- Since one of the idea of pagure is to make it easier for "drive-by" contribution to spec files, requiring to be a packager should only be there for maintainers, but pagure doesn't have this level of information/requirement, so we would need to find something or some place to add this requirement or see if that requirement still stands
sorry if I missed it, but pkgdb manages some more things where I not yet understand how this maps to the pagure dist-git workflow:
- Request a new package to be added to pagure dist-git/a package to be unretired - Requests for new (EPEL) branches - Orphan/Retire a package - Manage development status of collections (In development, live, EOL) - Configuring upstream release monitoring - Manage koshei integration
Kind regards Till
On Thu, Apr 20, 2017 at 08:33:23PM +0200, Till Maas wrote:
Hi,
On Thu, Apr 20, 2017 at 03:45:16PM +0200, Pierre-Yves Chibon wrote:
- With pkgdb out of the loop, we'll need to figure some things out:
- Where/How to store the contact info for bugzilla
- Not sure relying on pagure's ACLs there is the way to go since we would loose a level of granularity in the ACLs that I know people like and ask for (having commit w/o being on the CC list in bugzilla or being on the CC list w/o being a packager)
- How/when to require people be part of the packager group in FAS?
- Since one of the idea of pagure is to make it easier for "drive-by" contribution to spec files, requiring to be a packager should only be there for maintainers, but pagure doesn't have this level of information/requirement, so we would need to find something or some place to add this requirement or see if that requirement still stands
sorry if I missed it, but pkgdb manages some more things where I not yet understand how this maps to the pagure dist-git workflow:
- Request a new package to be added to pagure dist-git/a package to be unretired
New package to add == request for a new PoC of a new component in bugzilla
- Requests for new (EPEL) branches
== request for a new PoC in Fedora EPEL in bugzilla
- Orphan/Retire a package
Orphan == request to change the PoC of the component in bugzilla
- Manage development status of collections (In development, live, EOL)
This is going away with the modularity effort and could be ported to PDC in the mean time.
- Configuring upstream release monitoring
- Manage koshei integration
We could include these two in the substitute of our current pkgdb.
The only remaining item which I'm not entirely sure how to handle it is retiring a package.
Do note that this is all currently WIP or rather thoughts in progress :)
Pierre
On Thu, 2017-04-20 at 15:45 +0200, Pierre-Yves Chibon wrote:
- I think we will want to deprecate pkgdb entirely, so while the work
above is important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
Just to note - Bodhi uses pkgdb to figure out ACLs and who to e-mail about updates (possibly more, but that's what comes to mind) so we'd need to coordinate the EOL of pkgdb with a Bodhi update too.
On Thu, Apr 20, 2017 at 03:19:49PM -0400, Randy Barlow wrote:
On Thu, 2017-04-20 at 15:45 +0200, Pierre-Yves Chibon wrote:
- I think we will want to deprecate pkgdb entirely, so while the work
above is important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
Just to note - Bodhi uses pkgdb to figure out ACLs and who to e-mail about updates (possibly more, but that's what comes to mind) so we'd need to coordinate the EOL of pkgdb with a Bodhi update too.
Bodhi still sends email? Doesn't it rely on FMN now?
But yes that change will require coordination anyway indeed.
Pierre
On Fri, 2017-04-21 at 10:23 +0200, Pierre-Yves Chibon wrote:
Bodhi still sends email? Doesn't it rely on FMN now?
Yeah Bodhi sends e-mails directly still. In fact, I've got a rather vexing issue about it:
https://github.com/fedora-infra/bodhi/issues/1403
But yes that change will require coordination anyway indeed.
Cool.
On Thu, Apr 20, 2017 at 7:45 AM, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
I figured it has been a while since I reported progress on making pagure a front-end for dist-git.
.. snip ..
What needs to be done:
- Fix the sync script
- Make it *way* faster than it is
- Make it creates the project on pagure using the releng user rather than relying on the first contributor it finds in the list of maintainers
- Make fedmsg-genacls be triggered on pagure's fork fedmsg message so
that we re-generate the gitolite configuration file when someone forks a project (and thus give them access to their fork)
- Once above is done: call for more testers
Would git mirroring work well here? If so, it might be worth checking out grokmirror. (https://github.com/lfit/grokmirror)
.. snip ..
Have a nice day, Pierre
Hope this helps,
herlo
On Thu, Apr 20, 2017 at 01:51:14PM -0600, Clint Savage wrote:
On Thu, Apr 20, 2017 at 7:45 AM, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone, I figured it has been a while since I reported progress on making pagure a front-end for dist-git... snip ..
What needs to be done: - Fix the sync script  - Make it *way* faster than it is  - Make it creates the project on pagure using the releng user rather than   relying on the first contributor it finds in the list of maintainers  - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we   re-generate the gitolite configuration file when someone forks a project   (and thus give them access to their fork) - Once above is done: call for more testersWould git mirroring work well here? If so, it might be worth checking out grokmirror. (https://github.com/lfit/grokmirror)
Not entirely sure to see how that would work. This seems to mirror git repositories, but we are not mirroring them here, what we want to mirror is the ACLs from pkgdb to gitolite (that we have already) and to pagure (that we have but need to improve/optimize).
Could you expand on what your thoughts were? Maybe I'm miss-understanding them :)
Thanks, Pierre
On Fri, Apr 21, 2017 at 2:26 AM, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Thu, Apr 20, 2017 at 01:51:14PM -0600, Clint Savage wrote:
On Thu, Apr 20, 2017 at 7:45 AM, Pierre-Yves Chibon <
pingou@pingoured.fr>
wrote:
Good Morning Everyone, I figured it has been a while since I reported progress on makingpagure
a front-end for dist-git... snip ..
What needs to be done: - Fix the sync script  - Make it *way* faster than it is  - Make it creates the project on pagure using the releng userrather
than   relying on the first contributor it finds in the list of maintainers  - Make fedmsg-genacls be triggered on pagure's fork fedmsgmessage so
that we   re-generate the gitolite configuration file when someoneforks a
project   (and thus give them access to their fork) - Once above is done: call for more testersWould git mirroring work well here? If so, it might be worth checking
out
grokmirror. (https://github.com/lfit/grokmirror)
Not entirely sure to see how that would work. This seems to mirror git repositories, but we are not mirroring them here, what we want to mirror is the ACLs from pkgdb to gitolite (that we have already) and to pagure (that we have but need to improve/optimize).
Could you expand on what your thoughts were? Maybe I'm miss-understanding them :)
I am not completely sure how pagure worked under the covers, but I know it uses gitolite, at least that's what I'd read above. At The Linux Foundation, when I worked there, we mirrored bare repositories using grokmirror, and fast, too, with hundreds, sometimes thousands, of remotes as well.
Maybe the crux of the problem is around compiling the gitolite.conf (and its plethora of sub-configuration files), so this might not be a solution for you. But if it's about syncing between two git resources, grokmirror would provide that functionality. It's been a while since I've worked with it, but we could chat on IRC next week and maybe setup an example. Only if that sort of thing would help though.
herlo
Thanks, Pierre
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists. fedoraproject.org
El jue, 20-04-2017 a las 15:45 +0200, Pierre-Yves Chibon escribió:
Good Morning Everyone,
I figured it has been a while since I reported progress on making pagure a front-end for dist-git.
So here is a small status update.
What works:
- Well currently pagure is working as a front-end for dist-git in
stg: https://src.stg.fedoraproject.org/pagure/
- Hosting repos, browsing them, creating PR works all fine (w/ one ,
see below)
- Features we want to exclude have been turned off (no user
management on the pagure side)
What does not work:
- Syncing the ACLs from pkgdb to gitolite takes ~3 minutes in prod,
in stg, the updated script in stg (which sync the ACLs from pkgdb to gitolite and pagure) runs in ~30/35 minutes (if it doesn't crash, which my last run just did) This is a blocker since it means it takes 30 to 35 minutes for someone to get access to their git repo or (worse) their fork
- This pagure instance has the same issue as the main one, including
a heisenbug we're trying to track but have had no luck reproducing so far :(
What needs to be done:
- Fix the sync script
- Make it *way* faster than it is - Make it creates the project on pagure using the releng user rather than relying on the first contributor it finds in the list of maintainers - Make fedmsg-genacls be triggered on pagure's fork fedmsg message so that we re-generate the gitolite configuration file when someone forks a project (and thus give them access to their fork)
- Once above is done: call for more testers
In the future:
- I think we will want to deprecate pkgdb entirely, so while the work
above is important, ideally it won't be there for too long.
- With pkgdb out of the loop, we'll need to figure some things out:
- Where/How to store the contact info for bugzilla - Not sure relying on pagure's ACLs there is the way to go since we would loose a level of granularity in the ACLs that I know people like and ask for (having commit w/o being on the CC list in bugzilla or being on the CC list w/o being a packager) - How/when to require people be part of the packager group in FAS? - Since one of the idea of pagure is to make it easier for "drive-by" contribution to spec files, requiring to be a packager should only be there for maintainers, but pagure doesn't have this level of information/requirement, so we would need to find something or some place to add this requirement or see if that requirement still stands
This is all I can think of for now, I'll update this thread if I come up with more ideas/challenges.
critpath is stored in pkgdb and pulled into bodhi and other places. Was just updating critpath and it stuck out to me.
Dennis
infrastructure@lists.fedoraproject.org
-
Clint Savage -
Dennis Gilmore -
Paul W. Frields -
Pierre-Yves Chibon -
Randy Barlow -
Till Maas