I've written up a brief proposal about how "hidden" packages may be supported in Koji. The objective of this is to enable building EPEL packages in Koji. I wrote this up fairly quickly, and I'm sure I haven't thought through all the issues, but I wanted to get the ball rolling. Let me know if you have any questions/comments/ideas/issues relating to this proposal.
http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
Thanks, Mike
On Friday 14 March 2008, Mike Bonnet wrote:
I've written up a brief proposal about how "hidden" packages may be supported in Koji. The objective of this is to enable building EPEL packages in Koji. I wrote this up fairly quickly, and I'm sure I haven't thought through all the issues, but I wanted to get the ball rolling. Let me know if you have any questions/comments/ideas/issues relating to this proposal.
http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
Thanks, Mike
the tree will have to be nothing like /mnt/koji/packages
instead it will have to be like http://download.fedora.redhat.com/pub/fedora/linux/ so that you can use rsync and repo mirroring tools to sync the content and keep trees in sync. ill leave it up to Seth to explain more but hit micro repository option he is working on would allow us to pull the existing repodata intothe new repodata.
We really only need to have a command that can suck the repodata into the database. so we know about the packages.
Dennis
On Fri, Mar 14, 2008 at 6:55 PM, Dennis Gilmore dennis@ausil.us wrote:
On Friday 14 March 2008, Mike Bonnet wrote:
I've written up a brief proposal about how "hidden" packages may be supported in Koji. The objective of this is to enable building EPEL packages in Koji. I wrote this up fairly quickly, and I'm sure I haven't thought through all the issues, but I wanted to get the ball rolling. Let me know if you have any questions/comments/ideas/issues relating to this proposal.
http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
Thanks, Mike
the tree will have to be nothing like /mnt/koji/packages
instead it will have to be like http://download.fedora.redhat.com/pub/fedora/linux/ so that you can use rsync and repo mirroring tools to sync the content and keep trees in sync. ill leave it up to Seth to explain more but hit micro repository option he is working on would allow us to pull the existing repodata intothe new repodata.
We really only need to have a command that can suck the repodata into the database. so we know about the packages.
Another big issue will be that EL is split in so many 'interesting' ways. You need to be able to either have the build system know of every seperate channels from RHN for each variation EL-3,4,5 or have RHN somehow give a conglomerate channel to you of all the packages in one big bucket. The code from DAG's mrepo sort of does this but would need some work on getting it to play nicely
On Fri, 2008-03-14 at 19:17 -0600, Stephen John Smoogen wrote:
On Fri, Mar 14, 2008 at 6:55 PM, Dennis Gilmore dennis@ausil.us wrote:
On Friday 14 March 2008, Mike Bonnet wrote:
I've written up a brief proposal about how "hidden" packages may be supported in Koji. The objective of this is to enable building EPEL packages in Koji. I wrote this up fairly quickly, and I'm sure I haven't thought through all the issues, but I wanted to get the ball rolling. Let me know if you have any questions/comments/ideas/issues relating to this proposal.
http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
Thanks, Mike
the tree will have to be nothing like /mnt/koji/packages
instead it will have to be like http://download.fedora.redhat.com/pub/fedora/linux/ so that you can use rsync and repo mirroring tools to sync the content and keep trees in sync. ill leave it up to Seth to explain more but hit micro repository option he is working on would allow us to pull the existing repodata intothe new repodata.
We really only need to have a command that can suck the repodata into the database. so we know about the packages.
Another big issue will be that EL is split in so many 'interesting' ways. You need to be able to either have the build system know of every seperate channels from RHN for each variation EL-3,4,5 or have RHN somehow give a conglomerate channel to you of all the packages in one big bucket. The code from DAG's mrepo sort of does this but would need some work on getting it to play nicely
We're not going to be pulling packages straight from RHN, we'll be manually importing the packages. The proposal allows for multiple alternate directory trees, so it will be possible to have a tree to which we import the RHEL-4 binaries, a tree for RHEL-5 binaries, etc. When a new RHEL update is released, it'll be up to an admin to download the new packages and import them into the proper tree.
On Fri, 2008-03-14 at 19:55 -0500, Dennis Gilmore wrote:
On Friday 14 March 2008, Mike Bonnet wrote:
I've written up a brief proposal about how "hidden" packages may be supported in Koji. The objective of this is to enable building EPEL packages in Koji. I wrote this up fairly quickly, and I'm sure I haven't thought through all the issues, but I wanted to get the ball rolling. Let me know if you have any questions/comments/ideas/issues relating to this proposal.
http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
Thanks, Mike
the tree will have to be nothing like /mnt/koji/packages
instead it will have to be like http://download.fedora.redhat.com/pub/fedora/linux/ so that you can use rsync and repo mirroring tools to sync the content and keep trees in sync. ill leave it up to Seth to explain more but hit micro repository option he is working on would allow us to pull the existing repodata intothe new repodata.
I'm not really sure what you're talking about. *No one* will be mirroring the hidden packages in the case we're talking about (building EPEL in Koji), these will be RHEL binaries, and only available to the builders.
We really only need to have a command that can suck the repodata into the database. so we know about the packages.
We have a command that can suck data about packages in to the database. It's "koji import". The proposal is designed to work within the framework of that we already have in Koji, to try to minimize the number of changes and thus the time it'll take to implement.
The micro repository option Seth has talked about sounds interesting, but it doesn't really help with the issue of making packages available to Koji builders without making them available to the world.
On Saturday 15 March 2008, Mike Bonnet wrote:
On Fri, 2008-03-14 at 19:55 -0500, Dennis Gilmore wrote:
On Friday 14 March 2008, Mike Bonnet wrote:
I've written up a brief proposal about how "hidden" packages may be supported in Koji. The objective of this is to enable building EPEL packages in Koji. I wrote this up fairly quickly, and I'm sure I haven't thought through all the issues, but I wanted to get the ball rolling. Let me know if you have any questions/comments/ideas/issues relating to this proposal.
http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
Thanks, Mike
the tree will have to be nothing like /mnt/koji/packages
instead it will have to be like http://download.fedora.redhat.com/pub/fedora/linux/ so that you can use rsync and repo mirroring tools to sync the content and keep trees in sync. ill leave it up to Seth to explain more but hit micro repository option he is working on would allow us to pull the existing repodata intothe new repodata.
I'm not really sure what you're talking about. *No one* will be mirroring the hidden packages in the case we're talking about (building EPEL in Koji), these will be RHEL binaries, and only available to the builders.
What im talking about was the proposal put forward when we had the buildsys meeting the proposal that we all said sounded like the way to move forward. This is not that proposal. importing packages is the one thing that hurts koji from getting wider use outside of fedora. Some people will import the data. Most do not want to. Fedora will be mirroring RHEL content from RHN, people outside of fedora will be mirroring fedora and building on top of that.
We really only need to have a command that can suck the repodata into the database. so we know about the packages.
We have a command that can suck data about packages in to the database. It's "koji import". The proposal is designed to work within the framework of that we already have in Koji, to try to minimize the number of changes and thus the time it'll take to implement.
sure koji import --repodata --url=<http|ftp|file>://<host>/path/to/repodata
The micro repository option Seth has talked about sounds interesting, but it doesn't really help with the issue of making packages available to Koji builders without making them available to the world.
sure it does, we have an internal only tree thats on the phx internal network and not available outside of phx, just as we do now.
Dennis
On Sat, 2008-03-15 at 21:58 -0500, Dennis Gilmore wrote:
What im talking about was the proposal put forward when we had the buildsys meeting the proposal that we all said sounded like the way to move forward. This is not that proposal. importing packages is the one thing that hurts koji from getting wider use outside of fedora. Some people will import the data. Most do not want to. Fedora will be mirroring RHEL content from RHN, people outside of fedora will be mirroring fedora and building on top of that.
I'm not sure I see the problem here, necessarily.
okay here's how I see what Mike's proposal does and then what we eventually want to do:
Mike's Proposal: - gives us a way of building from the reposync'd rhel-trees that we now have on puppet1 w/o distributing these pkgs out to the world. Yay - gives us a way of building epel in koji so we can finally disable plague, also yay.
The eventual future: - have a way of koji to build from an arbitrary set of remote repos by: 1. syncing down the repodata from those remote repos and 'faking' an import 2. creating its own local repo out of the remote repos (or micro-repos) for external use.
There's nothing mutually exclusive about these two items and, afaict, there's no requirement that they be done at the same time, either.
I guess in short: - Dennis: I think your suggestion is one we should do but we do not require it immediately.
- Mike: The proposal sounds round enough to me. We've already got all of the production repos(no betas) in rhn for rhel4 and 5 for i386, x86_64 and ppc* synced over on puppet1. They resync every night. Is there anything else you need that I missed?
-sv
Dennis Gilmore wrote:
What im talking about was the proposal put forward when we had the buildsys meeting the proposal that we all said sounded like the way to move forward. This is not that proposal. importing packages is the one thing that hurts koji from getting wider use outside of fedora. Some people will import the data. Most do not want to. Fedora will be mirroring RHEL content from RHN, people outside of fedora will be mirroring fedora and building on top of that.
All that was really concluded at the meeting was that Mike and/or I would create a proposal. As I said in that meeting there are basically two approaches: import and hide or find a sane way to use external repos. Mike's proposal is the former.
On Fri, Mar 14, 2008 at 5:33 PM, Mike Bonnet mikeb@redhat.com wrote:
I've written up a brief proposal about how "hidden" packages may be supported in Koji. The objective of this is to enable building EPEL packages in Koji. I wrote this up fairly quickly, and I'm sure I haven't thought through all the issues, but I wanted to get the ball rolling. Let me know if you have any questions/comments/ideas/issues relating to this proposal.
http://people.redhat.com/mikeb/koji/koji-hidden-packages-proposal.html
Thanks, Mike
What can I do to help on this?
buildsys@lists.fedoraproject.org
-
Dennis Gilmore -
Mike Bonnet -
Mike McLean -
Seth Vidal -
Stephen John Smoogen