Hi, As some of you are aware the comps changes that anaconda wanted changed how comps works in yum which nicely BROKE Fedora Extras rawhide builds once the comps.xml changes landed in rawhide. So what do we do to fix it?
Jeremy and I tossed around a couple of ideas - none of them terribly attractive but we settled on this, I'd like to hear some input:
- make three packages and put them in the buildgroups repo: http://fedoraproject.org/buildgroups/ The packages would have no real payload but would have only a requires list that matches the 3 groups listed in: http://fedoraproject.org/buildgroups/4/i386/buildroots.xml
- modify mock to act on those packages instead of on group names. Possibly making that act be something you configure so we don't break everyone with the mock update.
- no longer have to worry about groups-file-format-changes screwing up mock and extras builds
So what do y'all think? That sound like a plan? Would someone like to make the packages for us while I go futz with mock?
thanks, -sv
seth vidal wrote:
Hi, As some of you are aware the comps changes that anaconda wanted changed how comps works in yum which nicely BROKE Fedora Extras rawhide builds once the comps.xml changes landed in rawhide. So what do we do to fix it?
Jeremy and I tossed around a couple of ideas - none of them terribly attractive but we settled on this, I'd like to hear some input:
make three packages and put them in the buildgroups repo: http://fedoraproject.org/buildgroups/ The packages would have no real payload but would have only a requires list that matches the 3 groups listed in: http://fedoraproject.org/buildgroups/4/i386/buildroots.xml
modify mock to act on those packages instead of on group names.
Possibly making that act be something you configure so we don't break everyone with the mock update.
- no longer have to worry about groups-file-format-changes screwing up
mock and extras builds
So what do y'all think? That sound like a plan? Would someone like to make the packages for us while I go futz with mock?
Old fedora.us and our current Extras packaging guidelines tell folks to test their builds against the minimum build environment pulled in by the artifical deps of the fedora-rpmdevtools package. You could just use fedora-rpmdevtools as the base package instead of creating yet another package to pull in minimum buildroot deps.
Note that we made an explicit decision NOT to include auto* as base deps in fedora-rpmdevtools. This is because different packages may require specific versions of autoconf or automake.
Warren Togami wtogami@redhat.com
Old fedora.us and our current Extras packaging guidelines tell folks to test their builds against the minimum build environment pulled in by the artifical deps of the fedora-rpmdevtools package. You could just use fedora-rpmdevtools as the base package instead of creating yet another package to pull in minimum buildroot deps.
We'd need a fair bit more stuff than is in fedora-rpmdevtools: - createrepo, buildsys-macros, python, rpm-python,
take a look at the contents of this file: http://fedoraproject.org/buildgroups/4/i386/buildroots.xml
that's what we need and that's a bit of a problem for fedora-rpmdevtools.
Note that we made an explicit decision NOT to include auto* as base deps in fedora-rpmdevtools. This is because different packages may require specific versions of autoconf or automake.
right and in the buildroots.xml we include all the automakes and autoconf.
I think we'd be better off with NOT confusing the purposes of fedora-rpmdevtools package and the mock build packages, not to mention that I'd like to make sure this is written so it can be used for !fedora distributions, too.
I know I'm not using mock just for building fedora-based things; I'm sure others aren't either.
-sv
seth vidal wrote:
Old fedora.us and our current Extras packaging guidelines tell folks to test their builds against the minimum build environment pulled in by the artifical deps of the fedora-rpmdevtools package. You could just use fedora-rpmdevtools as the base package instead of creating yet another package to pull in minimum buildroot deps.
We'd need a fair bit more stuff than is in fedora-rpmdevtools:
- createrepo, buildsys-macros, python, rpm-python,
take a look at the contents of this file: http://fedoraproject.org/buildgroups/4/i386/buildroots.xml
that's what we need and that's a bit of a problem for fedora-rpmdevtools.
OK no problem. So do you think this package should exist in the Extras tree, or a special case outside?
Warren Togami wtogami@redhat.com
On Dec 13, 2005, at 12:32 PM, seth vidal wrote:
OK no problem. So do you think this package should exist in the Extras tree, or a special case outside?
I'm thinking it should be like buildsys-macros, only really existing in the buildgroups repo.
-sv
That's what I'm picturing, too. Although I don't really like the idea of creating extra packages just to fake some dependencies... By the way, is there a place where the changes to comps.xml are documented? I'd like to know more about what's changing before I add (subtract?) any more from this conversation.
-Jeff
On Tue, 2005-12-13 at 17:14 -0400, Jeff Sheltren wrote:
On Dec 13, 2005, at 12:32 PM, seth vidal wrote:
OK no problem. So do you think this package should exist in the Extras tree, or a special case outside?
I'm thinking it should be like buildsys-macros, only really existing in the buildgroups repo.
-sv
That's what I'm picturing, too. Although I don't really like the idea of creating extra packages just to fake some dependencies... By the way, is there a place where the changes to comps.xml are documented? I'd like to know more about what's changing before I add (subtract?) any more from this conversation.
right now it's all in comps.py in yum cvs-HEAD.
however the short version, so far, is this.
There are 2 main elements: - category - group
groups can contain packages in modes 'mandatory, default, optional'. groups CANNOT contain other groups.
categories can contain other groups.
Things still up in the air: 1. whether or not categories can or should contain single package entries. 2. How/what yum will do with the categories at all. Right now the only thing using the categories is anaconda.
Both categories and groups are exposed via yum.comps right now. The comps/groupInfo object in yum is signficantly simpler than it was before.
hope this helps. -sv
seth vidal wrote:
Hi, As some of you are aware the comps changes that anaconda wanted changed how comps works in yum which nicely BROKE Fedora Extras rawhide builds once the comps.xml changes landed in rawhide. So what do we do to fix it?
I realize this might be an odd time for me to jump into the discussion, but a few concerns and questions leap to mind. Perhaps some of these stem from my imperfect knowledge of the situation; if so, I apologize.
First, isn't this an odd sort of incompatibility to introduce in yum? Even if this solution is applied in mock, that is no insurance against further incompatible yum changes in other areas.
It seems to me that this problem could be resolved at the repo level. Couldn't plague maintain its own copy of the rawhide repo sans comps?
I guess my points are: 1) not all mock users will be impacted by this issue (I won't, since I generate my own repos). 2) If this change is applied in mock, it will impact mock users unless the new behavior is optional.
Let me throw out another possibility. Add a mock config option named buildcomps containing a url. If this option is present, have mock parse that comps and issue the appropriate yum install commands instead of a groupinstall.
On Wed, 2005-12-14 at 15:10 -0500, Mike McLean wrote:
seth vidal wrote:
As some of you are aware the comps changes that anaconda wanted changed how comps works in yum which nicely BROKE Fedora Extras rawhide builds once the comps.xml changes landed in rawhide. So what do we do to fix it?
I realize this might be an odd time for me to jump into the discussion, but a few concerns and questions leap to mind. Perhaps some of these stem from my imperfect knowledge of the situation; if so, I apologize.
First, isn't this an odd sort of incompatibility to introduce in yum? Even if this solution is applied in mock, that is no insurance against further incompatible yum changes in other areas.
The incompatibility is more in the repo metadata than in yum -- yum 2.4.x (which is what's on the build hosts) doesn't understand the new comps format. And that's partly because the comps side of things has had a lot less thought and effort put into making sure that it covers the needs of the userbase. And I'm going to go out on a limb and say that I expect this _not_ to be the last time it changes... we're actually starting to use the information and so I'm all but certain that we'll find things that we hadn't considered.
It seems to me that this problem could be resolved at the repo level. Couldn't plague maintain its own copy of the rawhide repo sans comps?
It's more than just the devel repo -- it'll then be the fc5 repo and the devel repo. Plus the extras counterparts. Plus everyone on older releases trying to build against fc5/devel using mock.
I guess my points are:
- not all mock users will be impacted by this issue (I won't, since I
generate my own repos).
All mock users will be impacted because you're not guaranteed that the version of yum you have can handle the group information in the repository.
- If this change is applied in mock, it will impact mock users unless
the new behavior is optional.
Let me throw out another possibility. Add a mock config option named buildcomps containing a url. If this option is present, have mock parse that comps and issue the appropriate yum install commands instead of a groupinstall.
Then mock has to grow how to parse a comps file. And which format should be preferred? If we were to go that route, we're better off just putting a list of packages in the mock config file, but that's just painful from a maintenance perspective, IMHO
Jeremy
The incompatibility is more in the repo metadata than in yum -- yum 2.4.x (which is what's on the build hosts) doesn't understand the new comps format. And that's partly because the comps side of things has had a lot less thought and effort put into making sure that it covers the needs of the userbase. And I'm going to go out on a limb and say that I expect this _not_ to be the last time it changes... we're actually starting to use the information and so I'm all but certain that we'll find things that we hadn't considered.
While the new format is in certain ways better for the installer, you only need to change very few lines to make other apps use both the old format and the new format.
I think we should do more testing and put in more thoughts before pushing out changes for the comps.xml file. Especially if now further changes might get done over the next weeks. Shouldn't we then start a completely new file and only then add it to the other tools once its implementation is more complete?
I haven't chekd, but is the comps.xml file part of the repodata with defined content or is it just more "riding along" in the same directory?
Then mock has to grow how to parse a comps file. And which format should be preferred? If we were to go that route, we're better off just putting a list of packages in the mock config file, but that's just painful from a maintenance perspective, IMHO
Or just allow both ways. Doesn't make a too big difference until things get more standardized at some point.
regards,
Florian La Roche
Jeremy Katz wrote:
The incompatibility is more in the repo metadata than in yum -- yum 2.4.x (which is what's on the build hosts) doesn't understand the new comps format. And that's partly because the comps side of things has had a lot less thought and effort put into making sure that it covers the needs of the userbase. And I'm going to go out on a limb and say that I expect this _not_ to be the last time it changes... we're actually starting to use the information and so I'm all but certain that we'll find things that we hadn't considered.
I was thinking more along the lines of the "new yum can't parse old comps" issue. If it could, it seems like we could just migrate to the new yum in the buildsystem.
For that matter, if the old yum could be told to ignore comps on a per-repo basis, we could work around it that way.
Let me throw out another possibility. Add a mock config option named buildcomps containing a url. If this option is present, have mock parse that comps and issue the appropriate yum install commands instead of a groupinstall.
Then mock has to grow how to parse a comps file. And which format should be preferred? If we were to go that route, we're better off just putting a list of packages in the mock config file, but that's just painful from a maintenance perspective, IMHO
Well, there's pain all around. Personally I think having to rebuild packages every time I make a buildgroups change is painful. I also think that being required to have fake packages in my buildroots is annoying.
At least with my suggestion you can point mock at a comps file you are confident it will be able to parse. By making it a url, you can still centralize that portion of the config.
On Wed, 2005-12-14 at 17:40 -0500, Mike McLean wrote:
Jeremy Katz wrote:
The incompatibility is more in the repo metadata than in yum -- yum 2.4.x (which is what's on the build hosts) doesn't understand the new comps format. And that's partly because the comps side of things has had a lot less thought and effort put into making sure that it covers the needs of the userbase. And I'm going to go out on a limb and say that I expect this _not_ to be the last time it changes... we're actually starting to use the information and so I'm all but certain that we'll find things that we hadn't considered.
I was thinking more along the lines of the "new yum can't parse old comps" issue. If it could, it seems like we could just migrate to the new yum in the buildsystem.
Only if the new yum doesn't require pulling in lots of other stuff. And it's far better to just be able to use _any_ yum. Otherwise, eg, people won't be able to build for the devel tree on old releases without upgrading their yum.
For that matter, if the old yum could be told to ignore comps on a per-repo basis, we could work around it that way.
That should work -- set enablegroups=0 in your repo config file.
Let me throw out another possibility. Add a mock config option named buildcomps containing a url. If this option is present, have mock parse that comps and issue the appropriate yum install commands instead of a groupinstall.
Then mock has to grow how to parse a comps file. And which format should be preferred? If we were to go that route, we're better off just putting a list of packages in the mock config file, but that's just painful from a maintenance perspective, IMHO
Well, there's pain all around. Personally I think having to rebuild packages every time I make a buildgroups change is painful. I also think that being required to have fake packages in my buildroots is annoying.
At least with my suggestion you can point mock at a comps file you are confident it will be able to parse. By making it a url, you can still centralize that portion of the config.
mock doesn't do the parsing, though (or any downloading of files at all) -- all of that is in yum. To make mock do the parsing would need a) mock to be able to download files b) mock to be able to parse the comps file
That seems somewhat less than ideal to me
Jeremy
Jeremy Katz wrote:
On Wed, 2005-12-14 at 17:40 -0500, Mike McLean wrote:
For that matter, if the old yum could be told to ignore comps on a per-repo basis, we could work around it that way.
That should work -- set enablegroups=0 in your repo config file.
Is this not a viable solution, then? Mock only needs the groups from the (fairly stable) groups repo. We could set enablegroups=0 for all the other repos and make sure that the comps in the groups repo is safe.
Is this not a viable solution, then? Mock only needs the groups from the (fairly stable) groups repo. We could set enablegroups=0 for all the other repos and make sure that the comps in the groups repo is safe.
safe in what regard? We still need to add a bunch of intelligence to mock, then. It needs to know how to fetch it and parse it and that's just one more file format to deal with.
I do agree with your point of having to build new packages if you change the packages in your buildgroups but one nice thing you get from this is sign-able buildgroup specification. b/c, right now, packages can be signed, but the groups file cannot.
-sv
seth vidal wrote:
Is this not a viable solution, then? Mock only needs the groups from the (fairly stable) groups repo. We could set enablegroups=0 for all the other repos and make sure that the comps in the groups repo is safe.
safe in what regard? We still need to add a bunch of intelligence to mock, then. It needs to know how to fetch it and parse it and that's just one more file format to deal with.
The enablegroups=0 approach is separate from my suggestion of a buildcomps config option. Unless I am misunderstanding (which I admit is possible), you could:
- set enablegroups=0 for all repos except [groups] in the yum config that is embedded in the mock config. - make sure that the buildgroups.xml in the groups repo is understandable by the yum version installed on your mock system
You still have yum parsing the comps, but it only parses one (and that one is easier to control). Since the groups repo is so directly tied to mock, it seems reasonable to expect that buildgroups.xml should match your mock/yum setup.
I do agree with your point of having to build new packages if you change the packages in your buildgroups but one nice thing you get from this is sign-able buildgroup specification. b/c, right now, packages can be signed, but the groups file cannot.
Good points. I'm not 100% opposed to the fake packages, but the idea still kinda bugs me, especially if it becomes the only way.
The enablegroups=0 approach is separate from my suggestion of a buildcomps config option. Unless I am misunderstanding (which I admit is possible), you could:
- set enablegroups=0 for all repos except [groups] in the yum config
that is embedded in the mock config.
- make sure that the buildgroups.xml in the groups repo is
understandable by the yum version installed on your mock system
So then depending on what release the user is running mock and yum on then the buildgroups.xml file they are using will change?
So then we'll need more than one buildgroups repo for misc fedora building based on what distro the user is using? That seems to scale less well.
You still have yum parsing the comps, but it only parses one (and that one is easier to control). Since the groups repo is so directly tied to mock, it seems reasonable to expect that buildgroups.xml should match your mock/yum setup.
yes - but now you have to maintain 2 of these files.
Good points. I'm not 100% opposed to the fake packages, but the idea still kinda bugs me, especially if it becomes the only way.
wrt to rebuilding the package. What if I could give you a script that just took a spec file and spat out the noarch.rpm? Would that make your life easier and less irritated by the package-full-of-deps, thing?
-sv
seth vidal wrote:
You still have yum parsing the comps, but it only parses one (and that one is easier to control). Since the groups repo is so directly tied to mock, it seems reasonable to expect that buildgroups.xml should match your mock/yum setup.
yes - but now you have to maintain 2 of these files.
not if new yum could handle old comps.
wrt to rebuilding the package. What if I could give you a script that just took a spec file and spat out the noarch.rpm? Would that make your life easier and less irritated by the package-full-of-deps, thing?
The rebuilding isn't that big a deal, more of an annoyance. My main worry is the fake packages sitting in the repos. Using fake packages like this has always seemed like a hack to me. Sometimes it is necessary, but I hate to see it used as a central mechanism in the build system.
In my case the fake packages introduce a problem in bookkeeping. I can work around it, of course, but I'd rather avoid the mess.
Anyway, how about a compromise? See attached patch.
On Thu, 2005-12-15 at 15:50 -0500, Mike McLean wrote:
seth vidal wrote:
You still have yum parsing the comps, but it only parses one (and that one is easier to control). Since the groups repo is so directly tied to mock, it seems reasonable to expect that buildgroups.xml should match your mock/yum setup.
yes - but now you have to maintain 2 of these files.
not if new yum could handle old comps.
I do not think we'll be putting the old comps support into yum 2.5.X or later.
That'd be like adding 1.x headers support into yum 2.1 and higher.
in short, it'd be silly.
The whole point of changing the api is to be able to CHANGE it.
The rebuilding isn't that big a deal, more of an annoyance. My main worry is the fake packages sitting in the repos. Using fake packages like this has always seemed like a hack to me. Sometimes it is necessary, but I hate to see it used as a central mechanism in the build system.
Sure - it is a hack. All of building inside a chroot is a hack, too.
What's your point?
In my case the fake packages introduce a problem in bookkeeping. I can work around it, of course, but I'd rather avoid the mess.
what bookkeeping problem do they introduce?
Anyway, how about a compromise? See attached patch.
How is this a compromise?
-sv
seth vidal wrote:
How is this a compromise?
Your proposal was to have mock perform the prep install from a set of specially named fake packages rather than from comps groups. I believe this patch allows that.
My current thinking is that I'd like to keep using the comps groups and just make sure that my yum and my repos and in alignment. This patch allows that.
...or perhaps this is more or less what you had in mind all along, but I go the impression that mock support for using groups was in danger of disappearing.
Your proposal was to have mock perform the prep install from a set of specially named fake packages rather than from comps groups. I believe this patch allows that.
My current thinking is that I'd like to keep using the comps groups and just make sure that my yum and my repos and in alignment. This patch allows that.
two questions: 1. what does giving this option buy us? Other than more code to maintain, I mean. 2. why wouldn't you just go on using the older mock?
...or perhaps this is more or less what you had in mind all along, but I go the impression that mock support for using groups was in danger of disappearing.
very much so.
-sv
Hi,
Then mock has to grow how to parse a comps file. And which format should be preferred? If we were to go that route, we're better off just putting a list of packages in the mock config file, but that's just painful from a maintenance perspective, IMHO
That's what mach does, and I never saw it as a problem. There's no maintenance to be done - you give it a list of packages it needs to have installed for the "build" target and if the rpm dependencies are mapped ok it just works. This is one of the first changes Seth made when forking to mock IIRC.
Thomas
Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> her warm white belly in the life I'd lived had seen nothing finer <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/
On Thu, 2006-01-05 at 16:06 +0100, Thomas Vander Stichele wrote:
Hi,
Then mock has to grow how to parse a comps file. And which format should be preferred? If we were to go that route, we're better off just putting a list of packages in the mock config file, but that's just painful from a maintenance perspective, IMHO
That's what mach does, and I never saw it as a problem. There's no maintenance to be done - you give it a list of packages it needs to have installed for the "build" target and if the rpm dependencies are mapped ok it just works. This is one of the first changes Seth made when forking to mock IIRC.
B/c we didn't want the mock config to hold the list of packages in an internal config file on every machine. If we did we wouldn't be able to reasonably know that all the users of it were building on the same chroots for their own test builds.
-sv
buildsys@lists.fedoraproject.org
-
Florian La Roche -
Jeff Sheltren -
Jeremy Katz -
Mike McLean -
seth vidal -
seth vidal -
Thomas Vander Stichele -
Warren Togami