Arjan van de Ven wrote:

On Thu, 2005-01-27 at 09:28 +0100, Christof Damian wrote:

...

I just read about Red Hat ABE (Application Build Environment) which seems to be something similar to mach.

Hmmm, nice stuff.

Finally, someone has packaged up a KISSy chroot based build system, very cool.

Watch out for:

* rpm-4.1.1 won't create multilib packaging correctly on AS2.1. You ought to lose rpm-4.1.1 at your earliest opportunity. Move to rpm-4.2.2-0.8 (at least), and with external beecrypt and elfutils, and you should be fine.

* --aid is as good or bad as the rpmdb used. I don't yet see tools to manage rpmdb headers incrementally, and --justdb from the rpm CLI is a blunderbuss. Much better could be done. In fact has been done ...

* perl sux equally as much as python ;-)

the goals are very similar to mach, but mach uses apt which made it not suitable as basis for the ABE

I suggested to mach developers quite some time ago that --aid was more than enough to populate an empty chroot from rpm packages. Thanks for the manifest proof that, indeed, --aid is sufficient.

...

Will this be available or work on fedora? Has anyone tried it ?

it'll work for fc3 for sure; I tested that extensively ;) Older fedoras might need some minor tweaks

Aside from the issues above, I dunno of any rpm implementation problems that would prevent rhel-abe from being used for any/all of FCn, but there's a great deal of churn-and-burn packaging and rpm details that need to be accomodated. Probably that's what you mean by "tweaks".

There are later versions of rpm built for FC1/FC2/FC3 at ftp://ftp.rpm.org/pub/rpm/test that will never ever be released "officially" (due to lack of interest), but those packages might be useful as starting points for adjusting some of the "tweaks" between fc1/fc2/fc3/fc4.

Latest possible common version of rpm for all of fc1/fc2/fc3/fc4 would only help eliminate "tweaks".

Disclaimer: I dunno where rhel-abe came from, and I dunno where rhel-abe is going, and I've never used rhel-abe, or knew rhel-abe existed, until an hour ago.

So my comments are from examining the srpm, nothing more.

Good luck!

73 de Jeff

Hi,

• perl sux equally as much as python ;-)

Maybe even more :)

...

the goals are very similar to mach, but mach uses apt which made it not suitable as basis for the ABE

I suggested to mach developers quite some time ago that --aid was more than enough to populate an empty chroot from rpm packages. Thanks for the manifest proof that, indeed, --aid is sufficient.

If all you want to do is populate an empty chroot then it is. For a system like this you need more. There's no good way to resolve dependencies using only rpm. And when you are trying to do more than just "build against this known set of packages I have locally on my hard disk" it's insufficient.

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> when he died all he left us was alone <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

Hi Arjan,

...

I just read about Red Hat ABE (Application Build Environment) which seems to be something similar to mach.

the goals are very similar to mach, but mach uses apt which made it not suitable as basis for the ABE

I have to bite here :)

Given that a) mach has been used by some people over some time; b) people working on mach have repeatedly tried to discuss with Red Hat and said "hey, we'd like to work with you on a build system to be used by all", as part of the "community" project that is Fedora; but no attempts were made from Red Hat to unify forces; c) nothing more was ever offered as negative feedback on mach than "it uses apt" (a fact that is easily changeable, obviously); d) you could easily have asked "hey, can't mach be made to not use apt, but do (insert random feature you would like)"

why did the "NIH" syndrome that Red Hat sometimes displays wins out over the desire to involve the community in the "community" project ?

I am behind Red Hat and Fedora 100% of the way, against the flames of friends who really do not understand why this Fedora thing is all talk and no action. Things like this are just one of the many things symptomatic of the fact that Red Hat seems to want us to believe there's a community to be involved in, when in fact there is no such thing.

I realize that you probably don't care, and that you have a job to do, and it's already hard enough as it is, and sometimes it's just easier to Do Your Own Thing to Get The Job Done. And this is very much not a personal flame at you, just a flame at Red Hat in general.

To put it ghetto-style - when is RH going to stop talking the talk and start walking the walk ?

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> Say what you mean or you won't mean a thing to me <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

Thomas Vander Stichele wrote:

Hi Arjan,

...

...

I just read about Red Hat ABE (Application Build Environment) which seems to be something similar to mach.

the goals are very similar to mach, but mach uses apt which made it not suitable as basis for the ABE

I have to bite here :)

Given that a) mach has been used by some people over some time;

True for beehive as well.

b) people working on mach have repeatedly tried to discuss with Red Hat and said "hey, we'd like to work with you on a build system to be used by all", as part of the "community" project that is Fedora; but no attempts were made from Red Hat to unify forces;

Well Red Hat ain't exactly got a mouth, or more specifically, one mouth.

I spent several months attempting to unify forces, and I work at Red Hat. That is not "No attempt."

And I'm sure there are other, less feeble, attempts from Red Hat to unify than my efforts.

c) nothing more was ever offered as negative feedback on mach than "it uses apt" (a fact that is easily changeable, obviously);

Well clearly, having mach rely on a package that is not included in any Red Hat product, presents certain logistical difficulties. That should be obvious.

d) you could easily have asked "hey, can't mach be made to not use apt, but do (insert random feature you would like)"

I have said repeatedly that mach needs to lose the subliminal messages in the progress bars, even if they're s-o-o-o-o cute! ;-)

why did the "NIH" syndrome that Red Hat sometimes displays wins out over the desire to involve the community in the "community" project ?

Assimilating a 3rd -- nay 4th or 5th if I count [rs]-c-p and rpm -- dependency solver into the distro (i.e. apt) in order to accdomodate a poor design choice in mach makes little sense to me.

And, FWIW, I have suggested repeatedly that apt be added to FC internally to Red Hat in spite of the cost of attempting to maintain Yet Another Depsolver. The previous line basically summarizes the majority of the feedback that I have heard:

FC needs fewer depsolvers that work more reliably.

I am behind Red Hat and Fedora 100% of the way, against the flames of friends who really do not understand why this Fedora thing is all talk and no action. Things like this are just one of the many things symptomatic of the fact that Red Hat seems to want us to believe there's a community to be involved in, when in fact there is no such thing.

Heh, a casual reading seems to indicate that there is no community for Red Hat to be involved with, perhaps not what you intended. ;-)

I realize that you probably don't care, and that you have a job to do, and it's already hard enough as it is, and sometimes it's just easier to Do Your Own Thing to Get The Job Done. And this is very much not a personal flame at you, just a flame at Red Hat in general.

So flame away. Do you have any idea what it is like being bathed in flame for months and years, simply because you happen to work for Red Hat?

To put it ghetto-style - when is RH going to stop talking the talk and start walking the walk ?

When your momma takes off her combat boots, and not before. ;-)

73 de Jeff

seth vidal wrote:

Do you have any idea what it's like being bathed in flames for months and years at a time b/c you're trying to VOLUNTEER to help out an organization that appears to actively wish to deter your help? At least you get paid to get flamed. I get flamed and laughed at for free.

I warned you when I added yum to FC:

You are so so doomed!

And thanks for shouldering much of the depsolver diagnosis load with yum support. That is personally appreciated.

so pull the other one, its got bells on.

So you want to get paid to be flamed?

I'm sure that can be arranged if/when you wish.

Me, I want peace and quiet for a change, there's work to do.

73 de Jeff

On Thu, 2005-01-27 at 08:16 -0500, Jeff Johnson wrote:

Thomas Vander Stichele wrote:

...

c) nothing more was ever offered as negative feedback on mach than "it uses apt" (a fact that is easily changeable, obviously);

Well clearly, having mach rely on a package that is not included in any Red Hat product,

RH has the ability to change this at any time.

presents certain logistical difficulties. That should be obvious.

It is not - RH has had no problems in adding yum support and has no problem in adding and removing other packages at any time at RH's free will.

For example instead of adding yum and keeping up2date, RH could have tried to help apt. - IMO, this is all politics and not at all technically motivated.

And, FWIW, I have suggested repeatedly that apt be added to FC internally to Red Hat in spite of the cost of attempting to maintain Yet Another Depsolver. The previous line basically summarizes the majority of the feedback that I have heard:

FC needs fewer depsolvers that work more reliably.

Isn't the solution obvious? Implement one unified depsolver into rpm which behaves exactly as you envision it, instead.

Ralf

On Thu, 2005-01-27 at 08:52 -0500, seth vidal wrote:

...

RH has the ability to change this at any time.

ability? yes. willingness? no.

...

It is not - RH has had no problems in adding yum support and has no problem in adding and removing other packages at any time at RH's free will.

Do you know why they had no issue adding yum support? B/c it could be covered internally. If it broke and I wasn't around to fix it - they could take care of it.

100+ lines of C++ they were not interested in maintaining.

That should be 100K+ lines of C++

100 lines of c++ will just barely get you hello world :)

-sv

Ralf Corsepius wrote:

On Thu, 2005-01-27 at 08:52 -0500, seth vidal wrote:

...

...

RH has the ability to change this at any time.

ability? yes. willingness? no.

...

It is not - RH has had no problems in adding yum support and has no problem in adding and removing other packages at any time at RH's free will.

Do you know why they had no issue adding yum support? B/c it could be covered internally. If it broke and I wasn't around to fix it - they could take care of it.

100+ lines of C++ they were not interested in maintaining.

How comes, FE/fedora.us is able to maintain it?

I know apt's code is ... ... leaves a lot to be desired, but it doesn't require that much effort to maintain the package.

Also not true. The guy who maintained apt-rpm chose to write smartpm instead.

That sez' a whole lot about the maintainability of the apt code base. There are many known legacy issues with C++ as well, can't be helped, I'm certainly not complaining.

Or perhaps a whole lot about the politics of package management and vendors.

One never knows, and one cannot tell. <shrug>

...

...

For example instead of adding yum and keeping up2date, RH could have tried to help apt. - IMO, this is all politics and not at all technically motivated.

IMO you don't know what you're talking about.

I guess, I do ... I spent way too much time with rpmlib and apt.

Tried smartpm? Best damn depsolver that I've ever seen, does all the (imho) useful stuff that apt does (and yum/up2date do not, at least not yet, like back-tracking), without the C++ baggage and the Debian Borg politics.

But, by all means, if *you* like apt, then *you* should use apt. Use what works.

73 de Jeff

I know. If you're so convinced about smartrpm, why don't you include it into FC and consider to abandon up2date and yum, rsp. to adopt smartrpm's resolver into rpm, rsp. to change yum to use that?

1. b/c it's not just his opinion that counts. 2. b/c rh does not control yum development 3. b/c smartrpm, while a nice depsolver, seems to add A LOT of complexity for some highly limited gain to fedora.

Another feature I am missing in both apt and yum is a usable --download-only operation. apt-get has "-d" but insists on its "package name mangling", (FC3's) yum doesn't support it all, I don't know about smartrpm.

it's funny, you seem capable of programming except you've not contributed any patches, at all, to yum. Why not add in what you're looking for? It's not like parsing the metadata to do download-only is terribly difficult.

-sv

Ralf Corsepius wrote:

On Thu, 2005-01-27 at 09:36 -0500, Jeff Johnson wrote:

...

Ralf Corsepius wrote:

...

...

How comes, FE/fedora.us is able to maintain it?

I know apt's code is ... ... leaves a lot to be desired, but it doesn't require that much effort to maintain the package.

Also not true. The guy who maintained apt-rpm chose to write smartpm instead.

I know. If you're so convinced about smartrpm, why don't you include it into FC and consider to abandon up2date and yum, rsp. to adopt smartrpm's resolver into rpm, rsp. to change yum to use that?

What do you think I am, the FC God or something? I do think smartpm is a better biscuit, and I have told all the appropriate people.

What gets included in FC is an arduous and complicated negotiation, and is usually quite controversial, both with Red Hat and within Fedora.

I trust that the right decision will be made in the end.

...

Tried smartpm?

Yes, I tried it for a few hours, a couple of days ago.

Good.

As I already wrote some days ago, I am not (yet) convinced, at least I could not get familiar with it - Too much black magic involved.

May-be I should give it another try and dig a little deeper.

It's early yet for smartpm, yes. Use what works.

...

Best damn depsolver that I've ever seen, does all the (imho) useful stuff that apt does (and yum/up2date do not, at least not yet, like back-tracking),

Does smartrpm have equivalents to

apt-get source apt-get build-dep

These are the features I like about apt and which make apt interesting to me.

Another feature I am missing in both apt and yum is a usable --download-only operation. apt-get has "-d" but insists on its "package name mangling", (FC3's) yum doesn't support it all, I don't know about smartrpm.

No clue, I (blush) do not use apt, and I'm having a good day when the random rpmdb on my box happens to be sufficiently functional and populated that yum does not get too confused.

I'm pretty sure Gustavo would be willing to implement new and useful features in smartpm however.

73 de Jeff

Florian La Roche wrote:

One of the very good items about yum is that it is actually using the rpmlib depsolver.

Ditto wrto smartpm.

And up2date, and [rs]-c-p and apt and beehive and rhel-abe, for various definitions of "use" rpmlib, for completeness.

up2date is perhaps the most techically close to "using" rpmlib, not that that is any significant or serious advantage, rpmlib is not necessarily the best depsolver in the world.

Nor is it very hard to write a goal-driven loop, traversing edges in a dependency graph endlessly, which is all that rpmlib does.

What is needed is better policy, not universal "use", for choosing packages to install, from repos, from "cloning" from other machines, etc, as policy is better able to meet user expectations of "works", in a way that stable known mechanism will never be able to achieve.

rpmlib (and everything that uses) is quite stupid about choosing between multiple provides, for one obvious example. Multilib, and kernel's, are other known deficiencies in rp[mlib mechanism (or, if you will, the lack of better policy mechanisms for depsolvers).

But depsolver policy is a much more complicated topic than using rpmlib ... 73 de Jeff

seth vidal wrote:

...

Ditto wrto smartpm.

/me looks at the smartpm code.

it doesn't seem like it's using the rpm depsolver overly much. It's using it for the transaction - but it's solving the deps all by itself.

Yep. smartpm has a different, and more elegant, dependency graph and a better depsolver.

Nodes in a smartpm dependency graph are packages+operation, not just package.

That permits relations (in the topological sort sense) to be explicitly written between identically named nodes, unlike what rpmlib currently does, which is handle install/upgrade/erase operations as a function on the dependency graph.

So in a very real sense, smartpm has a better depsolver than rpmlib already. One noticeable deficiency in rpmlib is that erased packages are not topologically sorted, basically because I could not figure how to code the function that traversed the dependency graph with "package" nodes.

The solution to ordering erasures and upgrades is obvious (to me anyways) when the nodes are labeled with "package+operation", and so can be ordered more reliably and more correctly.

But yes, I can and will teach rpmlib the same trick. Coding with nodes and edges ain't hard at all, what takes thought is the concepts, which smartpm (and Gustavo) does better than rpmlib atm.

...

rpmlib (and everything that uses) is quite stupid about choosing between multiple provides, for one obvious example. Multilib, and kernel's, are other known deficiencies in rp[mlib mechanism (or, if you will, the lack of better policy mechanisms for depsolvers).

yep - which is why I'm looking forward to using the new check objects that paul was talking about - to better identify the requiring package.

"check objects" is a mystery to me, just as much of a mystery as rhel-abe was this morning.

73 de Jeff

-sv

How comes, FE/fedora.us is able to maintain it?

they don't. No one is patching apt. It's not being maintained even upstream. How far do you expect it to get?

I know apt's code is ... ... leaves a lot to be desired, but it doesn't require that much effort to maintain the package.

maintaining a package != maintaining the program.,

...

IMO you don't know what you're talking about.

I guess, I do ... I spent way too much time with rpmlib and apt.

really, then you know apt doesn't work, at all, with multilib.

You gonna fix that?

-sv

On Thu, 27 Jan 2005, Jeff Johnson wrote:

seth vidal wrote:

...

really, then you know apt doesn't work, at all, with multilib.

You gonna fix that?

I am, one of these days, probably with help from either Gustavo or Panu.

I wish you luck :) If you *really* want to mess with apt internals I do have ideas how it could be done (some of which I've tried but got tangled in the implementation details and dropped due to lack of time and interest). But really I think time would be better spent improving yum/smartpm to have the equivalent of build-dep, source and such operations.

- Panu -

On Friday 28 January 2005 18:22, Dag Wieers wrote:

On Fri, 28 Jan 2005, Panu Matilainen wrote:

...

But really I think time would be better spent improving yum/smartpm to have the equivalent of build-dep, source and such operations.

And make Smart work on older versions of python and with older rpmlib versions, at least RHEL3. RHN support is also on my wishlist.

Two things: CVS/SVN and Mailing List. Otherwise, we have no venue to contribute patches.

thanks,

On Thu, 27 Jan 2005, Ralf Corsepius wrote:

On Thu, 2005-01-27 at 08:52 -0500, seth vidal wrote:

...

...

RH has the ability to change this at any time.

ability? yes. willingness? no.

...

It is not - RH has had no problems in adding yum support and has no problem in adding and removing other packages at any time at RH's free will.

Do you know why they had no issue adding yum support? B/c it could be covered internally. If it broke and I wasn't around to fix it - they could take care of it.

100+ lines of C++ they were not interested in maintaining.

How comes, FE/fedora.us is able to maintain it?

Fedora.us has/had an upstream apt-rpm developer (some weird masochist sharing my mail-address :) maintaining it and writing all sorts of weird Lua-extensions to it to better fit the world of FC, external kernel-module packages and such.

I know apt's code is ... ... leaves a lot to be desired, but it doesn't require that much effort to maintain the package.

Maintaining the package ain't hard, but developing apt-rpm into various directions required by FC (multilib, new repodata format) is just about as fun as pulling teeth without anesthesia. The new repodata is something that would be sanely implementable into apt, multilib as used in FC and RHEL (namely packages with same nevr but different arch simultaenously installed) is something that doesn't fit nicely into it's design. And that's putting it somewhat mildly. I've actually tried various approaches to adding multilib support to apt with varying success, however none of work well enough to be actually usable.

- Panu -

On Fri, 28 Jan 2005, Ralf Corsepius wrote:

On Fri, 2005-01-28 at 11:58 +0200, Panu Matilainen wrote:

...

new repodata format)

IMO, this is more a matter of politics and willingness, but a technical requirement.

Technically, I don't see any need for apt to adopt yum's repodata format. Politically, this requirement is introduced by RH not wanting to add apt-repositories and fedora.us apparently being unable to set up complete repositories. If apt-repositories are cleverly set up, the additional overhead they introduce in addition to the original files becomes more or less negligible.

Sure, it can be viewed as politics/willingness or lack of thereof. I personally see the lack of repodata support in apt as an example of the standstill in apt's development which in turn hints that maybe it's time to move forward to something else :)

BTW: Even SuSE is available with apt. I wonder why they don't have the multilib issue - I guess they don't ship multilibs :)

Apt has zero problems with mixing 32bit and 64bit packages IF the packages have different names. Suse packages their 32bit stuff for x86_64 with something like libfoo32bit names which circumvents the whole problem. Apt just can't sanely support packages having same name but different arch being simultaneously installed (+ a bunch of other misc related details)

- Panu -

On Fri, 2005-01-28 at 12:30 +0100, Michael Schwendt wrote:

On Fri, 28 Jan 2005 11:34:09 +0100, Ralf Corsepius wrote:

...

Technically, I don't see any need for apt to adopt yum's repodata format. Politically, this requirement is introduced by RH not wanting to add apt-repositories and fedora.us apparently being unable to set up complete repositories.

Unless I'm misinformed, fedora.us even provides an apt-repository for pre-extras. What do you mean with "unable to set up complete repositories"?

SRPMS apt-repositories are missing for pre-extras.

This renders "apt-get source" and "apt-get build-dep" non-applicable to fedora.us hosted apt-repositories and therefore voids at least these aspects where apt is superior to yum.

I had asked Warren Togami to add them on PM and he answered: "There is little good reason to do so. Trying to limit the size of that repository because many mirror administrators see it as redundant."

This is not true, SRPMS apt-repositories are not redundant. Not having them implies loss of functionality to apt.

As Fedora.US had supplied SRPMS apt-repositories for FC < 3, omitting them for pre-extras also means a regression in functionality of fedora.us having been introduced with the switch from FE2 to pre- extras/FE3.

Ralf

On Fri, 2005-01-28 at 04:38 -0800, Rahul Sundaram wrote:

there is a reason its called *pre* extras

Yes, identifying and fixing defects like this.

Ralf

On Fri, 2005-01-28 at 13:47 +0100, Michael Schwendt wrote:

On Fri, 28 Jan 2005 13:31:01 +0100, Ralf Corsepius wrote:

...

I had asked Warren Togami to add them on PM and he answered: "There is little good reason to do so. Trying to limit the size of that repository because many mirror administrators see it as redundant."

This is not true, SRPMS apt-repositories are not redundant. Not having them implies loss of functionality to apt.

Well, then I think the Apt community may need to prove him wrong and do some lobbying.

FWIW, I would like to see the complete apt'table (pre-)Extras SRPMS repos available at fedora.us and its mirrors. The reasons have been outlined in this thread.

On Mon, 31 Jan 2005, Warren Togami wrote:

Ville Skyttä wrote:

...

FWIW, I would like to see the complete apt'table (pre-)Extras SRPMS repos available at fedora.us and its mirrors. The reasons have been outlined in this thread.

I have to admit that I don't understand if there is any purpose in this. "apt-get source" is only a convenience but otherwise is not very useful.

Well, 'apt-get install foobar' is only a convenience as well, you could just as well manually locate latest version of, download and install with rpm. I find it a major pain in the ass to manually locate the latest version of a given src.rpm when apt (or any similar tool) can do it automatically for you.

"apt-get build-dep" does not need SRPMS at all except the local SRPM, which you already have locally because you want to build it. Is there some aspect of this that I am failing to see? I fail to see the "need" in this.

You have to manually locate and download the SRPM first, and then use # apt-get build-dep /path/to/srpms/foobar-1.2-1.src.rpm compared to just # apt-get build-dep foobar

It IS a much more convenient for us who deal with SRPMS a lot. Because the lack of the SRPMS I keep a local apt-enabled mirror of FC to be able to access the SRPMS without all the manual work, which feels somewhat ridiculous to me.

From the perspective of mirror administrators it is somewhat painful to host redundant RPMS. I personally don't have the disk space to do this forever with all distributions.

I suppose it is tolerable to do only SRPMS.extras of the latest stable distribution. When FC4 happens, I will wipe the 3 SRPMS.extras. Is this acceptable? I will however not add the base and updates SRPMS.

That'd be a definite improvement over not having the SRPMS at all.

- Panu -

Ralf Corsepius wrote:

SRPMS apt-repositories are missing for pre-extras.

This renders "apt-get source" and "apt-get build-dep" non-applicable to fedora.us hosted apt-repositories and therefore voids at least these aspects where apt is superior to yum.

I had asked Warren Togami to add them on PM and he answered: "There is little good reason to do so. Trying to limit the size of that repository because many mirror administrators see it as redundant."

This is not true, SRPMS apt-repositories are not redundant. Not having them implies loss of functionality to apt.

And having them makes it a _lot_ easier to grab a package, fix something, build it, test it, then post a bug report w/patch.

SRPMS repositories _are_ important.

Eli --------------------. "If it ain't broke now, Eli Carter \ it will be soon." -- crypto-gram eli.carter(a)inet.com `-------------------------------------------------

------------------------------------------------------------------------ Confidentiality Notice: This e-mail transmission may contain confidential and/or privileged information that is intended only for the individual or entity named in the e-mail address. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or reliance upon the contents of this e-mail message is strictly prohibited. If you have received this e-mail transmission in error, please reply to the sender, so that proper delivery can be arranged, and please delete the message from your computer. Thank you. Tektronix Texas, LLC formerly Inet Technologies, Inc. ------------------------------------------------------------------------

On Mon, 2005-01-31 at 23:09 +0100, Enrico Scholz wrote:

rc040203@freenet.de (Ralf Corsepius) writes:

...

This renders "apt-get source" and "apt-get build-dep" non-applicable to fedora.us hosted apt-repositories and therefore voids at least these aspects where apt is superior to yum.

Although I love apt for its powerful configuration, I have to admit that 'apt-get build-dep' is not very usefully in the rpm-world:

• it detects only the requirements which were used to build the src.rpm, but not the deps which will be required

True, but ???

apt-get build-deps is useful for chasing bugs in rpms, e.g. when having notices something suspicious when using a binary rpm and trying to fix the cause.

Typical scenario: 1. Use a binary rpm, notice a problem. 2. su; apt-get build-dep <package> 3. apt-get source <package> 4. rpm -U <package>.src.rpm 5. edit <package>.spec|<package> 6. rpmbuild -ba <package>

Edit and rebuild the package. As run-time deps of the binary rpm and build-time deps of the package do not necessarily have to match, this is very convenient.

• extending functionality is a non-trivial task, a the dep-resolving should happen as non-root while the final package removal/installation must happen as root

Also true, but ... using a simple chroot is sufficient for the case above.

Ralf

On Tue, 2005-02-01 at 02:07 +0100, Enrico Scholz wrote:

rc040203@freenet.de (Ralf Corsepius) writes:

...

...

...

This renders "apt-get source" and "apt-get build-dep" non-applicable to fedora.us hosted apt-repositories and therefore voids at least these aspects where apt is superior to yum.

Although I love apt for its powerful configuration, I have to admit that 'apt-get build-dep' is not very usefully in the rpm-world:

• it detects only the requirements which were used to build the src.rpm, but not the deps which will be required

True, but ???

Ok, an example:

[conditionals in RPM specs]

Well, IMO, any rpm which is part of a distribution should be rebuildable by any arbitrary user/semi-newbie using rpmbuild --rebuild ... without having to provide any specific flags.

I.e. I consider rpm specs to be broken, which require special conditionals to match those setting having been used by the distributor.

Ralf

On Tue, 2005-02-01 at 09:58 +0100, Enrico Scholz wrote:

I just said that BuildRequires: must not be determined by the header of the src.rpm package.

"Must not" is a bit strong expression, the correctness of the build deps retrieved that way depends on what one wants to achieve. The SRPM header contains the build dependencies resolved one way or the another at the time the SRPM was built, that's it.

Depending on the use case, that may or may not meet the requirements. For the vast majority of packages (most do not do any conditionals/scripting mentioned earlier in this thread) the SRPM header info is "correct" in the same-as-in-the-specfile sense.

Granted, that's not enough for eg. build systems having certain requirements. But that doesn't make the data completely useless, one just has to be aware of the limitations.

On Fri, 2005-01-28 at 13:28 +0100, Dag Wieers wrote:

On Fri, 28 Jan 2005, Ralf Corsepius wrote:

...

...

new repodata format)

IMO, this is more a matter of politics and willingness, but a technical requirement.

Technically, I don't see any need for apt to adopt yum's repodata format. Politically, this requirement is introduced by RH not wanting to add apt-repositories and fedora.us apparently being unable to set up complete repositories. If apt-repositories are cleverly set up, the additional overhead they introduce in addition to the original files becomes more or less negligible.

Well, generating repositories demands a lot of a system. Generating 3 different repositories (apt, old-yum, new-yum) is really becoming an issue and prevents me from releasing updates flexibly. If I have to wait 30 to 45 minutes before I can starting syncing with a mirror, I delay that sometimes until the next time I'm online which could be 24h later.

Try aptate from apt4rpm (http://apt4rpm.sourceforge.net - Try the version from CVS-head, the released tarballs are troublesome) and feel free to ask me on PM in case of problems :-)

You have to know that I refuse to work with passphrase-less keys, so signing and syncing is an interactive step and requires my attention.

...

BTW: Even SuSE is available with apt. I wonder why they don't have the multilib issue - I guess they don't ship multilibs :)

BTW: The repositories at ftp://ftp.gwdg.de/pub/linux/suse/apt/SuSE (semi official SuSE apt repositories) are built by aptate/apt4rpm.

Ralf

On Fri, 2005-01-28 at 16:12 +0100, Dag Wieers wrote:

On Fri, 28 Jan 2005, Ralf Corsepius wrote:

...

On Fri, 2005-01-28 at 13:28 +0100, Dag Wieers wrote:

...

On Fri, 28 Jan 2005, Ralf Corsepius wrote:

...

Technically, I don't see any need for apt to adopt yum's repodata format. Politically, this requirement is introduced by RH not wanting to add apt-repositories and fedora.us apparently being unable to set up complete repositories. If apt-repositories are cleverly set up, the additional overhead they introduce in addition to the original files becomes more or less negligible.

Well, generating repositories demands a lot of a system. Generating 3 different repositories (apt, old-yum, new-yum) is really becoming an issue and prevents me from releasing updates flexibly. If I have to wait 30 to 45 minutes before I can starting syncing with a mirror, I delay that sometimes until the next time I'm online which could be 24h later.

Try aptate from apt4rpm (http://apt4rpm.sourceforge.net - Try the version from CVS-head, the released tarballs are troublesome) and feel free to ask me on PM in case of problems :-)

Afaics, aptate just calls genbasedir, yum-arch or createrepo. Each of them largely doing the same thing and reading 14GB worth of packages.

Not quite. aptate uses a cache to check for which rpms have been added to a repository since the last time it has been run, and then incrementally builds the apt-repositories.

Give it a try, initially building the repositories (setting up the cache) takes a lot of time, but updating repositories, after a couple of packages have been added to it is fast.

As far as yum is concerned, you are right. yum repositories are re-built instead of being incrementally built, however even they are only built if a package had been added or removed from the repository.

I am using it to build local apt repositories of external ftp sites, which do not provide apt repositories.

BTW: the GWDG site is several x 10GB in size. They apply a similar procedure as I do. Nightly cron jobs rsync'ing from extern, and running aptate in the end to add the changes.

Ralf

On Fri, Jan 28, 2005 at 11:58:00AM +0200, Panu Matilainen wrote:

On Thu, 27 Jan 2005, Ralf Corsepius wrote:

...

I know apt's code is ... ... leaves a lot to be desired, but it doesn't require that much effort to maintain the package.

Maintaining the package ain't hard, but developing apt-rpm into various directions required by FC (multilib, new repodata format) is just about as fun as pulling teeth without anesthesia.

The fun depends on whether you are the patient or the doctor ;)

The new repodata is something that would be sanely implementable into apt,

I think that's the least that needs to be done. As you say it is easily fixable, and also until then it can be taken care of server-side (where the question arises, what does the new repodata format really buy us other than being xml? I was under the impression that all depsolvers, rpm and deb and its cat were going to use it, turns out it's yum and up2date only).

multilib as used in FC and RHEL (namely packages with same nevr but different arch simultaenously installed) is something that doesn't fit nicely into it's design. And that's putting it somewhat mildly. I've actually tried various approaches to adding multilib support to apt with varying success, however none of work well enough to be actually usable.

There is a simple patch by the CERN folks used for ia64 by spliting apt's processing of i386 and ia64 into different package worlds (in apt-rpm's archives). Can that be used as for an initial multilib approach?

apt's lack of multilib is a PITA, but you must also consider that apt's development community has only one member coming from the multilib world, the masochist sharing Panu's mail address ... ;)

On Fri, 28 Jan 2005, seth vidal wrote:

...

I think that's the least that needs to be done. As you say it is easily fixable, and also until then it can be taken care of server-side (where the question arises, what does the new repodata format really buy us other than being xml? I was under the impression that all depsolvers, rpm and deb and its cat were going to use it, turns out it's yum and up2date only).

Do you think that's the way I wanted it? no.The point of the metadata format was to remove the duplicate implementations of the same data. But sometimes you end up that not everyone wants to do any work to implement the functionality in their program. What more can be done to convince them to do that?

So it's not a failing of the format, not as far as I've been told.

Only apt is not using the new metadata format, so all-in-all it has been very succesful. My only concern is that older distributions have been ignored. (yum 2.0, apt and up2date)

And even when apt is fixed, I still need to carry old-yum style metadata support (even though yum-arch is complaining and failing to understand that it is *NOT* obsolete) as long as we have yum 2.0 and up2date around in its current form on RHEL, RH, FC1 and FC2. (everything except FC3 :))

Fixing createrepo to provide old-yum metadata would be an acceptable workaround from the repository maintainer POV. Trying to get rid of repository maintainers is an alternative too :)

-- dag wieers, dag@wieers.com, http://dag.wieers.com/ -- [all I want is a warm bed and a kind word and unlimited power]

Almost got me. Fixing for old-yum would be bonus. Though, I think the true bonus would be incremental update. Still thinking about it though. Mass --resign is a burden. Rerun of metadata is a burden. If there could be a MANIFEST from build to sign to metadata to publish incrementally operating on stuff surrounding the list, then managing thousands of packages becomes less of a burden.

incremental update may not actually be any faster.

Think about it - how would you make sure the incremental update was right?

You'd have to: 1. read in all the xml 2. open and checksum(sha1, md5)

compare filenames + checksums to what you just obtained add/delete/modify as necessary write back out ALL of the xml.

so tell me again how much saving you're going to get?

-sv

The exercise is to attempt a method in which you save computation of md5 or sha1, as these are one of the time consuming steps of createrepo. The save would be in a 100k package repository: (100,000 - N) * Time(sum_calc), where N equals the number of packages that *need* to generate sums for. A parameterized list of package names passed into createrepo would be sufficient to figure out what composes the N list. An external process, such as a Manifest list, would then be used to mitigate a set of packages through the entire build process. Apt uses a md5sum cache, but having fine-tuned controlled of the process would be more stable and directed. This is how much saving you'd get for #2.

Let me know when you've figured it out but as it stands I don't think incrementally updating the metadata is very feasible.

-sv

On Jan 29, 2005, Axel Thimm Axel.Thimm@ATrpms.net wrote:

How about having multiple repodatas, the base one and small incremental ones, the incremental ones containing also package cancelations? As a side effect this would also reduce download bandwidth and thus make even clients/users happy (not only repo maintainers).

+1. Heck, make it +2.

I agree with Axel, for a change :-)

On Sat, Jan 29, 2005 at 02:23:25PM -0500, seth vidal wrote:

On Sat, 2005-01-29 at 17:11 -0200, Alexandre Oliva wrote:

...

On Jan 29, 2005, Axel Thimm Axel.Thimm@ATrpms.net wrote:

...

How about having multiple repodatas, the base one and small incremental ones, the incremental ones containing also package cancelations? As a side effect this would also reduce download bandwidth and thus make even clients/users happy (not only repo maintainers).

+1. Heck, make it +2.

I agree with Axel, for a change :-)

How would it reduce bandwidth - you'd have to download and parse multiple entries and you'd STILL have to do just a much work on the repo-side b/c you'd have to check all the packages for changes.

For N packages the ballanced load are log_2 N bins. Adding M packages touches only log_2 M bins. And the bins have a max size of 2^i packages where i goes from 0 to N-1. And the good news is you touch the bins with i < M, e.g. the small ones.

The statistical net effect is that for M package additions to arbitrary N you get log_2 M downloads of a total of 2M packages.

In relevant numbers:

o N~=4000, log_2 N~=12 You have 12 bins. o 10 security/bug fix updates, (statistically) only bins 0 to 4 are changed amounting to 32 packages. Clients download only 5 files worth of 32 packages in size.

Compare with the current situation, where you need to get the whole lot of N packages for each update.

For this to work you need to

o introduce package cancelation (anti-packages ;) o introduce multiple repodata components o keep a manifest of the last state and feed the repo creation system with the differences (packages lost, packages gained).

It's rather common and very efficient in high performance statistics of large sums.

On Sat, Jan 29, 2005 at 05:07:00PM -0500, seth vidal wrote:

...

For N packages the ballanced load are log_2 N bins. Adding M packages touches only log_2 M bins. And the bins have a max size of 2^i packages where i goes from 0 to N-1. And the good news is you touch the bins with i < M, e.g. the small ones.

The statistical net effect is that for M package additions to arbitrary N you get log_2 M downloads of a total of 2M packages.

In relevant numbers:

o N~=4000, log_2 N~=12 You have 12 bins. o 10 security/bug fix updates, (statistically) only bins 0 to 4 are changed amounting to 32 packages. Clients download only 5 files worth of 32 packages in size.

Compare with the current situation, where you need to get the whole lot of N packages for each update.

For this to work you need to

let's be clear - for this to work YOU need to. [...] But far be it from to halt the steady march of progress - when you get a chance to implement this stuff let me know.

Hey Seth, relax. This is just a suggested concept for improving things. Someone may pick it up, I didn't enforce it on YOU. ;)

Oh and once more - who is it gets the benefit from all this work? It sounds like it's mostly repo maintainers - not the users.

Did you miss the "User downloads 5 files in size of 32 package metadata _in total_ vs 4000"? E.g. the user will typically download less than 1% of what he's downloading now. It benefits by far more the user base (and perhaps mirror admins) than the repo creator.

...

o introduce package cancelation (anti-packages ;)

fat chance.

Sorry, my slang is off, does this mean "no way", or "already in development"? From the context of the rest I'd guess the first. ;)

...

o introduce multiple repodata components

which buys us not all that much other than complexity of debugging.

It buys you all the nice things already outlined.

...

o keep a manifest of the last state and feed the repo creation system with the differences (packages lost, packages gained).

And how do you feed the repo creation system this data? Where do you get it to begin with? The only way you know this information is if you already have it

But you do, this is about incremental updates to a repository, right?

• the only way you have it is if you checked all the packages for

what has changed. Are you beginning to see the loop here?

No.

If someone wants to combine createrepo and yum-arch into one program so it makes both at the same time that's fine - it's about an hour or two worth of work,

That's a complete other topic.

what you're describing above is considerably more, not to mention redesigning the depsolvers to deal with the new repository format.

It may even may it simpler, since you don't need to split it into more importnant and less important data and have file dependencies computed in two loops.

On Sat, Jan 29, 2005 at 05:54:48PM -0600, David Farning wrote:

Now, in most situations very few packages have been changed. Therefore, between primary.xml versions few <package>...</packages> have changed thus most of the information in primary.xml is redundant against previous versions of primary.xml.

Why all the complication when a general-purpose algorithm, rsync, already exists to solve this problem? Using rsync or librsync would require no changes to the repodata...

On Jan 29, 2005, "Charles R. Anderson" cra@WPI.EDU wrote:

Why all the complication when a general-purpose algorithm, rsync, already exists to solve this problem?

It doesn't do very well on compressed files, and there aren't a lot of rsync-enabled web proxies out there.

Dag Wieers wrote:

On Sun, 29 Jan 2005, Alexandre Oliva wrote:

...

On Jan 29, 2005, "Charles R. Anderson" cra@WPI.EDU wrote:

...

Why all the complication when a general-purpose algorithm, rsync, already exists to solve this problem?

It doesn't do very well on compressed files, and there aren't a lot of rsync-enabled web proxies out there.

You may like this RFE:

https://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=391

gzip has an --rsyncable option that is missing from the python zlib implementation. If python/zlib could be taught this, and createrepo is able to use it, the metadata would be rsyncable (improves transferspeed for repo maintainers/mirrors)

The necessary patch is at https://svn.uhulinux.hu/packages/dev/zlib/patches/02-rsync.patch

The patch is also in rpm-4.4., zlib bundled into -lrpmio, with a "rpmz_" uniqifier on zlib symbols.

73 de Jeff

Hi,

I think that if we reduce to amount of metadata necessary to transfer both users and maintainers will benefit.

They will consume less bandwidth and be able to serve more clients and the users will have faster downloads.

Oh and once more - who is it gets the benefit from all this work? It sounds like it's mostly repo maintainers - not the users.

On Jan 29, 2005, seth vidal skvidal@phy.duke.edu wrote:

users b/c problems will be put-near impossible to debug developers b/c they will have lost sanity writing and maintaining this code.

In other words, you have somehow proved that it's impossible to both have better behavior and still be maintainable? This is quite a breakthrough in software engineering! :-)

On Jan 30, 2005, seth vidal skvidal@phy.duke.edu wrote:

...

In other words, you have somehow proved that it's impossible to both have better behavior and still be maintainable? This is quite a breakthrough in software engineering! :-)

No, I'm just trying to keep folks, including myself, from doing a lot of work that ends up being a real serious pain for years on end.

Like trying to get everybody to agree on a format that is specifically designed to get every user to download a multi-megabyte file every time a repository has any change whatsoever?

If you're trying to avoid real serious pain for years on end too, we're on the same boat :-)

...

You mean like rpm is a pain? ;-)

hahah, yes, but that's your pain :)

-sv

On Jan 30, 2005, Jeff Johnson n3npq@nc.rr.com wrote:

More seriously, I'm like a weekend's work away from adding a look-aside cache to rpm -4.4.x (which has a relaible http/https stack using neon) that could be invoked asynchronously to yum as rpm -q http://host/path/to/N-V-R.A.rpm and then yum could read the header from the package as it was being downloaded

Err... By the time yum or any other depsolver decides to download a package, it's already got all the headers for all packages. And I hope you're not suggesting yum to get rpm to download *all* packages just because it needs headers. *That* would be a waste of bandwidth.

into /var/cache/yum/repo/packages since you already know the header byte range you are interested in from the xml metadata, thereby saving the bandwidth used by reading the header twice.

Hmm... I hope you're not saying yum actually fetches the header portion out of the rpm files for purposes of dep resolution. Although I realize the information in the .xml file makes it perfectly possible, it also makes it (mostly?) redundant. Having to download not only the big xml files but also all of the headers would suck in a big way!

I was thinking to myself that having to download only the compressed xml files might be a win (bandwidth-wise) over going though all of the headers like good old yum 2.0 did, at least in the short term, and for a repository that doesn't change too much.

But having to download the xml files *and* the rpm's headers upfront would make the repodata format a bit loser, because not only would you waste a lot of bandwidth with the xml files, that are much bigger than the header.info files, but also because fetching only the header portion out of the rpm files with byte-range downloads makes them non-cacheable by say squid.

I'd be very surprised if yum 2.1 actually worked this way. I expect far better from Seth, and from what I read during the design period of the metadata format, I understood that the point of the xml files was precisely to avoid having to download the hdr files in the first place. So why would they be needed? To get rpmlib to verify the transaction, perhaps?

That's a far bigger bandwidth saving than attempting to fragment primary.xml, which already has timestamp checks to avoid downloading the same file repeatedly

The problem is not downloading the same file repeatedly. The problem is that, after it is updated, you have to download the entire file again to get a very small amount of new information. Assuming a biggish repository like FC updates, development, pre-extras, extras or dag, freshrpms, at-rpms, newrpms, etc, it's a lot of wasted bandwidth.

On Jan 29, 2005, seth vidal skvidal@phy.duke.edu wrote:

How would it reduce bandwidth - you'd have to download and parse multiple entries and you'd STILL have to do just a much work on the repo-side b/c you'd have to check all the packages for changes.

The reduced bandwidth would be for the thousands of users who could download a 1KiB file with the changes since the last time they checked the repo, instead downloading 4MiB with about 1KiB of new information.

Sure, createrepo would have to look at previous versions of the repodata, see what changed since then (it could optionally use only file timestamps and sizes to check that files haven't changed, instead of having to read them entirely to compute checksums) and generate a new, incremental repository format.

What I'm thinking is that this incremental repodata tree would contain the relative location of the original repodata tree, such that whoever downloads the incremental repodata can get to the previous states, and so on, by following the paths given.

So we could put in a counter-based repository history with the following properties:

- after the first run of createrepo, repodata/repomd.xml points to repodata/0, without adding or removing anything.

- after the second run of createrepo, repodata/repomd.xml points to repodata/1, with a repomd.xml that points to ../0, and primary.xml et al files adding/removing packages from ../0

and so on.

every now and then, one could consolidate the multiple repodata subdirs into a single set of xml files. You could even do this every time, and have repomd.xml indicate that you can either get all the data from this single set of files, or the incremental history from this other file.

This sort of indirection in repomd.xml has one interesting additional side effects: if done properly, it would enable us to create composite and/or filtered repositories. Your composite repository would reference a base repository (or a set thereof) in repomd.xml, as well as package removals or additions so as to filter out packages from one repository that are say known to be incompatible, and additions from your own.

This may sure add a lot of complexity to the client side, but reducing daily downloads of rawhide/i386's primary.xml.gz and filelists.xml.gz (totaling 4MiB) by however many users track rawhide to a few KiB sounds like a pretty good idea to me.

On Fri, Jan 28, 2005 at 05:33:25PM +0200, Panu Matilainen wrote:

...

There is a simple patch by the CERN folks used for ia64 by spliting apt's processing of i386 and ia64 into different package worlds (in apt-rpm's archives). Can that be used as for an initial multilib approach?

Ugh, no thanks. Feel free to patch your version of apt with it, but from my POV it's just too ugly to live with.

Well, uglyness is the essence of coding. ;)

But, seriously, would that patch do its job and get apt up to multilib?

...

apt's lack of multilib is a PITA, but you must also consider that apt's development community has only one member coming from the multilib world, the masochist sharing Panu's mail address ... ;)

...and even I'm not really from "multilib world" since I don't have x86_64 boxes at home. My next system is going to be that but whether it happens this year, next year or the one after that I dunno :)

OK, then apt has no developer that could look after multilib support. No wonder that it is still not there (and will probably never make it). It's a catch22 (FC does not support apt, because apt has no multilib, and apt has no multilib, because FC has no (multilib) developers assigned to it :)

Thankfully yum and smart are growing fast enough to fill the gap in 64bit world.

Ralf Corsepius wrote:

On Thu, 2005-01-27 at 08:16 -0500, Jeff Johnson wrote:

...

Thomas Vander Stichele wrote:

...

...

c) nothing more was ever offered as negative feedback on mach than "it uses apt" (a fact that is easily changeable, obviously);

Well clearly, having mach rely on a package that is not included in any Red Hat product,

RH has the ability to change this at any time.

So does mach.

...

presents certain logistical difficulties. That should be obvious.

It is not - RH has had no problems in adding yum support and has no problem in adding and removing other packages at any time at RH's free will.

It's not obvious that mach (as is) will never function on any Red Hat (or Red Hat packages) based distro unless apt is included? I beg to differ.

Yum, because in python, and because developed more closely with other Red Hat tools like anaconda originally at YDL and now locally at Duke, was an easier cost/benefit decision.

apt, because in C++, with concepts like Suggests: and Recommends: and Enhances: and pinning and a back-tracking depsolver and alien etc etc, is a far far more complicated integration effort with a less clear cost/benefit analysis.

Red Hat has very few people able or willing to support apt, for historical, if not for other reasons.

For example instead of adding yum and keeping up2date, RH could have tried to help apt. - IMO, this is all politics and not at all technically motivated.

I have helped apt-rpm, and have helped smartpm, and have helped Red Carpet, and other distros technically with rpm for years.

And I work for Red Hat.

Does that count?

...

And, FWIW, I have suggested repeatedly that apt be added to FC internally to Red Hat in spite of the cost of attempting to maintain Yet Another Depsolver. The previous line basically summarizes the majority of the feedback that I have heard:

FC needs fewer depsolvers that work more reliably.

Isn't the solution obvious? Implement one unified depsolver into rpm which behaves exactly as you envision it, instead.

What do you think I am trying to do, grow bananas in Chapel Hill?

I have been trying to build the infrastructure for a unified package manager for *years*. And so has Red Hat, but the politics are insurmountable imho.

Where have you been?

73 de Jeff