On Wed, 2005-03-02 at 17:41 -0500, seth vidal wrote:

...

...

• something that watches cvs for things to be built
• something that makes/handles/cleans up the chroots
• something that spawns the builds
• something that deals with the results

Is it reasonable to focus on these as modules to be developed?

One more addendum - having the last 3 be a separate package end users could interact with would be very useful I suspect.

Definitely the second and some part of the third. Having the other two in the set of "stuff we let you install and have your own personal copy of the buildsystem" may be overkill. Although all of it should be available, it's just what we advertise makes sense for people to do as their setup.

All of it is useful. We want people to be able to set up their own build engines running their favorite not-really-supported platform, including their option to drive it automagically from our repositories, and to produce results web pages that look just like the canonical ones.

Jeremy Katz wrote :

...

• something that spawns the builds
• something that deals with the results

These two are likely to be fairly related. Perhaps even the same thing.

Yes and no : Since we're probably going to want to support different archs coming from different build machines some day, the build spawning should be per build host, whereas the dealing with the results will be partly per build host (the part that you consider fairly related) but also partly in a central location which will gather everything, right?

Other than that, I agree with Jeremy regarding the fact the builds should be explicitly requested and not automatic for every CVS commit, or even every commit matching certain criterias (i.e. a change in version and/or release).

Matthias

On Wed, 2 Mar 2005, Jeremy Katz wrote:

...

...

From what I can think breaking up the build system into:

• something that watches cvs for things to be built

One thing that comes to my mind is that you probably don't want to be watching CVS. Having it be an explicit "request a build now" makes more sense (which can then be integrated as a makefile target eventually, etc). I just tend to prefer having "do a build" be an explicit action rather than a side effect.

I agree, I would rather have a "cvs tag build" or "cvs tag build-test" or something like that. That will queue a build request and provide some sort of URL where one could watch the status.

The two big things are probably the "handle chroots" piece and "spawn builds". Especially if we want to go the route of a new chroot for every build. So I'd mostly focus on those two first and I think the other stuff will mostly fall out on its own.

But wait, there is more! Ok, so we have chroots, we're spawning build, what do we do with the resulting packages? what is their path through the process?

So far we have:

A. Buildroot provisioning - yum-based scriplet - users can run it themselves and create their own trees - for speed reasonsm can we assume that buildroots are generic (ie, have the devel stuff installed, but are not customized for the needs of any particular src.rpm build)

B. Spawning builds - assuming a queue of some sort of things that need building - do we have a master controller for builds or do we let all buildhosts fight to empty out the build queue? - once a buildroot is chosen: - we customize it according to the src.rpm's buildrequires - launch the "chroot ... rpmbuild --rebuild ..." job - stdout and stderr go to a log accessible online in real time? - extract the binary packages and drop them somewhere - after the build is done - dispose of the buildroot? - set up a new buildroot again (async?)

C. Package management - we have a bunch of new packages built for a particular tree - what is the qualification process? - QA? - puching stuff out?

Anybody else has any other big components we need to concentrate on?

Cristian

On Wed, 2 Mar 2005, Roland McGrath wrote:

...

- assuming a queue of some sort of things that need building
- do we have a master controller for builds or do we let all buildhosts
  fight to empty out the build queue?

You probably want a queue manager and scheduler to handle things like prioritization and failover.

-- Elliot

Not sure what this would achieve ? In mach, these three "target" names mean the following:

• minimal: a minimal set of packages that allows you to chroot into it

and run bash

• base: the same set of packages, but with all packages needed to make

the rpm db consistent added

• build: a bunch of additional packages that rpmbuild likes to have

(patch, gcc, ...)

Not sure what (some magic link to comps.xml) would bring more.

but instead of having to discern the list w/no relationship information from this:

# Fedora Core Development packages['fedora-development-i386-core'] = { 'dir': 'fedoracore-development-i386', 'minimal': 'bash glibc yum python createrepo', 'base': 'coreutils findutils openssh-server', 'build': 'dev rpm-build make gcc tar gzip patch ' + 'unzip bzip2 diffutils cpio elfutils', }

we can use an xml format that various folks are already very familiar with.

...

2. I talked to Jeremy some about this and I think if we do all rpmdb

transactions from OUTSIDE of the rpmdb and only build in the chroot then we should be able to safely juggle multiple rpmdb versions from host to chroot systems.

Yep - that's how mach 2 has always done it after long discussions with jbj. The alternative is to put specially compiled rpms in the root.

I'd also like to have no calls to the rpm cli binary in any buildroot system. we should never be making the buildroot with --nodeps or --force so I don't see a motive to use rpm for erasures or additions.

...

3. there's no reason to not develop a specialized script that uses the

yum modules that can be run by something like mach-helper for making chroots reasonably correctly.

What does "reasonably correctly" mean here ? I mean, is there anything wrong with the chroots that currently can be created with rpm -- root/apt-get .../yum --installroot=... ?

okay so what do we get out of making the buildsystem capable of using yum/apt-get/rpm--aid/whatever for doing the installs?

what's the perk? If we're building this for fedora why not just make a script that imports the yum modules and works out of the available infrastructure? Is there something that's needed in the yum modules to make this work?

-sv

well, this passes by the fact that a) comps.xml doesn't have this concept of minimal

why not? Just make a new group, call it minimal.

b) comps.xml IIRC has a completely different understanding of "base" than what I just said (the minimum self-consistent set of packages that give you bash)

again - call it chroot-base - but the point is the same.

c) there are distros mach is used for that do not have comps.xml files.

I'm not talking about using the distro provided comps.xml - I'm talking about using that format for specifying the packages installed in the chroots.

-sv

...

...

c) there are distros mach is used for that do not have comps.xml files.

I'm not talking about using the distro provided comps.xml - I'm talking about using that format for specifying the packages installed in the chroots.

What would be the advantage of this? You will have to maintain yet another configuration file with an ugly format (XML), and it will not work with other depsolvers like apt or smartpm.

Right, I'm having trouble trying to figure out why we're bothering with support for other depsolvers at this time. We just need to build now.

-sv

On Fri, 2005-03-04 at 16:52 +0100, Enrico Scholz wrote:

skvidal@phy.duke.edu (seth vidal) writes:

...

...

...

I'm not talking about using the distro provided comps.xml - I'm talking about using that format for specifying the packages installed in the chroots.

What would be the advantage of this? You will have to maintain yet another configuration file with an ugly format (XML), and it will not work with other depsolvers like apt or smartpm.

Right, I'm having trouble trying to figure out why we're bothering with support for other depsolvers at this time. We just need to build now.

When we want to build now, I do not understand why code for new technology (comps.xml) shall be added, while existing technology (manual package-lists) is already working...

comps.xml isn't new technology. All the support is there. No waiting. Zero-day.

-sv

...

comps.xml isn't new technology. All the support is there.

mach2 has already support for specifying the location of the comps.xml file? And the comps.xml files for the buildroots exist already?

yep, in yum.

it just has to use groupinstall :)

-sv

On Fri, 2005-03-04 at 18:37 +0100, Enrico Scholz wrote:

skvidal@phy.duke.edu (seth vidal) writes:

...

...

...

comps.xml isn't new technology. All the support is there.

mach2 has already support for specifying the location of the comps.xml file? And the comps.xml files for the buildroots exist already?

yep, in yum.

it just has to use groupinstall :)

Where can the location of the comps.xml files be specified? E.g. so that builds for FC3 use comps-A.xml while builds for devel use comps-B.xml? This works really out-of-the-box without adding code to mach2?

in the repository metadata.

you can even have a repository that has ONLY comps information.

-sv

On Tue, 2005-03-08 at 14:38 -1000, Warren Togami wrote:

I have no objection to yum groupinstall, but in my opinion none of the current defined groups are suitable for a minimal buildroot. There have been objections in the past to this with the opinion that the "Development" tools group and -devel packages should be assumed to be installed in this minimum buildroot. However this is a bad assumption because the set of -devel packages has been arbitrary, and dependencies don't make sure that particular tools exist in the buildroot.

As seth has pointed out before, we don't have to use the existing comps file, we can create our own groups, but in comps format. That is the real question, adding comps format support to the build system.

I agree with the rest about the minimal build environment, buildreqs for the rest.

On Wed, 9 Mar 2005, Panu Matilainen wrote:

Minimal buildroot isn't necessary for reproducible builds, a *consistently* populated buildroot is. You'll get a consistent environment by dropping in Base + Devel groups with yum groupinstall even with the stock comps.xml.

Providing consistent buildroots actually works against reproducible builds in the long term, because of the effect those buildroots have on the way people choose to package things.

For maximum quality control, packages should not be affected by having an unrelated (non-BuildRequires and non-base) package installed in the buildroot. If package X is is unrelated to the ongoing build of package Y, then package Y's build should not be affected by the absence OR presence of package X in the buildroot.

The root cause of the problem here is not really having consistent buildroots, but having improper packaging that doesn't account for all possible variables. One thing we have internally at Red Hat is a mass rebuild system that creates a buildroot with all packages installed, attempts rebuilds of all packages, and for the builds that succeed, it compares the resulting binary packages against the original ones to see if things like filelist or dependencies have changed. It'd be nice to get the equivalent of that for Fedora.

Best, -- Elliot

On Tue, 15 Mar 2005, seth vidal wrote:

ALL packages installed becomes a bit more complex when you think about fedora extras. ALL could become several thousand packages.

A random sampling should be sufficient.

Cheers, -- Elliot

On Tue, 15 Mar 2005, Elliot Lee wrote:

...

Minimal buildroot isn't necessary for reproducible builds, a *consistently* populated buildroot is. You'll get a consistent environment by dropping in Base + Devel groups with yum groupinstall even with the stock comps.xml.

Providing consistent buildroots actually works against reproducible builds in the long term, because of the effect those buildroots have on the way people choose to package things.

I agree... in theory...

For maximum quality control, packages should not be affected by having an unrelated (non-BuildRequires and non-base) package installed in the buildroot. If package X is is unrelated to the ongoing build of package Y, then package Y's build should not be affected by the absence OR presence of package X in the buildroot.

In the ideal world, yes. In the practical world, a very large amount of software does ./configure time autodetection of various libraries and other software which may or may not be present in a buildroot, many of it being conditionally enabled/disabled based on the presence or lack of libs/etc. avail.

This autodetection is good for Joe Blow downloading something and compiling/installing by hand into /usr/local per se. but it kindof works against rpm based builds in the way of reproduceability.

This puts a large part of the reproduceability factor square in the hands of the package maintainer. In order to get a reasonably good chance of having every rpm rebuild exactly the same regardless of what deps are present or absent in the buildroot, all package maintainers need to be come much more intimately involved with the rpms they maintain. This would require deeply inspecting all ./configure options with each release of the software, and being more involved with the underlying projects in question, and very closely analyzing the output of ./configure and trying to determine if there are any changes from upstream version to version and build to build.

While it could be argued "this is already the packager's responsibility", in reality it does not work well, and it isn't likely to ever work well as long as it is not automated in some fashion. Relying on humans to do all of this:

1) Puts a lot of extra burden on humans, whom are already already greatly overburdened.

2) Makes the single point of failure be the human. Very bad idea. Not scaleable. Humans make mistakes. Computers do not.

The most scaleable systems, are those which are as completely automated as possible, requiring as little to no human intervention as possible.

So my suggestion to those seeking a solution to this problem, is to look at how it can be eliminated or reduced through software automation. rpmdiff is an example of creative use of automation. Perhaps someone can brainstorm an automation tool that could be plugged into rpm or beehive or mach, etc.

The root cause of the problem here is not really having consistent buildroots, but having improper packaging that doesn't account for all possible variables.

Yep.

One thing we have internally at Red Hat is a mass rebuild system that creates a buildroot with all packages installed, attempts rebuilds of all packages, and for the builds that succeed, it compares the resulting binary packages against the original ones to see if things like filelist or dependencies have changed. It'd be nice to get the equivalent of that for Fedora.

If someone were to develop a tool that compared consecutive ./configure runs and reported major differences, that'd be cool. I don't know how difficult that'd be though. I suspect if it were easy someone might have done it by now, but who knows. ;o)

HTH

mharris@redhat.com ("Mike A. Harris") writes:

...

One thing we have internally at Red Hat is a mass rebuild system that creates a buildroot with all packages installed, attempts rebuilds of all packages, and for the builds that succeed, it compares the resulting binary packages against the original ones to see if things like filelist or dependencies have changed. It'd be nice to get the equivalent of that for Fedora.

If someone were to develop a tool that compared consecutive ./configure runs and reported major differences,

That's not a problem... just run a diff across the 'config.status' files. But what is with packages which do not use ./configure but have other kinds of build-time feature-checks?

Comparing resulting packages seems to be a more universal way for detecting missing BuildRequires:.

Enrico