hi folks, So right now I think things with mach and yum are working for building fedora extras. The packages _seem_ like they're coming out right and things seem functional enough. The second part that I need help and input on is the glue scripts and requirements for having automatic triggering of builds for packagers.
The questions I have: 1. if this is meant to run on the red hat boxes in the PHX coloc, what does that infrastructure look like? What features does it have? Can we assume all the build boxes have access to the cvs tree? Do we need to worry about pushing srpms around? 2. How do folks want packagers to send notices about builds? Just a cvs tag? A webpage? A gpg-signed email with specific content? A custom xmpp/jabber-client to send a custom message to a listening build client across an xmpp infrastructure? :) 3. What things am I missing or not understanding about what is needed from the build system? The requirements I've been working under are/were: - self hosting on Fedora Core - not crazy What else do I need to think about? 4. Who else is interested in working on this and getting things progressing more? The yum changes to mach are just a hackjob to get a problem solved for the short term. However, I'd like to continue down this general line of development. so <buffy>Where do we go from here?</buffy>
Thanks, -sv
seth vidal wrote :
- if this is meant to run on the red hat boxes in the PHX coloc, what
does that infrastructure look like? What features does it have? Can we assume all the build boxes have access to the cvs tree? Do we need to worry about pushing srpms around?
I'd guess that the lookaside cache is there exactly to avoid any srpm upload, and use it + the CVS instead.
- How do folks want packagers to send notices about builds? Just a
cvs tag? A webpage? A gpg-signed email with specific content? A custom xmpp/jabber-client to send a custom message to a listening build client across an xmpp infrastructure? :)
Yeah, a "jabber bot"! :-D For me, the easiest would be to trigger a build directly from a form on a web page. Ideally, be able to track the build from the same page too : Check status, view full build log once finished etc. Sounds easy, but most certainly isn't, especially if we plan on supporting other archs than i386/ x86_64 since it'll mean de-centralized builds.
- What things am I missing or not understanding about what is needed
from the build system? The requirements I've been working under are/were: - self hosting on Fedora Core - not crazy What else do I need to think about?
Depending on the answers to the questions above, all the flow between how/ when a packager requests a build, and how/when the packages appear in the ftp tree will need to be thought out. Hmm, "not crazy", you say? :-)
- Who else is interested in working on this and getting things
progressing more? The yum changes to mach are just a hackjob to get a problem solved for the short term. However, I'd like to continue down this general line of development. so <buffy>Where do we go from here?</buffy>
I still haven't had time to try out your modified mach2, but definitely want to, and want to adopt it since it'll solve all the small annoyances related to using apt (requiring the metadata, having all cached file names mangled). Keep up the good work Seth, it's much appreciated! As for the exact direction to take, it'll depend on RH official answers, what can and cannot be done with the build server(s).
Matthias
On Fri, 2005-03-18 at 09:50 +0100, Matthias Saou wrote:
seth vidal wrote :
- How do folks want packagers to send notices about builds? Just a
cvs tag? A webpage? A gpg-signed email with specific content? A custom xmpp/jabber-client to send a custom message to a listening build client across an xmpp infrastructure? :)
Yeah, a "jabber bot"! :-D For me, the easiest would be to trigger a build directly from a form on a web page. Ideally, be able to track the build from the same page too : Check status, view full build log once finished etc. Sounds easy, but most certainly isn't, especially if we plan on supporting other archs than i386/ x86_64 since it'll mean de-centralized builds.
I think requiring a tag of some kind in CVS to trigger a build would be cool: it ensures that builds are actually tagged.
Maybe something like this: "make tag" in a CVS checkout could do the tagging with the appropriate tag name derived from the package (N?)EVR(+branch?), as well as force- tag a special transient and moving BUILDME_DAMMIT'ish tag which the buildsys could look for. However, the buildsys needs to be able to figure out when to build based on new tags appearing in CVS or existing tags being moved to new revisions. That might be a bit tricky, but I think it's doable. Or there could be a special BUILDIT'ish file (possibly GPG signed) committed to CVS containing checksums of some kind and the tag to build.
Regarding the tracking, I agree that a web page would work pretty well. Some examples from Debian: http://packages.qa.debian.org/ http://buildd.debian.org/
A mail interface should also be there, I think a notification message after a build containing links to the hypothetical web page for the real contents (build logs, possibly some other reports like rpmlint, etc) as in above would be ok.
Triggering a build by sending mail sounds somewhat cumbersome, ditto having to surf to a specific web page and clicking around. Being able to do it with CVS would be ideal IMO.
I think requiring a tag of some kind in CVS to trigger a build would be cool: it ensures that builds are actually tagged.
Maybe something like this: "make tag" in a CVS checkout could do the tagging with the appropriate tag name derived from the package (N?)EVR(+branch?), as well as force- tag a special transient and moving BUILDME_DAMMIT'ish tag which the buildsys could look for. However, the buildsys needs to be able to figure out when to build based on new tags appearing in CVS or existing tags being moved to new revisions. That might be a bit tricky, but I think it's doable. Or there could be a special BUILDIT'ish file (possibly GPG signed) committed to CVS containing checksums of some kind and the tag to build.
problem: what if you only want a single arch built, how do you specify a single arch be built via the tag? How do you deal with test non-release builds? That's my concern about tag-based building, it seems kinda limited.
I'd think having tagged releases + some way of instructing the buildsystem that we want it built 1. for release 2. for a set of archs, would probably give us more flexibility.
Maybe some of beehive's caretakers can tell us how beehive gets build requests signaled?
-sv
On Sun, 2005-03-20 at 10:06 -0500, seth vidal wrote:
I think requiring a tag of some kind in CVS to trigger a build would be cool: it ensures that builds are actually tagged.
Maybe something like this: "make tag" in a CVS checkout could do the tagging with the appropriate tag name derived from the package (N?)EVR(+branch?), as well as force- tag a special transient and moving BUILDME_DAMMIT'ish tag which the buildsys could look for. However, the buildsys needs to be able to figure out when to build based on new tags appearing in CVS or existing tags being moved to new revisions. That might be a bit tricky, but I think it's doable. Or there could be a special BUILDIT'ish file (possibly GPG signed) committed to CVS containing checksums of some kind and the tag to build.
problem: what if you only want a single arch built, how do you specify a single arch be built via the tag? How do you deal with test non-release builds? That's my concern about tag-based building, it seems kinda limited.
I'd think having tagged releases + some way of instructing the buildsystem that we want it built 1. for release 2. for a set of archs, would probably give us more flexibility.
The second "special file" approach could be used for this, although there might be better and more user friendly approaches. Anyway, eg. something like this (disclaimer: unfiltered raw braindump):
echo 'tag archs buildtype' \ | gpg --clearsign > BUILD && cvs ci -m 'Build request' BUILD
(Yes, I'm aware that this doesn't contain enough uniquely identifying data so that it would make sense to sign it, think replayability elsewhere in the CVS repo. But don't get stuck wit that now :)
Synopsis: tag [archs [buildtype]]
tag: The tag to build. archs (optional): The archs to build for, ALL (or not present) means all supported. buildtype (optional): Type of build, enumeration of keywords. "release" or not present means a release build, add others as needed.
The buildsys could either "cvs up" and/or "cvs stat" all BUILD files, or be configured to receive notifications from the CVS server some way when they are changed, eg. mail.
Semi-offtopic: it would be good to prepare the infrastructure so that rpmbuild args could be passed to the build system some way. One example of potential packages that could benefit from this would be kernel module packages where the module specfiles wouldn't need changes between target kernel revisions, instead the target kernel(s) to build for would be identified by eg. a --define to rpmbuild.
Semi-offtopic: it would be good to prepare the infrastructure so that rpmbuild args could be passed to the build system some way. One example of potential packages that could benefit from this would be kernel module packages where the module specfiles wouldn't need changes between target kernel revisions, instead the target kernel(s) to build for would be identified by eg. a --define to rpmbuild.
mach lets you pass extras flags to rpmbuild internally, but I agree with you. I've been butting my head up against a few things in mach that I think, for the way the buildsystem is going to have to layout might need to be moved around a bit.
I'm hoping Thomas comes back from vacation or work or wherever he's been b/c I really would like to know what he thinks about some of this stuff. I had to hack up some of mach to get it to work the way it is now but I'd like to get things sorted out so we can either merge upstream or plan to make things match up at some later date.
-sv
seth vidal (skvidal@phy.duke.edu) said:
Maybe some of beehive's caretakers can tell us how beehive gets build requests signaled?
Explicit signals via a 'make build'. Users can pass a variable to build in a scratch destination. 'make build' signals the build system to attempt to check out the current n-v-r from CVS (in general, you need to do 'make tag' before 'make build'), and build in the default location for that branch of CVS.
Bill
seth vidal wrote:
problem: what if you only want a single arch built, how do you specify a single arch be built via the tag? How do you deal with test non-release builds? That's my concern about tag-based building, it seems kinda limited.
We shouldn't allow only a single arch to be built into the repository.
But for flexibility, if a package has a difficult time being fixed for an arch (after some effort is put into it), the packager should temporarily Exclude, build, and file a bug assigned to an arch-group of maintainers. For example Thorsten seems interested in making packages work on x86-64, while dwmw2 on ppc.
To answer your other question, we need both a tag and target repository.
Warren Togami wtogami@redhat.com
On Fri, 2005-03-18 at 01:52 -0500, seth vidal wrote:
So right now I think things with mach and yum are working for building fedora extras. The packages _seem_ like they're coming out right and things seem functional enough.
Yep, has seemed pretty reasonable to me in the poking at it I've done so far. mschwendt's idea of running a lot of already built packages through to see how they fare isn't a bad one. I'll try to set up one of my test boxes to do this over the course of this week (hopefully tomorrow). You don't have to worry about doing that right now :-)
The second part that I need help and input on is the glue scripts and requirements for having automatic triggering of builds for packagers.
The questions I have:
- if this is meant to run on the red hat boxes in the PHX coloc, what
does that infrastructure look like? What features does it have? Can we assume all the build boxes have access to the cvs tree? Do we need to worry about pushing srpms around?
I can't answer authoritatively on what the set up is, but I have a pretty good idea. As it stands right now, the build boxes are all x86_64 boxes with reasonable amounts of RAM and disk space. They should be able to access CVS (if not, that can be fixed) with the theory that you want the buildsystem to be given an explicit tag to build and then you check out off the tag, make the src.rpm and go.
We _do_ need to worry about how the binary packages end up. I'm not entirely sure what's best here. Internally, we use writing to directory trees that look like release/package-N/V-R/A (N is name, V is version, R is release, A is arch)
Then, the trees which end up on the FTP site are composed out of these directory trees. This has the nice feature of making the inheritance of builds from older releases a bit easier.
What is slightly more complicated with a scheme like this is how do you update the repodata after a build completes quickly with the new package info. Some of the "createrepo should be able to be done incrementally" people will probably come back out of the woodwork.
- How do folks want packagers to send notices about builds? Just a
cvs tag? A webpage? A gpg-signed email with specific content? A custom xmpp/jabber-client to send a custom message to a listening build client across an xmpp infrastructure? :)
Magic occurring at cvs tag time is less than ideal. People like it from the "oh, it looks simple" perspective, but there's a lot of other metadata that you sometimes need. Having it be a web form that you give all the appropriate info to seems reasonable to me. Or XML-RPC. Or something like that. Either gives you the ability to have a relatively simple makefile target for a 'make build' with the appropriate info you need to set[1]. This still all works off of a tag being made [2]
Then, after the build, we probably want to kick off a "build complete" mail at least to the originator of the build. Failed builds should similarly get a mail and the build logs need to be able to be easily accessible.
One other thing that springs to mind is the question of what arches to build packages for and whether a specific arch failing should block the build. My opinion, which matches how Core gets built, is that * Packages get built for all Extras arches. ExcludeArch/ExclusiveArch can be used for the (rare) things which need otherwise * Build failures on one arch block all arches. Otherwise, some arches will fall far behind and things like prereqs get really painful
- What things am I missing or not understanding about what is needed
from the build system? The requirements I've been working under are/were: - self hosting on Fedora Core - not crazy What else do I need to think about?
Security is probably one. Although "not crazy" probably covers that. :)
Otherwise, I can't think of big things which need thinking about. Perhaps easy setup such that it can be included in the kickstart configs for the build machines without any real difficulty. But how things look now doesn't seem bad for that.
And the bit that we talked about during LinuxWorld on how we want to make it possible and easy for developers to download the buildsystem and get it going on their own workstation for test builds. That then also enables other third party repositories (there won't be only one :-) to use it and get some consistency.
- Who else is interested in working on this and getting things
progressing more? The yum changes to mach are just a hackjob to get a problem solved for the short term. However, I'd like to continue down this general line of development. so <buffy>Where do we go from here?</buffy>
I'm interested... let me see how much time I can commit. I think where we go from here is mostly trying to get some of the glue pieces working.
Jeremy
[1] Even if this requires writing a (simple-ish) python app to run for kicking it off [2] Yes, I know that the tags aren't being done right now. I'll get that added to Makefile.common tonight or in the morning.
Yep, has seemed pretty reasonable to me in the poking at it I've done so far. mschwendt's idea of running a lot of already built packages through to see how they fare isn't a bad one. I'll try to set up one of my test boxes to do this over the course of this week (hopefully tomorrow). You don't have to worry about doing that right now :-)
You did this and it did reasonably okay, right?
We _do_ need to worry about how the binary packages end up. I'm not entirely sure what's best here. Internally, we use writing to directory trees that look like release/package-N/V-R/A (N is name, V is version, R is release, A is arch)
Then, the trees which end up on the FTP site are composed out of these directory trees. This has the nice feature of making the inheritance of builds from older releases a bit easier.
What is slightly more complicated with a scheme like this is how do you update the repodata after a build completes quickly with the new package info. Some of the "createrepo should be able to be done incrementally" people will probably come back out of the woodwork.
Those people have too much ram. If push comes to shove I can think of a few ways of doing this. createrepo will read symlinks as well, so we could do something grotty like a symlink-farm for packages per repo.
One other thing that springs to mind is the question of what arches to build packages for and whether a specific arch failing should block the build. My opinion, which matches how Core gets built, is that
- Packages get built for all Extras arches. ExcludeArch/ExclusiveArch
can be used for the (rare) things which need otherwise
- Build failures on one arch block all arches. Otherwise, some arches
will fall far behind and things like prereqs get really painful
This might be tricky. I'm worried about notifications b/t the buildsystems. Or are you thinking about having levels like:
buildsystem -> pops out rpm in some known location releasesystem -> determines if all arches have built and therefore if the pkg ever gets moved/copied/linked to the release tree it's targeted for.
And the bit that we talked about during LinuxWorld on how we want to make it possible and easy for developers to download the buildsystem and get it going on their own workstation for test builds. That then also enables other third party repositories (there won't be only one :-) to use it and get some consistency.
Here's what I'm thinking right now: If we can get a good working interface for bugzilla for package tracking. And we can define some new fields/tags for items in this interface then we should be able to let it work for us.
1. packager tags a release in cvs using make tag or whatever. 2. packager updates the package status in the bugzilla package tracker a. they mark it as build b. they mark it for what release (fc3, testing, rawhide, etc) c. they input the cvs tag to build from 3. build system, at regular intervals, queries this information via xml-rpc to bugzilla. It builds the packages and attaches the log reports (or links to the log reports) to the comments in the package tracking interface. 4. build system puts the finalized packages + other stuff in some path that's web accessible as you described above.(CAVEAT: some special casing for embargo'd builds will need to be put in place)
Advantages: 1. users cannot directly request builds so the system isn't overwhelmed 2. regular intervals means the user doesn't have to wait for some person to kick off a build. 3. Having one build master system scan and kick off builds on other machines/arches is not crazy OR having multiple build systems scan, mark the package as 'being built for arch foo on system bar' is also not outside the realm of possibility (though there might be race conditions there) 4. Reasonably scalable as more build systems/packagers are added
Disadvantages: 1. might be overstating the functionality available in the xml-rpc interface to bugzilla 2. Users cannot directly kick off builds, they have to wait (waaaaaaah) 3. Dealing with Embargo'd builds - gonna be a pain no matter what 4. Ordering of builds based on dependency 5. package tracking system does not yet exist.
thoughts? -sv
On Fri, 25 Mar 2005, seth vidal wrote:
<snip>
Disadvantages:
- might be overstating the functionality available in the xml-rpc interface to bugzilla
The xml-rpc functionality doesn't really need to be too complicated, I don't think. It's basically just:
* An interface to create a bug according to some "build request form". Simple "new bug" function, got it.
* An interface to walk through a set of bugs and pull interesting fields. Got it.
* An interface to add comments to existing bugs. Got it.
And what we don't got, dkl can create, if it's sensible.
- Users cannot directly kick off builds, they have to wait (waaaaaaah)
- Dealing with Embargo'd builds - gonna be a pain no matter what
- Ordering of builds based on dependency
How do you do this now? Is it trial and error, or do you have a heuristic that works?
- package tracking system does not yet exist.
--g
_____________________ ____________________________________________ Greg DeKoenigsberg ] [ the future masters of technology will have Community Relations ] [ to be lighthearted and intelligent. the Red Hat ] [ machine easily masters the grim and the ] [ dumb. --mcluhan Red Hat Summit ] [ New Orleans ] [ Learn. Network. Experience Open Source. June 1/2/3 2005 ] [ (And Make Your Boss Pay For It.) [ http://www.redhat.com/promo/summit/
- An interface to create a bug according to some "build request form".
Simple "new bug" function, got it.
- An interface to walk through a set of bugs and pull interesting fields.
Got it.
- An interface to add comments to existing bugs. Got it.
Maybe an interface to lock the bug for a few seconds while you do this update so two systems don't end up answering at the same time, but I'm guessing that's implicit.
And what we don't got, dkl can create, if it's sensible.
sensible is important.
- Users cannot directly kick off builds, they have to wait (waaaaaaah)
- Dealing with Embargo'd builds - gonna be a pain no matter what
- Ordering of builds based on dependency
How do you do this now? Is it trial and error, or do you have a heuristic that works?
pass mach a list of related pkgs and it builds them in the right order, installing the buildreqs it needs for the others it is building in that set.
So it sounds like the order of operations is: - package tracking system - xml-rpc interface - cvs tagging for extras - buildsystem scripts to pull/update this stuff.
sound right?
-sv
On Fri, 2005-03-25 at 05:09 -0500, seth vidal wrote:
Yep, has seemed pretty reasonable to me in the poking at it I've done so far. mschwendt's idea of running a lot of already built packages through to see how they fare isn't a bad one. I'll try to set up one of my test boxes to do this over the course of this week (hopefully tomorrow). You don't have to worry about doing that right now :-)
You did this and it did reasonably okay, right?
Yeah, almost everything succeeded. Just quick checking some of the failure logs showed missing BuildRequires being the reason, which seems sane. Thanks for poking to remind me.
One other thing that springs to mind is the question of what arches to build packages for and whether a specific arch failing should block the build. My opinion, which matches how Core gets built, is that
- Packages get built for all Extras arches. ExcludeArch/ExclusiveArch
can be used for the (rare) things which need otherwise
- Build failures on one arch block all arches. Otherwise, some arches
will fall far behind and things like prereqs get really painful
This might be tricky. I'm worried about notifications b/t the buildsystems. Or are you thinking about having levels like:
buildsystem -> pops out rpm in some known location releasesystem -> determines if all arches have built and therefore if the pkg ever gets moved/copied/linked to the release tree it's targeted for.
That's the easiest way to do it. A file gets dropped in specific location (BUILD-SUCCESS, BUILD-FAIL) that can just be watched for. We don't have to have instantaneous knowledge of build success.
And the bit that we talked about during LinuxWorld on how we want to make it possible and easy for developers to download the buildsystem and get it going on their own workstation for test builds. That then also enables other third party repositories (there won't be only one :-) to use it and get some consistency.
Here's what I'm thinking right now: If we can get a good working interface for bugzilla for package tracking. And we can define some new fields/tags for items in this interface then we should be able to let it work for us.
This could work. Although more "fields" in bugzilla always makes me a little bit wary. Even though I know dkl would end up just overloading existing fields in some cases and making them look different with template magic.
- build system puts the finalized packages + other stuff in some path that's web accessible as you described above.(CAVEAT: some special casing for embargo'd builds will need to be put in place)
Yeah, embargo'd stuff probably requires thought. There's not really a way to do it right now with the CVS repo either, though, so it's further out as a question
- Having one build master system scan and kick off builds on other machines/arches is not crazy OR having multiple build systems scan, mark the package as 'being built for arch foo on system bar' is also not outside the realm of possibility (though there might be race conditions there)
I think there is provision for having a build master machine? Cristian?
Disadvantages:
- might be overstating the functionality available in the xml-rpc interface to bugzilla
It just sounds like query is mostly needed. And for the nice, easy to use build target from the makefile, open bug/add comments. Those should both be present.
- Users cannot directly kick off builds, they have to wait (waaaaaaah)
So long as the polling is done frequently enough, this isn't a huge deal.
- Dealing with Embargo'd builds - gonna be a pain no matter what
See above.
- Ordering of builds based on dependency
FIFO. You need to request your builds in dep order. Sucks a little, but is easy to implement :)
Also, another thing I've thought about (and it's run through my head a few times, this is just the first time I've remembered to type it out). One thing that would be nice would be able to have multiple roots for the same release in mach easily (ie, without having to make copies of the config files with tweaked paths :-). At the same time, that's probably not something that matters in the short term as it could easily be added as an optimization later.
Jeremy
That's the easiest way to do it. A file gets dropped in specific location (BUILD-SUCCESS, BUILD-FAIL) that can just be watched for. We don't have to have instantaneous knowledge of build success.
Right now it's just if the package name/v-r dir shows up in the right 'stages' dir.
See: http://extras64.linux.duke.edu/needsign/development for examples.
It just sounds like query is mostly needed. And for the nice, easy to use build target from the makefile, open bug/add comments. Those should both be present.
<nod> I've not heard the latest status on the package tracking interface in bugzilla. Does it sound like it is near or far? Near == 2 weeks, Far == 2 months.
- Users cannot directly kick off builds, they have to wait (waaaaaaah)
So long as the polling is done frequently enough, this isn't a huge deal.
right now it's every 5 minutes.
- Ordering of builds based on dependency
FIFO. You need to request your builds in dep order. Sucks a little, but is easy to implement :)
agreed.
Also, another thing I've thought about (and it's run through my head a few times, this is just the first time I've remembered to type it out). One thing that would be nice would be able to have multiple roots for the same release in mach easily (ie, without having to make copies of the config files with tweaked paths :-). At the same time, that's probably not something that matters in the short term as it could easily be added as an optimization later.
I thought about that some. We'd need something that tracks what root it is in and make sure to request that same one again. For mach we'd need to open it up to the possibility that the root name is not the root location. I'm not sure how hard/easy that will be w/o looking more closely.
-sv
buildsys@lists.fedoraproject.org
-
Bill Nottingham -
Greg DeKoenigsberg -
Jeremy Katz -
Matthias Saou -
seth vidal -
Ville Skyttä -
Warren Togami