Hey folks, I've been doing a lot of tests today and I have some good news to report.
1. the xml-rpc communication is working pretty well. We can spawn builds out to different hosts than the queuer and get feedback on what broke in the build and/or why.
2. I've cleaned up the code and on my set of 2 systems (x86_64 and ppc) I can build for all 3 architectures w/o having to manually run anything.
You can see the code here: http://linux.duke.edu/~skvidal/misc/buildsys/
Gist of how it is used: The queuer runs, figures out what needs to be built by getting the list from the /cvs/extras/common/tobuild file. It preps for the build by: - checking out the tag from cvs - making all the necessary dirs (name/v-r/a) - making the srpm Then it dispatches the build to the right archwelder classes. These classes build the packages and put the files into the right places if they succeed; or output the logs if they fail. After the build runs the queuer will notify the person requesting the build of success or failure.
In the event of a failure on any one architecture the other builds are stopped and logs are reported.
That's the short version of how it works - a list of todos are at the top of the two important files. I've got a few more tests to do and then I'll probably make the 'tobuild' file open to anyone with cvs commit access so you can run 'make build' to request your own builds.
let me know what you think, even if I've wasted my time.
-sv
On Wed, 2005-05-04 at 00:40 -0400, seth vidal wrote:
Hey folks, I've been doing a lot of tests today and I have some good news to report.
- the xml-rpc communication is working pretty well. We can spawn builds
out to different hosts than the queuer and get feedback on what broke in the build and/or why.
- I've cleaned up the code and on my set of 2 systems (x86_64 and ppc)
I can build for all 3 architectures w/o having to manually run anything.
A couple of comments, some of which I'd even be willing to do the work for :)
1) CVS/tobuild - Could this code be separated into an XML-RPC client? So you'd have the Queuer be an XMLRPC server, and clients would connect to it and feed it build jobs. For Aurora at least, we're probably not going to have CVS of this type, and "tobuild" seems somewhat cumbersome.
I'd like to build a client that parses the output of Beehive emails and queues a build (so that whenever something gets built internally at Red Hat, it could get built into an Aurora build system)
There would be some type of authentication so the Queuer might only accept connections from a certain host[s], eventually by SSL certificate or something once that code gets done.
2) Config info - Man that's spread around a lot... Anyway, what's the suggested configuration tool to use? I don't really like ConfigParser all that much, but I don't want to do something else that people don't like either. How about a simple "config.py" file that contains the configuration as dict entries or something, and then the source would use "config.get('localarch')"? Fairly simple, but suggestions accepted.
3) Could you explain a bit about the flow? The client/server separation isn't extremely clear currently, such that the server includes the builder code as well and calls some of it. Which scripts do you launch on which machines? With which arguments? Could we separate the server/queuer code from the XMLRPC/builder bits explicitly?
let me know what you think, even if I've wasted my time.
Certainly not wasted...
Cheers, Dan
On Thu, 2005-05-05 at 09:37 -0400, Dan Williams wrote:
A couple of comments, some of which I'd even be willing to do the work for :)
- CVS/tobuild - Could this code be separated into an XML-RPC client?
So you'd have the Queuer be an XMLRPC server, and clients would connect to it and feed it build jobs. For Aurora at least, we're probably not going to have CVS of this type, and "tobuild" seems somewhat cumbersome.
This was definitely intended as the eventual direction. But we wanted to get something going quickly, and write a file was far simpler. :-) Especially since at first, there wasn't going to be the need for XML-RPC -- that came later as a need for kicking off the ppc builds.
Any help down this path would definitely be appreciated, otherwise it's at least on my todo list. The key is to keep the client as simple as possible so that we don't have a big list of deps like you end up with for the beehive client. Basic python + included modules should be fairly sane, though. And using the client certs for the fedora account system stuff seems like the obvious auth mechanism then.
Jeremy
On Thu, 2005-05-05 at 12:04 -0400, Jeremy Katz wrote:
On Thu, 2005-05-05 at 09:37 -0400, Dan Williams wrote:
A couple of comments, some of which I'd even be willing to do the work for :)
- CVS/tobuild - Could this code be separated into an XML-RPC client?
So you'd have the Queuer be an XMLRPC server, and clients would connect to it and feed it build jobs. For Aurora at least, we're probably not going to have CVS of this type, and "tobuild" seems somewhat cumbersome.
This was definitely intended as the eventual direction. But we wanted to get something going quickly, and write a file was far simpler. :-) Especially since at first, there wasn't going to be the need for XML-RPC -- that came later as a need for kicking off the ppc builds.
Any help down this path would definitely be appreciated, otherwise it's at least on my todo list. The key is to keep the client as simple as possible so that we don't have a big list of deps like you end up with for the beehive client. Basic python + included modules should be fairly sane, though. And using the client certs for the fedora account system stuff seems like the obvious auth mechanism then.
Ok, I'll try to do this in the next couple days then. Is there CVS somewhere for the scripts? I don't want to do the work and then do manual merges really.
Dan
On Thu, 2005-05-05 at 12:31 -0400, Dan Williams wrote:
On Thu, 2005-05-05 at 12:04 -0400, Jeremy Katz wrote:
On Thu, 2005-05-05 at 09:37 -0400, Dan Williams wrote:
A couple of comments, some of which I'd even be willing to do the work for :)
- CVS/tobuild - Could this code be separated into an XML-RPC client?
So you'd have the Queuer be an XMLRPC server, and clients would connect to it and feed it build jobs. For Aurora at least, we're probably not going to have CVS of this type, and "tobuild" seems somewhat cumbersome.
This was definitely intended as the eventual direction. But we wanted to get something going quickly, and write a file was far simpler. :-) Especially since at first, there wasn't going to be the need for XML-RPC -- that came later as a need for kicking off the ppc builds.
Any help down this path would definitely be appreciated, otherwise it's at least on my todo list. The key is to keep the client as simple as possible so that we don't have a big list of deps like you end up with for the beehive client. Basic python + included modules should be fairly sane, though. And using the client certs for the fedora account system stuff seems like the obvious auth mechanism then.
Ok, I'll try to do this in the next couple days then. Is there CVS somewhere for the scripts? I don't want to do the work and then do manual merges really.
code is in extras-buildsys-temp in /cvs/fedora
look in the dir 'automation' in that module.
If you want the code that does the 2-way auth via ssl certs you should look at m2crypto. CCing this to icon so he can tell you where to look for the code.
-sv
buildsys@lists.fedoraproject.org
-
Dan Williams -
Jeremy Katz -
seth vidal