buildsys September 2007

buildsys@lists.fedoraproject.org

18 participants
15 discussions

mock improvements
by Michael E Brown 04 Oct '07

04 Oct '07

So here is the list of things that have been requested lately and I'll be working on a few of these over the next few weeks. If anybody has any input, I'd take it. As I start on each, I'll most likely email the mailing list with the outline of what I'm doing. If anybody has existing patches for these (against current mock git), all the better... :) 1) more reliable mount/umount several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad.... 2) caching yum downloads several people have commented that the autocache stuff is great for speeding up builds, others say that it can sometimes be bad for reproducability, and that simply saving the yum cache dir would be better. 3) ccache integration This is a new one that I havent seen before, but should significantly speed up builds for people who often do rebuild-the-entire-distribution-type things. I'm told by some that this is bad for reproducability, but good for speeding up builds when you are just sanity checking or when that small reproducability hit doesnt matter. I've also seen lots of empirical data that ccache should not cause any problems. This will be have to be specifically enabled through a commandline or config file option, so those who care can turn it on/off. 4) distcc integration Pretty much the same case as ccache. Has more things that need thought than the ccache case, above, though. -- Michael

6 7

RE: problems with orphanskill feature
by Michael E Brown 27 Sep '07

27 Sep '07

Do you want to do the patch or me? It will be tomorrow morning before I can get to it. (or very late tonight.) -- Michael -----Original Message----- From: Clark Williams [mailto:clark.williams@gmail.com] Sent: Thu 9/27/2007 4:11 PM To: Brown, Michael E Cc: Jesse Keating; fedora-buildsys-list(a)redhat.com Subject: Re: problems with orphanskill feature -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael E Brown wrote: > On Thu, Sep 27, 2007 at 01:30:15PM -0500, Clark Williams wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Michael/Jesse, >> >> We're seeing odd failures with mock and the orphanskill feature. What seems to happen >> is that the '; mock-helper orphanskill <rootdir>' string is tacked onto a command >> which is passed to do_chroot() and after the main command is run, an attempt is made >> to run mock-helper (which is not installed in the chroot). So people are seeing a >> "File not found" message after a successful command. >> >> Now it looks like to me that part of the reason for an orphanskill was that the do() >> routine might hang until all the child processes are done, so I'm loathe to just run >> the orphanskill after the do_chroot() is finished (I suspect twisty lines of logic, >> all alike). Seems like we can do a couple of things: >> >> 1. Copy mock-helper into each chroot, so it's available for orphanskill >> 2. Back out the orphanskill logic and try again >> >> Option #1 is somewhat easy, if kinda ugly (not sure I like the idea of scattering a >> setuid-root program into all our build roots). Option #2 requires that we look at the >> code in all the do_* and do() routines to make sure that orphanskill runs when we >> need it to. Ideally I'd like to insure that orphanskill runs *outside* the chroot and >> that it's not needed to keep self.do() from hanging. >> >> What you guys think? > > How about we just run two commands in a row? I see the comment but don't > really see why. Line 973, we dont need to run orphanskill if it isnt > chroot. For the my.do_chroot() on line 975, it looks like we could just > do a my.do_chroot() followed by a normal os.system(). That was my first thought, but I was concerned that we might be missing something subtle in the timeout code (hence my email to you :)). If you think we can just run the orphanskill stuff after running the do_chroot() then I say that's the way to go. > > The problem it is trying to fix is if the rpmbuild process spawns child > processes that fork and never exit. I believe this was seen in some code > that was running in the rpmbuild as a unit test? > > We should also be cc-ing fedora-buildsys-list. (done) wups (looks shamefaced) > > I also understand Jesse's sentiment to just back it out. If it is going > to take more than a day or two to fix, we could just back it out. Let's try running it right after the rpmbuild. If that doesn't work right, then we can just comment it out while we take a closer look at it. Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG/BxuqA4JVb61b9cRAkAIAJ0cusVhhetlhnNJA5RWTkcD4cHG5wCcD5Rb aQo57BZBD1ME0FLDj6nARq4= =itpW -----END PGP SIGNATURE-----

2 1

Re: problems with orphanskill feature
by Michael E Brown 27 Sep '07

27 Sep '07

On Thu, Sep 27, 2007 at 01:30:15PM -0500, Clark Williams wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Michael/Jesse, > > We're seeing odd failures with mock and the orphanskill feature. What seems to happen > is that the '; mock-helper orphanskill <rootdir>' string is tacked onto a command > which is passed to do_chroot() and after the main command is run, an attempt is made > to run mock-helper (which is not installed in the chroot). So people are seeing a > "File not found" message after a successful command. > > Now it looks like to me that part of the reason for an orphanskill was that the do() > routine might hang until all the child processes are done, so I'm loathe to just run > the orphanskill after the do_chroot() is finished (I suspect twisty lines of logic, > all alike). Seems like we can do a couple of things: > > 1. Copy mock-helper into each chroot, so it's available for orphanskill > 2. Back out the orphanskill logic and try again > > Option #1 is somewhat easy, if kinda ugly (not sure I like the idea of scattering a > setuid-root program into all our build roots). Option #2 requires that we look at the > code in all the do_* and do() routines to make sure that orphanskill runs when we > need it to. Ideally I'd like to insure that orphanskill runs *outside* the chroot and > that it's not needed to keep self.do() from hanging. > > What you guys think? How about we just run two commands in a row? I see the comment but don't really see why. Line 973, we dont need to run orphanskill if it isnt chroot. For the my.do_chroot() on line 975, it looks like we could just do a my.do_chroot() followed by a normal os.system(). The problem it is trying to fix is if the rpmbuild process spawns child processes that fork and never exit. I believe this was seen in some code that was running in the rpmbuild as a unit test? We should also be cc-ing fedora-buildsys-list. (done) I also understand Jesse's sentiment to just back it out. If it is going to take more than a day or two to fix, we could just back it out. -- Michael

2 1

koji initialization, docs
by aberoham＠gmail.com 20 Sep '07

20 Sep '07

I have latest trunk version of koji, kojid, kojira, etc setup and running. But now what? Does any administration documentation exist? To figure out how to put gas in this engine do I really have to read the source? Thanks.

6 7

[PATCH koji] added koji-helper setuid program
by Enrico Scholz 17 Sep '07

17 Sep '07

This patch adds a 'koji-helper' setuid program which implements the following methods: * koji-helper rmrf <dir> removes everything under <dir>, inclusive <dir>. It does not cross filesystem borders * koji-helper rmtree <dir> removes everything under <dir>, but not <dir> itself. It does not cross filesystem borders Methods above are implemented to replace the python 'safe_rmtree()' method which was never safe, nor will work when 'kojid' is running as non-root. Signed-off-by: Enrico Scholz <enrico.scholz(a)informatik.tu-chemnitz.de> --- Makefile | 15 ++- builder/kojid | 53 +++-------- koji.spec | 3 - src/koji-helper.c | 260 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 286 insertions(+), 45 deletions(-) diff --git a/Makefile b/Makefile index cd88d4b..2a239fd 100644 --- a/Makefile +++ b/Makefile @@ -2,6 +2,8 @@ NAME=koji SPECFILE = $(firstword $(wildcard *.spec)) SUBDIRS = hub builder koji cli docs util www +sbindir = /usr/sbin + ifdef DIST DIST_DEFINES := --define "dist $(DIST)" endif @@ -52,11 +54,14 @@ ifndef TAG TAG=$(NAME)-$(VERSION)-$(RELEASE) endif -_default: - @echo "read the makefile" +all: src/koji-helper + +src/koji-helper: src/koji-helper.c + $(CC) $(CFLAGS) $< -o $@ clean: rm -f *.o *.so *.pyc *~ koji*.bz2 koji*.src.rpm + rm -f src/koji-helper rm -rf koji-$(VERSION) for d in $(SUBDIRS); do make -s -C $$d clean; done @@ -100,14 +105,16 @@ force-tag:: # @$(MAKE) tag TAG_OPTS="-F $(TAG_OPTS)" DESTDIR ?= / -install: +install: all @if [ "$(DESTDIR)" = "" ]; then \ echo " "; \ echo "ERROR: A destdir is required"; \ exit 1; \ fi - mkdir -p $(DESTDIR) + mkdir -p $(DESTDIR)$(sbindir) + + install -p -m0710 src/koji-helper $(DESTDIR)$(sbindir) for d in $(SUBDIRS); do make DESTDIR=`cd $(DESTDIR); pwd` \ -C $$d install; [ $$? = 0 ] || exit 1; done diff --git a/builder/kojid b/builder/kojid index 6e973fe..5940954 100755 --- a/builder/kojid +++ b/builder/kojid @@ -150,35 +150,17 @@ def log_output(path, args, outfile, uploadpath, cwd=None, logerror=0, append=0, outfd.close() return status[1] -def safe_rmtree(path, unmount=False, strict=True): +def safe_rmtree(path, strict=True, op='rmrf'): logger = logging.getLogger("koji.build") - #safe remove: with -xdev the find cmd will not cross filesystems - # (though it will cross bind mounts from the same filesystem) - if not os.path.exists(path): - logger.debug("No such path: %s" % path) - return - if unmount: - umount_all(path) - #first rm -f non-directories - logger.debug('Scrubbing files in %s' % path) - rv = os.system("find '%s' -xdev \\! -type d -print0 |xargs -0 rm -f" % path) - msg = 'file removal failed (code %r) for %s' % (rv,path) - if rv != 0: - logger.warn(msg) - if strict: - raise koji.GenericError, msg - else: - return rv - #them rmdir directories - #with -depth, we start at the bottom and work up - logger.debug('Scrubbing directories in %s' % path) - rv = os.system("find '%s' -xdev -depth -type d -print0 |xargs -0 rmdir" % path) - msg = 'dir removal failed (code %r) for %s' % (rv,path) - if rv != 0: - logger.warn(msg) - if strict: - raise koji.GenericError, msg - return rv + rc = os.spawnvp(os.P_WAIT, '/usr/sbin/koji-helper', ['/usr/sbin/koji-helper', op, path]) + if rc!=0: + msg = "directory removal failed (code %r) for %s" % (rc,path) + logger.warn(msg) + if strict: + raise koji.GenericError, msg + else: + return rc + return rc def umount_all(topdir): "Unmount every mount under topdir" @@ -635,7 +617,7 @@ class TaskManager(object): if age > 3600*24: #dir untouched for a day self.logger.info("Removing buildroot: %s" % desc) - if topdir and safe_rmtree(topdir, unmount=True, strict=False) != 0: + if topdir and safe_rmtree(topdir, strict=False) != 0: continue #also remove the config try: @@ -644,15 +626,7 @@ class TaskManager(object): self.logger.warn("%s: can't remove config: %s" % (desc, e)) elif age > 120: if rootdir: - try: - flist = os.listdir(rootdir) - except OSError, e: - self.logger.warn("%s: can't list rootdir: %s" % (desc, e)) - continue - if flist: - self.logger.info("%s: clearing rootdir" % desc) - for fn in flist: - safe_rmtree("%s/%s" % (rootdir,fn), unmount=True, strict=False) + safe_rmtree(rootdir, strict=False, op='rmtree') else: self.logger.debug("Recent buildroot: %s: %i seconds" % (desc,age)) self.logger.debug("Local buildroots: %d" % len(local_br)) @@ -1211,8 +1185,7 @@ class BaseTaskHandler(object): def removeWorkdir(self): if self.workdir is None: return - safe_rmtree(self.workdir, unmount=False, strict=True) - #os.spawnvp(os.P_WAIT, 'rm', ['rm', '-rf', self.workdir]) + os.spawnvp(os.P_WAIT, 'rm', ['rm', '-rf', self.workdir]) def wait(self, subtasks=None, all=False, failany=False): """Wait on subtasks diff --git a/koji.spec b/koji.spec index 13d3bf0..f14bb6e 100644 --- a/koji.spec +++ b/koji.spec @@ -16,7 +16,6 @@ Group: Applications/System URL: http://hosted.fedoraproject.org/projects/koji Source: %{name}-%{PACKAGE_VERSION}.tar.bz2 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) -BuildArch: noarch Requires: python-krbV >= 1.0.13 Requires: rpm-python Requires: pyOpenSSL @@ -89,6 +88,7 @@ koji-web is a web UI to the Koji system. %setup -q %build +make CFLAGS="$CFLAGS" CC="%__cc" all %install rm -rf $RPM_BUILD_ROOT @@ -125,6 +125,7 @@ rm -rf $RPM_BUILD_ROOT %files builder %defattr(-,root,root) +%attr(4710,root,kojibuilder) %_sbindir/koji-helper %{_sbindir}/kojid %{_initrddir}/kojid %config(noreplace) %{_sysconfdir}/sysconfig/kojid diff --git a/src/koji-helper.c b/src/koji-helper.c new file mode 100644 index 0000000..a3d0921 --- /dev/null +++ b/src/koji-helper.c @@ -0,0 +1,260 @@ +/* --*- c -*-- + * Copyright (C) 2007 Enrico Scholz <enrico.scholz(a)informatik.tu-chemnitz.de> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; version 3 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#define _GNU_SOURCE + +#ifndef MOCK_ROOT +# define MOCK_ROOT "/var/lib/mock" +#endif + +#include <sys/stat.h> +#include <string.h> +#include <stdio.h> +#include <stdlib.h> +#include <errno.h> +#include <unistd.h> +#include <dirent.h> +#include <fcntl.h> +#include <stdbool.h> + +static int __attribute__((__nonnull__(1, 2))) +safe_chdir(char const *path, struct stat const *exp_st) +{ + struct stat cur_st; + + if (strchr(path, '/')) { + fprintf(stderr, "safe_chdir(): invalid char in path '%s'\n", path); + return -1; + } + + if (strcmp(path, "..")==0) { + fprintf(stderr, "safe_chdir(): parent dir referred\n"); + return -1; + } + + if (chdir(path) < 0) { + fprintf(stderr, "chdir(%s): %s\n", path, strerror(errno)); + return -1; + } + + if (stat(".", &cur_st) < 0) { + fprintf(stderr, "stat(%s): %s\n", path, strerror(errno)); + return -2; + } + + if (cur_st.st_dev != exp_st->st_dev || + cur_st.st_ino != exp_st->st_ino) { + fprintf(stderr, "RACE: path '%s' changed before chdir()\n", path); + return -2; + } + + return 0; +} + +static int +rmrf_cwd(struct stat *cwd_st) +{ + DIR *cwd = opendir("."); + int rc = -1; + + if (!cwd) { + perror("opendir()"); + return -1; + } + + for (;;) { + struct dirent *ent = readdir(cwd); + struct stat st; + + if (!ent) + break; + + if (ent->d_name[0] == '.' && + (ent->d_name[1] == '\0'|| + (ent->d_name[1] == '.' && ent->d_name[2] == '\0'))) + continue; /* skip '.' and '..' */ + + if (lstat(ent->d_name, &st) < 0) { + fprintf(stderr, "rmrf_cwd: lstat(%s): %s\n", + ent->d_name, strerror(errno)); + continue; + } + + if (cwd_st && cwd_st->st_dev != st.st_dev) + continue; /* do not cross devices */ + else if (S_ISDIR(st.st_mode)) { + switch (safe_chdir(ent->d_name, &st)) { + case -1: continue; + case -2: break; + default: rmrf_cwd(&st); break; + } + + if (fchdir(dirfd(cwd)) < 0) { + perror("rmrf_cwd: fchdir()"); + goto err; + } + + if (rmdir(ent->d_name) < 0) { + fprintf(stderr, "rmrf_cwd: rmdir(%s): %s\n", + ent->d_name, strerror(errno)); + continue; + } + } else if (unlink(ent->d_name) < 0) { + fprintf(stderr, "rmrf_cwd: unlink(%s): %s\n", + ent->d_name, strerror(errno)); + continue; + } + } + + rc = 0; +err: + closedir(cwd); + return rc; +} + +static int +safe_chdir_subpath(char const *path_c, size_t path_len) +{ + char path[path_len+1]; + char *ptr = path; + int rc = 0; + + if (path_len == 0) + return 0; + + strncpy(path, path_c, path_len); + path[path_len] = '\0'; + + while (ptr) { + char *new_ptr = strsep(&ptr, "/"); + struct stat st; + + if (*new_ptr == '\0') + continue; /* skip empty path components + * (e.g. double /) */ + + if (lstat(new_ptr, &st) < 0) { + fprintf(stderr, "stat(%s): %s\n", + new_ptr, strerror(errno)); + rc = -1; + } else if (!S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode)) { + fprintf(stderr, "safe_chdir_subpath(): invalid mode of '%s': %04x\n", + new_ptr, st.st_mode); + rc = -1; + } else + rc = safe_chdir(new_ptr, &st); + + if (rc < 0) + break; + } + + return rc; +} + +/* Usage: rmrf <dir> */ +static int +do_rmrf(int argc, char *argv[], bool remove_parent_dir) +{ + char const *dir; + size_t dir_len; + char const *last_path; + int parent_fd; + + if (argc != 2) { + fprintf(stderr, "wrong number of parameters for 'rmrf' operation\n"); + return EXIT_FAILURE; + } + + dir = argv[1]; + + /* strip leading MOCK_ROOT; it's a little bit hacky but required to + * keep backward compatibility */ + if (strncmp(dir, MOCK_ROOT, sizeof(MOCK_ROOT)-1) == 0) + dir += sizeof(MOCK_ROOT)-1; + + while (*dir == '/') + ++dir; /* strip leading '/' */ + + dir_len = strlen(dir); + while (dir_len>0 && dir[dir_len-1] == '/') + --dir_len; /* strip trailing '/' */ + + last_path = dir + dir_len; + while (last_path > dir && last_path[-1] != '/') + --last_path; + + if (dir_len == 0) { + fprintf(stderr, "do_rmrf(): empty path\n"); + return EXIT_FAILURE; + } + if (dir_len > 255) { + fprintf(stderr, "pathname too long\n"); + return EXIT_FAILURE; + } + + + /* real work begins here... */ + + if (chdir(MOCK_ROOT) < 0) { + perror("chdir(<MOCK_ROOT>)"); + return EXIT_FAILURE; + } + + if (last_path > dir && /* else, it would be a noop */ + safe_chdir_subpath(dir, last_path - dir) < 0) + return EXIT_FAILURE; + + parent_fd = open(".", O_RDONLY|O_DIRECTORY); + if (parent_fd < 0) { + perror("open(<MOCK_ROOT>)"); + return EXIT_FAILURE; + } + + if (safe_chdir_subpath(last_path, dir+dir_len - last_path + 1) < 0) + return EXIT_FAILURE; + + /* we are now *in* the given path */ + if (rmrf_cwd(NULL) < 0) + return EXIT_FAILURE; + + if (fchdir(parent_fd) < 0) { + perror("fchdir(<parent>)"); + return EXIT_FAILURE; + } + + if (remove_parent_dir && + rmdir(last_path) < 0) + return EXIT_FAILURE; + + return EXIT_SUCCESS; +} + +int main(int argc, char *argv[]) +{ + if (argc < 2) { + fprintf(stderr, "not enough parameters\n"); + return EXIT_FAILURE; + } + + if (strcmp(argv[1], "rmrf") == 0) + return do_rmrf(argc-1, argv+1, true); + else if (strcmp(argv[1], "rmtree") == 0) + return do_rmrf(argc-1, argv+1, false); + else + fprintf(stderr, "unknown argument\n"); + + return EXIT_FAILURE; +}

2 2

[PATCH koji] Added support for building from git repositories
by Enrico Scholz 17 Sep '07

17 Sep '07

This patch adds support for building from git:// respositories. It is used like koji call build 'git://<REPOSITORY>?<PACKAGE>#<TAG>' <TARGET> The repository specified above must have the following layout: <BASE_DIR>/<REPOSITORY> |- common/ | `- .git/ |- ... |- <PACKAGE> | `- .git/ |- ... This means that every package is hold in an own git repository. There must be a 'common' repository which contains files required for building the srpm. There is no explicit support for branches; the patch just checks out the given tag and builds from it. Perhaps it can be enhanced to check whether branch-point is some child of the tag... TODO: the git and cvs handlers share some common, non trivial code. This should be generalized. TODO: use a separate user account for the 'git' and 'make srpm' operations; currently they are executed under the uid 'kojid' is running (which is 'root). Hence, a '%(/sbin/killall5)' in the spec file can bring down the box. Signed-off-by: Enrico Scholz <enrico.scholz(a)informatik.tu-chemnitz.de> --- builder/kojid | 130 +++++++++++++++++++++++++++++++++++++++++++- cli/koji | 9 ++- koji.spec | 2 - koji/__init__.py | 11 ++++ www/kojiweb/taskinfo.chtml | 7 ++ www/kojiweb/tasks.chtml | 1 6 files changed, 153 insertions(+), 7 deletions(-) diff --git a/builder/kojid b/builder/kojid index 5940954..bbb9774 100755 --- a/builder/kojid +++ b/builder/kojid @@ -1396,7 +1396,7 @@ class ChainBuildTask(BaseTaskHandler): subtasks = [] build_tasks = [] for src in build_level: - if src.startswith('cvs://'): + if src.startswith('cvs://') or src.startswith('git://'): task_id = session.host.subtask(method='build', arglist=[src, target, opts], parent=self.id) @@ -1437,8 +1437,10 @@ class BuildTask(BaseTaskHandler): raise koji.BuildError, "arch_override is only allowed for scratch builds" task_info = session.getTaskInfo(self.id) # only allow admins to perform non-scratch builds from srpm - if not src.startswith('cvs://') and not opts.get('scratch') \ - and not 'admin' in session.getUserPerms(task_info['owner']): + if not src.startswith('cvs://') and \ + not src.startswith('git://') and \ + not opts.get('scratch') and \ + not 'admin' in session.getUserPerms(task_info['owner']): raise koji.BuildError, "only admins may peform non-scratch builds from srpm" target_info = session.getBuildTarget(target) if not target_info: @@ -1496,6 +1498,8 @@ class BuildTask(BaseTaskHandler): if isinstance(src,str): if src.startswith('cvs://'): return self.getSRPMFromCVS(src) + if src.startswith('git://'): + return self.getSRPMFromGit(src) else: #assume this is a path under uploads return src @@ -1514,6 +1518,17 @@ class BuildTask(BaseTaskHandler): srpm = result['srpm'] return srpm + def getSRPMFromGit(self, url): + #TODO - allow different ways to get the srpm + task_id = session.host.subtask(method='buildSRPMFromGit', + arglist=[url], + label='srpm', + parent=self.id) + # wait for subtask to finish + result = self.wait(task_id)[task_id] + srpm = result['srpm'] + return srpm + def readSRPMHeader(self, srpm): #srpm arg should be a path relative to <BASEDIR>/work global options @@ -1767,6 +1782,115 @@ class TagBuildTask(BaseTaskHandler): exctype, value = sys.exc_info()[:2] session.host.tagNotification(False, tag_id, fromtag, build_id, user_id, ignore_success, "%s: %s" % (exctype, value)) raise e + + +class BuildSRPMFromGitTask(BaseTaskHandler): + + Methods = ['buildSRPMFromGit'] + _taskWeight = 0.75 + + def spec_sanity_checks(self, filename): + spec = open(filename).read() + for tag in ("Packager", "Distribution", "Vendor"): + if re.match("%s:" % tag, spec, re.M): + raise koji.BuildError, "%s is not allowed to be set in spec file" % tag + for tag in ("packager", "distribution", "vendor"): + if re.match("%%define\s+%s\s+" % tag, spec, re.M): + raise koji.BuildError, "%s is not allowed to be defined in spec file" % tag + + def handler(self,url): + if not url.startswith('git://'): + raise koji.BuildError("invalid git URL: %s" % url) + + # Hack it because it refuses to parse it properly otherwise + scheme, netloc, path, params, query, fragment = urlparse.urlparse('http'+url[3:]) + if not (netloc and path and fragment and query): + raise koji.BuildError("invalid git URL: %s" % url) + + # Steps: + # 1. GIT clone into tempdir + # 3. Run 'make srpm' + + gitdir = self.workdir + '/git' + koji.ensuredir(gitdir) + logfile = self.workdir + "/srpm.log" + uploadpath = self.getUploadDir() + sourcedir = '%s/%s' % (gitdir, query) + + cmd = ['git', 'clone', 'git://%s%s/%s' % (netloc, path, query), query] + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1): + output = "(none)" + try: + output = open(logfile).read() + except IOError: + pass + raise koji.BuildError, "Error with clone 'git://%s%s/%s: %s" % (netloc, path, query, output) + + cmd = ['git', 'clone', 'git://%s%s/common' % (netloc, path)] + self.logger.debug("executing: %s" % cmd) + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1): + output = "(none)" + try: + output = open(logfile).read() + except IOError: + pass + raise koji.BuildError, "Error with clone 'git://%s%s/common: %s" % (netloc, path, output) + + cmd = ['git', 'ls-files', '-t', '-s'] + log_output(cmd[0], cmd, logfile, uploadpath, cwd='%s/common' % gitdir, logerror=1, append=1) + + cmd = ['git', 'checkout', '-b', 'kojibuild', fragment] + self.logger.debug("executing: %s" % cmd) + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=sourcedir, logerror=1, append=1): + output = "(none)" + try: + output = open(logfile).read() + except IOError: + pass + raise koji.BuildError, "Error with checking out tag '%s' from git://%s%s/%s: %s" % (fragment, netloc, path, query, output) + + cmd = ['git', 'ls-files', '-t', '-s'] + log_output(cmd[0], cmd, logfile, uploadpath, cwd=sourcedir, logerror=1, append=1) + + spec_files = glob.glob("%s/*.spec" % sourcedir) + if len(spec_files) == 0: + raise koji.BuildError("No spec file found") + elif len(spec_files) > 1: + raise koji.BuildError("Multiple spec files found: %s" % spec_files) + spec_file = spec_files[0] + + # Run spec file sanity checks. Any failures will throw a BuildError + self.spec_sanity_checks(spec_file) + + #build srpm + cmd = ['make', '-C', sourcedir, 'srpm', '_KOJI=1.2.2', '_KOJI_TAG=%s' % fragment] + self.logger.debug("executing: %s" % cmd) + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1): + raise koji.BuildError, "Error building SRPM" + + srpms = glob.glob('%s/*.src.rpm' % sourcedir) + if len(srpms) == 0: + raise koji.BuildError, "No srpms found in %s" % sourcedir + elif len(srpms) > 1: + raise koji.BuildError, "Multiple srpms found in %s: %s" % (sourcedir, ", ".join(srpms)) + else: + srpm = srpms[0] + + # check srpm name + h = koji.get_rpm_header(srpm) + name = h[rpm.RPMTAG_NAME] + version = h[rpm.RPMTAG_VERSION] + release = h[rpm.RPMTAG_RELEASE] + srpm_name = "%(name)s-%(version)s-%(release)s.src.rpm" % locals() + if srpm_name != os.path.basename(srpm): + raise koji.BuildError, 'srpm name mismatch: %s != %s' % (srpm_name, os.path.basename(srpm)) + + #upload srpm and return + self.uploadFile(srpm) + return { + 'srpm' : "%s/%s" % (uploadpath, srpm_name), + 'log' : "%s/srpm.log" % uploadpath, + } class BuildSRPMFromCVSTask(BaseTaskHandler): diff --git a/cli/koji b/cli/koji index 718b6e5..4f05500 100755 --- a/cli/koji +++ b/cli/koji @@ -669,7 +669,7 @@ def handle_build(options, session, args): if build_opts.background: #relative to koji.PRIO_DEFAULT priority = 5 - if not source.startswith('cvs://'): + if not (source.startswith('cvs://') or source.startswith('git://')): # only allow admins to perform non-scratch builds from srpm if not opts['scratch'] and not session.hasPerm('admin'): parser.error(_("builds from srpm must use --scratch")) @@ -741,7 +741,7 @@ def handle_chain_build(options, session, args): if build_level: src_list.append(build_level) build_level = [] - elif src.startswith('cvs://'): + elif src.startswith('cvs://') or src.startswith('git://'): build_level.append(src) elif '/' not in src and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R @@ -2432,8 +2432,13 @@ def _parseTaskParams(session, method, task_id): if method == 'buildFromCVS': lines.append("CVS URL: %s" % params[0]) lines.append("Build Target: %s" % params[1]) + elif method == 'buildFromGit': + lines.append("GIT URL: %s" % params[0]) + lines.append("Build Target: %s" % params[1]) elif method == 'buildSRPMFromCVS': lines.append("CVS URL: %s" % params[0]) + elif method == 'buildSRPMFromGit': + lines.append("GIT URL: %s" % params[0]) elif method == 'multiArchBuild': lines.append("SRPM: %s/work/%s" % (options.topdir, params[0])) lines.append("Build Target: %s" % params[1]) diff --git a/koji.spec b/koji.spec index f14bb6e..7d4d643 100644 --- a/koji.spec +++ b/koji.spec @@ -49,7 +49,7 @@ Requires(post): /sbin/service Requires(preun): /sbin/chkconfig Requires(preun): /sbin/service Requires(pre): /usr/sbin/useradd -Requires: cvs +Requires: cvs git-core make Requires: rpm-build Requires: redhat-rpm-config Requires: createrepo >= 0.4.10 diff --git a/koji/__init__.py b/koji/__init__.py index d808126..5789220 100644 --- a/koji/__init__.py +++ b/koji/__init__.py @@ -1517,6 +1517,11 @@ def taskLabel(taskInfo): if source.startswith('cvs://'): source = source[source.rfind('/') + 1:] source = source.replace('#', ':') + elif source.startswith('git://'): + source = source[source.rfind('?') + 1:] + tmp = source[:source.find('#')] + source = source[source.find('#') + 1:] + source = '%s:%s' % (tmp, source[source.rfind('/') + 1:]) else: source = os.path.basename(source) extra = '%s, %s' % (target, source) @@ -1526,6 +1531,12 @@ def taskLabel(taskInfo): url = url[url.rfind('/') + 1:] url = url.replace('#', ':') extra = url + elif method == 'buildSRPMFromGit': + if taskInfo.has_key('request'): + url = taskInfo['request'][0] + url = url[url.rfind('/') + 1:] + url = url.replace('#', ':') + extra = url elif method == 'buildArch': if taskInfo.has_key('request'): srpm, tagID, arch = taskInfo['request'][:3] diff --git a/www/kojiweb/taskinfo.chtml b/www/kojiweb/taskinfo.chtml index 76df4bc..dd5cf72 100644 --- a/www/kojiweb/taskinfo.chtml +++ b/www/kojiweb/taskinfo.chtml @@ -67,6 +67,11 @@ Build Target: <a href="buildtargetinfo?name=$params[1]">$params[1]</a> #elif $task.method == 'buildSRPMFromCVS' CVS URL: $params[0] + #elif $task.method == 'buildFromGit' + GIT URL: $params[0] + Build Target: <a href="buildtargetinfo?name=$params[1]">$params[1]</a> + #elif $task.method == 'buildSRPMFromGit' + GIT URL: $params[0] #elif $task.method == 'multiArchBuild' SRPM: $params[0] Build Target: <a href="buildtargetinfo?name=$params[1]">$params[1]</a> @@ -276,7 +281,7 @@ $cgi.escape($result.faultString.strip()) <a href="getfile?taskID=$task.id&name=$urllib.quote($filename)">$filename</a> #end for #if $task.state not in ($koji.TASK_STATES.CLOSED, $koji.TASK_STATES.CANCELED, $koji.TASK_STATES.FAILED) and \ - $task.method in ('buildSRPMFromCVS', 'buildArch', 'createrepo') + $task.method in ('buildSRPMFromCVS', 'buildSRPMFromGit', 'buildArch', 'createrepo') <a href="watchlogs?taskID=$task.id">Watch logs</a> #end if diff --git a/www/kojiweb/tasks.chtml b/www/kojiweb/tasks.chtml index 70de06c..32ca6ad 100644 --- a/www/kojiweb/tasks.chtml +++ b/www/kojiweb/tasks.chtml @@ -104,6 +104,7 @@ All <option value="all" #if $method == 'all' then 'selected="selected"' else ''#>all</option> <option value="build" #if $method == 'build' then 'selected="selected"' else ''#>build</option> <option value="buildSRPMFromCVS" #if $method == 'buildSRPMFromCVS' then 'selected="selected"' else ''#>buildSRPMFromCVS</option> + <option value="buildSRPMFromGit" #if $method == 'buildSRPMFromGit' then 'selected="selected"' else ''#>buildSRPMFromGit</option> <option value="buildArch" #if $method == 'buildArch' then 'selected="selected"' else ''#>buildArch</option> <option value="buildNotification" #if $method == 'buildNotification' then 'selected="selected"' else ''#>buildNotification</option> <option value="tagBuild" #if $method == 'tagBuild' then 'selected="selected"' else ''#>tagBuild</option>

3 3

No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr'
by Sergio Monteiro Basto 15 Sep '07

15 Sep '07

Hi, I use updated fedora 7 what I could do ? pungi -c /etc/pungi/f7-everything.i386 stops width : INFO:pypungi.pungi:Running /usr/lib/anaconda-runtime/pkgorder /srv/pungi/f7/7/Everything/i386/os i386 Fedora ERROR:pypungi.pungi:Got an error from /usr/lib/anaconda-runtime/pkgorder ERROR:pypungi.pungi:Traceback (most recent call last): File "/usr/lib/anaconda-runtime/pkgorder", line 169, in <module> addGroups(ds, ["core", "base", "text-internet"]) File "/usr/lib/anaconda-runtime/pkgorder", line 105, in addGroups processTransaction(ds) File "/usr/lib/anaconda-runtime/pkgorder", line 77, in processTransaction ds.populateTs(keepold=0) File "/usr/lib/python2.5/site-packages/yum/depsolve.py", line 201, in populateTs self.downloadHeader(txmbr.po) File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 863, in downloadHeader cache=repo.http_caching != 'none', File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 599, in getHeader cache=cache, File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 567, in _getFile http_headers=headers, File "/usr/lib/python2.5/site-packages/urlgrabber/mirror.py", line 411, in urlgrab return self._mirror_try(func, url, kw) File "/usr/lib/python2.5/site-packages/urlgrabber/mirror.py", line 397, in _mirror_try return func_ref( *(fullurl,), **kwargs ) File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 927, in urlgrab return self._retry(opts, retryfunc, url, filename) File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 845, in _retry r = apply(func, (opts,) + args, {}) File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 915, in retryfunc fo._do_grab() File "/usr/lib/python2.5/site-packages/urlgrabber/grabber.py", line 1196, in _do_grab else: new_fo = open(self.filename, 'wb') IOError: [Errno 2] No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr' Thanks, -- Sérgio M. B.

6 23

Koji
by Oscar Bacho 14 Sep '07

14 Sep '07

Hi list koji is a great software, but which are case use for koji? in the college, we are trying to create a distribution based on Fedora. Att Oscar Bacho Instituto Tecnologico de Acapulco

3 3

[patch 3/3] Add a --prioritize option
by Mark McLoughlin 03 Sep '07

03 Sep '07

In the absence of kickstart support for specifying repo priorities, perhaps the best we can do is make the order of repos imply their priority. This patch adds a --prioritize command line option which causes pungi to assign priorities to each repo based on the order they were supplied. Signed-off-by: Mark McLoughlin <markmc(a)redhat.com> Index: pungi/pypungi/gather.py =================================================================== --- pungi.orig/pypungi/gather.py +++ pungi/pypungi/gather.py @@ -47,7 +47,7 @@ class PungiYum(yum.YumBase): pass class Gather(pypungi.PungiBase): - def __init__(self, config, ksparser): + def __init__(self, config, ksparser, prioritize = False): pypungi.PungiBase.__init__(self, config) # Set our own logging name space @@ -119,6 +119,12 @@ class Gather(pypungi.PungiBase): thisrepo.enablegroups = True if not repo.priority is None: thisrepo.priority = repo.priority + elif prioritize: + if hasattr(thisrepo, "priority"): + thisrepo.priority = len(self.ayum.repos.repos) + else: + self.logger.warning('The yum-priorities plugin is needed to support prioritizing repos') + prioritize = False self.ayum.repos.add(thisrepo) self.ayum.repos.enableRepo(thisrepo.id) self.ayum._getRepos(thisrepo=thisrepo.id, doSetup = True) Index: pungi/pungi =================================================================== --- pungi.orig/pungi +++ pungi/pungi @@ -62,7 +62,7 @@ def main(): # Actually do work. if not opts.sourceisos: if opts.do_all or opts.do_gather: - mygather = pypungi.gather.Gather(config, ksparser) + mygather = pypungi.gather.Gather(config, ksparser, opts.prioritize) mygather.getPackageObjects() mygather.downloadPackages() mygather.makeCompsFile() @@ -143,6 +143,8 @@ if __name__ == '__main__': help='disable gathering of source packages (optional)') parser.add_option("--sourceisos", default=False, action="store_true", dest="sourceisos", help='Create the source isos (other arch runs must be done)') + parser.add_option("--prioritize", default=False, action="store_true", dest="prioritize", + help='Prioritize yum repositories in the order they are specified') parser.add_option("-c", "--config", dest="config", help='Path to kickstart config file') --

1 0

[patch 2/3] Support repository priorities
by Mark McLoughlin 03 Sep '07

03 Sep '07

Support yum repository priorities supplied through kickstart. Signed-off-by: Mark McLoughlin <markmc(a)redhat.com> Index: pungi/pypungi/gather.py =================================================================== --- pungi.orig/pypungi/gather.py +++ pungi/pypungi/gather.py @@ -117,6 +117,8 @@ class Gather(pypungi.PungiBase): self.logger.info('URL for repo %s is %s' % (thisrepo.name, thisrepo.baseurl)) thisrepo.basecachedir = self.ayum.conf.cachedir thisrepo.enablegroups = True + if not repo.priority is None: + thisrepo.priority = repo.priority self.ayum.repos.add(thisrepo) self.ayum.repos.enableRepo(thisrepo.id) self.ayum._getRepos(thisrepo=thisrepo.id, doSetup = True) --

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys September 2007