buildsys November 2007

buildsys@lists.fedoraproject.org

25 participants
19 discussions

Pungi gathering - too many kernels
by Peter Åstrand 23 Nov '07

23 Nov '07

I have a KS file that looks like: %packages --nobase @core @base-x kernel @hardware-support -fedora-logos # Compose Needs anaconda-runtime #iscsi-initiator-utils memtest86+ #vnc-server %end Despite this, *all* kernels gets shipped: Pungi.Gather.INFO: Checking deps of xorg-x11-drv-nouveau.i386 yum.verbose.YumBase.DEBUG: Matched xorg-x11-server-Xorg - 1.3.0.0-33.fc8.i386 to require for xorg-x11-server-Xorg Pungi.Gather.INFO: Added xorg-x11-server-Xorg.i386 for xorg-x11-drv-nouveau.i386 yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-42.fc8.i586 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel-PAE - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel-PAE-debug - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel-debug - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel-debug - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-49.fc8.i586 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel-PAE-debug - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel-PAE - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau Pungi.Gather.INFO: Added kernel-PAE.i686 for xorg-x11-drv-nouveau.i386 Pungi.Gather.INFO: Added kernel-debug.i686 for xorg-x11-drv-nouveau.i386 Pungi.Gather.INFO: Added kernel-PAE-debug.i686 for xorg-x11-drv-nouveau.i386 Pungi.Gather.INFO: Added kernel.i586 for xorg-x11-drv-nouveau.i386 Pungi.Gather.INFO: Added kernel.i686 for xorg-x11-drv-nouveau.i386 Is this really correct? The normal "kernel" package provides kernel-drm-nouveau, so why is it necessary to pull in all other kernel packages? To me, it looks like the dependency resolver is not satisfied with one package fulfilling the dependencies, but rather includes all. Regards, --- Peter Åstrand ThinLinc Chief Developer Cendio AB http://www.cendio.se Wallenbergs gata 4 583 30 Linköping Phone: +46-13-21 46 00

3 2

Koji patches for no longer creating hardlinks for package repodata
by Jesse Keating 17 Nov '07

17 Nov '07

We've talked about ways of making new repo tasks faster, and one of the big speedups is no longer making a tree of hardlinks to run createrepo over. Recently createrepo gained[1] the ability to read from an include list. Paired with the option to specify a baseurl to make everything relative from, we can now run createrepo directly on the packages/ dir, using the include file (generated from the list of packages we /would/ have hardlinked), use an http url as the baseurl (so that static-repos continue to work) and skip the hardlink step all together. An added bonus is that when a static-repo expires, the metadata you may be working with will continue to reference valid package URLS so your transaction will continue without failure. While I was working with this code, I also noticed that we can stop creating a buildsys-build rpm and instead rely on the groupdata in the repodata and just use 'groupinstall buildsys-build' as are chroot init command. Add to that we were unnecessarily linking comps.xml files into each arch directory of a repo directory, when instead we can just reference the one we create in groups/ when creating repodata. Since we no longer need to make the buildgroups rpm, the PrepRepo task in kojid is no longer needed. I took the one part that it was still usefully doing (asking the hub to prep the repo dirs) and move it up to the NewRepo task. This saves a task creation and pickup/handoff. I introduced a config item in kojid.conf, which is pkgurl. This is used as the baseurl component to the repodata and should be the url which leads to the packages/ directory. Finally, since the repodata itself will continue to live in the same expected locations, builders can continue to consume the repodata via nfs. But since the repodata itself references http, yum will use http to download the packages to populate the buildroot. We just have to ensure that the url used in pkgurl is not only valid in the outside world (so that static-repos continue to work) but valid within the buildsystem structure as well. I've committed my changes to a git tree (cloned from tip today) here: http://jkeating.fedorapeople.org/git/koji/ Feedback would be welcome. We're going to get the createrepo change into RHEL5.2 (along with --update support) with the idea that when we refresh the Fedora buildsystem machines in early December, we'll go with RHEL5 and perhaps a newer createrepo package. Ideally we'd get the changes I've made into koji beforehand so that we can release a new build of koji during the refresh. [1] https://lists.dulug.duke.edu/pipermail/rpm-metadata/2007-November/000812.ht… -- Jesse Keating Fedora -- All my bits are free, are yours?

1 1

adding debug files to tree with pungi
by Doug Chapman 16 Nov '07

16 Nov '07

I have been working on building Fedora 8 ia64 trees with pungi. I have it working for the normal binary and source rpms but I want to be able to use it to add the debuginfo rpms to the tree. The closest I have been able to get this to work was to add *debuginfo to the %packages section of my pungi .ks file. This however just adds the debuginfo packages in 8/ia64/os/Packages/ instead of 8/ia64/debuginfo/ where I find them on other Fedora distros. Can this be done with pungi? btw, I am running the F8 version of pungi: pungi-1.1.9-1.fc8 thanks, - Doug

2 1

RE: potential patch
by Michael E Brown 16 Nov '07

16 Nov '07

Better patch: @@ -264,7 +264,7 @@ def main(retParams): log_cfg = ConfigParser.ConfigParser() logging.config.fileConfig(log_ini) log_cfg.read(log_ini) - except (IOError, OSError), e: + except (IOError, OSError, ConfigParser.NoSectionError), e: log.error("Could not find required logging config file: %s" % log_ini) sys.exit(50) About branching: If they ever upgrade fedoraproject.org to a non-ancient version of git, that would be fine. With git >= 1.5, you can delete remote branches. With older git, there is no way to remove a remote branch. Are there any other changes you are needing for cross-building? -- Michael -----Original Message----- From: Clark Williams [mailto:williams@redhat.com] Sent: Fri 11/16/2007 9:03 AM To: Brown, Michael E Subject: potential patch -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael, I'm porting our cross-build environment to use the new mock (0.8.8+) and ran into a small issues. Evidently the ConfigParser stuff doesn't barf if the input configfile doesn't exist, so you get a very strange error about "no formatters section exists" or something similar. What do you think of the change below? I'm still struggling with how I should push these sorts of changes back to the main git repo. I wonder if we shouldn't have michael,clark, and jessie branches up there so we could just push to that for review then merge to master? Dunno... Clark - From 5add76128fcc10555264392307d5da998a05401b Mon Sep 17 00:00:00 2001 From: Clark Williams <williams(a)redhat.com> Date: Thu, 15 Nov 2007 09:20:01 -0600 Subject: [PATCH] fix for no logging.ini situation - --- src/mock.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/src/mock.py b/src/mock.py index 08832dd..9954dcf 100755 - --- a/src/mock.py +++ b/src/mock.py @@ -260,12 +260,15 @@ def main(retParams): # reconfigure logging in case config file was overridden log_ini = os.path.join(config_path, config_opts["log_config_file"]) + if not os.path.exists(log_ini): + log.error("Could not find required logging config file: %s" % log_ini) + sys.exit(50) try: log_cfg = ConfigParser.ConfigParser() logging.config.fileConfig(log_ini) log_cfg.read(log_ini) except (IOError, OSError), e: - - log.error("Could not find required logging config file: %s" % log_ini) + log.error("Error configuring logging: %s" % e) sys.exit(50) # set up logging format strings - -- 1.5.3.4 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFHPbFJHyuj/+TTEp0RAga1AJ0ddE0ReUVdJpqGwSIHuFQqmhimEACgpej2 WX/yAWX3mo+J7GoS20t9JLg= =OQg0 -----END PGP SIGNATURE-----

3 2

What replaced plague
by John Villalovos 15 Nov '07

15 Nov '07

I get the impression that another program has replaced plague for doing the distributed build stuff. Can someone tell me what it is and a link to a site with more info? Thanks! John

2 1

Mock 0.8.x available
by Michael E Brown 15 Nov '07

15 Nov '07

Mock users, I have posted a new version of mock for review and comments. RPM download: http://linux.dell.com/libsmbios/download/mock/mock-0.8.1/ Git repository: http://linux.dell.com/git/mock-v2.git Compatibility: -- Mock 0.8.x+ has been written on Fedora 7 and is intended for use on Fedora 7 and higher. A limited amount of testing has been done on FC6, so please report any FC6-specific bugs and we will make best-effort attempts to fix them. -- I have tried to maintain a high degree of compatibility with mock 0.7 in cmdline args and New stuff: -- plugin system to better modularize things -- new plugins: -- Yum cache -- root cache -- ccache -- bind mount -- root cache (formerly called autocache) is now significantly smaller in size -- speed increases: mock 0.8 is now in almost every case minutes faster than 0.7, especially for multiple builds. -- uses python logging.py module for increased logging flexibility -- new command: 'mock install PACKAGE' to run a 'yum install PACKAGE' inside the buildroot -- expanded command: 'mock installdeps RPM_FILE' can now install deps for a local binary RPM. Formerlly, only SRPMS were supported. -- new option: "--cleanup-after" that can be used with "--resultdir". This will do a cleanup of the buildroot after the build. This is enabled by default such that any '--resultdir' builds will be automatically cleaned up -- All of the config files delivered with mock have been tested and should work. (The test case compiles mock using each of the config files delivered with mock.) Changed: -- cmdline requires "rebuild" argument when rebuilding srpms. Previously you could just pass srpm name. -- output has slightly changed. Mock is now slightly more verbose. -- formerly /dev from the host was bind-mounted into the chroot. This is now not enabled by default, but can be configured easily per-chroot using the bind plugin. See the 'defaults.cfg' files for config details. -- '-r CONFIG' option can no longer accept full config filenames ("config.cfg"). Leave off the '.cfg' for mock 0.8+. Formerly, config ('-r' option) could be specified using either "config.cfg" or "config" and mock would look for /etc/mock/config.cfg, automatically adding the '.cfg', if necessary. -- logs are not overwritten or truncated for --no-clean or --resultdir builds. -- cmdline options removed: --autocache, --rebuild-cache: use "--enable-plugin" or "--disable-plugin" instead --verbose, --debug, --quiet: the log files contain all the information. --verbose could possibly return. --statedir: statedir didnt have much point in life and has gone away Config files: -- Old config files will, by and large, still work. There are several options in the old config files that are no longer applicable now that 'mock-helper' has went away. The next release of mock should have warnings enabled if it sees that your config file has obsolete options. Internal Changes: -- now modularized -- mock-helper is gone. Instead there is a setuid-wrapper that calls mock.py. This vastly simplifies development. Plugin Notes: -- the plugin system is somewhat rudimentary at this time. If this becomes something that is truly useful for out-of-tree plugins or third-parties, we can formalize the interface some more. As of now, the plugin interface should have most of the hooks to do useful stuff, but some other things like the 'conduit' interface that yum uses for plugins is not present. This would formalize the interface a bit more and make sure the plugins dont grope too badly into internals they shouldnt. Future plans: -- Upstream mock git will be updated to this version next week after we sort out branching for the 0.7 release. -- This version of mock will be checked into Fedora 9 next week. -- yum integration: plan to try to use the yum API rather than the cmdline. this will enable several optimizations and speed ups due to not redownloading the yum metadata multiple times. -- config file format change: at some future point, we probably will switch to an .ini format config file. SPECIAL SECURITY WARNING: 1) default /usr/bin/mock owner is root:mock with permissions 4550 (-rwsrwx---). Thus, it may only be run by the 'root' user or members of the 'mock' group. DO NOT PUT UNTRUSTED USERS IN THE 'MOCK' GROUP! The current code (as well as all previous versions) has many easily-used mechanisms that an untrusted user could use to elevate their local privileges. IT IS NOT THE GOAL OF THE MOCK DEVELOPERS TO MAKE MOCK A TOOL THAT CAN BE USED BY UNTRUSTED LOCAL USERS. 2) All 'rpmbuild' commands, as well as the 'rpm' command to install and unpack the to-be-built SRPM are run as the user running 'mock' with no elevated privileges. These 'rpm' and 'rpmbuild' commands are run in the chroot environment with these lower privileges. Thus, it should be relatively safe to use mock to compile code from untrusted sources. That being said, MOCK HAS NOT BEEN THOROUGHLY AUDITED TO GUARANTEE THE SAFETY OF THIS. -- Michael

7 13

RHEL 5.1
by Rob Myers 09 Nov '07

09 Nov '07

I know RHEL is a bit off topic for this list, but bear with me. I import all the RHEL5 bits into koji and then use koji to build rpms for RHEL5. The release of RHEL 5.1 gave me a few kinks to work out: The RHEL5 supplementary channel includes some noarch packages which do not come with source rpms. This breaks the assumption in mash that excludearch and exclusive arch can safely be initialized on SRPMS only. I know that mash is not designed for this use case, but I thought it might be worth pointing out. The patch I used is below. diff --git a/mash/__init__.py b/mash/__init__.py index b02d6c8..6e7a496 100644 --- a/mash/__init__.py +++ b/mash/__init__.py @@ -209,6 +209,18 @@ class Mash: # now deal with noarch for pkg in noarch: for target_arch in self.config.arches: + + # if excludearch is not set this build likely has no src.rpm + # so set excludearch and exclusivearch from the binary + if pkg['build_id'] fg not in excludearch: + path = os.path.join(koji.pathinfo.build(builds_hash[pkg['build_id']]), koji.pathinfo.rpm(pkg)) + fn = open(path, 'r') + hdr = koji.get_rpm_header(fn) + excludearch[pkg['build_id']] = hdr['EXCLUDEARCH'] + exclusivearch[pkg['build_id']] = hdr['EXCLUSIVEARCH'] + fn.close() + continue + if (excludearch[pkg['build_id']] and has_any(masharch.compat[target_arch], excludearch[pkg['build_id']])) or \ (exclusivearch[pkg['build_id']] and not has_any(masharch.compat[target_arch], exclusivearch[pkg['build_id']])): print "Excluding %s.%s from %s due to EXCLUDEARCH/EXCLUSIVEARCH" % (pkg['name'], pkg['arch'], target_arch) Another problem was that I had already imported RHEL 5.1 beta, which has the same rpms in RHEL 5.1 signed with a different key- one that I do not have access to. Since I wanted the rpms written out with the release key, I added a function take the 5.1 release sighdrs and add them to the previously imported RHEL 5.1 beta rpms. Would anyone else ever need this functionality? The patch I used is below. diff --git a/hub/kojihub.py b/hub/kojihub.py index f9ba39a..c5a1261 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -4706,6 +4706,15 @@ class RootExports(object): context.session.assertPerm('sign') return add_rpm_sig(an_rpm, base64.decodestring(data)) + def addRPMSigByPath(self, an_rpm, path): + """Store a signature header for an rpm + + an_rpm: N-V-R.a (for example) + path: the path to the signed rpm + """ + context.session.assertPerm('sign') + return add_rpm_sig(an_rpm, koji.rip_rpm_sighdr(path)) + findBuildID = staticmethod(find_build_id) getTagID = staticmethod(get_tag_id) getTag = staticmethod(get_tag) Comments? rob.

3 4

[Patch] for pungi-1.1.8-1.fc8 build on fedora 7
by Sergio Monteiro Basto 06 Nov '07

06 Nov '07

Hi, To fix this error: Pungi.Gather:INFO: Finished downloading packages. Traceback (most recent call last): File "/usr/bin/pungi", line 178, in <module> main() File "/usr/bin/pungi", line 68, in main mygather.makeCompsFile() File "/usr/lib/python2.5/site-packages/pypungi/gather.py", line 397, in makeCompsFile ourcomps.writelines(compslines[start:end]) UnboundLocalError: local variable 'start' referenced before assignment

2 2

RFC - Patch - koji SCM generalization
by Rob Myers 06 Nov '07

06 Nov '07

attached is a patch that attempts to generalize checking out the components used to build an SRPM. this patch supports CVS, GIT, and SVN, but only CVS and SVN have been tested. the idea is to provide the infrastructure for different SCM systems to be configured at run-time so that users can choose their favorite system. is there a better approach? did i miss something obvious? general comments? thanks. rob.

2 8

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys November 2007