buildsys November 2010

buildsys@lists.fedoraproject.org

26 participants
25 discussions

[BUG] Koji and Lustre
by Thomas Guthmann 09 Sep '11

09 Sep '11

Hey guys, We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py) when used with a Lustre filesystem. This function parses all attributes of every file of a build and is used when you want to display build.log/root.log through the web interface. Everything returned by listTaskOutput is returned through XML-RPC and as a result we had this : An error has occurred while processing your request. Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC limits'> Traceback (most recent call last): File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in publish_object return old_publish_object(req, object) File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 412, in publish_object return publish_object(req,util.apply_fs_data(object, req.form, req=req)) File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line 439, in apply_fs_data return object(**args) File "/usr/share/koji-web/scripts/index.py", line 649, in getfile output = server.listTaskOutput(taskID, stat=True) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718, in _callMethod raise err Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC limits'> The issue comes from the st_dev value gathered by getStat (stat). In Lustre this value can be very high and that's why it complains. To fix that, we used the same condition than st_size, we cast the value as a string. See attached patch. Example (see 'Device:' the value after the '/'): # stat build.log File: `build.log' Size: 106021 Blocks: 208 IO Block: 2097152 regular file Device: e04ae70eh/3763005198d Inode: 721664 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache) Access: 2010-07-13 10:44:40.000000000 +1000 Modify: 2010-07-12 12:54:15.000000000 +1000 Change: 2010-07-12 12:54:52.000000000 +1000 # That's what is read by listTaskOutput relfilename=build.log ATTR=st_atime GETATTR=1278981616 relfilename=build.log ATTR=st_blksize GETATTR=2097152 relfilename=build.log ATTR=st_blocks GETATTR=208 relfilename=build.log ATTR=st_ctime GETATTR=1278903292 relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\ relfilename=build.log ATTR=st_gid GETATTR=48 relfilename=build.log ATTR=st_ino GETATTR=721664 relfilename=build.log ATTR=st_mode GETATTR=33188 relfilename=build.log ATTR=st_mtime GETATTR=1278903255 relfilename=build.log ATTR=st_nlink GETATTR=1 relfilename=build.log ATTR=st_rdev GETATTR=0 relfilename=build.log ATTR=st_size GETATTR=106021 relfilename=build.log ATTR=st_uid GETATTR=48 Hope it helps, lost a good amount of time on that one :) Cheers, Thomas

2 3

Strange mock build failure due to typo
by Paul Howarth 24 Jan '11

24 Jan '11

Hi, today I have been preparing an update to perl-Math-Pari and came across a very strange build failure whilst doing a local mock build on a Fedora 13 x86_64 host. My package built successfully on x86_64 but when I tried to build for i386, the build failed during %setup but without any diagnostics. The SRPM was installed but no attempt to install its build requirements was made. The root.log showed an exit status of 0 for all commands that had been run. After much experimentation bisecting the changes I had made, I discovered that a typo in the changelog entry was the culprit: I had set the year to 2100 instead of 2010. So it would appear that somewhere in the mock/yum/rpm stack there may be a year 2038 problem waiting to bite us (though I suspect there may not be too many 32-bit builds happening by then). Seriously though, it would be nice to have better diagnostics for this and perhaps an rpmlint check for changelog entries in the future? Paul.

2 1

Speeding up koji-generated mock builds
by Alan Franzoni 15 Dec '10

15 Dec '10

Hello, I'm trying to get our koji instance faster; since we build a lot of packages, we'd like to reduce the latency. There're some things that usually make mock faster: the options I tinkered with and seem to improve performance are root_cache, yum_cache and tmpfs. All of those are disabled by default when generating mock config through koji, and there seems no way to override them: https://fedorahosted.org/koji/browser/koji/__init__.py#L1276 So here come the questions: 1) Am i missing anything? 2) are all those options intentionally locked out? 3) would a patch which let them be configured be accepted into your git repo? 4) any other way to speedup mock? Thanks! -- Alan Franzoni -- contact me at public(a)[mysurname].eu

5 7

make iso failed
by Vijay N. Majagaonkar 07 Dec '10

07 Dec '10

Hi all, I have updated my system to f14 and now i am not able to build iso out of koji, mock : mock-1.1.6-1.fc14.noarch python : python-2.7-8.fc14.1.x86_64 koji : koji-1.4.0-4.fc14.noarch [Error] Building Installation Images: ######################################## 100.0% DEBUG util.py:267: Going to replace isolinux/isolinux.cfg with /etc/revisor/conf.d/revisor-proxyone-isolinux.cfg DEBUG util.py:267: Deleted the old isolinux.cfg DEBUG util.py:267: Inserted the new isolinux.cfg DEBUG util.py:267: Removing files and directories that do not need to be on the media: DEBUG util.py:267: Traceback (most recent call last): DEBUG util.py:267: File "/usr/lib/python2.4/site-packages/revisor/__init__.py", line 441, in run DEBUG util.py:267: self.base.run() DEBUG util.py:267: File "/usr/lib/python2.4/site-packages/revisor/base.py", line 106, in run DEBUG util.py:267: self.cli.run() DEBUG util.py:267: File "/usr/lib/python2.4/site-packages/revisor/cli.py", line 44, in run DEBUG util.py:267: self.base.lift_off() DEBUG util.py:267: File "/usr/lib/python2.4/site-packages/revisor/base.py", line 1172, in lift_off DEBUG util.py:267: self.buildInstallationMedia() DEBUG util.py:267: File "/usr/lib/python2.4/site-packages/revisor/base.py", line 1487, in buildInstallationMedia DEBUG util.py:267: self.plugins.exec_hook('post_exec_buildinstall') DEBUG util.py:267: File "/usr/lib/python2.4/site-packages/revisor/plugins.py", line 191, in exec_hook DEBUG util.py:267: exec("self.%s.%s()" % (plugin,hook)) DEBUG util.py:267: File "<string>", line 1, in ? DEBUG util.py:267: File "/usr/lib/python2.4/site-packages/revisor/modremove/__init__.py", line 61, in post_exec_buildinstall DEBUG util.py:267: for file in self.cfg.rm: DEBUG util.py:267: TypeError: iteration over non-sequence [/Error] Is this something to do with python version, if I downgrade the version to previous version that is f12 it works correct, do i need to patch these packages make it work under f14 ? Thanks for your help V!jay

2 4

[PATCH 1/3] Install build deps with yum-builddep.
by Ville Skyttä 04 Dec '10

04 Dec '10

No longer need to screen-scrape resolvedep and feed that to yum install, and we have a chance to get BuildConflicts handing "for free" (when RHBZ #614191 is done in yum(-builddep)). --- mock.spec.in | 2 +- py/mock/backend.py | 33 ++++++++++++++++++++++----------- 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/mock.spec.in b/mock.spec.in index e2580ca..0949975 100644 --- a/mock.spec.in +++ b/mock.spec.in @@ -18,7 +18,7 @@ Source: https://fedorahosted.org/mock/attachment/wiki/MockTarballs/%{name}-%{ver URL: http://fedoraproject.org/wiki/Projects/Mock BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildArch: noarch -Requires: python >= 2.4, yum >= 2.4, tar, pigz, python-ctypes, python-decoratortools, usermode +Requires: python >= 2.4, yum >= 2.4, yum-utils >= 1.1.9, tar, pigz, python-ctypes, python-decoratortools, usermode Requires: createrepo Requires(pre): shadow-utils BuildRequires: python-devel diff --git a/py/mock/backend.py b/py/mock/backend.py index ba0ad14..81a98d0 100644 --- a/py/mock/backend.py +++ b/py/mock/backend.py @@ -69,6 +69,7 @@ class Root(object): self.chroot_file_contents = config['files'] self.chroot_setup_cmd = config['chroot_setup_cmd'] self.yum_path = '/usr/bin/yum' + self.yum_builddep_path = '/usr/bin/yum-builddep' self.macros = config['macros'] self.more_buildreqs = config['more_buildreqs'] self.cache_topdir = config['cache_topdir'] @@ -444,23 +445,28 @@ class Root(object): """figure out deps from srpm. call yum to install them""" try: self.uidManager.becomeUser(0, 0) + + def _yum_and_check(cmd): + output = self._yum(cmd, returnOutput=1) + for line in output.split('\n'): + if line.lower().find('No Package found for'.lower()) != -1: + raise mock.exception.BuildError, "Bad build req: %s. Exiting." % line + + # first, install pre-existing deps and configured additional ones arg_string = self.preExistingDeps for hdr in mock.util.yieldSrpmHeaders(srpms, plainRpmOk=1): # get text buildreqs - a = mock.util.requiresTextFromHdr(hdr) - b = mock.util.getAddtlReqs(hdr, self.more_buildreqs) - for item in mock.util.uniqReqs(a, b): + for item in mock.util.getAddtlReqs(hdr, self.more_buildreqs): arg_string = arg_string + " '%s'" % item - - # everything exists, okay, install them all. - # pass build reqs (as strings) to installer if arg_string != "": - output = self._yum('resolvedep %s' % arg_string, returnOutput=1) - for line in output.split('\n'): - if line.lower().find('No Package found for'.lower()) != -1: - raise mock.exception.BuildError, "Bad build req: %s. Exiting." % line + # everything exists, okay, install them all. + # pass build reqs (as strings) to installer + _yum_and_check('resolvedep %s' % arg_string) # nothing made us exit, so we continue self._yum('install %s' % arg_string, returnOutput=1) + + # install actual build dependencies + _yum_and_check("builddep '%s'" % "' '".join(srpms)) finally: self.uidManager.restorePrivs() @@ -676,7 +682,12 @@ class Root(object): if not self.online: cmdOpts = "-C" - cmd = '%s --installroot %s %s %s' % (self.yum_path, self.makeChrootPath(), cmdOpts, cmd) + # invoke yum-builddep instead of yum if cmd is builddep + exepath = self.yum_path + if cmd.startswith("builddep "): + exepath = self.yum_builddep_path + cmd = cmd[len("builddep "):] + cmd = '%s --installroot %s %s %s' % (exepath, self.makeChrootPath(), cmdOpts, cmd) self.root_log.debug(cmd) output = "" try: -- 1.7.2.3

3 9

How do folks use an SCM?
by Jon Stanley 03 Dec '10

03 Dec '10

So I have an interesting situation. I have an SCM (CVS, don't laugh) that requires kerberized gserver authentication. How do I use Koji with this? I don't mind embedding a password for a user that has read-only access to the repo somewhere, but I really don't want to if I can avoid it. Also, with the interesting requirement of a Makefile with target srpm, how do folks generate that for externally developed packages? Frankly, most of the packages that we're going to build are rebuilds of RHEL content with minor changes (sometimes a patch, sometimes just pathname changes, etc), so generating an SRPM and feeding it directly to koji is easier than maintaining some SCM layout that's foreign to us and a lookaside cache. Note that the reason we want to use koji is build reproducibility, but we'll be saving the SRPM's used in some location.

2 1

for help
by Jinze Xue 03 Dec '10

03 Dec '10

I want to move all data from a old koji build system to a new os, what's the best way to do? and who have documents about configs and steps to build cvs system for a koji build system ? send me ,thanks very much !!!!!!!

2 1

A question on Fedora Koji infrastructure
by Alan Franzoni 28 Nov '10

28 Nov '10

Hello everybody, lately we're experiencing some strange and random issues with koji on fc14 64 bit; we're still tracking down some errors and we'll later submit tickets, but my question is: which OS uses koji.fedoraproject.org main hub and builders? Is the host machine running fedora, centos, rhel or what? -- Alan Franzoni -- contact me at public(a)[mysurname].eu

3 2

Koji build slaves configuration
by Alan Franzoni 23 Nov '10

23 Nov '10

Hello, after successfully using koji on a single machine we've hit a performance bottleneck, hence we're extending our build infrastracture by adding new builders. There's a step which is not clear to me: /mnt/koji on the main machine should be remotely mounted read-only to the builders, right? is there any preferred mount method (nfs/sshfs/...) which is known not to be a source of issues? -- Alan Franzoni -- contact me at public(a)[mysurname].eu

6 6

growing /mnt/koji/work/tasks
by Alan Franzoni 23 Nov '10

23 Nov '10

Hello, after upgrading to fc14 and koji 1.4 I'm still partially experiencing an issue with the "tasks" directory; it keeps growing and contains very old builds. In the old thread it was suggested that this shouldn't happen if kojira is running: http://www.mail-archive.com/buildsys@lists.fedoraproject.org/msg00472.html In fact now that kojira is running and everything is updated the cli-build dir is perfectly clean, but the tasks dir content seems to stay there (forever?) alo. I can't see any error in kojira.log, and all files are owned by apache. koji-gc is scheduled and runs every night. Where should I look at to track my problems? -- Alan Franzoni -- contact me at public(a)[mysurname].eu

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys November 2010