buildsys June 2011

buildsys@lists.fedoraproject.org

11 participants
9 discussions

[BUG] Koji and Lustre
by Thomas Guthmann 09 Sep '11

09 Sep '11

Hey guys, We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py) when used with a Lustre filesystem. This function parses all attributes of every file of a build and is used when you want to display build.log/root.log through the web interface. Everything returned by listTaskOutput is returned through XML-RPC and as a result we had this : An error has occurred while processing your request. Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC limits'> Traceback (most recent call last): File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in publish_object return old_publish_object(req, object) File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 412, in publish_object return publish_object(req,util.apply_fs_data(object, req.form, req=req)) File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line 439, in apply_fs_data return object(**args) File "/usr/share/koji-web/scripts/index.py", line 649, in getfile output = server.listTaskOutput(taskID, stat=True) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718, in _callMethod raise err Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC limits'> The issue comes from the st_dev value gathered by getStat (stat). In Lustre this value can be very high and that's why it complains. To fix that, we used the same condition than st_size, we cast the value as a string. See attached patch. Example (see 'Device:' the value after the '/'): # stat build.log File: `build.log' Size: 106021 Blocks: 208 IO Block: 2097152 regular file Device: e04ae70eh/3763005198d Inode: 721664 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache) Access: 2010-07-13 10:44:40.000000000 +1000 Modify: 2010-07-12 12:54:15.000000000 +1000 Change: 2010-07-12 12:54:52.000000000 +1000 # That's what is read by listTaskOutput relfilename=build.log ATTR=st_atime GETATTR=1278981616 relfilename=build.log ATTR=st_blksize GETATTR=2097152 relfilename=build.log ATTR=st_blocks GETATTR=208 relfilename=build.log ATTR=st_ctime GETATTR=1278903292 relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\ relfilename=build.log ATTR=st_gid GETATTR=48 relfilename=build.log ATTR=st_ino GETATTR=721664 relfilename=build.log ATTR=st_mode GETATTR=33188 relfilename=build.log ATTR=st_mtime GETATTR=1278903255 relfilename=build.log ATTR=st_nlink GETATTR=1 relfilename=build.log ATTR=st_rdev GETATTR=0 relfilename=build.log ATTR=st_size GETATTR=106021 relfilename=build.log ATTR=st_uid GETATTR=48 Hope it helps, lost a good amount of time on that one :) Cheers, Thomas

2 3

Kojid unable to regen-repo in Fedora 15
by Anthony Joseph Messina 05 Aug '11

05 Aug '11

I'm running a Kojid instance, freshly upgraded from Fedora 14 -> 15. My koji hub is still running Fedora 14. When running a regen-repo, I now get this error: 2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in runTask response = (handler.run(),) File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run return self.handler(*self.params,**self.opts) File "/usr/sbin/kojid", line 2552, in handler self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo) File "/usr/sbin/kojid", line 2606, in create_local_repo % parseStatus(status, ' '.join(cmd)) GenericError: failed to create repo: /usr/bin/createrepo -vd -o /tmp/koji/tasks/1796/1796/repo -u http://messinet.com/koji-mnt/packages -i /mnt/koji/repos/dist-f14-build/221/x86_64/pkglist -g /mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat /mnt/koji/packages/ exited with status 1 Any ideas on where I should go? Thanks in advance. -A -- Anthony - http://messinet.com - http://messinet.com/~amessina/gallery 8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E

4 10

[PATCH] Add support for custom checkout command
by Jan Pazdziora 20 Jul '11

20 Jul '11

Hello please consider included patch which adds support for custom command to get the sources from SCM. The reason why we needed something like this is the fact that in the Spacewalk project, we have multiple packages in one git repo, in different directories. To get .spec and .tar.gz for a particular package, a slightly different workflow is needed than koji generally supports. In our case, I wanted to be able to just use git format to get the .tar.gz from the remote repo (no local checkout needed (even if we might have one eventually to speed things up)), but before that a bit more work needs to be done to find out which subdirectory the package is in, similar to what the tito tool does (even if we do not use tito directly). Seeing that the allowed_scms already has fourth option to specify the alternative to make sources, and since we don't need to run the checkout at all, I've just added fifth option to specify custom script, which then does all the heavy lifting -- it checks if it already has the .spec and .tar.gz cached (as we build Spacewalk for four OSes from one source), it retrieves them from git.fedorahosted.org if it does not, and generally populates the sourcedir properly, short-cutting any checkout and make sources. I'd appreciate your comments about the patch, or inclusion of the patch to koji master as is. The patch has been in use at koji.spacewalkproject.org (albeit on top of koji 1.4, so in the /usr/sbin/kojid) for a week or so and it seems to achieve the goal properly. Thank you. >From 8e4072de758926a58c93226a44b08eda8b7ded87 Mon Sep 17 00:00:00 2001 From: Jan Pazdziora <jpazdziora(a)redhat.com> Date: Wed, 8 Jun 2011 10:48:08 +0200 Subject: [PATCH] Support custom checkout command to populate sourcedir from SCM. The configuration is via the fifth part of the allowed_scms configuration value: allowed_scms=git.fedorahosted.org:/git/spacewalk.git:0::/usr/local/bin/get-tgz-from-git The custom script is passed the sourcedir, host, repository, and revision as parameters. --- koji/daemon.py | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletions(-) diff --git a/koji/daemon.py b/koji/daemon.py index b6f775a..07e1be7 100644 --- a/koji/daemon.py +++ b/koji/daemon.py @@ -192,6 +192,7 @@ class SCM(object): self.revision = fragment self.use_common = True self.source_cmd = ['make', 'sources'] + self.checkout_cmd = None for scmtype, schemes in SCM.types.items(): if self.scheme in schemes: @@ -275,6 +276,9 @@ class SCM(object): else: # there was nothing after the trailing :, so they don't want to run a source_cmd at all self.source_cmd = None + if len(scm_tuple) >= 5: + if scm_tuple[4]: + self.checkout_cmd = scm_tuple[4].split(',') break else: self.logger.warn('Ignoring incorrectly formatted SCM host:repository: %s' % allowed_scm) @@ -311,7 +315,10 @@ class SCM(object): raise koji.BuildError, 'Error running %s command "%s", see %s for details' % \ (self.scmtype, ' '.join(cmd), os.path.basename(logfile)) - if self.scmtype == 'CVS': + if self.checkout_cmd: + module_checkout_cmd = self.checkout_cmd + [sourcedir, self.host, self.repository, self.revision] + + elif self.scmtype == 'CVS': pserver = ':pserver:%s@%s:%s' % ((self.user or 'anonymous'), self.host, self.repository) module_checkout_cmd = ['cvs', '-d', pserver, 'checkout', '-r', self.revision, self.module] common_checkout_cmd = ['cvs', '-d', pserver, 'checkout', 'common'] -- 1.7.4.4 -- Jan Pazdziora Principal Software Engineer, Satellite Engineering, Red Hat

1 1

--install issue with mock-1.1.10-1.fc14
by Paul B Schroeder 29 Jun '11

29 Jun '11

I have a F14 system with all of the latest updates, including the latest mock. I --init my chroot with: $ mock -r fedora-14-x86_64 --init And then try to install additional packages in my chroot with: mock -r fedora-14-x86_64 --verbose --install "revisor-cli revisor-comps revisor-isolinux" And get: ERROR: Command failed: # ['/usr/bin/yum', '--installroot', '/var/lib/mock/fedora-14-x86_64/root/', 'install', 'revisor-cli revisor-comps revisor-isolinux', '--setopt=tsflags=nocontexts'] Error: Nothing to do When I do a 'yum downgrade' to mock-1.1.5-1.fc14, I am able to install the additional packages without issue. The error seems to be coming from yum, but only causes an issue with the latest mock. Has anybody seen this issue or have any idea as to what is going on? Thanks...Paul... -- --- Paul B Schroeder <paul.schroeder "at" vbridges "dot" com> Virtual Bridges, Inc.

2 2

What is the difference among those tags in koji?
by yanchuan 28 Jun '11

28 Jun '11

Hi, What is the difference among thos tags in the  buildsystem web of Fedora http://koji.fedoraproject.org/koji? There are 13 tags for Fedora 15,and I'm confused with the relationship among those tags.I know dist-f15-updates is inherited from dist-f15,and dist-f15-override is   inherited from dist-f15-updates.I what to know which package should goes to dist-f15-updates? In what situation, a package should goes to dist-f15-override but not dist-f15-updates? Where can I find some relevant docs?I'm new here. Thanks. Yanchuan

2 1

Repos not regenerating
by Aditya Patawari 24 Jun '11

24 Jun '11

I have a personal instance of Koji. I added certain external repos and tried to build a package but got a "waitrepo (6)" and build never started. On trying to regenerate the repos, I got the following error. GenericError: failed to merge repos: /usr/libexec/kojid/mergerepos -a i386 -b /mnt/koji/repos/centos5-mail-hosting-stable-build/419/i386/blocklist -o /tmp/koji/tasks/1331/1331/repo -g /mnt/koji/repos/centos5-mail-hosting-stable-build/419/groups/comps.xml -r file:///tmp/koji/tasks/1331/1331/repo_419_premerge/ -r https://pulp.abc.com/pulp/repos/centos/5/os/x86_64/ -r https://pulp.abc.com/pulp/repos/centos/5/updates/x86_64/ -r https://pulp.abc.com/pulp/repos/centos/5/updates/i386/ -r https://pulp.abc.com/pulp/repos/centos/5/os/i386/ -r https://pulp.abc.com/pulp/repos/epel/5/x86_64/ -r https://pulp.abc.com/pulp/repos/epel/5/i386/ exited with status 1 Please help. -- Aditya Patawari http://blog.adityapatawari.com/ https://fedoraproject.org/wiki/User:Adimania India

1 2

Kojihub fails after F14 -> f15 upgrade
by Anthony Joseph Messina 09 Jun '11

09 Jun '11

After upgrading from Fedora 14 to Fedora 15, I'm having the following issue when trying to access my koji web install. I use Kerberos for my private Koji instance. I've see this: http://www.redhat.com/archives/fedora-buildsys-list/2008-July/msg00022.html I've checked my pg_hba.conf (it's the same as it was a few hours ago with a working Fedora 14 Koji setup). Of course, now I'm using PostgreSQL 9. I run the PostgreSQL service on the same box and am able to access other PostgreSQL stuff without issue, using Kerberos. Any help would be appreciated. Thanks in advance. -Anthony MOD_PYTHON ERROR ProcessId: 26326 Interpreter: 'messinet.com' ServerName: 'messinet.com' DocumentRoot: '/var/www/egroupware/sitemgr/sitemgr-site' URI: '/koji/' Location: None Directory: '/usr/share/koji-web/scripts/' Filename: '/usr/share/koji-web/scripts/index.py' PathInfo: '' Phase: 'PythonHandler' Handler: 'kojiweb.publisher' Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch default=default_handler, arg=req, silent=hlist.silent) File "/usr/lib64/python2.7/site-packages/mod_python/importer.py", line 1229, in _process_target result = _execute_target(config, req, object, arg) File "/usr/lib64/python2.7/site-packages/mod_python/importer.py", line 1128, in _execute_target result = object(arg) File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 39, in handler return mod_python.publisher.handler(req) File "/usr/lib64/python2.7/site-packages/mod_python/publisher.py", line 213, in handler published = publish_object(req, object) File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 34, in publish_object return old_publish_object(req, _genHTML(req, 'error.chtml')) File "/usr/share/koji-web/lib/kojiweb/util.py", line 69, in _genHTML req._values['mavenEnabled'] = req._session.mavenEnabled() File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1510, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1740, in _callMethod return proxy.__getattr__(name)(*args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1312, in single_request response.msg, ProtocolError: <ProtocolError for messinet.com/kojihub: 500 Internal Server Error> MODULE CACHE DETAILS Accessed: Wed Jun 8 19:04:04 2011 Generation: 0 _mp_0dec3ca8c086f5baed01b0d5504fa2b0 { FileName: '/usr/share/koji-web/scripts/index.py' Instance: 1 [IMPORT] Generation: 1 Modified: Thu Dec 16 15:13:17 2010 Imported: Wed Jun 8 19:04:04 2011 } -- Anthony - http://messinet.com - http://messinet.com/~amessina/gallery 8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E

3 3

Koji host selection/load balancing
by Anthony Towns 07 Jun '11

07 Jun '11

Hi, The host selection algorithm in koji/daemon.py (TaskManager.checkRelAvail) is designed to choose a host that's in the top half of hosts by most available capacity. I think it doesn't quite work as might be expected though -- in the case where you have five hosts all with a capacity of 3.0, you could have the following behaviour: A takes job, capacities: [3.0, 3.0, 3.0, 3.0, 2.0] (median = 3.0) B takes job, capacities: [3.0, 3.0, 3.0, 2.0, 2.0] (median = 3.0) C takes job, capacities: [3.0, 3.0, 2.0, 2.0, 2.0] (median = 2.0) A takes job, capacities: [3.0, 3.0, 2.0, 2.0, 1.0] (median = 2.0) B takes job, capacities: [3.0, 3.0, 2.0, 1.0, 1.0] (median = 2.0) C takes job, capacities: [3.0, 3.0, 1.0, 1.0, 1.0] (median = 1.0) A takes job, capacities: [3.0, 3.0, 1.0, 1.0] (median = 3.0) D takes job, capacities: [3.0, 2.0, 1.0, 1.0] (median = 2.0) D takes job, capacities: [3.0, 1.0, 1.0, 1.0] (median = 1.0) B takes job, capacities: [3.0, 1.0, 1.0] (median = 1.0) C takes job, capacities: [3.0, 1.0] (median = 3.0) So eleven jobs get divided up as: 3 on A 3 on B 3 on C 2 on D 0 on E This is because the >=median test doesn't actually ensure this host is in the top half, just that there's a host in the top half that's no better than this one. I think a better test would be ==best or >median, in which case you'd get: A takes job, capacities: [3.0, 3.0, 3.0, 3.0, 2.0] B takes job, capacities: [3.0, 3.0, 3.0, 2.0, 2.0] C takes job, capacities: [3.0, 3.0, 2.0, 2.0, 2.0] D takes job, capacities: [3.0, 2.0, 2.0, 2.0, 2.0] E takes job, capacities: [2.0, 2.0, 2.0, 2.0, 2.0] Patch attached for consideration. Cheers, aj -- Anthony Towns <atowns(a)redhat.com>

1 0

koji troubles building
by Clint Savage 02 Jun '11

02 Jun '11

Hi all, I know a few of you have helped me with my buildout of koji over the past couple weeks. Thank you all for your help! I've been making good progress on the buildout until last night, when I got stuck with a very interesting problem. At this point, I've searched high and low, and done everything I can think of to address this problem. I'm still stuck here, 24-hours later, with no solution. After a good 6 hours of work, I finally got all of the dependencies working for a build of a tool called 'pydf'. A colorized version of df, written in python. I recently added it to Fedora's koji. When I attempt to build this rpm from a src.rpm, I get through all of the deps. It seems about the time that mock is to write data to the chroot is when things start to break. The machine I am running is Fedora 15, x86_64, fully updated as of tonight. Koji is setup to only build x86_64 rpms. Everything is on just one machine for now. $ koji -c ~/.koji/config-goose build dist-g6 rpmbuild/SRPMS/pydf-9-3.el6.src.rpm ..snip.. 865 buildArch (pydf-9-3.el6.src.rpm, noarch): open (kojibuilder1) -> FAILED: BuildError: error building package (arch noarch), mock exited with status 1; see build.log for more information 0 free 1 open 0 done 1 failed 864 build (dist-g6, pydf-9-3.el6.src.rpm): open (kojibuilder1) -> FAILED: BuildError: error building package (arch noarch), mock exited with status 1; see build.log for more information 0 free 0 open 0 done 2 failed When I look at the logs, the most interesting is the 'mock_output.log' State Changed: setup ERROR: Exception(/mnt/koji/work/cli-build/1306989306.986684.cIQpcXny/pydf-9-3.el6.src.rpm) Config(dist-g6-build-116-205) 0 minutes 1 seconds INFO: Results and/or logs in: /var/lib/mock/dist-g6-build-116-205/result Traceback (most recent call last): File "/usr/sbin/mock", line 491, in <module> def do_buildsrpm(config_opts, chroot, options, args): File "/usr/sbin/mock", line 794, in main do_rebuild(config_opts, chroot, args) File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at 0x01F9EB90>", line 3, in do_rebuild File "/usr/lib/python2.7/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/sbin/mock", line 463, in do_rebuild chroot.build(srpm, timeout=config_opts['rpmbuild_timeout']) File "<peak.util.decorators.rewrap wrapping mock.backend.build at 0x01F8EB90>", line 3, in build File "/usr/lib/python2.7/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/lib/python2.7/site-packages/mock/backend.py", line 530, in build gid=self.chrootgid, File "/usr/lib/python2.7/site-packages/mock/backend.py", line 449, in doChroot shell=shell, *args, **kargs ) File "<peak.util.decorators.rewrap wrapping mock.util.do at 0x01F895F0>", line 3, in do File "/usr/lib/python2.7/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/lib/python2.7/site-packages/mock/util.py", line 292, in do preexec_fn = preexec, File "/usr/lib64/python2.7/subprocess.py", line 672, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1202, in _execute_child raise child_exception OSError: [Errno 13] Permission denied Everything above the 'Status Changed: setup' appears to be normal. Some things that were suggested to me in #koji. - SELinux permissions. Currently in permissive mode. - The user is not in the mock group - all users allowed to use koji are in the mock group (clints,kojibuilder to be exact) I cannot see any reason this is failing, so I am hoping to find some insight here. Also, I should note that I did run across some abrt dumps which looked interesting and will be happy to post, if desired. Thank you all for your continued help. I sure hope this issue is easier to solve than it currently looks. Cheers, Clint

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys June 2011