buildsys February 2015

buildsys@lists.fedoraproject.org

20 participants
26 discussions

koji's mergerepos does not consider epoch tag
by Atanas Zhelev 19 Feb '15

19 Feb '15

Hi all, i'm using external repository with various vendor supplied rpm's in koji. One of those rpm's had it's versioning schema changed and it uses epoch tag now. After regeneration of the build tags koji does not see the new package. It looks like mergerepos ignores the epoch tag. The "old" rpm which actually has a higher version number if epoch is not considered is included instead. Can you offer any advice? Thanks -- BR Atanas

3 5

docker-rpm-builder stable released - build native RPMs through docker
by Alan Franzoni 19 Feb '15

19 Feb '15

Hello, I had given a shot to this project some months ago; after a bit of refactoring, it's now usable. (to the Fedora/Redhat people: if you feel I'm hijacking this ml with my announcements, just let me know; since it's a low-traffic ml and highly relevant to the topic, I thought it the best place where to place such announcements) https://github.com/alanfranz/docker-rpm-builder RPM packages for Centos/RHEL and Fedora are available now. Key features: - fast RPM building without chroot unpacking/file copy/etc - templating support for specfiles - interactive mode when a build fails - build straight from a directory without an intermediate source RPM (optional: you can still rebuild a .src.rpm) - works on any host Linux distro that supports docker and has Python 2.7 - leverages existing RPM toolchain, you don't need to learn a new language - root filesystem is fully writeable during the build If anybody happens to try it, let me know what you think about it. I'll be doing a few benchmarks in the next weeks. Bye! -- My development blog: ollivander.franzoni.eu . @franzeur on Twitter contact me at public(a)[mysurname].eu

3 2

Fw: Mock, autosigning
by Clark Williams 16 Feb '15

16 Feb '15

Second send, since I managed to use the old buildsys list address in my previous forward of this message. Sigh... I've forwarded Julian's proposed plugin for mock to perform signing since he wasn't sure where best to send it. I have *not* reviewed this patch yet but will be doing so in the near future. Help reviewing from anyone with signing expertise in the Fedora infrastructure would be appreciated. Clark Begin forwarded message: Date: Tue, 10 Feb 2015 01:30:03 +0000 From: Julien Ballet <lta(a)fb.com> To: "clark.williams(a)gmail.com" <clark.williams(a)gmail.com> Subject: Mock, autosigning Hi, I’ve been wandering around the different mock pages (wiki/trac), and I haven’t found any indications about the preferred way to submit patches for mock. The wiki page tells to go to Trac, and trac doesn’t say anything :) So please forgive me for looking at git logs and reaching out to the most recent/active contributor :) I’ve been working on a small patch to mock that adds automatic signing of built package. It also introduces a small api change, so please let me know if it’s wrong and if you have a better idea :) Let me also know if there’s a better place to post patches :) Regards, Lta.

2 1

How should mock behave when DNF is not available?
by Miroslav Suchý 11 Feb '15

11 Feb '15

I am working on switching rawhide targets to DNF. However on EL* is DNF not available. Therefore you will not be able to build rawhide packages on EL7- using standard mock configuration. I see several options to offer to user: 1) Discourage user from doing that by providing no hints how to workaround it. 2) Tell him that DNF is not available and he must run mock with "--yum" option. And warn him that he may get different depsolving. 3) Tell him that DNF is not available and fall back automatically to yum and warn him that he may get different depsolving. Unless somebody will have better idea I would go with (2). Ideas? Comments? Mirek

4 7

[PATCH] add missed permission check in RootExports.wrapperRPM
by Chenxiong Qi 10 Feb '15

10 Feb '15

From: Chenxiong Qi <cqi(a)redhat.com> according to the docstring and the behavior in other similar methods, permission should be checked against current session before increasing priority. RootExports.wrapperRPM has no such step. The process of adjusting priority gets refactored to avoid duplicated code. --- hub/kojihub.py | 60 +++++++++++++++++++++---------------------------------- 1 files changed, 23 insertions(+), 37 deletions(-) diff --git a/hub/kojihub.py b/hub/kojihub.py index 92fc889..61fb498 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -7002,6 +7002,20 @@ class RootExports(object): context.session.assertPerm('admin') return make_task('restartHosts', [], priority=priority) + def _adjust_priority(self, priority): + """Adjust priority relative to the default priority + + priority: the amount to increase (or decrease) the task priority, + relative to the default priority; higher values mean lower + priority. Only admins have the right to specify a negative + priority here, otherwise, ActionNotAllowed will be raised. + returns final priority + """ + if priority < 0: + if not context.session.hasPerm('admin'): + raise koji.ActionNotAllowed, 'only admins may create high-priority tasks' + return koji.PRIO_DEFAULT + priority + def build(self, src, target, opts=None, priority=None, channel=None): """Create a build task @@ -7015,10 +7029,7 @@ class RootExports(object): opts = {} taskOpts = {} if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, 'only admins may create high-priority tasks' - taskOpts['priority'] = koji.PRIO_DEFAULT + priority + taskOpts['priority'] = self._adjust_priority(priority) if channel: taskOpts['channel'] = channel return make_task('build',[src, target, opts],**taskOpts) @@ -7040,10 +7051,7 @@ class RootExports(object): opts = {} taskOpts = {} if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, 'only admins may create high-priority tasks' - taskOpts['priority'] = koji.PRIO_DEFAULT + priority + taskOpts['priority'] = self._adjust_priority(priority) if channel: taskOpts['channel'] = channel @@ -7067,10 +7075,7 @@ class RootExports(object): opts = {} taskOpts = {} if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, 'only admins may create high-priority tasks' - taskOpts['priority'] = koji.PRIO_DEFAULT + priority + taskOpts['priority'] = self._adjust_priority(priority) if channel: taskOpts['channel'] = channel @@ -7111,7 +7116,7 @@ class RootExports(object): taskOpts = {} if priority: - taskOpts['priority'] = koji.PRIO_DEFAULT + priority + taskOpts['priority'] = self._adjust_priority(priority) taskOpts['channel'] = channel return make_task('wrapperRPM', [url, build_target, build, None, opts], **taskOpts) @@ -7132,10 +7137,7 @@ class RootExports(object): raise koji.GenericError, "Maven support not enabled" taskOpts = {} if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, 'only admins may create high-priority tasks' - taskOpts['priority'] = koji.PRIO_DEFAULT + priority + taskOpts['priority'] = self._adjust_priority(priority) if channel: taskOpts['channel'] = channel @@ -7166,10 +7168,7 @@ class RootExports(object): opts = {} taskOpts = {} if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, 'only admins may create high-priority tasks' - taskOpts['priority'] = koji.PRIO_DEFAULT + priority + taskOpts['priority'] = self._adjust_priority(priority) if channel: taskOpts['channel'] = channel @@ -7190,12 +7189,7 @@ class RootExports(object): taskOpts = {'channel': img_type} taskOpts['arch'] = arch if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, \ - 'only admins may create high-priority tasks' - - taskOpts['priority'] = koji.PRIO_DEFAULT + priority + taskOpts['priority'] = self._adjust_priority(priority) return make_task(img_type, [name, version, arch, target, ksfile, opts], **taskOpts) @@ -7208,12 +7202,8 @@ class RootExports(object): context.session.assertPerm('image') taskOpts = {'channel': 'image'} if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, \ - 'only admins may create high-priority tasks' + taskOpts['priority'] = self._adjust_priority(priority) - taskOpts['priority'] = koji.PRIO_DEFAULT + priority if not opts.has_key('scratch') and not opts.has_key('indirection_template_url'): raise koji.ActionNotAllowed, 'Non-scratch builds must provide url for the indirection template' @@ -7228,12 +7218,8 @@ class RootExports(object): context.session.assertPerm('image') taskOpts = {'channel': 'image'} if priority: - if priority < 0: - if not context.session.hasPerm('admin'): - raise koji.ActionNotAllowed, \ - 'only admins may create high-priority tasks' + taskOpts['priority'] = self._adjust_priority(priority) - taskOpts['priority'] = koji.PRIO_DEFAULT + priority if not opts.has_key('scratch') and not opts.has_key('ksurl'): raise koji.ActionNotAllowed, 'Non-scratch builds must provide ksurl' -- 1.7.1

1 0

[PATCH 1/4] Prepare for the new sources file format
by Mathieu Bridon 10 Feb '15

10 Feb '15

From: Mathieu Bridon <bochecha(a)daitauha.fr> The current format for the 'sources file is as follows: $hash $filename We're eventually going to move to: $hashtype $hash $filename This commit just prepares us for that, allowing to parse both formats. If the 'sources' file doesn't contain the $hashtype, it is assumed to be the one defined in the config file as lookasidehash. --- src/pyrpkg/__init__.py | 10 +++++++++- src/pyrpkg/sources.py | 9 +++++---- test/test_sources.py | 2 +- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/src/pyrpkg/__init__.py b/src/pyrpkg/__init__.py index 5c4dbc5..f9606fe 100644 --- a/src/pyrpkg/__init__.py +++ b/src/pyrpkg/__init__.py @@ -1568,7 +1568,15 @@ class Commands(object): # Default to putting the files where the module is if not outdir: outdir = self.path - for (csum, file) in self._read_sources(): + + for entry in self._read_sources(): + if len(entry) == 2: + csum, file = entry + hashtype = self.lookasidehash + + else: + hashtype, csum, file = entry + # See if we already have a valid copy downloaded outfile = os.path.join(outdir, file) if os.path.exists(outfile): diff --git a/src/pyrpkg/sources.py b/src/pyrpkg/sources.py index c4263ad..8d8b277 100644 --- a/src/pyrpkg/sources.py +++ b/src/pyrpkg/sources.py @@ -42,10 +42,11 @@ def _parse_line(line): stripped_line = line.strip() if not stripped_line: return [] - entries = stripped_line.split(' ', 1) - if len(entries) != 2: - raise ValueError("Malformed line: %r." % line) - return entries + entries = stripped_line.split(' ') + if len(entries) in (2, 3): + return entries + + raise ValueError("Malformed line: %r." % line) def _format_line(entry): diff --git a/test/test_sources.py b/test/test_sources.py index 997ab83..71b49fb 100644 --- a/test/test_sources.py +++ b/test/test_sources.py @@ -14,7 +14,7 @@ class formatLineTestCase(unittest.TestCase): def test_wrong_number_of_fields(self): WRONG_ENTRIES = [ ('foo'), - ('foo', 'bar', 'foo'), + ('foo', 'bar', 'foo', 'bar'), ] for entry in WRONG_ENTRIES: self.assertRaises(ValueError, sources._format_line, entry) -- 2.1.0

1 5

status of signed rawhide
by Miroslav Suchý 09 Feb '15

09 Feb '15

What is the status of signing packages in rawhide? Is every package now signed? I am asking whether I should put gpgcheck=1 in mock configs. Mirek

2 1

Mock and DNF - the plan
by Miroslav Suchý 09 Feb '15

09 Feb '15

Hi, today we [1] had a post-DevConf meeting and discussed the best approach to switch Mock to use DNF as default. The resolution is that: * just after F22 git branching (which should happen tomorrow) I will change fedora-rawhide-*.cfg files in mock to use DNF. For everything else YUM will be still used. And YUM will be general default. * Michael McLean will adjust builroot tags in koji code to allow setting of additional keys/values and it will be used by Koji to generate mock config which will either use DNF or YUM. * The koji.fedoraproject.org will not be changed right now, but likely in 1-3 months. * During this transient period local mock builds for rawhide can differ from koji builds due different depsolving. I will prepare announcement for devel mailing list so people are aware of this and encourage people to report any problem they experience. Hopefully if there will be some problems we will have time to address them before the new code land in Koji. This change only affects future rawhide (Fedora 23) targets and it will not affect older Fedora and EPELs. [1] me, Dennis Gilmore, Peter Robinson, Jan Zeleny, Michael Mraka, Michael McLean and Daniel Mach -- Miroslav Suchy, RHCE, RHCDS Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

1 0

[PATCH] simplify the statistics of packagesbyuser by a single XMLRPC call
by Chenxiong Qi 07 Feb '15

07 Feb '15

From: Chenxiong Qi <cqi(a)redhat.com> Original implementation sends multiple times of calls to listPackages to gather the number of packages of each user, then multiple database queries and unnecessary operations in listPackages that are not related to the statistics itself will happen repeatedly while iterating users. This patch simplifies this process by providing a higher level API, which requires only one database query against the tag_packages table, and any client is able to get the statistics data by a single XMLRPC call. --- hub/kojihub.py | 12 ++++++++++++ www/kojiweb/index.py | 7 ++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/hub/kojihub.py b/hub/kojihub.py index 92fc889..d5e9aa0 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -9332,6 +9332,18 @@ class RootExports(object): values=locals(), opts=queryOpts) return query.iterate() + def reportPackagesByUser(self): + '''Report of packages by user + + Return a mapping from user.id to the number of packages + ''' + fields = ('user_id', 'packages_count') + sql = ''' +SELECT owner, COUNT(package_id) FROM tag_packages +GROUP BY owner ORDER BY owner''' + rows = _multiRow(sql, None, fields) + return dict(((str(row['user_id']), row['packages_count']) for row in rows)) + class BuildRoot(object): diff --git a/www/kojiweb/index.py b/www/kojiweb/index.py index ff15a5f..279e6fc 100644 --- a/www/kojiweb/index.py +++ b/www/kojiweb/index.py @@ -1919,13 +1919,10 @@ def packagesbyuser(environ, start=None, order=None): maxPackages = 1 users = server.listUsers() + data = server.reportPackagesByUser() - server.multicall = True for user in users: - server.count('listPackages', userID=user['id'], with_dups=True) - packageCounts = server.multiCall() - - for user, [numPackages] in zip(users, packageCounts): + numPackages = data.get(str(user['id']), 0) user['packages'] = numPackages if numPackages > maxPackages: maxPackages = numPackages -- 1.7.1

1 0

[PATCH] simplify the statistics of packagesbyuser by a single XMLRPC call
by Chenxiong Qi 07 Feb '15

07 Feb '15

From: Chenxiong Qi <cqi(a)redhat.com> Original implementation sends multiple times of calls to listPackages to gather the number of packages of each user, then multiple database queries and unnecessary operations in listPackages that are not related to the statistics itself will happen repeatedly while iterating users. This patch simplifies this process by providing a higher level API, which requires only one database query against the tag_packages table, and any client is able to get the statistics data by a signle XMLRPC call. --- .gitignore | 2 ++ hub/kojihub.py | 12 ++++++++++++ www/kojiweb/index.py | 7 ++----- 3 files changed, 16 insertions(+), 5 deletions(-) diff --git a/.gitignore b/.gitignore index 4857165..3a8896f 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ *.pyc *.pyo tests/test.py +*.swp +*~ diff --git a/hub/kojihub.py b/hub/kojihub.py index 92fc889..d5e9aa0 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -9332,6 +9332,18 @@ class RootExports(object): values=locals(), opts=queryOpts) return query.iterate() + def reportPackagesByUser(self): + '''Report of packages by user + + Return a mapping from user.id to the number of packages + ''' + fields = ('user_id', 'packages_count') + sql = ''' +SELECT owner, COUNT(package_id) FROM tag_packages +GROUP BY owner ORDER BY owner''' + rows = _multiRow(sql, None, fields) + return dict(((str(row['user_id']), row['packages_count']) for row in rows)) + class BuildRoot(object): diff --git a/www/kojiweb/index.py b/www/kojiweb/index.py index ff15a5f..279e6fc 100644 --- a/www/kojiweb/index.py +++ b/www/kojiweb/index.py @@ -1919,13 +1919,10 @@ def packagesbyuser(environ, start=None, order=None): maxPackages = 1 users = server.listUsers() + data = server.reportPackagesByUser() - server.multicall = True for user in users: - server.count('listPackages', userID=user['id'], with_dups=True) - packageCounts = server.multiCall() - - for user, [numPackages] in zip(users, packageCounts): + numPackages = data.get(str(user['id']), 0) user['packages'] = numPackages if numPackages > maxPackages: maxPackages = numPackages -- 1.7.1

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys February 2015