buildsys June 2015

buildsys@lists.fedoraproject.org

21 participants
24 discussions

koji-gc Not Deleting Builds after Trashcan?
by Dustin C. Hatch 19 Jun '15

19 Jun '15

I noticed today that I've been getting the same "build is about to be deleted" email from koji-gc in my local Koji environment for several weeks. When I investigated, I found that koji-gc has been tagging it into the trashcan, but after the grace period, instead of deleting it, it just untags it. Then, after the next delay, it puts it back in the trashcan and the process repeats. Anyone have any idea why this would be happening? % koji list-history --build=firemon-wf-server-8.1.1-2.fmos.el7 Wed Apr 29 16:57:03 2015 firemon-wf-server-8.1.1-2.fmos.el7 tagged into fmos-803-testing by dhatch Fri May 8 22:13:21 2015 firemon-wf-server-8.1.1-2.fmos.el7 untagged from fmos-803-testing by kojiadmin Sat May 9 22:32:34 2015 firemon-wf-server-8.1.1-2.fmos.el7 tagged into trashcan by kojiadmin Sat May 16 22:41:18 2015 firemon-wf-server-8.1.1-2.fmos.el7 untagged from trashcan by kojiadmin Sun May 17 22:44:55 2015 firemon-wf-server-8.1.1-2.fmos.el7 tagged into trashcan by kojiadmin Sun May 24 22:26:06 2015 firemon-wf-server-8.1.1-2.fmos.el7 untagged from trashcan by kojiadmin Mon May 25 22:23:12 2015 firemon-wf-server-8.1.1-2.fmos.el7 tagged into trashcan by kojiadmin Mon Jun 1 22:12:04 2015 firemon-wf-server-8.1.1-2.fmos.el7 untagged from trashcan by kojiadmin Tue Jun 2 22:22:52 2015 firemon-wf-server-8.1.1-2.fmos.el7 tagged into trashcan by kojiadmin Tue Jun 9 22:37:15 2015 firemon-wf-server-8.1.1-2.fmos.el7 untagged from trashcan by kojiadmin Wed Jun 10 22:26:38 2015 firemon-wf-server-8.1.1-2.fmos.el7 tagged into trashcan by kojiadmin Wed Jun 17 22:19:34 2015 firemon-wf-server-8.1.1-2.fmos.el7 untagged from trashcan by kojiadmin Thu Jun 18 22:34:38 2015 firemon-wf-server-8.1.1-2.fmos.el7 tagged into trashcan by kojiadmin [still active] $ rpm -qa 'koji*' koji-hub-1.9.0-10.el7.gitcd45e886.noarch koji-hub-plugin-mash-1.3.3-1.el7.centos.noarch koji-web-1.9.0-10.el7.gitcd45e886.noarch koji-builder-1.9.0-10.el7.gitcd45e886.noarch koji-utils-1.9.0-10.el7.gitcd45e886.noarch koji-1.9.0-10.el7.gitcd45e886.noarch koji-gc.conf is attached Thanks in advance for any insight, -- ♫Dustin

1 0

Switch to DNF in Koji?
by Miroslav Suchý 18 Jun '15

18 Jun '15

Hi, several months ago I talked to dgilmore and probinso and we talked about using DNF in Koji and start building F22 packages using DNF resolver. The conlusion was that it is quite late in cycle. And it is better to start with plain Mock and leave user to test it on their local workstatations. Mock is using DNF for fedora-rawhide-* targets for some time already. And I received none complaints. Therefore I'm raising question: Can we switch Koji to use DNF for building rawhide targets? Of course F22- will still use yum for building. It would be great if we can make this change before mass rebuils for F23. -- Miroslav Suchy, RHCA Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

6 15

New koji plugin - containerbuild
by Pavol Babincak 12 Jun '15

12 Jun '15

Hello, I'd like to announce new Koji plugin[1] - containerbuild. It uses osbs[2] library to talk to builder which runs OpenShift 3[3] to create a container build and import the image back to Koji. Current version 0.3.1 was the first one which we managed to run on the other than developers machine. Plugin isn't packaged in Fedora yet but you can create an rpm with the spec file in the project if you are inter Feel free to test it and let me know if you have any questions. [1] https://github.com/release-engineering/koji-containerbuild [2] https://github.com/DBuildService/osbs.git [3] http://www.openshift.org/ -- Pavol Babincak Release Engineering, Red Hat

2 2

Koji hub policy - restrict builds to package owner
by Pat Riehecky 11 Jun '15

11 Jun '15

I'm looking to restrict what a non-admin can build a bit more strongly.... Basically, I'm aiming towards: if you are not the package owner for this package under this tag you can't build it or tag it into that tag. I don't see a test under the listed policy tests[1] that can help me towards that goal. Any suggestions? Pat [1]https://fedoraproject.org/wiki/Koji/Policies#Available_tests -- Pat Riehecky Scientific Linux developer Fermi National Accelerator Laboratory www.fnal.gov www.scientificlinux.org

1 0

[PATCH] use mergerepo_c when available
by Dennis Gilmore 06 Jun '15

06 Jun '15

mergerepo_c has a --koji flag to use the behaviour of kojis inbuilt fork of mergerepo. lets use it if it is available. Signed-off-by: Dennis Gilmore <dennis(a)ausil.us> --- builder/kojid | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/builder/kojid b/builder/kojid index 4e53fb5..8c0b20d 100755 --- a/builder/kojid +++ b/builder/kojid @@ -4305,7 +4305,11 @@ class CreaterepoTask(BaseTaskHandler): repos.append(ext_url) blocklist = self.repodir + '/blocklist' - cmd = ['/usr/libexec/kojid/mergerepos', '-a', arch, '-b', blocklist, '-o', self.outdir] + if os.path.isfile('/usr/bin/mergerepo_c'): + cmd = ['/usr/bin/mergerepo_c', '--koji'] + else: + cmd = ['/usr/libexec/kojid/mergerepos'] + cmd.extend(['-a', arch, '-b', blocklist, '-o', self.outdir]) if os.path.isfile(groupdata): cmd.extend(['-g', groupdata]) for repo in repos: -- 2.4.0

2 2

[PATCH] Make the templated footer configurable.
by Ralph Bean 06 Jun '15

06 Jun '15

We ran into a problem[1] where old footers would cause syntax errors when interpreted as templates. This makes that behavior configurable and defaults to the old literal interpretation. [1] https://lists.fedoraproject.org/pipermail/buildsys/2015-May/004751.html --- www/conf/web.conf | 5 +++++ www/kojiweb/includes/footer.chtml | 5 +++++ www/kojiweb/wsgi_publisher.py | 1 + www/lib/kojiweb/util.py | 1 + 4 files changed, 12 insertions(+) diff --git a/www/conf/web.conf b/www/conf/web.conf index 171349b..38f0b61 100644 --- a/www/conf/web.conf +++ b/www/conf/web.conf @@ -24,3 +24,8 @@ LoginTimeout = 72 # Secret = CHANGE_ME LibPath = /usr/share/koji-web/lib + +# If set to True, then the footer will be included literally. +# If False, then the footer will be included as another Kid Template. +# Defaults to True +LiteralFooter = True diff --git a/www/kojiweb/includes/footer.chtml b/www/kojiweb/includes/footer.chtml index 684e5a4..8e22022 100644 --- a/www/kojiweb/includes/footer.chtml +++ b/www/kojiweb/includes/footer.chtml @@ -8,8 +8,13 @@ #set $localfooterpath=$util.themePath("extra-footer.html", local=True) #if os.path.exists($localfooterpath) +#if $literalFooter +#set $localfooter="".join(open($localfooterpath).readlines()) +#$localfooter +#else #include $localfooterpath #end if +#end if </div> </div> diff --git a/www/kojiweb/wsgi_publisher.py b/www/kojiweb/wsgi_publisher.py index 4f58b8e..e790815 100644 --- a/www/kojiweb/wsgi_publisher.py +++ b/www/kojiweb/wsgi_publisher.py @@ -69,6 +69,7 @@ class Dispatcher(object): ['KojiFilesURL', 'string', 'http://localhost/kojifiles'], ['KojiTheme', 'string', None], ['KojiGreeting', 'string', 'Welcome to Koji Web'], + ['LiteralFooter', 'boolean', True], ['WebPrincipal', 'string', None], ['WebKeytab', 'string', '/etc/httpd.keytab'], diff --git a/www/lib/kojiweb/util.py b/www/lib/kojiweb/util.py index ddee29a..d2777b1 100644 --- a/www/lib/kojiweb/util.py +++ b/www/lib/kojiweb/util.py @@ -54,6 +54,7 @@ def _initValues(environ, title='Build System Info', pageID='summary'): values['title'] = title values['pageID'] = pageID values['currentDate'] = str(datetime.datetime.now()) + values['literalFooter'] = environ['koji.options'].get('LiteralFooter', True) themeCache.clear() themeInfo.clear() themeInfo['name'] = environ['koji.options'].get('KojiTheme', None) -- 2.4.2

2 1

[PATCH 1/2] Add CLI for runroot plugin.
by Ralph Bean 06 Jun '15

06 Jun '15

From: Daniel Mach <dmach(a)redhat.com> --- cli/koji | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/cli/koji b/cli/koji index 8e0f7b2..6cdecaa 100755 --- a/cli/koji +++ b/cli/koji @@ -6442,6 +6442,71 @@ def handle_moshimoshi(options, session, args): if u.get("krb_principal", None) is not None: print "Authenticated via Kerberos principal %s" % (u["krb_principal"]) + +def handle_runroot(options, session, args): + "[admin] Run a command in a buildroot" + usage = _("usage: %prog runroot [options] <tag> <arch> <command>") + usage += _("\n(Specify the --help global option for a list of other help options)") + parser = OptionParser(usage=usage) + parser.disable_interspersed_args() + parser.add_option("-p", "--package", action="append", default=[], help=_("make sure this package is in the chroot")) + parser.add_option("-m", "--mount", action="append", default=[], help=_("mount this directory read-write in the chroot")) + parser.add_option("--skip-setarch", action="store_true", default=False, + help=_("bypass normal setarch in the chroot")) + parser.add_option("-w", "--weight", type='int', help=_("set task weight")) + parser.add_option("--channel-override", help=_("use a non-standard channel")) + parser.add_option("--task-id", action="store_true", default=False, + help=_("Print the ID of the runroot task")) + parser.add_option("--use-shell", action="store_true", default=False, + help=_("Run command through a shell, otherwise uses exec")) + parser.add_option("--repo-id", type="int", help=_("ID of the repo to use")) + + (opts, args) = parser.parse_args(args) + + if len(args) < 3: + parser.error(_("Incorrect number of arguments")) + activate_session(session) + tag = args[0] + arch = args[1] + if opts.use_shell: + # everything must be correctly quoted + command = ' '.join(args[2:]) + else: + command = args[2:] + task_id = session.runroot(tag, arch, command, + channel=opts.channel_override, + packages=opts.package, mounts=opts.mount, + repo_id=opts.repo_id, + skip_setarch=opts.skip_setarch, + weight=opts.weight) + if opts.task_id: + print task_id + + try: + while True: + # wait for the task to finish + if session.taskFinished(task_id): + break + time.sleep(options.poll_interval) + except KeyboardInterrupt: + # this is probably the right thing to do here + print "User interrupt: canceling runroot task" + session.cancelTask(task_id) + return + output = None + if "runroot.log" in session.listTaskOutput(task_id): + output = session.downloadTaskOutput(task_id, "runroot.log") + if output: + sys.stdout.write(output) + info = session.getTaskInfo(task_id) + if info is None: + sys.exit(1) + state = koji.TASK_STATES[info['state']] + if state in ('FAILED', 'CANCELED'): + sys.exit(1) + return + + def handle_help(options, session, args): "List available commands" usage = _("usage: %prog help [options]") -- 2.4.2

2 2

[PATCH] Add the hub plugin that accompanies the kojid runroot plugin.
by Ralph Bean 06 Jun '15

06 Jun '15

--- plugins/runroot_hub.py | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 plugins/runroot_hub.py diff --git a/plugins/runroot_hub.py b/plugins/runroot_hub.py new file mode 100644 index 0000000..a666919 --- /dev/null +++ b/plugins/runroot_hub.py @@ -0,0 +1,61 @@ +#koji hub plugin +# There is a kojid plugin that goes with this hub plugin. The kojid builder +# plugin has a config file. This hub plugin has no config file. + + +from koji.context import context +from koji.plugin import export +import koji +import random +import sys + +#XXX - have to import kojihub for mktask +sys.path.insert(0, '/usr/share/koji-hub/') +from kojihub import mktask, get_tag, get_all_arches + +__all__ = ('runroot',) + + +def get_channel_arches(channel): + """determine arches available in channel""" + chan = context.handlers.call('getChannel', channel, strict=True) + ret = {} + for host in context.handlers.call('listHosts', channelID=chan['id'], enabled=True): + for a in host['arches'].split(): + ret[koji.canonArch(a)] = 1 + return ret + +@export +def runroot(tagInfo, arch, command, channel=None, **opts): + """ Create a runroot task """ + context.session.assertPerm('runroot') + taskopts = { + 'priority': 15, + 'arch': arch, + } + + taskopts['channel'] = channel or 'runroot' + + if arch == 'noarch': + #not all arches can generate a proper buildroot for all tags + tag = get_tag(tagInfo) + if not tag['arches']: + raise koji.GenericError, 'no arches defined for tag %s' % tag['name'] + + #get all known arches for the system + fullarches = get_all_arches() + + tagarches = tag['arches'].split() + + # If our tag can't do all arches, then we need to + # specify one of the arches it can do. + if set(fullarches) - set(tagarches): + chanarches = get_channel_arches(taskopts['channel']) + choices = [x for x in tagarches if x in chanarches] + if not choices: + raise koji.GenericError, 'no common arches for tag/channel: %s/%s' \ + % (tagInfo, taskopts['channel']) + taskopts['arch'] = koji.canonArch(random.choice(choices)) + + return mktask(taskopts,'runroot', tagInfo, arch, command, **opts) + -- 2.4.2

2 1

koji question -- why does this build not show a specific git commit?
by Matthew Miller 06 Jun '15

06 Jun '15

Most build tasks in Koji, like http://koji.fedoraproject.org/koji/taskinfo?taskID=9390300 have a specific git commit in the source parameter. For example in this case, ad5df7bf04ca75c5415c480436d0ec6ff5d05097. But *this* one: http://koji.fedoraproject.org/koji/taskinfo?taskID=9250546 just has "HEAD", which is less useful since that moves. I see that it's from a chainbuild, but when I look at the list of chainbuilds, I see that most of those appear to use specific commits too. What's special/weird about this one? -- Matthew Miller <mattdm(a)fedoraproject.org> Fedora Project Leader

4 5

[PATCH] Add CLI for runroot plugin.
by Daniel Mach 06 Jun '15

06 Jun '15

--- cli/koji | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/cli/koji b/cli/koji index 8e0f7b2..6cdecaa 100755 --- a/cli/koji +++ b/cli/koji @@ -6442,6 +6442,71 @@ def handle_moshimoshi(options, session, args): if u.get("krb_principal", None) is not None: print "Authenticated via Kerberos principal %s" % (u["krb_principal"]) + +def handle_runroot(options, session, args): + "[admin] Run a command in a buildroot" + usage = _("usage: %prog runroot [options] <tag> <arch> <command>") + usage += _("\n(Specify the --help global option for a list of other help options)") + parser = OptionParser(usage=usage) + parser.disable_interspersed_args() + parser.add_option("-p", "--package", action="append", default=[], help=_("make sure this package is in the chroot")) + parser.add_option("-m", "--mount", action="append", default=[], help=_("mount this directory read-write in the chroot")) + parser.add_option("--skip-setarch", action="store_true", default=False, + help=_("bypass normal setarch in the chroot")) + parser.add_option("-w", "--weight", type='int', help=_("set task weight")) + parser.add_option("--channel-override", help=_("use a non-standard channel")) + parser.add_option("--task-id", action="store_true", default=False, + help=_("Print the ID of the runroot task")) + parser.add_option("--use-shell", action="store_true", default=False, + help=_("Run command through a shell, otherwise uses exec")) + parser.add_option("--repo-id", type="int", help=_("ID of the repo to use")) + + (opts, args) = parser.parse_args(args) + + if len(args) < 3: + parser.error(_("Incorrect number of arguments")) + activate_session(session) + tag = args[0] + arch = args[1] + if opts.use_shell: + # everything must be correctly quoted + command = ' '.join(args[2:]) + else: + command = args[2:] + task_id = session.runroot(tag, arch, command, + channel=opts.channel_override, + packages=opts.package, mounts=opts.mount, + repo_id=opts.repo_id, + skip_setarch=opts.skip_setarch, + weight=opts.weight) + if opts.task_id: + print task_id + + try: + while True: + # wait for the task to finish + if session.taskFinished(task_id): + break + time.sleep(options.poll_interval) + except KeyboardInterrupt: + # this is probably the right thing to do here + print "User interrupt: canceling runroot task" + session.cancelTask(task_id) + return + output = None + if "runroot.log" in session.listTaskOutput(task_id): + output = session.downloadTaskOutput(task_id, "runroot.log") + if output: + sys.stdout.write(output) + info = session.getTaskInfo(task_id) + if info is None: + sys.exit(1) + state = koji.TASK_STATES[info['state']] + if state in ('FAILED', 'CANCELED'): + sys.exit(1) + return + + def handle_help(options, session, args): "List available commands" usage = _("usage: %prog help [options]") -- 2.4.2

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys June 2015