From what I gather, mash is the tool of choice for composing usable
repositories out of koji's build efforts. What advice do y'all have for integrating the two? Presently, I've got mash running via cron every 15m but that hardly seems optimal as it's a lot of busy work if I'm not building packages for several days and conversely adds more latency into my build + deploy + test timeline if a build finishes right after mash does. Seems like a hook or callback is in order, but I've not seen any write ups on this subject. Also, I had a rpm built by koji which then made it over to my repo via mash, but then I got busy for a few days on other tasks. When I came back it appears that my rpm has disappeared from the repo. I'm guessing kojira cleaned it up, but I was under the impression that such cleanup wouldn't propagate to the repo.
-- John Florian
On Monday, November 04, 2013 05:28:28 PM John.Florian@dart.biz wrote:
From what I gather, mash is the tool of choice for composing usable repositories out of koji's build efforts. What advice do y'all have for integrating the two? Presently, I've got mash running via cron every 15m but that hardly seems optimal as it's a lot of busy work if I'm not building packages for several days and conversely adds more latency into my build + deploy + test timeline if a build finishes right after mash does. Seems like a hook or callback is in order, but I've not seen any write ups on this subject. Also, I had a rpm built by koji which then made it over to my repo via mash, but then I got busy for a few days on other tasks. When I came back it appears that my rpm has disappeared from the repo. I'm guessing kojira cleaned it up, but I was under the impression that such cleanup wouldn't propagate to the repo.
I use a very custom koji-maint [1] script, but you could take a look at the mashCreateRepos function for some ideas. I run it from a systemd timer unit once a day. You could have systemd run it more often if you wanted, since the design goal was to only have mash create new repositories *if* the file date from the previously created repo is older that the last package tagged in the specific Koji build target.
-A
1. https://messinet.com/trac/browser/koji-maint/koji-maint
amessina@messinet.com wrote on 11/04/2013 18:56:08:
On Monday, November 04, 2013 05:28:28 PM John.Florian@dart.biz wrote:
From what I gather, mash is the tool of choice for composing usable repositories out of koji's build efforts. What advice do y'all have
for
integrating the two? Presently, I've got mash running via cron every
15m
but that hardly seems optimal as it's a lot of busy work if I'm not building packages for several days and conversely adds more latency
into my
build + deploy + test timeline if a build finishes right after mash
does.
Seems like a hook or callback is in order, but I've not seen any
write ups
on this subject. Also, I had a rpm built by koji which then made it
over
to my repo via mash, but then I got busy for a few days on other
tasks.
When I came back it appears that my rpm has disappeared from the
repo.
I'm guessing kojira cleaned it up, but I was under the impression
that
such cleanup wouldn't propagate to the repo.
I use a very custom koji-maint [1] script, but you could take a look at
the
mashCreateRepos function for some ideas. I run it from a systemd timer
unit
once a day. You could have systemd run it more often if you
wanted,since the
design goal was to only have mash create new repositories *if* the file
date
from the previously created repo is older that the last package tagged
in the
specific Koji build target.
-A
-- Anthony - http://messinet.com - http://messinet.com/~amessina/gallery 8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
Thanks for that link. Looks to have all sorts of handy features. Package signing is another item on my checklist and I see you've got code in there for that too. I'll have to hack on this (or so it appears) to get it working with certificate-based authentication. Would you be interested in any such patches?
-- John Florian
On Tuesday, November 05, 2013 10:46:55 AM John.Florian@dart.biz wrote:
amessina@messinet.com wrote on 11/04/2013 18:56:08:
On Monday, November 04, 2013 05:28:28 PM John.Florian@dart.biz wrote:
From what I gather, mash is the tool of choice for composing usable repositories out of koji's build efforts. What advice do y'all have for integrating the two? Presently, I've got mash running via cron every 15m but that hardly seems optimal as it's a lot of busy work if I'm not building packages for several days and conversely adds more latency into my build + deploy + test timeline if a build finishes right after mash does. Seems like a hook or callback is in order, but I've not seen any write ups on this subject. Also, I had a rpm built by koji which then made it over to my repo via mash, but then I got busy for a few days on other tasks. When I came back it appears that my rpm has disappeared from the repo. I'm guessing kojira cleaned it up, but I was under the impression that such cleanup wouldn't propagate to the repo.
I use a very custom koji-maint [1] script, but you could take a look at the mashCreateRepos function for some ideas. I run it from a systemd timer unit once a day. You could have systemd run it more often if you wanted,since the design goal was to only have mash create new repositories *if* the file date from the previously created repo is older that the last package tagged in the specific Koji build target.
-A
-- Anthony - http://messinet.com - http://messinet.com/~amessina/gallery 8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
Thanks for that link. Looks to have all sorts of handy features. Package signing is another item on my checklist and I see you've got code in there for that too. I'll have to hack on this (or so it appears) to get it working with certificate-based authentication. Would you be interested in any such patches?
Sure, John. I would appreciate patches. Thanks. Though I'd like to see most of this type of work end up upstream in koji-gc or kojira ;)
I wish I knew Python...
-A
amessina@messinet.com wrote on 11/05/2013 11:08:49:
On Tuesday, November 05, 2013 10:46:55 AM John.Florian@dart.biz wrote:
amessina@messinet.com wrote on 11/04/2013 18:56:08:
On Monday, November 04, 2013 05:28:28 PM John.Florian@dart.biz
wrote:
From what I gather, mash is the tool of choice for composing
usable
repositories out of koji's build efforts. What advice do y'all
have for
integrating the two? Presently, I've got mash running via cron
every
15m but that hardly seems optimal as it's a lot of busy work if I'm
not
building packages for several days and conversely adds more
latency into
my build + deploy + test timeline if a build finishes right after
mash
does. Seems like a hook or callback is in order, but I've not seen any
write
ups on this subject. Also, I had a rpm built by koji which then made
it
over to my repo via mash, but then I got busy for a few days on other
tasks.
When I came back it appears that my rpm has disappeared from the
repo.
I'm guessing kojira cleaned it up, but I was under the impression
that
such cleanup wouldn't propagate to the repo.
I use a very custom koji-maint [1] script, but you could take a look
at
the mashCreateRepos function for some ideas. I run it from a
systemd
timer unit once a day. You could have systemd run it more often if
you
wanted,since the design goal was to only have mash create new repositories *if* the file date from the previously created repo is
older
that the last package tagged in the specific Koji build target.
-A
-- Anthony - http://messinet.com -
http://messinet.com/~amessina/gallery
8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
Thanks for that link. Looks to have all sorts of handy features.
Package
signing is another item on my checklist and I see you've got code in
there
for that too. I'll have to hack on this (or so it appears) to get it working with certificate-based authentication. Would you be
interested in
any such patches?
Sure, John. I would appreciate patches. Thanks. Though I'd like to see most of this type of work end up upstream in koji-gc or kojira ;)
I wish I knew Python...
I know Python very well, I wish I knew my way around koji. :-)
It's been quite a battle trying to follow the various wikis, blogs and etc. on getting a setup going and there seems to be even less written on how to actually use it in practice. Part of my difficulty probably stems from having built my own tool chain and using that for so long; I'm biased on approach. I'm trying to convert now to mainstream practices because I want a more automated setup where git commits to my numerous projects yields rpms which yield repos which yield custom Fedora Live images which yield VM builds and automated testing all along the way in a huge continuous integration loop.
-- John Florian
On 11/04/2013 05:28 PM, John.Florian@dart.biz wrote:
From what I gather, mash is the tool of choice for composing usable repositories out of koji's build efforts. What advice do y'all have for integrating the two? Presently, I've got mash running via cron every 15m but that hardly seems optimal as it's a lot of busy work if I'm not building packages for several days and conversely adds more latency into my build + deploy + test timeline if a build finishes right after mash does.
You might consider mimicking kojira. That is, monitor the tags you care about for changes and only regenerate when you need to. It's not a perfect system, but it's much better than every X minutes.
Seems like a hook or callback is in order, but I've not seen any write ups on this subject.
That is also a possibility. The hub code has plenty of callback hooks.
Also, I had a rpm built by koji which then made it over to my repo via mash, but then I got busy for a few days on other tasks. When I came back it appears that my rpm has disappeared from the repo. I'm guessing kojira cleaned it up, but I was under the impression that such cleanup wouldn't propagate to the repo.
I guess that depends on how the repo is generated. If the mash copies them rpms to its own location, then nothing in koji should touch them. If OTOH mash references the koji path directly in the generated repo, then the file could disappear if koji deletes it.
(koji builds are generally deleted by koji-gc. kojira doesn't delete builds, it just manages yum repos)
buildsys@lists.fedoraproject.org
-
Anthony Joseph Messina -
John.Florian＠dart.biz -
Mike McLean