Hi,
Dennis Gilmore has been talking for some time about how he'd like to run Mash in Koji tasks, instead of the set of scripts releng currently uses.
So I started working on this, and I got to the point where it actually works, but I have a few questions before I can finish it and submit the patches.
1. What would be an appropriate weight for this new task?
At the moment I'm just using the default, but given that mashing can be quite intensive, it might be better to increase that. I have no idea on what would be the right number, though.
2. What would be the appropriate permissions to run this task?
Right now, the task requires the user to be an admin. Should I introduce a new 'mash' permission? Is there an existing 'releng' permission I should use?
3. What to do with the output?
Mash will create the repositories for all configured architectures, corresponding to a certain Koji tag.
As such, its output is much bigger that, say, a build task.
The task could upload everything back to the Hub, but that could take a significant time and might hammer the Hub?
Dennis was saying that another possibility was to mount /mnt/koji as read-write on the builder, and inside the chroot, so that the task could write its output directly there.
What would be the preferred option?
24.09.2013 07:38, Mathieu Bridon wrote:
Hi,
Dennis Gilmore has been talking for some time about how he'd like to run Mash in Koji tasks, instead of the set of scripts releng currently uses.
So I started working on this, and I got to the point where it actually works, but I have a few questions before I can finish it and submit the patches.
Did you seen this plugin mash_and_spacewalk_sync ? https://github.com/philicious/koji-scripts-and-plugins
On Tue, 2013-09-24 at 10:56 +0300, Viacheslav Dubrovskyi wrote:
24.09.2013 07:38, Mathieu Bridon wrote:
Hi,
Dennis Gilmore has been talking for some time about how he'd like to run Mash in Koji tasks, instead of the set of scripts releng currently uses.
So I started working on this, and I got to the point where it actually works, but I have a few questions before I can finish it and submit the patches.
Did you seen this plugin mash_and_spacewalk_sync ? https://github.com/philicious/koji-scripts-and-plugins
I had not.
That being said, Fedora doesn't use spacewalk, so that wouldn't really be what we need?
24.09.2013 11:02, Mathieu Bridon wrote:
On Tue, 2013-09-24 at 10:56 +0300, Viacheslav Dubrovskyi wrote:
24.09.2013 07:38, Mathieu Bridon wrote:
Hi,
Dennis Gilmore has been talking for some time about how he'd like to run Mash in Koji tasks, instead of the set of scripts releng currently uses.
So I started working on this, and I got to the point where it actually works, but I have a few questions before I can finish it and submit the patches.
Did you seen this plugin mash_and_spacewalk_sync ? https://github.com/philicious/koji-scripts-and-plugins
I had not.
That being said, Fedora doesn't use spacewalk, so that wouldn't really be what we need?
I just found that something similar has already been doing.
IMHO, It would be nice to have it out of the box. And I'm ready to help with testing.
On 09/24/2013 12:38 AM, Mathieu Bridon wrote:
Hi,
Dennis Gilmore has been talking for some time about how he'd like to run Mash in Koji tasks, instead of the set of scripts releng currently uses.
So I started working on this, and I got to the point where it actually works, but I have a few questions before I can finish it and submit the patches.
- What would be an appropriate weight for this new task?
At the moment I'm just using the default, but given that mashing can be quite intensive, it might be better to increase that. I have no idea on what would be the right number, though.
We use 1.5 for createrepo, so probably higher than that, but probably not much higher. Maybe 2? You can always tune that value later.
- What would be the appropriate permissions to run this task?
Right now, the task requires the user to be an admin. Should I introduce a new 'mash' permission? Is there an existing 'releng' permission I should use?
That really depends on intended usage. I don't see a permission similar to 'releng' in koji.fp.o, but adding new perms is easy.
- What to do with the output?
Mash will create the repositories for all configured architectures, corresponding to a certain Koji tag.
As such, its output is much bigger that, say, a build task.
The task could upload everything back to the Hub, but that could take a significant time and might hammer the Hub?
Dennis was saying that another possibility was to mount /mnt/koji as read-write on the builder, and inside the chroot, so that the task could write its output directly there.
What would be the preferred option?
The createrepo tasks upload their generated repodata to the hub. Uploading is much less intensive for the hub since the fast_upload patches went in.
I recommend avoiding rw mounts in the builders if you can.
On Tue, 24 Sep 2013 12:38:44 +0800 Mathieu Bridon bochecha@fedoraproject.org wrote:
Hi,
Dennis Gilmore has been talking for some time about how he'd like to run Mash in Koji tasks, instead of the set of scripts releng currently uses.
So I started working on this, and I got to the point where it actually works, but I have a few questions before I can finish it and submit the patches.
- What would be an appropriate weight for this new task?
At the moment I'm just using the default, but given that mashing can be quite intensive, it might be better to increase that. I have no idea on what would be the right number, though.
Yeah, not sure either... but I wonder if it wouldn't make sense to make this it's own channel so it could go to specific builders.
In the Fedora case we would want that as our mashes are very memory intensive (mostly due to deltarpm generation), so if somehow an arm builder got it, not sure how long it would take it to finish (if ever).
- What would be the appropriate permissions to run this task?
Right now, the task requires the user to be an admin. Should I introduce a new 'mash' permission? Is there an existing 'releng' permission I should use?
Admin would probibly be ok for a first cut.
- What to do with the output?
Mash will create the repositories for all configured architectures, corresponding to a certain Koji tag.
As such, its output is much bigger that, say, a build task.
The task could upload everything back to the Hub, but that could take a significant time and might hammer the Hub?
Dennis was saying that another possibility was to mount /mnt/koji as read-write on the builder, and inside the chroot, so that the task could write its output directly there.
What would be the preferred option?
Not sure. For some data to toss in tho, a current rawhide mash takes up about 163GB. Branched f20 is currently around 153GB.
So, in addition to the question of if it should try and upload to the hub or not, it needs to make sure and have enough local working space to hold all that (unless it's writing to a rw /mnt/koji for that).
kevin
Hi, and thanks for the feedback Kevin,
On Thu, 2013-09-26 at 10:32 -0600, Kevin Fenzi wrote:
On Tue, 24 Sep 2013 12:38:44 +0800 Mathieu Bridon bochecha@fedoraproject.org wrote:
- What would be an appropriate weight for this new task?
At the moment I'm just using the default, but given that mashing can be quite intensive, it might be better to increase that. I have no idea on what would be the right number, though.
Yeah, not sure either... but I wonder if it wouldn't make sense to make this it's own channel so it could go to specific builders.
In the Fedora case we would want that as our mashes are very memory intensive (mostly due to deltarpm generation), so if somehow an arm builder got it, not sure how long it would take it to finish (if ever).
Good point, I'll work on that.
- What to do with the output?
[... snip ...]
Not sure. For some data to toss in tho, a current rawhide mash takes up about 163GB. Branched f20 is currently around 153GB.
So, in addition to the question of if it should try and upload to the hub or not, it needs to make sure and have enough local working space to hold all that (unless it's writing to a rw /mnt/koji for that).
So first of all, do the Fedora builders have that kind of space available?
Mike, given the above, do you still think uploading is the best path forward?
Thanks,
On Wed, 2013-10-02 at 11:53 +0800, Mathieu Bridon wrote:
On Thu, 2013-09-26 at 10:32 -0600, Kevin Fenzi wrote:
On Tue, 24 Sep 2013 12:38:44 +0800 Mathieu Bridon bochecha@fedoraproject.org wrote:
- What to do with the output?
[... snip ...]
Not sure. For some data to toss in tho, a current rawhide mash takes up about 163GB. Branched f20 is currently around 153GB.
So, in addition to the question of if it should try and upload to the hub or not, it needs to make sure and have enough local working space to hold all that (unless it's writing to a rw /mnt/koji for that).
So first of all, do the Fedora builders have that kind of space available?
To add some more numbers, I have just finished a mash run inside a Koji task, uploading the results at the end.
Number of SRPM: 1079 Total size of the mash output: 4.5GB
That is, a much smaller repository than Fedora. ;)
The task took just over 3 hours to complete.
The upload itself upload took... about 2 hours and 50 minutes.
The Hub and the builder are two vms on the same local network.
If we assume proportionality between size and time, it would take about 96 hours just to upload the results of a F20 mash.
That seems... quite long? :-/
Mike, given the above, do you still think uploading is the best path forward?
On 10/01/2013 11:53 PM, Mathieu Bridon wrote:
On Thu, 2013-09-26 at 10:32 -0600, Kevin Fenzi wrote:
Not sure. For some data to toss in tho, a current rawhide mash takes up about 163GB. Branched f20 is currently around 153GB.
So, in addition to the question of if it should try and upload to the hub or not, it needs to make sure and have enough local working space to hold all that (unless it's writing to a rw /mnt/koji for that).
So first of all, do the Fedora builders have that kind of space available?
Mike, given the above, do you still think uploading is the best path forward?
Can we break that space consumption down a bit?
In the normal Koji createrepo case (which I based my recommendation on) the yum repo is created on a builder from rpms on /mnt/koji. The only thing that is uploaded at the end is the yum metadata, which is only a fraction of the size of the repo itself.
I realize with mash there is more going one (depending on the options), but still. How much truly new content is generated by mashing? (As opposed to content that is merely referenced or arranged)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thu, 03 Oct 2013 09:54:26 -0400 Mike McLean mikem@redhat.com wrote:
On 10/01/2013 11:53 PM, Mathieu Bridon wrote:
On Thu, 2013-09-26 at 10:32 -0600, Kevin Fenzi wrote:
Not sure. For some data to toss in tho, a current rawhide mash takes up about 163GB. Branched f20 is currently around 153GB.
So, in addition to the question of if it should try and upload to the hub or not, it needs to make sure and have enough local working space to hold all that (unless it's writing to a rw /mnt/koji for that).
So first of all, do the Fedora builders have that kind of space available?
Mike, given the above, do you still think uploading is the best path forward?
Can we break that space consumption down a bit?
In the normal Koji createrepo case (which I based my recommendation on) the yum repo is created on a builder from rpms on /mnt/koji. The only thing that is uploaded at the end is the yum metadata, which is only a fraction of the size of the repo itself.
I realize with mash there is more going one (depending on the options), but still. How much truly new content is generated by mashing? (As opposed to content that is merely referenced or arranged)
Mike we need the full tree mashed with RPMS as its what we push to the mirrors, just the metadata is not sufficient. ideally we would hardlink the rpms into /mnt/koji/packages/
it is kinda like how we used to make repos in koji many years ago. I really am not sure whats the best way to do it. ultimately I want to run pungi in koji as well, making all of the Fedora releng tasks happen in koji.
Dennis
buildsys@lists.fedoraproject.org
-
Dennis Gilmore -
Kevin Fenzi -
Mathieu Bridon -
Mike McLean -
Viacheslav Dubrovskyi