So here is the list of things that have been requested lately and I'll be working on a few of these over the next few weeks. If anybody has any input, I'd take it. As I start on each, I'll most likely email the mailing list with the outline of what I'm doing.
If anybody has existing patches for these (against current mock git), all the better... :)
1) more reliable mount/umount several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad....
2) caching yum downloads several people have commented that the autocache stuff is great for speeding up builds, others say that it can sometimes be bad for reproducability, and that simply saving the yum cache dir would be better.
3) ccache integration This is a new one that I havent seen before, but should significantly speed up builds for people who often do rebuild-the-entire-distribution-type things. I'm told by some that this is bad for reproducability, but good for speeding up builds when you are just sanity checking or when that small reproducability hit doesnt matter. I've also seen lots of empirical data that ccache should not cause any problems. This will be have to be specifically enabled through a commandline or config file option, so those who care can turn it on/off.
4) distcc integration Pretty much the same case as ccache. Has more things that need thought than the ccache case, above, though.
-- Michael
On Tue, 2007-09-25 at 14:11 -0500, Michael E Brown wrote:
So here is the list of things that have been requested lately and I'll be working on a few of these over the next few weeks. If anybody has any input, I'd take it. As I start on each, I'll most likely email the mailing list with the outline of what I'm doing.
If anybody has existing patches for these (against current mock git), all the better... :)
- more reliable mount/umount
several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad....
- caching yum downloads
several people have commented that the autocache stuff is great for speeding up builds, others say that it can sometimes be bad for reproducability, and that simply saving the yum cache dir would be better.
- ccache integration
This is a new one that I havent seen before, but should significantly speed up builds for people who often do rebuild-the-entire-distribution-type things. I'm told by some that this is bad for reproducability, but good for speeding up builds when you are just sanity checking or when that small reproducability hit doesnt matter. I've also seen lots of empirical data that ccache should not cause any problems. This will be have to be specifically enabled through a commandline or config file option, so those who care can turn it on/off.
- distcc integration
Pretty much the same case as ccache. Has more things that need thought than the ccache case, above, though.
A method for cleaning up stale/orphaned processes that get created during a build was proposed a while ago:
https://bugzilla.redhat.com/show_bug.cgi?id=221351
I'll leave it to you as to whether or not this is the correct implementation, but I certainly think it's a good idea. I see rogue processes consuming resources on the build machines all the time. The process-cleanup code should probably be run before mock exits, whether the build completed successfully or not.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Mike Bonnet wrote:
On Tue, 2007-09-25 at 14:11 -0500, Michael E Brown wrote:
So here is the list of things that have been requested lately and I'll be working on a few of these over the next few weeks. If anybody has any input, I'd take it. As I start on each, I'll most likely email the mailing list with the outline of what I'm doing.
If anybody has existing patches for these (against current mock git), all the better... :)
- more reliable mount/umount
several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad....
- caching yum downloads
several people have commented that the autocache stuff is great for speeding up builds, others say that it can sometimes be bad for reproducability, and that simply saving the yum cache dir would be better.
- ccache integration
This is a new one that I havent seen before, but should significantly speed up builds for people who often do rebuild-the-entire-distribution-type things. I'm told by some that this is bad for reproducability, but good for speeding up builds when you are just sanity checking or when that small reproducability hit doesnt matter. I've also seen lots of empirical data that ccache should not cause any problems. This will be have to be specifically enabled through a commandline or config file option, so those who care can turn it on/off.
- distcc integration
Pretty much the same case as ccache. Has more things that need thought than the ccache case, above, though.
A method for cleaning up stale/orphaned processes that get created during a build was proposed a while ago:
https://bugzilla.redhat.com/show_bug.cgi?id=221351
I'll leave it to you as to whether or not this is the correct implementation, but I certainly think it's a good idea. I see rogue processes consuming resources on the build machines all the time. The process-cleanup code should probably be run before mock exits, whether the build completed successfully or not.
Didn't this go in (orphanskill) in the last round of updates?
Clark
On Tue, Sep 25, 2007 at 02:11:12PM -0500, Michael E Brown wrote:
So here is the list of things that have been requested lately and I'll be working on a few of these over the next few weeks. If anybody has any input, I'd take it. As I start on each, I'll most likely email the mailing list with the outline of what I'm doing.
Maybe you can also look at bug #235141 (nearly 6 months old), about not removing files with the immutable bit set. The workaround posted there works fine, b.t.w., but I'm not sure if this is the ultimate solution.
- more reliable mount/umount
several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad....
And /dev/pts and /proc.
- caching yum downloads
several people have commented that the autocache stuff is great for speeding up builds, others say that it can sometimes be bad for reproducability, and that simply saving the yum cache dir would be better.
Only as an extra option, I assume.
- more reliable mount/umount
several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad....
And /dev/pts and /proc.
Yes... this is one should be pushed way up in priority if possible... building any apps around mock will lead to many a developer wondering what just happened to their '/dev'... as is the case right now for me.... atleast 10 times in the last few hours.
Thanks.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
BJ Dierkes wrote:
- more reliable mount/umount
several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad....
And /dev/pts and /proc.
Yes... this is one should be pushed way up in priority if possible... building any apps around mock will lead to many a developer wondering what just happened to their '/dev'... as is the case right now for me.... atleast 10 times in the last few hours.
Thanks.
I'm looking at BZ 250985 now.
I'd like to add code to do two things:
1. Find where we're exiting without umounting 2. Add some paranoia code in the startup logic to not screw up if #1 fails (like is happening now).
To that end, would you run one of your failure cases with mock --debug and send me the output?
Thanks, Clark
On Oct 4, 2007, at 11:09 AM, Clark Williams wrote:
I'd like to add code to do two things:
- Find where we're exiting without umounting
- Add some paranoia code in the startup logic to not screw up if
#1 fails (like is happening now).
To that end, would you run one of your failure cases with mock -- debug and send me the output?
Looking at this a bit further... I can't recreate the issue using mock alone. One of the issues I fixed in my app yesterday (after commenting on this thread) was that I wasn't properly terminating spawned child processes (i.e. mock). Therefore, mock was still running a job at the time that I attempted to re-run it from my app.
That said, I think the main focus here is #2... ensuring that if the bind mounts from the host are indeed still mounted, divert from cleaning the chroot (or add logic to clean everything except those points that are mounted).
On top of this however, there does not appear to be anything in place to protect against two users (or the same user) using the same mock config file, and therefore the same /var/lib/mock/<chroot>. If everyone knows how mock works they would use a --uniqueext for each build to protect from cleaning a chroot that is currently in use (and has host bind mounts). But that isn't likely, and isn't safe.
If you look at the 'def clean()' method, mock isn't even _umounting '/ dev' at all before it cleans the chroot:
=== def clean(self): """clean out chroot with extreme prejudice :)""" self.state("clean")
self.root_log('Cleaning Root') if os.path.exists('%s/%s' % (self.rootdir, 'proc')): self._umount('proc') if os.path.exists('%s/%s' % (self.rootdir, 'dev/pts')): self._umount('dev/pts')
if os.path.exists(self.basedir): cmd = '%s -rf %s' % (self.config['rm'], self.basedir) (retval, output) = self.do(cmd)
if retval != 0: error(output) if os.path.exists(self.rootdir): raise RootError, "Failed to clean basedir, exiting" ===
Should be:
=== --- /home/wdierkes/mock 2007-10-03 20:01:03.000000000 -0500 +++ /usr/bin/mock 2007-10-04 15:05:14.000000000 -0500 @@ -193,6 +193,8 @@ self._umount('proc') if os.path.exists('%s/%s' % (self.rootdir, 'dev/pts')): self._umount('dev/pts') + if os.path.exists('%s/%s' % (self.rootdir, 'dev')): + self._umount('dev')
if os.path.exists(self.basedir): cmd = '%s -rf %s' % (self.config['rm'], self.basedir) ===
After adding this change, /dev no longer gets borked. I've added this patch/comment to BZ 250985.
I would suggest adding some logic to say, determine if a build is currently in progress (using a specific config/chroot) and if so die out, but suggest the user add a --uniqueext.... or something to that effect.
Thanks.
Michael E Brown wrote:
So here is the list of things that have been requested lately and I'll be working on a few of these over the next few weeks. If anybody has any input, I'd take it. As I start on each, I'll most likely email the mailing list with the outline of what I'm doing.
Some kind of locking maybe so that when someone is running "mock -r <root> shell" and "mock -r <root> <srpm>" is run that the latter doesn't run?
buildsys@lists.fedoraproject.org
-
BJ Dierkes -
Clark Williams -
Jos Vos -
Michael E Brown -
Mike Bonnet -
Orion Poplawski