Hi,
I've recently setup a build machine for staging builds to go into our in-house koji. I installed mock and setup the build scratch space in /data/mock (perms set to 2775, root:mock).
I setup an alias for all users for mock so that everybody gets there own buildroot:
alias mock='mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME'
Creating a buildroot is fine:
$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --init INFO: mock.py version 1.0.7 starting... State Changed: init plugins State Changed: start State Changed: lock buildroot State Changed: clean State Changed: init State Changed: lock buildroot Mock Version: 1.0.7 INFO: Mock Version: 1.0.7 INFO: enabled root cache State Changed: unpacking root cache INFO: enabled yum cache State Changed: cleaning yum metadata INFO: enabled ccache State Changed: running yum
But when I attempt to clean it, this happens:
$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean INFO: mock.py version 1.0.7 starting... State Changed: init plugins State Changed: start State Changed: lock buildroot State Changed: clean Traceback (most recent call last): File "/usr/sbin/mock", line 693, in ? main(retParams) File "/usr/sbin/mock", line 567, in main chroot.clean() File "<peak.util.decorators.rewrap wrapping mock.backend.clean at 0x0A926848>", line 3, in clean File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/lib/python2.4/site-packages/mock/backend.py", line 134, in clean mock.util.rmtree(self.basedir) File "<peak.util.decorators.rewrap wrapping mock.util.rmtree at 0x0A91A2A8>", line 3, in rmtree File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in rmtree shutil.rmtree(path, *args, **kargs) File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'
My mock config looks like this for dist-rhel5-x86_64: config_opts['chroothome'] = '/builddir' config_opts['use_host_resolv'] = False config_opts['basedir'] = '/data/mock' config_opts['rpmbuild_timeout'] = 86400 config_opts['yum.conf'] = '[main]\ncachedir=/var/cache/yum\ndebuglevel=1\nlogfile=/var/log/yum.log \nreposdir=/dev/null\nretries=20\nobsoletes=1\ngpgcheck=0\nassumeyes=1\n \n# repos\n\n[build]\nname=build\nbaseurl=http://my-koji-server/repos/dist-r hel5-build/latest/$basearch\n' config_opts['chroot_setup_cmd'] = 'groupinstall build' config_opts['target_arch'] = 'x86_64' config_opts['root'] = 'dist-rhel5'
Has anyone else encountered/fixed this?
Thanks in advance, Greg
This e-mail message contains information which is confidential and may be privileged. It is intended for use by the addressee only. If you are not the intended addressee, we request that you notify the sender immediately and delete or destroy this e-mail message and any attachment(s), without copying, saving, forwarding, disclosing or using its contents in any other way. TomTom N.V., TomTom International BV or any other company belonging to the TomTom group of companies will not be liable for damage relating to the communication by e-mail of data, documents or any other information.
On 03/07/2011 04:17 AM, Greg Trahair wrote:
But when I attempt to clean it, this happens:
$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
...
File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in rmtree shutil.rmtree(path, *args, **kargs) File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'
It's hard to know without more data. The error is happening within os.rmtree.
Have a look at the remains that were not deleted. Is there anything unusual about them? Is it possible another process wrote files to directory during the clean operation? Does the problem repeat?
Hi Mike,
The directory is completely empty, and mock --init and mock --clean give the same error. Even a mock --no-clean --init gives the error.
The strange thing is that I can (as my unprivileged user) just run rmdir /data/mock/dist-rhel5-greg with no problems, so permissions are not the culprit. SELinux is disabled.
I can recreate the problem every time I run mock --init, I have to rmdir the buildroot before I can continue testing builds.
After a little testing, it seems that /data/mock being NFS is an issue, as I just changed the buildroot back to /var/lib/mock and all seems ok. Both /data/mock and /var/lib/mock are set 2775 root:mock.
I can't find any limitations of os.rmtree an nfs.
Regards, Greg
-----Original Message----- From: buildsys-bounces@lists.fedoraproject.org [mailto:buildsys-bounces@lists.fedoraproject.org] On Behalf Of Mike McLean Sent: Monday, March 07, 2011 7:55 PM To: buildsys@lists.fedoraproject.org Subject: Re: Mock 'Directory not empty' error during --clean
On 03/07/2011 04:17 AM, Greg Trahair wrote:
But when I attempt to clean it, this happens:
$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
...
File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in rmtree shutil.rmtree(path, *args, **kargs) File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'
It's hard to know without more data. The error is happening within os.rmtree.
Have a look at the remains that were not deleted. Is there anything unusual about them? Is it possible another process wrote files to directory during the clean operation? Does the problem repeat? -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
Ah, you're being bitten by root-squash. This means that any thread running as root gets mapped to the user nobody on the NFS server and in general gets denied access to regular user files.
Do you have the ability to control how the NFS partition is exported or is it controlled by your IT group? If you have control over the NFS server you can export it with no_root_squash and things should work properly.
I don't have a good solution for running mock on a root-squashed NFS partition at the moment.
Clark
On Tue, 8 Mar 2011 09:45:00 +0100 "Greg Trahair" Greg.Trahair@tomtom.com wrote:
Hi Mike,
The directory is completely empty, and mock --init and mock --clean give the same error. Even a mock --no-clean --init gives the error.
The strange thing is that I can (as my unprivileged user) just run rmdir /data/mock/dist-rhel5-greg with no problems, so permissions are not the culprit. SELinux is disabled.
I can recreate the problem every time I run mock --init, I have to rmdir the buildroot before I can continue testing builds.
After a little testing, it seems that /data/mock being NFS is an issue, as I just changed the buildroot back to /var/lib/mock and all seems ok. Both /data/mock and /var/lib/mock are set 2775 root:mock.
I can't find any limitations of os.rmtree an nfs.
Regards, Greg
-----Original Message----- From: buildsys-bounces@lists.fedoraproject.org [mailto:buildsys-bounces@lists.fedoraproject.org] On Behalf Of Mike McLean Sent: Monday, March 07, 2011 7:55 PM To: buildsys@lists.fedoraproject.org Subject: Re: Mock 'Directory not empty' error during --clean
On 03/07/2011 04:17 AM, Greg Trahair wrote:
But when I attempt to clean it, this happens:
$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
...
File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in rmtree shutil.rmtree(path, *args, **kargs) File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/data/mock/dist-rhel5-greg'
It's hard to know without more data. The error is happening within os.rmtree.
Have a look at the remains that were not deleted. Is there anything unusual about them? Is it possible another process wrote files to directory during the clean operation? Does the problem repeat? -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
I thought this originally, but the NFS server is setup with no_root_squash already (netapp equiv - root=<host>).
I've worked-around the problem by just using a local disk, but does anyone else use mock buildroots on NFS?
-----Original Message----- From: Clark Williams [mailto:clark.williams@gmail.com] Sent: Tuesday, March 08, 2011 4:55 PM To: Discussion of Fedora build system Cc: Greg Trahair Subject: Re: Mock 'Directory not empty' error during --clean
Ah, you're being bitten by root-squash. This means that any thread running as root gets mapped to the user nobody on the NFS server and in general gets denied access to regular user files.
Do you have the ability to control how the NFS partition is exported or is it controlled by your IT group? If you have control over the NFS server you can export it with no_root_squash and things should work properly.
I don't have a good solution for running mock on a root-squashed NFS partition at the moment.
Clark
On Tue, 8 Mar 2011 09:45:00 +0100 "Greg Trahair" Greg.Trahair@tomtom.com wrote:
Hi Mike,
The directory is completely empty, and mock --init and mock --clean
give
the same error. Even a mock --no-clean --init gives the error.
The strange thing is that I can (as my unprivileged user) just run
rmdir
/data/mock/dist-rhel5-greg with no problems, so permissions are not
the
culprit. SELinux is disabled.
I can recreate the problem every time I run mock --init, I have to
rmdir
the buildroot before I can continue testing builds.
After a little testing, it seems that /data/mock being NFS is an
issue,
as I just changed the buildroot back to /var/lib/mock and all seems
ok.
Both /data/mock and /var/lib/mock are set 2775 root:mock.
I can't find any limitations of os.rmtree an nfs.
Regards, Greg
-----Original Message----- From: buildsys-bounces@lists.fedoraproject.org [mailto:buildsys-bounces@lists.fedoraproject.org] On Behalf Of Mike McLean Sent: Monday, March 07, 2011 7:55 PM To: buildsys@lists.fedoraproject.org Subject: Re: Mock 'Directory not empty' error during --clean
On 03/07/2011 04:17 AM, Greg Trahair wrote:
But when I attempt to clean it, this happens:
$ mock -r dist-rhel5-x86_64 --uniqueext $LOGNAME --clean
...
File "/usr/lib/python2.4/site-packages/mock/util.py", line 82, in rmtree shutil.rmtree(path, *args, **kargs) File "/usr/lib64/python2.4/shutil.py", line 172, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/python2.4/shutil.py", line 170, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty:
'/data/mock/dist-rhel5-greg'
It's hard to know without more data. The error is happening within os.rmtree.
Have a look at the remains that were not deleted. Is there anything unusual about them? Is it possible another process wrote files to directory during the clean operation? Does the problem repeat? -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On 03/08/2011 11:06 AM, Greg Trahair wrote:
I thought this originally, but the NFS server is setup with no_root_squash already (netapp equiv - root=<host>).
I've worked-around the problem by just using a local disk, but does anyone else use mock buildroots on NFS?
I've used nfs for a lot of things, but I'm having trouble seeing the value for buildroots. They're not really all that big and local drives are generally faster, so why bother?
Nfs has always been flaky and finicky. Could be that some race, or alternately caching is causing the rmdir call to incorrectly fail because something still thinks those deleted files are in there. If so you might try tweaking the mount options (noac perhaps?) or try a different kernel.
Is it 100% repeatable? Is it always the same directory that fails to remove? Is the remainder always the same.
When it does happen, does a second mock --clean continue to fail repeatedly?
buildsys@lists.fedoraproject.org
-
Clark Williams -
Greg Trahair -
Mike McLean