Hello,
When setting up a new Koji setup, I submit a regen-repo task which is picked up by the single builder host that I have setup, which fails with a message in the kojid.log file:
<snip> 2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161, in runTask response = (handler.run(),) File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 3536, in handler raise koji.GenericError, "Repo directory missing: %s" % path GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085 </snip>
This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host inside /mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and the builder host share with each other. Is there any cause for this error to occur when the directory referenced as missing does actually exist?
Thanks,
Kirk Harr Systems and Network Engineer End Point Corporation
On 02/12/2015 03:00 PM, Kirk Harr wrote:
When setting up a new Koji setup, I submit a regen-repo task which is picked up by the single builder host that I have setup, which fails with a message in the kojid.log file:
<snip> 2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161, in runTask response = (handler.run(),) File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 3536, in handler raise koji.GenericError, "Repo directory missing: %s" % path GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085 </snip>
This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host inside /mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and the builder host share with each other. Is there any cause for this error to occur when the directory referenced as missing does actually exist?
The test that appears to be failing is:
if not os.path.isdir(path): raise koji.GenericError, "Repo directory missing: %s" % path
This test happens just after the hub call to host.repoInit (which creates the stub repo dir with basic data).
A few possibilities: 1) some sort of race issue where the new directory is still unknown to the builder 2) some sort of permission error where the kojid process cannot see the directory (does gluster do anything like "root squash"?) 3) the mount isn't set up on the builder after all 4) some sort of path mismatch (e.g. mounted in the wrong place)
Hi Mike,
Thanks for your response, for the possibilities:
1) This is possible, if so is there a way to retry the regen-repo task using the same repo ID so that it can try with the directory in place? 2) GlusterFS does have root-squash functionality, but it is disabled by default and is off in this case. 3) I confirmed the mount is in place and I can edit files on the remote disk from the builder. 4) Confirmed that the builder has the volume mounted at /mnt/koji
Any ideas are appreciated as I have been working with this setup for a bit now and I cannot get any further into the setup or building RPMs until I can figure out why it cannot build its repository information.
Thanks,
Kirk Harr Systems and Network Engineer End Point Corporation
On 2/13/2015 10:15 AM, Mike McLean wrote:
On 02/12/2015 03:00 PM, Kirk Harr wrote:
When setting up a new Koji setup, I submit a regen-repo task which is picked up by the single builder host that I have setup, which fails with a message in the kojid.log file:
<snip> 2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161, in runTask response = (handler.run(),) File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 3536, in handler raise koji.GenericError, "Repo directory missing: %s" % path GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085 </snip>
This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host inside /mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and the builder host share with each other. Is there any cause for this error to occur when the directory referenced as missing does actually exist?
The test that appears to be failing is:
if not os.path.isdir(path): raise koji.GenericError, "Repo directory missing: %s" % path
This test happens just after the hub call to host.repoInit (which creates the stub repo dir with basic data).
A few possibilities:
- some sort of race issue where the new directory is still unknown to
the builder 2) some sort of permission error where the kojid process cannot see the directory (does gluster do anything like "root squash"?) 3) the mount isn't set up on the builder after all 4) some sort of path mismatch (e.g. mounted in the wrong place)
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On 02/13/2015 01:53 PM, Kirk Harr wrote:
Hi Mike,
Thanks for your response, for the possibilities:
- This is possible, if so is there a way to retry the regen-repo task
using the same repo ID so that it can try with the directory in place?
No, but you could stick a nice sleep (and probably some debugging output) just after the call to self.session.host.repoInit in NewRepoTask.handler() (in kojid).
Use self.logger.warn() to log anything you want to check on, probably before and after a time.sleep(60) or similar.
Things you might want to log: self.logger.warn("Got repo: %r", path) self.logger.warn("Stat says: %r", os.lstat(path)) ...etc
- GlusterFS does have root-squash functionality, but it is disabled by
default and is off in this case.
- I confirmed the mount is in place and I can edit files on the remote
disk from the builder.
Aside: RW should not be required for normal use, unless you are using the runroot plugin. Unless that is the case, I would recommend not granting the builders write permission to /mnt/koji
- Confirmed that the builder has the volume mounted at /mnt/koji
Any ideas are appreciated as I have been working with this setup for a bit now and I cannot get any further into the setup or building RPMs until I can figure out why it cannot build its repository information.
Thanks,
Kirk Harr Systems and Network Engineer End Point Corporation
On 2/13/2015 10:15 AM, Mike McLean wrote:
On 02/12/2015 03:00 PM, Kirk Harr wrote:
When setting up a new Koji setup, I submit a regen-repo task which is picked up by the single builder host that I have setup, which fails with a message in the kojid.log file:
<snip> 2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161, in runTask response = (handler.run(),) File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 3536, in handler raise koji.GenericError, "Repo directory missing: %s" % path GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085 </snip>
This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host inside /mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and the builder host share with each other. Is there any cause for this error to occur when the directory referenced as missing does actually exist?
The test that appears to be failing is:
if not os.path.isdir(path): raise koji.GenericError, "Repo directory missing: %s" % path
This test happens just after the hub call to host.repoInit (which creates the stub repo dir with basic data).
A few possibilities:
- some sort of race issue where the new directory is still unknown to
the builder 2) some sort of permission error where the kojid process cannot see the directory (does gluster do anything like "root squash"?) 3) the mount isn't set up on the builder after all 4) some sort of path mismatch (e.g. mounted in the wrong place)
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
Hi Mike,
I added those logging lines to the kojid python and then when attempting a repository refresh the output that is caught in the log is: <snip> 2015-02-26 17:44:14,886 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1161, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 157, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 3408, in handler self.logger.warn("Stat says: %r", os.lstat(path)) OSError: [Errno 61] No data available: '/mnt/koji/repos/dist-centos6-build/1127’ </snip> Its appears the lstat test on the path is returning this error, but I am not familiar with OSError 61. I was able to run ls on the directory referenced, so it definitely exists but I am starting to believe there is something else preventing the file read from that location. Any idea what could cause an error of this type?
Thanks again for your help in tracing down the cause of this problem.
Kirk Harr Systems and Network Engineer End Point Corporation
On Feb 19, 2015, at 4:19 PM, Mike McLean mikem@redhat.com wrote:
self.logger.warn("Got repo: %r", path) self.logger.warn("Stat says: %r", os.lstat(path))
On 02/26/2015 05:49 PM, Kirk Harr wrote:
Hi Mike,
I added those logging lines to the kojid python and then when attempting a repository refresh the output that is caught in the log is:
<snip> 2015-02-26 17:44:14,886 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1161, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 157, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 3408, in handler self.logger.warn("Stat says: %r", os.lstat(path)) OSError: [Errno 61] No data available: '/mnt/koji/repos/dist-centos6-build/1127’ </snip> Its appears the lstat test on the path is returning this error, but I am not familiar with OSError 61. I was able to run ls on the directory referenced, so it definitely exists but I am starting to believe there is something else preventing the file read from that location. Any idea what could cause an error of this type?
selinux maybe?
Did you trying inserting a long sleep after the repoInit call? I think you said that the directory was there (presumably on the builder's mount) when you looked for it after the fact. So either 1) there is some race issue where the builder does not see the fs update in time 2) somehow the kojid process does not have access to read it (even if your root shell does)
Maybe try running 'watch ls -l /mnt/koji/repos/dist-centos6-build/' in a root shell on the builder while you replicate the problem (to see if you can observe a lag.
Thanks again for your help in tracing down the cause of this problem.
Kirk Harr Systems and Network Engineer End Point Corporation
On Feb 19, 2015, at 4:19 PM, Mike McLean mikem@redhat.com wrote:
self.logger.warn("Got repo: %r", path) self.logger.warn("Stat says: %r", os.lstat(path))
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
Hi Mike,
After troubleshooting the issue for a bit, we were able to get the Koji setup working, and can now generate the build root repositories and build packages. The main issue was that the builders needed to use /tmp/koji as the ‘workdir’ which then removed any of the strange permissions errors that would occur when the builder would attempt to process the task. I had originally had this set to the shared volume for the Kojira/Koji-hub server on /mnt/koji, which was not working as expected.
Thanks for your assistance in tracing it down,
Kirk Harr Systems and Network Engineer End Point Corporation
On Mar 27, 2015, at 2:33 PM, Mike McLean mikem@redhat.com wrote:
On 02/26/2015 05:49 PM, Kirk Harr wrote:
Hi Mike,
I added those logging lines to the kojid python and then when attempting a repository refresh the output that is caught in the log is:
<snip> 2015-02-26 17:44:14,886 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1161, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 157, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 3408, in handler self.logger.warn("Stat says: %r", os.lstat(path)) OSError: [Errno 61] No data available: '/mnt/koji/repos/dist-centos6-build/1127’ </snip> Its appears the lstat test on the path is returning this error, but I am not familiar with OSError 61. I was able to run ls on the directory referenced, so it definitely exists but I am starting to believe there is something else preventing the file read from that location. Any idea what could cause an error of this type?
selinux maybe?
Did you trying inserting a long sleep after the repoInit call? I think you said that the directory was there (presumably on the builder's mount) when you looked for it after the fact. So either
- there is some race issue where the builder does not see the fs update in time
- somehow the kojid process does not have access to read it (even if your root shell does)
Maybe try running 'watch ls -l /mnt/koji/repos/dist-centos6-build/' in a root shell on the builder while you replicate the problem (to see if you can observe a lag.
Thanks again for your help in tracing down the cause of this problem.
Kirk Harr Systems and Network Engineer End Point Corporation
On Feb 19, 2015, at 4:19 PM, Mike McLean mikem@redhat.com wrote:
self.logger.warn("Got repo: %r", path) self.logger.warn("Stat says: %r", os.lstat(path))
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
buildsys@lists.fedoraproject.org
-
Kirk Harr -
Mike McLean