Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
== Client log == koji regen-repo centos-6-build Regenerating repo for tag centos-6-build Watching tasks (this may be safely interrupted)... 4888 newRepo (centos-6-build): free 4888 newRepo (centos-6-build): free -> open (builder-vbox2.b2pweb.com) 4889 createrepo (x86_64): free 4890 createrepo (i386): free 4889 createrepo (x86_64): free -> open (builder-vbox2.b2pweb.com) 4890 createrepo (i386): free -> open (builder-vbox2.b2pweb.com) 4889 createrepo (x86_64): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 2 open 0 done 1 failed 4888 newRepo (centos-6-build): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 1 open 0 done 2 failed 4890 createrepo (i386): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 0 open 0 done 3 failed
4888 newRepo (centos-6-build) failed
== Builder log == /usr/sbin/kojid --verbose --fg 2013-11-15 14:46:47,576 [INFO] {12485} koji.build:73 Starting up 2013-11-15 14:47:03,363 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4891 2013-11-15 14:47:03,711 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:03,775 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,053 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:19,117 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,312 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4892 2013-11-15 14:47:19,662 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636} 2013-11-15 14:47:19,727 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4892, 'weight': 1.5} 2013-11-15 14:47:19,728 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,925 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4893 2013-11-15 14:47:20,277 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636, 4893: 12638} 2013-11-15 14:47:20,342 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4893, 'weight': 1.5} 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4892, 'weight': 1.5} 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:27,662 [WARNING] {12636} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2845, in handler self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo) File "/usr/sbin/kojid", line 2895, in create_local_repo status = log_output(self.session, cmd[0], cmd, logfile, self.getUploadDir(), logerror=True) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in log_output incremental_upload(session, remotename, outfd, uploadpath) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in incremental_upload fast_incremental_upload(session, fname, fd, path, retries, logger) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in fast_incremental_upload result = session.rawUpload(contents, offset, path, fname, overwrite=True) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:33,658 [WARNING] {12638} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2845, in handler self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo) File "/usr/sbin/kojid", line 2895, in create_local_repo status = log_output(self.session, cmd[0], cmd, logfile, self.getUploadDir(), logerror=True) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in log_output incremental_upload(session, remotename, outfd, uploadpath) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in incremental_upload fast_incremental_upload(session, fname, fd, path, retries, logger) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in fast_incremental_upload result = session.rawUpload(contents, offset, path, fname, overwrite=True) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:35,625 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636, 4893: 12638} 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True} 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:693 Waking up task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True} 2013-11-15 14:47:35,691 [INFO] {12485} koji.TaskManager:846 Task 4892 (pid 12636) exited with status 0 2013-11-15 14:47:35,715 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7036 (task 4892) 2013-11-15 14:47:35,780 [INFO] {12485} koji.TaskManager:846 Task 4893 (pid 12638) exited with status 0 2013-11-15 14:47:35,803 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7037 (task 4893) 2013-11-15 14:47:36,640 [WARNING] {12568} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2811, in handler results = self.wait(subtasks.values(), all=True, failany=True) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 234, in wait return dict(self.session.host.taskWaitResults(self.id,subtasks)) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:51,152 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:51,215 [INFO] {12485} koji.TaskManager:846 Task 4891 (pid 12568) exited with status 0 2013-11-15 14:47:51,239 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7035 (task 4891)
On 11/15/13 5:48 AM, Didier Fabert wrote:
Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
== Client log == koji regen-repo centos-6-build Regenerating repo for tag centos-6-build Watching tasks (this may be safely interrupted)... 4888 newRepo (centos-6-build): free 4888 newRepo (centos-6-build): free -> open (builder-vbox2.b2pweb.com) 4889 createrepo (x86_64): free 4890 createrepo (i386): free 4889 createrepo (x86_64): free -> open (builder-vbox2.b2pweb.com) 4890 createrepo (i386): free -> open (builder-vbox2.b2pweb.com) 4889 createrepo (x86_64): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 2 open 0 done 1 failed 4888 newRepo (centos-6-build): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 1 open 0 done 2 failed 4890 createrepo (i386): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 0 open 0 done 3 failed
4888 newRepo (centos-6-build) failed
== Builder log == /usr/sbin/kojid --verbose --fg 2013-11-15 14:46:47,576 [INFO] {12485} koji.build:73 Starting up 2013-11-15 14:47:03,363 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4891 2013-11-15 14:47:03,711 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:03,775 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,053 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:19,117 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,312 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4892 2013-11-15 14:47:19,662 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636} 2013-11-15 14:47:19,727 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4892, 'weight': 1.5} 2013-11-15 14:47:19,728 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,925 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4893 2013-11-15 14:47:20,277 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636, 4893: 12638} 2013-11-15 14:47:20,342 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4893, 'weight': 1.5} 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4892, 'weight': 1.5} 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:27,662 [WARNING] {12636} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2845, in handler self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo) File "/usr/sbin/kojid", line 2895, in create_local_repo status = log_output(self.session, cmd[0], cmd, logfile, self.getUploadDir(), logerror=True) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in log_output incremental_upload(session, remotename, outfd, uploadpath) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in incremental_upload fast_incremental_upload(session, fname, fd, path, retries, logger) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in fast_incremental_upload result = session.rawUpload(contents, offset, path, fname, overwrite=True) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:33,658 [WARNING] {12638} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2845, in handler self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo) File "/usr/sbin/kojid", line 2895, in create_local_repo status = log_output(self.session, cmd[0], cmd, logfile, self.getUploadDir(), logerror=True) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in log_output incremental_upload(session, remotename, outfd, uploadpath) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in incremental_upload fast_incremental_upload(session, fname, fd, path, retries, logger) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in fast_incremental_upload result = session.rawUpload(contents, offset, path, fname, overwrite=True) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:35,625 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636, 4893: 12638} 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True} 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:693 Waking up task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True} 2013-11-15 14:47:35,691 [INFO] {12485} koji.TaskManager:846 Task 4892 (pid 12636) exited with status 0 2013-11-15 14:47:35,715 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7036 (task 4892) 2013-11-15 14:47:35,780 [INFO] {12485} koji.TaskManager:846 Task 4893 (pid 12638) exited with status 0 2013-11-15 14:47:35,803 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7037 (task 4893) 2013-11-15 14:47:36,640 [WARNING] {12568} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2811, in handler results = self.wait(subtasks.values(), all=True, failany=True) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 234, in wait return dict(self.session.host.taskWaitResults(self.id,subtasks)) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:51,152 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:51,215 [INFO] {12485} koji.TaskManager:846 Task 4891 (pid 12568) exited with status 0 2013-11-15 14:47:51,239 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7035 (task 4891)
koji.LockError is only raised in two places, both around fnctl.lockf():
http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
The fcntl manpage says:
ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).
Are you mounting /mnt/koji over nfs? Maybe the nfslock service died? Is something else holding a lot of locks?
On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
On 11/15/13 5:48 AM, Didier Fabert wrote:
Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
koji.LockError is only raised in two places, both around fnctl.lockf():
http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
The fcntl manpage says:
ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).
Are you mounting /mnt/koji over nfs? Maybe the nfslock service died? Is something else holding a lot of locks?
Yes, you're right, nfslock was dead on NFS server (el5) After restarting nfslock service on my NFS server, all is OK.
Thanks a lot Mike to pointing me to the right direction.
The problem appeared today again, although I set up in kojira to keep repos for 1 week. The root log's end is this:
DEBUG util.py:314: Executing command: /usr/bin/repoquery -c /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} DEBUG util.py:264: Traceback (most recent call last): DEBUG util.py:264: File "/usr/bin/repoquery", line 1510, in <module> DEBUG util.py:264: main(sys.argv) DEBUG util.py:264: File "/usr/bin/repoquery", line 1504, in main DEBUG util.py:264: repoq.runQuery(regexs) DEBUG util.py:264: File "/usr/bin/repoquery", line 982, in runQuery DEBUG util.py:264: pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs) DEBUG util.py:264: File "/usr/bin/repoquery", line 903, in matchPkgs DEBUG util.py:264: pkgs = self.returnPkgList(patterns=items) DEBUG util.py:264: File "/usr/bin/repoquery", line 856, in returnPkgList DEBUG util.py:264: pkgs = self.pkgSack.returnNewestByNameArch(**kwargs) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in <lambda> DEBUG util.py:264: pkgSack = property(fget=lambda self: self._getSacks(), DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in _getSacks DEBUG util.py:264: self.repos.populateSack(which=repos) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 309, in populateSack DEBUG util.py:264: self.doSetup() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 134, in doSetup DEBUG util.py:264: self.retrieveAllMD() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 92, in retrieveAllMD DEBUG util.py:264: downloading = repo._commonRetrieveDataMD_list(mdtypes) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in _commonRetrieveDataMD_list DEBUG util.py:264: os.rename(local, local + '.old.tmp') DEBUG util.py:264: OSError: [Errno 2] No such file or directory DEBUG util.py:354: Child return code was: 1 DEBUG util.py:314: Executing command: ['/bin/umount', '-n', '-l', '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
The server is: koji-hub-1.8.0-1.el6.noarch
The builder version is: koji-builder-1.8.0-1.fc17.noarch
Any suggestion, where to start to debug?
Thank you, Peter
----- Eredeti üzenet -----
Feladó: "Didier Fabert" didier.fabert@gmail.com Címzett: "Discussion of Fedora build system" buildsys@lists.fedoraproject.org Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19 Tárgy: Re: Koji regen-repo error
On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
On 11/15/13 5:48 AM, Didier Fabert wrote:
Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
koji.LockError is only raised in two places, both around fnctl.lockf():
http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
The fcntl manpage says:
ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).
Are you mounting /mnt/koji over nfs? Maybe the nfslock service died? Is something else holding a lot of locks?
Yes, you're right, nfslock was dead on NFS server (el5) After restarting nfslock service on my NFS server, all is OK.
Thanks a lot Mike to pointing me to the right direction.
-- Didier -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On 12/5/13 9:36 AM, Bojtos Péter wrote:
The problem appeared today again, although I set up in kojira to keep repos for 1 week. The root log's end is this:
DEBUG util.py:314: Executing command: /usr/bin/repoquery -c /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} DEBUG util.py:264: Traceback (most recent call last): DEBUG util.py:264: File "/usr/bin/repoquery", line 1510, in <module> DEBUG util.py:264: main(sys.argv) DEBUG util.py:264: File "/usr/bin/repoquery", line 1504, in main DEBUG util.py:264: repoq.runQuery(regexs) DEBUG util.py:264: File "/usr/bin/repoquery", line 982, in runQuery DEBUG util.py:264: pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs) DEBUG util.py:264: File "/usr/bin/repoquery", line 903, in matchPkgs DEBUG util.py:264: pkgs = self.returnPkgList(patterns=items) DEBUG util.py:264: File "/usr/bin/repoquery", line 856, in returnPkgList DEBUG util.py:264: pkgs = self.pkgSack.returnNewestByNameArch(**kwargs) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in <lambda> DEBUG util.py:264: pkgSack = property(fget=lambda self: self._getSacks(), DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in _getSacks DEBUG util.py:264: self.repos.populateSack(which=repos) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 309, in populateSack DEBUG util.py:264: self.doSetup() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 134, in doSetup DEBUG util.py:264: self.retrieveAllMD() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 92, in retrieveAllMD DEBUG util.py:264: downloading = repo._commonRetrieveDataMD_list(mdtypes) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in _commonRetrieveDataMD_list DEBUG util.py:264: os.rename(local, local + '.old.tmp') DEBUG util.py:264: OSError: [Errno 2] No such file or directory DEBUG util.py:354: Child return code was: 1 DEBUG util.py:314: Executing command: ['/bin/umount', '-n', '-l', '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
The server is: koji-hub-1.8.0-1.el6.noarch
The builder version is: koji-builder-1.8.0-1.fc17.noarch
This looks like a problem with mock or repoquery. What version of those do you have installed on your builder?
Any suggestion, where to start to debug?
Thank you, Peter
----- Eredeti üzenet -----
Feladó: "Didier Fabert" didier.fabert@gmail.com Címzett: "Discussion of Fedora build system" buildsys@lists.fedoraproject.org Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19 Tárgy: Re: Koji regen-repo error
On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
On 11/15/13 5:48 AM, Didier Fabert wrote:
Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
koji.LockError is only raised in two places, both around fnctl.lockf():
http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
The fcntl manpage says:
ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).
Are you mounting /mnt/koji over nfs? Maybe the nfslock service died? Is something else holding a lot of locks?
Yes, you're right, nfslock was dead on NFS server (el5) After restarting nfslock service on my NFS server, all is OK.
Thanks a lot Mike to pointing me to the right direction.
-- Didier -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
Mock version is: mock-1.1.32-1.fc17.noarch
Repoquery is in the following package: yum-utils-1.1.31-4.fc17.noarch
Peter
----- Eredeti üzenet -----
Feladó: "Mike Bonnet" mikeb@redhat.com Címzett: buildsys@lists.fedoraproject.org Elküldött üzenetek: Csütörtök, 2013. December 5. 18:43:48 Tárgy: Re: Koji regen-repo error
On 12/5/13 9:36 AM, Bojtos Péter wrote:
The problem appeared today again, although I set up in kojira to keep repos for 1 week. The root log's end is this:
DEBUG util.py:314: Executing command: /usr/bin/repoquery -c /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} DEBUG util.py:264: Traceback (most recent call last): DEBUG util.py:264: File "/usr/bin/repoquery", line 1510, in <module> DEBUG util.py:264: main(sys.argv) DEBUG util.py:264: File "/usr/bin/repoquery", line 1504, in main DEBUG util.py:264: repoq.runQuery(regexs) DEBUG util.py:264: File "/usr/bin/repoquery", line 982, in runQuery DEBUG util.py:264: pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs) DEBUG util.py:264: File "/usr/bin/repoquery", line 903, in matchPkgs DEBUG util.py:264: pkgs = self.returnPkgList(patterns=items) DEBUG util.py:264: File "/usr/bin/repoquery", line 856, in returnPkgList DEBUG util.py:264: pkgs = self.pkgSack.returnNewestByNameArch(**kwargs) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in <lambda> DEBUG util.py:264: pkgSack = property(fget=lambda self: self._getSacks(), DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in _getSacks DEBUG util.py:264: self.repos.populateSack(which=repos) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 309, in populateSack DEBUG util.py:264: self.doSetup() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 134, in doSetup DEBUG util.py:264: self.retrieveAllMD() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 92, in retrieveAllMD DEBUG util.py:264: downloading = repo._commonRetrieveDataMD_list(mdtypes) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in _commonRetrieveDataMD_list DEBUG util.py:264: os.rename(local, local + '.old.tmp') DEBUG util.py:264: OSError: [Errno 2] No such file or directory DEBUG util.py:354: Child return code was: 1 DEBUG util.py:314: Executing command: ['/bin/umount', '-n', '-l', '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
The server is: koji-hub-1.8.0-1.el6.noarch
The builder version is: koji-builder-1.8.0-1.fc17.noarch
This looks like a problem with mock or repoquery. What version of those do you have installed on your builder?
Any suggestion, where to start to debug?
Thank you, Peter
----- Eredeti üzenet -----
Feladó: "Didier Fabert" didier.fabert@gmail.com Címzett: "Discussion of Fedora build system" buildsys@lists.fedoraproject.org Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19 Tárgy: Re: Koji regen-repo error
On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
On 11/15/13 5:48 AM, Didier Fabert wrote:
Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
koji.LockError is only raised in two places, both around fnctl.lockf():
http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
The fcntl manpage says:
ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).
Are you mounting /mnt/koji over nfs? Maybe the nfslock service died? Is something else holding a lot of locks?
Yes, you're right, nfslock was dead on NFS server (el5) After restarting nfslock service on my NFS server, all is OK.
Thanks a lot Mike to pointing me to the right direction.
-- Didier -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
Dear alll,
I applied Remi's patch from the bugzilla ticket: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=969776 It solved the problem. It should be in a newer mock package in fedora/EPEL.
Thank you for your help, Peter
----- Eredeti üzenet -----
Feladó: "Bojtos Péter" ptr@ulx.hu Címzett: "Discussion of Fedora build system" buildsys@lists.fedoraproject.org Elküldött üzenetek: Csütörtök, 2013. December 5. 19:16:23 Tárgy: Re: Koji regen-repo error
Mock version is: mock-1.1.32-1.fc17.noarch
Repoquery is in the following package: yum-utils-1.1.31-4.fc17.noarch
Peter
----- Eredeti üzenet -----
Feladó: "Mike Bonnet" mikeb@redhat.com Címzett: buildsys@lists.fedoraproject.org Elküldött üzenetek: Csütörtök, 2013. December 5. 18:43:48 Tárgy: Re: Koji regen-repo error
On 12/5/13 9:36 AM, Bojtos Péter wrote:
The problem appeared today again, although I set up in kojira to keep repos for 1 week. The root log's end is this:
DEBUG util.py:314: Executing command: /usr/bin/repoquery -c /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} DEBUG util.py:264: Traceback (most recent call last): DEBUG util.py:264: File "/usr/bin/repoquery", line 1510, in <module> DEBUG util.py:264: main(sys.argv) DEBUG util.py:264: File "/usr/bin/repoquery", line 1504, in main DEBUG util.py:264: repoq.runQuery(regexs) DEBUG util.py:264: File "/usr/bin/repoquery", line 982, in runQuery DEBUG util.py:264: pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs) DEBUG util.py:264: File "/usr/bin/repoquery", line 903, in matchPkgs DEBUG util.py:264: pkgs = self.returnPkgList(patterns=items) DEBUG util.py:264: File "/usr/bin/repoquery", line 856, in returnPkgList DEBUG util.py:264: pkgs = self.pkgSack.returnNewestByNameArch(**kwargs) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in
<lambda> DEBUG util.py:264: pkgSack = property(fget=lambda self: self._getSacks(), DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in _getSacks DEBUG util.py:264: self.repos.populateSack(which=repos) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 309, in populateSack DEBUG util.py:264: self.doSetup() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 134, in doSetup DEBUG util.py:264: self.retrieveAllMD() DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/repos.py", line 92, in retrieveAllMD DEBUG util.py:264: downloading = repo._commonRetrieveDataMD_list(mdtypes) DEBUG util.py:264: File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in _commonRetrieveDataMD_list DEBUG util.py:264: os.rename(local, local + '.old.tmp') DEBUG util.py:264: OSError: [Errno 2] No such file or directory DEBUG util.py:354: Child return code was: 1 DEBUG util.py:314: Executing command: ['/bin/umount', '-n', '-l', '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
The server is: koji-hub-1.8.0-1.el6.noarch
The builder version is: koji-builder-1.8.0-1.fc17.noarch
This looks like a problem with mock or repoquery. What version of those do you have installed on your builder?
Any suggestion, where to start to debug?
Thank you, Peter
----- Eredeti üzenet -----
Feladó: "Didier Fabert" didier.fabert@gmail.com Címzett: "Discussion of Fedora build system" buildsys@lists.fedoraproject.org Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19 Tárgy: Re: Koji regen-repo error
On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
On 11/15/13 5:48 AM, Didier Fabert wrote:
Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
koji.LockError is only raised in two places, both around fnctl.lockf():
http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
The fcntl manpage says:
ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).
Are you mounting /mnt/koji over nfs? Maybe the nfslock service died? Is something else holding a lot of locks?
Yes, you're right, nfslock was dead on NFS server (el5) After restarting nfslock service on my NFS server, all is OK.
Thanks a lot Mike to pointing me to the right direction.
-- Didier -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
buildsys@lists.fedoraproject.org
-
Didier FABERT -
Mike Bonnet -
Peter Bojtos