Hey,
TL;DR - after another out-of-space incident, Koji is back online accepting builds.
I created new 600 GB volume and copied over the content of the /mnt/koji plus moved there directories "packages" and "external-repos" from /mnt/tmp (ephemeral volume which was filling up).
Filesystem Size Used Avail Use% Mounted on /dev/xvde1 5.7G 3.0G 2.7G 53% / none 1.8G 0 1.8G 0% /dev/shm tmpfs 4.0G 44M 4.0G 2% /tmp /dev/xvdab1 591G 378G 183G 68% /mnt/koji /dev/xvdt1 50G 27G 21G 56% /mnt2/koji /dev/xvdm1 148G 108G 33G 77% /mnt2/xvdm /dev/xvdj1 367G 30G 322G 9% /mnt/tmp
Old volumes xvdt1 and xvdm1 are mounted under /mnt2 and can be removed and returned to the EBS pool once we confirm koji is working again.
First, I moved everything under single volume until I realized this was deadly slow. For building processes, local storage must be used. This was the reason why there was such a mess with symlinks. But due to fact that the instance was small flavor, ephemeral volume is small and limiting. Fortunately I found "packages" dirs together with "external-repos" 'moveable' to EBS volume (Koji is sensitive in multi-volume setups beacuse heavy use of hardlinks). Builds can be slower in this setup. We should have enough space now (183 G + 322 G).
I tested a mash of Katello repos and everything went fine. However, when I attempted to run the Katello build job I got the following:
http://ci.theforeman.org/job/katello_packaging_build_matrix_rpm_rubygem_kate...
152275 build (katello-nightly-rhel6, ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm): open (kojibuilder2) -> FAILED: Fault: <Fault 1: 'Traceback (most recent call last):\n File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1114, in runTask\n response = (handler.run(),)\n File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 146, in run\n return self.handler(*self.params,**self.opts)\n File "/usr/sbin/kojid", line 731, in handler\n h = self.readSRPMHeader(srpm)\n File "/usr/sbin/kojid", line 802, in readSRPMHeader\n fo = koji.openRemoteFile(relpath, **opts)\n File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1376, in openRemoteFile\n fo = open(fn)\nIOError: [Errno 116] Stale file handle: '/mnt/koji/work/cli-build/1410382239.066185.bHVVRsri/ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm'\n'> 0 free 0 open 0 done 1 failed
Eric
On Wed, Sep 10, 2014 at 4:52 PM, Lukas Zapletal lzap@redhat.com wrote:
Hey,
TL;DR - after another out-of-space incident, Koji is back online accepting builds.
I created new 600 GB volume and copied over the content of the /mnt/koji plus moved there directories "packages" and "external-repos" from /mnt/tmp (ephemeral volume which was filling up).
Filesystem Size Used Avail Use% Mounted on /dev/xvde1 5.7G 3.0G 2.7G 53% / none 1.8G 0 1.8G 0% /dev/shm tmpfs 4.0G 44M 4.0G 2% /tmp /dev/xvdab1 591G 378G 183G 68% /mnt/koji /dev/xvdt1 50G 27G 21G 56% /mnt2/koji /dev/xvdm1 148G 108G 33G 77% /mnt2/xvdm /dev/xvdj1 367G 30G 322G 9% /mnt/tmp
Old volumes xvdt1 and xvdm1 are mounted under /mnt2 and can be removed and returned to the EBS pool once we confirm koji is working again.
First, I moved everything under single volume until I realized this was deadly slow. For building processes, local storage must be used. This was the reason why there was such a mess with symlinks. But due to fact that the instance was small flavor, ephemeral volume is small and limiting. Fortunately I found "packages" dirs together with "external-repos" 'moveable' to EBS volume (Koji is sensitive in multi-volume setups beacuse heavy use of hardlinks). Builds can be slower in this setup. We should have enough space now (183 G + 322 G).
-- Later, Lukas #lzap Zapletal
-- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
I am not hitting this attempting a build through Jenkins:
Created task: 152480 Task info: http://koji.katello.org/koji/taskinfo?taskID=152480 Watching tasks (this may be safely interrupted)... 152480 build (katello-nightly-rhel6, katello-1.5.0-14.201409110135git71a484b.el6.src.rpm): free 152480 build (katello-nightly-rhel6, katello-1.5.0-14.201409110135git71a484b.el6.src.rpm): free -> open (kojibuilder2) 152483 buildArch (katello-1.5.0-14.201409110135git71a484b.el6.src.rpm, noarch): free 152483 buildArch (katello-1.5.0-14.201409110135git71a484b.el6.src.rpm, noarch): free -> open (kojibuilder1) 152483 buildArch (katello-1.5.0-14.201409110135git71a484b.el6.src.rpm, noarch): open (kojibuilder1) -> closed 0 free 1 open 1 done 0 failed 152480 build (katello-nightly-rhel6, katello-1.5.0-14.201409110135git71a484b.el6.src.rpm): open (kojibuilder2) -> FAILED: Fault: <Fault 1: "<type 'exceptions.OSError'>: [Errno 18] Invalid cross-device link"> 0 free 0 open 1 done 1 failed
On Wed, Sep 10, 2014 at 5:13 PM, Eric D Helms ericdhelms@gmail.com wrote:
I tested a mash of Katello repos and everything went fine. However, when I attempted to run the Katello build job I got the following:
http://ci.theforeman.org/job/katello_packaging_build_matrix_rpm_rubygem_kate...
152275 build (katello-nightly-rhel6, ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm): open (kojibuilder2) -> FAILED: Fault: <Fault 1: 'Traceback (most recent call last):\n File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1114, in runTask\n response = (handler.run(),)\n File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 146, in run\n return self.handler(*self.params,**self.opts)\n File "/usr/sbin/kojid", line 731, in handler\n h = self.readSRPMHeader(srpm)\n File "/usr/sbin/kojid", line 802, in readSRPMHeader\n fo = koji.openRemoteFile(relpath, **opts)\n File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1376, in openRemoteFile\n fo = open(fn)\nIOError: [Errno 116] Stale file handle: '/mnt/koji/work/cli-build/1410382239.066185.bHVVRsri/ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm'\n'> 0 free 0 open 0 done 1 failed
Eric
On Wed, Sep 10, 2014 at 4:52 PM, Lukas Zapletal lzap@redhat.com wrote:
Hey,
TL;DR - after another out-of-space incident, Koji is back online accepting builds.
I created new 600 GB volume and copied over the content of the /mnt/koji plus moved there directories "packages" and "external-repos" from /mnt/tmp (ephemeral volume which was filling up).
Filesystem Size Used Avail Use% Mounted on /dev/xvde1 5.7G 3.0G 2.7G 53% / none 1.8G 0 1.8G 0% /dev/shm tmpfs 4.0G 44M 4.0G 2% /tmp /dev/xvdab1 591G 378G 183G 68% /mnt/koji /dev/xvdt1 50G 27G 21G 56% /mnt2/koji /dev/xvdm1 148G 108G 33G 77% /mnt2/xvdm /dev/xvdj1 367G 30G 322G 9% /mnt/tmp
Old volumes xvdt1 and xvdm1 are mounted under /mnt2 and can be removed and returned to the EBS pool once we confirm koji is working again.
First, I moved everything under single volume until I realized this was deadly slow. For building processes, local storage must be used. This was the reason why there was such a mess with symlinks. But due to fact that the instance was small flavor, ephemeral volume is small and limiting. Fortunately I found "packages" dirs together with "external-repos" 'moveable' to EBS volume (Koji is sensitive in multi-volume setups beacuse heavy use of hardlinks). Builds can be slower in this setup. We should have enough space now (183 G + 322 G).
-- Later, Lukas #lzap Zapletal
-- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
candlepin@lists.stg.fedorahosted.org
-
Eric D Helms -
Lukas Zapletal