Hello,
lately, COPR pending job queues are holding jobs for pretty long time (even hours). This is a buggy behaviour and we will be doing our best to fix this issue in the following days.
Thank your for your patience COPR team
Hey,
On Wed, Nov 1, 2017 at 3:42 PM, Michal Novotny clime@redhat.com wrote:
Hello,
lately, COPR pending job queues are holding jobs for pretty long time (even hours). This is a buggy behaviour and we will be doing our best to fix this issue in the following days.
as we have found out yesterday. One of the main issue was 'createrepo.lock.lock' file in @rubygems/rubygems/srpm-builds/ directory on copr-backend that caused 'createrepo' command to hand indefinitely after a srpm build and hence the builder that was allocated for rubygems srpm builds was never released afterwards. Together with #160 SRPM build may be allocated on multiple builders at once https://pagure.io/copr/copr/issue/160, this probably was causing the really long waiting times. Our queueing mechaism will still need to be (and will be) improved but this was the main issue very likely.
Sorry for the waiting COPR team
Hi Clime,
Le 03/11/2017 à 06:39, Michal Novotny a écrit :
Hey,
On Wed, Nov 1, 2017 at 3:42 PM, Michal Novotny <clime@redhat.com mailto:clime@redhat.com> wrote:
Hello, lately, COPR pending job queues are holding jobs for pretty long time (even hours). This is a buggy behaviour and we will be doing our best to fix this issue in the following days.
as we have found out yesterday. One of the main issue was 'createrepo.lock.lock' file in @rubygems/rubygems/srpm-builds/ directory on copr-backend that caused 'createrepo' command to hand indefinitely after a srpm build and hence the builder that was allocated for rubygems srpm builds was never released afterwards. Together with #160 SRPM build may be allocated on multiple builders at once https://pagure.io/copr/copr/issue/160, this probably was causing the really long waiting times. Our queueing mechaism will still need to be (and will be) improved but this was the main issue very likely.
Sorry for the waiting COPR team
All ppc64le plateforms are in pending mode since yesterday, is it relative to the above ?
Regards, Jean-Marc
Hello Jean-Marc,
answer is below...
On Tue, Nov 28, 2017 at 11:20 AM, Jean-Marc Liger < jean-marc.liger@parisdescartes.fr> wrote:
Hi Clime, Le 03/11/2017 à 06:39, Michal Novotny a écrit :
Hey,
On Wed, Nov 1, 2017 at 3:42 PM, Michal Novotny clime@redhat.com wrote:
Hello,
lately, COPR pending job queues are holding jobs for pretty long time (even hours). This is a buggy behaviour and we will be doing our best to fix this issue in the following days.
as we have found out yesterday. One of the main issue was 'createrepo.lock.lock' file in @rubygems/rubygems/srpm-builds/ directory on copr-backend that caused 'createrepo' command to hand indefinitely after a srpm build and hence the builder that was allocated for rubygems srpm builds was never released afterwards. Together with #160 SRPM build may be allocated on multiple builders at once https://pagure.io/copr/copr/issue/160, this probably was causing the really long waiting times. Our queueing mechaism will still need to be (and will be) improved but this was the main issue very likely.
Sorry for the waiting COPR team
All ppc64le plateforms are in pending mode since yesterday, is it relative to the above ?
No, not really. This is an OpenStack problem as we are not able to spawn any new ppc64le builder from COPR. We will try to build a new ppc64le image based on Fedora27 and see if it helps and if not, we will give another try to solve it together with OpenStack folks.
clime
Regards, Jean-Marc
copr-devel@lists.stg.fedorahosted.org