Hello packagers & devs,
we have just deployed a new release of COPR eco-system. The changes made are mainly internal but should help to solve some issues we have been tackling recently. The most important upgrades are:
1) copr-dist-git service is now parallel ("multi-process"). That means that if there is a problem with a job import, the other jobs won't be blocked. Also we added import-job timing out so if import gets stuck, it doesn't get stuck forever.
2) Waiting-queue logic has been rewritten. I believe there will be not much of a perceived performance improvement but we managed to cut the relevant code length almost by half and simplify some bits.
3) We have fixed appstream meta-data building - a bug that also influenced forking. When a project was forked the rpm re-signing process kept looping. Other actions were blocked by this. Hence, in addition to fix the root cause (the meta-data building), we also needed to fix the forking logic to handle the errors better.
4) We have introduced a first bits of support for modularity
So that's it. I hope COPR will serve you well. COPR team
On Thursday, August 18, 2016 1:30:57 PM CEST Michal Novotny wrote:
- Waiting-queue logic has been rewritten. I believe there will be not much
of a perceived performance improvement but we managed to cut the relevant code length almost by half and simplify some bits.
Hms, thanks for the info. That commit 1c51da66151df0e1add7ca877568d16584793bef removed a lot of code in scheduler, which was a debugged with care and took (really) non-trival efforts to get it working right ...
What exactly changed, is there link for patch review to simplify reading, to follow reasoning behind?
Does that mean that we no longer preallocate workers (looks like 'max_workers' backend parameter disappeared)? Where is the queue mainained now (python retask is missing)?
Pavel
Hello Pavel,
On Thu, Aug 25, 2016 at 2:21 PM, Pavel Raiskup praiskup@redhat.com wrote:
On Thursday, August 18, 2016 1:30:57 PM CEST Michal Novotny wrote:
- Waiting-queue logic has been rewritten. I believe there will be not
much
of a perceived performance improvement but we managed to cut the relevant code length almost by half and simplify some bits.
Hms, thanks for the info. That commit 1c51da66151df0e1add7ca877568d1 6584793bef removed a lot of code in scheduler, which was a debugged with care and took (really) non-trival efforts to get it working right ...
What exactly changed, is there link for patch review to simplify reading, to follow reasoning behind?
Does that mean that we no longer preallocate workers (looks like
'max_workers' backend parameter disappeared)? Where is the queue mainained now (python retask is missing)?
the queue is now maintained only in frontend and backend is served with tasks one by one. Workers are being spawned "on demand". Their maximum count is determined by configured maximum number of virtual machines that can be allocated. The code emerged as a reaction to the problem described in https://bugzilla.redhat.com/ show_bug.cgi?id=1344805.
Pavel
On Thursday, August 25, 2016 9:43:53 PM CEST Michal Novotny wrote:
the queue is now maintained only in frontend and backend is served with tasks one by one. Workers are being spawned "on demand". Their maximum count is determined by configured maximum number of virtual machines that can be allocated. The code emerged as a reaction to the problem described in https://bugzilla.redhat.com/ show_bug.cgi?id=1344805.
Thanks for the link, but - ouch :( that sounds like a non-trivial step back. At least for those who don't have super powerful Open Stack cloud which would give them instances within 10s.
Pavel
copr-devel@lists.fedorahosted.org
-
Michal Novotny -
Pavel Raiskup