I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to value 100 after beta freeze is over.
The above MBS setting controls how many parallel component builds MBS can run at the same time. Current value is only 20. That value was defined 3 years ago, in March 2017 when there were much fewer modules in Fedora. Our Koji has capacity to run many more builds. Currently we have 158 Koji builders in default channel that are able to run RPM builds. Their total capacity is 566. Therefore it should be safe to increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since Koji builds submitted by MBS have lower priority than default for non-modular component builds submitted by packagers, so packager-submitted builds will take precedence over builds submitted by MBS.
-- Mikolaj Izdebski
On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski mizdebsk@redhat.com wrote:
I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to value 100 after beta freeze is over.
The above MBS setting controls how many parallel component builds MBS can run at the same time. Current value is only 20. That value was defined 3 years ago, in March 2017 when there were much fewer modules in Fedora. Our Koji has capacity to run many more builds. Currently we have 158 Koji builders in default channel that are able to run RPM builds. Their total capacity is 566. Therefore it should be safe to increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since Koji builds submitted by MBS have lower priority than default for non-modular component builds submitted by packagers, so packager-submitted builds will take precedence over builds submitted by MBS.
Does the koji numbers above take into account architectures? While have 158 builders they are spread out over a lot of architectures with ~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and I guess ~70 x86_64 builders. So while we could build a lot more x86_64 builds at once, could we completely swamp the other arches and actually cause problems for even completing the x86_64 ones?
If the architecture would be a problem, should we look at the smaller/slowest number of builders and work out what its capacity is and not go much above that?
-- Mikolaj Izdebski _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On Mon, Mar 11, 2019 at 10:43 AM Stephen John Smoogen smooge@gmail.com wrote:
On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski mizdebsk@redhat.com wrote:
I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to value 100 after beta freeze is over.
The above MBS setting controls how many parallel component builds MBS can run at the same time. Current value is only 20. That value was defined 3 years ago, in March 2017 when there were much fewer modules in Fedora. Our Koji has capacity to run many more builds. Currently we have 158 Koji builders in default channel that are able to run RPM builds. Their total capacity is 566. Therefore it should be safe to increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since Koji builds submitted by MBS have lower priority than default for non-modular component builds submitted by packagers, so packager-submitted builds will take precedence over builds submitted by MBS.
Does the koji numbers above take into account architectures? While have 158 builders they are spread out over a lot of architectures with ~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and I guess ~70 x86_64 builders. So while we could build a lot more x86_64 builds at once, could we completely swamp the other arches and actually cause problems for even completing the x86_64 ones?
If the architecture would be a problem, should we look at the smaller/slowest number of builders and work out what its capacity is and not go much above that?
That should not be a problem. Koji can handle thousands of builds submitted at the same time. This happens, for example, during mass rebuilds.
-- Mikolaj Izdebski
On Mon, 11 Mar 2019 at 06:55, Mikolaj Izdebski mizdebsk@redhat.com wrote:
On Mon, Mar 11, 2019 at 10:43 AM Stephen John Smoogen smooge@gmail.com wrote:
On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski mizdebsk@redhat.com wrote:
I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to value 100 after beta freeze is over.
The above MBS setting controls how many parallel component builds MBS can run at the same time. Current value is only 20. That value was defined 3 years ago, in March 2017 when there were much fewer modules in Fedora. Our Koji has capacity to run many more builds. Currently we have 158 Koji builders in default channel that are able to run RPM builds. Their total capacity is 566. Therefore it should be safe to increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since Koji builds submitted by MBS have lower priority than default for non-modular component builds submitted by packagers, so packager-submitted builds will take precedence over builds submitted by MBS.
Does the koji numbers above take into account architectures? While have 158 builders they are spread out over a lot of architectures with ~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and I guess ~70 x86_64 builders. So while we could build a lot more x86_64 builds at once, could we completely swamp the other arches and actually cause problems for even completing the x86_64 ones?
If the architecture would be a problem, should we look at the smaller/slowest number of builders and work out what its capacity is and not go much above that?
That should not be a problem. Koji can handle thousands of builds submitted at the same time. This happens, for example, during mass rebuilds.
Yeah.. sorry I didn't explain things clearly. Koji can handle it but does it make the developer experience worse across the board? We could do a mass rebuild every day and koji wouldn't break a sweat.. but developers actually wanting to get stuff out of Fedora would hate it because the queues for builds would be so long. My questions should have stated that.
On Mon, Mar 11, 2019 at 12:41 PM Stephen John Smoogen smooge@gmail.com wrote:
On Mon, 11 Mar 2019 at 06:55, Mikolaj Izdebski mizdebsk@redhat.com wrote:
On Mon, Mar 11, 2019 at 10:43 AM Stephen John Smoogen smooge@gmail.com wrote:
On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski mizdebsk@redhat.com wrote:
I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to value 100 after beta freeze is over.
The above MBS setting controls how many parallel component builds MBS can run at the same time. Current value is only 20. That value was defined 3 years ago, in March 2017 when there were much fewer modules in Fedora. Our Koji has capacity to run many more builds. Currently we have 158 Koji builders in default channel that are able to run RPM builds. Their total capacity is 566. Therefore it should be safe to increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since Koji builds submitted by MBS have lower priority than default for non-modular component builds submitted by packagers, so packager-submitted builds will take precedence over builds submitted by MBS.
Does the koji numbers above take into account architectures? While have 158 builders they are spread out over a lot of architectures with ~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and I guess ~70 x86_64 builders. So while we could build a lot more x86_64 builds at once, could we completely swamp the other arches and actually cause problems for even completing the x86_64 ones?
If the architecture would be a problem, should we look at the smaller/slowest number of builders and work out what its capacity is and not go much above that?
That should not be a problem. Koji can handle thousands of builds submitted at the same time. This happens, for example, during mass rebuilds.
Yeah.. sorry I didn't explain things clearly. Koji can handle it but does it make the developer experience worse across the board? We could do a mass rebuild every day and koji wouldn't break a sweat.. but developers actually wanting to get stuff out of Fedora would hate it because the queues for builds would be so long. My questions should have stated that.
As I mentioned in the first email, builds submitted by MBS have lower priority than default. If all Koji builders are loaded with modular builds then developer may need to wait a bit before one of them completes. But once any builder has free capacity, it will pick up developer-submitted builds before MBS-submitted ones.
-- Mikolaj Izdebski
+1 also. thank you for answering my questions.
On Mon, 11 Mar 2019 at 14:31, Kevin Fenzi kevin@scrye.com wrote:
+1 to the change here.
If it gives us any problems we can adjust it later...
kevin
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Mikolaj Izdebski -
Stephen John Smoogen