If we had a "koji benchmark" tool, what would you like it to do?
I've written an Ansible playbook that automatically sets up Koji according to our documentation, and I want to exercise the hub to see how it performs. We can use a benchmark tool to understand changes that could impact performance. As one example, we probably want to compare CentOS 7, 8, and Fedora.
What scalability problems have you encountered with Koji in the wild?
I'm thinking of things like: - Tagging thousands of builds quickly - Contention on the sessions table - Maintaining thousands of builders - Ensuring kojira runs in a timely manner
When we have test data, even if the benchmarks are fairly synthetic, it's easier to see the impact of code changes before we ship.
(As an aside, I started writing a really simple "echo" and "login" benchmark tool using the txkoji library, but I found that Twisted's built-in XML-RPC client only supports HTTP/1.0, so no keepalives!)
- Ken
Dne 05. 06. 20 v 22:12 Ken Dreyer napsal(a):
What scalability problems have you encountered with Koji in the wild?
* How long it takes to generate SRPM? - when I hit that, it was in fact issue in (dist-)git. * How long it takes from building package A until I can submit build of package B which build-requires that package A? (real life issue with mergerepo). * How long it takes to run Mock with current distribution config?
Hi,
On 6/5/20 10:12 PM, Ken Dreyer wrote:
What scalability problems have you encountered with Koji in the wild?
* How long does it take to build a docker image/VM?
This goes beyond Koji, I guess, but we're seeing a slowdown of qemu processes over time and builds start timing out until you reboot the builder. We've been unable to track this issue down so far.
Cheers, Alex
* How long run of koji-gc takes * How quickly are +- no-op tasks consumed (tagNotification) * What is the impact of queue size to task waiting time (There is an observation that koji doesn't scale well if there are thousands of tasks in the queue)
po 8. 6. 2020 v 15:35 odesílatel Alex alex.m.lists2@gmail.com napsal:
Hi,
On 6/5/20 10:12 PM, Ken Dreyer wrote:
What scalability problems have you encountered with Koji in the wild?
- How long does it take to build a docker image/VM?
This goes beyond Koji, I guess, but we're seeing a slowdown of qemu processes over time and builds start timing out until you reboot the builder. We've been unable to track this issue down so far.
Cheers, Alex _______________________________________________ koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.o...
On Fri, Jun 05, 2020 at 02:12:51PM -0600, Ken Dreyer wrote:
If we had a "koji benchmark" tool, what would you like it to do?
I've written an Ansible playbook that automatically sets up Koji according to our documentation, and I want to exercise the hub to see how it performs. We can use a benchmark tool to understand changes that could impact performance. As one example, we probably want to compare CentOS 7, 8, and Fedora.
Are you looking to just test the hub? Or also builders?
And a lot of this will depend on the database... and the builders and the storage speed, etc.
The fedora koji db is public, so you're welcome to use that, but not sure it's a 'common' case. :)
What scalability problems have you encountered with Koji in the wild?
I'm thinking of things like:
- Tagging thousands of builds quickly
- Contention on the sessions table
- Maintaining thousands of builders
- Ensuring kojira runs in a timely manner
- tons of side tags, how long to newrepo them all - patalogical packages like texlive hitting timeouts uploading, etc. - writing out signatures of a ton of packages. - response time for various web pages on the hub?
When we have test data, even if the benchmarks are fairly synthetic, it's easier to see the impact of code changes before we ship.
Yeah.
kevin
On Sat, Jun 13, 2020 at 6:14 PM Kevin Fenzi kevin@scrye.com wrote:
On Fri, Jun 05, 2020 at 02:12:51PM -0600, Ken Dreyer wrote:
If we had a "koji benchmark" tool, what would you like it to do?
I've written an Ansible playbook that automatically sets up Koji according to our documentation, and I want to exercise the hub to see how it performs. We can use a benchmark tool to understand changes that could impact performance. As one example, we probably want to compare CentOS 7, 8, and Fedora.
Are you looking to just test the hub? Or also builders?
And a lot of this will depend on the database... and the builders and the storage speed, etc.
You're right. Thanks to everyone who responded so far! This made me realize I should think a little bigger here. Originally I was just thinking about benchmarking the hub alone because there are a lot of things to measure and improve there (like mass-tagging), but several of the responses made me think it makes sense to track more processes end-to-end.
For example, SRPM generation would require dist-git and it's related to how efficiently we perform Git operations. Off the top of my head, the Koji SCM class simply runs "git clone -n", and we could probably change that to always run shallow clones instead so that we're not fetching so much history and unrelated branches.
Image generation is another important one, because it would implicitly test compatibility and performance of qemu and docker (eventually buildah).
- Ken
koji-devel@lists.stg.fedorahosted.org
-
Alex -
Ken Dreyer -
Kevin Fenzi -
Miroslav Suchý -
Tomas Kopecek