On Thu, May 28, 2020 at 8:19 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:

Good Morning Everyone,

I know this question has already been raised a few times, but I think we should raise it once more: what do we see as future for loopabull?

It is currently triggered on 4 topics (3 from prod and 1 from stg) to do basically three actions:

• Flag commit successfully built in koji, in other words it adds these flags to dist-git: https://src.fedoraproject.org/rpms/mingw-filesystem/c/717f2a929bd25b62a0427e...
• Flag when the Fedora CI start testing a PR
• Flag when the Fedora CI finished testing a PR (and thus reports Pass/Fail)

Wait, wait, wait, what?! These aren't things natively supported by the software we use (Koji and Zuul)? It seems like the place to start here would be to move this functionality to the right places. A koji plugin and an extension for the Zuul pagure driver to do those things would make a ton more sense here...

On Thu, 2020-05-28 at 08:30 -0400, Neal Gompa wrote:

On Thu, May 28, 2020 at 8:19 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:

...

Good Morning Everyone,

I know this question has already been raised a few times, but I think we should raise it once more: what do we see as future for loopabull?

It is currently triggered on 4 topics (3 from prod and 1 from stg) to do basically three actions:

• Flag commit successfully built in koji, in other words it adds these flags to dist-git: https://src.fedoraproject.org/rpms/mingw-filesystem/c/717f2a929bd25b62a0427e...
• Flag when the Fedora CI start testing a PR
• Flag when the Fedora CI finished testing a PR (and thus reports Pass/Fail)

Wait, wait, wait, what?! These aren't things natively supported by the software we use (Koji and Zuul)? It seems like the place to start here would be to move this functionality to the right places. A koji plugin and an extension for the Zuul pagure driver to do those things would make a ton more sense here...

Why, though?

I quite like the design of small consumer components to do specific jobs in response to messages. Lots of the bits I maintain work that way. Why is it necessarily better for Koji to grow the ability to change stuff in dist-git than to have a little go-between to do the job?

On Thu, May 28, 2020 at 10:31 AM Adam Williamson adamwill@fedoraproject.org wrote:

Why, though?

I quite like the design of small consumer components to do specific jobs in response to messages. Lots of the bits I maintain work that way. Why is it necessarily better for Koji to grow the ability to change stuff in dist-git than to have a little go-between to do the job?

One nice property of doing these things as separate services is that they are more resilient to cascading failures.

Tomas found some problems recently with Koji failing to retry sending messages when a messagebus connection gets into a certain (rare) state where it is unavailable but proton is confused. Koji ends up dropping those messages on the floor, but in the meantime there is more pressure on the koji-hub application. I could envision a worse scenario where Koji has to send even more "events" reliably to a bunch of other services that could be responding very slowly, etc.

The same thing happens with Pagure Git hooks - unfortunately the more we pile in there, the slower everything becomes.

- Ken

On Fri, May 29, 2020 at 03:18:58PM +0200, Clement Verna wrote:

On Thu, 28 May 2020 at 14:27, Pierre-Yves Chibon <[1]pingou@pingoured.fr> wrote:

 Good Morning Everyone,

 I know this question has already been raised a few times, but I think we
 should
 raise it once more: what do we see as future for loopabull?

 It is currently triggered on 4 topics (3 from prod and 1 from stg) to do
 basically
 three actions:
 - Flag commit successfully built in koji, in other words it adds these
 flags
   to dist-git:
  
 [2]https://src.fedoraproject.org/rpms/mingw-filesystem/c/717f2a929bd25b62a0427e8e5c3792a0939dbfce
 - Flag when the Fedora CI start testing a PR
 - Flag when the Fedora CI finished testing a PR (and thus reports
 Pass/Fail)

 Upstream released yesterday a 0.0.7 release which brings supports for
 fedora-messaging (contributed by your servitor).
 Looking at the code, it should be python3 compatible, but it doesn't say
 specifically in the setup.py and I honestly don't remember if I've
 tested that
 or not.
 The package has been orphaned in Fedora for over 10 months and has thus
 been
 retired.

 I've had a chat with upstream yesterday and they are still interested in
 the
 project but more as a pet project and their time is just like the rest
 of us,
 limited for pet projects these days.
 That being said the code base is really quite small and involves
 technologies
 we're already using in other places (python-pika, celery, rabbitmq,
 ansible...)
 so there isn't really anything new there.

 One of its limitation currently is with secrets, how to pass/specify
 them.
 This is something we could circumvent via ansible-vault or so, but it
 needs a
 little investigation.

 I basically see three ways forward with this:
 - We continue with loopabull and we need to check its python3 support,
 how to
   deal with secrets, if we can get it to run in openshift & so on.
 - We look for something else, similar. The requirements being:
   - Run a task when seeing a message in our message bus
   - Handle secrets
   - Scalable up/down
   - Runnable in openshift is a bonus
   - Preferably in a language we can debug (python++, potentially rust)
 - We write something that fits our needs and requirements

There is a PR[0] in fedora-messaging to add a 'run' callback that would let you execute any command, I think that might be a nice solution and I think it would meet most of the requirements.   [0] - [3]https://github.com/fedora-infra/fedora-messaging/pull/163

Isn't that the equivalent of having us write a custom fedora-messaging consumer for each task we want to automate? In a way I like this, it's simple(r), really straight forward, constraint and self-contained. There is no risk that a previous task prevents a following one to be executed. On the other hand it also means that if we move to, say fed-messaging, in the future, we will have to convert more consumers.

If that's a trade off we're willing to live with, then I'm ok with it as well.

Pierre

On Thu, May 28, 2020 at 02:18:38PM +0200, Pierre-Yves Chibon wrote:

Good Morning Everyone,

I know this question has already been raised a few times, but I think we should raise it once more: what do we see as future for loopabull?

It is currently triggered on 4 topics (3 from prod and 1 from stg) to do basically three actions:

• Flag commit successfully built in koji, in other words it adds these flags to dist-git: https://src.fedoraproject.org/rpms/mingw-filesystem/c/717f2a929bd25b62a0427e...
• Flag when the Fedora CI start testing a PR
• Flag when the Fedora CI finished testing a PR (and thus reports Pass/Fail)

Upstream released yesterday a 0.0.7 release which brings supports for fedora-messaging (contributed by your servitor). Looking at the code, it should be python3 compatible, but it doesn't say specifically in the setup.py and I honestly don't remember if I've tested that or not. The package has been orphaned in Fedora for over 10 months and has thus been retired.

I've had a chat with upstream yesterday and they are still interested in the project but more as a pet project and their time is just like the rest of us, limited for pet projects these days. That being said the code base is really quite small and involves technologies we're already using in other places (python-pika, celery, rabbitmq, ansible...) so there isn't really anything new there.

One of its limitation currently is with secrets, how to pass/specify them. This is something we could circumvent via ansible-vault or so, but it needs a little investigation.

I basically see three ways forward with this:

• We continue with loopabull and we need to check its python3 support, how to deal with secrets, if we can get it to run in openshift & so on.
• We look for something else, similar. The requirements being:
  • Run a task when seeing a message in our message bus
  • Handle secrets
  • Scalable up/down
  • Runnable in openshift is a bonus
  • Preferably in a language we can debug (python++, potentially rust)
• We write something that fits our needs and requirements

Would you know something that fits our requirements and that we could just run? If not, do you have a preferred way between options #1 and #3?

So I took a few hours and started poking at this. I've ended up with: https://pagure.io/fedora-infra/toddlers (a bunch of small programs running around: toddlers :))

It's basically, one fedora-messaging consumer and plugins.

Some of its advantages: - We can run it in openshift easily and we should be able to scale it up/down as needed - Managing secrets is just like managing secrets for all our apps in openshift. - Relies on our main rabbitmq server and thus has monitoring (while loopabull ran its own, which granted we could monitor but that's something we've not done while we have this in place for our main cluster and easily configurable via ansible).

I would still like to add some unit-tests for this but from empirical testing (ie: running it locally) it is doing what it should be doing and the two/three tasks loopabull have already been ported.

Let me know what you think of it! :)

Pierre