I wonder, is it possible for Fedora Infrastructure team to implement announcements of new repository pushes over Fedora Message Bus[1]?
By new repository pushes I mean updates to 'fedora', 'updates' and 'updates-testing' repositories for all Fedora releases as well as updates to 'branched' and 'rawhide' trees.
The benefits of this solution: 1. Mirrors that download from dl.fp.o could start syncing very soon after an update is pushed (after a random timeout to decrease the load). 2. Those mirrors wouldn't have to periodically query the server for updates. 3. The duration when those mirrors are outdated would be shorter.
I can just guess about other people, but in our office I believe the message bus would help a lot. Currently we query the master server every 4 hours for updates. That presents a lot of bandwidth/IO load on both sides. Using the message bug there would be no useless queries. Also it would help us stay up-to-date more often that we currently are (if 'branched' is updated an hour after our last query, there are 3 hours wasted until we run rsync again).
This solution could be extended in the future to all Fedora public mirrors, so that they could announce using our message bus when they have finished syncing and users that sync from that particular mirror would know when to start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my private mirror, I would wait for their announcement on our message bus).
What do you think?
[1] join #fedora-fedmsg and stay there for a while, if you don't know what message bus is
Love the idea. Please describe how to hook up applications to receive such notifications. I'd love to have MirrorManager's update-master-directory-list cronjob listen for such instead.
-- Matt Domsch Technology Strategist Dell | Office of the CTO ________________________________________ From: infrastructure-bounces@lists.fedoraproject.org [infrastructure-bounces@lists.fedoraproject.org] On Behalf Of Kamil Paral [kparal@redhat.com] Sent: Monday, September 10, 2012 7:56 AM To: infrastructure@lists.fedoraproject.org Subject: announcing repository updates over Fedora Message Bus
I wonder, is it possible for Fedora Infrastructure team to implement announcements of new repository pushes over Fedora Message Bus[1]?
By new repository pushes I mean updates to 'fedora', 'updates' and 'updates-testing' repositories for all Fedora releases as well as updates to 'branched' and 'rawhide' trees.
The benefits of this solution: 1. Mirrors that download from dl.fp.o could start syncing very soon after an update is pushed (after a random timeout to decrease the load). 2. Those mirrors wouldn't have to periodically query the server for updates. 3. The duration when those mirrors are outdated would be shorter.
I can just guess about other people, but in our office I believe the message bus would help a lot. Currently we query the master server every 4 hours for updates. That presents a lot of bandwidth/IO load on both sides. Using the message bug there would be no useless queries. Also it would help us stay up-to-date more often that we currently are (if 'branched' is updated an hour after our last query, there are 3 hours wasted until we run rsync again).
This solution could be extended in the future to all Fedora public mirrors, so that they could announce using our message bus when they have finished syncing and users that sync from that particular mirror would know when to start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my private mirror, I would wait for their announcement on our message bus).
What do you think?
[1] join #fedora-fedmsg and stay there for a while, if you don't know what message bus is _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Love the idea. Please describe how to hook up applications to receive such notifications. I'd love to have MirrorManager's update-master-directory-list cronjob listen for such instead.
I don't have any hands-on experience with Fedora Message Bus. But we can ask the Messaging SIG guys [1]. On their mindmap they even have MirrorManager mentioned as one of their use cases.
My naive idea is that there would be a simple script (Python or whatever the qpid library has bindings for) that would connect to the bus and wait for the user-defined event, then trigger a user-defined command. We could distribute the script as part of the MirrorManager toolkit. I don't have any MirrorManager experience either, so maybe this could be easily integrated into the existing set of scripts.
This is basically why I propose this, to learn more about the topic and see whether there is someone willing to go ahead and try to implement this.
On Sep 10, 2012, at 7:15, Kamil Paral kparal@redhat.com wrote:
Love the idea. Please describe how to hook up applications to receive such notifications. I'd love to have MirrorManager's update-master-directory-list cronjob listen for such instead.
I don't have any hands-on experience with Fedora Message Bus. But we can ask the Messaging SIG guys [1]. On their mindmap they even have MirrorManager mentioned as one of their use cases.
My naive idea is that there would be a simple script (Python or whatever the qpid library has bindings for) that would connect to the bus and wait for the user-defined event, then trigger a user-defined command. We could distribute the script as part of the MirrorManager toolkit. I don't have any MirrorManager experience either, so maybe this could be easily integrated into the existing set of scripts.
This is basically why I propose this, to learn more about the topic and see whether there is someone willing to go ahead and try to implement this.
[1] http://fedoraproject.org/wiki/Messaging_SIG _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
I wrote some scripts to do similar functionality a few years ago when we were talking AMQP. Users could subscribe to packages and get gnome notifications when bohdi or koji events took place. I also actually made that mindmap at the Toronto fudcon a few years back. Maybe I can look into this again in the next week when I get back from vacation.
Brennan Ashton
On Mon, Sep 10, 2012 at 08:56:26AM -0400, Kamil Paral wrote:
I wonder, is it possible for Fedora Infrastructure team to implement announcements of new repository pushes over Fedora Message Bus[1]?
By new repository pushes I mean updates to 'fedora', 'updates' and 'updates-testing' repositories for all Fedora releases as well as updates to 'branched' and 'rawhide' trees.
Some of this should already be happening. There is a "org.fedoraproject.prod.bodhi.mashtask.sync.done" message that should be getting emitted, but we don't have a way to check the history and verify yet. https://bugzilla.redhat.com/show_bug.cgi?id=853252 should take care of that.
"branched" and "rawhide" are another story. We'd need to add "fedmsg-logger" statements to some of the scripts in http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure where yet.
The benefits of this solution:
- Mirrors that download from dl.fp.o could start syncing very soon after an update is pushed (after a random timeout to decrease the load).
- Those mirrors wouldn't have to periodically query the server for updates.
- The duration when those mirrors are outdated would be shorter.
I can just guess about other people, but in our office I believe the message bus would help a lot. Currently we query the master server every 4 hours for updates. That presents a lot of bandwidth/IO load on both sides. Using the message bug there would be no useless queries. Also it would help us stay up-to-date more often that we currently are (if 'branched' is updated an hour after our last query, there are 3 hours wasted until we run rsync again).
This solution could be extended in the future to all Fedora public mirrors, so that they could announce using our message bus when they have finished syncing and users that sync from that particular mirror would know when to start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my private mirror, I would wait for their announcement on our message bus).
What do you think?
This sounds great for the future. I've been working on docs this week (not quite done yet). I'll have docs out soon with examples on how to do something like this.
What do you think?
This sounds great for the future. I've been working on docs this week (not quite done yet). I'll have docs out soon with examples on how to do something like this.
Exciting! Please don't forget to share on planet.fp.o once you have something. This is great stuff.
On Mon, Sep 10, 2012 at 12:07:12PM -0400, Kamil Paral wrote:
Exciting! Please don't forget to share on planet.fp.o once you have something. This is great stuff.
Updated docs:
- http://fedmsg.rtfd.org/ - http://fedmsg.readthedocs.org/en/latest/consuming.html
I'm going to hold off sharing on the planet until we can get a port open for external message consumption of the production bus. Right now, only staging is available (with practically no activity).
After the freeze, hopefully we can get that in place. An announcement pointing folks to the docs will be more timely then. :)
On Mon, Sep 10, 2012 at 11:30:15AM -0400, Ralph Bean wrote:
"branched" and "rawhide" are another story. We'd need to add "fedmsg-logger" statements to some of the scripts in http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure where yet.
Just an update:
I talked with dgilmore about this today and he'd like to wait until a "composedb" app is written so we can have all the fedmsg statements in a nice, centralized place (not strewn out across all the releng scripts).
On Wed, Sep 12, 2012 at 11:38:50AM -0400, Ralph Bean wrote:
On Mon, Sep 10, 2012 at 11:30:15AM -0400, Ralph Bean wrote:
"branched" and "rawhide" are another story. We'd need to add "fedmsg-logger" statements to some of the scripts in http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure where yet.
Just an update:
I talked with dgilmore about this today and he'd like to wait until a "composedb" app is written so we can have all the fedmsg statements in a nice, centralized place (not strewn out across all the releng scripts).
How far along is the composedb? Considering the patch to enable fedmsg for rawhide/branched mashes is a few trivial 1-liners[0], it seems like a quick win to just apply/deploy them. The transition to the composedb will be transparent to the message consumers, as the topics won't change.
Ralph currently doesn't have privs to commit/test them, but I'd be happy to help.
luke
[0]: http://threebean.org/patches/0001-Add-fedmsg-logger-statements-to-compose-sc...
infrastructure@lists.fedoraproject.org