Hi all,
within the whole Meltdown and Spectre story I realized that Koji pushes even security updates to batched only, not directly to stable. In concrete case we have the firefox update [1], which already received 10 positive karma and many users complaining that it takes so long to get it out as a stable update. Shouldn't we change that behaviour to get such security updates out fast when maintainer relies on autopush when karma threshold is reached?
Greetings, Christian
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2018-276558ff6f
On 01/07/2018 01:38 PM, Christian Dersch wrote:
Hi all,
within the whole Meltdown and Spectre story I realized that Koji pushes even security updates to batched only, not directly to stable. In
The critera for bypassing batched is if the update is marked "urgent".
concrete case we have the firefox update [1], which already received 10 positive karma and many users complaining that it takes so long to get it out as a stable update. Shouldn't we change that behaviour to get such security updates out fast when maintainer relies on autopush when karma threshold is reached?
If they are urgent sure... perhaps this should have been so marked? Or the maintainer/submittor can request stable anytime after it has enough karma.
Anyhow, I have pushed it to request stable and will do another f27 push after the current ones finish to get it out today.
kevin
Kevin Fenzi wrote:
The critera for bypassing batched is if the update is marked "urgent".
The problem is, this appears to be insufficient.
I really don't understand why we do this "batched" thing to begin with. Users who want to batch updates have always been able to do it, GNOME Software will even do it for them. Now, those users who want to batch their updates are forced to follow Fedora rel-eng's schedule for the batches rather than being able to pick their own weekday, so how does the server- side batching help them? And those users (like me) who want to get their updates, including security fixes (!) as we see here, as soon as they passed testing are now screwed.
If people insist on that "batched" misfeature, can we please at least get a fast track repository that contains all the batched updates (but no updates that are still in testing and have not been batched yet!)?
Kevin Kofler
On 01/07/2018 08:01 PM, Kevin Kofler wrote:
Kevin Fenzi wrote:
The critera for bypassing batched is if the update is marked "urgent".
The problem is, this appears to be insufficient.
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent. To save users downloads of repodata.
Users who want to batch updates have always been able to do it, GNOME Software will even do it for theNow, those users who want to batch their updates are forced to follow Fedora rel-eng's schedule for the batches rather than being able to pick their own weekday, so how does the server- side batching help them? And those users (like me) who want to get their updates, including security fixes (!) as we see here, as soon as they passed testing are now screwed.
rel-eng's schedule is a cron job at 03:00 on tuesday (so the batch appears on wed's pushes).
There was some discussion about changing the gnome batching based on the Fedora batching, but I don't know whats happened there.
I haven't seen a bunch of urgent updates get blocked by this process. Do you have more data for updates that hit this?
If people insist on that "batched" misfeature, can we please at least get a fast track repository that contains all the batched updates (but no updates that are still in testing and have not been batched yet!)?
I would be very much against additional repos like this.
kevin
Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon as they passed testing no matter whether they actually are urgent.
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save users downloads of repodata.
This does not work in practice because there is almost always at least one urgent update that requires downloading the whole repodata. (And also because maintainers are free to skip batched without giving a reason. I always do this because I consider batching a disservice to my users.)
rel-eng's schedule is a cron job at 03:00 on tuesday (so the batch appears on wed's pushes).
There was some discussion about changing the gnome batching based on the Fedora batching, but I don't know whats happened there.
But what if this is a company that wants to do weekly updates on Monday morning in order to be free from disruptions for the working week? Then they will get the updates up to almost 2 weeks late rather than up to 1 week as both you and they intend.
Batching really needs to be left to the client side!
I haven't seen a bunch of urgent updates get blocked by this process. Do you have more data for updates that hit this?
I have empirically seen security updates end up in the batches, but I have not checked each of them to see whether it actually went through "batched" or just happened to go out on that day.
But again, I think it is essential for users to be able to opt to getting updates without arbitrary delays.
If people insist on that "batched" misfeature, can we please at least get a fast track repository that contains all the batched updates (but no updates that are still in testing and have not been batched yet!)?
I would be very much against additional repos like this.
Why? It would allow you to keep the server-side batching while still allowing those users like me to opt out of it. And the repodata download size for fast track would be minimal if updates that went out to stable get removed from fast track.
Even RHEL has a fast track repository.
Kevin Kofler
On Mon, 08 Jan 2018 19:53:01 +0100 Kevin Kofler kevin.kofler@chello.at wrote:
Batching really needs to be left to the client side!
Right.
I did wonder what was going on with the updates...
dnf-cron, gnome-software etc could all *default* to weekly.
But if I'm doing a dnf update, I want the updates *now*.
On 01/08/2018 10:53 AM, Kevin Kofler wrote:
Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon as they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo. If we update a repo for some minor enhancements it means everyone in the world has to pay for that. If we just push all those out every tuesday and don't update those unless there's something urgent we save everyone a lot of bandwith and us computing time/resources.
To save users downloads of repodata.
This does not work in practice because there is almost always at least one urgent update that requires downloading the whole repodata. (And also because maintainers are free to skip batched without giving a reason. I always do this because I consider batching a disservice to my users.)
There are definitely more days when there are no updates for a particular repo now. Of course there would be even more if you (or those who do likewise) wouldn't skip batched, but probibly we need to explain why more clearly.
...snip...
I would be very much against additional repos like this.
Why? It would allow you to keep the server-side batching while still allowing those users like me to opt out of it. And the repodata download size for fast track would be minimal if updates that went out to stable get removed from fast track.
because it would be a ton more infrastructure and resources.
kevin
Kevin Fenzi wrote:
You also don't want updates-testing to even exist right?
That is not true. I want to leave the decision whether and for how long an update needs to be tested to the package maintainer instead of enforcing minimum testing requirements in the software, because the software can never understand the exact context. Removing updates-testing entirely is not what I want! But this is unrelated to the current issue of artificially delaying updates that satisfy all the criteria for being pushed to stable.
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
So to save people the download, you make a change that totally defeats the point of dnf checking for updates every hour to begin with?
If we update a repo for some minor enhancements it means everyone in the world has to pay for that. If we just push all those out every tuesday and don't update those unless there's something urgent we save everyone a lot of bandwith and us computing time/resources.
This does not work in practice because there are always updates that are not batched.
There are definitely more days when there are no updates for a particular repo now. Of course there would be even more if you (or those who do likewise) wouldn't skip batched, but probibly we need to explain why more clearly.
Are there really? The last couple days, there were basically daily pushes with around 2 updates each.
The batching only makes the daily pushes smaller and not empty, which does not help at all for reducing repodata download size, because there are still no repodata deltas implemented.
because it would be a ton more infrastructure and resources.
Really? Bodhi composes (or triggers the compose of, let's please not discuss the technical details down to that level) 2 repositories per release at each push, of which one (updates-testing) already works pretty much the way the fast track repository would work (updates transit there, but leave again when they reach the next level). How would adding a third level require a ton more resources? It would just need some small changes to the Bodhi implementation ("pushes to batched" would have to be actually pushed, to the fast track repository) and could otherwise use all the existing mirroring infrastructure. And users would be able to opt in to getting stable updates without batching. And even if those users keep the regular repodata downloads enabled, the downloads for fast track would still be much smaller than the repodata downloads for all of stable. And having fast track available would also reduce the proportion of updates that skip batched. (I know I would think twice about skipping batched for my updates if I knew that the users have a way to opt out of the batching. Right now, I don't even consider using batched.)
I see only advantages of such an approach, for minimal infrastructure costs. Almost everything you need is already there!
Kevin Kofler
January 9, 2018 9:59 PM, "Kevin Kofler" kevin.kofler@chello.at wrote:
Kevin Fenzi wrote:
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
So to save people the download, you make a change that totally defeats the point of dnf checking for updates every hour to begin with?
If we update a repo for some minor enhancements it means everyone in the world has to pay for that. If we just push all those out every tuesday and don't update those unless there's something urgent we save everyone a lot of bandwith and us computing time/resources.
This does not work in practice because there are always updates that are not batched.
There are definitely more days when there are no updates for a particular repo now. Of course there would be even more if you (or those who do likewise) wouldn't skip batched, but probibly we need to explain why more clearly.
Are there really? The last couple days, there were basically daily pushes with around 2 updates each.
So, if there ARE updates to be pushed (marked urgent?) and we update metadata ANYWAY, this is a good point to flush everything from batched in this push.
On 01/10/2018 04:01 AM, Tomasz Torcz wrote:
So, if there ARE updates to be pushed (marked urgent?) and we update metadata ANYWAY, this is a good point to flush everything from batched in this push.
We could yeah. We could also only ever push when there are urgent updates...
kevin
On 01/09/2018 12:57 PM, Kevin Kofler wrote:
Kevin Fenzi wrote:
You also don't want updates-testing to even exist right?
That is not true. I want to leave the decision whether and for how long an update needs to be tested to the package maintainer instead of enforcing minimum testing requirements in the software, because the software can never understand the exact context. Removing updates-testing entirely is not what I want! But this is unrelated to the current issue of artificially delaying updates that satisfy all the criteria for being pushed to stable.
Agreed. Thanks for clarifying.
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
So to save people the download, you make a change that totally defeats the point of dnf checking for updates every hour to begin with?
It doesn't do that though. dnf wants the latest metadata so it can let users use that cache for things like searching for packages or listing them or the like.
If we update a repo for some minor enhancements it means everyone in the world has to pay for that. If we just push all those out every tuesday and don't update those unless there's something urgent we save everyone a lot of bandwith and us computing time/resources.
This does not work in practice because there are always updates that are not batched.
I... have seen updates pushes that do not take place when I have been pushing updates, so I assert you are incorrect. True, it doesn't happen as often as I was hoping, but it does and has happened.
There are definitely more days when there are no updates for a particular repo now. Of course there would be even more if you (or those who do likewise) wouldn't skip batched, but probibly we need to explain why more clearly.
Are there really? The last couple days, there were basically daily pushes with around 2 updates each.
At least one of those cases was me pushing firefox, right after a f27-updates push just finished, so yeah, it only had 2 updates in it.
The batching only makes the daily pushes smaller and not empty, which does not help at all for reducing repodata download size, because there are still no repodata deltas implemented.
because it would be a ton more infrastructure and resources.
Really? Bodhi composes (or triggers the compose of, let's please not discuss the technical details down to that level)
...snip...
but the technical details are what matters here. ;) Making another repo, building drpms, and doing all the compose will take time, disk space and cpu cycles and slow all other updates pushes down.
Anyhow, I don't want this to be a back and forth between just us, I'd like to hear some other opinions and proposals and have FESCo decide on some adjustment ehre.
I agree the current setup is not ideal and personally I'm open to adjusting/changing it, but I don't know that I think we should just drop batched entirely. We should come up with something that balances all the various needs (you want updates as soon as they are tested, I want not to update metadata on people all the time, etc).
kevin
Kevin Fenzi wrote:
On 01/09/2018 12:57 PM, Kevin Kofler wrote:
Kevin Fenzi wrote:
If we update a repo for some minor enhancements it means everyone in the world has to pay for that. If we just push all those out every tuesday and don't update those unless there's something urgent we save everyone a lot of bandwith and us computing time/resources.
This does not work in practice because there are always updates that are not batched.
I... have seen updates pushes that do not take place when I have been pushing updates, so I assert you are incorrect. True, it doesn't happen as often as I was hoping, but it does and has happened.
This week, there have been almost daily nonempty update pushes (listing only the SRPMs here, and only the updates that affected me): * Jan 10 (previous batch) * Jan 11 (not batched, gtk3 and microcode-ctl) * Jan 12 (not batched, kernel, dhcp, dnfdragora, hplip, webkitgtk4) * none on Sat Jan 13 (some updates from RPM Fusion, but those have separate repodata, so they don't count) * Jan 14/15 (night, not batched, bluez, python-nbconvert, python-qtconsole) * Jan 15/16 (night, not batched, llvm with all its revdeps, hplip again) and now today (Jan 17)'s batch includes (for me) 135 (!) binary packages (less if you count the SRPMs, but in the end, the binary packages are what really matter).
I don't see how this is helpful.
Kevin Kofler
Thanks, Kevin. Knowing when the updates are actually going out adds important context to this discussion.
On Wed, Jan 17, 2018 at 7:30 AM, Kevin Kofler kevin.kofler@chello.at wrote:
I don't see how this is helpful.
Kevin has a point here. Clearly what we have now is, in practice, not working as intended.
Michael
On 01/17/2018 08:47 AM, mcatanzaro@gnome.org wrote:
Clearly what we have now is, in practice, not working as intended.
The original intent as I understood it from the thread long ago[0] was to reduce the number of updates that go out on non-Tuesdays, and make most updates happen on Tuesdays. The data that Kevin cited seems to be accomplishing that purpose.
[0] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Randy Barlow wrote:
The original intent as I understood it from the thread long ago[0] was to reduce the number of updates that go out on non-Tuesdays, and make most updates happen on Tuesdays. The data that Kevin cited seems to be accomplishing that purpose.
But whom does this help? There are still updates going out daily, the repodata download cost is still there, the notifications too if you aren't doing client-side batching (and if you are, you don't need server-side batching to begin with).
The only difference I see is that some fixes take longer to reach the users and that the huge Tuesday (Wednesday morning for most users) pushes are a huge pain to go through if you want to actually check what you are updating. (Am I the only one who bothers reading update notes?)
Kevin Kofler
On 18. Jan 2018, at 16:13, Kevin Kofler kevin.kofler@chello.at wrote:
Am I the only one who bothers reading update notes?
Nope.
BK
On 01/18/2018 10:13 AM, Kevin Kofler wrote:
But whom does this help? There are still updates going out daily, the repodata download cost is still there, the notifications too if you aren't doing client-side batching (and if you are, you don't need server-side batching to begin with).
It would help people with more minimal package sets more often. On days with just a handful of updates, it's possible that some users don't use any of them and thus don't have to get notified.
There is some discussion around possibly forming a formal policy around batching updates[0]. If we did batch updates more then users would more often get days when they don't need to download metadata, which would be nice.
Randy Barlow wrote:
It would help people with more minimal package sets more often. On days with just a handful of updates, it's possible that some users don't use any of them and thus don't have to get notified.
Even if they don't get notified, they still have to download the whole metadata in the background.
Kevin Kofler
I wrote:
This week, there have been almost daily nonempty update pushes (listing only the SRPMs here, and only the updates that affected me):
- Jan 10 (previous batch)
- Jan 11 (not batched, gtk3 and microcode-ctl)
- Jan 12 (not batched, kernel, dhcp, dnfdragora, hplip, webkitgtk4)
- none on Sat Jan 13 (some updates from RPM Fusion, but those have separate repodata, so they don't count)
- Jan 14/15 (night, not batched, bluez, python-nbconvert, python-qtconsole)
- Jan 15/16 (night, not batched, llvm with all its revdeps, hplip again)
and now today (Jan 17)'s batch includes (for me) 135 (!) binary packages (less if you count the SRPMs, but in the end, the binary packages are what really matter).
And today (Jan 18), the day after the batch, there's yet another nonempty push: redhat-rpm-config/nim-srpm-macros, cmake, jnr-*, python-sphinx, twolame moved from RPM Fusion to Fedora.
Kevin Kofler
Kevin Fenzi kevin@scrye.com wrote:
[…]
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo. If we update a repo for some minor enhancements it means everyone in the world has to pay for that. If we just push all those out every tuesday and don't update those unless there's something urgent we save everyone a lot of bandwith and us computing time/resources.
[…]
BTW, are there technical reasons why the metadata is updated en bloc and not incrementally like for example delta RPMs, or is it just that nobody bothered to implement something like that yet for metadata?
Tim
On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
Kevin Fenzi kevin@scrye.com wrote:
[…]
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo. If we update a repo for some minor enhancements it means everyone in the world has to pay for that. If we just push all those out every tuesday and don't update those unless there's something urgent we save everyone a lot of bandwith and us computing time/resources.
[…]
BTW, are there technical reasons why the metadata is updated en bloc and not incrementally like for example delta RPMs, or is it just that nobody bothered to implement something like that yet for metadata?
There is no implementation that I know of. It's been talked about for many years, but not been implemented.
kevin
On Wed, 2018-01-10 at 08:06 -0800, Kevin Fenzi wrote:
On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
BTW, are there technical reasons why the metadata is updated en bloc and not incrementally like for example delta RPMs, or is it just that nobody bothered to implement something like that yet for metadata?
There is no implementation that I know of. It's been talked about for many years, but not been implemented.
We talked a bit about using casync for delta metadata at Flock last summer, and I even did a small proof of concept to see what kind of savings we'd see, but there were a couple of issues that came up:
* casync creates a large number of very small files (roughly 4k files for yesterday's metadata) that mirrors would need to sync * Users would need to download a significant portion of those files to delta against (anywhere from 0 to all of them, depending on how similar their metadata is)
Downloading any significant number of tiny files over http/1.x is really, really slow as the downloads are serial.
On reflection, one idea that might fix both problems would be if casync could concatenate the cacnk files into one large block (cablk, perhaps), and have the caidx file contain the location of each cacnk in the block file. Using http ranges, you would then be able to download just the parts of the file you wanted (though I have not actually tested whether specifying a large list of http ranges will download faster than serially downloading each individual file).
FWIW, casyncing Jan 4th's metadata over Dec 15th's downloads 9.6MB (in just over 1k cacnk files), while downloading the same metadata directly would total 16MB. More frequent updates would be smaller, but I'm not convinced the difference from an average updates push to another would be less than 2MB.
Jonathan
On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
BTW, are there technical reasons why the metadata is updated en bloc and not incrementally like for example delta RPMs, or is it just that nobody bothered to implement something like that yet for metadata?
This has been discussed for a very long time: https://bugzilla.redhat.com/show_bug.cgi?id=850896
On Jan 9, 2018, at 9:59 AM, Kevin Fenzi kevin@scrye.com wrote:
On 01/08/2018 10:53 AM, Kevin Kofler wrote: Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon as they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata updates fast? This shouldn't be particularly hard -- a tool like casync or even svn should work pretty well. Or it could be a simple ad-hoc thing. Have the mirrors serve up both the whole metadata blob and a list of metadata changes. The client could grab the list of changes from last time, compute a bitwise-identical blob, and verify the signature on that, deltarpm style. (But on the decompressed data, of course -- no need to repeat the mistakes of the past.)
It seems that all this batched stuff is a rather weak hack to work around the extreme inefficiency of Fedora's metadata distribution model.
On 10 January 2018 at 14:23, Andrew Lutomirski luto@mit.edu wrote:
On Jan 9, 2018, at 9:59 AM, Kevin Fenzi kevin@scrye.com wrote:
On 01/08/2018 10:53 AM, Kevin Kofler wrote: Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon as they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata updates fast? This shouldn't be particularly hard -- a tool like casync or even svn should work pretty well. Or it could be a simple
This sounds a lot like the Atomic project and how it does things...
ad-hoc thing. Have the mirrors serve up both the whole metadata blob and a list of metadata changes. The client could grab the list of changes from last time, compute a bitwise-identical blob, and verify the signature on that, deltarpm style. (But on the decompressed data, of course -- no need to repeat the mistakes of the past.)
It seems that all this batched stuff is a rather weak hack to work around the extreme inefficiency of Fedora's metadata distribution model.
All things are a weak hack to work around some inefficiency and an extremely important process to deal with needs. Trying to label something without taking into consideration of the other is where most arguments go pear shaped.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, Jan 10, 2018 at 11:38 AM, Stephen John Smoogen smooge@gmail.com wrote:
On 10 January 2018 at 14:23, Andrew Lutomirski luto@mit.edu wrote:
On Jan 9, 2018, at 9:59 AM, Kevin Fenzi kevin@scrye.com wrote:
On 01/08/2018 10:53 AM, Kevin Kofler wrote: Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon
as
they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
I really don't understand why we do this "batched" thing to begin
with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people
and
are urgent.
But the users were already able to opt to update only weekly. So why
force a
fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata updates fast? This shouldn't be particularly hard -- a tool like casync or even svn should work pretty well. Or it could be a simple
This sounds a lot like the Atomic project and how it does things...
Maybe some of Atomic's infrastructure could be used to distribute metadata for regular old Fedora.
--Andy
On 10 January 2018 at 14:46, Andrew Lutomirski luto@mit.edu wrote:
On Wed, Jan 10, 2018 at 11:38 AM, Stephen John Smoogen smooge@gmail.com wrote:
On 10 January 2018 at 14:23, Andrew Lutomirski luto@mit.edu wrote:
On Jan 9, 2018, at 9:59 AM, Kevin Fenzi kevin@scrye.com wrote:
On 01/08/2018 10:53 AM, Kevin Kofler wrote: Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon as they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
> I really don't understand why we do this "batched" thing to begin > with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata updates fast? This shouldn't be particularly hard -- a tool like casync or even svn should work pretty well. Or it could be a simple
This sounds a lot like the Atomic project and how it does things...
Maybe some of Atomic's infrastructure could be used to distribute metadata for regular old Fedora.
OK clearly I should not try to help with a single sentence email as it didn't help anyone. I should have asked more questions on what you meant by metadata and what you meant by distribution and how svn would have been used instead. I should have also asked if you knew enough about how atomic did things to give an informed opinion on how that was similar to the request. Without that we ended up talking past each other.
On Wed, Jan 10, 2018 at 12:01 PM, Stephen John Smoogen smooge@gmail.com wrote:
On 10 January 2018 at 14:46, Andrew Lutomirski luto@mit.edu wrote:
On Wed, Jan 10, 2018 at 11:38 AM, Stephen John Smoogen smooge@gmail.com wrote:
On 10 January 2018 at 14:23, Andrew Lutomirski luto@mit.edu wrote:
On Jan 9, 2018, at 9:59 AM, Kevin Fenzi kevin@scrye.com wrote:
On 01/08/2018 10:53 AM, Kevin Kofler wrote: Kevin Fenzi wrote: > Well, if this firefox update was urgent, shouldn't it have been > marked > urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon as they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
>> I really don't understand why we do this "batched" thing to begin >> with. > > To reduce the constant flow of updates that are very minor or affect > very few mixed in with the major updates that affect lots of people > and > are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata updates fast? This shouldn't be particularly hard -- a tool like casync or even svn should work pretty well. Or it could be a simple
This sounds a lot like the Atomic project and how it does things...
Maybe some of Atomic's infrastructure could be used to distribute metadata for regular old Fedora.
OK clearly I should not try to help with a single sentence email as it didn't help anyone. I should have asked more questions on what you meant by metadata and what you meant by distribution and how svn would have been used instead.
SVN is probably a poor model, but imagine that the various metadata files (repodata, I think) were check in, uncompressed, to an SVN repo. Then the client could do 'svn up' and take advantage of delta compression. I suspect the server load would be too high, though.
Basically, what I'm getting at is that Fedora seems to distribute comps, prestodelta, primary, filelists, and updateinfo, and they add up to 20MB compressed for the updates repo. AFAICT it gets re-downlaoded from scratch every time, even though the actual list of changes is probably minimal from day to day.
I should have also asked if you knew enough about how atomic did things to give an informed opinion on how that was similar to the request. Without that we ended up talking past each other.
I don't know too much about the details. I suspect that, if the .xml.gz files were, instead, one file per package, then they could be served up from an ostree repo.
On Wed, Jan 10, 2018, at 2:38 PM, Stephen John Smoogen wrote:
This sounds a lot like the Atomic project and how it does things...
Atomic could mean either (rpm)-ostree or Docker/OCI. In the Docker/OCI world search isn't standardized yet AIUI but there's progress on that upstream. I am not aware of the state of things overall there - it'd be really interesting to have someone who is clued in comment.
I do ostree and rpm, which is used for Atomic Host (and the Atomic Workstation). Now we're in the midst of a deeply fundamental rework: https://github.com/projectatomic/rpm-ostree/issues/1081
And very specifically on this topic: https://github.com/projectatomic/rpm-ostree/issues/1127
(That issue includes my thoughts on e.g. ostree vs git for metadata)
That said, if someone is actually planning to *work* on this, the best place to discuss is http://lists.rpm.org/pipermail/rpm-ecosystem/ as you're more likely to have the relevant people involved. I do plan to take some of my ideas from the above rpm-ostree issue to that list, but it's going to come after implementing jigdo.
One followup that should help people understand things:
When someone pushes an update to a package that isn't in Atomic Host (or Workstation), *and* one is using rpm-ostree in "pure ostree" mode (i.e. you never ran `rpm-ostree install`), then checking for updates just uses libostree, which like any sane image system, makes this a very cheap operation.
In the case of libostree, checking for "no changes" is just a few HTTP requests for tiny files.
Say for example you've got a bunch of CentOS containers on your FAH system (IMO this is not just a valid use case but one I'd encourage) - you don't see an available update whenever a new Firefox or whatever appears.
We were also doing ~two week batching for FAH updates before Bodhi started doing batching...the interaction between those really could be better...that's a whole other topic.
Anyways though to finish the explanation: the second you do `rpm-ostree install libvirt` for example (as I have done on my home server), you're doing *both* systems; now libdnf comes into play, and it's the same RPM metadata from Fedora with all the same performance penalty. Smoothing out this dramatic cliff of a transition is part of the reason I'm working on rpm-ostree jigdo.
On Wed, Jan 10, 2018 at 2:23 PM, Andrew Lutomirski luto@mit.edu wrote:
On Jan 9, 2018, at 9:59 AM, Kevin Fenzi kevin@scrye.com wrote:
On 01/08/2018 10:53 AM, Kevin Kofler wrote: Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
Urgency is always in the eye of the beholder. I as a user consider all security updates "urgent", and in addition, I want ALL updates as soon as they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent.
But the users were already able to opt to update only weekly. So why force a fixed schedule on them?
To save all the Fedora users in the world from having to update metadata for minor changes. Since there's a hourly dnf makecache every user in the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata updates fast? This shouldn't be particularly hard -- a tool like casync or even svn should work pretty well. Or it could be a simple ad-hoc thing. Have the mirrors serve up both the whole metadata blob and a list of metadata changes. The client could grab the list of changes from last time, compute a bitwise-identical blob, and verify the signature on that, deltarpm style. (But on the decompressed data, of course -- no need to repeat the mistakes of the past.)
It seems that all this batched stuff is a rather weak hack to work around the extreme inefficiency of Fedora's metadata distribution model.
This was actually prototyped two years ago with zsync: https://github.com/rh-lab-q/deltametadata-prototype
It worked, but it was slow since it wasn't implemented in librepo and of course we never got it implemented in the infrastructure.
Details: https://github.com/rh-lab-q/deltametadata-prototype/wiki/Deltametadata-of-re...
The Plan of 2017 that never got done: https://github.com/rh-lab-q/deltametadata-prototype/wiki/Plan-2017
On Mon, Jan 8, 2018 at 5:39 PM, Kevin Fenzi kevin@scrye.com wrote:
On 01/07/2018 08:01 PM, Kevin Kofler wrote:
Kevin Fenzi wrote:
The critera for bypassing batched is if the update is marked "urgent".
The problem is, this appears to be insufficient.
Well, if this firefox update was urgent, shouldn't it have been marked urgent?
I thought for some reason that all updates marked as security were automatically urgent, maybe I'm misremembering, but if not it might be good to do that as a RFE that way all security updates go out non batched.
I really don't understand why we do this "batched" thing to begin with.
To reduce the constant flow of updates that are very minor or affect very few mixed in with the major updates that affect lots of people and are urgent. To save users downloads of repodata.
Users who want to batch updates have always been able to do it, GNOME Software will even do it for theNow, those users who want to batch their updates are forced to follow Fedora rel-eng's schedule for the batches rather than being able to pick their own weekday, so how does the server- side batching help them? And those users (like me) who want to get their updates, including security fixes (!) as we see here, as soon as they passed testing are now screwed.
rel-eng's schedule is a cron job at 03:00 on tuesday (so the batch appears on wed's pushes).
There was some discussion about changing the gnome batching based on the Fedora batching, but I don't know whats happened there.
I haven't seen a bunch of urgent updates get blocked by this process. Do you have more data for updates that hit this?
If people insist on that "batched" misfeature, can we please at least get a fast track repository that contains all the batched updates (but no updates that are still in testing and have not been batched yet!)?
I would be very much against additional repos like this.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Mon, Jan 8, 2018 at 8:17 PM, Peter Robinson pbrobinson@gmail.com wrote:
I thought for some reason that all updates marked as security were automatically urgent, maybe I'm misremembering, but if not it might be good to do that as a RFE that way all security updates go out non batched.
Most security updates are simply not urgent, and we really don't want to pester users with these on a daily basis. So it would be nice if security updates went through batched by default.
Still, this update was marked as a high-severity security update. It probably makes sense that those should skip batched, in addition to updates marked as urgent.
Michael
On Tue, Jan 09, 2018 at 02:17:43AM +0000, Peter Robinson wrote:
I thought for some reason that all updates marked as security were automatically urgent, maybe I'm misremembering, but if not it might be good to do that as a RFE that way all security updates go out non batched.
There was some discussion. There are plenty of updates which have some security implication but which aren't really urgent. I'd rather them stay described as non-urgent security updates than people start not calling them security updates. When they _are_ important to get to users quickly, they should be marked that way. But that's always a tradeoff -- more time in testing gives more of a chance to find regressions or other probems.
Also, I think everyone in this discussion on _this_ list who would like updates faster should probably be using updates-testing. Or at least _looking_ at updates-testing. You can always pull individual updates from there on a per-package basis, and doing this helps everyone else.
Matthew Miller wrote:
Also, I think everyone in this discussion on _this_ list who would like updates faster should probably be using updates-testing. Or at least _looking_ at updates-testing. You can always pull individual updates from there on a per-package basis, and doing this helps everyone else.
I want tested updates faster. I don't want untested updates.
I also don't think it would be a productive use of my time to go through all testing updates every day to see which ones are batched for stable. This is something that could easily be automated by software (i.e., by having Bodhi push the updates to a fast track repository), why should I be doing this by hand?
What I do normally do is read the update notes of every update that I am about to apply. This is also a reason why I hate the batching, because the batches are huge and painful to check through. I prefer more frequent smaller pushes that I can easily read through. But what I can definitely tell you is that most updates do not contain enough information in the update notes to really know their impact, so using that as a basis for applying or not applying some update from updates-testing is not going to scale either.
By the way, the reason I did not voice these complaints in the thread initially announcing this change is that I was both too angry and too busy at that time to come up with a polite reply, and besides, the change was already implemented when announced anyway.
Kevin Kofler
within the whole Meltdown and Spectre story I realized that Koji pushes even security updates to batched only, not directly to stable. In
The critera for bypassing batched is if the update is marked "urgent".
Or once you push it stable and it's queued for batched the maintainer then has an option to push stable in the same spot as usual, just needs the maintainer to push the button again.
concrete case we have the firefox update [1], which already received 10 positive karma and many users complaining that it takes so long to get it out as a stable update. Shouldn't we change that behaviour to get such security updates out fast when maintainer relies on autopush when karma threshold is reached?
If they are urgent sure... perhaps this should have been so marked? Or the maintainer/submittor can request stable anytime after it has enough karma.
Anyhow, I have pushed it to request stable and will do another f27 push after the current ones finish to get it out today.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On 01/08/2018 01:36 AM, Kevin Fenzi wrote:
On 01/07/2018 01:38 PM, Christian Dersch wrote:
Hi all,
within the whole Meltdown and Spectre story I realized that Koji pushes even security updates to batched only, not directly to stable. In
The critera for bypassing batched is if the update is marked "urgent".
concrete case we have the firefox update [1], which already received 10 positive karma and many users complaining that it takes so long to get it out as a stable update. Shouldn't we change that behaviour to get such security updates out fast when maintainer relies on autopush when karma threshold is reached?
If they are urgent sure... perhaps this should have been so marked? Or the maintainer/submittor can request stable anytime after it has enough karma.
Anyhow, I have pushed it to request stable and will do another f27 push after the current ones finish to get it out today.
Seems to be my fault then, didn't know that the urgent/high state has the meaning there.
ma.
devel@lists.stg.fedoraproject.org
-
Andrew Clayton -
Andrew Lutomirski -
Benjamin Kircher -
Christian Dersch -
Colin Walters -
Jonathan Dieter -
Kevin Fenzi -
Kevin Kofler -
Martin Stransky -
Matthew Miller -
mcatanzaro＠gnome.org -
Neal Gompa -
Peter Robinson -
Randy Barlow -
Samuel Sieb -
Stephen John Smoogen -
Tim Landscheidt -
Tomasz Torcz