Is something going on with the mirrors today?
Today, I see we are getting this error on an F10 install using preupgrade:
Cannot download the kickstart file...
Regards, Gerry
Gerry Reno wrote:
Is something going on with the mirrors today?
Today, I see we are getting this error on an F10 install using preupgrade:
Cannot download the kickstart file...
Regards, Gerry
Can anyone else verify this? We just tried this preupgrade again with a different mirror and we're still seeing the same error.
Regards, Gerry
Is something going on with the mirrors today?
Yes. The sync (from master to mirrors) may be in progress for F11 beta release next Tuesday: tens of GB per mirror. The floodgates for rawhide have been opened (changes that were held back during the freeze for F11 beta): more hundreds of packages. It's also a weekend, spring break on many academic calendars in the US, time for the final rounds of the US NCAA college basketball tournament, etc. The reliability/supervision of mirrors might be even more erratic than usual.
John Reiser wrote:
Is something going on with the mirrors today?
Yes. The sync (from master to mirrors) may be in progress for F11 beta release next Tuesday: tens of GB per mirror. The floodgates for rawhide have been opened (changes that were held back during the freeze for F11 beta): more hundreds of packages. It's also a weekend, spring break on many academic calendars in the US, time for the final rounds of the US NCAA college basketball tournament, etc. The reliability/supervision of mirrors might be even more erratic than usual.
terrific. We've been trying numerous mirrors and they all appear to have the same problem. So I guess nobody can perform any upgrades today. They should not screw up every mirror while doing these mirror updates. Some mirrors should still be left operational and then update them later.
Regards, Gerry
On 03/28/2009 10:40 AM, John Reiser wrote:
Is something going on with the mirrors today?
Yes. The sync (from master to mirrors) may be in progress for F11 beta release next Tuesday: tens of GB per mirror. The floodgates for rawhide have been opened (changes that were held back during the freeze for F11 beta): more hundreds of packages. It's also a weekend, spring break on many academic calendars in the US, time for the final rounds of the US NCAA college basketball tournament, etc. The reliability/supervision of mirrors might be even more erratic than usual.
Also, the Mozilla guys released a Firefox security update yesterday afternoon (we have updates already out for all supported releases), and many of our mirrors are also mozilla mirrors...
John Reiser wrote:
Yes. The sync (from master to mirrors) may be in progress for F11 beta release next Tuesday: tens of GB per mirror. The floodgates for rawhide have been opened (changes that were held back during the freeze for F11 beta): more hundreds of packages. It's also a weekend, spring break on many academic calendars in the US, time for the final rounds of the US NCAA college basketball tournament, etc. The reliability/supervision of mirrors might be even more erratic than usual.
Try the European mirrors then. :-)
Kevin Kofler
Once upon a time, Kevin Kofler kevin.kofler@chello.at said:
John Reiser wrote:
Yes. The sync (from master to mirrors) may be in progress for F11 beta release next Tuesday: tens of GB per mirror. The floodgates for rawhide have been opened (changes that were held back during the freeze for F11 beta): more hundreds of packages. It's also a weekend, spring break on many academic calendars in the US, time for the final rounds of the US NCAA college basketball tournament, etc. The reliability/supervision of mirrors might be even more erratic than usual.
Try the European mirrors then. :-)
Or one run by somebody who's team is already out (or was never in to begin with)! :-)
Kevin Kofler wrote:
John Reiser wrote:
... spring break on many academic calendars in the US, time for the final rounds of the US NCAA college basketball tournament, ...
Try the European mirrors then. :-)
yum, preupgrade, and related FedoraProject software often does its best to hide such details from the user, and to make it inconvenient to choose anything other than the default round-robin by geo-IP. [jigdo is an exception.]
For instance, by default yumdownloader generates no record of which mirror you get upon HTTP 302 [or 303] redirection from the master mirror. In practice you see the ultimate URL only if the connection [attempt] actually times out. This enables the mirror administrator(s) to hide from users in most cases of poor service (missing file, stale file, slower than necessary, ...)
On Sat, Mar 28, 2009 at 09:18:42PM -0700, John Reiser wrote:
Kevin Kofler wrote:
John Reiser wrote:
... spring break on many academic calendars in the US, time for the final rounds of the US NCAA college basketball tournament, ...
Try the European mirrors then. :-)
yum, preupgrade, and related FedoraProject software often does its best to hide such details from the user, and to make it inconvenient to choose anything other than the default round-robin by geo-IP. [jigdo is an exception.]
The mirror choice algorithm is far more complicated than round-robin by geo-IP. Of course, it isn't perfect, but is a decent approximation most of the time.
For instance, by default yumdownloader generates no record of which mirror you get upon HTTP 302 [or 303] redirection from the master mirror. In practice you see the ultimate URL only if the connection [attempt] actually times out.
Patches or enhancement bugs filed suggesting ways in which the tools you're interested would be more verbose are welcome.
This enables the mirror administrator(s) to hide from users in most cases of poor service (missing file, stale file, slower than necessary, ...)
This, my friend, is just a pot shot. We're _lucky_ to have so many mirrors, and in most cases, mirrors with very responsive administrators; not to mention the Infrastructure team that keeps it all humming. If you have genuine problems with particular mirrors, feel free to let the folks in #fedora-admin know, or file a ticket in the Fedora Infrastructure trac (https://fedorahosted.org/fedora-infrastructure) with the details.
Thanks, Matt Fedora Mirror Wrangler, and proud of it! :-)
Matt Domsch wrote:
On Sat, Mar 28, 2009 at 09:18:42PM -0700, John Reiser wrote:
For instance, by default yumdownloader generates no record of which mirror you get upon HTTP 302 [or 303] redirection from the master mirror. In practice you see the ultimate URL only if the connection [attempt] actually times out.
Patches or enhancement bugs filed suggesting ways in which the tools you're interested would be more verbose are welcome.
This enables the mirror administrator(s) to hide from users in most cases of poor service (missing file, stale file, slower than necessary, ...)
This, my friend, is just a pot shot. We're _lucky_ to have so many mirrors, and in most cases, mirrors with very responsive administrators; not to mention the Infrastructure team that keeps it all humming. If you have genuine problems with particular mirrors, feel free to let the folks in #fedora-admin know, or file a ticket in the Fedora Infrastructure trac (https://fedorahosted.org/fedora-infrastructure) with the details.
As a mostly end-user of the mirror system, I consider the remark to be on-target and as specific as possible with the current mirror system and usage. It is not a pot shot.
Rescue mode and install from physical DVD are important to me. I try to help make them better by testing during development. During the alpha period for Fedora 11, I have composed install DVD via pungi from rawhide on about 20 days. Usually I try both i386 and x86_64. About 40% of the time things work very well. The downloading required by the compose goes quickly, and testing the resulting DVD verifies a fix, sheds new light on a continuing problem, or reveals a new bug. About 40% of the time things go OK: average download speed, perhaps a snag that is easily fixable, perhaps progress on rescue mode, etc. About 20% of the time the results are poor. Downloading takes a long time, fails, or fetches stale data; or the compose or booting or rescue fails, sometimes in mysterious ways. The exact cause is murky, and frustrating to determine. The mirror system ought to be near the bottom of the list of suspects, but it isn't. Somehow it feels like the mirror system is providing about 90% over-all reliability when I'm expecting 97-99%.
I start the pungi compose a few hours after receiving the rawhide report. As long as there are fewer than 100 packages or so, then I consider this to be time enough for changes to propagate through the mirror system. I precede the compose with "yum update" to check for changes to pungi, anaconda, yum, mkinitrd, and rpm; and if found then I update them before the compose. Other packages wait for update until after the compose, which is done as yum localupdate from /var/cache/pungi/rawhide/packages. [Thus, a very low-tech mirror for propagating daily changes to two places. My connection is cable modem with 12Mbit/s advertised but often <= 1MByte/s.]
What do I see? Many times success, but too often murk. Different packages changed from rawhide report to "yum update" to pungi download, and not just the big spikes for translation string updates, mass keyed signings, deliberate release holds, etc. Sporadic delays of 10 seconds or more for establishing connections, as viewed in System Monitor. Throughput varying from 80KB/s to 3MB/s for observable durations on non-small files. Most importantly: little indication of why, and little hope of finding out.
Pungi does report "Connection timed out" so that is obvious. The log file sometimes contains one of: Requested Range Not Satisfiable Package does not match intended download HTTP Error 404: Not Found with an associated URL. That's good, but the same message may appear consecutively for more than one mirror. Where does the blame lie: with each listed mirror, or with the master, or with pungi (or its delegate)? Which mirror gets credit for succeeding with this package, thus *NOT* appearing in a complaint at the moment? When the package that is downloaded is not the version listed in rawhide report: why?
The mirror system provides me with much good use, but enough not-so-good use that I grouse about it.
On Sun, Mar 29, 2009 at 01:57:36PM -0700, John Reiser wrote:
I start the pungi compose a few hours after receiving the rawhide report. As long as there are fewer than 100 packages or so, then I consider this to be time enough for changes to propagate through the mirror system.
I wish it were that simple. Perhaps if we had push-mirroring in place, it would be. As it stands, each mirror schedules its own syncs. Some every hour or so, some every 6, some once a day. Rawhide being the fastest churning piece on the mirror system, exposes these latencies in ways that the other content simply doesn't.
During the F11-Alpha cycle, some bugs in MirrorManager have been discovered and fixed which may have lead to more redirections to stale mirrors. These should be reduced, but I can't promise they'll be completely eliminated. MM shouldn't return stale mirrors more than ~6 hours stale. If you need a more deterministic update latency for your specific test purposes, I recommend you point your baseurl entries at download.fedora.redhat.com.
Thanks, Matt
Once upon a time, John Reiser jreiser@BitWagon.com said:
For instance, by default yumdownloader generates no record of which mirror you get upon HTTP 302 [or 303] redirection from the master mirror. In practice you see the ultimate URL only if the connection [attempt] actually times out. This enables the mirror administrator(s) to hide from users in most cases of poor service (missing file, stale file, slower than necessary, ...)
Maybe yum should be changed to send the end-user's email address, so us nefarious mirror admins can make downloads "slower than necessary" just for you.
I spent several hours this weekend getting my mirror ready for F11 Beta and then trying to catch up on rawhide (which had huge churn after the beta freeze was released). I guess I could have gone without sleep and cut a few hours off the total time; would that have made you happier?
On Sat, 2009-03-28 at 21:18 -0700, John Reiser wrote:
[jigdo is an exception.]
jigdo is no exception. The URLs listed in the jigdo file use mirrormanager to redirect you to a mirror using the same algorithm as yum getting a mirrorlist.
Jesse Keating wrote:
On Sat, 2009-03-28 at 21:18 -0700, John Reiser wrote:
[jigdo is an exception.]
jigdo is no exception. The URLs listed in the jigdo file use mirrormanager to redirect you to a mirror using the same algorithm as yum getting a mirrorlist.
The results were vastly different when I used jigdo two months ago. With jigdo(-lite) I get to [have to] choose one or more specific mirrors (or meta-mirrors) if I wish. I also saw the actual resulting mirror for each package as it downloaded, and it was easy to recognize one or two mirrors that were broken, and to use ^C to hasten a re-choosing instead of waiting for timeout. With yum tools the choice of repo is mostly hidden in a file that must be edited beforehand, and the actual resulting server for each package often is not revealed.
John Reiser wrote:
yum, preupgrade, and related FedoraProject software often does its best to hide such details from the user, and to make it inconvenient to choose anything other than the default round-robin by geo-IP. [jigdo is an exception.]
I just hardcode a known good mirror in the .repo files (and keep the mirrorlist as fallback).
That said, the mirrorlists actually work pretty well these days. I remember the days when all the .repo files were configured to default to a single heavily overloaded server at Red Hat. Then we got a round robin of some random mirrors all over the world, usually with disastrous resulting bandwidth. Now we get local, usually fast mirrors by default.
Kevin Kofler