I'm getting 404 errors from several places to which mirrormanager wants to send me to get https://download.fedoraproject.org/pub/alt/atomic/stable/Cloud-Images/x86_64... (Specifically, mirrors.rit.edu and mirrors.kernel.org.) Presumbably, these just haven't synced yet. But doesn't mirrormanager get feedback about when mirrors are fresh?
On Thu, Jan 28, 2016 at 08:41:19AM -0500, Matthew Miller wrote:
I'm getting 404 errors from several places to which mirrormanager wants to send me to get https://download.fedoraproject.org/pub/alt/atomic/stable/Cloud-Images/x86_64... (Specifically, mirrors.rit.edu and mirrors.kernel.org.) Presumbably, these just haven't synced yet. But doesn't mirrormanager get feedback about when mirrors are fresh?
I can try to give a few reasons (excuses) why you were redirected to a wrong file.
* The file does not exist anymore on the master mirror. Not being 100% sure how the file names are constructed but it seems to include a date and the file on the master mirror (and a few other mirrors includes a '.2'). So it seems like the file was recreated. Using the new file name I am redirected to a mirror which has the file.
* /pub/alt (also known as the MM category 'Fedora Other') on the master mirror is only scanned once per day (12:15 UTC). As the scanning takes a very long time and as it creates a high I/O load the less important MM categories like 'Fedora Other' and 'Fedora Archive' are only scanned once a day.
* mirrors.rit.edu and mirrors.kernel.org are both running report_mirror to update the status of their mirror after the sync. We trust mirrors running report_mirror. We only get the information that they claim the transmitted directories are up to date. The actual content is not checked when using report_mirror. In the current case both mirrors have been crawled by since the last report_mirror, but at the time of the crawl the file 'Fedora-Cloud-Atomic-23-20160127.2.x86_64.qcow2' (with the '.2') was not yet part of the database as it has a time of 'Jan 27 17:29' which is after the previous master mirror scan. By now the new file should be part of the database and soon the crawler should detect it also on the mirrors.
So there are many reasons why it did fail but the main reason probably is, that due to large amount of files it is difficult to have real-time view of what is on the mirrors and the master mirror. Finding the right balance between massive amounts of I/O on the master and mirrors and freshness of the MM database is the hard part.
Adrian
On Thu, Jan 28, 2016 at 03:53:55PM +0100, Adrian Reber wrote:
- /pub/alt (also known as the MM category 'Fedora Other') on the master mirror is only scanned once per day (12:15 UTC). As the scanning takes a very long time and as it creates a high I/O load the less important MM categories like 'Fedora Other' and 'Fedora Archive' are only scanned once a day.
This suggests that it's probably time to think about moving Atomic out of /pub/alt.
Hi,
I'm getting 404 errors from several places to which mirrormanager wants to send me to get https://download.fedoraproject.org/pub/alt/atomic/stable/Cloud-Images/x86_64... (Specifically, mirrors.rit.edu and mirrors.kernel.org.) Presumbably, these just haven't synced yet. But doesn't mirrormanager get feedback about when mirrors are fresh?
Well, yes, but it looks like this is a file that was (recently) removed (it's also gone from dl.fedoraproject.org, the master mirrors). So at that point, all bets are off, as I don't think mirrormanager deletes anything from its database, and it has seen that the mirror had the bits at some point.
With kind regards, Patrick Uiterwijk Fedora Infra
On Thu, Jan 28, 2016 at 09:53:58AM -0500, Patrick Uiterwijk wrote:
Well, yes, but it looks like this is a file that was (recently) removed (it's also gone from dl.fedoraproject.org, the master mirrors).
Ah! So the question is: why does the web site point to the wrong thing?
----- 元のメッセージ -----
On Thu, Jan 28, 2016 at 09:53:58AM -0500, Patrick Uiterwijk wrote:
Well, yes, but it looks like this is a file that was (recently) removed (it's also gone from dl.fedoraproject.org, the master mirrors).
Ah! So the question is: why does the web site point to the wrong thing?
Indeed. And for that, I'll refer you to our great friends at the Websites team.
With kind regards, Patrick Uiterwijk Fedora Infra
On Thu, 28 Jan 2016 10:20:14 -0500 (EST) Patrick Uiterwijk puiterwijk@redhat.com wrote:
----- 元のメッセージ -----
On Thu, Jan 28, 2016 at 09:53:58AM -0500, Patrick Uiterwijk wrote:
Well, yes, but it looks like this is a file that was (recently) removed (it's also gone from dl.fedoraproject.org, the master mirrors).
Ah! So the question is: why does the web site point to the wrong thing?
Indeed. And for that, I'll refer you to our great friends at the Websites team.
FWIW, I think this was all due to the compose needing some manual help and thus ending up with a '2' in the name instead of what was expected.
I think it's all cleared up now.
kevin
infrastructure@lists.fedoraproject.org