Hello, this is a repost of a message to fedora-devel, as this seems to be the place for it.
From: Jesse Keating jkeating@redhat.com
On Friday 08 June 2007 14:24:47 Anthony Bryan wrote:
I was hoping Fedora could investigate using Metalinks for their ISO downloads. Metalink is an XML format for listing all the ways you can get a file or collection of files (mirrors + their location, rsync, p2p) along with checksums to automatically repair a file in case of error, signatures, language, OS/arch, and other metadata. It's mainly used for large files like ISOs, where errors can be very frustrating.
It's supported by about 20 programs on unix, mac, and win, including aria2 (already in the Fedora repos). It's used by openSUSE, OpenOffice.org, cURL, and many other distributions.
Here's a screenshot of a Metalink download in the DownThemAll Firefox extension (nightly build). What you don't see are all the mirrors and checksums. http://code.downthemall.net/maierman/metaselect4.png
This is something interesting, and I wonder if we could make use of MirrorManager ( https://hosted.fedoraproject.org/projects/mirrormanager ) to have dynamic .metalink files created with updated mirror readiness info. Certainly something that looks worth looking into.
That would be quite nice, no one else has dynamic .metalinks on a large scale. When I got the F7 ISO, I noticed it would fit in well w/ the download pages which tell which mirrors have which releases. I think it would make things less frustrating for end users trying to get things, and hopefully create less strain on mirrors. Certain metalink clients will download from domestic mirrors first, if country info is in there, which should hopefully be more efficient for everyone.
What can we do to make this happen? Is this the type of thing that's easier for the maintainer of MirrorManager to add, or should we supply a patch?
Here's the current tools people have done, if that helps - http://www.metalinker.org/implementation.html#generate
Metalink Editor - in Python, GUI cURL - they use a short Perl script that makes them based on location. Simba/RoPkg::Metalink - Perl Bouncer - there's a patch for it. Metalink tools - CLI, C++
-- (( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
On Sat, Jun 09, 2007 at 04:22:24AM -0400, Anthony Bryan wrote:
Hello, this is a repost of a message to fedora-devel, as this seems to be the place for it.
From: Jesse Keating jkeating@redhat.com
On Friday 08 June 2007 14:24:47 Anthony Bryan wrote:
I was hoping Fedora could investigate using Metalinks for their ISO downloads. Metalink is an XML format for listing all the ways you can get a file or collection of files (mirrors + their location, rsync, p2p) along with checksums to automatically repair a file in case of error, signatures, language, OS/arch, and other metadata. It's mainly used for large files like ISOs, where errors can be very frustrating.
It's supported by about 20 programs on unix, mac, and win, including aria2 (already in the Fedora repos). It's used by openSUSE, OpenOffice.org, cURL, and many other distributions.
Here's a screenshot of a Metalink download in the DownThemAll Firefox extension (nightly build). What you don't see are all the mirrors and checksums. http://code.downthemall.net/maierman/metaselect4.png
This is something interesting, and I wonder if we could make use of MirrorManager ( https://hosted.fedoraproject.org/projects/mirrormanager ) to have dynamic .metalink files created with updated mirror readiness info. Certainly something that looks worth looking into.
That would be quite nice, no one else has dynamic .metalinks on a large scale. When I got the F7 ISO, I noticed it would fit in well w/ the download pages which tell which mirrors have which releases. I think it would make things less frustrating for end users trying to get things, and hopefully create less strain on mirrors. Certain metalink clients will download from domestic mirrors first, if country info is in there, which should hopefully be more efficient for everyone.
What can we do to make this happen? Is this the type of thing that's easier for the maintainer of MirrorManager to add, or should we supply a patch?
I saw some articles on metalink and spent time time looking at it a few weeks ago. Good stuff.
Patches are certainly welcome. What I picture is a new application 'generate_metalinks' along the lines of the generate_mirrorlists application within mirrormanager that connects to the database, finds what it wants, and generates static .metalink text files. You'd probably only want to metalink the .ISOs, of which there are plenty. No sense making .metalink files for every file in the (presently) 660GB file system.
Then we also need to generate static web pages that include HTML links to all the .metalink files. And link to those from somewhere in the mirrors.fedoraproject.org/ namespace, maybe mirrors.fp.o/metalink/index.html plus all the .metalink files in a dir there.
Then we add it into the update-static-content script that runs at the top of the hour, so they're dynamically generated.
Here's the current tools people have done, if that helps - http://www.metalinker.org/implementation.html#generate
Metalink Editor - in Python, GUI cURL - they use a short Perl script that makes them based on location. Simba/RoPkg::Metalink - Perl Bouncer - there's a patch for it. Metalink tools - CLI, C++
As mm is written in python + TurboGears, it would be great if the metalink creator tool was also in python. :-)
Source to mm is available here: https://hosted.fedoraproject.org/projects/mirrormanager
Thanks, Matt
infrastructure@lists.fedoraproject.org