Well, the basic idea of Metalinks is simple. The main server will host an XML files which describes the mirrors and validation data (SHA1 digest). http://sourceforge.net/apps/trac/metalinks/wiki/WhatAreMetalinks The user needs a special download client (or the DownloadThemAll Firefox plugin) which will interpret the XML, choose the mirror and validate the download. Most administrators tend to just write their own script to generate the XML files. Here are some of the other options: - Use a command line tool (we have created one) which can do the hashing for you: http://sourceforge.net/apps/trac/metalinks/ - Use a mirroring system, such as mirrorbrain: http://mirrorbrain.org/ This will test you mirrors, validate your files etc. It is a complete package when it comes to mirroring, but probably harder to deploy. - Post metadata (sha1sum * > SHA1SUMS) and use the dynmirror.net online service to create metalinks. Dynmirror is meant to allow multiple people to create ad-hoc mirrors for popular content (you provide the validation data, anybody can mirror). You can also use the metalink in combination with XSLT to generate a download page. But you would have to have a web-admin who finds something like that cool to do. Hope this information helps. If you ever find that mirroring becomes a pain, then you may want to read this mail again and see if it can solve a problem then ;) Thanks for reading! Bram On Sun, 2010-01-24 at 12:10 -0800, Greg Newby wrote:

On Thu, Jan 21, 2010 at 01:05:59PM +0100, Bram Neijt wrote:

...

Hi all,

I'm a developer on the metalinks project and I would like to see if I could help with project gutenberg file distribution/mirroring by introducing metalinks or adding dynamically generated metalinks using dynmirror.net.

Sorry your message got missed the first time, Bram. We haven't used the -w list much lately....the -p and -d are more active.

...

Metalinks are XML files which describe multiple mirrors and optionally multiple files. They help with distributing the load over the different mirrors and provide a more care-free download of multiple files. It is not a P2P system, although it is easy to include a link to a torrent file or other P2P links.

This would definitely be of interest, though it's not really a pain point for us. Our good friends at iBiblio have plenty of bandwidth, and most of our downloads are not that large.

We would not mind seamlessly directing traffic to geographically appropriate download locations, though. Currently, it's something the human needs to do explicitely.

It would be good to hear more from you about how this might integrate with www.gutenberg.org. -- Greg

...

More information is available at: http://sourceforge.net/apps/trac/metalinks/wiki/WhatAreMetalinks and http://metalinker.org/

I'm going to be on a vacation the upcoming week, but I would like to hear from all of you if I could help with implementing metalinks for the gutenberg project and/or answer any questions you may have.

Greetings, Bram Neijt

PS I first wrote this message to webmaster2010, but that returned with a failure.

I think this is fixed, now. At least for webmaster2010@pglaf.org

Dr. Gregory B. Newby Chief Executive and Director Project Gutenberg Literary Archive Foundation http://gutenberg.org A 501(c)(3) not-for-profit organization with EIN 64-6221541 gbnewby@pglaf.org