[gutvol-w] Re: Distributing metalinks for the Gutenberg project

bram bneijt at gmail.com
Thu Feb 4 09:47:36 PST 2010


Well, the basic idea of Metalinks is simple. The main server will host
an XML files which describes the mirrors and validation data (SHA1
digest).
http://sourceforge.net/apps/trac/metalinks/wiki/WhatAreMetalinks

The user needs a special download client (or the DownloadThemAll Firefox
plugin) which will interpret the XML, choose the mirror and validate the
download.

Most administrators tend to just write their own script to generate the
XML files. Here are some of the other options:

- Use a command line tool (we have created one) which can do the hashing
for you: http://sourceforge.net/apps/trac/metalinks/

- Use a mirroring system, such as mirrorbrain: http://mirrorbrain.org/
  This will test you mirrors, validate your files etc. It is a complete
package when it comes to mirroring, but probably harder to deploy.

- Post metadata (sha1sum * > SHA1SUMS) and use the dynmirror.net online
service to create metalinks. Dynmirror is meant to allow multiple people
to create ad-hoc mirrors for popular content (you provide the validation
data, anybody can mirror).

You can also use the metalink in combination with XSLT to generate a
download page. But you would have to have a web-admin who finds
something like that cool to do.

Hope this information helps. If you ever find that mirroring becomes a
pain, then you may want to read this mail again and see if it can solve
a problem then ;)

Thanks for reading!

Bram

On Sun, 2010-01-24 at 12:10 -0800, Greg Newby wrote:
> On Thu, Jan 21, 2010 at 01:05:59PM +0100, Bram Neijt wrote:
> > Hi all,
> > 
> > I'm a developer on the metalinks project and I would like to see if I
> > could help with project gutenberg file distribution/mirroring by
> > introducing metalinks or adding dynamically generated metalinks using
> > dynmirror.net.
> 
> Sorry your message got missed the first time, Bram.  We haven't
> used the -w list much lately....the -p and -d are more active.
> 
> > Metalinks are XML files which describe multiple mirrors and optionally
> > multiple files. They help with distributing the load over the different
> > mirrors and provide a more care-free download of multiple files. It is
> > not a P2P system, although it is easy to include a link to a torrent
> > file or other P2P links.
> 
> This would definitely be of interest, though it's not really
> a pain point for us.  Our good friends at iBiblio have plenty
> of bandwidth, and most of our downloads are not that large.
> 
> We would not mind seamlessly directing traffic to geographically 
> appropriate download locations, though.  Currently, it's
> something the human needs to do explicitely.  
> 
> It would be good to hear more from you about how this might
> integrate with www.gutenberg.org.
>   -- Greg
> 
> > More information is available at:
> > http://sourceforge.net/apps/trac/metalinks/wiki/WhatAreMetalinks
> > and
> > http://metalinker.org/
> > 
> > I'm going to be on a vacation the upcoming week, but I would like to
> > hear from all of you if I could help with implementing metalinks for the
> > gutenberg project and/or answer any questions you may have.
> > 
> > 
> > Greetings,
> >   Bram Neijt
> > 
> > PS I first wrote this message to webmaster2010, but that returned with
> > a failure.
> 
> I think this is fixed, now.  At least for webmaster2010 at pglaf.org
> 
> Dr. Gregory B. Newby
> Chief Executive and Director
> Project Gutenberg Literary Archive Foundation http://gutenberg.org
> A 501(c)(3) not-for-profit organization with EIN 64-6221541
> gbnewby at pglaf.org
> 





More information about the gutvol-w mailing list