- gutvol-d - lists.pglaf.org

How does the gutenberg search actually work?
by Robert Cicconetti 05 Mar '13

05 Mar '13

How does one search for an exact title in the PG archive? I went to look for a book I worked on ("Brother Against Brother", 40541) and had to dig through 3 pages of fairy books to find it despite specifying the exact title. Quotes don't seem to work. Searching for the second title word "against" doesn't return any results at all... I know I can use google but I'd rather use the tools actually hooked into the PG catalog... Thanks! -Bob

3 2

and this is what we have devolved to
by bowerbird 24 Feb '13

24 Feb '13

and this is what we have devolved to. "if you want something from project gutenberg, you have to ask marcello." it's a damn shame. -bowerbird

2 1

Programmatically build a catalog database?
by Petite Abeille 20 Feb '13

20 Feb '13

Hello, (Hopefully this is the proper mailing list for such topic. Let me know otherwise.) I would like to build a local database of the Gutenberg catalog. The 'Gutenberg Feeds' page [1] lists the following resource to help achieve that programmatically: (1) All books in one huge file, in " superseded DCMI recommendation" format (2) A separate file for each book, in "current DCMI recommendation" format (3) A RSS Feed, in rss version="0.91" format So far: (1) sports all the Gutenberg assets, and is handy for the initial database build. But this looks a bit overkill for a day to day synchronization. (2) seems more appropriate than (1) for daily updates, but sports a different format: "current DCMI recommendation" vs. " superseded DCMI recommendation" (3) is a bit a blast from the past, but at least provides a list of new resources daily. Sadly there is no explicit link to (2), so one has to infer it from the <link> information. Questions: - Is there a version of (1) in the same format as (2)? Assuming the "current DCMI recommendation" is the canonical representation. That would save one from dealing with two different formats, or hacking (1) to get all the references to (2) and then hammer PG to get the individual files in format (2). - Why are (1) and (2) in different formats? - Is there an alternative feed that lists the rdf resource explicitly? An Atom feed perhaps? Apologies if these are FAQs, but I couldn't locate an unambiguous archive of this mailing list. Is GNAME a good proxy for the list postings? http://dir.gmane.org/gmane.culture.literature.e-books.gutenberg.volunteers Alternatively, is there a more straightforward way to build a local database of PG's assets? Perhaps I'm missing something :) Thanks in advance for any pointers. Cheers, PA. [1] http://www.gutenberg.org/wiki/Gutenberg:Feeds

3 14

jumpy mouse, crappy software
by bowerbird 15 Feb '13

15 Feb '13

my apologies for the double-post. been having trouble with a jumpy mouse. but why would an e-mail program send out two exact same e-mails within a few seconds? and why would listserve software go ahead and post two exact same messages _ever_, let alone within a few seconds of each other? my gawd we live in an age of crappy software. i mean, i can understand hardware going bad; physical things wear out; it's the way of the world. but crappy software? that's inexcusable, and on us. -bowerbird

1 0

harts and flowers
by bowerbird 15 Feb '13

15 Feb '13

greg said: > what's cors? sorry, i just assumed you would know. also, have you heard of google? or wikipedia? top hit at google: > http://en.wikipedia.org/wiki/Cross-origin_resource_sharing basically, this python program retrieves the e-text just fine: > #!/usr/bin/python > import urllib > print 'Content-type: text/html\n' > f=urllib.urlopen("http://snowy.arsc.alaska.edu/gutenberg/3/3/3/3333/3333- 8.txt") > s=f.read() > f.close() > print s but on the other hand, this (defanged) javascript will _not_ get the file: > !doctype html> > html> > head> > meta charset=utf-8 /> > title>CORS example > /title> > script type="text/javascript"> > > var client = new XMLHttpRequest(); > > client.onreadystatechange = function () { > if (this.readyState == 4 && this.status == 200) { > alert('' + this.responseText.substring(0,30)); > } > }; > > client.open('get', 'http://snowy.arsc.alaska.edu/gutenberg/3/3/3/3333/3333-8.txt'); > > client.send(); > /script> > /head> > body> > p id="hello">cors_snowy > /p> > /body> > /html> the error-log reports: > XMLHttpRequest cannot load > http://snowy.arsc.alaska.edu/gutenberg/3/3/3/3333/3333-8.txt. > Origin http://mywebsitenamehere.com is not allowed by Access-Control-Allow-Origin. this javascript _would_ work if it was being served from snowy, because then the javascript would be requesting a resource from the domain from which it came, which is always allowed. but if we serve it from another domain, the policy against cors (cross-domain origin resource sharing) disallows the request. however, you can have snowy _allow_ such cors requests. it has something to do with the headers that you send out. if you authorize these cors requests, then javascript will be able to get content from snowy, in the same way python can, from _any_ domain from which it might be served. for myself, i've worked around this problem. (i simply have javascript instruct my domain to have python go get the file, which it can do, and then i have python store it on my domain, and then -- since it's now on my domain -- javascript can get it.) but javascript coders shouldn't have to do this two-step shuffle. and they won't, if you have snowy authorize cors requests... -bowerbird

1 0

harts and flowers
by bowerbird 15 Feb '13

15 Feb '13

greg said: > what's cors? sorry, i just assumed you would know. also, have you heard of google? or wikipedia? top hit at google: > http://en.wikipedia.org/wiki/Cross-origin_resource_sharing basically, this python program retrieves the e-text just fine: > #!/usr/bin/python > import urllib > print 'Content-type: text/html\n' > f=urllib.urlopen("http://snowy.arsc.alaska.edu/gutenberg/3/3/3/3333/3333- 8.txt") > s=f.read() > f.close() > print s but on the other hand, this (defanged) javascript will _not_ get the file: > !doctype html> > html> > head> > meta charset=utf-8 /> > title>CORS example > /title> > script type="text/javascript"> > > var client = new XMLHttpRequest(); > > client.onreadystatechange = function () { > if (this.readyState == 4 && this.status == 200) { > alert('' + this.responseText.substring(0,30)); > } > }; > > client.open('get', 'http://snowy.arsc.alaska.edu/gutenberg/3/3/3/3333/3333-8.txt'); > > client.send(); > /script> > /head> > body> > p id="hello">cors_snowy > /p> > /body> > /html> the error-log reports: > XMLHttpRequest cannot load > http://snowy.arsc.alaska.edu/gutenberg/3/3/3/3333/3333-8.txt. > Origin http://mywebsitenamehere.com is not allowed by Access-Control-Allow-Origin. this javascript _would_ work if it was being served from snowy, because then the javascript would be requesting a resource from the domain from which it came, which is always allowed. but if we serve it from another domain, the policy against cors (cross-domain origin resource sharing) disallows the request. however, you can have snowy _allow_ such cors requests. it has something to do with the headers that you send out. if you authorize these cors requests, then javascript will be able to get content from snowy, in the same way python can, from _any_ domain from which it might be served. for myself, i've worked around this problem. (i simply have javascript instruct my domain to have python go get the file, which it can do, and then i have python store it on my domain, and then -- since it's now on my domain -- javascript can get it.) but javascript coders shouldn't have to do this two-step shuffle. and they won't, if you have snowy authorize cors requests... -bowerbird

1 0

i hart you
by bowerbird 14 Feb '13

14 Feb '13

james, yes, you have discovered that marcello cares more about fighting the indiscriminate downloading robots than taking care of the real humans... but something tells me angel michael is entertained by the notion that there's "indiscriminate scraping" of p.g. e-texts. but i suppose that, as the lead dog -- (a little sledder talk for newby) -- gutenberg.org gets lots of headwind. so... just choose one of the mirrors! i'd suggest that greg can point you to a certain mirror that's more in line with the assured-access that don remarked should be associated with an endeavor like this one founded by michael hart, an attitude i'd guess rubbed off on greg. because i'll personally attest that urllib is working just fine to grab files from it. but, if i might suggest an improvement, greg, could you authorize c.o.r.s. for it? (at least it appears to me it ain't working.) i'd need nothing more than "get" access, which i think poses absolutely no danger. but also, greg, just to avoid cors hassles, maybe i should re-request one of those accounts you so often promise to people. (but feel free to embargo any javascript i'd serve until you could verify its safety.) i'm about to (finally) make some big moves, and although i no longer feel any need to include project gutenberg, my hart is not so cold that i feel a need to exclude it either. so if you want to come along for the ride, just crawl in the back seat. and buckle up. -bowerbird

2 1

Re: [gutvol-d] [Sugar-devel] Problems downloading PG titles without a web browser
by James Simmons 14 Feb '13

14 Feb '13

Daniel, Thanks for your tips. I was able to add a user agent string like this: OLPCReader/1.0; +mailto:nicestep@gmail.com and that fixed the problem. The code has been committed to Git and I'll release a new .xo file shortly. I don't know that I'll submit a path for network.py as the user agent string is not a problem for most uses. James Simmons On Sun, Feb 10, 2013 at 7:05 PM, Daniel Narvaez <dwnarvaez(a)gmail.com> wrote: > Actually it seems like you might be able to do it with urllib too. > > http://docs.python.org/2/library/urllib.html > > Search for urlopener, there is an example. > > On Monday, 11 February 2013, Daniel Narvaez wrote: > >> >> http://git.sugarlabs.org/sugar-toolkit-gtk3/sugar-toolkit-gtk3/blobs/master… >> >> That code *looks* trivial to port to urllib2. You could do that, submit >> the patch and in the meantime just copy and paste the ported code in your >> activity (it doesn't seem to have dependencies on other toolkit stuff). >> Well, unless it's a gtk2 activity, in that case you can skip the "submit >> the patch" step :) >> >> On Monday, 11 February 2013, James Simmons wrote: >> >>> Marcello (and everyone else who replied): >>> >>> It looks like the mirrors have the same policy. I'm currently using >>> this to download from the URL: >>> >>> >>> http://doc.sugarlabs.org/epydocs/sugar.network.GlibURLDownloader-class.html >>> >>> I'm not seeing a way to put a user agent in using this code. It looks >>> like urllib2 supports it. I used the other because I was already using it >>> for collaboration and it gives me a way to do a progress report on the >>> download. I appreciate everyone's detective work and suggestions. >>> >>> James Simmons >>> >>> >>> On Sun, Feb 10, 2013 at 1:22 PM, Marcello Perathoner < >>> marcello(a)perathoner.de> wrote: >>> >>>> On 02/10/2013 06:46 PM, James Simmons wrote: >>>> >>>>> I had created an Activity for the One Laptop Per Child project which >>>>> downloads and reads the Plain Text version of PG titles. I use the >>>>> offline >>>>> catalogue and some Python code to list available titles in the >>>>> catalogue >>>>> that match a search string. For instance the child might enter >>>>> "Twain" and >>>>> get a list of all the books by and about Twain. Then the child can >>>>> download the book using the same program. >>>>> >>>>> This has been working well for years but lately it has stopped working. >>>>> When I try to debug it I see it trying to download this URL for >>>>> instance: >>>>> >>>>> http://www.gutenberg.org/dirs/**1/1/119/119.zip<http://www.gutenberg.org/dirs/1/1/119/119.zip> >>>>> >>>> >>>> >>>> Supply a user-agent that clearly defines your app and provides a way to >>>> contact you eg. >>>> >>>> OLPCReader/1.0; +http://www.olpc-reader.org/**app-info.html<http://www.olpc-reader.org/app-info.html> >>>> >>>> OLPCReader/1.0; +mailto:me@example.com >>>> >>>> The standard Python-urllib user agent will not do! >>>> >>>> >>>> or >>>> >>>> >>>> use a mirror site. See: www.gutenberg.org/MIRRORS.ALL >>>> >>>> >>>> >>>> >>>> >>>> Regards >>>> >>>> -- >>>> Marcello Perathoner >>>> webmaster(a)gutenberg.org >>>> >>> >>> >> >> -- >> Daniel Narvaez >> >> > > -- > Daniel Narvaez > >

1 0

Proposal for augmented structure for PG
by Jon Hurst 12 Feb '13

12 Feb '13

I would like to propose an augmented structure for PG. The aim of this structure is to realise Greg's stated desire to include ebooks that are contributed outside of the normal WWer workflow. The basic premise is that the PG library as it stands becomes just one of multiple contributor libraries hosted by PG, with the gutenberg.org interface rejigged to catalogue all of these libraries. In terms of customer interface, the customer gets a searchable catalogue of works. When they access a work, they are presented with a list of which contributor libraries contain expressions of that work, each with some blurb describing the properties of that expression. Thus, for example, the "Huckleberry Finn" page might list two expressions in the WWer library, one in the DP library, one in the Adcock library, an RST derived one in the Marcello library and a ZML derived one in the Bowerbird library. Library owners would be responsible for maintaining their own libraries. Anything in any of the libraries is fair game for other contributors to build on. Contributors may choose to pool resources and maintain common root versions/ master formats, e.g RST advocates might maintain a set of RST files in Mercurial, possibly even grouping together to build a common RST derived library. Customers would be able to list all the books in a particular library, so if they thought the Bowerbird "Huckleberry Finn" was the best formatted ebook they have ever seen they would be encouraged to see what other Bowerbird ebooks there were. If they thought it was awful, they would avoid Bowerbird works in the future. You could even allow customers to blacklist certain libraries from their future results. Library owners would also get the ability to recommend expressions in other libraries to anyone who is visiting their library, creating a useful reputation chain. The WWers should not be upset as nothing is being erased. They have the biggest library, and they are already entirely responsible for its maintenance. It is already maintained to strict standards, and if those standards stack up, it should be the go to library. DP should be pleased. They would get to do CSS3 and make their own epubs if they want to, and they would get to maintain their own library, something that has been talked about wistfully on the DP forums for a _long_ time. Individual contributors should be pleased. They can produce work to specifications driven by their own beliefs in the formats that they believe are superior, and if they are right about the quality of their work being superior, they will garner a following.

5 11

Problems downloading PG titles without a web browser
by James Simmons 12 Feb '13

12 Feb '13

I had created an Activity for the One Laptop Per Child project which downloads and reads the Plain Text version of PG titles. I use the offline catalogue and some Python code to list available titles in the catalogue that match a search string. For instance the child might enter "Twain" and get a list of all the books by and about Twain. Then the child can download the book using the same program. This has been working well for years but lately it has stopped working. When I try to debug it I see it trying to download this URL for instance: http://www.gutenberg.org/dirs/1/1/119/119.zip Put this URL in any web browser and it will download "A Tramp Abroad" by Mark Twain. However, try and download the same URL using my Python code and you get this: Forbidden You don't have permission to access /dirs/1/1/119/119.zip on this server. ------------------------------ Apache Server at www.gutenberg.org Port 80 My Activity is a much more convenient way of downloading books than a web browser. For one thing, it gives the downloaded file a more intelligent name than 119.zip. I would like for it to work again as it did before. Suggestions? James Simmons

6 10