June 2006 - gutvol-d - lists.pglaf.org

Automated readability scores for PG eBooks
by Greg Newby 26 Jun '06

26 Jun '06

Feedback/input would be valued. I've been corresopnding with Simon Ronald at RocketReader.com to see about integrating readability scores into the main PG book catalog. Because we don't have a lot of subject cataloging, one value of this is that it does a good job of identifing children's eBooks (they tend to be "easy"). This is also usable for people seeking to develop literacy or provide literacy instruction, by providing a way of reading something "harder" or "easier" as desired. Take a look at the list below (ten hardest, then ten easiest). The first score is overall, followed by a set of scores that made it up. I had provided some earlier feedback on how "hard" books were not necessarily prose, which is part of what Dr. Ronald is responding to. If you have feedback on the results, or my idea for adding these scores as an element of the catalog search results, please chime in! -- Greg ----- Forwarded message from "Dr. Simon Ronald" ----- Subject: Further Readability Results Date: Tue, 20 Jun 2006 03:34:26 +0930 Hello Greg, Here are some further "hardest and easiest" books based on a recent run. The run required 1 hour and 49 minutes to complete. This run classified 15,099 books - being a full scan of the English books. We incorporated a ordered list detection algorithm - some of the books contained (sometimes very noisy) lists of items - we found 162 books in total that were list based. It should be noted that we classified the entire book as list or "not list" based on a threshold -> if the book was a list then each separate line was considered a sentence for the purposes of readability. In time we will incorporate intra book list detection to allow the readability methodology to vary depending on the context within the book. It should also be noted that some of the HTML versions may well contain markup hints such as the use of the <ol> or <ul> HTML tag, we could use these and other tags to improve the quality of sentence chunking. Each entry has a series of 12 percentiles listed after the main readability percentile. These percentiles correspond to the 12 readability attributes in this order. bigword density short word density (-) wordsPerSentences syllablesPerWords profainwordsPerWords numbersPerWords mostCommon1000WordsPerWord (-) commascharsPerWords wordsPerParagraphs letterFrequencyDistributionError adjacentLetterPairsFrequencyDistributionError uniqueStemmedWordsPerWord; 99.914 95 90 95 97 0 79 86 84 79 88 85 90 Note on the Resemblances and Differences in the Structure and the Development of the Brain in Man and Apes (etext2354) 99.907 96 93 90 98 0 71 96 94 49 80 70 96 Original Letters and Biographic Epitomes (etext13203) 99.907 89 86 96 88 95 82 71 94 81 80 67 67 The Great Conspiracy, Volume 7 (etext7139) 99.904 85 87 93 86 0 86 97 75 78 88 78 99 A Biography of Edmund Spenser (etext6937) 99.897 92 90 95 92 88 69 76 90 80 48 60 78 Memoirs of the Court of St. Cloud (Being secret letters from a gentleman at Paris to a nobleman in London) \xe2\x8 0\x94 Volume 1 (etext3892) 99.897 82 92 32 87 88 93 98 87 68 93 83 85 Graf von Loeben and the Legend of Lorelei (etext11066) 99.894 96 93 89 98 80 91 84 97 84 88 36 27 The Modern Regime, Volume 2 (etext2582) 99.887 92 88 73 90 92 82 77 89 45 97 67 76 An Enquiry Concerning the Principles of Taste, and of the Origin of our Ideas of Beauty, etc. (etext13485) 99.887 91 89 88 92 0 64 75 99 96 88 67 96 Giordano Bruno (etext4228) 99.884 99 95 92 99 88 79 84 34 87 66 70 74 Monism as Connecting Religion and Science (etext9199) 99.881 93 91 75 92 94 64 67 77 94 88 67 80 Rise of the Dutch Republic, the \xe2\x80\x94 Volume 22: 1574-76 (etext4824) 99.874 91 92 23 93 80 80 98 95 70 80 91 74 The Principal Navigations, Voyages, Traffiques and Discoveries of the English Nation \xe2\x80\x94 Volume 01 (etext7182) 99.868 97 95 74 99 97 81 91 77 64 27 36 78 Gilbertus Anglicus (etext16155) 99.868 86 95 63 86 0 98 90 89 50 93 89 80 Cessions of Land by Indian Tribes to the United States: Illustrated by Those in the State of Indiana^M (etext17148 ) 99.858 87 87 95 87 80 56 79 93 63 66 83 73 An Essay towards Fixing the True Standards of Wit, Humour, Railery, Satire, and Ridicule (1744) (etext16233) 99.858 91 96 71 90 95 99 99 99 3 2 85 65 Noteworthy Families (Modern Science) (etext17128) 99.858 99 99 5 99 95 92 99 99 68 0 81 87 Roget's Thesaurus of English Words and Phrases (etext10681) 99.854 97 92 89 98 0 83 81 79 95 97 60 57 Eighteenth Brumaire of Louis Bonaparte (etext1346) 99.844 99 99 84 99 80 98 93 14 61 88 83 55 Venereal Diseases in New Zealand (1922) (etext15352) 99.844 96 91 98 96 88 80 74 98 97 66 64 9 Act, Declaration, & Testimony for the Whole of our Covenanted Reformation, as Attained to, and Established in Britain and Ireland; Particularly Betwixt the Years 1638 and 1649, Inclusive (etext13200) 99.844 90 89 96 90 0 79 78 69 87 66 70 97 Dr. Bullivant (etext9249) 99.831 90 88 94 89 80 71 61 96 90 66 30 72 Memoirs of the Court of St. Cloud (Being secret letters from a gentleman at Paris to a nobleman in London) \xe2\x8 0\x94 Volume 7 (etext3898) 99.831 99 99 87 99 99 92 88 27 54 93 94 34 Three Contributions to the Theory of Sex (etext14969) 99.831 95 88 91 92 80 61 67 72 83 93 64 70 Superstition Unveiled (etext15696) 99.831 95 93 45 97 80 89 97 94 43 13 72 80 Aboriginal American Authors (etext9188) 99.824 89 92 88 88 0 90 92 56 54 80 94 84 Transactions of the American Society of Civil Engineers, Vol. LXVIII, Sept. 1910 (etext18012) 99.824 87 99 40 90 0 99 95 94 3 88 92 97 On the Origin of Species (etext8205) 99.798 91 89 93 90 88 69 60 89 86 48 47 74 Memoirs of the Court of St. Cloud (Being secret letters from a gentleman at Paris to a nobleman in London) \xe2\x8 0\x94 Volume 3 (etext3894) 99.798 90 87 85 88 0 81 64 96 84 48 81 94 The Lives of the Twelve Caesars, Volume 11: Titus (etext6396) 99.798 90 86 97 90 0 81 76 90 94 27 76 82 The evolution of English lexicography (etext11694) 99.798 73 81 96 78 98 56 54 95 72 66 89 96 A Modest Proposal (etext1080) 99.798 95 96 41 97 0 95 96 79 54 98 78 78 Webster's March 7th Speech/Secession (etext1663) 99.798 91 92 47 92 92 50 87 87 87 66 64 89 Rise of the Dutch Republic, the \xe2\x80\x94 Volume 01: Introduction I (etext4801) 99.798 88 86 77 88 95 50 73 73 97 80 72 83 Rise of the Dutch Republic, the \xe2\x80\x94 Volume 26: 1577, part III (etext4828) 99.785 95 85 98 90 88 75 71 92 94 66 56 32 The Auchensaugh Renovation of the National Covenant and (etext12381) 99.785 93 92 81 92 88 92 85 98 84 80 24 16 The Modern Regime, Volume 1 (etext2581) 99.785 70 78 92 75 92 81 79 94 87 48 52 82 The Mayflower and Her Log; July 15, 1620-May 6, 1621 \xe2\x80\x94 Volume 5 (etext4105) Easiest 4.176 2 1 11 2 0 0 1 36 9 48 78 11 The Song of the Blood-Red Flower (etext12935) 4.176 0 0 11 0 0 0 15 48 8 2 94 6 Six Little Bunkers at Grandma Bell's (etext14623) 4.176 14 8 27 12 0 0 9 15 10 27 41 7 Melbourne House, Volume 2 (etext12964) 4.176 7 3 6 4 0 0 5 3 9 66 86 23 The Romantic (etext13292) 4.176 17 5 14 10 0 0 1 12 48 48 6 23 The Girl from Montana (etext15274) 4.176 7 6 13 7 0 0 11 27 22 27 56 9 Jess of the Rebel Trail (etext15382) 4.176 5 3 8 3 0 0 0 2 45 66 60 25 Stories of American Life and Adventure (etext15597) 4.176 3 8 13 6 0 0 12 3 72 1 78 13 Kazan (etext10084) 4.176 11 5 8 8 0 0 12 1 9 48 94 11 The Second Honeymoon (etext17446) 4.176 13 9 10 12 0 0 13 8 3 48 36 23 The Circus Boys on the Plains : or, the Young Advance Agents Ahead of the Show (etext2478) 4.176 19 11 11 15 0 0 10 26 42 27 9 1 The Captives (etext3601) 4.176 0 1 12 1 0 0 2 3 49 27 90 29 Old Granny Fox (etext4980) 4.176 0 0 6 0 0 0 2 0 19 27 89 54 Sleepy-Time Tales: the Tale of Fatty Coon (etext5701) 4.176 0 0 12 1 0 0 10 6 31 2 93 40 The Adventures of Johnny Chuck (etext5844) 4.176 2 3 11 3 0 0 7 1 12 5 85 54 Tale of Brownie Beaver (etext6754) 4.176 12 7 19 8 0 0 9 7 55 5 52 13 The City of Fire (etext7008) 4.176 23 9 10 15 0 0 13 16 18 13 3 29 The Man with Two Left Feet^M (etext7471) 4.176 4 5 13 5 0 0 8 15 40 5 80 21 Way of the Lawless (etext9903) 3.931 11 7 11 8 0 0 14 7 36 2 85 9 The Hunted Woman (etext11328) 3.931 4 1 14 2 0 0 1 34 44 27 64 3 Mary Marie (etext11143) 3.931 12 8 22 11 0 0 10 20 15 13 41 13 Contrary Mary (etext17938) 3.931 0 0 3 0 0 0 0 1 0 98 97 29 The New McGuffey First Reader (etext1489) 3.931 4 7 18 6 0 0 7 33 21 13 56 9 Martin Pippin in the Apple Orchard (etext2032) 3.931 8 9 8 8 0 0 1 12 48 13 64 23 Twenty-Two Goblins (etext2290) 3.931 9 8 9 8 0 0 7 5 43 27 72 13 God's Country\xe2\x80\x94And the Woman (etext4585) 3.931 12 11 11 13 0 0 6 12 60 27 13 14 The Valley of Silent Men (etext4707) 3.931 5 2 23 3 0 0 6 16 10 13 80 23 The Boy Scout Camera Club, or, the Confession of a Photograph (etext7356) 3.931 8 9 15 10 0 0 17 4 6 5 81 21 Bob Cook and the German Spy (etext9899) 3.676 16 6 8 10 0 0 25 3 3 13 76 13 The Three Sisters (etext11876) 3.676 7 2 10 3 0 0 3 17 43 27 78 11 His Second Wife (etext17259) 3.676 6 5 17 6 0 0 14 33 24 5 60 3 Michael O'Halloran (etext9489) 3.384 12 7 6 10 0 0 28 0 20 27 13 32 The Sheriff's Son (etext17043) 3.384 1 0 17 1 0 0 9 36 11 5 88 9 The Bobbsey Twins in the Great West (etext5952) 3.384 6 3 7 4 0 0 3 24 20 13 52 34 Pan (etext7214) 3.384 0 0 5 0 0 0 2 1 20 5 93 59 Five Little Friends (etext7801) 3.384 0 1 7 1 0 0 13 0 11 1 88 53 The Tale of Sandy Chipmunk (etext9462) 3.109 0 0 6 0 0 0 0 19 0 2 93 50 Boy Blue and His Friends (etext16046) 3.109 8 3 10 4 0 0 1 30 19 27 75 6 Wanderers (etext7762) 2.874 1 2 56 2 0 0 0 33 32 2 9 5 Twilight Land (etext1751) 2.874 0 0 13 0 0 0 11 38 9 1 91 6 The Curlytops on Star Island (etext5989) 2.666 0 0 10 0 0 0 6 44 9 13 76 7 The Bobbsey Twins at Home (etext18420) 2.666 0 0 34 0 0 0 0 1 43 48 64 5 The King of Ireland's Son (etext3495) 2.460 7 11 12 11 0 0 8 3 70 5 19 16 Baree, Son of Kazan (etext4748) 2.460 10 9 11 10 0 0 2 15 9 13 52 21 Samuel the Seeker (etext5961) 2.255 6 1 2 2 0 0 28 1 7 80 30 14 Plays (etext10623) 2.255 12 13 14 14 0 0 11 4 23 5 30 14 King of the Khyber Rifles (etext6066) 2.255 3 3 19 3 0 0 6 11 13 13 64 23 Riders of the Silences (etext9867) 1.917 4 2 7 3 0 0 11 4 7 48 89 6 Anne Severn and the Fieldings (etext10817) 1.785 0 0 15 0 0 0 0 9 13 5 72 36 Fifty Famous Stories Retold (etext18442) 1.507 8 4 17 6 0 0 21 0 24 13 19 16 The Light in the Clearing (etext14150) 1.507 9 7 21 8 0 0 3 2 30 27 13 14 Voyages of Dr. Dolittle (etext1154) 1.507 0 0 2 0 0 0 43 22 3 27 30 2 Six Plays (etext5618) 1.391 4 6 18 6 0 0 9 0 24 5 67 6 The Secret Garden (etext17396) 1.391 12 9 8 10 0 0 16 7 21 13 2 19 Black Jack (etext9925) 1.080 10 4 14 6 0 0 8 9 14 5 24 19 The Gay Cockade (etext16433) 0.742 4 4 14 4 0 0 5 3 55 2 6 16 Isobel : a Romance of the Northern Trail (etext6715) 0.440 0 0 5 0 0 0 2 2 0 1 97 16 Bunny Rabbit's Diary (etext16982) 0.281 6 3 6 4 0 0 22 2 8 5 19 6 Mary Olivier: a Life (etext9366) Cheers, Dr. Simon Ronald CEO The Leader in High Performance Reading Level 2, 25 Gresham Street, Adelaide, SA, Australia, 5000 GPO Box 944, Adelaide SA 5001 Ph. +61 8 8410 2771 Fax. +61 8 8125 6679 1133 Broadway, Suite 706 New York, NY 10010 Ph: (646) 736 7673 (New York) Ph: (415) 992 5412 (California) Fax: (877) 731 4410 (toll free) ____________________________________ ----- End forwarded message ----- ----- End forwarded message -----

3 3

re: [gutvol-d] ebooks libre et gratuits
by Bowerbird＠aol.com 26 Jun '06

26 Jun '06

unlimited distribution rocks big time... major fucking concept... -bowerbird

1 0

re: [gutvol-d] Automated readability scores for PG eBooks
by Bowerbird＠aol.com 26 Jun '06

26 Jun '06

greg said: > one value of this is that it does > a good job of identifing children's eBooks > (they tend to be "easy"). checklist said: > bigword density > short word density (-) > wordsPerSentences > syllablesPerWords > profainwordsPerWords > numbersPerWords > mostCommon1000WordsPerWord (-) > commascharsPerWords > wordsPerParagraphs > letterFrequencyDistributionError > adjacentLetterPairsFrequencyDistributionError > uniqueStemmedWordsPerWord; aren't scientists silly? :+) look, greg, if you want a list of children's e-books, or a list of "easy" e-books, or any kind of list of books, just ask the distributed proofreaders people for the list... they'll give you a long list of books, any kind of list you want, and you won't have to do one little bit of fancy-ass statistics... i'm serious, they can give a list with p.g. e-text numbers and meaningful notes, and funny little stories, and _everything_... much more vivid than your boring-ass statistics... :+) -bowerbird

1 0

the end of the line
by Bowerbird＠aol.com 25 Jun '06

25 Jun '06

as i watch all these p.g. e-texts float across my screen, i just can't help but have some thoughts recur to me... in the old days, when -- for some very good reasons -- a p.g. e-text was considered to be an _amalgamation_ of different versions of a book (even when it really was not, a fiction advised by p.g. legal counsel at that early time), that gave a good reason to remove end-line hyphenation and reflow text (without hyphenation) to p.g. margination. after all, hyphenation mostly causes problems in e-books. in the current era, however, where most p.g. e-texts are pegged to a specific version of a book (and where, for the most part, the scans are now retained to cement this direct correspondence), it no longer makes sense to discard the line-breaks, or even the end-line hyphenation, to be frank. yes, end-line hyphenation should be _marked_ in some way, so it can be automatically eliminated, but the _default_action_ should be to retain it. it would defeat the purpose of saving the line-breaks if you didn't also retain end-line hyphenation, because the goal here would be to duplicate the print version. (don't bother arguing that there would never be such a desire; maybe you'd never have any need for it, but _someone_ might. i can think of half-a-dozen such reasons -- want to hear them?) if you want to see what the future of electronic-books looks like, see the "digital reprints" that jose menendez has been producing. > http://www.ibiblio.org/ebooks/Mabie/ > http://www.ibiblio.org/ebooks/Cather/ > http://www.ibiblio.org/ebooks/Einstein/ the deep links to the actual .pdf "digital reprints" are these: > http://www.ibiblio.org/ebooks/Mabie/Books_Culture.pdf > http://www.ibiblio.org/ebooks/Cather/Antonia/Antonia.pdf > http://www.ibiblio.org/ebooks/Einstein/Einstein_Relativity.pdf aside from the unfortunate fact that jose is using the .pdf format (a format which makes it far to difficult to repurpose the content), these "digital reprints" carve out an awesome model for e-books. they replicate the original paper-book to a high degree of fidelity, and do so using a small percentage of the disk-space of the scans. yet because it is an e-book, it gives all the benefits that they give. (at least it _would_, if it wasn't a .pdf. but that part can be fixed.) and the secret of these "digital reprints" is extremely simple, folks; all that jose has done is merely to retain the original line-breaks... so, once again, i recommend and request that you start retaining this valuable information, instead of intentionally tossing it away. (it is very ironic, to me, that distributed proofreaders _retains_ the line-breaks during their proofing -- because it makes that process so much easier -- but then they discard the line-breaks! hey, there might be some end-users out there who need 'em too!) honestly, folks, when i look at your p.g. e-texts, what i see is that they're gonna be thrown on the trashpile one day -- maybe soon. in a world that is awash in scans, and where o.c.r. is a commodity, it'll be trivial to convert those scans to text. so if someone needs to have the ability to duplicate the print version -- i.e., they _need_ to have the line-break information you are routinely discarding -- they'll simply o.c.r. the scans again. they will be required to do that, because your e-texts simply won't do the job that they want done... that's not to say that your p.g. e-texts will be _completely_ worthless. as an independent digitization, they'll go a long way toward helping to move any new o.c.r. effort up to an absurdly high level of accuracy. but since the absurd level of accuracy can be applied to either e-text, and since the new effort will have retained the line-break information, that will be the one that's retained. the p.g. e-text will be thrown away. and it would break my heart to see all your hard work just thrown away. on the other hand, if y'all started retaining that line-break information, then it'd be _your_ version which would be kept (because of its primacy), and the new o.c.r. effort would just be seen as a tool to increase accuracy. if project gutenberg wants to remain as the premiere library in cyberspace, you're going to have to fix this glitch, and do it quickly. mark my words... -bowerbird p.s. at some of you aren't good at reading between the lines, i'll tell you that i intend to mount such a massive o.c.r. effort, so the question about which version, p.g. or not, receives the higher accuracy is a very real one. i don't want to challenge the p.g. library, _unless_ you've made it deficient. i'm trying to help you by giving you this advice before it becomes crucial...

9 10

re: [gutvol-d] the end of the line
by Bowerbird＠aol.com 23 Jun '06

23 Jun '06

i said: > (a format which makes it far to difficult to repurpose the content), haha. "far to difficult". i made a boo-boo. > p.s. at some of you aren't good at reading between the lines "at some of you"...? wow. two mistakes (at least) in one post. good thing it's friday... ;+) -bowerbird

1 0

David's in progress list
by Norm Wolcott 22 Jun '06

22 Jun '06

I have been unable to download david's in progress list. It stops at Abbott and just sits there and never finisihes. Is it posted anywhere else? nwolcott2(a)post.harvard.edu

5 5

re: [gutvol-d] scraping the p.g. default .txt files
by Bowerbird＠aol.com 22 Jun '06

22 Jun '06

i said: > well, i have scraped the p.g. default .txt files -- > http://www.gutenberg.org/files/#####/#####.txt > -- from #10000 up, and surprisingly _quickly_. of course, the idea is to rework these e-texts into z.m.l. format. although i won't be able to get started on that for a few weeks, and it will probably take me about 6 months to finish them all, i did have a chance to do just a wee bit of experimentation... so, to see a review of the transformation of one such p.g. e-text: > http://snowy.arsc.alaska.edu/bowerbird/misc/screen1.html -bowerbird

1 0

chapter-headings linked to the table-of-contents
by Bowerbird＠aol.com 21 Jun '06

21 Jun '06

i see that carlo, one of the smarter p.g. people, has started doing one of the things i suggested some time back -- having each chapter header link to the table of contents. well-done, carlo... > http://www.gutenberg.org/files/18627/18627-h/18627-h.htm#table chalk up another "i told you so". -bowerbird

1 0

scraping the p.g. default .txt files
by Bowerbird＠aol.com 21 Jun '06

21 Jun '06

well, i have scraped the p.g. default .txt files -- http://www.gutenberg.org/files/#####/#####.txt -- from #10000 up, and surprisingly _quickly_. text is indeed compact. even when not zipped. a few notes. circa #18644 is the most recent? really? i thought we were up close to #20000? i take it .aus and .eur are in that count? please relabel human genome files! not really .txt! out of each 1,000 e-texts, about 150 are a.w.o.l. -- different types (e.g., mp3) or something or other, reducing these 8,644 down to some 7,000 or so. plus before i process further, i will toss out the non-english and other pesky variants... let's get it working on the simple ones first. which might take the 7,000 down to 6,000. i'd thought of it initially as a mere pilot-test, but it's looking more like split-half reliability. (i choose 10,000+ only because filenames were generated with a one-line template.) anyway, i chunked those files into folders of 1,000 e-texts each, because that was the size where my old machine starting choking, but o.s.x. seems to handle folders just fine even when the number of files inside is 5,000+... so i might consolidate the folders further, but in the meantime, the results lead to good news. each set of 1,000 e-texts takes roughly 300 megs, so the entire set of 20,000 would be about 6 gigs, meaning they will fit comfortably on today's dvd. and that is without any compression at all, baby. if we figure in compression, and tomorrow's dvd, we're talking an impressive library on a single disc. and a _huge_ library in a case containing 10 discs... -bowerbird

1 0

scoo bee doo bee bee doo
by Bowerbird＠aol.com 21 Jun '06

21 Jun '06

jon said: > Now, doesn't 232332 (base 6) sound much more impressive? yes, it does. especially if you're special enough to know that -- in base 6 lingo -- 3 is articulated as "bee", and 2 is "doo" except when it occurs at the start of a "word" in which case it is pronounced "scoo", meaning this number is vocalized as "scoo bee doo bee bee doo". -bowerbird p.s. personally, i like base 7 -- 112211 -- a lot because it's great to have m.c. palindrome on the ones and twos... p.p.s. there are 10 types of people in this world -- those who understand base 2 and those who don't.

1 0