- gutvol-d - lists.pglaf.org

re: [gutvol-d] Automated readability scores for PG eBooks
by Bowerbird＠aol.com 26 Jun '06

26 Jun '06

greg said: > one value of this is that it does > a good job of identifing children's eBooks > (they tend to be "easy"). checklist said: > bigword density > short word density (-) > wordsPerSentences > syllablesPerWords > profainwordsPerWords > numbersPerWords > mostCommon1000WordsPerWord (-) > commascharsPerWords > wordsPerParagraphs > letterFrequencyDistributionError > adjacentLetterPairsFrequencyDistributionError > uniqueStemmedWordsPerWord; aren't scientists silly? :+) look, greg, if you want a list of children's e-books, or a list of "easy" e-books, or any kind of list of books, just ask the distributed proofreaders people for the list... they'll give you a long list of books, any kind of list you want, and you won't have to do one little bit of fancy-ass statistics... i'm serious, they can give a list with p.g. e-text numbers and meaningful notes, and funny little stories, and _everything_... much more vivid than your boring-ass statistics... :+) -bowerbird

1 0

the end of the line
by Bowerbird＠aol.com 25 Jun '06

25 Jun '06

as i watch all these p.g. e-texts float across my screen, i just can't help but have some thoughts recur to me... in the old days, when -- for some very good reasons -- a p.g. e-text was considered to be an _amalgamation_ of different versions of a book (even when it really was not, a fiction advised by p.g. legal counsel at that early time), that gave a good reason to remove end-line hyphenation and reflow text (without hyphenation) to p.g. margination. after all, hyphenation mostly causes problems in e-books. in the current era, however, where most p.g. e-texts are pegged to a specific version of a book (and where, for the most part, the scans are now retained to cement this direct correspondence), it no longer makes sense to discard the line-breaks, or even the end-line hyphenation, to be frank. yes, end-line hyphenation should be _marked_ in some way, so it can be automatically eliminated, but the _default_action_ should be to retain it. it would defeat the purpose of saving the line-breaks if you didn't also retain end-line hyphenation, because the goal here would be to duplicate the print version. (don't bother arguing that there would never be such a desire; maybe you'd never have any need for it, but _someone_ might. i can think of half-a-dozen such reasons -- want to hear them?) if you want to see what the future of electronic-books looks like, see the "digital reprints" that jose menendez has been producing. > http://www.ibiblio.org/ebooks/Mabie/ > http://www.ibiblio.org/ebooks/Cather/ > http://www.ibiblio.org/ebooks/Einstein/ the deep links to the actual .pdf "digital reprints" are these: > http://www.ibiblio.org/ebooks/Mabie/Books_Culture.pdf > http://www.ibiblio.org/ebooks/Cather/Antonia/Antonia.pdf > http://www.ibiblio.org/ebooks/Einstein/Einstein_Relativity.pdf aside from the unfortunate fact that jose is using the .pdf format (a format which makes it far to difficult to repurpose the content), these "digital reprints" carve out an awesome model for e-books. they replicate the original paper-book to a high degree of fidelity, and do so using a small percentage of the disk-space of the scans. yet because it is an e-book, it gives all the benefits that they give. (at least it _would_, if it wasn't a .pdf. but that part can be fixed.) and the secret of these "digital reprints" is extremely simple, folks; all that jose has done is merely to retain the original line-breaks... so, once again, i recommend and request that you start retaining this valuable information, instead of intentionally tossing it away. (it is very ironic, to me, that distributed proofreaders _retains_ the line-breaks during their proofing -- because it makes that process so much easier -- but then they discard the line-breaks! hey, there might be some end-users out there who need 'em too!) honestly, folks, when i look at your p.g. e-texts, what i see is that they're gonna be thrown on the trashpile one day -- maybe soon. in a world that is awash in scans, and where o.c.r. is a commodity, it'll be trivial to convert those scans to text. so if someone needs to have the ability to duplicate the print version -- i.e., they _need_ to have the line-break information you are routinely discarding -- they'll simply o.c.r. the scans again. they will be required to do that, because your e-texts simply won't do the job that they want done... that's not to say that your p.g. e-texts will be _completely_ worthless. as an independent digitization, they'll go a long way toward helping to move any new o.c.r. effort up to an absurdly high level of accuracy. but since the absurd level of accuracy can be applied to either e-text, and since the new effort will have retained the line-break information, that will be the one that's retained. the p.g. e-text will be thrown away. and it would break my heart to see all your hard work just thrown away. on the other hand, if y'all started retaining that line-break information, then it'd be _your_ version which would be kept (because of its primacy), and the new o.c.r. effort would just be seen as a tool to increase accuracy. if project gutenberg wants to remain as the premiere library in cyberspace, you're going to have to fix this glitch, and do it quickly. mark my words... -bowerbird p.s. at some of you aren't good at reading between the lines, i'll tell you that i intend to mount such a massive o.c.r. effort, so the question about which version, p.g. or not, receives the higher accuracy is a very real one. i don't want to challenge the p.g. library, _unless_ you've made it deficient. i'm trying to help you by giving you this advice before it becomes crucial...

9 10

re: [gutvol-d] the end of the line
by Bowerbird＠aol.com 23 Jun '06

23 Jun '06

i said: > (a format which makes it far to difficult to repurpose the content), haha. "far to difficult". i made a boo-boo. > p.s. at some of you aren't good at reading between the lines "at some of you"...? wow. two mistakes (at least) in one post. good thing it's friday... ;+) -bowerbird

1 0

David's in progress list
by Norm Wolcott 22 Jun '06

22 Jun '06

I have been unable to download david's in progress list. It stops at Abbott and just sits there and never finisihes. Is it posted anywhere else? nwolcott2(a)post.harvard.edu

5 5

re: [gutvol-d] scraping the p.g. default .txt files
by Bowerbird＠aol.com 22 Jun '06

22 Jun '06

i said: > well, i have scraped the p.g. default .txt files -- > http://www.gutenberg.org/files/#####/#####.txt > -- from #10000 up, and surprisingly _quickly_. of course, the idea is to rework these e-texts into z.m.l. format. although i won't be able to get started on that for a few weeks, and it will probably take me about 6 months to finish them all, i did have a chance to do just a wee bit of experimentation... so, to see a review of the transformation of one such p.g. e-text: > http://snowy.arsc.alaska.edu/bowerbird/misc/screen1.html -bowerbird

1 0

chapter-headings linked to the table-of-contents
by Bowerbird＠aol.com 21 Jun '06

21 Jun '06

i see that carlo, one of the smarter p.g. people, has started doing one of the things i suggested some time back -- having each chapter header link to the table of contents. well-done, carlo... > http://www.gutenberg.org/files/18627/18627-h/18627-h.htm#table chalk up another "i told you so". -bowerbird

1 0

scraping the p.g. default .txt files
by Bowerbird＠aol.com 21 Jun '06

21 Jun '06

well, i have scraped the p.g. default .txt files -- http://www.gutenberg.org/files/#####/#####.txt -- from #10000 up, and surprisingly _quickly_. text is indeed compact. even when not zipped. a few notes. circa #18644 is the most recent? really? i thought we were up close to #20000? i take it .aus and .eur are in that count? please relabel human genome files! not really .txt! out of each 1,000 e-texts, about 150 are a.w.o.l. -- different types (e.g., mp3) or something or other, reducing these 8,644 down to some 7,000 or so. plus before i process further, i will toss out the non-english and other pesky variants... let's get it working on the simple ones first. which might take the 7,000 down to 6,000. i'd thought of it initially as a mere pilot-test, but it's looking more like split-half reliability. (i choose 10,000+ only because filenames were generated with a one-line template.) anyway, i chunked those files into folders of 1,000 e-texts each, because that was the size where my old machine starting choking, but o.s.x. seems to handle folders just fine even when the number of files inside is 5,000+... so i might consolidate the folders further, but in the meantime, the results lead to good news. each set of 1,000 e-texts takes roughly 300 megs, so the entire set of 20,000 would be about 6 gigs, meaning they will fit comfortably on today's dvd. and that is without any compression at all, baby. if we figure in compression, and tomorrow's dvd, we're talking an impressive library on a single disc. and a _huge_ library in a case containing 10 discs... -bowerbird

1 0

scoo bee doo bee bee doo
by Bowerbird＠aol.com 21 Jun '06

21 Jun '06

jon said: > Now, doesn't 232332 (base 6) sound much more impressive? yes, it does. especially if you're special enough to know that -- in base 6 lingo -- 3 is articulated as "bee", and 2 is "doo" except when it occurs at the start of a "word" in which case it is pronounced "scoo", meaning this number is vocalized as "scoo bee doo bee bee doo". -bowerbird p.s. personally, i like base 7 -- 112211 -- a lot because it's great to have m.c. palindrome on the ones and twos... p.p.s. there are 10 types of people in this world -- those who understand base 2 and those who don't.

1 0

Re: [gutvol-d] 20000 (decimal) represented in other bases -- Impact on PG
by Joshua Hutchinson 21 Jun '06

21 Jun '06

And below is an example of true geek humor. Us geeks are having a good chuckle. Everyone else is scratching their heads, saying, "What the *bleep* are they talking about!?" Josh PS And the Google nerds are busy searching for the meanings of the esoteric phrases... ;) > ----- Original Message ----- > From: "Marcello Perathoner" <marcello(a)perathoner.de> > To: "Jon Noring" <jon(a)noring.name>, "Project Gutenberg Volunteer Discussion" <gutvol-d(a)lists.pglaf.org> > Subject: Re: [gutvol-d] 20000 (decimal) represented in other bases -- Impact on PG > Date: Wed, 21 Jun 2006 20:15:09 +0200 > > > Jon Noring wrote: > > > Therefore, I recommend to PG that if human nature is important, and > > bigger is better, then PG should report the number of books it has in > > a lower base. > > Wasn't it Donald E. Knuth who celebrated his 1,000,000th birthday? > (base 2) > > > We should count our books in base t where t == 2^(12/18) > > That would make all those computations about our keeping up with Moore's > Law much simpler: If we have to add a new digit each new year, we are on > schedule. > > > I hope the advertising industry won't wisen up to this: everything would > start to cost $10¹ and you'll have to read the fine print to find out > the number base. > > > ¹) base the real price > > > -- > Marcello Perathoner > webmaster(a)gutenberg.org > > _______________________________________________ > gutvol-d mailing list > gutvol-d(a)lists.pglaf.org > http://lists.pglaf.org/listinfo.cgi/gutvol-d >

1 0

!@! Just TWO Books Needed for 20,000!!!
by Michael Hart 21 Jun '06

21 Jun '06

Anyone got anything coming in the next THREE hours??? ;-) Thanks!!! Give the world eBooks in 2006!!! Michael S. Hart Founder Project Gutenberg Blog at http://hart.pglaf.org

3 2