April 2010 - gutvol-d - lists.pglaf.org

freeliterature.org
by Marc D'Hooghe 15 Apr '10

15 Apr '10

Hi there, I started a couple of weeks ago with a new site in support of PG. http://www.freeliterature.org. Two goals: two spread information about free e-books and literature on the web (extensive link list) - and the possibility to help producing e-text by proofreading. You can download the scans of a book of your choice, and the text to proof is sent on demand. Enjoy. Marc.

1 0

slightly off topic, first post, scanning
by Sparr 15 Apr '10

15 Apr '10

This is my first post to the list. I come here for advice not directly related to PG. I have a number of physical books that have had their spines cut off and I need to scan them so they can be OCR'd and stored digitally. I am encountering two problems. All of the bulk page-feed scanners I have tried have one or both of the following issues: Duplex scanning happens simultaneously, so extreme contrast changes (such as borders) from one side of the page show through (about 5%) to the other side. Remnants of glue left on the edge of a page (from the removed spine) get stuck to the inside of the scanner or feeder, ruining the scans of subsequent pages until the scanner is cleaned. Can anyone recommend a solution to either of these problems that can hopefully be applied automatically or in a batch operation? Reduction of human intervention/effort is my goal.

5 6

[SPAM] re: !@! #17135 Twas The Night Before Christmas
by Bowerbird＠aol.com 15 Apr '10

15 Apr '10

gardner said: > So there's two more random data points in BB's column. thanks, gardner. but there's no need to be "random". the big sites using p.g. e-books are very easy to find, because they are... um... big. the first, and foremost, of course, was "black mask". david moynihan had the scripts down to a science, and even offered them to p.g. at one point in time, an offer that was declined, for some stupid reason that was not _just_ stupid, but asininely ridiculous. (and no, i don't even know what that reason was.) next up is "manybooks". matthew's converters are not nearly as good as david's, but he seems to have many loyal users who pick up their books from him, probably because he's always been very good about supporting the widest possible array of machinery... both of these providers have been using books from p.g. going back for many years, so the fact that they were using the plain-text versions might be due to their history... that is, they might just be in a rut... but the newest big provider on the block nowadays is "feedbooks". and as far as i can tell, hadrien also uses the plain-text version as his "starter" version... makes sense, since he stores the text in a database, so it does no good to have somebody else's markup. the number of files being download from these sites these days -- thanks to the kindle/iphone/ipad trio -- is downright _stunning_. hadrien at feedbooks says he's running at about 75,000 downloads every _day_, with 2.5 million in march. several individual books have over 10k downloads, some 20k, and one 30k... *** dakretz said: > Does anyone know of any epublisher other than PG > that *does* distribute the html we provide? well, in a way. apple grabs the .epubs from here. if i'm not mistaken, marcello uses the .html file, if one exists. but if not, he uses the plain-text... i'm quite sure the .epub files are full of ugliness. and mike cook and his "epubbooks" site might use the .html file as his "starter", you'd have to ask him. he hasn't done books "en masse", though, and thus hasn't had to face the problems from inconsistencies. -bowerbird

1 0

Question about Scanned Books
by Michael McDermott 14 Apr '10

14 Apr '10

Archive.org has many DJVU files of books that have lapsed into the public domain. Would it comply with PG's guidelines to take one of these volumes (the one I was thinking of has a copyright date of 1915 and can be found at http://www.archive.org/details/worksmartinluth00spaegoog)? The important elements here are: 1) I do not have a copy of the paper edition 2) This is a scan of a work that, by all appearances, qualifies having been published in the US before 1923 3) Was digitized by Google Would this work or would its ancestry cause problems? -- Michael McDermott www.mad-computer-scientist.com

3 2

so let's talk about my collaborative proofreading site, part 2
by Bowerbird＠aol.com 13 Apr '10

13 Apr '10

here's more info on my collaborative proofreading site... *** to see what we're talking about, you can visit this u.r.l.: > http://z-m-l.com/go/sitka/editr.pl *** we talked about 4 main topics last week: > navigating the pages... > certifying a page as clean... > searching the book for a string... > feel the power with the "command" field... under the 4th topic -- the command field -- we discussed 3 of the commands you can issue: > showmap... > concat... > showcustom... today we'll discuss a few more commands... *** blubberbaby... you'll remember that i also discussed how you can implement spellcheck functionality in your workflow. one key to this is creating a custom dictionary for each specific book that you are digitizing, one that contains the words unique to that particular book... at first, you'll have a "bad-words" list, which contains low-frequency words not found in a regular dictionary. the other list is the "good-words" list, which contains high-frequency words plus those in a regular dictionary. the process of correcting the book is one of _moving_ items on the "bad-words" list to the "good-words" list, either by certifying o.c.r. did recognize them correctly, or by correcting the misrecognition to what it should be. (or, in the case of an error by the publisher, correcting it.) what is handy, for this process, is knowing which pages have words that are contained on the "bad-words" list. you _could_ navigate through each of the pages, to see which ones have flagged words, which are shown in red. but why not have the computer just tell us what they are? voila the next command, christened "blubberbaby", to honor alaska, for this sitka book. enter "blubberbaby" in the search-field and click "find", and in a little while -- it's not yet unoptimized, so it's about 20 seconds -- you will be shown a page that includes all of the pages that have words which are still on the "bad-words" list... from that display-page, you can use the links there to open a number of these pages -- each in its own tab -- and work on them to deal with all of the flagged words. questionable words should be handled in preprocessing, for the most part, so if the workflow is designed correctly, you won't need to use this "blubberbaby" command often. but it's useful to have it, so you can do the check if desired. and if questionable words were not fixed in preprocessing, then you'll find "blubberbaby" to be even more important. *** pairsearch... you'll remember when i was discussing _inconsistencies_ in the sitka book that i used the "bad-words" list to find possible problems. specifically, when two variants of a word (usually a name) came up sorted next to each other, it was easy to spot 'em and tell that they needed checking. here are a few of them, so you can see what i mean... > Globokoe************** > Golobokoe************** > ... > Golofnin************** > Golovin************** > ... > Hagemeister************** > Hagmeister************** it's pretty obvious that these _might_ be inconsistencies... not all of them are. for instance, "golofnin" and "golvin" were -- apparently -- the names of two different people. but the others were errors made by the original printer, errors that coulda been caught (i caught 'em) and fixed. what you have to do, though, to check these pairs out, is go to the actual pages where they appear, and read the text, so as to determine the correct course of action. now, with the search capability, it's fairly easy to do that. you just enter each term, and then click on the links to open up the pages where that term appears. fairly easy. but that can get a bit tiresome if you have a lot to check. so i programmed this "pairsearch" command to help out. you enter the command "pairsearch", followed by pairs of terms that you want to search for, and the program presents the relevant pages to help you make a decision. so, for instance, for the three pairs above, you'd enter: > pairsearch hagemeister hagmeister golofnin golovin globokoe golobokoe the search-terms can be separated by spaces or line-ends. the output from that search is appended to this message. the lines are long, and will likely wrap, so it's also here: > http://z-m-l.com/go/sitka/pairsearch-output.html the pagenames aren't linked now, but eventually will be. this "pairsearch" command can be extremely useful in resolving inconsistencies within the book, both those introduced by o.c.r. and those by the original publisher. one more note... remember that publishers back in the old days didn't have the wonderful tools that we now have at our disposal, so it's no wonder that they had some problems when it came to words like "globokoe" and "golobokoe", or russian names. i'm sure if i had to use the primitive tools they had back then, i'd be making 3 times as many errors as they made, or more... *** end-page-hyphenates d.p. has proofers mark end-page-hyphenates with an asterisk. i'm not sure why they feel that's necessary. the computer can find end-page-hyphenates just fine. here's a routine to do it. put the command "end-page-hyphenates" in the search-field, and then click "find", and you'll get a list of where they occur. the list has links for both pages, containing both fragments... for this book, you'll get this: > sitkap002.txt ... and ... sitkap003.txt > sitkap007.txt ... and ... sitkap008.txt > sitkap018.txt ... and ... sitkap019.txt > sitkap019.txt ... and ... sitkap020.txt > sitkap021.txt ... and ... sitkap022.txt > sitkap027.txt ... and ... sitkap028.txt > sitkap043.txt ... and ... sitkap044.txt > sitkap051.txt ... and ... sitkap052.txt > sitkap077.txt ... and ... sitkap078.txt > sitkap079.txt ... and ... sitkap080.txt > sitkap083.txt ... and ... sitkap084.txt > sitkap087.txt ... and ... sitkap088.txt > sitkap102.txt ... and ... sitkap103.txt those pagenames are clickable, and take you to that page... there isn't a lot of reason you need to check those fragments, since the computer will also rejoin 'em if you unwrap the text. but if i didn't include this functionality, you know _someone_ would say "yeah, but your system doesn't do _this_, does it?" so now i can say, "well yes, as a matter of fact, it _does_..." *** so, we've added "blubberbaby" and "pairsearch" commands, as well as "end-page-hyphenates"; that's enough for today. by now, you should have a pretty good feel on how we will continue to implement functionalities as they are needed... we'll discuss more stuff as i get it put into place... -bowerbird p.s. here's the output from the "pairsearch" command above: > .....here it is, in order of appearance in the book: > > globokoe ... sitkap002.txt ... the inlet at Ozerskoe Redoubt and Globokoe (Deep) Lake; the island-studded > hagemeister ... sitkap006.txt ... ngland English Francisco Georgeson Hagemeister Jamestown Kashavaroffs Katle > hagemeister ... sitkap009.txt ... g instructions previously given to Hagemeister, instructing him to find the > golofnin ... sitkap032.txt ... r the command of Captain Vasili M. Golofnin, who was widely known for his a > golofnin ... sitkap034.txt ... stant, nor one doctor's pupil.'?? Golofnin soon left Sitka to return to St > hagmeister ... sitkap042.txt ... ills of Golden California. Captain Hagmeister came to re- lieve him, and in > golofnin ... sitkap045.txt ... to trade with the Kolosh [45-1] Golofnin, Voyage of the Sloop "Kamchatka > golofnin ... sitkap060.txt ... ccording to the account of Captain Golofnin, it was an establishment well b > golovin ... sitkap072.txt ... erica, by Captain-Lieutenant P. N. Golovin, pp. 72-73. [[72]] > globokoe ... sitkap072.txt ... other at the Ozer- skoe Redoubt on Globokoef[72-2] (Deep) Lake, ground the > golobokoe ... sitkap072.txt ... f the present improvement. [72-2] Golobokoe Lake was sounded to a depth cf > hagemeister ... sitkap075.txt ... nuary 11, 1818. Leonti Andreanvich Hagemeister, Jan. 11, 1818, to Oct. 24, > globokoe ... sitkap105.txt ... mountainside. The Redoubt and the Globokoe Lake.-- Southwest from Sitka ab > globokoe ... sitkap106.txt ... re in the rocky wall which divided Globokoe, or Deep Lake, from the sea, an > > > .....and sorted, by search-term: > > globokoe ... sitkap002.txt ... the inlet at Ozerskoe Redoubt and Globokoe (Deep) Lake; the island-studded > globokoe ... sitkap072.txt ... other at the Ozer- skoe Redoubt on Globokoef[72-2] (Deep) Lake, ground the > globokoe ... sitkap105.txt ... mountainside. The Redoubt and the Globokoe Lake.-- Southwest from Sitka ab > globokoe ... sitkap106.txt ... re in the rocky wall which divided Globokoe, or Deep Lake, from the sea, an > > golobokoe ... sitkap072.txt ... f the present improvement. [72-2] Golobokoe Lake was sounded to a depth cf > > golofnin ... sitkap032.txt ... r the command of Captain Vasili M. Golofnin, who was widely known for his a > golofnin ... sitkap034.txt ... stant, nor one doctor's pupil.'?? Golofnin soon left Sitka to return to St > golofnin ... sitkap045.txt ... to trade with the Kolosh [45-1] Golofnin, Voyage of the Sloop "Kamchatka > golofnin ... sitkap060.txt ... ccording to the account of Captain Golofnin, it was an establishment well b > > golovin ... sitkap072.txt ... erica, by Captain-Lieutenant P. N. Golovin, pp. 72-73. [[72]] > > hagemeister ... sitkap006.txt ... ngland English Francisco Georgeson Hagemeister Jamestown Kashavaroffs Katle > hagemeister ... sitkap009.txt ... g instructions previously given to Hagemeister, instructing him to find the > hagemeister ... sitkap075.txt ... nuary 11, 1818. Leonti Andreanvich Hagemeister, Jan. 11, 1818, to Oct. 24, > > hagmeister ... sitkap042.txt ... ills of Golden California. Captain Hagmeister came to re- lieve him, and in > > > .....and sorted again, this time in the order in which they were entered: > > hagemeister ... sitkap006.txt ... ngland English Francisco Georgeson Hagemeister Jamestown Kashavaroffs Katle > hagemeister ... sitkap009.txt ... g instructions previously given to Hagemeister, instructing him to find the > hagemeister ... sitkap075.txt ... nuary 11, 1818. Leonti Andreanvich Hagemeister, Jan. 11, 1818, to Oct. 24, > > hagmeister ... sitkap042.txt ... ills of Golden California. Captain Hagmeister came to re- lieve him, and in > > > golofnin ... sitkap032.txt ... r the command of Captain Vasili M. Golofnin, who was widely known for his a > golofnin ... sitkap034.txt ... stant, nor one doctor's pupil.'?? Golofnin soon left Sitka to return to St > golofnin ... sitkap045.txt ... to trade with the Kolosh [45-1] Golofnin, Voyage of the Sloop "Kamchatka > golofnin ... sitkap060.txt ... ccording to the account of Captain Golofnin, it was an establishment well b > > golovin ... sitkap072.txt ... erica, by Captain-Lieutenant P. N. Golovin, pp. 72-73. [[72]] > > > globokoe ... sitkap002.txt ... the inlet at Ozerskoe Redoubt and Globokoe (Deep) Lake; the island-studded > globokoe ... sitkap072.txt ... other at the Ozer- skoe Redoubt on Globokoef[72-2] (Deep) Lake, ground the > globokoe ... sitkap105.txt ... mountainside. The Redoubt and the Globokoe Lake.-- Southwest from Sitka ab > globokoe ... sitkap106.txt ... re in the rocky wall which divided Globokoe, or Deep Lake, from the sea, an > > golobokoe ... sitkap072.txt ... f the present improvement. [72-2] Golobokoe Lake was sounded to a depth cf > > > --30--

1 0

[SPAM] PGDP down?
by Greg Weeks 13 Apr '10

13 Apr '10

Is PGDP down, or is something with just my link? -- Greg Weeks http://durendal.org:8080/greg/

4 3

Project Gutenberg DVD Release Candidate now available for testing
by Aaron Cannon 13 Apr '10

13 Apr '10

Hi all. Finally, I have a release candidate of the new Gutenberg DVD available for testing. Currently, it is only available via BitTorrent. Please download and test, and let me know if you find any bugs. At this point, I'm not looking for suggestions on which new titles to add, but any other feedback is welcome and appreciated. I would not yet recommend distribution of this DVD image for any purpose other than testing. Please do not seed this torrent after April 30, because by then, we should have an official version available. This image is an .iso for a dvd-9, so you will either need a way to open or mount .iso files, or a dual layer DVD burner. You can download the torrent via the following magnet link: magnet:?xt=urn:btih:JB4PMPIXNTYMFCUZABAYYTX56JWQINPI or if your client doesn't speak magnet links, you can download the torrent file from: http://www.fireantproductions.com/pgdvd201004-rc1.torrent Thanks in advance for all the feedback. Aaron Cannon

2 2

spell-check functionality and the sitka book
by Bowerbird＠aol.com 12 Apr '10

12 Apr '10

in discussing how to bring about a spell-check functionality, i talked about the "bad-words" list as a part of that workflow. i posted the "bad-words" list from the o.c.r. for the sitka book. > http://z-m-l.com/go/sitka/sitka-reversedictionary.txt i talked about how the proofing process can be envisioned as movement from the "bad-words" list to the "good-words" list. now that rfrank has posted the "final" version at p.g., we can do the same "dictionary-check" procedure on his finished book: > http://z-m-l.com/go/sitka/sitka-reverseposted.txt you'll see i have introduced blank lines into these files, so as to coordinate their lines so they can be merged into one file. this will allow us to see how each word traversed the process. the merged file is here: > http://z-m-l.com/go/sitka/sitka-reverse-review.html *** go ahead and take a quick look at the words... one thing you might notice is that a number of the words are marked with asterisks. those are the ones that are suspicious, in that they appear to be _variants_ of each other, which might be a good indication of either (1) an o.c.r. misrecognition, or (2) an inconsistency in the p-book which should be corrected... (some people use levenshtein edit-distance to find these words. that's nice, but a plain old review of a sorted list works well too.) i checked those variants against the p-book, and many of them were indeed errors. (the ones marked "ok" at the end were right.) that's how i found many of these consistency errors rfrank missed. oh, by the way, there is still at least one more of those errors that i didn't mention earlier... so if anyone wants to go looking for it... ok, so let's go on to look at the list in other ways... *** for each individual word, we're gonna see how it was handled... the first group are some words that had garbage characters. those words didn't have any direct equivalent in the final file, at least none that were close enough to be sorted similarly... once we get into the lowercase words, we get some matching. and that continues as we get into the words with an initial cap, and on into the compound-words, and then into the numbers... focus on the initial-cap words -- which are primarily names -- and you'll see that the vast majority of these were recognized correctly, in that they persisted through to the final version... about 85% of the initial-cap words were recognized correctly. the same is true of the compound-words, with few exceptions. and most numbers seem to have been recognized correctly too. but lowercase words are more of a mixed bag. some were right, it's true, but a relatively high percentage of them were incorrect, as evidenced by the fact they were changed, one way or another. only about half the lowercase words were recognized correctly. you might remember that i had predicted precisely this pattern. most lowercase o.c.r. words that are in the "bad-words" list are generally misrecognitions, while most initial-cap words are not. indeed, the percentage of correct lowercase words in this o.c.r. was much higher than is normal, because this was not typical o.c.r., in the sense that the scans were clean, but also because rfrank probably did some preprocessing on the raw o.c.r. text, which we can tell because it had very few garbage characters... so what we see is that roughly 75% of these words were _correct_, despite the fact they were "bad-words" (i.e., not in the dictionary). they weren't in the dictionary, but they were "good" in this book... that so many "bad-words" can be correct and unique to the book is why it's important to use a "custom" book-specific dictionary. only about 25% of the "bad-words" were actually misrecognitions. to the extent that you can narrow down your _flagging_ of the "bad-words" to the ones that are _really_ bad, you can relieve your proofers of a _lot_ of unnecessary flags, which is _good_, because false flags sap the attention of proofers unnecessarily. once you've done this analysis a number of times, like i have, you'll come to recognize that it is a very important analysis... -bowerbird p.s. hey, dkretz, thanks for the shoutout! but one correction! i wasn't "baited" into "crossing a line" that got be banned at d.p. i never get "baited" into _anything_. i always know what i'm doing. and i've been banned from enough places that i know how it works, so it wasn't that i "crossed a line". again, i know what i'm doing... no, if i get banned from somewhere, it's something i _anticipated_, and after a consideration of that outcome, decided it didn't matter. which is _not_ to say that i _like_ to get banned, or that i _try_ to, but is _rather_ to say that i won't allow myself be banned _unless_ i have decided that i don't really care whether i'm banned or not... as for "crossing a line", there's no need for it. even though people will generally say that i broke some technical rule and that is why i was banned, the truth of the matter is that one only gets banned if one pisses off the person with the power to push the ban button. it has nothing at all to do with "the rules"... it's just raw emotion... oh, and let me say one more thing; it's _nice_ that d.p. still lets me come to their forums and read them. if they tried to prevent that, i _could_ get around it, but it's a hassle. so i thank them for that...

1 0

more notes from the merry-go-round
by Bowerbird＠aol.com 12 Apr '10

12 Apr '10

gosh i love a merry-go-round... :+( *** here's a few notes, repeated too many times already, but whatchagonnado? if this doesn't make sense to you, that's probably because it wasn't intended for you, so pay no mind. *** any heavy bracket markup like [i]italics[/i] is obtrusive, and that's why proofers complain about it, justifiably... use light markup, like _italics_, and they won't complain. better yet, have them do the proofing with an .html field -- using a text-area field only when editing is required -- and you can use _real_ italics (and highlight it with color). you can also use _color_ to flag your questionable words, which makes the flagging a whole lot harder to "miss"... *** spacey double-quotes are _easy_ to solve if you pay heed to the _paragraphs_. ergo... pay heed to the paragraphs! yes, paragraphs cross page-boundaries... but so what? you start with the first file, and keep track of paragraphs while you proceed to the second, and the third, and so on. there's a ton of redundancy in the quotes -- open, close, open, close, open, close. make use of that redundancy... it's not rocket-science; it's not even difficult programming. get over the mind-blockage you have on this topic. *** and once you _do_ get over that mind-blockage, you just might see that books that use _single-quotes_ for dialog aren't really all that different. "but", you are sputtering, "yes they are, because contractions cause big difficulties!" poppycock. here's a file with a list of contractions (among other stuff): > http://z-m-l.com/customdictionary.txt that file's been up since june of 2007. use that list intelligently to control those contractions... possessives also use the single-quote, but they're easy to deal with as well; just do a little thinking about them. don't give up so easily. _try._ you'll find it's not as hard as you thought. and if you actually honestly _try_, and hit a wall anyway, show me your actual honest efforts, and i will help you... *** "probable markup on this page"? are you purposely trying to be vague? how about listing the italicized words, _specifically_? geez! -bowerbird

1 0

Newby/Hart at Illinois symposium April 15-16
by Greg Newby 12 Apr '10

12 Apr '10

For those in the region, this might be of interest: http://50years.lis.illinois.edu/ PGLAF CEO Greg Newby will join PG founder Michael Hart at a symposium on the U. Illinois campus. Registration is free but limited. The panel with Michael & Greg is scheduled for Thursday April 15 from 1:30-3pm. -- Greg

1 1