[I'm forwarding the following message by "NetWorker" posted to The eBook Community. Any followup to the specific points NetWorker raises are maybe better posted over there, especially if you'd like NetWorker to see your comments ("ebook-community" at YahooGroups). NetWorker is very thorough... Jon] NetWorker wrote [a few days prior]:

Project Gutenberg e-texts may yet have a role to play in the production of high-quality e-books. Rising to the bait, I have placed a hold on my local library's one(!) copy of Frankenstein, which I will scan when it arrives. I will try to create a highly structured e-text (certainly not as fine as what Jon did with "My Antonia"). I will then try to find a way to preprocess both the OCR'ed text and the Project Gutenberg e-text in such a way that the two files can be meaningfully "diffed." Hopefully, I can come up with a method that will allow existing PG e-texts to be an automated "proofread" of next generation Public Domain e-books.

Boy, _that_ was an interesting experience! The project goals: As an e-book consumer, I want an e-book that contains _lots_ of metadata; the more the better. I want the metadata to be patterned, so that I can use automated tools to manage a collection; sorting by author, genre, publication date, publisher, contributors such as editors and illustrators, etc. I also want the actual text to be marked-up in such a way that 1) I can view the text with all the presentational richness traditionally associated with a paper book, if I choose to do so, 2) I can convert unambiguously from one markup language to another, and 3) so that I can do a structural analysis of the book using automated tools. I also want a mechanism to know if apparent errors in the text are due to transcription errors or the author's intent -- this can be accomplished by including source information in the metadata, or providing access to page scans; both would be preferable. Project Gutenberg e-texts satisfy none of these wishes; to create an e-book which _does_ satisfy them pretty much requires starting from scratch. Scanning technologies are quite advanced these days, but OCR is still not 100% accurate, and automated spell checking can only go so far. Clearly the most time-consuming -- and most error-prone -- part of producing a reasonably accurate e-book is proof-reading by a human being. My goal was to discover if Project Gutenberg e-texts, which are presumably fairly accurate as to the _words_, if nothing else, could be used as yet another automated preprocessing step to reduce typographical errors to a minimum before the actual proof-reading begins. The process: To test my theory, I decided to use the novel _Frankenstein_, by Mary Wollstonecraft Shelly. _Frankenstein_ is clearly in the public domain, is known to have at least two versions, and has been the subject of a fair amount of discussion on this list in the recent past. I obtained a copy of Frankenstein from the public library; it was published in the "Barnes & Noble Classics" series in 2000. I was fairly pleased with the edition, as it was printed in a rather old-seeming type-face which gave the appearance that it was in fact a photo reproduction of a much older text; it seemed likely that it had not gone through much in the way of re-editing to modern conventions. I scanned and OCR'ed the book using ABBYY FineReader. I then did a spell-check of the book from within FineReader so I could compared "misspelled" words to the actual scanned image. I then saved the text as an HTML file. In the past I have written a couple of programs to help in the creation of e-books. TidyeBook is based on the HTML Tidy code base. It fixes some of the inaccurate HTML produced by ABBYY, strips headers and footers but leaves page numbers intact, if invisible, and merges broken paragraphs when it can do so without question. html2txt, based on an earlier C++ version of HTML Tidy, takes an HTML document and reduces it to simple text similar to that used by Project Gutenberg. Next I ran "frankenstein.html" through TidyeBook to clean up the HTML. I then hand-edited the HTML to fix paragraph breaks not fixed in the automated process, or which should not have been broken. I also fixed those instances where hyphenated words spanned a page break (very easy to do given the output of TidyeBook). I then generated an Impoverished Text Format version of the HTML text using html2txt. My strategy was to use the Gnu "diff" program to detect differences between the simplified version of my work product, and the Project Gutenberg version. Because "diff" is line-oriented I needed to normalize the two texts so there was a greater likelihood that lines would be correctly matched. I did this by writing yet another program (this could probably have been done more efficiently by a Perl or AWK script, but I am not very familiar with scripting languages, but am a highly proficient C/C++ programmer; it was easiest for me to use the tools at my disposal). The new program would reduce each file to lines of no more than 60 characters (the shorter the line the easier for a human to find the difference detected). Additionally, the program would start a new line whenever it encountered what is conventionally accepted as sentence-ending punctuation (!.?) or two newline characters in a row, which would signal the beginning of a new paragraph. All whitespace was reduced to a space character, including multiple whitespace characters. I used the new program to normalize the text produced by html2txt and that of frank14.txt from Project Gutenberg. I then compared the two resultant files using gnu diff and Microsoft's WinDiff. The results: I was quite surprised to find literally thousands of differences between the two texts. Most of the differences were changes in punctuation and capitalization. Many em-dashes were converted to semicolons or omitted altogether, and many semicolons were converted to commas. Some words capitalized in my scan (eg. Paradise) were converted to lower case (paradise). Some phrases were "fixed" ("our uncle Thomas's book" became "our Uncle Thomas' book"; "an European" became "a European"). Some words were Americanized ("tranquillise" became "tranquillize") yet other words are not ("favourite" remained "favourite"). In an attempt to discover the source of these differences, I visited a number of not-so-local libraries, and checked out a number of different printings of _Frankenstein_. Two of the most interesting are Leonard Wolf's _The Annotated Frankenstein_, Clarkson N. Potter, 1977, which claims that "In order to ensure the authenticity of the text, we arranged with the Library of Congress in Washington, D.C., to microfilm a copy of the first edition. That text has been reproduced in this volume by the photo-offset process," and the Penguin Classics edition which includes an appendix identifying the differences between the 1818 and 1831 editions (while significant, they are neither as pervasive nor as substantive as has been earlier suggested). Neither of these editions contained the punctuation or spelling changes of the Project Gutenberg edition. One of the books I checked out, rather serendipitously as it turns out, was the Bantam Classic edition, which was first published in 1981. Of all the editions I consulted, only the Bantam edition contains virtually all of the changes I noted in the Project Gutenberg e-text. The PG edition is apparently based on Mary Shelly's revised 1831 edition (although it has lost both the "Author's Introduction to the Standard Novels Edition (1831)" by Mary Shell, and the "Preface" to the 1818 edition by Percy Shelly). I thus believe that the PG edition is based on the Bantam Classic edition of 1981. <sidebar> Interestingly, copyright law provides protection to changed versions of public domain texts if those changes are of a nature that they are more than mechanical and provide some modicum of creativity. Clearly, the punctuation changes are not merely mechanical, and in some cases actually change subtly the nuances of the text. Ironically, of all the textual bases that Project Gutenberg could have used for its e-text of _Frankenstein_, it choose the one which is apparently still protected by copyright! </sidebar> I modified my text normalization program to discard all punctuation except hyphens and underscores (and of course excluding the sentence-ending punctuation mentioned earlier). This reduced the noise to signal ratio enough that the differences started to become meaningful, although it still resulted in at least 500 differences. It allowed me to discover a handful of OCR errors that had been missed by the earlier automated methods, and I have so far also found a handful of errors in the PG text ("But must finish." should be "But I must finish.", "every sight ... seem still to" should be "every sight ... seems still to", "destroy radiant innocence" should be "destroy such radiant innocence", etc.) Conclusions: Of course, the goal of this exercise was not to establish the provenance of the Project Gutenberg e-text of _Frankenstein_, nor to discover if there are any errors in the PG e-text, but to determine if there was an automated method of reducing errors in newly scanned e-books for which a Project Gutenberg e-text already exists. I'm afraid the jury is still out on this question. If the texts are a different as the PG edition of _Frankenstein_ and virtually all other editions, the process of sorting through the chaff to find the grain of wheat may not be worthwhile; I believe that the OCR errors discovered so far were blatant enough that they would have been easily discovered in the first proof-reading, and I believe that human proof-reading will always be required no matter how good our automated tools become. Some time ago I produced an HTML e-book version of Mark Twain's _Pudd'n'head Wilson_. I believe I will put that e-book through that same process. I will then attempt a new scan of some other, perhaps more obscure, PD work that already has a PG version. Having at least three data points, I will report again later. p.s. -- If someone from Project Gutenberg wants my diff file to update the PG e-text, I will be happy to e-mail it to you; it is approximately 95k in size.