- gutvol-d - lists.pglaf.org

let's step through this process
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

don said: > Trac i've explained to y'all many times how to do this job simply. yet you continue to flock to the most complicated processes. so go ahead and do it that way; you will get what you deserve. *** for the people who wanna actually get something accomplished, though, i'll step through a process to show how it can be done... *** here's a copy of "pride and prejudice" at archive.org: > http://archive.org/details/harvardclassicss03elio you want to use a scan-set from archive.org or google, because those are canonical copies available to the public. no one has to "take your work" that this is an edition that is true to the author, because it'd been sitting in a library. and no one can accuse you of "doctoring" the scans, since the scan-set is being housed by an entity external to you, and can be fully accessed independently of your influence. a scan-set from a book you bought at barnes&noble has _neither_ of those two necessary and sufficient qualities. *** to work on that edition, i set up this folder on my site: > http://zenmagiclove.com/prhpr first off, i copied over all of the scans from archive.org. "pride and prejudice" was reprinted with another book, meaning that it didn't actually even begin until page 141, so the scans start with page 141, and continue up to 512. > http://zenmagiclove.com/prhpr/prhprp141.jpg > http://zenmagiclove.com/prhpr/prhprp512.jpg (512 is actually a blank page, but we need an even number.) you will find the original .djvu o.c.r. text-file in that folder: > http://zenmagiclove.com/prhpr/harvardclassicss03elio_djvu.txt i also re-saved that file with a new name, befitting its new folder: > http://zenmagiclove.com/prhpr/prhpr-000.zml i did some reworking of the file, mostly to show pagebreaks: > http://zenmagiclove.com/prhpr/prhpr-001.zml i think you can pretty clearly see this is just a text-evolution. then i did more work, for correct paging, and some niceties: > http://zenmagiclove.com/prhpr/prhpr-002.zml i think you can still see that this is just more text-evolution. then i wrote a program to turn that .zml file into rough .html: > http://zenmagiclove.com/prhpr/prhpr-002h.html and again, the relationship to the earlier .zml file is very direct. then i did a few global-type changes to start cleaning the o.c.r.: > http://zenmagiclove.com/prhpr/prhpr-003.zml then, to show how simple it can be to use a diffing procedure, i wrote a python script that compares 002.zml against 003.zml: > http://zenmagiclove.com/prhpr/docomp.py if you run that script -- by just clicking on that link there -- you'll see the lines that were edited within the first 2000 lines. the top line's from prhpr-002.zml, bottom from prhpr-003.zml. there is also a "change-line" below the other 2 lines, which shows where the lines differ (as sometimes it's hard to see the change). for instance, the change-lines of the diffs on lines 176 and 183 reveal that i globally deleted the space in front of a semicolon... the leftmost part of the change-line is a link that opens the scan for the relevant page in another window. (it will continue to open the scan in a named window, so you should make sure that it _is_ a window, not a tab, so you'll see both windows at the same time.) i have coded compare programs which are much more powerful, but this one serves its purpose as a little app to demo comparison. more tomorrow. but for now, the takeaway: none of this is hard. -bowerbird p.s. there's something very revealing in those .html versions, so take a little look at them and see if you can tell me what that is...

1 0

diffing as a superior methodogy to word-by-word proofing
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

robert said: > These appends seem to me to imply that > diffing parallel/independently produced texts > is a faster and more efficient way > to correct the OCR process > than sequential manual checking. much faster. much more efficient. > Has anyone (possibly including DP) > ever tried any of the above, and > documented the results in a scientifically valid way? well, yeah. just one week from today, it will be 7 full years since i posted this: > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-03,3 > Does DP work that way anyway? nope. and they didn't want to learn, either. because, at the end of 2006, i posted this in a d.p. forum: > http://www.pgdp.net/phpBB2/viewtopic.php?t=24008 *** much faster. much more efficient. absolutely no comparison. it's a complete waste of time to proof by reading word-by-word. *** by now it should be clear to everyone why i am getting so tired of repeating myself. -bowerbird

1 0

straight quotes
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

carlo said: > For example, one cannot assume that > a ' between letters is an apostrophe, hence > rendered with a right single quotation mark > (do you know the standard exception in English?) no, carlo, i don't know the exception. what is it? -bowerbird

2 1

might have a suitable dead trees edition
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

greg said: > There ARE master scans available for a number of eBooks. not for "classics" that were posted early and are downloaded often. > I'm going to visit B&N today to see whether they > might have a suitable dead trees edition for this purpose. it'd be a _very_ bad idea to use such a version as a "master". the good part about that idea is that a modern printing will give very clean o.c.r. if that o.c.r. was then proofed against a _worthy_ scan-set -- one that was published "back then" -- it _might_ (emphasis on "might") save a bunch of proofing... (depending on the source used by the current-day publisher). it's worth an experiment, but i would be very sensitive to whether it was actually working well, and quit if it wasn't. -bowerbird

1 0

blahblahblahblah
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

jon said: > 1. Develop a methodology for nominating master scans > where no record of provenance if extant text exists. > > 2. Source master scans and upload to PG. > > 3. Run master scans through DP until P2 > using LOTE non-clothing exception. > Capture P2 output and diff it against extant text > to produce the RTT. Note that if P2 output is perfect > the RTT _is_ P2 output. > > 4. Diff RTT against extant text > to produce comprehensive errata list, > and deliver to WWs so they can update the e-texts. > > And that, for the moment, is the limit of my ambition. > As Bowerbird points out, I have more than likely > just described an impossible scenario, even though > from a technical point of view it is completely trivial. well, yes, it's "impossible" because the people who would be necessary to make it work have no interest in doing it. but even if they did, this isn't really "completely trivial"... the twist is your assumption that there _must_ be a solid and usable provenance for the existing e-texts. for some of the classic e-texts, though, the "source" was an amalgamation of several different editions... so you're not gonna find a scan-set for "provenance". and for others, the "source" document was a _reprint_ done by the "classics" imprint of a corporate publisher. so you're not going to find a scan-set for those, either. (and even if you did, you wouldn't want to use it, since those imprints often "took liberties" with the real text.) therefore, steps 3 and 4 in your list simply won't work, not in the way that you've described them. furthermore, the whitewashers aren't gonna accept any "error-reports" that might be based on your work, because your scan-set wasn't the actual source of the currently-existing e-text. (the fact that the currently-existing file _has_no_source_ is of little concern to them, or to anybody else, it'd seem, except to some of us obsessive-compulsive fuss-budgets, although i've predicted, for some time, that will change.) the simple solution, of course, would be to replace the unsourced file with the one that has a solid provenance. but, as we see, p.g. doesn't seem to want to do that... -bowerbird p.s. it's also worth stating that even a replacement will _not_ be a cure-all. many end-users get their files from "republishers" -- manybooks.com, feedbooks.com, etc. -- and i'm not sure if those republishers go back and re-do the low-numbered "classics", even when they're replaced.

1 0

Improving Dracula?
by Peter Hatch 26 Sep '12

26 Sep '12

On Wed, Sep 26, 2012 at 10:35 AM, Greg Newby <gbnewby(a)pglaf.org> wrote: > My suggestion has been to start from scratch with a new scan set, > rather than trying to fix existing eBooks. It's OK to choose > a title we already have. I'm going to visit B&N today to see whether > they might have a suitable dead trees edition for this purpose. This reminds me, I spent some time looking into editions of Dracula, and it could really use a new scan. I found details of the editions at http://www.cesnur.org/2003/dracula/I.htm - there exists an 1899 American edition that is worse than the original, due to numerous minor changes and one significant alteration - from cesnur.org: That alteration occurs in chapter four near the end of Harker’s diary entry for June 29. He is describing a conversation he overheard between the Count and the three women who also resided in the castle. In the English edition, Dracula says, "Wait. Have patience. Tomorrow night, tomorrow night, is yours!" while the American edition reads, "Wait! Have patience! Tonight is mine. Tomorrow night is yours!" The origin and meaning of this change, the American edition implying that Dracula will feed off of Harker, the only hint of Dracula feeding on a male, has become an issue in understanding the text. The Gutenberg edition of Dracula is based off the American text, as is every free scan I could find. -- Peter Hatch

1 0

straight quotes
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

mark- it's entirely possible to convert quotes between straight and curly. there are some edge cases, but they are _extremely_uncommon_. i've detailed how to do it here many times. look in the archives if you really want to do it. i'm tired of explaining it repeatedly. the thing is, at project gutenberg, each book-producer can decide for themselves whether they want to use straight or curly quotes, so you're stuck with whatever they decide. unless you do the work. i'm truly sorry if this offends your typographic sensibilities, but nobody is gonna force the book-producers to curl their quotes. so you might just try to get used to taking them like they come. i suggest you think about more important matters in the world. there must be some. -bowerbird

1 0

blahblahblah
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

roger said: > I wonder if this paragraph > summarizes the basics of > what has been proposed: i think the thing you left out, roger, is that jon wants to "update" the files of the books downloaded most often. but those are the very files which the whitewashers are least likely to replace, at least as far as i have been able to tell. unless the updater did the original one. or unless the updater is a whitewasher... (and if anyone sees _any_ evidence to the contrary, i would certainly love to hear it.) the whitewashers are quite willing to accept "new" versions of these classic old chestnuts, but they're going to give them new numbers, and not overwrite the currently-existing files. and the other thing you left out, roger, is that jon isn't interested in doing this _unless_ the old chestnuts are replaced. he doesn't want to upload different files, let alone do this experiment elsewhere. at least that's what i heard him saying... so it appears that we are at an impasse, which is probably why you hear the sound... ...of crickets chirping... -bowerbird

1 0

eb.readingroo.ms
by don kretz 26 Sep '12

26 Sep '12

I can scavenge some code from DP-IT and offer a facility for viewing images and text in parallel using the WordPress installation at eb.readingroo.ms . A couple issues poked up in the process of checking this out. 1. It is possible to use the Trac installation that Greg set up on the same server. But it's written in Python, in which I'm but a novice, and anyway it would be preferable to set up a web service to query for text and images using something like JSON or XML/RPC. If someone (i.e. Roger) could look into using the Trac component architecture (Google for info; looks reasonable to me) we could collaborate on the API contracts and make it available broadly from anywhere (and in my specific case PHP and javascript.) 2. Just about any practical use I can think of requires synchronizing images and text, which for any technique I can think of requires that a.) the images be available and named according to page number (which is true if the PG naming convention was employed), and also b.) that the page numbers be recognizably embedded in the text stream (which happens sometimes in the html version, but I've never seen it in the plain text stream, which is more useful for our purposes.) What one would wish for is the text as it exists before the Post-Processors start their work; but unfortunately that's not available at all on PG, and stands behind the Wall of Status Quo on DP. This needs further discussion. Don

1 0

meanwhile, elsewhere birds are singing
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

some of you have checked out my jaguar demo. thanks. some of you got some errors on those external files -- like pride and prejudice, and the sherlock holmes book -- but i've fixed the input files if you wanna try again... the "october" version will be even more solid. see ya then. -bowerbird p.s. i'll be taking this to kickstarter soon. if the funding for my project(s) comes in, my code will go open-source.

1 0