roger said:
my sampling of the kind of errors left behind by BB's process as far as it went.
thanks for the fuller exposition, roger... just as a note, before we begin, i'd like to issue one minor note of protest that these errors are labeled as "bb:", when, actually, they were errors that were in _the_o.c.r._... it's not as if i _introduced_ these errors... it's just that my preprocessing failed to find and fix them. there's a difference... i don't have a suggestion for an alternative, and i see the reason for that nomenclature, just want people to grok the bias in the label. *** on a continued note... it's also the case that my preprocessing _did_ find-and-fix some 250 errors in the o.c.r., so i think the fact that it missed 21 is acceptable, as a first-pass. it can probably be improved, too, but i believe a 92% reduction in errors is nothing to sneeze at. even if i missed _100_, that would be a 70% reduction in o.c.r. errors. *** now that we're done with the minor protest... for the folks who aren't paying full attention, the whole point behind my argument is that a minimum of effort can fix a _lot_ of errors. i have never claimed it can find _all_ of them -- indeed, that would be a form of magic -- or that it's a _sufficient_ mechanism, by itself. this "minimum of effort" absolutely _needs_ to be followed by a beta-reading component. although i have never failed to stress that need, sometimes people seem to forget it completely. please remember to consider the full picture... once you do, you'll understand my rejoinder: my beta-readers woulda caught these errors. (just like roger's smoothreader caught them.) having said that, it is of some concern to me that this seems to change the error-rate so it becomes worse than 1-error-per-10-pages. gonna have to see what i can do about that...
With the addition of a good smoothreader, many of these diffs would have disappeared
right. (well, i'd say _all_ of them, but i'll settle with "many of them" if you insist that it's true, as we're talking about imaginary beta-readers, in my case, so i can't be certain of their skills, but if you say they're not perfect, i won't argue.) these results are from a midpoint in the workflow. while all the errors you reported are very real -- or almost all of them, anyway -- they are from classes that i have oft-acknowledged as ones that are missed in my _preprocessing_... there's a good reason i use that particular word -- _preprocessing_ -- because it _emphasizes_ the point that this doesn't create a final product. let's look at those acknowledged weakspots:
stealth scannos missed italics splotches spelling discrepancies
"stealth scannos" are something my preprocessing will never catch. that's why it needs beta-readers. "missed italics" will depend on abbyy's .rtf quality. if you don't use abbyy, you'll do all of it manually. and if you suck at it, like me, the results will suck. it's not that i cannot imagine coding a routine that could check for stealth scannos, or italics. i could develop some ideas and take a stab at it. i just wouldn't expect that i'd be very successful, in cost-benefit terms. you can do lotsa checks for stealth scannos, but you get too many false alarms. and my whole point is that you utilize cost-benefit. it's easy -- and efficient -- for beta-readers to catch stealth scannos, and even missed italics... so let them do that job! that's the wisest course. "splotches" are something else that's hard to catch, sometimes even impossible. i'll look at these errors, and see if they are the type of thing that _could_ be detected with some programmed routines. but still, it's obvious that some can't, which is -- yet again -- why you need to have beta-readers in the workflow. as for "spelling discrepancies", i do a spellcheck, but if both forms pass, then i let both of them go. and i didn't do consistency checking on this book. i _will_ make it part of my process, when i finally formalize it, but i didn't do it on this book, nope. *** so this wasn't a fair fight. roger used a smoothreader, and i didn't. so _of_course_ his results will be better... roger might've even done a word-by-word proofing, similar to the kind done at distributed proofreading. that can give even _better_ results. sometimes not, but sometimes it _will_, there's no question about it. where the question comes in is whether it's worth it, in terms of the time and energy that it takes people. go take a look at a project page over at d.p., and see how much time it takes the proofers to step through the pages of a round. you can see a page being saved every minute, or two, or three, so a 256-page book -- like "betty lee" -- can take about 600 minutes to go through a single round. that's 10 hours of work! for one round! is this a wise use of time and energy? i sure don't think so. when a beta-reader does a book, at least they get the pleasure of actually _reading_ it... those proofers over at d.p. don't even get that benefit. i don't think the time and energy that they _volunteer_ is being used in the best possible cost-benefit manner. and _that_ is what this discussion should be about... *** all in all, these results don't surprise me one bit. this o.c.r. had more scannos and splotches than i usually find, but the scans weren't all that hot, as you'll readily see just by paging through 'em. maybe i'll run 'em through scan-tailor, just to see if i can then get better o.c.r. out of them... but if roger will promise to make available his data -- i.e., scans, r.t.f. -- for his future books, i will formalize my preprocessing into a system, so he could run regular tests of its efficiency... -bowerbird p.s. now, notes on each of the individual errors... stealth scannos -- i leave these for the beta-readers...
RF: some of them, and give Ramon's message, BB: some of them, and give Earn on's message,
shoulda caught this, as an unexpected mid-sentence cap.
RF: Betty and next to Peggy Pollard, who, it BB: Betty and nest to Peggy Pollard, who, it
crap. i did catch one "nest/next" scanno, but forgot to check the rest of the file for another.
RF: a thing to work for that being president BB: a tiling to work for that being president
i'd guess a check for "tiling" would be a worthy one.
RF: the back. Mary Emma could not go with BB: the back. Mary Emma could hot go with
"could hot" and "can hot" checks will surely be worthy.
RF: problems. From Lucia's manner, she BB: problems. From Lucia's manlier, she
a "manlier" check might well be worthy.
RF: of the page and below was a brief resume BB: of the page and below war; a brief resume
a check for "war" might be good, or might give false alarms.
RF: I'm the crossest girl you ever saw, so far as mere looks BB: I'm the Grossest girl you ever saw, so far as mere looks
shoulda caught this, as an unexpected mid-sentence cap. *** spelling discrepancies -- these are just because i made mistakes
RF: of those still, quiet stiletto exchanges BB: of those still, quiet stilletto
i probably gave this the o.k. because i don't know how to spell it.
RF: tonsillitis. Betty saw her and overheard BB: tonsilitis. Betty saw her and overheard
i probably gave this the o.k. because i don't know how to spell it. *** splotches -- not much i can do about these
RF: packed a thin chiffon dress, while BB: packed, a thin chiffon dress, while
can't see how i could ever devise a test to catch that.
RF: this, Miss Betty Lee!" BB: this,' Miss Betty Lee!"
didn't do a check of the balancing of single-quotemarks. *** missed italics -- not much i can do about these
RF: wouldn't do _one thing_. She is sweet BB: wouldn't do one thing. She is sweet
i don't know how to code a test for italics.
RF: other times too, but _always_ then, BB: other times too, but always then, before
i suck at finding italics. i missed 40 cases, not? *** missing quote marks -- my tests should've caught these
RF: won't you?" BB: won't you?
shoulda found this, unless both quotemarks were missing.
RF: who sat down. "How is your mother BB: who sat down. How is your mother
shoulda found this, unless both quotemarks were missing. *** extra quote marks -- an ironic coincidence of splotches
RF: little habit of dropping in when BB: little habit of 'dropping in' when
two splotches happened to do something that made sense. *** levenshtein check -- i should include this in my process
RF: are the Sevillas and where do they live? BB: are the Savillas and where do they live?
i got so confused on this name, but thought i checked 'em all. *** guiguts-catchable errors -- i just plain forgot these tests
RF: sometimes! I can't study! Come over here BB: sometimes! I can't study I Come over
shoulda caught this, as an unexpected mid-sentence cap.
RF: who reads the sport page." BB: who reads the! sport page."
shoulda caught this, as an unexpected sentence-starting lower.
RF: know." BB: know," (at end of paragraph)
shoulda caught this, as an improper paragraph-termination. *** a bug in BB's generator -- not "a bug"; indicates a continued quote
RF: like my residence here. BB: like my residence here." "
not a bug. will be deleted before the product goes "final".