- gutvol-d - lists.pglaf.org

why isn't distributed proofreaders helping fadedpage?
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

why isn't distributed proofreaders helping fadedpage? the site needs proofers, and d.p. has an _excess_, to the degree that they are actively attempting to stunt the work being done by the p1 volunteers... so why not send some people over to fadedpage? rfrank has done a lot for distributed proofreaders. and fadedpage is showing d.p. how they can make a roundless system work, and that it _does_ work... so why is d.p. being so stingy? help your brother! -bowerbird p.s. and to start, some of you people who are now working at fadedpage can occasionally make a post in a forum at d.p. inviting people to try fadedpage...

1 0

how to create a spellcheck workflow
by Bowerbird＠aol.com 08 Apr '10

08 Apr '10

ok, well gee... i had gotten the firm impression that rfrank had read the d.p. forum thread where the development of their "wordcheck" was discussed (thoroughly -- 30 pages). d.p. calls it "wordcheck", but it's basically spellcheck. so why did i think he had read it? because he incorporated several of the things which i had suggested (to no good effect) in that thread... i won't bother reciting all of those particulars, since he might well have come up with the ideas himself, and nobody really cares anyway, not even me... but i mention it because now i have been convinced that roger must _not_ have read that thread, based on some current discussions on his fadedpage site... or, if he did read it, he didn't "get it", but given those current discussions, he might now be more receptive. so i will run through a quick little "how-to" refresher on how to design and build a spellcheck functionality, and incorporate it into the overall workflow of a book, so roger has the benefit of my wisdom here... ;+) as you'll see, this will hit a very wide variety of topics. note that what we're talking about here is _primarily_ executed during _preprocessing_, but there are some follow-on thoughts that apply to the proofing as well. *** 0. set o.c.r. parameters correctly. don't dehyphenate! name your files wisely. look both ways before crossing. 1. the first thing you should do with your o.c.r. results is to make the few global changes you can do _blindly_. the very first one is to strip trailing spaces from all lines. another will be to replace two spaces with a single space. yet another is to change a linebreak-doublequote-space combination to eliminate the space, since it's superfluous. likewise, change all cases of space-doublequote-linebreak. you get the drift. oh, by the way, don't do what d.p. does! _retain_ the runheads, and pagenumbers; you need 'em... 2. the next thing you should do is clean up the runheads. this has little bearing on our general "spellcheck" topic, but i include it here because it's always your second step. 3. the third thing on your list is to fix all paragraphing... again, not much to do with spellcheck, but it _is_ step #3. 4. now we can focus quite specifically on spellcheck stuff. we will take the o.c.r. and run it through a program i wrote which pulls out all the words _not_ present in its dictionary. i think rfrank has his own program that does the same thing, or something similar enough. i'll make mine available too. this is the first draft of your "bad-words" list for this book. (note this is not the same as how d.p. defines "bad words".) in this regard, use a good dictionary in this first check here. (this is something that rfrank hasn't done correctly thus far.) the dictionary i use is quite good, and it can be found here: > http://z-m-l.com/go/regulardictionary.txt just so you have a feel for the output from this program, i have posted it for the "sitka" book we've been discussing: > http://z-m-l.com/go/sitka/sitka-reversedictionary.txt that output was generated in 5 seconds, so it's pretty fast... 5. you'll see, on viewing your list of supposed "bad-words", that a bunch are not "bad-words" at all. some of 'em will be character-names, or jargon specific to your particular book. some will be hyphenated fragments, some compound-words. you can delete these words from the list now, if you want, but you shouldn't necessarily feel a great need to do that. if you're looking at the list of words from the "sitka" book, you will also notice that i separated the initial-caps words from all-lowercase ones. there is a good reason for that. due to proper names, _most_ initial-cap words are correct, whereas most of the all-lowercase words are _incorrect_... separating the lists makes it easier to focus your attention. 6. your dictionary-check program should also spit out the _frequency_ of each bad-word. you'll use that information to cull some words from this list of "bad-words". this you _will_ want to do, most definitely, so _sort_ on frequency. i fought tooth-and-nail on this with d.p. (i lost, of course), but you can take this to the bank that as long as there are a mere 4-plus occurrences of a specific string in the o.c.r., you can (i.e., should) delete it from this list of "bad-words". yes, some of the words _might_indeed_ be bad, but unless you're positive of that, you should delete 'em from the list. and yes, this means that those words will _not_ be flagged. but trust me, if there's 4 or more occurrences of a scanno, your proofers will find at least _one_ . and whenever they find _one_ "bad-word" that wasn't on that "bad-word" list, you will automatically search the rest of the book, and thus find those _other_ occurrences as well, so you can fix them. i did _not_ include frequency information in my "sitka" list, because i didn't want to make everything so bloody obvious, and because i want you to discover the importance of that frequency data for yourself, so it burns itself in your brain. 7. when you narrow your focus to the words that are _not_ in the dictionary, and which occur only two or three times in the book, you'll find you can be very productive fixing errors. for many of the words, it'll be obvious what they should be... building a tool that will take you _immediately_ to each word, plus show the scan alongside, will turn you into a _machine,_ an awesome and devastatingly efficient error-fixing machine. for this very first pass, i recommend you look only at words you're confident are scanning errors. (they're easy to spot.) on your next pass, you can look at more questionable words. also pay attention to words with several variants that'll thus sort next to each other. (see the asterisks in the "sitka" list.) it'll almost certainly be the case that one variant is a scanno. (97 times out of 100, it is the one with fewer occurrences.) also, in a system where you're gonna have proofers doing a word-by-word proofing, don't even bother to look at words which look kinda reasonable. and don't ever bother to view any words with only _one_ occurrence. those will be flagged, and it's no less efficient to have the proofer see if it's correct than to have _you_ see if it's correct. you wanna be efficient in preprocessing. efficiency is _the_point_ of preprocessing! 8. the next thing you want to look at are compound-words. my tool also separates compound-words to their own listing. as per usual, perusal of the compound-words will show that some are obviously correct, others are obviously incorrect, and a bunch where a judgment can only happen with a scan. the other thing about compound-words, which you will want your tools to handle, is a check against the rest of the book, to find any other instances of the compound where the parts are separated by a space (two words) or joined (one word)... that information will help you decide how to treat the word. 9. the other thing you're gonna check is end-line hyphenates. remember that i told you _not_ to have the o.c.r. rejoin them. my philosophy is to _retain_ the end-line hyphenates through my final product, but i'm not arguing for that position _here_. you can rejoin the end-line hyphenates if you want to do that. just don't do it until _after_ all of the proofing is done, because having original linebreaks makes a page much easier to proof. and besides, in determining whether or not the rejoined word contains a dash or not, we need to have uncompromised data. if your o.c.r. program destroys that data (by joining the word in the way that _its_ dictionary dictates), then you might just be doing a disservice to the way that the _book_ did things... for your spellcheck, however, you can ignore all of that stuff. internal to your spellcheck tool, rejoin the end-line hyphenate by eliminating the linebreak, and test the resultant compound. if it passes, fine. if not, try again with the dash removed too. if it _still_ doesn't pass, flag _both_ portions of the compound. (note that if you are using _my_ dictionary, mentioned above, it contains no compounds, so you would skip that first check.) 10. so at this point in time, you have a great "bad-words" list. that is half your battle. (that's right, just _half_.) so now you will make your "good-words" list. to do this, just run the text through your dictionary-checker using your "bad-words" list as the dictionary. thus, the output will be all of the words in your text which are _not_ included on your "bad-words" list. (or you can just use my tool, which can also create this list.) from now on, this "good-words" list will be your dictionary... got that? you're not using the huge dictionary file any more. you'll use the much-more-compact "good-words" list instead. so now you have a "bad-words" list and a "good-words" list, which -- taken together -- comprise all words in your book. there are other jobs that you'll do during preprocessing, but this is all the spellcheck work that's needed during that stage. 11. so now we will move on from preprocessing to proofing, which takes us to "flagging" -- highlighting possible errors... a word should be flagged if it appears in the "bad-words" list. a word _might_ be flagged if it's not on the "good-words" list, perhaps in a different way; for instance, yellow instead of red. notice that this is a slightly more nuanced way to do flagging. rather than flag _everything_ that _might_ be wrong, we are gonna flag _only_ the things we really _suspect_ are wrong... underflagging is better than overflagging, because too many flags makes us complacent; we start to check only the flags. it's impossible for your mind to ignore the fact that most of the flags are not actually errors, so it comes to expect that if something is _not_ flagged, it certainly won't be an error. but if you underflag, and the proofer spots an error that was _not_ flagged, it primes them to be attentive to everything... 12. the other thing that's extremely important here is that, when the book is finished, we should have resolved all flags. every word in the book will be there on the "good-word" list, and the "bad-word" list will have shrunk until it disappeared. that is, every bad-word will have been checked, and if it was "ok", it will have been moved to the "good-word" list, and if it was not "ok", it would have been _changed_, which will also remove it from the bad-words list. the words do _not_ have to be physically removed from the "bad-words" list, but if we check every word in the book on the "good-words" list, and find it fine, then the "bad-words" list will be eliminated. this complete "good-words" list is _useful,_ because we can run the full book against spellcheck at any time, and it will come out totally clean. so we do that check periodically, so we know that we haven't compromised the book's accuracy. oh, and just so you'll know, it's quite easy to write the code that does this check. you simply sort the words in the book, eliminating duplicates; then you sort the "good-words" list (if it's not already sorted), and eliminate its duplicates too (shouldn't be any); then the 2 outputs should be _identical._ this lets us envision the proofing process as movement of all words on the "bad-words" list to the "good-words" list. (put that image in your head; the visualization has utility.) to help facilitate that movement, you need to make it easy for proofers to put words on the "good-words" list, which is why -- on my proofing site -- i let them add all the words for a single page to the "good-words" list with one button-click. (another option is a button-click for each individual word.) the flip-side is that, in order to have a page be considered as "finished", all flagged words on that page _must_ be cleared. remember, words move from "bad-words" to "good-words". that's good enough for now. any questions on this? ;+) -bowerbird

2 1

Re: the sitka book from fadedpage is now posted
by Bowerbird＠aol.com 07 Apr '10

07 Apr '10

dakretz said: > This comes up all the time. it has been suggested before, yes... > a. It's socially unacceptable by acclamation. ...and is usually deemed socially unacceptable, true... but i'm not sure that that couldn't be turned around, assuming people are genuinely looking to be tested, in a sincere desire to improve. it is a bit repugnant, in my opinion... but so is the secretive collection of data on proofing accuracy that rfrank is doing now, especially since he's not revealing that to _anyone_, including the very people he is collecting data on... > b. All you prove is that the proofer did or > didn't catch an error you already knew about. well, i don't necessarily agree with that. i'd believe your ability to catch the introduced errors would be highly correlated with your general overall accuracy. so if you value that metric, there's one good purpose. (but, for the record, i believe that metric is valueless.) further, the ability to detect the error _immediately_, and show it to the proofer on-the-spot might well be the very best feedback necessary to get their attention, an argument which hasn't been fully considered before. so there _could_ be some real value in this technique. now, i certainly wouldn't do such error-injection on a proofer without their express approval, because it is entirely too _sneaky_ when you are doing it that way, and jeopardizes the trust-relationship, which is vital. but if a proofer _asked_ for it, i think it would be ok... i never subscribed to the "i need to have some errors to keep myself from betting bored" philosophy... but for a proofer who does, this could well be an answer. bottom line, though, i just don't think it's necessary... -bowerbird

2 1

Re: : the sitka book from fadedpage is now posted
by Bowerbird＠aol.com 07 Apr '10

07 Apr '10

rfrank has responded, over on the fadedpage forums, to the note i sent yesterday on the posted "sitka" book. as i might have expected, he fell on his sword and took full blame for the errors that were present... in some part, that's correct, because some errors _had_ been reported, and he simply failed to check the forum. but that's not really the proper take-away message here. a proofer can miss things, and still hit the "done" button, just because they honestly felt they'd caught everything. even the best proofer, with an accurate self-perception. 2 of the 6 errors that i reported were cases just like that. if the only person who's gonna serve as "back-up" for that proofer is the post-processor, that puts too much stress on the post-processor. that's exactly what d.p. has done, and that's why they have so few volunteers for that task... rfrank is willing to take on that stress, which is why he has fallen on his sword. but that's the wrong approach to take. rfrank is also improving his tools, so they catch those glitches. that's good, and it's part of the reason i reported these errors. plus he'd already improved his workflow, to catch _italics_... his improved workflow might work, _if_ the o.c.r. recognizes the styling correctly. but on any books with heavy formatting, going back and reinserting the formatting might be a real pain. a better approach, in my view, would be the one that rfrank started with, when he put up his site, which is to encourage the volunteers to do _both_ the proofing and the formatting. it's really _not_ that difficult to do both these tasks together. this is especially true if you give people a _formatted_display_, because then the obtrusive markup is cleared from the screen, and it's replaced by a rendering that resembles the actual scan. i demonstrated this technique with my own proofing site, and showed the additional strength that questionable words can be highlighted in a different color, maximizing the value of a flag. *** rfrank said: > But the most important thing I've concluded is that > the majority of reportable errors in Sitka are chargeable > to the post-processor (me) and not the roundless system. see, there's the "experimenter bias" that i was talking about; he'd rather take the blame himself than blame his system... i believe in the roundless system too. i believe in it so much, so strongly, that i believe the evidence can stand up for itself. > We don't have nearly enough participation by a realistic > cross-section of typical users here at fadedpage to > conclude anything based on real science or real statistics. > All I can say is that I am liking the roundless system more and > more. It seems to be doing its part well and continues to improve. there isn't a "cross-section" of "typical" proofers over at fadedpage, it's true. the proofers there are probably much better than average. and, as shown, even these better-than-average proofers can _miss_ errors. we're all human. we make mistakes. we need to be checked. -bowerbird

4 4

the sitka book from fadedpage is now posted
by Bowerbird＠aol.com 05 Apr '10

05 Apr '10

the sitka book, which was produced over at fadedpage.com, was posted over the weekend, so i will finish my version and do a comparison on it. the fadedpage/p.g. version is here: > http://www.gutenberg.org/files/31862/31862.txt i can, however, already report that the posted version _does_ contain a few errors. from page 19 in the original p-book: > http://z-m-l.com/go/sitka/sitkap019.png "yukali" is incorrectly entered as "yuhali", and "prazdnik" is incorrectly given as "prasdnik". in addition, both words are italicized in the original, but unitalicized in the p.g. version, for a grand total of 4 errors on this single page. (troubling is the fact that "yukali" had another occurrence in the book, and the similarity of these two non-dictionary words in the book's word-list _should've_ triggered closer examination.) ironically, there was a note in the forum about this page, saying that it was marked as "done" incorrectly after _no_ proofing, due to an inadvertent click on a wrong button, a note which also then pointed out these changes which needed to be done, but evidently rfrank missed that note. and with nobody else doing that page, the errors persisted. plus there are a few other errors i have already noticed... the word "sheetkah", on page 25, is missing its italics: > http://z-m-l.com/go/sitka/sitkap025.png and the period after "december 14th" should be a comma, in the footnote (#25) which was on page 80 in the p-book: > http://z-m-l.com/go/sitka/sitkap080.png so there are some things worth pointing out about all of this... the first is that, by _my_ standard of one-error-per-10-pages, these 6 errors are _not_ a damning indictment, not nearly so... in and of themselves, these errors are trivial. don't forget that. of course, you don't like to concentrate four errors on one page. but stuff happens. so in terms of this particular case, for _me_, i don't think it's a big deal. however, in terms of the _workflow_, the fact that a page with 4 errors can float through the system without _anyone_ having had a "second look" at it -- or, indeed, even a _first_ look -- is not a good sign, not a good sign at all. moreover, according to clear results of a poll i ran over at d.p., the majority of people over there believe that 5 errors in a whole _book_ is right at the maximum that they are willing to tolerate as "acceptable". even worse, some of them -- in a defiant act of wishful thinking -- actually have convinced themselves that they really do _attain_ that level of accuracy with the books they do! now, clearly that's ridiculous, and they're just fooling themselves; they don't actually _know_ how many errors are in their books, so they just let themselves believe that there aren't any errors there. but in spite of this break from reality, their _expressed_desire_ is that they release books that have 5 or fewer errors in them... viewed from that perspective, this performance -- with 4 errors on a single page and (at least) 6 in a posted book -- is _bad_... -bowerbird p.s. i'd point to the note that was left about that page, except that rfrank seems to take down the thread for a book once it is posted... once again, this data is important for a proper analysis of the test.

1 0

Full text catalog search
by Joyce Wilson 05 Apr '10

05 Apr '10

The "Full Text" search option here <http://www.gutenberg.org/catalog/world/search> is broken, and has been for at least a year. If the thinking is that it doesn't need to be fixed because some of the alternative searches at the bottom of this page <http://www.gutenberg.org/catalog/> are acceptable substitutes, then it makes sense to remove it as an option on the "Advanced Search" page. To have the option there but never find any results with it is just confusing. Regards, Joyce

1 0

Re: a horizontal proofing interface, line by line
by Bowerbird＠aol.com 03 Apr '10

03 Apr '10

jim said: > suggest you keep working in this direction > as I think it would make a contribution. well, yes and no. it's good for doing a word-by-word proofing... but, as i've shown, that methodology is overkill, at least when it is practiced over an entire book that was properly and aggressively preprocessed, as upwards of 90% of the lines are already perfect. it is good at pulling out the other 10% of the lines, however, and subjecting them to a closer scrutiny. plus it does have utility in doing a _comparison_ between two files, as is shown in this screenshot: > http://z-m-l.com/go/sitka/findlines.png that screenshot also gives you some hints about how i go about slicing the lines, although that's not really too difficult to figure out on your own. -bowerbird p.s. the pinkish lines in that screenshot are lines that have a diff between the two different versions. the numbers by the asterisks are a simple counter for the number of diffs cumulative to that point...

1 0

re: a horizontal proofing interface, line by line
by Bowerbird＠aol.com 03 Apr '10

03 Apr '10

jim said: > Do you have software to automatically > slice the bitmap and align the txt? > Or are you slicing the bitmap by hand? i don't do anything by hand, except pheed my phat phace, wave to my neighbors as i drive the car around our streets, pet my cat and dog, and pleasure myself and loved ones... *** juliet said: > Implementing this kind of line-by-line interface > efficiently is only possible if one has > word boundary information (actually, the line boundary) > from the OCR. That kind of information was not available > when DP started and to add it into the current DP would > be so much effort as to make it out of the question. my software doesn't need any such information from o.c.r. *** dakretz said: > I would rather have a scrolling semi-transparent > gun-slit overlay on the image that would synchronize with > the cursor in the text. I can't proof with so little context. i'm confused. the whole page of context is there, both in the form of the (sliced) image and the (sliced) text... but please, take the text and images i've provided and provide a demo of the system that you have described. > http://z-m-l.com/go/sitka/sitkap002.txt > http://z-m-l.com/go/sitka/sitkap002.png > http://z-m-l.com/go/lines -bowerbird

3 2

a horizontal proofing interface, line by line
by Bowerbird＠aol.com 02 Apr '10

02 Apr '10

here's a take on a horizontal proofing interface, where the scan is sliced up into individual lines: > http://z-m-l.com/go/lines/pagelines.html might be good if you use an iphone, for instance. -bowerbird

6 6

Re: jim replies to his e-mail
by Bowerbird＠aol.com 02 Apr '10

02 Apr '10

keith said: > I use the link given and I personally did not like > the layout or interface, but then again it just me. saying you didn't like it is different from saying that "it's broken". but not that much more informative... > To be honest I could not make heads or tails > of what I was seeing or exactly what to do. ok, that's a little bit more clear. or maybe it's not. i think most people would know that you make fixes in the edit-field, so as to match the scan. but maybe you need to be familiar with some other proofing systems in order to use that experience... at any rate, yes, training will certainly be available. thanks for the feedback. :+) -bowerbird

1 0