December 2011 - gutvol-d - lists.pglaf.org

pontifications from mount high horse -- #1496
by Bowerbird＠aol.com 28 Dec '11

28 Dec '11

here is a jumble of responses to various things that were said recently. i think it's got a pretty good flow, so i haven't bothered to put it in chronological order. but if you think that misrepresents anything, say so... *** don said: > I've scanned and reviewed the proofing on a lot of books, > all ABBYY-OCR'ed, and one error per 10 pages exhibits > good book selection Abbyy management skills I wish I had, > or maybe books with 12 lines per page. i was talking about _after_ the preprocessing, not before... raw o.c.r. has a typical rate of about 1 error per page on the relatively clean and simple and straightforward books. on a simple book, like the "betty lee" that roger was using, you can fix about 90% of those errors in roughly one hour. (assuming you have a good tool, which ain't hard to build.) another hour (or two) will fix 80% of the remaining errors, meaning you are left with ~2% of the original o.c.r. errors. at that point, the wise course of action is to send the book to the smoothreaders, and count on them to find the rest... for the encyclopedia stuff you do, don, it takes more work. more time, and more passes, and less accuracy at the end. how much, i can't say, because i don't have the experience. > I'm pretty sure we could do better work, quicker, if we > proofed in parallel, then compared (and merged) results i don't think the "serial/parallel" distinction matters much. as long as the work of both individuals gets incorporated -- which it does, in both -- i can see no quality difference. as for parallel being "quicker", it will be, _while_proofing,_ since person "a" can start at the same time as person "1". but once you add in the time for "comparison and merge", you've likely lost any advantage you had, and maybe more. i don't think anyone here has summoned up more evidence for the value of comparison/merge than i have, but if you are arguing that it's quicker and better, i probably disagree. (most of my research involved a pre-existing digitization, meaning that there was _no_ cost to generate that corpus. and sure, in _that_ type of situation, it is clearly "quicker".) *** roger said: > there are several more regex checks that > would have helped and I'll bake those in. a reg-ex path is almost certainly the wrong way to go. yeah, a small number are necessary and efficient, but as you get more and more, their signal-to-noise ratio starts to rot, and you waste your time on false alarms. the most effective means is also the easiest one of all, namely a straightforward spellcheck. i looked closely, and documented it carefully and extensively on this list, so i can tell you spellcheck does most of what you need. where reg-ex shines is with punctuation abnormalities. but if you're having those kinds of problems, then you will need to have help from smoothreaders _anyway_... *** don said: > Are you trying to proof and format > at the same time? I did that experiment > several years ago with the same effect. sadly, it will take a long time to erase the d.p. myths... the reason proofing and formatting do not get along at d.p. is because their system mixes pseudo-markup with plain-text which is shown in an .html text-area... the pseudo-markup _interferes_ with the plain-text -- since, face it, markup is inherently obstructive -- so yes, that makes "proofing-while-formatting" hard. but also causing the difficulty is the simple fact that plain-text does not resemble the p-book page-scan. the font is monospaced... also, the paragraphs are (a) not indented, and (b) separated by a blank line... italics are not displayed. blockquotes aren't set off. headings aren't centered. poetry is formatted right. in short, there is _nothing_ about their interface that aids in the task of _either_ proofing _or_ formatting! so if you try to do _both_, plus deal with the obstacles of in-line markup, you're really fighting a losing battle. and the answer is as obvious as the nose on your face: > http://z-m-l.com/go/triple2010.png look at the middle pane. it looks similar to the scan. it's not an "exact" match, but it's getting pretty close. the paragraphs _look_like_ the ones on the page-scan. the blocks of text between the two match quite nicely. the fonts are fairly equivalent, the leading is identical, there are curly-quotes, matching the ones on the scan. and the italics, if there were any, would _be_ italicized. so the only real discrepancy is the lack of justification. with this interface, you can indeed proof-and-format at the same time, and actually succeed. at least i can. i think results would be better than the d.p. interface. because there's no drag on proofing-and-formatting. that doesn't mean i think it's a good idea to do both... as i said earlier, it's important to keep a laser-focus... but a person _could_ do both, if they really wanted to. so yes, don, i agree with you that one shouldn't try to proof-and-format at the same time. but _not_ due to the "it can't be done" reasoning that you have implied, a message that's been promulgated by d.p. ineptitude. d.p. just uses a bad interface that makes it difficult... *** roger said: > Yes, because removing formatting > only to have to put it back in > seems counterproductive, > especially since Abbyy's output > has a higher catch percentage > than I believe a foofer does. well, ok, first of all, part of your reasoning is based on _your_ failure on this task, and i don't think your performance is too useful. as far as i know, you do too little formatting on d.p. to be _experienced_ at it... and yes, spotting italics is a task that takes practice... i'd also expect that, having been humbled, you'd be careful next time, and thus better. so before too long at all, you'd be great at it. (even so, even the great ones do miss some.) to be sure, i am _not_ saying that it helps to "try harder" or "be careful" or "pay attention". to me, it seems like explicit consciousness might well hinder the task, not facilitate it... spotting italics might be a right-brain thing; i've noticed that sometimes i will view a page, and "see nothing", so i click to the next page, but then i "have a feeling" that i missed one, so i'll turn back, and sure enough, there it is. so, like i said, it feels like a right-brain thing. and thus it might be better to be "unfocused". you can just look at a page and get a gestalt that tells you if the slant of an italic is there. whatever the case, it's definitely the situation that you get better the more often you do it. further, when i was doing it frequently, i got _faster_ at it, even as my accuracy improved. (so i tried to force myself to go even faster, but it didn't work. had to happen naturally.) but anyway, now i am out of practice again. having said all of that, however, i recommend that people use the .rtf version, since it does do a fairly good job these days on the italics... you still need to check, though, so on a book like this one, with easy and infrequent italics, it won't matter too much if you do .rtf or not. it will be the same amount of work either way. but on a book that has tons of difficult italics, it's well worth running a test to see how well abbyy recognizes them, because it could save a lot of time and energy, and boost accuracy. again, i am out of the habit these days, but i went through the whole book, to see how accurate i would be, and how long it takes. i spent an hour doing it, and found only 110 or 120 of the 160 italics you said are there. (that includes the ones i had marked earlier, whenever i encountered 'em "along the way".) so that's an accuracy-rate of a mere 70%-75%. even with another go, i might not find 'em all. so, from my perspective, it's a no-brainer to at least let abbyy _try_ to find them for me... moreover, i'd defer this task to collaborators if i was working in a distributed environment. > And having two foofers go through > each page of a book like this > seems to be a waste of resources. well, that's a separate question. it depends on how much energy you are willing to expend to _guarantee_ that you've got an error-free book. personally, i don't think italics are _that_ vital; nonetheless, i wouldn't want to miss more than a half-dozen in a 240-page book like this one. and i cannot say, without doing some research, how good smoothreaders are at catching these. (although i hasten to point out that, unlike d.p., my smoothreaders view text and scan together, so it'd be easier for them to detect-and-check.) heck, to be honest, i can't even say that i know what kind of accuracy the d.p. formatters have. i just assume their 2 rounds must be adequate. but i don't really have _research_ to back it up. nobody does. it's too hard to create a criterion. *** don said: > It will be interesting to see > what your testing reveals. > I remember that, at the time, > abbyy's accuracy with formatting > location and boundaries was not good, > and that one could spend more time > fixing what's there than to > put it there in the first place. gosh, more b.s. posing as "accumulated wisdom" from that flaccid-fountain-of-fail known as d.p. here's the truth, rather than a recycled d.p. myth. abbyy is now _pretty_good_ at recognizing italics. even back in the day, when this myth took shape, abbyy did a _fair_ job, if you gave it decent scans. the problem was that some people at d.p. threw crap scans at abbyy, then said "it doesn't work"... of course o.c.r. isn't gonna give you good results from a scan that looks like your baby kutzed on it! it's optical character recognition, not mind-reading! on some of those scans, _i_ couldn't read the type. so how could anyone expect o.c.r. to do the trick? i am continually amazed at how _well_ abbyy does. even on the current "betty lee" book, if you look at the places where there were glitches in the o.c.r., then look at the corresponding place on the scan, you usually see the splotch that caused the glitch. this scan-set was not up to the usual quality level for scans done by roger. must've been a bad copy. and here's another tip which many of you know: use scan-tailor to clean up your scans for o.c.r. (don knows -- as he always advises it to others.) i don't know all the ins-and-outs of scan-tailor, so i can't give any advice on optimizing results, but it proved itself during my initial experiments. so do a test-run of a few scans to see if your o.c.r. gets good results. if not, do mods with scan-tailor. > Also consider how the abbyy formatting data > could be used by software after your proofing, > to consider each location where abbyy found > formatting and whether you had put formatting > there; and resolve the differences efficiently. now _this_ is a good suggestion. myself, i would use the abby italic information right off the bat, without any hesitation at all... but if someone is reluctant to do that, they can certainly use that information _after_the_fact_. for instance, as i reported above, i located 113-119 places where "betty lee" has italics. (119 places, to be exact, found in 113 lines.) so i've appended a list of those lines with italics. a similar list can be generated from abbyy info, and the two lists then compared to test for diffs. the diff results could also be used for _training_, i.e., i could view lines where i missed the italics, and perhaps pick up a clue on how to improve... *** don said: > The best way to get past the tedium is to come up > with an objective means for comparing results > derived from different, and differences in, the process. doesn't seem like that should be too difficult... > Until then we're stuck with our prejudices. doesn't seem like a very appealing alternative... -bowerbird p.s. cases of italics that i found in "betty lee"... > to have _everything_ talked over in the family! I > "True enough. But you don't have to tell _all_ > Lucia (pronounced 'Lu _chee_ a,' in Italian > and as for her she was going to _choose_ her > I _am_ expecting it to put them to flight! I thank > most of them fine, and both as _worthy_ and as > _interested_ in their own team's winning. Do not > haven't lost it. Oh, wouldn't that be _awful?"_ > Mother's that has _grand_ notes in it and I use it > "Oh, _do_ you?" laughed Betty. "This is so > "He gives the _full_ address inside. It's > haps just to joke me -- and while I _thought_ that > other things, I never found out till the _middle_of_ > _my_sophomore_year_ that junior orchestra only > meant _second_ to the senior orchestra, sort of a > Probably it _was_ "nerve," but she had not meant > Clara probably thought that Betty was _gloating_ > enough tact to be president of _anything"_ > lunch room side by side. "I _thought_ it was > Woods, rather proudly. "As soon as _he_ gets > stuff _ing._ But I'd never get tired of it and I > "She'll copy Lucia, and it will do _her_ good to > dent of Lyon 'Y', _you_ would be going home > _junior_ for president!" > Come over here on the _chaise_longue_ and let me > broader than the ordinary _chaise_longue._ Golden > "If you only understood Italian, Betty! _Che_ > _peccato!_ That means 'What a pity' -- for I'll > he is on an African _safari;_ he wrote me just > "I _casually_mention_ hearing from my father > mother. She doesn't care a _centime_ for him; > stuff that I'm most scared for fear she _will_ start > him and tell him just to come over and _get_ her!" > Betty. I believe I _will_ write and tell my father > _were_ connected with Ramon -- and she had > everybody could select her own doll. And _these_ > _"I_ wouldn't care for such a friend," said Dotty. > I _knew,_ I should scarcely give any information > that died. Isn't it _pitiful?_ So I just sent Bessie > the kind that _she_ has to _help."_ > the kind that _she_ has to _help."_ > _rather_ you wouldn't tell Mother. I don't know > changed, provided she _could_ change it now. She > at first and they _are_ good folks, Betty -- at least > They were so _good_ to me! I'm going to give > in the _little_ basket till Christmas morning, Mrs. > chance that _he_ may get work the first of the > sang Latin hymns and songs, the _Adeste_Fidelis,_ > "It's a shame you aren't going to _your_ grand- > to _make_anything,_ I don't see, but I'll appre- > _never_ forget you. Couldn't if I tried!" > called it in the old days when _she_ was a little > _"We_ read things," importantly said Amy > "Funny she wants _you_ Mumsy, when she has > in trouble. Well, Lucia wanted _her;_ perhaps > "Did you see that _tr-ragic_ face, Betty?" asked > _your_ side," she told Carolyn Gwynne. > "It wasn't _anything!_ I got knocked down and > "We certainly could, and _crede_mihi,_ Betty, > to do, and you can't do everything, _crede_mihi!"_ > _"'Crede_mihi'_ -- I can't," laughed Betty. "Do > you suppose _'mihi'_ ought to come before > _'crede?'_ Oh, yes, imperative first!" > _"'O_tempora,_o_mores!'"_ replied Carolyn, > my word for it' isn't enough like 'believe _me'_ > like _ubi_est?_ or _Quid_loquor?_ > like _ubi_est?_ or _Quid_loquor?_ > _Quid_agis?_--_O_miserabile_me!_--_horribile_ > _Quid_agis?_--_O_miserabile_me!_--_horribile_ > _dictu_--_age_vero_--_da_operam,_ and other expres- > centrated on basketball "to beat _us."_ And beat > think that he owned her! She _liked_ Jack. > couldn't help it. Jack _must_ be all right! Why, > Betty had wondered if her own parents _could_ > where she was. It _was_ nonsense. She would go > home when she got ready. But she _would_ > near dinner time it is, and Mother _will_ be > "Oh, you _did!"_ exclaimed Doris. "Tell us > facetious about it. Isn't that so, _mon_cher_ > _papa?"_" "" > _wild_set_ of the _society_bunch!_ What do you > _wild_set_ of the _society_bunch!_ What do you > "Don't _ask_ me, Doris. I don't like _any_ of > "Don't _ask_ me, Doris. I don't like _any_ of > cally. "Would she _let_ me, do you suppose?" > her. Perhaps it was Chet's remark about _not_ > _forgetting_ that misled her! She was dressed, > if there _was_ a pretty good current in the river, > frowned again. _Could_ her father tell him? Then > "Oh, Father, you _told_ him! And I know he's > "Ramon Balinsky Sevilla is not _in_ Detroit!" > friend met him. _Now_ are you satisfied?" > new dress, _"so_ becoming," she said. "Betty, I > for it in others, and _have_a_care,_ Betty. That > _a_la_ victrola, though Jack apologized for not > de cook. Dat young lady in dere's done _passed_ > _out!_ An' de butleh -- he gone, too." > "No, honey, not daid. No, you jus' _keep_out._ > me, she would know the way around _this_ house > _Mother,_ was still up! > Then there _was_ something in the remarks > "I thought he _might_ call up to see if I had gotten > Betty was not quite sure just _how_ she could > think it was _terrible!"_" " " > cause _you_ didn't know what _was_ happening." > cause _you_ didn't know what _was_ happening." > just _saw_ going to pieces little by little and hav- > reach Italy and how he was having the _palazzo_ > opened in Milan. Now _that_ may mean some- > so _lonely,_ Mrs. Lee!" And your mother asked > _"Would_ you, Lucia? I wish you would stay. > to have a real old Italian _palazzo_ to come to > them the Sevillas. At least _he_ had not found > daughter -- _perhaps,_ Lucia thought. But the > perhaps, some day. I thank you for _trusting_ > annual Lyon High _Star._ It was the custom to > with her _Star_ under her arm, Lucia in the > thing. And she's _got_ to!" > not. Of course, we could not mention _that_ before

1 0

pontifications from mount high horse -- #1495
by Bowerbird＠aol.com 28 Dec '11

28 Dec '11

roger said: > I actually use a tool I've written (ppsmq) to > turn straight quotes in a text into curly quotes. it seems like you don't remember that i was the one who scoffed when you insisted this couldn't be done accurately. so i explained it very carefully. it was either here on this listserve, or in the d.p. forums, or probably both places, if you wanted me to locate that. anyway, i'm glad you finally saw the light. don said: > Double-quotes are an interesting case. > The editor WordPress interestingly automatically > converts them to curly-quotes without even asking, > and with great accuracy. gee, don, it seems like you don't remember this either. and yes, i had to explain all of this to you as well, don... and i did. so the "great accuracy" shouldn't surprise you. it is a very straightforward -- and simple -- task. really. > It even does a great job on single-quotes. that's a little bit more difficult. but only a very little bit. it's generally the same process. > I need to look at the algorithms when I get some time. so yes, you did forget. so yes, i will explain it once again... quotemarks on the left side of a line are opening quotemarks. quotemarks on the right side of a line are closing quotemarks. more generally, a quotemark which has whitespace to its _left_ is an opening quotemark. a closing quotemark has whitespace to its _right_. brackets, braces, and parens should be ignored. (note that any markup is another thing that should be ignored.) within a paragraph, the flow goes open/close/open/close, etc. so any "spacey" quotes within such a sequence are predictable. paragraphs span pagebreaks, so do not let that confuse you. mismatch means there's an error _somewhere_... could be a dropped or improperly-inserted quote, bad paragraphing, a doublequote o.c.r. misrecognized as singlequote (or a pair), a doublequote misrecognized as a number (e.g., 11, 44, 77), a mistake in the p-book, or some other freakish occurrence, but you can be sure there's some kind of error somewhere, and that you're gonna have to bring in a human to resolve it. a paragraph's final close-quote that's missing _can_be_ ok, but only if the following paragraph begins with a quotemark. in the case of singlequotes, the apostrophes in contractions must be dropped from the analysis, which is usually simple, as those apostrophes do not have whitespace on either side. however, the apostrophes which indicate _possessives_ are _sometimes_ at the end of a word, so that's a complication. you will also find that sometimes there is an apostrophe appearing at the beginning of a word which nonetheless should display with a closing-curl, a la "the roaring '20s". slang typography can also produce a myriad of wrinkles... keep in mind that all of these finer points i just discussed pale in significance to the fact that the big stuff is simple. most of the time, what you do will end up being error-free. and if there's a glitch, it will draw your attention naturally. the structures here follow fairly strict rules, so any collapse causes something else to fail, and that causes more failure, and so on, to the point where the flaw will become apparent. and your beta-readers, absorbing the content, are a last line. make sure they know that they're looking for this "petty" stuff, because you repaired all the "obvious" stuff a long time back... > one I'll solicit community improvement suggestions for. > It's the regex I use to find problematic quotes. ... > /([^\s\(-]?)"(\s*)([^\\]*?(\\.[^\\]*)*)(\s*)("|^\r)([^\s\)\.\,\?;:-]?)/gim reg-ex is the wrong way to go on this... just write the code... i've told you that before. and here i am, saying it once again. no wonder i'm getting bored repeating myself over and over. you should have learned this lesson 5 years ago. 5 years ago! if you do it the wrong way, of course it's gonna seem difficult. just do it the right way, and you'll discover that it's very easy... here's the python code to split the book into paragraphs: > theparagraphs=thebook.split("\n\n") then walk through the array of paragraphs, examining each. you just specify each quotemark as open, close, or spacey... if something doesn't fit -- e.g., it's supposed to be a "close", but it appears at a line-start, or has whitespace to its left -- you know that you have a glitch, which will need to be fixed. otherwise, you're good to go. do it, and stop wasting time... -bowerbird

1 0

pontifications from mount high horse -- #1494
by Bowerbird＠aol.com 28 Dec '11

28 Dec '11

by the way, if you want to get up to speed on this dialog and others like it, you need to familiarize yourself with don's "twisted". don, how about uploading the newest version?, and then letting everyone know where to get it. if you never use this tool, you'll never know it's cool, and it'll never be improved. show me you're not stupid... -bowerbird

1 0

pontifications from mount high horse -- #1493
by Bowerbird＠aol.com 28 Dec '11

28 Dec '11

roger said: > presenting the text to the user > one page at a time > fits on the computer's screen. i understand what you're trying to say. but... that's an after-the-fact rationalization. as evidenced by every p.g. presentation of an entire book on a single web-page, users are quite accustomed to scrolling. and there are times when it is the _best_ way to examine some particular aspect... for example, consider this specific display: > http://z-m-l.com/misc/doublequote-check.html that's an extremely useful presentation from the standpoint of checking quotes, which i think everyone will find obvious. there are two notable things on that page, both of them quite subtle, so bonus points to anyone who can point out either of them. -bowerbird

1 0

pontifications from mount high horse -- #1492
by Bowerbird＠aol.com 27 Dec '11

27 Dec '11

ok, i've written some posts that've been "on hold", with others still in the process of being composed, but i'll let this thought jump the queue for #1492. *** roger said: > I've come to the conclusion that > working at the page-level is sufficient but inefficient, and > that working at the book-level is efficient but insufficient. despite the fact that roger hasn't collected nearly enough data to be coming to conclusions, his statement is _fairly_ accurate, albeit highly trite and quite misguided. first, the page-level _is_ "inefficient"; however, it's necessary. further, the page-level is _not_ "sufficient; there are _plenty_ of decisions which simply cannot be made at the page-level... roger's done enough postprocessing that he can name a few. second, the book-level _is_ "efficient"; that's been my _point_ for literally _years_ now. i'm glad roger finally came around. but nobody has _ever_ claimed the book-level is "sufficient"; i've always said that "smoothreaders" are a vital component. these "beta readers" can catch stuff that even proofers miss, because they are reading the book for the _content_ within. and again, roger's well aware of their valuable contribution. so much for the "highly trite" constructive criticism. now let us go on to the "quite misguided" counterpart. roger said: > I concatenated all the text files first > and did the global corrections. > A lot of improvements were made quickly. > When I was convinced I had done all I could, > I burst that back into individual pages > and loaded it into PPE, the page-at-a-time editor > which presents the image, and edit window, > and an analysis window simultaneously. i find it quite interesting that roger feels the need to explain the exact characteristics of his "ppe" machine. perhaps roger didn't monitor this list 18 months ago, when i presented this screenshot of such an interface: > http://z-m-l.com/go/triple2010.png > http://z-m-l.com/go/triple2010.html (ironically, that screenshot was done as i was working on the "sitka" book, which had been scanned by roger.) that's dated march 29th of 2010. but the reason that it has a "2010" there in the filename is because i had uploaded an identical version back on july 7th, 2007: > http://z-m-l.com/go/triple.png > http://z-m-l.com/go/triple.html the 2010 version has more buttons, labeled as "buttons", because i have plenty of ideas for routines one could run, but note the 2007 version even has a dedicated button for every word which you might want to add to the dictionary, which is a nice touch, doncha think? i've been here a while. not that i claim any originality on such on interface, since the earliest o.c.r. program i ever used had a similar one... there are also lots of "obviousness" considerations here... i am only pointing out that roger is now fishing in a hole where i have already caught some nice trout for breakfast. but what's more important is that roger seems to believe that this interface belongs to the "page-at-a-time" mode. nothing could be further from the truth! i use this mode when i'm doing my "book-level" editing... roger seems to believe that book-level editing can only be done in a text-editor, using a concatenated text-file, and that it disallows any reference to the page-images... wrong! it _can_ be done that way, but not to its full effect, and you're always taking a risk if you make an edit _blindly_. (at times the risk is so minuscule that it would be stupid to waste any time "verifying" the change against the scan -- e.g., when you delete the spaces in front of commas -- but there's still a "risk" that any specific edit is incorrect.) so, like i said, for almost all the book-level editing i do, i use the very same tool that i use for page-level editing, an app that shows the text on one side, scan on the other. (it didn't take me long to abandon the 3-pane interface for my tools, since all machines don't have huge screens, though i still use it occasionally on my cinema-monitor.) so what does differentiate "book-level" from "page-level"? the way that you navigate the text-file is one factor. another is the amount of information you can access. starting with the second one, my book-level tools can access the entire book. so if you wanna know whether "car-fare" appears as "car fare" elsewhere in the book, a book-level tool will be able to inform you about that. how many times did "chet" mis-scan as "chef"? three. is "kamon" really "ramon", or another name? yes, no. "page-level" doesn't even let you ask such questions... "book-level" encourages them, and then answers them. but just as important, if not more, is the _navigation_, or how you approach the text, and handle the tasks... when i do something at "book-level", i do _that_thing_, and not any other thing. my attention is laser-focused. if i am correcting scannos, that's _all_ that i am doing... if i'm verifying paragraphing, that's the only thing i do. if i'm checking for italics, checking for italics is all i do. if i notice something else, out of the corner of my eye, i might fix it right away, _but_only_ if i won't lose focus! otherwise, i will just make a note, and continue on-task. because it is your _focus_ which makes book-level rock. i might be jumping all 'round the book, from page 16 to page 242, back to page 88, to page 7, to page 181, 211, 125, 39, and 123. and then i'm done with one task, so i go on to the next task, which is yet another whirlwind. but my attention is always solid, because i'm _focused_. contrast this with the "page-level" approach. the essence of a "page-by-page" method is that you do all the tasks for one page, and then go to the next page, where you'll do all the tasks for _that_ page, and so on... there's _no_focus_ on a task, none!, with page-by-page. that's why it's inefficient. it's also why it's error-prone. without an explicit checklist of "do this, that, and thus", a person is bound to forget to execute one task or other. heck, even when you have a laser-focus, you miss stuff! i did a focused check of the paragraphing in "betty lee", where that was the _only_ thing i looked at in the book, and i _still_ managed to make two mistakes nonetheless. that's why we need more than one set of eyes on a book. but if you fracture the focus of your proofers, you can have a half-dozen of 'em check a book, and still have bugs in it. anyway... one of my "on-hold" posts is an extensive treatment of book-level versus page-level, so i'll just finish that off and send it, rather than say anything more about it now. i just wanted to show that roger was very misguided here about what constitutes "book-level" versus "page-level". -bowerbird

1 0

Re: [gutvol-d] updated PP editor
by Bowerbird＠aol.com 26 Dec '11

26 Dec '11

ho ho ho! happy christmas eve! :+) ok, busy day for me, and i'm sure all of you too, so i'd guess you'll appreciate if i cut to the chase, and leave the extended discussion for next week. *** for now, i'll conditionally agree i missed 20 errors. but that number is kind of meaningless without also stating the number of errors that i _found_. if i found 2, while missing 20, that's extremely bad. if i found 2000, only missing 20, it's extraordinary. i would estimate that i found 80-200, meaning that my accuracy level was at the 80%-90% range, which is in the ballpark for a typical p1 proofer, _except_ i only worked for an hour, far less than a p1 round. moreover, i didn't perform the full range of checks. most significantly, i did not check any quotemarks. that's because i thought roger's preprocessing had probably _found_and_fixed_ all of those problems. however, since _7_ of them showed up as "missed", i decided that i should go back and do that check... more on the results of that check coming up later... *** the other info that roger needs to tell us about is how many errors that _he_ found and missed, and reveal how many "uniques" we each found... when we have those numbers for both proofings, we can then compute how many errors _remain_, which is the thing that we _really_ want to know, so we can decide if another proofing is "worth it". this is what don was talking about, except that the situation isn't nearly as vague as he believes. indeed, the specificity you can obtain from just a mere _handful_ of data can be quite startling, if you ensure that you select your data carefully. (this often means creating a test to produce it.) *** one of the things you should absolutely remember is that if you have found an error of a certain type, you _must_ check for more which might be lurking. i did that, with the errors of mine that roger listed. he found that i'd missed some "lie for he" scannos. and sure enough, a search turned up another one: > chance that lie may get work the first of the roger listed a paragraph-termination error i missed. and a search revealed another two or three more: > The World Syndicate Publishing Co, > be in it, Finny, I mean, > but were giving the Christmas hymn beginning, and, most notably for this book, there were also the double-quote errors roger discovered i'd missed.... the systematic search for those was quite interesting. *** the typographer for this p-book was badly hungover. the rate of errors in this p-book is surprisingly high, including an atypical number of outright misspellings. but the number of doublequote glitches is astounding. i have to verify them and write 'em up, but i'd guess that i found at least a dozen more, and maybe two... so -- at least in one sense -- you could say i missed even more errors than roger computed. but of course if i had done this doublequote check in the first place, and done it right, i wouldn't have missed any of these, as doublequotes _are_ a type of error that's detectable. so that check must definitely be a part of our routine... likewise with paragraph-terminators, also detectable... stealth scannos are a mixed situation. there are tests that can be done which _do_ help find stealth scannos. but these tests often pull up "false alarms" as well, and thus the cost-benefit ratio from doing them is unclear. that's why my general orientation is _not_ to do them, but rather to leave them for the smoothreaders to find, since detecting stealth scannos is often _easy_ for them. and that's what it boils down to -- to make this _easy_. *** more later, maybe today or tomorrow, and next week... watch out for flying reindeer! :+) -bowerbird

3 6

ho ho ho, merry christmas
by Bowerbird＠aol.com 26 Dec '11

26 Dec '11

here's what i found in my christmas stocking... > http://zenmagiclove.com/cgi-bin/bkdig/bkdig00.py -bowerbird p.s. roger and don, you're doing a fine job of educating each other with interesting dialog... you're teaching yourselves more than i ever did. so for as long as you each make one post a day, continuing a conversation, i won't interrupt you with my pontifications from mount high-horse. i'll only post in my threads. do we have a deal? :+)

1 0

book of james -- 004
by Bowerbird＠aol.com 24 Dec '11

24 Dec '11

james said: > I would guess this is > not the work flow > you had in mind. um, no, it's not... :+) and so i just spent a big whole bunch of hours for no good reason. but hey, it's my own fault. :+) i broke my very own hard-and-fast rule about being explicit and demanding (and demandingly explicit) on chain-of-custody with a collaborative document. (it's actually the #1 cause of headaches in that arena. it's even worse than an asshole boss, because it can make you hate friends. and i "supposedly" know it. and yet i proceeded without being explicit enough. my bad.) *** anyway... my first reaction is just to post the work i've done, and say "if it helps you, fine, go ahead and use it, and if it doesn't, you're no worse off than before," and leave it at that. and i will post the work, no doubt about it. but i feel compelled to give you a little bit of advice, as well, whether you asked for it or not. first... recently, i said this: > rewrapping is evil. > it just makes it harder for the next guy... unfortunately, in your case, james, "the next guy" is _you_. as far as i can tell, you have rewrapped the entire file; however, you still must _proof_ it. rewrapping makes that proofing 27 times harder. second... unless you decide that em-dashes are "immaterial", restoring the missing em-dashes alone will _double_ the time it takes you to clean this text. i kid you not. you really need to re-do the o.c.r. to fix that problem. you _could_ grab the em-dashes from the abbyy.gz file available on the archive.org website, but you will _still_ be missing the utf8 stuff, and that's a huge deal. using the "compose" tip makes that go faster, yes, but you'll save gobs of time if you don't have to do that job. em-dashes and utf8 -- alone -- mean that it is simply a bad investment of your time to continue on your path. that's my experience. and it was hard-won. *** like i said, i'm probably gonna just walk away from this. even so, i advise you to start anew. if you feel like you _must_ retain the 80 pages you've already "fixed", fine. but you still have 350 pages more to go, and if you do 'em inefficiently, you're gonna be wasting a _lot_ of your time. and even the 80 pages that you've already "done" are not -- as far as i can tell -- dependably solid. i did a check for space-doublequote-space, to see if i would find any, and i did. if you missed stuff that is _that_ easy to spot, you probably missed a lot of other more subtle stuff too. even good proofers miss 10%-20% the first time through. so you really need someone to do a second proofing, but with rewrapped lines, it will be _very_ hard to find anyone. (and realistically, with a text this difficult, you might well need three proofings, or even four, before it's really solid.) so anyway, that's my feedback, solicited or not... :+) -bowerbird

2 1

Re: [gutvol-d] updated PP editor
by Bowerbird＠aol.com 24 Dec '11

24 Dec '11

roger said: > The DHYP button dehyphenates the page > so spellchecking has whole words to check. minor question, for roger... why not have the spellchecker join the hyphenates, solely to do its checking, and leave the text intact? *** major questions, for everyone... can someone give me a 2012 justification for the reductionistic attitude toward the proofing task? why are we still doling out pages one-at-a-time? in 2002, i understand why d.p. was conceptualized in such a fashion... o.c.r. wasn't nearly as reliable, so there were corrections necessary on every page, and each one might demand a chunk of your time. but today, you can clean an entire book in an hour -- in one pass, or in six 10-minute sessions, say -- and send it to the smoothreaders for a final check... so why don't you guys want people to do it that way? smoothreaders are lots easier to recruit from scratch. i've been beating this drum for a long time now, and maybe my own noise has deafened me to hear you, so i'm seriously asking you for an answer and i will promise to do my best to hear what you have to say, because i honestly believe there _is_ no justification. so if you believe you have one, can i hear it please? *** for my side, i scraped roger's text and images, and have already cleaned the book and placed it online, as if for smoothreaders. i didn't even bother to do a final spellcheck, or an exhaustive _italics_ search, so those are two tasks you could do if you want to... i'm quite confident this meets my standard criterion, which is a rate of 1 error or less in every 10 pages... by the way, if you wanted to clean roger's text as well, i've appended a list of questionable things to examine. some of 'em are undoubtedly right, but i'd look at 'em anyway, if i were to seriously take the job of this book. (which is to say i didn't actually examine all of this list.) and there were other errors i'd already fixed before i generated this list, so it won't be an exhaustive list... anyway, here you go... > http://z-m-l.com/go/betle/betlep123.html > http://z-m-l.com/go/betle/betle.zml -bowerbird p.s. list of easily-found possible-errors in "betty lee"... > %ut > *Y > Adeste > Ah's > Ain > And'me > Avalking > afteh > agis > ar.e > arivals > artiele > atternoon > audtorium > Bam > Barnett > Baruett > Bradshaw > Buxton > bing > blackeyed > bootleggeh > boydom > butleh > C > Carlin > Caroline > Chanucey > Chauneey > Chetr > Chistmas > Christinas > Cie > Crede > cheks > cher > childf > couse > cousip > crede > D > Dat > Dey > Dodie > Dorrance > Dorranee > Dorry > de > dem > dere's > dictu > dinneh > doin > doublequick > drew'her > drummajor > E > Einma > Eosy > est > evidentally > Fascisti > Ferris > Fidelis > Finn > fatted > flyin > foh > G > Galilee > Geewhilikers > Glaus > Gr > Grood > Gwynne > git > goin > goodhumoredly > goodnight > guestspolitely > gwine > H'm > Hallowe'en > Hm > Howland > Huxley's > Huxleys > he'worked > hehse'f > high'spirits > hise'f > horribile > hurrried > hurryf > hurryin > ing > io > ioas > ivere > Josie > jus > Kose > LeEoy > Lena > LeRoy > Lovel > Lyon"Y > leftiit > longue > loquor > M > Mary > Maytime > Mm > Monday > mawnin > mayn't > me.v > mihi > miserabile > mon > muddly > mustn't > mysteriouser > Nup > neded > nevah > nosense > nuffin > O > O.K. > Onee > Ophelia > Orme > o > ol > on's > operam > orchestra"'next > oughtn't > Papa's > pail."I'm > palazso > palazzo > peccato > playin > prosecomp > proudj > puritanic > pusson > quaveringly > Rardon > r > re > Savilia > Savilla's > Savillas > Sehna > Selma > Sevilla's > Sevillas > sauve > schooltime > searchingiy > selfconscious > semed > serius > sewin > shound > siicceeded > slangily > slep > somthing > spect > spuzzy > suah > Titania > Tommy > t > t'ook > taMng > tempora > than,the > that.would > the'best > thore > tion > tiredlooking > tnat > togeteher > tol > trubble > ubi > under,his > unecessary > vero > victrola > vocabularly > Well,/remember > Willie > wanta > was'glad > welldressed > weren > wlio > woodchopper's > wouldn > wouldn'd > xnay > Y > Yes'm > year's > yo'se'f > yonr > > -Laura? > as-of > Bim-bang > car-fare? > co-operate > direc-i > distinguished-looking > E-r-r-boom > easy-going > entered,-her > faculty-versusstudent > G-od > G-ood > G-wynne > How-de-do > Howdo-you-do > have-something > how-do-you-do > hymn-book > ice-cream > inter-plays > it-go > junior-boys > keen-looking > Lu-chee-a > leave-takings > Mid-years > mid-years > non-sorority > One-two-three-go > pos-i-tive-ly > read-Cicero > room-and > school-girl > semi-order > senior-girls > Tooral-looral-loo-oo-oo-oo > Toot-toot > Trade-Last > tear-stained > termed-social > think-a > thought-Father > tooral-looral > tooral-looral-loo-oo-oo-oo > tr-ragic > try-out > twelve-thirty > What's-his-name > Ye-ah

3 3

book of james -- 003
by Bowerbird＠aol.com 23 Dec '11

23 Dec '11

james said: > Carlo, > I defined a compose key on my Linux box last night and > it works great. I can finish the book twice as fast now. carlo is handing out the holiday gifts, isn't he? :+) thanks again, carlo, for the djvu-text-extraction info... *** james, i just wanna make sure we're on the same page. (see how i did that?) ;+) i'm cleaning the text now, and i will hand it over to you sometime next week. that's my understanding, anyway. is it your understanding as well? i want to be positively _certain_ that we do _not_ have the situation where we're working on separate tracks, because merging 2 sets of corrections is a nightmare. good version control is an absolute requirement. -bowerbird

3 3