- gutvol-d - lists.pglaf.org

Re: [gutvol-d] updated PP editor
by Bowerbird＠aol.com 26 Dec '11

26 Dec '11

ho ho ho! happy christmas eve! :+) ok, busy day for me, and i'm sure all of you too, so i'd guess you'll appreciate if i cut to the chase, and leave the extended discussion for next week. *** for now, i'll conditionally agree i missed 20 errors. but that number is kind of meaningless without also stating the number of errors that i _found_. if i found 2, while missing 20, that's extremely bad. if i found 2000, only missing 20, it's extraordinary. i would estimate that i found 80-200, meaning that my accuracy level was at the 80%-90% range, which is in the ballpark for a typical p1 proofer, _except_ i only worked for an hour, far less than a p1 round. moreover, i didn't perform the full range of checks. most significantly, i did not check any quotemarks. that's because i thought roger's preprocessing had probably _found_and_fixed_ all of those problems. however, since _7_ of them showed up as "missed", i decided that i should go back and do that check... more on the results of that check coming up later... *** the other info that roger needs to tell us about is how many errors that _he_ found and missed, and reveal how many "uniques" we each found... when we have those numbers for both proofings, we can then compute how many errors _remain_, which is the thing that we _really_ want to know, so we can decide if another proofing is "worth it". this is what don was talking about, except that the situation isn't nearly as vague as he believes. indeed, the specificity you can obtain from just a mere _handful_ of data can be quite startling, if you ensure that you select your data carefully. (this often means creating a test to produce it.) *** one of the things you should absolutely remember is that if you have found an error of a certain type, you _must_ check for more which might be lurking. i did that, with the errors of mine that roger listed. he found that i'd missed some "lie for he" scannos. and sure enough, a search turned up another one: > chance that lie may get work the first of the roger listed a paragraph-termination error i missed. and a search revealed another two or three more: > The World Syndicate Publishing Co, > be in it, Finny, I mean, > but were giving the Christmas hymn beginning, and, most notably for this book, there were also the double-quote errors roger discovered i'd missed.... the systematic search for those was quite interesting. *** the typographer for this p-book was badly hungover. the rate of errors in this p-book is surprisingly high, including an atypical number of outright misspellings. but the number of doublequote glitches is astounding. i have to verify them and write 'em up, but i'd guess that i found at least a dozen more, and maybe two... so -- at least in one sense -- you could say i missed even more errors than roger computed. but of course if i had done this doublequote check in the first place, and done it right, i wouldn't have missed any of these, as doublequotes _are_ a type of error that's detectable. so that check must definitely be a part of our routine... likewise with paragraph-terminators, also detectable... stealth scannos are a mixed situation. there are tests that can be done which _do_ help find stealth scannos. but these tests often pull up "false alarms" as well, and thus the cost-benefit ratio from doing them is unclear. that's why my general orientation is _not_ to do them, but rather to leave them for the smoothreaders to find, since detecting stealth scannos is often _easy_ for them. and that's what it boils down to -- to make this _easy_. *** more later, maybe today or tomorrow, and next week... watch out for flying reindeer! :+) -bowerbird

3 6

ho ho ho, merry christmas
by Bowerbird＠aol.com 26 Dec '11

26 Dec '11

here's what i found in my christmas stocking... > http://zenmagiclove.com/cgi-bin/bkdig/bkdig00.py -bowerbird p.s. roger and don, you're doing a fine job of educating each other with interesting dialog... you're teaching yourselves more than i ever did. so for as long as you each make one post a day, continuing a conversation, i won't interrupt you with my pontifications from mount high-horse. i'll only post in my threads. do we have a deal? :+)

1 0

book of james -- 004
by Bowerbird＠aol.com 24 Dec '11

24 Dec '11

james said: > I would guess this is > not the work flow > you had in mind. um, no, it's not... :+) and so i just spent a big whole bunch of hours for no good reason. but hey, it's my own fault. :+) i broke my very own hard-and-fast rule about being explicit and demanding (and demandingly explicit) on chain-of-custody with a collaborative document. (it's actually the #1 cause of headaches in that arena. it's even worse than an asshole boss, because it can make you hate friends. and i "supposedly" know it. and yet i proceeded without being explicit enough. my bad.) *** anyway... my first reaction is just to post the work i've done, and say "if it helps you, fine, go ahead and use it, and if it doesn't, you're no worse off than before," and leave it at that. and i will post the work, no doubt about it. but i feel compelled to give you a little bit of advice, as well, whether you asked for it or not. first... recently, i said this: > rewrapping is evil. > it just makes it harder for the next guy... unfortunately, in your case, james, "the next guy" is _you_. as far as i can tell, you have rewrapped the entire file; however, you still must _proof_ it. rewrapping makes that proofing 27 times harder. second... unless you decide that em-dashes are "immaterial", restoring the missing em-dashes alone will _double_ the time it takes you to clean this text. i kid you not. you really need to re-do the o.c.r. to fix that problem. you _could_ grab the em-dashes from the abbyy.gz file available on the archive.org website, but you will _still_ be missing the utf8 stuff, and that's a huge deal. using the "compose" tip makes that go faster, yes, but you'll save gobs of time if you don't have to do that job. em-dashes and utf8 -- alone -- mean that it is simply a bad investment of your time to continue on your path. that's my experience. and it was hard-won. *** like i said, i'm probably gonna just walk away from this. even so, i advise you to start anew. if you feel like you _must_ retain the 80 pages you've already "fixed", fine. but you still have 350 pages more to go, and if you do 'em inefficiently, you're gonna be wasting a _lot_ of your time. and even the 80 pages that you've already "done" are not -- as far as i can tell -- dependably solid. i did a check for space-doublequote-space, to see if i would find any, and i did. if you missed stuff that is _that_ easy to spot, you probably missed a lot of other more subtle stuff too. even good proofers miss 10%-20% the first time through. so you really need someone to do a second proofing, but with rewrapped lines, it will be _very_ hard to find anyone. (and realistically, with a text this difficult, you might well need three proofings, or even four, before it's really solid.) so anyway, that's my feedback, solicited or not... :+) -bowerbird

2 1

Re: [gutvol-d] updated PP editor
by Bowerbird＠aol.com 24 Dec '11

24 Dec '11

roger said: > The DHYP button dehyphenates the page > so spellchecking has whole words to check. minor question, for roger... why not have the spellchecker join the hyphenates, solely to do its checking, and leave the text intact? *** major questions, for everyone... can someone give me a 2012 justification for the reductionistic attitude toward the proofing task? why are we still doling out pages one-at-a-time? in 2002, i understand why d.p. was conceptualized in such a fashion... o.c.r. wasn't nearly as reliable, so there were corrections necessary on every page, and each one might demand a chunk of your time. but today, you can clean an entire book in an hour -- in one pass, or in six 10-minute sessions, say -- and send it to the smoothreaders for a final check... so why don't you guys want people to do it that way? smoothreaders are lots easier to recruit from scratch. i've been beating this drum for a long time now, and maybe my own noise has deafened me to hear you, so i'm seriously asking you for an answer and i will promise to do my best to hear what you have to say, because i honestly believe there _is_ no justification. so if you believe you have one, can i hear it please? *** for my side, i scraped roger's text and images, and have already cleaned the book and placed it online, as if for smoothreaders. i didn't even bother to do a final spellcheck, or an exhaustive _italics_ search, so those are two tasks you could do if you want to... i'm quite confident this meets my standard criterion, which is a rate of 1 error or less in every 10 pages... by the way, if you wanted to clean roger's text as well, i've appended a list of questionable things to examine. some of 'em are undoubtedly right, but i'd look at 'em anyway, if i were to seriously take the job of this book. (which is to say i didn't actually examine all of this list.) and there were other errors i'd already fixed before i generated this list, so it won't be an exhaustive list... anyway, here you go... > http://z-m-l.com/go/betle/betlep123.html > http://z-m-l.com/go/betle/betle.zml -bowerbird p.s. list of easily-found possible-errors in "betty lee"... > %ut > *Y > Adeste > Ah's > Ain > And'me > Avalking > afteh > agis > ar.e > arivals > artiele > atternoon > audtorium > Bam > Barnett > Baruett > Bradshaw > Buxton > bing > blackeyed > bootleggeh > boydom > butleh > C > Carlin > Caroline > Chanucey > Chauneey > Chetr > Chistmas > Christinas > Cie > Crede > cheks > cher > childf > couse > cousip > crede > D > Dat > Dey > Dodie > Dorrance > Dorranee > Dorry > de > dem > dere's > dictu > dinneh > doin > doublequick > drew'her > drummajor > E > Einma > Eosy > est > evidentally > Fascisti > Ferris > Fidelis > Finn > fatted > flyin > foh > G > Galilee > Geewhilikers > Glaus > Gr > Grood > Gwynne > git > goin > goodhumoredly > goodnight > guestspolitely > gwine > H'm > Hallowe'en > Hm > Howland > Huxley's > Huxleys > he'worked > hehse'f > high'spirits > hise'f > horribile > hurrried > hurryf > hurryin > ing > io > ioas > ivere > Josie > jus > Kose > LeEoy > Lena > LeRoy > Lovel > Lyon"Y > leftiit > longue > loquor > M > Mary > Maytime > Mm > Monday > mawnin > mayn't > me.v > mihi > miserabile > mon > muddly > mustn't > mysteriouser > Nup > neded > nevah > nosense > nuffin > O > O.K. > Onee > Ophelia > Orme > o > ol > on's > operam > orchestra"'next > oughtn't > Papa's > pail."I'm > palazso > palazzo > peccato > playin > prosecomp > proudj > puritanic > pusson > quaveringly > Rardon > r > re > Savilia > Savilla's > Savillas > Sehna > Selma > Sevilla's > Sevillas > sauve > schooltime > searchingiy > selfconscious > semed > serius > sewin > shound > siicceeded > slangily > slep > somthing > spect > spuzzy > suah > Titania > Tommy > t > t'ook > taMng > tempora > than,the > that.would > the'best > thore > tion > tiredlooking > tnat > togeteher > tol > trubble > ubi > under,his > unecessary > vero > victrola > vocabularly > Well,/remember > Willie > wanta > was'glad > welldressed > weren > wlio > woodchopper's > wouldn > wouldn'd > xnay > Y > Yes'm > year's > yo'se'f > yonr > > -Laura? > as-of > Bim-bang > car-fare? > co-operate > direc-i > distinguished-looking > E-r-r-boom > easy-going > entered,-her > faculty-versusstudent > G-od > G-ood > G-wynne > How-de-do > Howdo-you-do > have-something > how-do-you-do > hymn-book > ice-cream > inter-plays > it-go > junior-boys > keen-looking > Lu-chee-a > leave-takings > Mid-years > mid-years > non-sorority > One-two-three-go > pos-i-tive-ly > read-Cicero > room-and > school-girl > semi-order > senior-girls > Tooral-looral-loo-oo-oo-oo > Toot-toot > Trade-Last > tear-stained > termed-social > think-a > thought-Father > tooral-looral > tooral-looral-loo-oo-oo-oo > tr-ragic > try-out > twelve-thirty > What's-his-name > Ye-ah

3 3

book of james -- 003
by Bowerbird＠aol.com 23 Dec '11

23 Dec '11

james said: > Carlo, > I defined a compose key on my Linux box last night and > it works great. I can finish the book twice as fast now. carlo is handing out the holiday gifts, isn't he? :+) thanks again, carlo, for the djvu-text-extraction info... *** james, i just wanna make sure we're on the same page. (see how i did that?) ;+) i'm cleaning the text now, and i will hand it over to you sometime next week. that's my understanding, anyway. is it your understanding as well? i want to be positively _certain_ that we do _not_ have the situation where we're working on separate tracks, because merging 2 sets of corrections is a nightmare. good version control is an absolute requirement. -bowerbird

3 3

updated PP editor
by Roger Frank 23 Dec '11

23 Dec '11

I've made some changes to the Page Proofing Editor and I'll freeze the code and use PPE to proofread two books I've scanned with Abbyy. The major changes are all to the analysis window contents. I've added visible spell checking, with suspect words highlighted in light blue. Other colors on the analysis window indicate other suspect conditions. I've added quite a few from the excellent set in gutcheck and some others from pptxt. There are more I will add over time, but I'm anxious to use it for a few books. I've also added a way for the user to add good words from any page. I see this most often for proper names. By the way, the text is scanned with end-of-line hyphenation intact. The DHYP button dehyphenates the page so spellchecking has whole words to check. Feedback always welcomed. I'll report back again after I've used it on an entire text. --Roger

1 0

Seeking advice on formatting of magazine-type content
by Roger Lee 23 Dec '11

23 Dec '11

I've previously completed a reasonable number of proof-readings of book content and now I've begun preparing the content of a large annual of weekly magazines - I have already received my copyright clearance. My annual was published in 1905-6 and is in fair condition, but not suitable for flat-bed scanning; the pages are somewhat yellowed. I'm using a LG LSM-100 Scanner Mouse and I've completed initial scanning of the first issue of this magazine. I'm creating separate PNG files for each complete page and of each distinct illustration; these need further work. Based on my previous proof-reading, I'm working on correcting the initial OCR text in plain text files. I'm using the folder structure as follows... Annual pngs (of annual only) text (of annual only) Issue 1 pngs (of Issue 1 only) text (of Issue 1 only) (etc) Each weekly magazine comprises fictional and non-fictional material. Initially, I plan to treat each issue of about 16 pages as a project and the annual as a separate project. I'm also thinking that where articles are episodic, I feel it is worth considering side projects in which the episodes for a given article are assembled as a unit. The annual has a good table of contents using page numbers; in the ebook format, the page numbers might usefully be converted into issue number and article-in-issue number (like 1395.1). In the first issue, I've seen only one page reference to an issue pre-dating this annual. In the Project Gutenberg catalog, I see there are quite a number of other magazines, so I can use them as a guide though I realize there are experienced final-formatters out there. I would welcome pointers to relevant information to help me proceed and, if possible, contact with someone who has worked on this kind of material. How do I go about creating the project structure for others to help with proof-reading, etc, in due course? Where can I find advice on cleaning up illustrations? Thanks for your patience, Roger

5 9

Re: [gutvol-d] book of james -- 001
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

for want of better text, i've now loaded in the original o.c.r. for the "book of james". > http://z-m-l.com/go/bhaga/bhagap123.html this isn't the edit interface, it's just a skeleton to display the o.c.r./scan combo for each page. resizing your browser-window resizes the scan. so james, if you want me to pursue this at all, send me your corrected files. otherwise, i will just be done with it. i'd love to see an xhtml tutorial on this book... *** roger said: > http://etext.ws/ppe.php yes! roger's back! :+) > different users with different browsers. iphone, no. ipad, also no, unfortunately. doesn't appear to size itself very well... > http://z-m-l.com/misc/cinema-screen-shot.png lotta unused space on my 23-inch screen. anyway, all you people who bellyache about "open source" need to step up to the plate... go over and help roger code something cool. show that you really _deserve_ open source... *** don said: > Is this closer? i don't think so. i'd say you're getting colder. but i _will_ read that story about the turnip... *** keith said: > elves aha! that's great. i didn't think the gerbils would have enough manual dexterity... but _elves_ are very handy with their... hands... > Now, if I could only get then to program. oh, they don't have to be able to code at all. just tell them to run these commands: > pandoc -i criterion.html -o markdown.md > pandoc -i criterion.html -o rst.rst > pandoc -i markdown.md -o markdown.html > pandoc -i rst.rst -o rst.html if criterion.html and markdown.html and rst.html are not "sufficiently similar", let me know why not. *** carlo said: > djvutxt --page=123 djvufile.djvu 123.txt ok, i get it now. it's on _my_ machine. i thought you issued it on archive.org. i can't thank you enough for that, carlo; this ends a years-long headache for me. -bowerbird

4 5

Re: [gutvol-d] book of james -- 002
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

james- there is now cleaner text on those web-pages: > http://z-m-l.com/go/bhaga/bhagap123.html this text is from the .djvu file, so the pagebreaks are much more accurate than they were before... i was hoping the em-dashes would reappear too, but no luck there. perhaps someone will offer to re-do the o.c.r. i will also play with the abbyy.gz file at archive.org, and see what i can coax out of that, but i'm having too much fun tinkering with the .djvu text for now. like i said, hang loose until after christmas... :+) -bowerbird

1 0

book of james -- 002
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

james said: > I started doing the page-at-a-time thing and gave up. > Your pages are already better than mine because > I used Tesseract and archive.org uses ABBY Fine Reader. except the o.c.r. from archive.org, in this case, is screwed up. it's missing its em-dashes. i've dealt with this problem before, and it's less work to re-do the o.c.r. than to fix the em-dashes. will someone with a good version of abbyy please re-do this o.c.r.? > 2). This book really requires a way to enter UTF-8 characters. if someone does the o.c.r. for you, they can specify utf8 output... > If I could just stick a circumflex above a's, u's, and i's > (both lower and upper case) that would be 99% of what I need. if you can pull out a list of the words that require circumflexes, we can create a script that does a global change in one swoop. > (after de-hyphenating do not dehyphenate! the program will do that for you. > re-wrapping do not rewrap! if you need to rewrap, the program can do it. rewrapping is evil. it just makes it harder for the next guy... -bowerbird

3 2