April 2010 - gutvol-d - lists.pglaf.org

my report on that sitka posting
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

ok folks, here's my book report on the "sitka" book which was done over at fadedpage, and posted to p.g. it's #31862 if you want to go and take a look at it: > http://www.gutenberg.org/files/31862/31862.txt *** i'll discuss the high points in the body of this message; appended you'll find the data documenting these points... *** 1. ...stealth scanno... the first thing to note is that a sharp-eyed proofer caught a stealth scanno. my system, which relies on spellcheck heavily, is susceptible to stealth scannos, so i'm always interested in their frequency, and whether or not they will cause a serious problem in comprehension. in this book, there was 1 stealth scanno, so they were infrequent in this book. further, it wasn't a bad one... a "heeded" was misrecognized as "needed", so no big deal. *** 2. ...publisher corrections made; here's a list of 15... rfrank made his digitization philosophy very clear here, as he made a number of _corrections_ to the original book. and i agree! i'm firmly against the "transcription" model, which holds that we should merely reproduce the print-book. i maintain we are _republishing_ the book, and therefore i strongly believe we need to correct any errors we find. and like me, roger doesn't bother with a "list of changes", either. i don't think it's necessary to make such a list. they're boring. just make the change and keep on moving... however, there is one aspect to note about this, which is that means rfrank sets a higher standard for his work, and if we find errors in the p-book which he should've found, then he is now accountable for having missed those errors. *** 3. ...more fixes, requiring research outside the book... to his credit, rfrank went out of his way to check things, so as to make sure that he was _only_ fixing actual errors, and not introducing bad changes. sure, it's not too hard to look up a sailor's name on wikipedia to spell it right, or to check that a word means "an edit from the czar", but it takes time, and it improves the output, so give kudos... *** 4. ...he even did "corrections" i might not have done... as is often the case when you adopt a "re-publication" philosophy as opposed to "transcription", some of the changes that you make might not get universal agreement. but that comes with the territory, so you roll with it. *** 5. ...and some i _certainly_ wouldn't have done... and you continue rolling, even when disagreement is thick. *** 6. ...and yet he didn't do some that i woulda done... everybody has their own opinions... :+) i won't count these as "errors". but _i_would_ do them differently... *** 7. ...and i disagree on how he did some formatting... once again, everybody's got an opinion... for the first 2 cases shown, i put a comma after the period, just so the following lowercase letter wouldn't throw a flag every time i did that check. the third case is a bit more complicated... i believe that the formatting rules used by rfrank -- and d.p. in general -- are slightly misguided... first of all, they are arbitrary, with wiggle-room, which means they're hard to understand and interpret. i think rules that are firmly grounded work better... second, in many situations the rules instruct us to put the formatting toggles _inside_ the word itself (at least inside the punctuation which surrounds it). in this case, for instance, the _italicized_ word is placed inside some _non-italicized_ parentheses. so an italic letter butts a non-italic parenthesis. but _none_ of the rendering engines we have today (like browsers and e-book apps) are capable enough to make that look reasonably good, let alone nice. indeed, most of the time it looks absolutely dreadful; a shift from non-italic to italic (and back) is ugly. so even when it doesn't "make sense" grammatically, my recommendation is to italicize the entire string, it looks better, so that's what i do; you should too. *** 8. ...the previous errors... ok, now we get down to the brass tacks. were there any errors in the posted book? well, yeah. i've already pointed out 6. they are repeated below, to refresh memory. but there are more errors... *** 9. ...there are hyphenation differences... rfrank has clearly shown that he is more than willing to make changes to the book, so as to correct errors, and for consistency. for instance, he changed several cases of the name "wrangel", consistent to "wrangell". so if we now _find_ errors or inconsistencies, even if they were present in the original book, we'll hold him accountable for not having made the proper corrections. it's only fair. and yes, the book had several hyphenation inconsistencies. rfrank may have found and fixed some, i don't know, but i _do_ know that there were some that he did _not_ fix... do you like the word "sealion"? or is "sea-lion" better? i like the second. the book was split 50/50. rfrank did 3 of them one way, then left the final 1 the other way... i'd call the error-count 2, but you can call it 1, or 3. there were 2 cases of "far-off" and 1 case of "far off", so i'd go with "far-off", and raise the error-count by 1. there was 1 "far-seeing" and 1 "far seeing". ironically, they're both on the same page. talk about short-sighted. call 1 of 'em an error. there was 1 "guest-house" and 1 "guesthouse", so 1 error. and 1 "ice-houses" and 1 "icehouses", so again, 1 error... so let's say 6 hyphenation errors, due to inconsistency... *** 10. ...and the other consistency errors... i show 11 other consistency errors, all on names... feel free to argue about any of them. if you can find someone who's willing to argue them, that is. so far, 17 new errors... *** 11. ...and finally, the last 2 errors... there was 1 case of some missed formatting, as shown clearly in the scan for this page: > http://z-m-l.com/go/sitka/sitkap025.png _barabaras_, barabaras, ^^^^^^^^^^ it's humorous to note that this missed italic was on the same page as the missed italic i reported before. but nobody said anything about "barabaras" earlier... last but not least, 1 misspelled word: "gooch-heat", which should have been "gooch-haet" -- wolf-house... yes, it was printed wrong, but it shoulda been caught. so 2 more, for a sum of 19 new errors, plus the original 6, making a grand total of 25. that's twice my usual standard -- 1 error per 10 pages -- but since _most_ of these errors were also present in the p-book (along with many more which were corrected by rfrank), i'd call this a good digitization. *** in conclusion... many people will say that most of these errors are trivial. i wouldn't argue with them. at the same time, i'm compulsive enough to fix the trivial. i don't want any errors in my books. even "trivial" ones... i'm guessing that roger will want to fix these errors too. i'm also guessing that roger will say most of these errors can be laid at the feet of the postprocessor, so that his "roundless" system has passed this test with flying colors. and, in a sense, he's right, in that most of these errors _should_ be located within the realm of the postprocessor. (or -- to once again stress my model -- the preprocessor.) but to give the postprocessor the _luxury_ of enough time to do the job of catching errors like these, we must have the proofers perform as many duties as they possibly can. when a page can conceivably be seen by only _one_ proofer, then a postprocessor simply can't have enough faith that each page is correct, and will inevitably perform checks they would not do if they believed every page was solid... if you're worried about simple things like missing italics, you just won't have the focus to look for more subtle stuff. use your proofers to make sure every page is rock-solid. give the postprocessor the luxury to _polish_ the book... -bowerbird p.s. here's a "bonus error", thrown in for good measure. "blockhouse" isn't capitalized elsewhere in the book, and if we did capitalize it here, we'd capitalize "upper" too. > near the site of the upper blockhouse. Her successor, > near the site of the upper Blockhouse. Her successor, > ===========================^^^^^^^^^^^^^^^^^^^^^^^^^^ =============================================================== 1. ...stealth scanno... > and was needed. Captain A. Holmes A'Court, > and was heeded. Captain A. Holmes A'Court, > ========^================================= 2. ...publisher corrections made; here's a list of 15... > Narative of a Voyage Round the World. > Narrative of a Voyage Round the World. > ===^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > pupil.' > pupil.'" > =======^ > an air of prosperity prevaded the place. > an air of prosperity pervaded the place. > ======================^^================ > Her successor, the sceond Princess > Her successor, the second Princess > ====================^^============ > in these waters bought skins for mere trifles, some for > in these waters, bought skins for mere trifles, some for > ===============^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > bath house > a bath house > ^^^^^=^^^^ > island but > islands but > ======^^^^ > ungents, combine to make > unguents, combine to make > ===^^^^^^^^^^^^^^^^^^^^^ > Hudson Bay Co. the Russian ships that sailed > Hudson's Bay Co. the Russian ships that sailed > ======^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > of the educational institution of Sitka. > of the educational institutions of Sitka. > ========================^^^^^^^ > The liquid refreshments serve to him > The liquid refreshments served to him > =============================^^^^^^^ > the villianous liquor called "hoochinoo" > the villainous liquor called "hoochinoo" > ========^^============================== > island to the broad Pacific. What were the thoughts > islands to the broad Pacific. What were the thoughts > ======^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > or "Ranche," there is choice of two streets, > or "Ranche," there is a choice of two streets, > ======================^^^^^^^^^^^^^^^^^^^^^^ > and three who names I do not know. > and three whose names I do not know. > =============^^^^^^^^^^^^^^^^^^^^^ 3. ...more fixes, requiring research outside the book... > Priest Vasili Michaeloff Ocueredin, > Priest Vasili Michaeloff Ocheredin, > ===========================^======= > of the harvest. Her captain, Entienne Marchand, > of the harvest. Her captain, Etienne Marchand, > ==============================^^^^=^^^^^^^^^^^ > ukaze, > ukase, > ===^== 4. ...he even did "corrections" i might not have done... > all the Russians to the Republic > all the Russias to the Republic > ==============^^^^^^^^^^^^^^^^^ > of the American troops, Gen. Jeff C. Davis, > of the American troops, Gen. Jefferson C. Davis, > =================================^^^^^^^^^^ > is called Kosters Trail. The first > is called Koster's Trail. The first > ================^^^^^^^^^^^^^^^^^^ 5. ...and some i _certainly_ wouldn't have done... > articles, carved with totemic design > articles, carved with totemic designs > ==================================== 6. ...and yet he didn't do some that i woulda done... > --The Author > The Author > ^^^^^^^^^^^ > in the Bering Sea, > in Bering Sea > ===^^^^^^^^^^ 7. ...and i disagree on how he did some formatting... > to the Hudson's Bay Co., the Russian ships that > to the Hudson's Bay Co. the Russian ships that > =======================^^^^^^^^^^^^^^^^^^^^^^^ > Total, 400. Ib., p. 52. > Total, 400. Ib. p. 52. > ===============^^^^^^^ > of New Archangel _(Novo Arkangelsk),_ > of New Archangel (_Novo Arkangelsk_,) > =================^^===============^^^ 8. ...the previous errors... > _yukali_ (dried salmon), > yuhali (dried salmon), > ^^^^^^^^^^^^^^^^^^^^^^ > _prazdnik_ (holiday) > prasdnik (holiday) > ^^^^^^^^^^^^^^^^^^ > _Sheetkah_, > Sheetkah, > ^^^=^^^^^ > Seattle Intelligencer, December 14th, 1868; > Seattle Intelligencer, December 14th. 1868; > ======================^^^^^^ 9. ...there are hyphenation differences... sea-lions (#19, but at line-end) sealion (#29) sealion (#48) sea-lion (#101) > and sea-lion meat from Kodiak, and > and sealion meat from Kodiak, and > =======^^^^^^^^^^^^^^^^^^^^^^^^^^ > adorned with sea-lion heads > adorned with sealion heads > ================^^^^^^^^^^ far off (#35) far-off (#42) far-off (#80) > in the far-off possession of the Czar. > in the far off possession of the Czar. > ==========^=========================== far seeing (#16) far-seeing (#16) (yes, inconsistent on the same page) > wealthiest and most far-seeing of the leaders > wealthiest and most far seeing of the leaders > =======================^^^^^^^^^^^^^^^^^^^^^^ > being entertained in the guest-house were ...but yet... > went to the guesthouse of the kwan. All the > The ice-houses were near the outlet of ...but yet... > icehouses was laden on the ship 250 tons, and 10. ...and the other consistency errors... > in Chicagof Island, sent his ship's boat > in Chichagoff Island, sent his ship's boat > =======^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Admiral Chicagof, Minister > Admiral Chichagof, Minister > ============^^^^^^^^^^^^^^ > and Chicagof." (A Voyage Round the World, Lisianski, p. 235.)] > and Chichagof." (A Voyage Round the World, Lisianski, p. 235.)] > ========^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > of Golden California. Captain Hagemeister came to relieve him, > of Golden California. Captain Hagmeister came to relieve him, > =================================^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > of St. Michael > of St. Michaels > ============== > stir the lust for vengeance. The Keeksitties, > stir the lust for vengeance. The Keeksittis, > ==========================================^^ > [Footnote 19: Globokoe Lake was sounded to > [Footnote 19: Golobokoe Lake was sounded to > ===============^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Prince Dmitri Maksoutoff, Dec. 2, 1863, to Oct. 18, 1867.] > Prince Dmitri Maksoutof, Dec. 2, 1863, to Oct. 18, 1867.] > =======================^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Annahootz, the friendly Kokwanton war chief, > Annahootz, the friendly Kokwantan war chief, > ===============================^============ for these two, we'd have to research which variant is correct: ...in one place, we have one spelling... > the mouth of the Indian River, or _Kolosh Ryeku_. ...but in another place it's spelled differently... > the river, known as the _Kolosh Ryeka_, by the Russians ...in one place, we have one spelling... > Captain Leontius Andreanovich Hagemeister ...but in another place it's spelled differently... > Leonti Andreanvich Hagemeister, Jan. 11, 1818, to Oct. 24, 1818. ...getting now down to some smaller points... ...for consistency, and clarity, i'd change these lines... > Total, 400, Ib. p. 52. > their own account. Id. Vol. 2, p. 38. ...i would use "ibid" in both, replacing the "ib." and "id."... ...but i wouldn't call a "failure" to do this "an error"... ...for consistency, i would also change this reference... > Russian American Archives. Corr. Vol. I, p. 275. ...so that it matched with this reference... > Russian American Archives, Correspondence, Vol. II, No. 108. ...but i wouldn't count a "failure" to do so as "an error"... ...and, to finish off this consistency section... ...i'd do research to find out how these archives are named... > Russian American Archives. Corr. Vol. I, p. 275. > their own account. Id. Vol. 2, p. 38. ...because in the top line, it's "vol. i", roman-style... ...while in the bottom line, it's "vol. 2", arabic-style... ...obviously, one of these versions is incorrect... ...but so much for the trivial obsessive-compulsive points... 11. ...and finally, the last 2 errors... _barabaras_, barabaras, ^^^^^^^^^^ gooch-heat gooch-haet ========^^

1 0

so let's talk about my collaborative proofreading site
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

ok, so let's talk about my collaborative proofreading site... yeah, that's a real knee-slapper, isn't it? :+) ok, i'll talk, and y'all can just sit and listen. or go watch t.v. *** to see what i'm talking about, you can visit this u.r.l.: > http://z-m-l.com/go/sitka/editr.pl when you go there, you're proofing the "sitka" book... unlike other proofing systems, pages are not "assigned" to you. you can go and proof any page you wanna proof. even go back the next day and proof it again, if you like. you'll see a row of buttons at the top-left of the screen, which includes one that says "go", and "prev", and "next"... navigating the pages... "prev" and "next" are what you'd expect. they take you to the previous page or the next page in the book, obviously. at the end of that row of buttons, you'll see an edit-field that has a number in it. (or perhaps a letter and numbers.) that number is the pagenumber. (and, ergo, the filename.) if you want to jump to a specific page, put its number in the edit-field, and then click the "go" button. boom, you're there. the ability to navigate the book at will, and to proof any page you like, can be extremely powerful once you learn to use it. certifying a page as clean... there's also an "ok" button. that's what you will click when you've proofed a page and found nothing at all to change. by clicking "ok", you indicate you certify the page is clean. searching the book for a string... next to "ok" is another edit-field and a "find" button next to it. this field lets you enter a search-term, and then when you click "find", a screen appears that lists pages with that search-term... the line containing the word is shown, with a link to its page. if you want to visit any of these pages, you can open them in a separate tab/window. and it's fine to open a bunch at one time. for instance, say you want to check the sitka chapter-headers. enter the term "chapter" in the edit-field, and then click "find", and you will get a list that includes the table of contents page as well as each of the nine pages where each chapter starts... (red lines are a case-insensitive hit; black are case-sensitive.) there are many other ways you can use the search functionality, but we'll save the discussion about all of those for a later time... to return from the search-results page back to a proofing page, just use the "back" button in your browser. you'll have to clear the search-term out of the edit-field, or it'll do the search again. (i should make it so it only does the search when you click the "find" button, but that's not the way it works now, sorry folks.) feel the power with the "command" field... the edit-field where you enter your search-term also serves as a "command" field. i'll eventually have a number of commands that you can issue, but for now there's just a couple of them... showmap... the first command is "showmap". enter that, and click "find". the program will show the "map" of files comprising the book. each one is a clickable link, so -- as before -- you can open any pages you like in separate tabs and proof them just fine... ("listcat", short for "list catalog", is a synonym that works too.) concat... another command is "concat". enter that, and click "find", and the program will concatenate all the text files into one, and put them on a web-page for you. this will allow you to look at the entire book, save it to your machine, and so on... you can also use the browser's "find" command, so "concat" is useful when you need more context to a "find" operation than the single line output from the native find command... showcustom... a third command is "showcustom". this command spits out the "custom dictionary" that has been created for the book. *** that's enough for now. give you a little toy to play with over the weekend, if you like. we'll discuss more stuff next week. anyway, thanks for the little chat. so, how was the t.v. show? -bowerbird

1 0

Re: how to create a spellcheck workflow
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

gardner said: > what use is preserving or fixing the page headings? > If they can be mechanically fixed, they were not worth much. well, to take your second sentence first, the runheads cannot _always_ be mechanically fixed. sometimes they contain text that describes the current point in the chapter, like an outline, and aren't just limited to a boring recitation of title and author. and returning to the question, runheads are useful since they help you keep your bearings in the book. even when they are nothing but title/author, they help keep recto/verso straight... plus -- especially when you're working on multiple projects -- it's useful to have the reminders about which book you're in... because without 'em, after a time, the pages all look the same. > Do you include or exclude running headings in > your word-count dictionary analysis? Does it matter? never paid much attention, so i guess it doesn't matter much. but runheads are definitely excluded from search operations. -bowerbird

1 0

why isn't distributed proofreaders helping fadedpage?
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

why isn't distributed proofreaders helping fadedpage? the site needs proofers, and d.p. has an _excess_, to the degree that they are actively attempting to stunt the work being done by the p1 volunteers... so why not send some people over to fadedpage? rfrank has done a lot for distributed proofreaders. and fadedpage is showing d.p. how they can make a roundless system work, and that it _does_ work... so why is d.p. being so stingy? help your brother! -bowerbird p.s. and to start, some of you people who are now working at fadedpage can occasionally make a post in a forum at d.p. inviting people to try fadedpage...

1 0

how to create a spellcheck workflow
by Bowerbird＠aol.com 08 Apr '10

08 Apr '10

ok, well gee... i had gotten the firm impression that rfrank had read the d.p. forum thread where the development of their "wordcheck" was discussed (thoroughly -- 30 pages). d.p. calls it "wordcheck", but it's basically spellcheck. so why did i think he had read it? because he incorporated several of the things which i had suggested (to no good effect) in that thread... i won't bother reciting all of those particulars, since he might well have come up with the ideas himself, and nobody really cares anyway, not even me... but i mention it because now i have been convinced that roger must _not_ have read that thread, based on some current discussions on his fadedpage site... or, if he did read it, he didn't "get it", but given those current discussions, he might now be more receptive. so i will run through a quick little "how-to" refresher on how to design and build a spellcheck functionality, and incorporate it into the overall workflow of a book, so roger has the benefit of my wisdom here... ;+) as you'll see, this will hit a very wide variety of topics. note that what we're talking about here is _primarily_ executed during _preprocessing_, but there are some follow-on thoughts that apply to the proofing as well. *** 0. set o.c.r. parameters correctly. don't dehyphenate! name your files wisely. look both ways before crossing. 1. the first thing you should do with your o.c.r. results is to make the few global changes you can do _blindly_. the very first one is to strip trailing spaces from all lines. another will be to replace two spaces with a single space. yet another is to change a linebreak-doublequote-space combination to eliminate the space, since it's superfluous. likewise, change all cases of space-doublequote-linebreak. you get the drift. oh, by the way, don't do what d.p. does! _retain_ the runheads, and pagenumbers; you need 'em... 2. the next thing you should do is clean up the runheads. this has little bearing on our general "spellcheck" topic, but i include it here because it's always your second step. 3. the third thing on your list is to fix all paragraphing... again, not much to do with spellcheck, but it _is_ step #3. 4. now we can focus quite specifically on spellcheck stuff. we will take the o.c.r. and run it through a program i wrote which pulls out all the words _not_ present in its dictionary. i think rfrank has his own program that does the same thing, or something similar enough. i'll make mine available too. this is the first draft of your "bad-words" list for this book. (note this is not the same as how d.p. defines "bad words".) in this regard, use a good dictionary in this first check here. (this is something that rfrank hasn't done correctly thus far.) the dictionary i use is quite good, and it can be found here: > http://z-m-l.com/go/regulardictionary.txt just so you have a feel for the output from this program, i have posted it for the "sitka" book we've been discussing: > http://z-m-l.com/go/sitka/sitka-reversedictionary.txt that output was generated in 5 seconds, so it's pretty fast... 5. you'll see, on viewing your list of supposed "bad-words", that a bunch are not "bad-words" at all. some of 'em will be character-names, or jargon specific to your particular book. some will be hyphenated fragments, some compound-words. you can delete these words from the list now, if you want, but you shouldn't necessarily feel a great need to do that. if you're looking at the list of words from the "sitka" book, you will also notice that i separated the initial-caps words from all-lowercase ones. there is a good reason for that. due to proper names, _most_ initial-cap words are correct, whereas most of the all-lowercase words are _incorrect_... separating the lists makes it easier to focus your attention. 6. your dictionary-check program should also spit out the _frequency_ of each bad-word. you'll use that information to cull some words from this list of "bad-words". this you _will_ want to do, most definitely, so _sort_ on frequency. i fought tooth-and-nail on this with d.p. (i lost, of course), but you can take this to the bank that as long as there are a mere 4-plus occurrences of a specific string in the o.c.r., you can (i.e., should) delete it from this list of "bad-words". yes, some of the words _might_indeed_ be bad, but unless you're positive of that, you should delete 'em from the list. and yes, this means that those words will _not_ be flagged. but trust me, if there's 4 or more occurrences of a scanno, your proofers will find at least _one_ . and whenever they find _one_ "bad-word" that wasn't on that "bad-word" list, you will automatically search the rest of the book, and thus find those _other_ occurrences as well, so you can fix them. i did _not_ include frequency information in my "sitka" list, because i didn't want to make everything so bloody obvious, and because i want you to discover the importance of that frequency data for yourself, so it burns itself in your brain. 7. when you narrow your focus to the words that are _not_ in the dictionary, and which occur only two or three times in the book, you'll find you can be very productive fixing errors. for many of the words, it'll be obvious what they should be... building a tool that will take you _immediately_ to each word, plus show the scan alongside, will turn you into a _machine,_ an awesome and devastatingly efficient error-fixing machine. for this very first pass, i recommend you look only at words you're confident are scanning errors. (they're easy to spot.) on your next pass, you can look at more questionable words. also pay attention to words with several variants that'll thus sort next to each other. (see the asterisks in the "sitka" list.) it'll almost certainly be the case that one variant is a scanno. (97 times out of 100, it is the one with fewer occurrences.) also, in a system where you're gonna have proofers doing a word-by-word proofing, don't even bother to look at words which look kinda reasonable. and don't ever bother to view any words with only _one_ occurrence. those will be flagged, and it's no less efficient to have the proofer see if it's correct than to have _you_ see if it's correct. you wanna be efficient in preprocessing. efficiency is _the_point_ of preprocessing! 8. the next thing you want to look at are compound-words. my tool also separates compound-words to their own listing. as per usual, perusal of the compound-words will show that some are obviously correct, others are obviously incorrect, and a bunch where a judgment can only happen with a scan. the other thing about compound-words, which you will want your tools to handle, is a check against the rest of the book, to find any other instances of the compound where the parts are separated by a space (two words) or joined (one word)... that information will help you decide how to treat the word. 9. the other thing you're gonna check is end-line hyphenates. remember that i told you _not_ to have the o.c.r. rejoin them. my philosophy is to _retain_ the end-line hyphenates through my final product, but i'm not arguing for that position _here_. you can rejoin the end-line hyphenates if you want to do that. just don't do it until _after_ all of the proofing is done, because having original linebreaks makes a page much easier to proof. and besides, in determining whether or not the rejoined word contains a dash or not, we need to have uncompromised data. if your o.c.r. program destroys that data (by joining the word in the way that _its_ dictionary dictates), then you might just be doing a disservice to the way that the _book_ did things... for your spellcheck, however, you can ignore all of that stuff. internal to your spellcheck tool, rejoin the end-line hyphenate by eliminating the linebreak, and test the resultant compound. if it passes, fine. if not, try again with the dash removed too. if it _still_ doesn't pass, flag _both_ portions of the compound. (note that if you are using _my_ dictionary, mentioned above, it contains no compounds, so you would skip that first check.) 10. so at this point in time, you have a great "bad-words" list. that is half your battle. (that's right, just _half_.) so now you will make your "good-words" list. to do this, just run the text through your dictionary-checker using your "bad-words" list as the dictionary. thus, the output will be all of the words in your text which are _not_ included on your "bad-words" list. (or you can just use my tool, which can also create this list.) from now on, this "good-words" list will be your dictionary... got that? you're not using the huge dictionary file any more. you'll use the much-more-compact "good-words" list instead. so now you have a "bad-words" list and a "good-words" list, which -- taken together -- comprise all words in your book. there are other jobs that you'll do during preprocessing, but this is all the spellcheck work that's needed during that stage. 11. so now we will move on from preprocessing to proofing, which takes us to "flagging" -- highlighting possible errors... a word should be flagged if it appears in the "bad-words" list. a word _might_ be flagged if it's not on the "good-words" list, perhaps in a different way; for instance, yellow instead of red. notice that this is a slightly more nuanced way to do flagging. rather than flag _everything_ that _might_ be wrong, we are gonna flag _only_ the things we really _suspect_ are wrong... underflagging is better than overflagging, because too many flags makes us complacent; we start to check only the flags. it's impossible for your mind to ignore the fact that most of the flags are not actually errors, so it comes to expect that if something is _not_ flagged, it certainly won't be an error. but if you underflag, and the proofer spots an error that was _not_ flagged, it primes them to be attentive to everything... 12. the other thing that's extremely important here is that, when the book is finished, we should have resolved all flags. every word in the book will be there on the "good-word" list, and the "bad-word" list will have shrunk until it disappeared. that is, every bad-word will have been checked, and if it was "ok", it will have been moved to the "good-word" list, and if it was not "ok", it would have been _changed_, which will also remove it from the bad-words list. the words do _not_ have to be physically removed from the "bad-words" list, but if we check every word in the book on the "good-words" list, and find it fine, then the "bad-words" list will be eliminated. this complete "good-words" list is _useful,_ because we can run the full book against spellcheck at any time, and it will come out totally clean. so we do that check periodically, so we know that we haven't compromised the book's accuracy. oh, and just so you'll know, it's quite easy to write the code that does this check. you simply sort the words in the book, eliminating duplicates; then you sort the "good-words" list (if it's not already sorted), and eliminate its duplicates too (shouldn't be any); then the 2 outputs should be _identical._ this lets us envision the proofing process as movement of all words on the "bad-words" list to the "good-words" list. (put that image in your head; the visualization has utility.) to help facilitate that movement, you need to make it easy for proofers to put words on the "good-words" list, which is why -- on my proofing site -- i let them add all the words for a single page to the "good-words" list with one button-click. (another option is a button-click for each individual word.) the flip-side is that, in order to have a page be considered as "finished", all flagged words on that page _must_ be cleared. remember, words move from "bad-words" to "good-words". that's good enough for now. any questions on this? ;+) -bowerbird

2 1

Re: the sitka book from fadedpage is now posted
by Bowerbird＠aol.com 07 Apr '10

07 Apr '10

dakretz said: > This comes up all the time. it has been suggested before, yes... > a. It's socially unacceptable by acclamation. ...and is usually deemed socially unacceptable, true... but i'm not sure that that couldn't be turned around, assuming people are genuinely looking to be tested, in a sincere desire to improve. it is a bit repugnant, in my opinion... but so is the secretive collection of data on proofing accuracy that rfrank is doing now, especially since he's not revealing that to _anyone_, including the very people he is collecting data on... > b. All you prove is that the proofer did or > didn't catch an error you already knew about. well, i don't necessarily agree with that. i'd believe your ability to catch the introduced errors would be highly correlated with your general overall accuracy. so if you value that metric, there's one good purpose. (but, for the record, i believe that metric is valueless.) further, the ability to detect the error _immediately_, and show it to the proofer on-the-spot might well be the very best feedback necessary to get their attention, an argument which hasn't been fully considered before. so there _could_ be some real value in this technique. now, i certainly wouldn't do such error-injection on a proofer without their express approval, because it is entirely too _sneaky_ when you are doing it that way, and jeopardizes the trust-relationship, which is vital. but if a proofer _asked_ for it, i think it would be ok... i never subscribed to the "i need to have some errors to keep myself from betting bored" philosophy... but for a proofer who does, this could well be an answer. bottom line, though, i just don't think it's necessary... -bowerbird

2 1

Re: : the sitka book from fadedpage is now posted
by Bowerbird＠aol.com 07 Apr '10

07 Apr '10

rfrank has responded, over on the fadedpage forums, to the note i sent yesterday on the posted "sitka" book. as i might have expected, he fell on his sword and took full blame for the errors that were present... in some part, that's correct, because some errors _had_ been reported, and he simply failed to check the forum. but that's not really the proper take-away message here. a proofer can miss things, and still hit the "done" button, just because they honestly felt they'd caught everything. even the best proofer, with an accurate self-perception. 2 of the 6 errors that i reported were cases just like that. if the only person who's gonna serve as "back-up" for that proofer is the post-processor, that puts too much stress on the post-processor. that's exactly what d.p. has done, and that's why they have so few volunteers for that task... rfrank is willing to take on that stress, which is why he has fallen on his sword. but that's the wrong approach to take. rfrank is also improving his tools, so they catch those glitches. that's good, and it's part of the reason i reported these errors. plus he'd already improved his workflow, to catch _italics_... his improved workflow might work, _if_ the o.c.r. recognizes the styling correctly. but on any books with heavy formatting, going back and reinserting the formatting might be a real pain. a better approach, in my view, would be the one that rfrank started with, when he put up his site, which is to encourage the volunteers to do _both_ the proofing and the formatting. it's really _not_ that difficult to do both these tasks together. this is especially true if you give people a _formatted_display_, because then the obtrusive markup is cleared from the screen, and it's replaced by a rendering that resembles the actual scan. i demonstrated this technique with my own proofing site, and showed the additional strength that questionable words can be highlighted in a different color, maximizing the value of a flag. *** rfrank said: > But the most important thing I've concluded is that > the majority of reportable errors in Sitka are chargeable > to the post-processor (me) and not the roundless system. see, there's the "experimenter bias" that i was talking about; he'd rather take the blame himself than blame his system... i believe in the roundless system too. i believe in it so much, so strongly, that i believe the evidence can stand up for itself. > We don't have nearly enough participation by a realistic > cross-section of typical users here at fadedpage to > conclude anything based on real science or real statistics. > All I can say is that I am liking the roundless system more and > more. It seems to be doing its part well and continues to improve. there isn't a "cross-section" of "typical" proofers over at fadedpage, it's true. the proofers there are probably much better than average. and, as shown, even these better-than-average proofers can _miss_ errors. we're all human. we make mistakes. we need to be checked. -bowerbird

4 4

the sitka book from fadedpage is now posted
by Bowerbird＠aol.com 05 Apr '10

05 Apr '10

the sitka book, which was produced over at fadedpage.com, was posted over the weekend, so i will finish my version and do a comparison on it. the fadedpage/p.g. version is here: > http://www.gutenberg.org/files/31862/31862.txt i can, however, already report that the posted version _does_ contain a few errors. from page 19 in the original p-book: > http://z-m-l.com/go/sitka/sitkap019.png "yukali" is incorrectly entered as "yuhali", and "prazdnik" is incorrectly given as "prasdnik". in addition, both words are italicized in the original, but unitalicized in the p.g. version, for a grand total of 4 errors on this single page. (troubling is the fact that "yukali" had another occurrence in the book, and the similarity of these two non-dictionary words in the book's word-list _should've_ triggered closer examination.) ironically, there was a note in the forum about this page, saying that it was marked as "done" incorrectly after _no_ proofing, due to an inadvertent click on a wrong button, a note which also then pointed out these changes which needed to be done, but evidently rfrank missed that note. and with nobody else doing that page, the errors persisted. plus there are a few other errors i have already noticed... the word "sheetkah", on page 25, is missing its italics: > http://z-m-l.com/go/sitka/sitkap025.png and the period after "december 14th" should be a comma, in the footnote (#25) which was on page 80 in the p-book: > http://z-m-l.com/go/sitka/sitkap080.png so there are some things worth pointing out about all of this... the first is that, by _my_ standard of one-error-per-10-pages, these 6 errors are _not_ a damning indictment, not nearly so... in and of themselves, these errors are trivial. don't forget that. of course, you don't like to concentrate four errors on one page. but stuff happens. so in terms of this particular case, for _me_, i don't think it's a big deal. however, in terms of the _workflow_, the fact that a page with 4 errors can float through the system without _anyone_ having had a "second look" at it -- or, indeed, even a _first_ look -- is not a good sign, not a good sign at all. moreover, according to clear results of a poll i ran over at d.p., the majority of people over there believe that 5 errors in a whole _book_ is right at the maximum that they are willing to tolerate as "acceptable". even worse, some of them -- in a defiant act of wishful thinking -- actually have convinced themselves that they really do _attain_ that level of accuracy with the books they do! now, clearly that's ridiculous, and they're just fooling themselves; they don't actually _know_ how many errors are in their books, so they just let themselves believe that there aren't any errors there. but in spite of this break from reality, their _expressed_desire_ is that they release books that have 5 or fewer errors in them... viewed from that perspective, this performance -- with 4 errors on a single page and (at least) 6 in a posted book -- is _bad_... -bowerbird p.s. i'd point to the note that was left about that page, except that rfrank seems to take down the thread for a book once it is posted... once again, this data is important for a proper analysis of the test.

1 0

Full text catalog search
by Joyce Wilson 05 Apr '10

05 Apr '10

The "Full Text" search option here <http://www.gutenberg.org/catalog/world/search> is broken, and has been for at least a year. If the thinking is that it doesn't need to be fixed because some of the alternative searches at the bottom of this page <http://www.gutenberg.org/catalog/> are acceptable substitutes, then it makes sense to remove it as an option on the "Advanced Search" page. To have the option there but never find any results with it is just confusing. Regards, Joyce

1 0

Re: a horizontal proofing interface, line by line
by Bowerbird＠aol.com 03 Apr '10

03 Apr '10

jim said: > suggest you keep working in this direction > as I think it would make a contribution. well, yes and no. it's good for doing a word-by-word proofing... but, as i've shown, that methodology is overkill, at least when it is practiced over an entire book that was properly and aggressively preprocessed, as upwards of 90% of the lines are already perfect. it is good at pulling out the other 10% of the lines, however, and subjecting them to a closer scrutiny. plus it does have utility in doing a _comparison_ between two files, as is shown in this screenshot: > http://z-m-l.com/go/sitka/findlines.png that screenshot also gives you some hints about how i go about slicing the lines, although that's not really too difficult to figure out on your own. -bowerbird p.s. the pinkish lines in that screenshot are lines that have a diff between the two different versions. the numbers by the asterisks are a simple counter for the number of diffs cumulative to that point...

1 0