- gutvol-d - lists.pglaf.org

Project Gutenberg DVD Release Candidate now available for testing
by Aaron Cannon 13 Apr '10

13 Apr '10

Hi all. Finally, I have a release candidate of the new Gutenberg DVD available for testing. Currently, it is only available via BitTorrent. Please download and test, and let me know if you find any bugs. At this point, I'm not looking for suggestions on which new titles to add, but any other feedback is welcome and appreciated. I would not yet recommend distribution of this DVD image for any purpose other than testing. Please do not seed this torrent after April 30, because by then, we should have an official version available. This image is an .iso for a dvd-9, so you will either need a way to open or mount .iso files, or a dual layer DVD burner. You can download the torrent via the following magnet link: magnet:?xt=urn:btih:JB4PMPIXNTYMFCUZABAYYTX56JWQINPI or if your client doesn't speak magnet links, you can download the torrent file from: http://www.fireantproductions.com/pgdvd201004-rc1.torrent Thanks in advance for all the feedback. Aaron Cannon

2 2

spell-check functionality and the sitka book
by Bowerbird＠aol.com 12 Apr '10

12 Apr '10

in discussing how to bring about a spell-check functionality, i talked about the "bad-words" list as a part of that workflow. i posted the "bad-words" list from the o.c.r. for the sitka book. > http://z-m-l.com/go/sitka/sitka-reversedictionary.txt i talked about how the proofing process can be envisioned as movement from the "bad-words" list to the "good-words" list. now that rfrank has posted the "final" version at p.g., we can do the same "dictionary-check" procedure on his finished book: > http://z-m-l.com/go/sitka/sitka-reverseposted.txt you'll see i have introduced blank lines into these files, so as to coordinate their lines so they can be merged into one file. this will allow us to see how each word traversed the process. the merged file is here: > http://z-m-l.com/go/sitka/sitka-reverse-review.html *** go ahead and take a quick look at the words... one thing you might notice is that a number of the words are marked with asterisks. those are the ones that are suspicious, in that they appear to be _variants_ of each other, which might be a good indication of either (1) an o.c.r. misrecognition, or (2) an inconsistency in the p-book which should be corrected... (some people use levenshtein edit-distance to find these words. that's nice, but a plain old review of a sorted list works well too.) i checked those variants against the p-book, and many of them were indeed errors. (the ones marked "ok" at the end were right.) that's how i found many of these consistency errors rfrank missed. oh, by the way, there is still at least one more of those errors that i didn't mention earlier... so if anyone wants to go looking for it... ok, so let's go on to look at the list in other ways... *** for each individual word, we're gonna see how it was handled... the first group are some words that had garbage characters. those words didn't have any direct equivalent in the final file, at least none that were close enough to be sorted similarly... once we get into the lowercase words, we get some matching. and that continues as we get into the words with an initial cap, and on into the compound-words, and then into the numbers... focus on the initial-cap words -- which are primarily names -- and you'll see that the vast majority of these were recognized correctly, in that they persisted through to the final version... about 85% of the initial-cap words were recognized correctly. the same is true of the compound-words, with few exceptions. and most numbers seem to have been recognized correctly too. but lowercase words are more of a mixed bag. some were right, it's true, but a relatively high percentage of them were incorrect, as evidenced by the fact they were changed, one way or another. only about half the lowercase words were recognized correctly. you might remember that i had predicted precisely this pattern. most lowercase o.c.r. words that are in the "bad-words" list are generally misrecognitions, while most initial-cap words are not. indeed, the percentage of correct lowercase words in this o.c.r. was much higher than is normal, because this was not typical o.c.r., in the sense that the scans were clean, but also because rfrank probably did some preprocessing on the raw o.c.r. text, which we can tell because it had very few garbage characters... so what we see is that roughly 75% of these words were _correct_, despite the fact they were "bad-words" (i.e., not in the dictionary). they weren't in the dictionary, but they were "good" in this book... that so many "bad-words" can be correct and unique to the book is why it's important to use a "custom" book-specific dictionary. only about 25% of the "bad-words" were actually misrecognitions. to the extent that you can narrow down your _flagging_ of the "bad-words" to the ones that are _really_ bad, you can relieve your proofers of a _lot_ of unnecessary flags, which is _good_, because false flags sap the attention of proofers unnecessarily. once you've done this analysis a number of times, like i have, you'll come to recognize that it is a very important analysis... -bowerbird p.s. hey, dkretz, thanks for the shoutout! but one correction! i wasn't "baited" into "crossing a line" that got be banned at d.p. i never get "baited" into _anything_. i always know what i'm doing. and i've been banned from enough places that i know how it works, so it wasn't that i "crossed a line". again, i know what i'm doing... no, if i get banned from somewhere, it's something i _anticipated_, and after a consideration of that outcome, decided it didn't matter. which is _not_ to say that i _like_ to get banned, or that i _try_ to, but is _rather_ to say that i won't allow myself be banned _unless_ i have decided that i don't really care whether i'm banned or not... as for "crossing a line", there's no need for it. even though people will generally say that i broke some technical rule and that is why i was banned, the truth of the matter is that one only gets banned if one pisses off the person with the power to push the ban button. it has nothing at all to do with "the rules"... it's just raw emotion... oh, and let me say one more thing; it's _nice_ that d.p. still lets me come to their forums and read them. if they tried to prevent that, i _could_ get around it, but it's a hassle. so i thank them for that...

1 0

more notes from the merry-go-round
by Bowerbird＠aol.com 12 Apr '10

12 Apr '10

gosh i love a merry-go-round... :+( *** here's a few notes, repeated too many times already, but whatchagonnado? if this doesn't make sense to you, that's probably because it wasn't intended for you, so pay no mind. *** any heavy bracket markup like [i]italics[/i] is obtrusive, and that's why proofers complain about it, justifiably... use light markup, like _italics_, and they won't complain. better yet, have them do the proofing with an .html field -- using a text-area field only when editing is required -- and you can use _real_ italics (and highlight it with color). you can also use _color_ to flag your questionable words, which makes the flagging a whole lot harder to "miss"... *** spacey double-quotes are _easy_ to solve if you pay heed to the _paragraphs_. ergo... pay heed to the paragraphs! yes, paragraphs cross page-boundaries... but so what? you start with the first file, and keep track of paragraphs while you proceed to the second, and the third, and so on. there's a ton of redundancy in the quotes -- open, close, open, close, open, close. make use of that redundancy... it's not rocket-science; it's not even difficult programming. get over the mind-blockage you have on this topic. *** and once you _do_ get over that mind-blockage, you just might see that books that use _single-quotes_ for dialog aren't really all that different. "but", you are sputtering, "yes they are, because contractions cause big difficulties!" poppycock. here's a file with a list of contractions (among other stuff): > http://z-m-l.com/customdictionary.txt that file's been up since june of 2007. use that list intelligently to control those contractions... possessives also use the single-quote, but they're easy to deal with as well; just do a little thinking about them. don't give up so easily. _try._ you'll find it's not as hard as you thought. and if you actually honestly _try_, and hit a wall anyway, show me your actual honest efforts, and i will help you... *** "probable markup on this page"? are you purposely trying to be vague? how about listing the italicized words, _specifically_? geez! -bowerbird

1 0

Newby/Hart at Illinois symposium April 15-16
by Greg Newby 12 Apr '10

12 Apr '10

For those in the region, this might be of interest: http://50years.lis.illinois.edu/ PGLAF CEO Greg Newby will join PG founder Michael Hart at a symposium on the U. Illinois campus. Registration is free but limited. The panel with Michael & Greg is scheduled for Thursday April 15 from 1:30-3pm. -- Greg

1 1

my report on that sitka posting
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

ok folks, here's my book report on the "sitka" book which was done over at fadedpage, and posted to p.g. it's #31862 if you want to go and take a look at it: > http://www.gutenberg.org/files/31862/31862.txt *** i'll discuss the high points in the body of this message; appended you'll find the data documenting these points... *** 1. ...stealth scanno... the first thing to note is that a sharp-eyed proofer caught a stealth scanno. my system, which relies on spellcheck heavily, is susceptible to stealth scannos, so i'm always interested in their frequency, and whether or not they will cause a serious problem in comprehension. in this book, there was 1 stealth scanno, so they were infrequent in this book. further, it wasn't a bad one... a "heeded" was misrecognized as "needed", so no big deal. *** 2. ...publisher corrections made; here's a list of 15... rfrank made his digitization philosophy very clear here, as he made a number of _corrections_ to the original book. and i agree! i'm firmly against the "transcription" model, which holds that we should merely reproduce the print-book. i maintain we are _republishing_ the book, and therefore i strongly believe we need to correct any errors we find. and like me, roger doesn't bother with a "list of changes", either. i don't think it's necessary to make such a list. they're boring. just make the change and keep on moving... however, there is one aspect to note about this, which is that means rfrank sets a higher standard for his work, and if we find errors in the p-book which he should've found, then he is now accountable for having missed those errors. *** 3. ...more fixes, requiring research outside the book... to his credit, rfrank went out of his way to check things, so as to make sure that he was _only_ fixing actual errors, and not introducing bad changes. sure, it's not too hard to look up a sailor's name on wikipedia to spell it right, or to check that a word means "an edit from the czar", but it takes time, and it improves the output, so give kudos... *** 4. ...he even did "corrections" i might not have done... as is often the case when you adopt a "re-publication" philosophy as opposed to "transcription", some of the changes that you make might not get universal agreement. but that comes with the territory, so you roll with it. *** 5. ...and some i _certainly_ wouldn't have done... and you continue rolling, even when disagreement is thick. *** 6. ...and yet he didn't do some that i woulda done... everybody has their own opinions... :+) i won't count these as "errors". but _i_would_ do them differently... *** 7. ...and i disagree on how he did some formatting... once again, everybody's got an opinion... for the first 2 cases shown, i put a comma after the period, just so the following lowercase letter wouldn't throw a flag every time i did that check. the third case is a bit more complicated... i believe that the formatting rules used by rfrank -- and d.p. in general -- are slightly misguided... first of all, they are arbitrary, with wiggle-room, which means they're hard to understand and interpret. i think rules that are firmly grounded work better... second, in many situations the rules instruct us to put the formatting toggles _inside_ the word itself (at least inside the punctuation which surrounds it). in this case, for instance, the _italicized_ word is placed inside some _non-italicized_ parentheses. so an italic letter butts a non-italic parenthesis. but _none_ of the rendering engines we have today (like browsers and e-book apps) are capable enough to make that look reasonably good, let alone nice. indeed, most of the time it looks absolutely dreadful; a shift from non-italic to italic (and back) is ugly. so even when it doesn't "make sense" grammatically, my recommendation is to italicize the entire string, it looks better, so that's what i do; you should too. *** 8. ...the previous errors... ok, now we get down to the brass tacks. were there any errors in the posted book? well, yeah. i've already pointed out 6. they are repeated below, to refresh memory. but there are more errors... *** 9. ...there are hyphenation differences... rfrank has clearly shown that he is more than willing to make changes to the book, so as to correct errors, and for consistency. for instance, he changed several cases of the name "wrangel", consistent to "wrangell". so if we now _find_ errors or inconsistencies, even if they were present in the original book, we'll hold him accountable for not having made the proper corrections. it's only fair. and yes, the book had several hyphenation inconsistencies. rfrank may have found and fixed some, i don't know, but i _do_ know that there were some that he did _not_ fix... do you like the word "sealion"? or is "sea-lion" better? i like the second. the book was split 50/50. rfrank did 3 of them one way, then left the final 1 the other way... i'd call the error-count 2, but you can call it 1, or 3. there were 2 cases of "far-off" and 1 case of "far off", so i'd go with "far-off", and raise the error-count by 1. there was 1 "far-seeing" and 1 "far seeing". ironically, they're both on the same page. talk about short-sighted. call 1 of 'em an error. there was 1 "guest-house" and 1 "guesthouse", so 1 error. and 1 "ice-houses" and 1 "icehouses", so again, 1 error... so let's say 6 hyphenation errors, due to inconsistency... *** 10. ...and the other consistency errors... i show 11 other consistency errors, all on names... feel free to argue about any of them. if you can find someone who's willing to argue them, that is. so far, 17 new errors... *** 11. ...and finally, the last 2 errors... there was 1 case of some missed formatting, as shown clearly in the scan for this page: > http://z-m-l.com/go/sitka/sitkap025.png _barabaras_, barabaras, ^^^^^^^^^^ it's humorous to note that this missed italic was on the same page as the missed italic i reported before. but nobody said anything about "barabaras" earlier... last but not least, 1 misspelled word: "gooch-heat", which should have been "gooch-haet" -- wolf-house... yes, it was printed wrong, but it shoulda been caught. so 2 more, for a sum of 19 new errors, plus the original 6, making a grand total of 25. that's twice my usual standard -- 1 error per 10 pages -- but since _most_ of these errors were also present in the p-book (along with many more which were corrected by rfrank), i'd call this a good digitization. *** in conclusion... many people will say that most of these errors are trivial. i wouldn't argue with them. at the same time, i'm compulsive enough to fix the trivial. i don't want any errors in my books. even "trivial" ones... i'm guessing that roger will want to fix these errors too. i'm also guessing that roger will say most of these errors can be laid at the feet of the postprocessor, so that his "roundless" system has passed this test with flying colors. and, in a sense, he's right, in that most of these errors _should_ be located within the realm of the postprocessor. (or -- to once again stress my model -- the preprocessor.) but to give the postprocessor the _luxury_ of enough time to do the job of catching errors like these, we must have the proofers perform as many duties as they possibly can. when a page can conceivably be seen by only _one_ proofer, then a postprocessor simply can't have enough faith that each page is correct, and will inevitably perform checks they would not do if they believed every page was solid... if you're worried about simple things like missing italics, you just won't have the focus to look for more subtle stuff. use your proofers to make sure every page is rock-solid. give the postprocessor the luxury to _polish_ the book... -bowerbird p.s. here's a "bonus error", thrown in for good measure. "blockhouse" isn't capitalized elsewhere in the book, and if we did capitalize it here, we'd capitalize "upper" too. > near the site of the upper blockhouse. Her successor, > near the site of the upper Blockhouse. Her successor, > ===========================^^^^^^^^^^^^^^^^^^^^^^^^^^ =============================================================== 1. ...stealth scanno... > and was needed. Captain A. Holmes A'Court, > and was heeded. Captain A. Holmes A'Court, > ========^================================= 2. ...publisher corrections made; here's a list of 15... > Narative of a Voyage Round the World. > Narrative of a Voyage Round the World. > ===^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > pupil.' > pupil.'" > =======^ > an air of prosperity prevaded the place. > an air of prosperity pervaded the place. > ======================^^================ > Her successor, the sceond Princess > Her successor, the second Princess > ====================^^============ > in these waters bought skins for mere trifles, some for > in these waters, bought skins for mere trifles, some for > ===============^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > bath house > a bath house > ^^^^^=^^^^ > island but > islands but > ======^^^^ > ungents, combine to make > unguents, combine to make > ===^^^^^^^^^^^^^^^^^^^^^ > Hudson Bay Co. the Russian ships that sailed > Hudson's Bay Co. the Russian ships that sailed > ======^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > of the educational institution of Sitka. > of the educational institutions of Sitka. > ========================^^^^^^^ > The liquid refreshments serve to him > The liquid refreshments served to him > =============================^^^^^^^ > the villianous liquor called "hoochinoo" > the villainous liquor called "hoochinoo" > ========^^============================== > island to the broad Pacific. What were the thoughts > islands to the broad Pacific. What were the thoughts > ======^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > or "Ranche," there is choice of two streets, > or "Ranche," there is a choice of two streets, > ======================^^^^^^^^^^^^^^^^^^^^^^ > and three who names I do not know. > and three whose names I do not know. > =============^^^^^^^^^^^^^^^^^^^^^ 3. ...more fixes, requiring research outside the book... > Priest Vasili Michaeloff Ocueredin, > Priest Vasili Michaeloff Ocheredin, > ===========================^======= > of the harvest. Her captain, Entienne Marchand, > of the harvest. Her captain, Etienne Marchand, > ==============================^^^^=^^^^^^^^^^^ > ukaze, > ukase, > ===^== 4. ...he even did "corrections" i might not have done... > all the Russians to the Republic > all the Russias to the Republic > ==============^^^^^^^^^^^^^^^^^ > of the American troops, Gen. Jeff C. Davis, > of the American troops, Gen. Jefferson C. Davis, > =================================^^^^^^^^^^ > is called Kosters Trail. The first > is called Koster's Trail. The first > ================^^^^^^^^^^^^^^^^^^ 5. ...and some i _certainly_ wouldn't have done... > articles, carved with totemic design > articles, carved with totemic designs > ==================================== 6. ...and yet he didn't do some that i woulda done... > --The Author > The Author > ^^^^^^^^^^^ > in the Bering Sea, > in Bering Sea > ===^^^^^^^^^^ 7. ...and i disagree on how he did some formatting... > to the Hudson's Bay Co., the Russian ships that > to the Hudson's Bay Co. the Russian ships that > =======================^^^^^^^^^^^^^^^^^^^^^^^ > Total, 400. Ib., p. 52. > Total, 400. Ib. p. 52. > ===============^^^^^^^ > of New Archangel _(Novo Arkangelsk),_ > of New Archangel (_Novo Arkangelsk_,) > =================^^===============^^^ 8. ...the previous errors... > _yukali_ (dried salmon), > yuhali (dried salmon), > ^^^^^^^^^^^^^^^^^^^^^^ > _prazdnik_ (holiday) > prasdnik (holiday) > ^^^^^^^^^^^^^^^^^^ > _Sheetkah_, > Sheetkah, > ^^^=^^^^^ > Seattle Intelligencer, December 14th, 1868; > Seattle Intelligencer, December 14th. 1868; > ======================^^^^^^ 9. ...there are hyphenation differences... sea-lions (#19, but at line-end) sealion (#29) sealion (#48) sea-lion (#101) > and sea-lion meat from Kodiak, and > and sealion meat from Kodiak, and > =======^^^^^^^^^^^^^^^^^^^^^^^^^^ > adorned with sea-lion heads > adorned with sealion heads > ================^^^^^^^^^^ far off (#35) far-off (#42) far-off (#80) > in the far-off possession of the Czar. > in the far off possession of the Czar. > ==========^=========================== far seeing (#16) far-seeing (#16) (yes, inconsistent on the same page) > wealthiest and most far-seeing of the leaders > wealthiest and most far seeing of the leaders > =======================^^^^^^^^^^^^^^^^^^^^^^ > being entertained in the guest-house were ...but yet... > went to the guesthouse of the kwan. All the > The ice-houses were near the outlet of ...but yet... > icehouses was laden on the ship 250 tons, and 10. ...and the other consistency errors... > in Chicagof Island, sent his ship's boat > in Chichagoff Island, sent his ship's boat > =======^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Admiral Chicagof, Minister > Admiral Chichagof, Minister > ============^^^^^^^^^^^^^^ > and Chicagof." (A Voyage Round the World, Lisianski, p. 235.)] > and Chichagof." (A Voyage Round the World, Lisianski, p. 235.)] > ========^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > of Golden California. Captain Hagemeister came to relieve him, > of Golden California. Captain Hagmeister came to relieve him, > =================================^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > of St. Michael > of St. Michaels > ============== > stir the lust for vengeance. The Keeksitties, > stir the lust for vengeance. The Keeksittis, > ==========================================^^ > [Footnote 19: Globokoe Lake was sounded to > [Footnote 19: Golobokoe Lake was sounded to > ===============^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Prince Dmitri Maksoutoff, Dec. 2, 1863, to Oct. 18, 1867.] > Prince Dmitri Maksoutof, Dec. 2, 1863, to Oct. 18, 1867.] > =======================^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Annahootz, the friendly Kokwanton war chief, > Annahootz, the friendly Kokwantan war chief, > ===============================^============ for these two, we'd have to research which variant is correct: ...in one place, we have one spelling... > the mouth of the Indian River, or _Kolosh Ryeku_. ...but in another place it's spelled differently... > the river, known as the _Kolosh Ryeka_, by the Russians ...in one place, we have one spelling... > Captain Leontius Andreanovich Hagemeister ...but in another place it's spelled differently... > Leonti Andreanvich Hagemeister, Jan. 11, 1818, to Oct. 24, 1818. ...getting now down to some smaller points... ...for consistency, and clarity, i'd change these lines... > Total, 400, Ib. p. 52. > their own account. Id. Vol. 2, p. 38. ...i would use "ibid" in both, replacing the "ib." and "id."... ...but i wouldn't call a "failure" to do this "an error"... ...for consistency, i would also change this reference... > Russian American Archives. Corr. Vol. I, p. 275. ...so that it matched with this reference... > Russian American Archives, Correspondence, Vol. II, No. 108. ...but i wouldn't count a "failure" to do so as "an error"... ...and, to finish off this consistency section... ...i'd do research to find out how these archives are named... > Russian American Archives. Corr. Vol. I, p. 275. > their own account. Id. Vol. 2, p. 38. ...because in the top line, it's "vol. i", roman-style... ...while in the bottom line, it's "vol. 2", arabic-style... ...obviously, one of these versions is incorrect... ...but so much for the trivial obsessive-compulsive points... 11. ...and finally, the last 2 errors... _barabaras_, barabaras, ^^^^^^^^^^ gooch-heat gooch-haet ========^^

1 0

so let's talk about my collaborative proofreading site
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

ok, so let's talk about my collaborative proofreading site... yeah, that's a real knee-slapper, isn't it? :+) ok, i'll talk, and y'all can just sit and listen. or go watch t.v. *** to see what i'm talking about, you can visit this u.r.l.: > http://z-m-l.com/go/sitka/editr.pl when you go there, you're proofing the "sitka" book... unlike other proofing systems, pages are not "assigned" to you. you can go and proof any page you wanna proof. even go back the next day and proof it again, if you like. you'll see a row of buttons at the top-left of the screen, which includes one that says "go", and "prev", and "next"... navigating the pages... "prev" and "next" are what you'd expect. they take you to the previous page or the next page in the book, obviously. at the end of that row of buttons, you'll see an edit-field that has a number in it. (or perhaps a letter and numbers.) that number is the pagenumber. (and, ergo, the filename.) if you want to jump to a specific page, put its number in the edit-field, and then click the "go" button. boom, you're there. the ability to navigate the book at will, and to proof any page you like, can be extremely powerful once you learn to use it. certifying a page as clean... there's also an "ok" button. that's what you will click when you've proofed a page and found nothing at all to change. by clicking "ok", you indicate you certify the page is clean. searching the book for a string... next to "ok" is another edit-field and a "find" button next to it. this field lets you enter a search-term, and then when you click "find", a screen appears that lists pages with that search-term... the line containing the word is shown, with a link to its page. if you want to visit any of these pages, you can open them in a separate tab/window. and it's fine to open a bunch at one time. for instance, say you want to check the sitka chapter-headers. enter the term "chapter" in the edit-field, and then click "find", and you will get a list that includes the table of contents page as well as each of the nine pages where each chapter starts... (red lines are a case-insensitive hit; black are case-sensitive.) there are many other ways you can use the search functionality, but we'll save the discussion about all of those for a later time... to return from the search-results page back to a proofing page, just use the "back" button in your browser. you'll have to clear the search-term out of the edit-field, or it'll do the search again. (i should make it so it only does the search when you click the "find" button, but that's not the way it works now, sorry folks.) feel the power with the "command" field... the edit-field where you enter your search-term also serves as a "command" field. i'll eventually have a number of commands that you can issue, but for now there's just a couple of them... showmap... the first command is "showmap". enter that, and click "find". the program will show the "map" of files comprising the book. each one is a clickable link, so -- as before -- you can open any pages you like in separate tabs and proof them just fine... ("listcat", short for "list catalog", is a synonym that works too.) concat... another command is "concat". enter that, and click "find", and the program will concatenate all the text files into one, and put them on a web-page for you. this will allow you to look at the entire book, save it to your machine, and so on... you can also use the browser's "find" command, so "concat" is useful when you need more context to a "find" operation than the single line output from the native find command... showcustom... a third command is "showcustom". this command spits out the "custom dictionary" that has been created for the book. *** that's enough for now. give you a little toy to play with over the weekend, if you like. we'll discuss more stuff next week. anyway, thanks for the little chat. so, how was the t.v. show? -bowerbird

1 0

Re: how to create a spellcheck workflow
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

gardner said: > what use is preserving or fixing the page headings? > If they can be mechanically fixed, they were not worth much. well, to take your second sentence first, the runheads cannot _always_ be mechanically fixed. sometimes they contain text that describes the current point in the chapter, like an outline, and aren't just limited to a boring recitation of title and author. and returning to the question, runheads are useful since they help you keep your bearings in the book. even when they are nothing but title/author, they help keep recto/verso straight... plus -- especially when you're working on multiple projects -- it's useful to have the reminders about which book you're in... because without 'em, after a time, the pages all look the same. > Do you include or exclude running headings in > your word-count dictionary analysis? Does it matter? never paid much attention, so i guess it doesn't matter much. but runheads are definitely excluded from search operations. -bowerbird

1 0

why isn't distributed proofreaders helping fadedpage?
by Bowerbird＠aol.com 09 Apr '10

09 Apr '10

why isn't distributed proofreaders helping fadedpage? the site needs proofers, and d.p. has an _excess_, to the degree that they are actively attempting to stunt the work being done by the p1 volunteers... so why not send some people over to fadedpage? rfrank has done a lot for distributed proofreaders. and fadedpage is showing d.p. how they can make a roundless system work, and that it _does_ work... so why is d.p. being so stingy? help your brother! -bowerbird p.s. and to start, some of you people who are now working at fadedpage can occasionally make a post in a forum at d.p. inviting people to try fadedpage...

1 0

how to create a spellcheck workflow
by Bowerbird＠aol.com 08 Apr '10

08 Apr '10

ok, well gee... i had gotten the firm impression that rfrank had read the d.p. forum thread where the development of their "wordcheck" was discussed (thoroughly -- 30 pages). d.p. calls it "wordcheck", but it's basically spellcheck. so why did i think he had read it? because he incorporated several of the things which i had suggested (to no good effect) in that thread... i won't bother reciting all of those particulars, since he might well have come up with the ideas himself, and nobody really cares anyway, not even me... but i mention it because now i have been convinced that roger must _not_ have read that thread, based on some current discussions on his fadedpage site... or, if he did read it, he didn't "get it", but given those current discussions, he might now be more receptive. so i will run through a quick little "how-to" refresher on how to design and build a spellcheck functionality, and incorporate it into the overall workflow of a book, so roger has the benefit of my wisdom here... ;+) as you'll see, this will hit a very wide variety of topics. note that what we're talking about here is _primarily_ executed during _preprocessing_, but there are some follow-on thoughts that apply to the proofing as well. *** 0. set o.c.r. parameters correctly. don't dehyphenate! name your files wisely. look both ways before crossing. 1. the first thing you should do with your o.c.r. results is to make the few global changes you can do _blindly_. the very first one is to strip trailing spaces from all lines. another will be to replace two spaces with a single space. yet another is to change a linebreak-doublequote-space combination to eliminate the space, since it's superfluous. likewise, change all cases of space-doublequote-linebreak. you get the drift. oh, by the way, don't do what d.p. does! _retain_ the runheads, and pagenumbers; you need 'em... 2. the next thing you should do is clean up the runheads. this has little bearing on our general "spellcheck" topic, but i include it here because it's always your second step. 3. the third thing on your list is to fix all paragraphing... again, not much to do with spellcheck, but it _is_ step #3. 4. now we can focus quite specifically on spellcheck stuff. we will take the o.c.r. and run it through a program i wrote which pulls out all the words _not_ present in its dictionary. i think rfrank has his own program that does the same thing, or something similar enough. i'll make mine available too. this is the first draft of your "bad-words" list for this book. (note this is not the same as how d.p. defines "bad words".) in this regard, use a good dictionary in this first check here. (this is something that rfrank hasn't done correctly thus far.) the dictionary i use is quite good, and it can be found here: > http://z-m-l.com/go/regulardictionary.txt just so you have a feel for the output from this program, i have posted it for the "sitka" book we've been discussing: > http://z-m-l.com/go/sitka/sitka-reversedictionary.txt that output was generated in 5 seconds, so it's pretty fast... 5. you'll see, on viewing your list of supposed "bad-words", that a bunch are not "bad-words" at all. some of 'em will be character-names, or jargon specific to your particular book. some will be hyphenated fragments, some compound-words. you can delete these words from the list now, if you want, but you shouldn't necessarily feel a great need to do that. if you're looking at the list of words from the "sitka" book, you will also notice that i separated the initial-caps words from all-lowercase ones. there is a good reason for that. due to proper names, _most_ initial-cap words are correct, whereas most of the all-lowercase words are _incorrect_... separating the lists makes it easier to focus your attention. 6. your dictionary-check program should also spit out the _frequency_ of each bad-word. you'll use that information to cull some words from this list of "bad-words". this you _will_ want to do, most definitely, so _sort_ on frequency. i fought tooth-and-nail on this with d.p. (i lost, of course), but you can take this to the bank that as long as there are a mere 4-plus occurrences of a specific string in the o.c.r., you can (i.e., should) delete it from this list of "bad-words". yes, some of the words _might_indeed_ be bad, but unless you're positive of that, you should delete 'em from the list. and yes, this means that those words will _not_ be flagged. but trust me, if there's 4 or more occurrences of a scanno, your proofers will find at least _one_ . and whenever they find _one_ "bad-word" that wasn't on that "bad-word" list, you will automatically search the rest of the book, and thus find those _other_ occurrences as well, so you can fix them. i did _not_ include frequency information in my "sitka" list, because i didn't want to make everything so bloody obvious, and because i want you to discover the importance of that frequency data for yourself, so it burns itself in your brain. 7. when you narrow your focus to the words that are _not_ in the dictionary, and which occur only two or three times in the book, you'll find you can be very productive fixing errors. for many of the words, it'll be obvious what they should be... building a tool that will take you _immediately_ to each word, plus show the scan alongside, will turn you into a _machine,_ an awesome and devastatingly efficient error-fixing machine. for this very first pass, i recommend you look only at words you're confident are scanning errors. (they're easy to spot.) on your next pass, you can look at more questionable words. also pay attention to words with several variants that'll thus sort next to each other. (see the asterisks in the "sitka" list.) it'll almost certainly be the case that one variant is a scanno. (97 times out of 100, it is the one with fewer occurrences.) also, in a system where you're gonna have proofers doing a word-by-word proofing, don't even bother to look at words which look kinda reasonable. and don't ever bother to view any words with only _one_ occurrence. those will be flagged, and it's no less efficient to have the proofer see if it's correct than to have _you_ see if it's correct. you wanna be efficient in preprocessing. efficiency is _the_point_ of preprocessing! 8. the next thing you want to look at are compound-words. my tool also separates compound-words to their own listing. as per usual, perusal of the compound-words will show that some are obviously correct, others are obviously incorrect, and a bunch where a judgment can only happen with a scan. the other thing about compound-words, which you will want your tools to handle, is a check against the rest of the book, to find any other instances of the compound where the parts are separated by a space (two words) or joined (one word)... that information will help you decide how to treat the word. 9. the other thing you're gonna check is end-line hyphenates. remember that i told you _not_ to have the o.c.r. rejoin them. my philosophy is to _retain_ the end-line hyphenates through my final product, but i'm not arguing for that position _here_. you can rejoin the end-line hyphenates if you want to do that. just don't do it until _after_ all of the proofing is done, because having original linebreaks makes a page much easier to proof. and besides, in determining whether or not the rejoined word contains a dash or not, we need to have uncompromised data. if your o.c.r. program destroys that data (by joining the word in the way that _its_ dictionary dictates), then you might just be doing a disservice to the way that the _book_ did things... for your spellcheck, however, you can ignore all of that stuff. internal to your spellcheck tool, rejoin the end-line hyphenate by eliminating the linebreak, and test the resultant compound. if it passes, fine. if not, try again with the dash removed too. if it _still_ doesn't pass, flag _both_ portions of the compound. (note that if you are using _my_ dictionary, mentioned above, it contains no compounds, so you would skip that first check.) 10. so at this point in time, you have a great "bad-words" list. that is half your battle. (that's right, just _half_.) so now you will make your "good-words" list. to do this, just run the text through your dictionary-checker using your "bad-words" list as the dictionary. thus, the output will be all of the words in your text which are _not_ included on your "bad-words" list. (or you can just use my tool, which can also create this list.) from now on, this "good-words" list will be your dictionary... got that? you're not using the huge dictionary file any more. you'll use the much-more-compact "good-words" list instead. so now you have a "bad-words" list and a "good-words" list, which -- taken together -- comprise all words in your book. there are other jobs that you'll do during preprocessing, but this is all the spellcheck work that's needed during that stage. 11. so now we will move on from preprocessing to proofing, which takes us to "flagging" -- highlighting possible errors... a word should be flagged if it appears in the "bad-words" list. a word _might_ be flagged if it's not on the "good-words" list, perhaps in a different way; for instance, yellow instead of red. notice that this is a slightly more nuanced way to do flagging. rather than flag _everything_ that _might_ be wrong, we are gonna flag _only_ the things we really _suspect_ are wrong... underflagging is better than overflagging, because too many flags makes us complacent; we start to check only the flags. it's impossible for your mind to ignore the fact that most of the flags are not actually errors, so it comes to expect that if something is _not_ flagged, it certainly won't be an error. but if you underflag, and the proofer spots an error that was _not_ flagged, it primes them to be attentive to everything... 12. the other thing that's extremely important here is that, when the book is finished, we should have resolved all flags. every word in the book will be there on the "good-word" list, and the "bad-word" list will have shrunk until it disappeared. that is, every bad-word will have been checked, and if it was "ok", it will have been moved to the "good-word" list, and if it was not "ok", it would have been _changed_, which will also remove it from the bad-words list. the words do _not_ have to be physically removed from the "bad-words" list, but if we check every word in the book on the "good-words" list, and find it fine, then the "bad-words" list will be eliminated. this complete "good-words" list is _useful,_ because we can run the full book against spellcheck at any time, and it will come out totally clean. so we do that check periodically, so we know that we haven't compromised the book's accuracy. oh, and just so you'll know, it's quite easy to write the code that does this check. you simply sort the words in the book, eliminating duplicates; then you sort the "good-words" list (if it's not already sorted), and eliminate its duplicates too (shouldn't be any); then the 2 outputs should be _identical._ this lets us envision the proofing process as movement of all words on the "bad-words" list to the "good-words" list. (put that image in your head; the visualization has utility.) to help facilitate that movement, you need to make it easy for proofers to put words on the "good-words" list, which is why -- on my proofing site -- i let them add all the words for a single page to the "good-words" list with one button-click. (another option is a button-click for each individual word.) the flip-side is that, in order to have a page be considered as "finished", all flagged words on that page _must_ be cleared. remember, words move from "bad-words" to "good-words". that's good enough for now. any questions on this? ;+) -bowerbird

2 1

Re: the sitka book from fadedpage is now posted
by Bowerbird＠aol.com 07 Apr '10

07 Apr '10

dakretz said: > This comes up all the time. it has been suggested before, yes... > a. It's socially unacceptable by acclamation. ...and is usually deemed socially unacceptable, true... but i'm not sure that that couldn't be turned around, assuming people are genuinely looking to be tested, in a sincere desire to improve. it is a bit repugnant, in my opinion... but so is the secretive collection of data on proofing accuracy that rfrank is doing now, especially since he's not revealing that to _anyone_, including the very people he is collecting data on... > b. All you prove is that the proofer did or > didn't catch an error you already knew about. well, i don't necessarily agree with that. i'd believe your ability to catch the introduced errors would be highly correlated with your general overall accuracy. so if you value that metric, there's one good purpose. (but, for the record, i believe that metric is valueless.) further, the ability to detect the error _immediately_, and show it to the proofer on-the-spot might well be the very best feedback necessary to get their attention, an argument which hasn't been fully considered before. so there _could_ be some real value in this technique. now, i certainly wouldn't do such error-injection on a proofer without their express approval, because it is entirely too _sneaky_ when you are doing it that way, and jeopardizes the trust-relationship, which is vital. but if a proofer _asked_ for it, i think it would be ok... i never subscribed to the "i need to have some errors to keep myself from betting bored" philosophy... but for a proofer who does, this could well be an answer. bottom line, though, i just don't think it's necessary... -bowerbird

2 1