this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** ok, time to start in with our "final checks" on this book... you might recall that we noticed -- right off the bat -- that the o.c.r. was having a problem misrecognizing the letter "i", especially in conjunction with a lowercase "l" (where the pair were mistaken as a capital-h), and at the start of many words, where the lowercase "i" was often misrecognized as a capital... we took care of all the "li"-misrecognized-as-capital-h errors when we reviewed the spellcheck, as those words got flagged. but the improperly-cased i-words passed spellcheck just fine, so we need another check, to catch them and similar errors... this check is a relatively easy one to do -- you just look for capitalized words which are not at the start of a sentence... the problem is that this check gives too many false-alarms, because it finds _names_, not just improperly-cased words. on the other hand, you can turn that _flaw_ into an _asset_. you might remember that we _want_ to find names, because we want to add 'em to our customized spellcheck dictionary. it ends up that this routine is a _great_ way to locate names. what you do is you search for mid-sentence capitalized words. when you find one, you check it against the regular dictionary. if it's not there, you put the word on the "possible names" list. if the word is in the dictionary, you flag it as a "possible error". sometimes the error is due to an inappropriate upper-casing. other times it can also be due to some missing punctuation, or a period that was misrecognized as a comma, and so on... but most of the time in this book, it was inappropriate casing, so the refinement to our routine is an extremely welcome one. this refinement doesn't remove all of the false-alarms (or all of the "misses", for that matter, if you know signal-detection). there are a good number of personal names that _do_ appear in a dictionary, such as "smith" and "brown" and "tanner", etc. (there's a humorous example in our book here, which dotes on the many contributions from the classic epic poet "homer", a name that passes spellcheck _nicely_, praise be to babe ruth, albeit probably not in a way the greek storyteller anticipated.) likewise, there are sometimes mid-sentence initial-caps that are _not_ o.c.r. errors. (you might recall that the current book gave honorific capitalization to "idealism", "realism", and such.) but all in all, this is a very worthy routine, which finds errors in a good number, without too many misses or false-alarms... so that'll be the first part of our program today. *** the second part will check the balancing of our doublequotes. you might remember that i fixed the doublequotes manually, in the initial clean-up, since there weren't very many of them. but it's always good to have the computer go back and check your work, make sure you did it correctly. ("to err is human.") besides, there might be some that the o.c.r. just plain missed. or there might be some that the _printer_ accidentally forgot. the best check, for doublequotes, is to split on paragraphs, and check that each one starts with an open-doublequote (if it has any doublequotes), and then alternates between close-doublequotes and open-doublequotes, ending with -- preferably -- a doublequote. it is grammatically correct to not have the close-doublequote when a quoted passage continues with the next paragraph. but, for the most part, throughout the file, we're gonna expect alternation between an open-doublequote and a close-doublequote, so we can write a routine that actively tests for that specific pattern... to be exact, we will set up a print "toggle", and _output_ any text between an open-doublequote and a close-doublequote. and we'll wave a flag if we encounter two open-doublequotes in a row, or (especially) two consecutive close-doublequotes. *** the third part of this miscellaneous clean-up program today will check any ambiguous paragraphing at the top of a page. the o.c.r. at archive.org has no information about a paragraph that starts at the top of a page, but we need to know that info, so we'll check anything that looks like it involves such a thing. there are three tip-offs. one is sentence-ending punctuation at the end of a page. another is a capital letter at the top of a page. last of all, a third is a short line at the end of a page. any of those tip-offs will result in a check by us of the scan... as you can tell from those tip-offs, this check involves both text at the bottom of one page _and_ at the top of the next. for the goal of getting proper paragraphing, it's enough to view the text at the top of "the next page" -- if it's indented, you have a new paragraph, and if not, you don't -- but you also want to check the text at the bottom of "the one page", because you sometimes find the o.c.r. made an error there. (indeed, it happened once, maybe twice, in the current book.) this check will tell you the pages that you'll need to review, and you will check the pagescans to make your judgment, and then edit the text-file appropriately, if there is a need. *** the final check performed by today's program will be on the runheads. i include these in the z.m.l. version of an e-book, so i want them to be correct. we expect they'll be repetitive, so we ignore 'em if they are, print them if/when they change. *** here's the program:

http://zenmarkuplanguage.com/grapes112.py http://zenmarkuplanguage.com/grapes112.txt

the output is kinda sloppy, so i won't repeat it here. just run the program yourself if you want to see it... let's review our findings on the 4 things we checked... suffice it to say that there were a lot of places where the o.c.r. misrecognized a lowercase-i as uppercase, and a few other places where we had an inappropriate mid-sentence capped word, for one reason or another. what the program also revealed is that i'd made some errors when i manually "fixed" spacey-doublequotes... that's the glitch with "manual" fixes -- human error... just a couple of errors, yes. but errors nonetheless... third, it listed out _16_ possible page-top paragraphs, about half of which will require a reference to the scan. those on 4-5, 7-8, 45-46, 111-112, 112-113 do not, because we can tell the trigger was fired unnecessarily. a few of the rest obviously constitute a new paragraph, specifically those on 168-169, 171-172, and 251-252. and we're tipped to an o.c.r. error at the bottom of 208. finally, we see that we have a couple runheads that have misrecognitions in them, so we'll have to fix those too... *** based on this output, i did corrections to the text-file:

http://zenmarkuplanguage.com/grapes005.txt

*** so here's the same program, run on the new "005" file:

http://zenmarkuplanguage.com/grapes113.py http://zenmarkuplanguage.com/grapes113.txt

as you can see from the output, i didn't catch all of the bugs when i did my edits. again with the human error. but this teaches us something that's extremely important. after you do edits -- automatic or manual, but especially the manual kind -- you need to re-do the check, to ensure that (1) you made all of the changes you intended to make, and that (2) you didn't make any other unintended changes. *** anyway, so i did the corrections _again_ to the text-file:

http://zenmarkuplanguage.com/grapes006.txt

and here's the check:

http://zenmarkuplanguage.com/grapes114.py http://zenmarkuplanguage.com/grapes114.txt

ok, it finally looks clean. whew. thus ends today's lesson. -bowerbird