- gutvol-d - lists.pglaf.org

taking on the textbook gougers
by Bowerbird＠aol.com 26 Sep '11

26 Sep '11

"we're sick and tired of being gouged, and we're not gonna take it anymore". > https://torrentfreak.com/pirate-service-makes-textbook-rentals-last-forever… > http://librarypirate.me/ -bowerbird

2 1

these grapes are sweet -- lesson #17
by Bowerbird＠aol.com 26 Sep '11

26 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** i forgot to tell you i fixed that bug in the search routine. and i'm running it with the latest edited version, which -- because we're out of sequence a bit -- is grapes006.txt... > http://zenmarkuplanguage.com/grapes111.py > http://zenmarkuplanguage.com/grapes111.txt everything that's returned in the "notfound" list looks ok, which means we've finally got clean text. what we need to do now is to include the "notfound" words in the "specials" dictionary, so we can create the dictionary that will constitute _the_custom_dictionary_ for this book... *** after that, the next step is a complete visual run-through; that is, we'll go through every page of our text and check it against its pagescan. there are a couple of objects here: 1) to check that all our _formatting_ is correct throughout, such as paragraphs, blockquotes, headers, and so on, and 2) to find and insert any _styling_ that appears in the book, such as italics, bold, and so on, and 3) a final clean-up q/c. for this task, it's best to use the page-viewer: > http://zenmarkuplanguage.com/grapes203.py -bowerbird

1 0

these grapes are sweet -- lesson #16
by Bowerbird＠aol.com 25 Sep '11

25 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** another demo-only lesson, again with our page-viewer: > http://zenmarkuplanguage.com/grapes203.py there are several big advancements in this version... first, there's now a "toc" button, which shows you a list of the pages (with links) that head a section. (in z.m.l., this is a line preceded by 4 blank lines.) so you have more navigational power than before. next, when you do a search now, it not only lists the _pages_ that contain your search-term (with links), it now lists the actual _line_ that contains the term, which gives a heckuva lot more context on the hit... finally, in a development that increases your power tremendously, you can now specify a _text-file_ as your search term (it must have the ".txt" extension), and the program performs the search of the pages using every _line_ contained within that text-file... (with the restriction that each line is just one word.) say there's a file "simple.txt" containing these lines: > love > books > wonder > mystery > dry-as-dust if you were to give "simple.txt" as the search term, then the program will search for all of those terms. as you can see, such a file exists on my website: > http://zenmarkuplanguage.com/simple.txt so try entering "simple.txt" as your search-term. bingo! a list (with links) of 134 lines with at least one of the five search-terms, from pages 7 to 277. nifty. useful. powerful. that's all for this lesson... -bowerbird

1 0

these grapes are sweet -- lesson #15
by Bowerbird＠aol.com 24 Sep '11

24 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** ok, time to start in with our "final checks" on this book... you might recall that we noticed -- right off the bat -- that the o.c.r. was having a problem misrecognizing the letter "i", especially in conjunction with a lowercase "l" (where the pair were mistaken as a capital-h), and at the start of many words, where the lowercase "i" was often misrecognized as a capital... we took care of all the "li"-misrecognized-as-capital-h errors when we reviewed the spellcheck, as those words got flagged. but the improperly-cased i-words passed spellcheck just fine, so we need another check, to catch them and similar errors... this check is a relatively easy one to do -- you just look for capitalized words which are not at the start of a sentence... the problem is that this check gives too many false-alarms, because it finds _names_, not just improperly-cased words. on the other hand, you can turn that _flaw_ into an _asset_. you might remember that we _want_ to find names, because we want to add 'em to our customized spellcheck dictionary. it ends up that this routine is a _great_ way to locate names. what you do is you search for mid-sentence capitalized words. when you find one, you check it against the regular dictionary. if it's not there, you put the word on the "possible names" list. if the word is in the dictionary, you flag it as a "possible error". sometimes the error is due to an inappropriate upper-casing. other times it can also be due to some missing punctuation, or a period that was misrecognized as a comma, and so on... but most of the time in this book, it was inappropriate casing, so the refinement to our routine is an extremely welcome one. this refinement doesn't remove all of the false-alarms (or all of the "misses", for that matter, if you know signal-detection). there are a good number of personal names that _do_ appear in a dictionary, such as "smith" and "brown" and "tanner", etc. (there's a humorous example in our book here, which dotes on the many contributions from the classic epic poet "homer", a name that passes spellcheck _nicely_, praise be to babe ruth, albeit probably not in a way the greek storyteller anticipated.) likewise, there are sometimes mid-sentence initial-caps that are _not_ o.c.r. errors. (you might recall that the current book gave honorific capitalization to "idealism", "realism", and such.) but all in all, this is a very worthy routine, which finds errors in a good number, without too many misses or false-alarms... so that'll be the first part of our program today. *** the second part will check the balancing of our doublequotes. you might remember that i fixed the doublequotes manually, in the initial clean-up, since there weren't very many of them. but it's always good to have the computer go back and check your work, make sure you did it correctly. ("to err is human.") besides, there might be some that the o.c.r. just plain missed. or there might be some that the _printer_ accidentally forgot. the best check, for doublequotes, is to split on paragraphs, and check that each one starts with an open-doublequote (if it has any doublequotes), and then alternates between close-doublequotes and open-doublequotes, ending with -- preferably -- a doublequote. it is grammatically correct to not have the close-doublequote when a quoted passage continues with the next paragraph. but, for the most part, throughout the file, we're gonna expect alternation between an open-doublequote and a close-doublequote, so we can write a routine that actively tests for that specific pattern... to be exact, we will set up a print "toggle", and _output_ any text between an open-doublequote and a close-doublequote. and we'll wave a flag if we encounter two open-doublequotes in a row, or (especially) two consecutive close-doublequotes. *** the third part of this miscellaneous clean-up program today will check any ambiguous paragraphing at the top of a page. the o.c.r. at archive.org has no information about a paragraph that starts at the top of a page, but we need to know that info, so we'll check anything that looks like it involves such a thing. there are three tip-offs. one is sentence-ending punctuation at the end of a page. another is a capital letter at the top of a page. last of all, a third is a short line at the end of a page. any of those tip-offs will result in a check by us of the scan... as you can tell from those tip-offs, this check involves both text at the bottom of one page _and_ at the top of the next. for the goal of getting proper paragraphing, it's enough to view the text at the top of "the next page" -- if it's indented, you have a new paragraph, and if not, you don't -- but you also want to check the text at the bottom of "the one page", because you sometimes find the o.c.r. made an error there. (indeed, it happened once, maybe twice, in the current book.) this check will tell you the pages that you'll need to review, and you will check the pagescans to make your judgment, and then edit the text-file appropriately, if there is a need. *** the final check performed by today's program will be on the runheads. i include these in the z.m.l. version of an e-book, so i want them to be correct. we expect they'll be repetitive, so we ignore 'em if they are, print them if/when they change. *** here's the program: > http://zenmarkuplanguage.com/grapes112.py > http://zenmarkuplanguage.com/grapes112.txt the output is kinda sloppy, so i won't repeat it here. just run the program yourself if you want to see it... let's review our findings on the 4 things we checked... suffice it to say that there were a lot of places where the o.c.r. misrecognized a lowercase-i as uppercase, and a few other places where we had an inappropriate mid-sentence capped word, for one reason or another. what the program also revealed is that i'd made some errors when i manually "fixed" spacey-doublequotes... that's the glitch with "manual" fixes -- human error... just a couple of errors, yes. but errors nonetheless... third, it listed out _16_ possible page-top paragraphs, about half of which will require a reference to the scan. those on 4-5, 7-8, 45-46, 111-112, 112-113 do not, because we can tell the trigger was fired unnecessarily. a few of the rest obviously constitute a new paragraph, specifically those on 168-169, 171-172, and 251-252. and we're tipped to an o.c.r. error at the bottom of 208. finally, we see that we have a couple runheads that have misrecognitions in them, so we'll have to fix those too... *** based on this output, i did corrections to the text-file: > http://zenmarkuplanguage.com/grapes005.txt *** so here's the same program, run on the new "005" file: > http://zenmarkuplanguage.com/grapes113.py > http://zenmarkuplanguage.com/grapes113.txt as you can see from the output, i didn't catch all of the bugs when i did my edits. again with the human error. but this teaches us something that's extremely important. after you do edits -- automatic or manual, but especially the manual kind -- you need to re-do the check, to ensure that (1) you made all of the changes you intended to make, and that (2) you didn't make any other unintended changes. *** anyway, so i did the corrections _again_ to the text-file: > http://zenmarkuplanguage.com/grapes006.txt and here's the check: > http://zenmarkuplanguage.com/grapes114.py > http://zenmarkuplanguage.com/grapes114.txt ok, it finally looks clean. whew. thus ends today's lesson. -bowerbird

1 0

these grapes are sweet -- lesson #14
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** of course, if there is no lesson #13, then lesson #14 essentially is the unlucky one... so let us use this lesson to discuss a bug... i gave you the list of words that were "notfound" in our book. i know, i know, i'm giving you a lot of dense lists of stuff, so i really can't blame you if your eyes just gloss over, but _still_, i was testing to see if you actually look at this stuff or not... evidently "not", or you woulda noticed these were "notfound": > conceit > conceived > concentration > conception > conceptions > concern > concerned > concerning > concert > conclusion > conclusions well, quite obviously, those words should be in our dictionary. and -- again, obviously -- they _are_. but for _some_ reason, our program didn't find them, so we've gotta track that down... it ends up, it was caused by a rejoined end-of-line hyphenate. (not just an end-of-line hyphenate, but an end-of-page one.) the hyphenate "con-crete" occurs across pages 144-145, and "crete" on page 145 was misrecognized with an initial capital. this caused our code, as written, to misbehave just a little bit, because that mid-word capital-letter confused our sort logic, resulting in a glitch on all the "con..." words after "concrete"... i will provide a fix in the next version of the spellchecker code. *** meanwhile, i've fixing the newly-exposed errors in our text: > http://zenmarkuplanguage.com/grapes004.txt after a few more checks and corrections, it'll be good to go... *** for now though, the code is good enough that we can install it in our page-by-page viewer-program. you can see that here: > http://zenmarkuplanguage.com/grapes202.py i'm "color-coding" the words now. first of all, the "common" words are rendered in light-gray. the list of those "common words" -- 555 of 'em -- is here: > http://z-m-l.com/common555.txt it's amazing how much of the book is these 555 words... next, the british words are blue, "regular" words are gray, "special" words are green, hyphenates are light-blue, and "notfound" words are rendered in red, for best visibility... thus, all the problem words can be noticed at first glance. i've also installed a _search_ capacity... just enter the term to find, and then click the button labeled "sf" ("search for"). you'll be transported to the next page containing the term. clicking "sf" again will take you to the next occurrence, etc. you also get a list of links to other pages with that term... our "engine" for page-by-page stuff is coming right along... -bowerbird

1 0

these grapes are sweet -- lesson #13
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** there is no lesson #13. -bowerbird

1 0

Re: [gutvol-d] these grapes are sweet -- lesson #12
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

roger's right. so is keith. but i am walking a slightly different tight-rope, in that i'm working the elementary-school level, where i'm explaining these concepts to "a novice" and providing actual working code in a language in which i've never ever programmed before now. so there are plenty of more-complicated niceties which are being glossed over at the present time. in spite of that fact, i'm showing how you can take some very crude code and nevertheless do the job of digitization, from raw o.c.r. to polished e-book. because there are people in power over at d.p. who continue to tell their volunteers "it cannot be done", even though both roger and i have proven _it_can_. it is resolutely immoral how much time and energy -- donated by those kind volunteers in good faith -- is being _wasted_ by leaders who are lying to them... -bowerbird

1 0

these grapes are sweet -- lesson #12
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. i hope you enjoyed the brief interlude between our lessons... *** ok, we've done all of our "first-pass" cleaning, so now it's time for us to move on to spellcheck. so let's code a spellchecker... the first thing you need to know is that even though i call it "a spellchecker", it's very different from the spellchecker that you are used to, the one you have inside your wordprocessor. within that environment, a spellchecker has two components: 1) a routine to flag words that are not in the dictionary, and 2) a routine to provide a "suggestion" on the correct spelling. we need the first component. but we don't need the second, since the question about the "correctness" of the spelling is answered by viewing the pagescan, to see what was printed. luckily for us, the first routine is extremely easy to program, while the second is usually much more difficult and complex. so we get to do the simple part and not the hard part. great! *** the first thing we need is a "dictionary" to use as our standard. again, this is not "a dictionary" per se, because there are no _definitions_ included in it, but merely a long list of words... you can find many dictionaries online. here's the one i use: > http://z-m-l.com/go/regulardictionary.txt if you take a look at that, you'll see it's a simple list of words, each one on a line by itself, arranged in alphabetical order... it's reasonably sized, at just under 800k, with 81,500+ words. i've got another dictionary -- i think it's the one rfrank uses in his routines -- that's 1.5 megs. bigger isn't always better. first, it takes more machine-time to use a bigger dictionary. second, and more importantly, you're more likely to miss a real o.c.r. error that just happens to spell some other word. i even think my dictionary is too big... there are "words" in it that only a scrabble player could love. but it's too much work to refine it, and i don't think it's causing errors, so i keep it... as an interesting sidenote, many of the dictionaries online were formed originally by the "moby" research, which used e-texts from project gutenberg as their underlying corpus. *** on the face of it, it's brain-dead simple to check a book. first, you need to split the book up into "words", which you can do my changing all the spaces into linebreaks, and then splitting the file by linebreaks. easy enough... then, a looped routine like this seems like it should work: > if eachword not in thedictionary then print eachword that code has big bugs in it, though. can you spot them? first off, if you looked at the dictionary, it's all-lower-case, so we need to convert eachword into its lower-case version. second, the dictionary contains no punctuation characters, so we need to strip all the leading and trailing punctuation. (anything with internal punctuation is automatically "bad".) but the biggest bug is misrecognitions like "ng" _will_ be found in the dictionary, e.g., in every word ending in "ing". to prevent such "subwords" from being found, we need to slap a linebreak at the start and at the end of eachword, so it will only register a hit if it matches a complete "line". after those little changes, this routine will "work" just fine. it's not a _good_ routine, though, because it has to search too many words -- from the top of the dictionary down -- for every bookword before finding a hit. and on bad words, it has to search the entire dictionary to know it's "not there". *** now, because the dictionary is sorted alphabetically, we can improve our routine so that it is significantly more efficient. for instance, we could split the dictionary into an array, with all the words that start with each letter as a separate item... i.e., all "a" words go in one item, all "b" words, all "c" words... then, we'd just have to search that one specific array-item... (for a word starting with "c", we'd search the "c-word" item.) that would cut down on the unnecessary searching a bunch. *** but we can do even better by sorting the words in our book, so that we will then be dealing with _two_ alphabetical lists. in addition, sorting the words in our book allows us to easily discard the duplicates, so we only search for each word once. here's how we do the check, and why it's so bloody efficient... we have 2 arrays -- our dictionarywords and our bookwords. we set our pointers to start at the top of each list, and we then compare the dictionaryword with the bookword. if the dictionaryword and the bookword match each other, we put the bookword on the list we keep of "found words", and we then move on to the next word in both of our lists. if the dictionaryword is "less than" the bookword -- that is, if it comes _before_ the bookword in an alphabetized list -- then we go on to the next word in our dictionaryword list... finally, if the dictionaryword is "greater than" the bookword, we know we've gone past where the bookword would've been -- if it _were_ in the dictionary -- so it must not be in there, thus we will place that bookword in our "not-found" list, and then move on to the next bookword in our list of bookwords. this is simple logic to code. (easier to program than explain.) *** so now let's go on to the next step in our thinking about this. the "not-found" list is very important, of course. indeed, you could be forgiven for thinking it is the only one that matters. but you would be wrong; the "found" list is extremely valuable. once we have the initial list of "found" words, we're _cooking_. that's because we now have a "dictionary" for this unique book. this file is the one we will use in our spellchecker from now on. because this file is significantly smaller than the full dictionary, our spellcheck routine will work very speedily, which means we can incorporate it as a regular part of all our cleaning routines. so if/when we inadvertently introduce errors, we will catch them. you may find yourself resisting this idea. keep thinking about it, until you come to the realization it _is_ the smartest thing to do. *** there are two other wrinkles i threw into the spellcheck routine. the first is that i included a check for _british_ spellings, which was necessary for this book. (if the file wouldn't have had any, it wouldn't have mattered. since it did, if i had _not_ included the british dictionary, those words wouldn't been "not-found".) the british words get listed separately, so if there are only a few, you can check to see whether they were simple misrecognitions. when there are many, as you can see here, you are assured that the author utilised such spellings with the fullest of intentions... the british dictionary is here: > http://z-m-l.com/go/britishdictionary.txt the other wrinkle is a file i call "specials.txt". this file contains "special" stuff that isn't in the regulardictionary.txt, but which is not unusual to encounter when digitizing a book, things like roman numerals, arabic numbers, famous names, and so forth, much of which requires that its check be done _case-sensitive_. the "specials" dictionary is here: > http://z-m-l.com/go/specials.txt take a good look at the items there, so you get a feel for them... *** ok, so let's run the code: > http://zenmarkuplanguage.com/grapes110.py > http://zenmarkuplanguage.com/grapes110.txt we get a list of "notfound", "british", "specials" and "found" words. i have found it useful to split the "notfound" list into 2 segments, depending upon whether the word starts with an initial capital... the ones which do are usually _names_, and (like most of the text) they generally tend to have been recognized by the o.c.r. correctly. and yes indeed, that appears to be exactly the case we have here: > Arcady > Dante's > De > Emerson's > Farge's > Florio's > Gary's > Goethe's > Hamilton > Haymarket > Hegel's > Hfe > Hke > Hmits > Holinshed's > Hterature > Hve > Hving > Lowell's > MDCCCCII > Mabie > Melos > Norse > Plato's > Quincey's > Selfexpression > Shakespeare's > Spenser's > Symonds's > Tennyson's > Text-books > They'll > Tolstoi's > Wordsworth we do find a few words which do not look like names, but look instead like o.c.r. errors, including a bunch of those starting with a capital-h... further, "selfexpression" clearly just needs to have its hyphen restored. *** next, words which were notfound that start with a lowercase letter are most typically misrecognitions, and that was true in the current case... > acquirv > ageo > appletree > bhnd > certainlytrue > clearlyenough > comparacively > conceit > conceived > concentration > conception > conceptions > concern > concerned > concerning > concert > conclusion > conclusions > conditlons > de > dehvered > eightysix > enhghtenment > en« > escape.[1 > farreaching > generahsations > h'abitually > halfbelieved > halfunderstood > happyformulation > hfe > highminded > hves > illdirected > largelydeveloped > manysided > mediaevalism > nomena > oftrepeated > oi > oijly > onlylive > onlysearching > overmatched > overweighted > personahty > phe > pious-minded > potentiahties > powerfullyorganised > rationahsing > richment > selfsacrifice > selfsatisfaction > throug > uhimate > unbroken- > vitahty > wellwrought > whlch > witliout > wo;-ks most of the words are o.c.r. errors. some of 'em are hyphenates that appear to have lost their hyphen. others are terms that were at the end of a line where a speck was misrecognized as a hyphen, so they were improperly joined (e.g., certainlytrue, clearlyenough). and still others are terms that were end-of-line hyphenates which must _retain_ their hyphen if/when they become rejoined together. but, as with the capitalized words, these words are rather typical. so, in many ways, this looks like the type of list you would expect... but there were some things about this list which jumped out at me. the first is "they'll". you might have seen that i have _no_contractions_ in my dictionary. that's because i've stored them in a separate file, one which i didn't have the current version of the program read in or utilize. just so you know, here's that list of contractions: > http://z-m-l.com/go/contractions.txt so i expected to see a lot more than one contraction appear on this list. i checked why, and found, to my surprise, that this book contains _no_ contractions -- save for that "they'll", which is embedded in a quote -- and that's why nothing else showed up. you learn every book is unique. another thing which jumped out at me is that i expected more names... there are a few, but usually you see more. ditto with compound-words; i have _none_ in my dictionary, so every one should've been listed here. the reason for these last two strange phenomena is quite humorous. remember that "specials" dictionary, which i suggested you review? it was formed, in large part, on the basis of my experience when i did this mabie book in an earlier experiment. indeed, look at the words that turned up in the "specials" list, and you'll see a number of names, as well as _all_ of the compound-words which are present in the book, from "book-lover" to "world-spirit". i briefly considered taking those compound-words out of the "specials" file, just so it wouldn't look like i was "cheating". but i decided against it, because it _does_ illustrate that -- as you do more and more books -- your dictionaries become more and more complete, so you end up with more accurate results... besides, all of those compound-words can easily be checked _inside_ of the program, and verified, if you like, simply by checking each side of the dash to see if they are valid words, and they are, in this book... (the one "exception" might be "self-centred", a problem that we could solve quite easily simply by adding the word to our british dictionary.) so much for the spellcheck lesson. do you have any questions for me? -bowerbird

3 2

Amazon finally offers Library Lending
by James Adcock 21 Sep '11

21 Sep '11

Just got this PR message and tried it out: http://phx.corporate-ir.net/phoenix.zhtml?c=176060 <http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle> &p=irol-newsArticle First I had to find my library card, then I had to retrieve my password, then the library checkout process takes you to Overdrive, and then you are automatically taken to the Amazon Up-Sell site to download the book you are lending from your library. Oh, and would you like to buy a book we have pre-selected for you at the same time? Other than that it works great if you pick a book which hasn't been checked out already (most were.) And reading a modern title I found I didn't enjoy it any more and probably less than the older titles from PG. I have a Sony EPUB reader so I could already read library checkout books on that, but I like reading on the Kindle better, and the wireless download to Kindle is much more user-friendly than the Sony EPUB reader library- book-by-USB-cord process.

1 0

these grapes are sweet -- lesson #11
by Bowerbird＠aol.com 16 Sep '11

16 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** the major advances attained in our last lesson were that we 1) are now able to chunk the text in terms of its _pages_, and 2) link to a particular scan, or even pull it in, if we so desire... a question you might have is "where are the scans mounted?" one answer is that you can mount them anywhere you like... in the case of my code here, i linked to scans from this book which have been located on one of my websites for ages now. (indeed, this set of scans is _the_original_, from google, who released this book as their very first public-domain example. back then, the project was named "google print", which is why each scan bears a "watermark" in its margin which says that.) one big reason i linked to my scans is that they were already mounted, but another was that they are named intelligently; the number in the filename of each scan is its pagenumber. but of course, this approach won't serve _you_, at least not if you do another book for which i haven't mounted scans... there is another easy answer, though... recall that we are cleaning the o.c.r. text for a book scanned by archive.org. so the scans are located at archive.org! you might object that the scans in that scan-set are combined in a .zip file. that is correct. but they can still be accessed individually. to see how, take another look at the archive.org viewer-app: > http://www.archive.org/stream/booksculture00mabiuoft#page/123 that displays the scan for page 123 (and perhaps 122 too, depending whether you're using the 1-up or 2-up modes.) if you right-click on the scan for page 123, and then choose the selection to "copy image address", you should get this: > http://ia600300.us.archive.org/BookReader/BookReaderImages.php? > zip=/1/items/booksculture00mabiuoft/booksculture00mabiuoft_jp2.zip > &file=booksculture00mabiuoft_jp2/booksculture00mabiuoft_0129.jp2 > &scale=3.02920443101712&rotate=0 that's a u.r.l., a very long one. (i broke it up into pieces.) and it ends up that you can ignore the last part of it -- the part that gives the "scale" and "rotate" components. this u.r.l. gives us the address for the scan for page 123. you could be forgiven if you're a bit confused, since there isn't _any_ occurrence of "123" inside of that u.r.l., at least until i remind you that there's an offset of "6" in this book. sure enough, 123+6=129, and you _do_ see a "129" there. so that u.r.l. tells us how to address the other pagescans in this book. if we wanted to show the scan for page 234, for instance, we'd simply change that 129 to 240 (234+6). thus, for another book, you'll need to figure out the offset. one complication is that some books have an offset that _changes_ in the course of the book, due to pages which were unnumbered. (this happens with illustration plates.) in such cases, you'll need to work with a "variable offset". it's not difficult to figure it out; it's just a pain in the ass. of course, another option is just to download the scans in the .zip file, rename the files more intelligently, and mount them in a location over which you have control... this will give you a chance to downscale the images too. because you should be aware that archive.org scans are bloated... for instance, their scan for page 123 is 221k, whereas the .jpg mounted on my site for page 123 is 66k. so if you summon a page many times, it'll be downloaded repeatedly, which could impact any bandwidth limitations you might be suffering under... so if you think you might work a lot on the pages, it's best to download the scans... why waste those precious mobile minutes 4 times as fast? perhaps you think that's a first-world problem? re-think. the people on the poor side of the digital divide will suffer even more from the archive.org obsession with pagescans instead of digital text, which is lighter _and_ more flexible. and i don't think we should sweep it under the rug, either... having said all that, i will also report that the java-based viewer-app at archive.org works like a charm on an ipad. so -- if you are lucky enough to afford it -- appreciate it. getting back to that issue of where the scans are stored, it's also possible to grab google's scans programmatically just use the same tactic of getting the u.r.l. for the scan, and plug it into your program. you might also find that google's a bit better about naming its files intelligently. *** so, enough about that. let's get back to our mabie book... in our last lesson, i viewed scans to answer some questions about a handful of edits, so i made those fixes to the text... i also decided to do the manual editing of the front-matter. this book was typical of most, in that the recognition of the front-matter is very spotty. most of the words are correct -- which is a big improvement over the way it used to be -- but the formatting is bad, with far too many blank lines... you can see the original o.c.r. front-matter here: > http://zenmarkuplanguage.com/grapes000.txt up until this last edit, i substituted in some dummy-text: > http://zenmarkuplanguage.com/grapes002.txt and you can view my hand-edited front-matter here: > http://zenmarkuplanguage.com/grapes003.txt so that's now the newest version of our text: grapes003.txt. *** this book was most decidedly quite friendly to us, because the first page of its body-text has the pagenumber of "7", meaning that we had 6 pages of front-matter to play with. sometimes we need to shift the front-matter to the back -- just while we're cleaning the text -- so we can _start_ on "page 1" and have that text be the first text in the file. but we didn't have to play that little trick, thank goodness. you will notice that i did "rearrange" some of the pages, and i dropped the text from the cover (we didn't need it). plus in an e-book, you don't have "recto" and "verso", and blank pages are unnecessary, so i eliminate 'em. in general, just play around with it, and make it work... *** ok, now for some code... our page-oriented advances from the last lesson allow us to do something kinda cool, for the sake of a little demo. here we go: > http://zenmarkuplanguage.com/grapes201.py i keep hoping marcello will barge in here to make fun of my python skills. i have even tried to bait him with code that's _not_ how a "real" python programmer would do it. but this code is _really_ not up to snuff, so i ain't even gonna give you the source today. nonetheless -- this little demo program just might get your juices flowing. it shows one page of text, with its pagescan alongside, and lets you click buttons to go backward or forward... (or just enter a pagenumber in the box to jump there.) that's pretty simple, to be sure, but it can also serve as an _engine_ for features which are much more powerful. if you'd like to see some examples of that, just ask me. or if you want to know the nature of the problems with the code in this program (and the last), ask about that... or if you're wondering if all this can be accomplished in a more effective manner with an offline application -- the answer is "yes", as i've been saying all along -- then we can talk about that... or not... as you prefer... *** one thing that this page-based method tells us is that there are various ways you can approach a digitization. you can set your focus at the level of _lines_ of text, which proves to be the _best_ level most of the time, in my opinion. you can work on the book as a whole. you can make the _page_ the primary unit of analysis. you can digitize at the level of _individual_words_, and you can concentrate on the _paragraph_ as your entity. now, of course, there's no requirement that you work _exclusively_ at any particular level. some checks can be accomplished more effectively at a certain level, so you should choose your focus appropriate for the task. so in our next lesson, we're gonna drill down to words, and code up an actual spell-check routine for our text. big fun for the weekend! :+) -bowerbird

1 0