September 2011 - gutvol-d - lists.pglaf.org

these grapes are sweet -- lesson #13
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** there is no lesson #13. -bowerbird

1 0

Re: [gutvol-d] these grapes are sweet -- lesson #12
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

roger's right. so is keith. but i am walking a slightly different tight-rope, in that i'm working the elementary-school level, where i'm explaining these concepts to "a novice" and providing actual working code in a language in which i've never ever programmed before now. so there are plenty of more-complicated niceties which are being glossed over at the present time. in spite of that fact, i'm showing how you can take some very crude code and nevertheless do the job of digitization, from raw o.c.r. to polished e-book. because there are people in power over at d.p. who continue to tell their volunteers "it cannot be done", even though both roger and i have proven _it_can_. it is resolutely immoral how much time and energy -- donated by those kind volunteers in good faith -- is being _wasted_ by leaders who are lying to them... -bowerbird

1 0

these grapes are sweet -- lesson #12
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. i hope you enjoyed the brief interlude between our lessons... *** ok, we've done all of our "first-pass" cleaning, so now it's time for us to move on to spellcheck. so let's code a spellchecker... the first thing you need to know is that even though i call it "a spellchecker", it's very different from the spellchecker that you are used to, the one you have inside your wordprocessor. within that environment, a spellchecker has two components: 1) a routine to flag words that are not in the dictionary, and 2) a routine to provide a "suggestion" on the correct spelling. we need the first component. but we don't need the second, since the question about the "correctness" of the spelling is answered by viewing the pagescan, to see what was printed. luckily for us, the first routine is extremely easy to program, while the second is usually much more difficult and complex. so we get to do the simple part and not the hard part. great! *** the first thing we need is a "dictionary" to use as our standard. again, this is not "a dictionary" per se, because there are no _definitions_ included in it, but merely a long list of words... you can find many dictionaries online. here's the one i use: > http://z-m-l.com/go/regulardictionary.txt if you take a look at that, you'll see it's a simple list of words, each one on a line by itself, arranged in alphabetical order... it's reasonably sized, at just under 800k, with 81,500+ words. i've got another dictionary -- i think it's the one rfrank uses in his routines -- that's 1.5 megs. bigger isn't always better. first, it takes more machine-time to use a bigger dictionary. second, and more importantly, you're more likely to miss a real o.c.r. error that just happens to spell some other word. i even think my dictionary is too big... there are "words" in it that only a scrabble player could love. but it's too much work to refine it, and i don't think it's causing errors, so i keep it... as an interesting sidenote, many of the dictionaries online were formed originally by the "moby" research, which used e-texts from project gutenberg as their underlying corpus. *** on the face of it, it's brain-dead simple to check a book. first, you need to split the book up into "words", which you can do my changing all the spaces into linebreaks, and then splitting the file by linebreaks. easy enough... then, a looped routine like this seems like it should work: > if eachword not in thedictionary then print eachword that code has big bugs in it, though. can you spot them? first off, if you looked at the dictionary, it's all-lower-case, so we need to convert eachword into its lower-case version. second, the dictionary contains no punctuation characters, so we need to strip all the leading and trailing punctuation. (anything with internal punctuation is automatically "bad".) but the biggest bug is misrecognitions like "ng" _will_ be found in the dictionary, e.g., in every word ending in "ing". to prevent such "subwords" from being found, we need to slap a linebreak at the start and at the end of eachword, so it will only register a hit if it matches a complete "line". after those little changes, this routine will "work" just fine. it's not a _good_ routine, though, because it has to search too many words -- from the top of the dictionary down -- for every bookword before finding a hit. and on bad words, it has to search the entire dictionary to know it's "not there". *** now, because the dictionary is sorted alphabetically, we can improve our routine so that it is significantly more efficient. for instance, we could split the dictionary into an array, with all the words that start with each letter as a separate item... i.e., all "a" words go in one item, all "b" words, all "c" words... then, we'd just have to search that one specific array-item... (for a word starting with "c", we'd search the "c-word" item.) that would cut down on the unnecessary searching a bunch. *** but we can do even better by sorting the words in our book, so that we will then be dealing with _two_ alphabetical lists. in addition, sorting the words in our book allows us to easily discard the duplicates, so we only search for each word once. here's how we do the check, and why it's so bloody efficient... we have 2 arrays -- our dictionarywords and our bookwords. we set our pointers to start at the top of each list, and we then compare the dictionaryword with the bookword. if the dictionaryword and the bookword match each other, we put the bookword on the list we keep of "found words", and we then move on to the next word in both of our lists. if the dictionaryword is "less than" the bookword -- that is, if it comes _before_ the bookword in an alphabetized list -- then we go on to the next word in our dictionaryword list... finally, if the dictionaryword is "greater than" the bookword, we know we've gone past where the bookword would've been -- if it _were_ in the dictionary -- so it must not be in there, thus we will place that bookword in our "not-found" list, and then move on to the next bookword in our list of bookwords. this is simple logic to code. (easier to program than explain.) *** so now let's go on to the next step in our thinking about this. the "not-found" list is very important, of course. indeed, you could be forgiven for thinking it is the only one that matters. but you would be wrong; the "found" list is extremely valuable. once we have the initial list of "found" words, we're _cooking_. that's because we now have a "dictionary" for this unique book. this file is the one we will use in our spellchecker from now on. because this file is significantly smaller than the full dictionary, our spellcheck routine will work very speedily, which means we can incorporate it as a regular part of all our cleaning routines. so if/when we inadvertently introduce errors, we will catch them. you may find yourself resisting this idea. keep thinking about it, until you come to the realization it _is_ the smartest thing to do. *** there are two other wrinkles i threw into the spellcheck routine. the first is that i included a check for _british_ spellings, which was necessary for this book. (if the file wouldn't have had any, it wouldn't have mattered. since it did, if i had _not_ included the british dictionary, those words wouldn't been "not-found".) the british words get listed separately, so if there are only a few, you can check to see whether they were simple misrecognitions. when there are many, as you can see here, you are assured that the author utilised such spellings with the fullest of intentions... the british dictionary is here: > http://z-m-l.com/go/britishdictionary.txt the other wrinkle is a file i call "specials.txt". this file contains "special" stuff that isn't in the regulardictionary.txt, but which is not unusual to encounter when digitizing a book, things like roman numerals, arabic numbers, famous names, and so forth, much of which requires that its check be done _case-sensitive_. the "specials" dictionary is here: > http://z-m-l.com/go/specials.txt take a good look at the items there, so you get a feel for them... *** ok, so let's run the code: > http://zenmarkuplanguage.com/grapes110.py > http://zenmarkuplanguage.com/grapes110.txt we get a list of "notfound", "british", "specials" and "found" words. i have found it useful to split the "notfound" list into 2 segments, depending upon whether the word starts with an initial capital... the ones which do are usually _names_, and (like most of the text) they generally tend to have been recognized by the o.c.r. correctly. and yes indeed, that appears to be exactly the case we have here: > Arcady > Dante's > De > Emerson's > Farge's > Florio's > Gary's > Goethe's > Hamilton > Haymarket > Hegel's > Hfe > Hke > Hmits > Holinshed's > Hterature > Hve > Hving > Lowell's > MDCCCCII > Mabie > Melos > Norse > Plato's > Quincey's > Selfexpression > Shakespeare's > Spenser's > Symonds's > Tennyson's > Text-books > They'll > Tolstoi's > Wordsworth we do find a few words which do not look like names, but look instead like o.c.r. errors, including a bunch of those starting with a capital-h... further, "selfexpression" clearly just needs to have its hyphen restored. *** next, words which were notfound that start with a lowercase letter are most typically misrecognitions, and that was true in the current case... > acquirv > ageo > appletree > bhnd > certainlytrue > clearlyenough > comparacively > conceit > conceived > concentration > conception > conceptions > concern > concerned > concerning > concert > conclusion > conclusions > conditlons > de > dehvered > eightysix > enhghtenment > en« > escape.[1 > farreaching > generahsations > h'abitually > halfbelieved > halfunderstood > happyformulation > hfe > highminded > hves > illdirected > largelydeveloped > manysided > mediaevalism > nomena > oftrepeated > oi > oijly > onlylive > onlysearching > overmatched > overweighted > personahty > phe > pious-minded > potentiahties > powerfullyorganised > rationahsing > richment > selfsacrifice > selfsatisfaction > throug > uhimate > unbroken- > vitahty > wellwrought > whlch > witliout > wo;-ks most of the words are o.c.r. errors. some of 'em are hyphenates that appear to have lost their hyphen. others are terms that were at the end of a line where a speck was misrecognized as a hyphen, so they were improperly joined (e.g., certainlytrue, clearlyenough). and still others are terms that were end-of-line hyphenates which must _retain_ their hyphen if/when they become rejoined together. but, as with the capitalized words, these words are rather typical. so, in many ways, this looks like the type of list you would expect... but there were some things about this list which jumped out at me. the first is "they'll". you might have seen that i have _no_contractions_ in my dictionary. that's because i've stored them in a separate file, one which i didn't have the current version of the program read in or utilize. just so you know, here's that list of contractions: > http://z-m-l.com/go/contractions.txt so i expected to see a lot more than one contraction appear on this list. i checked why, and found, to my surprise, that this book contains _no_ contractions -- save for that "they'll", which is embedded in a quote -- and that's why nothing else showed up. you learn every book is unique. another thing which jumped out at me is that i expected more names... there are a few, but usually you see more. ditto with compound-words; i have _none_ in my dictionary, so every one should've been listed here. the reason for these last two strange phenomena is quite humorous. remember that "specials" dictionary, which i suggested you review? it was formed, in large part, on the basis of my experience when i did this mabie book in an earlier experiment. indeed, look at the words that turned up in the "specials" list, and you'll see a number of names, as well as _all_ of the compound-words which are present in the book, from "book-lover" to "world-spirit". i briefly considered taking those compound-words out of the "specials" file, just so it wouldn't look like i was "cheating". but i decided against it, because it _does_ illustrate that -- as you do more and more books -- your dictionaries become more and more complete, so you end up with more accurate results... besides, all of those compound-words can easily be checked _inside_ of the program, and verified, if you like, simply by checking each side of the dash to see if they are valid words, and they are, in this book... (the one "exception" might be "self-centred", a problem that we could solve quite easily simply by adding the word to our british dictionary.) so much for the spellcheck lesson. do you have any questions for me? -bowerbird

3 2

Amazon finally offers Library Lending
by James Adcock 21 Sep '11

21 Sep '11

Just got this PR message and tried it out: http://phx.corporate-ir.net/phoenix.zhtml?c=176060 <http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle> &p=irol-newsArticle First I had to find my library card, then I had to retrieve my password, then the library checkout process takes you to Overdrive, and then you are automatically taken to the Amazon Up-Sell site to download the book you are lending from your library. Oh, and would you like to buy a book we have pre-selected for you at the same time? Other than that it works great if you pick a book which hasn't been checked out already (most were.) And reading a modern title I found I didn't enjoy it any more and probably less than the older titles from PG. I have a Sony EPUB reader so I could already read library checkout books on that, but I like reading on the Kindle better, and the wireless download to Kindle is much more user-friendly than the Sony EPUB reader library- book-by-USB-cord process.

1 0

these grapes are sweet -- lesson #11
by Bowerbird＠aol.com 16 Sep '11

16 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** the major advances attained in our last lesson were that we 1) are now able to chunk the text in terms of its _pages_, and 2) link to a particular scan, or even pull it in, if we so desire... a question you might have is "where are the scans mounted?" one answer is that you can mount them anywhere you like... in the case of my code here, i linked to scans from this book which have been located on one of my websites for ages now. (indeed, this set of scans is _the_original_, from google, who released this book as their very first public-domain example. back then, the project was named "google print", which is why each scan bears a "watermark" in its margin which says that.) one big reason i linked to my scans is that they were already mounted, but another was that they are named intelligently; the number in the filename of each scan is its pagenumber. but of course, this approach won't serve _you_, at least not if you do another book for which i haven't mounted scans... there is another easy answer, though... recall that we are cleaning the o.c.r. text for a book scanned by archive.org. so the scans are located at archive.org! you might object that the scans in that scan-set are combined in a .zip file. that is correct. but they can still be accessed individually. to see how, take another look at the archive.org viewer-app: > http://www.archive.org/stream/booksculture00mabiuoft#page/123 that displays the scan for page 123 (and perhaps 122 too, depending whether you're using the 1-up or 2-up modes.) if you right-click on the scan for page 123, and then choose the selection to "copy image address", you should get this: > http://ia600300.us.archive.org/BookReader/BookReaderImages.php? > zip=/1/items/booksculture00mabiuoft/booksculture00mabiuoft_jp2.zip > &file=booksculture00mabiuoft_jp2/booksculture00mabiuoft_0129.jp2 > &scale=3.02920443101712&rotate=0 that's a u.r.l., a very long one. (i broke it up into pieces.) and it ends up that you can ignore the last part of it -- the part that gives the "scale" and "rotate" components. this u.r.l. gives us the address for the scan for page 123. you could be forgiven if you're a bit confused, since there isn't _any_ occurrence of "123" inside of that u.r.l., at least until i remind you that there's an offset of "6" in this book. sure enough, 123+6=129, and you _do_ see a "129" there. so that u.r.l. tells us how to address the other pagescans in this book. if we wanted to show the scan for page 234, for instance, we'd simply change that 129 to 240 (234+6). thus, for another book, you'll need to figure out the offset. one complication is that some books have an offset that _changes_ in the course of the book, due to pages which were unnumbered. (this happens with illustration plates.) in such cases, you'll need to work with a "variable offset". it's not difficult to figure it out; it's just a pain in the ass. of course, another option is just to download the scans in the .zip file, rename the files more intelligently, and mount them in a location over which you have control... this will give you a chance to downscale the images too. because you should be aware that archive.org scans are bloated... for instance, their scan for page 123 is 221k, whereas the .jpg mounted on my site for page 123 is 66k. so if you summon a page many times, it'll be downloaded repeatedly, which could impact any bandwidth limitations you might be suffering under... so if you think you might work a lot on the pages, it's best to download the scans... why waste those precious mobile minutes 4 times as fast? perhaps you think that's a first-world problem? re-think. the people on the poor side of the digital divide will suffer even more from the archive.org obsession with pagescans instead of digital text, which is lighter _and_ more flexible. and i don't think we should sweep it under the rug, either... having said all that, i will also report that the java-based viewer-app at archive.org works like a charm on an ipad. so -- if you are lucky enough to afford it -- appreciate it. getting back to that issue of where the scans are stored, it's also possible to grab google's scans programmatically just use the same tactic of getting the u.r.l. for the scan, and plug it into your program. you might also find that google's a bit better about naming its files intelligently. *** so, enough about that. let's get back to our mabie book... in our last lesson, i viewed scans to answer some questions about a handful of edits, so i made those fixes to the text... i also decided to do the manual editing of the front-matter. this book was typical of most, in that the recognition of the front-matter is very spotty. most of the words are correct -- which is a big improvement over the way it used to be -- but the formatting is bad, with far too many blank lines... you can see the original o.c.r. front-matter here: > http://zenmarkuplanguage.com/grapes000.txt up until this last edit, i substituted in some dummy-text: > http://zenmarkuplanguage.com/grapes002.txt and you can view my hand-edited front-matter here: > http://zenmarkuplanguage.com/grapes003.txt so that's now the newest version of our text: grapes003.txt. *** this book was most decidedly quite friendly to us, because the first page of its body-text has the pagenumber of "7", meaning that we had 6 pages of front-matter to play with. sometimes we need to shift the front-matter to the back -- just while we're cleaning the text -- so we can _start_ on "page 1" and have that text be the first text in the file. but we didn't have to play that little trick, thank goodness. you will notice that i did "rearrange" some of the pages, and i dropped the text from the cover (we didn't need it). plus in an e-book, you don't have "recto" and "verso", and blank pages are unnecessary, so i eliminate 'em. in general, just play around with it, and make it work... *** ok, now for some code... our page-oriented advances from the last lesson allow us to do something kinda cool, for the sake of a little demo. here we go: > http://zenmarkuplanguage.com/grapes201.py i keep hoping marcello will barge in here to make fun of my python skills. i have even tried to bait him with code that's _not_ how a "real" python programmer would do it. but this code is _really_ not up to snuff, so i ain't even gonna give you the source today. nonetheless -- this little demo program just might get your juices flowing. it shows one page of text, with its pagescan alongside, and lets you click buttons to go backward or forward... (or just enter a pagenumber in the box to jump there.) that's pretty simple, to be sure, but it can also serve as an _engine_ for features which are much more powerful. if you'd like to see some examples of that, just ask me. or if you want to know the nature of the problems with the code in this program (and the last), ask about that... or if you're wondering if all this can be accomplished in a more effective manner with an offline application -- the answer is "yes", as i've been saying all along -- then we can talk about that... or not... as you prefer... *** one thing that this page-based method tells us is that there are various ways you can approach a digitization. you can set your focus at the level of _lines_ of text, which proves to be the _best_ level most of the time, in my opinion. you can work on the book as a whole. you can make the _page_ the primary unit of analysis. you can digitize at the level of _individual_words_, and you can concentrate on the _paragraph_ as your entity. now, of course, there's no requirement that you work _exclusively_ at any particular level. some checks can be accomplished more effectively at a certain level, so you should choose your focus appropriate for the task. so in our next lesson, we're gonna drill down to words, and code up an actual spell-check routine for our text. big fun for the weekend! :+) -bowerbird

1 0

these grapes are sweet -- lesson #10
by Bowerbird＠aol.com 15 Sep '11

15 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** in the last lesson, we culminated with a program that gave us a search capability on the original o.c.r. text-file of our book. we can make that program work on our edited text as well. the newest version of our edited text is here: > http://zenmarkuplanguage.com/grapes002.txt you can use your comparison tool to see the changes made from the previous version... but i'll tell you that one set of changes was to ensure all of the pagenumbers are correct. and since the pagenumbers are correct, we can show them, (rather than the linenumber in the file) for lines we display. so we'll do that. we'll show the pagenumber for each line, and we'll include the linenumber _on_that_specific_page_... (so we kinda know if the line is up top, middle, or bottom.) plus, as we said earlier, having this information be correct means that -- once we pull out any particular line of text -- we can show (or link to) the _scan_ of the page containing it. so let's incorporate these additions into our search program. the search program, now working on grapes002.txt, is here: > http://zenmarkuplanguage.com/grapes109.py > http://zenmarkuplanguage.com/grapes109.txt as you can see, once you run a search, each located line is preceded by the pagenumber:linenumber -- plus the line (in its entirety) has been made into a link to its page-scan. click on a line to refer to its scan to resolve any questions. for an example of this, consider that -- when i am doing edits on the text -- if there is something about which i am uncertain, i'll mark it with a "??", so i can return to it later. when i was doing the "first-pass" edits to this file, i didn't bother to look at pagescans, but if something was unclear, and i couldn't make the decision without seeing the scan, i just used the "??" notation. so, now that we have links to view the pagescan for any line found by the search routine, you can play along with me and search for those "??" lines... the tool will show you these: > 052:10 _____ the 'Myth of the Soul' ?? I discovered > 092:18 _____ spell which they all possess, ?? -- the > 100:06 _____ sonalities ?? the atmosphere of such re- > 105:22 _____ Kourotrophos ?? the mother of corn and > 138:18 _____ ?? climate; intensity of heat and light; > 146:22 _____ characterise ?? the man of culture will be > 266:18 _____ us. They disclose wide ?? diversity of > 275:11 _____ in the educational ?? of the indi- it's simple enough to click the line, so as to view its scan. for instance, clicking the top line will display the scan for page 52, and we know the line is #10, so midway down... and yes, the scan tells me that there _was_ a comma there (which is what i had been unsure about). so i will fix that. indeed, i can now fix all of those lines in similar fashion... *** another thing i've added is the ability to list all of the lines on a particular page. just enter "p#123" in the search-box, for instance, and all the lines for page 123 will be shown... and all of the lines are links to the scan for page 123, yes, but in this case, we obviously know we wanna see the scan, so the program just displays it, without us having to tell it. *** if you are paying attention, all this should give you lots of ideas about what could be done next, so think about that... -bowerbird

1 0

these grapes are sweet -- lesson #09
by Bowerbird＠aol.com 14 Sep '11

14 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** i showed you the list of edits which i made in my "first pass", all of which i found using various searches in my text-editor. the auto-change that stripped the trailing space off every line touched... well... every line, quite obviously, so in that sense, every single line of the original o.c.r. text-file was changed, albeit in a terribly uninteresting way. if we look at non-trivial auto-changes, there were far fewer... we auto-changed floating punctuation, but that was merely a couple lines for most of the marks, which the exception of (1) linebreak doublequotes, where 19 lines were fixed, and (2) semicolons, where some 444+ instances were corrected, which is a big number, but it remains an uninteresting edit. mid-line floating doublequotes got changed manually, with roughly 62 of those corrected. in most books, i would _not_ correct those glitches by hand, but since there were so few... but again, all of these are boring in the extreme... probably the most interesting set of edits were those which involved "weird" characters. using reg-ex, it's easy enough to locate such characters in a single search, if you choose... which means that it's easy to write a program to list the lines that contain any such characters. that's what we'll do today, so you can actually see the lines located with that routine... this is a useful program that you'll want to run on any file when you begin to work on it, to see what problems it has. it helps you zero in on the most common o.c.r. problems... and again at several points in your workflow, you'll want to run it again, just to make sure that you haven't introduced any of these weird characters into your text by accident... this program is a modification of the last one, adding in an array of characters that we want to flag, then looping through the lines of the file and listing any of those lines which contain a flag characters. you can run this program, or grab the code, here: > http://zenmarkuplanguage.com/grapes103.py > http://zenmarkuplanguage.com/grapes103.txt *** again, that code lists the lines by their order in the file... (it displays the linenumber for your convenience as well, since many text-editors will let you jump to a certain line. if you don't know of such a feature, you should look for it.) now, we might also want to group the lines according to the flag character, so we see all the lines that have a dollarsign (for instance) listed together, so we'll rework the code a bit. you can run this new program, or view the code, here: > http://zenmarkuplanguage.com/grapes104.py > http://zenmarkuplanguage.com/grapes104.txt this produces a nice grouped list, plus it also shows us the flagged terms which produced _no_ instances from the file. as you can see, most of the listed lines _are_ o.c.r. errors. so, for the most part, once we've cleaned up these glitches, whenever we run this program again later in our workflow, it should produce very few lines. it _will_, however, produce _some_ lines. at least it should. for instance, we're flagging "11", because in a lot of books, o.c.r. misrecognizes the end of words like _they'll_ or _we'll_ as an _11_... (you'd think it'd be smarter than that, but no.) indeed, as you can see from the output of our little program, there was one such misrecognition in this file, on line #4938. it's also the case, however, that there will be a lot of lines where "11" is the correct recognition, like the pagenumber for page 11, for instance. and for 111 and 211 too. d'uh. don't forget 112, 113, 114, 115, 116, 117, 118, and 119... but we see no "111" in this file, as the pagenumber for 111 was misrecognized as "iii". we also have no 115, and we see (because we searched for "ii" as well) that 116 and 118 were misrecognized, with the "11" seen as a pair of lowercase i's. also, pagenumber 113 has a signature number we'll erase... (this informs us we need a search to find that phenomenon, which will be a reg-ex looking for number-space-number.) likewise with that "ii". few english words have an "ii" pair, but imagine if you were digitizing a book about "skiing"... the goal is to catch o.c.r. errors but minimize false-alarms. our grouped listing also shows, in this text, that asterisks were usually a misrecognized doublequote (or singlequote). and we see we had a few glitches where "w" was seen as "v/". also note that we had problems involving the insertioncaret. of specific import is that some of these caret problems were on pagenumbers, which will be particularly important to us... *** the linenumbers of the file, which we've been listing, are good. but there's better positional information inside the text itself, i.e., pagenumbers (plus, by extension, per-page linenumbers). the pagenumbers give us the vital ability to _display_the_scan_ for any particular line of text, so we wanna correct 'em _now_, so we start benefiting immediately from that great capability. so let us write some code to help us correct the pagenumbers... that's rather simple to do, because our expectation is that they will march along for us in a sequence that's entirely predictable. so we'll just look closely for any departures from that sequence. *** the routine that helps clean up pagenumbers is found here: > http://zenmarkuplanguage.com/grapes105.py > http://zenmarkuplanguage.com/grapes105.txt you can puzzle through the code later if you like, but for now just look at the output. what you'll see is a line of "xxxxxxxx" wherever we get a departure from the next-expected-number. the program also shows us any short lines that it encounters, which helps us immensely to see where the problems might lie. it also lists the actual contents of the line, not just the number which it expected to find. sometimes a line "reduces down to" the expected number, but it has some garbage characters in it, and we're gonna wanna eliminate those garbage characters... and as before, this is a program that we're gonna wanna re-run every so often, to make sure we didn't mess up any pagenumbers. indeed, we might want to leave this routine in all our programs, and have it be silent as long as everything checks out fine, but have it bring notice to our attention if it suddenly finds an error. so, let's run this code against the file that i edited, to check it. > http://zenmarkuplanguage.com/grapes106.py > http://zenmarkuplanguage.com/grapes106.txt oops! as we can see, i fixed most of the pagenumber problems, but not _all_ of them. there's still a bug on pagenumber 61, and pagenumber 183 is still missing-in-action, and pagenumber 228 was misrecognized by the o.c.r. as "328", so we have to fix that... we also have a listing for linenumber 5957 and linenumber 5959, where a footnote number is confusing our routine. we can fix the code so that it ignores such footnote numbers in the future. *** so far, we've taken the general approach of _searching_for_ the particular character(s) which might be an o.c.r. misrecognition. that's an ok approach if we know what we should look for, but sometimes we don't. so we can take the _opposite_ approach, and say "these are _good_ characters, so ignore all of them, but show me any characters which are _not_ these good characters". that's the code we've got here: > http://zenmarkuplanguage.com/grapes107.py > http://zenmarkuplanguage.com/grapes107.txt what we find when we run that is some high-bit characters, which we were not expecting to find in 7-bit o.c.r. output... remember how i told you that archive.org was "losing" their em-dashes, and i had to badger them for years to fix that? well, when they "fixed" it, they did so by introducing a utf8 character into their otherwise-all-7-bit-files. what idiots! these are technocrats of the very worst kind, ones who will introduce unnecessary complexity into the simplest of files. oh well, it's just another global change, so it's fairly easy to ignore this particular obstacle the technocrats threw at us... besides em-dashes, there are 2 other lines that show up... those were also high-bit characters that got put in the file. but we've isolated them, so it's easy enough to delete 'em... *** now that we've got search-related routines in the program, might as well code first pass at a general search capability. that's here: > http://zenmarkuplanguage.com/grapes108.py > http://zenmarkuplanguage.com/grapes108.txt now we can search the file for any word we want... this is pretty simplistic, but it's much farther along than we were before we coded it, so let's enjoy our progress... -bowerbird

1 0

Re: these grapes are sweet -- lesson #08
by Bowerbird＠aol.com 14 Sep '11

14 Sep '11

hunter said: > i am the lead developer for guiguts well, hunter... given the poor probability d.p. will change its antiquated workflow, it's great that a person like you is willing to try to smooth and improve the tools... thank you for sharing. -bowerbird

1 0

these grapes are sweet -- lesson#08
by hmonroe.pglaf＠huntermonroe.com 14 Sep '11

14 Sep '11

I am the lead developer for guiguts and have made some progress in cleaning up its many bugs. I would be happy to include any useful scripts from Bowerbird or anyone else. Any volunteers with perl experience to help with clean up bugs and adding new features would be appreciated. We are providing a Windows executable already and can provide a Mac or Linux executable if someone volunteers to build it. See the latest release notes below. Hunter ------------- This is a major version which will become 1.0 once the bugs are ironed out (see this discussion http://www.pgdp.net/phpBB2/viewtopic.php?t=48584) Note: upgrading to this version requires a complete reinstall (save your header.txt and setting.rc files). Relative to version 0.2.10 there are several major new features. First, all the HTML checks can be run with a single click, and the output is clickable in most cases. Second, HTML and CSS validation can now be done on your own computer (and PGTEI as well) and there are checks for unused CSS and image issues (using rfrank's pphtml, ppimage, and pptxt scripts). Third, there is now an option to view text and images side-by-side without having to click on "See Image" for each page. For instance you move forward or back one page for both text and image with the "<" and ">" buttons on the status bar. Also, the Auto Show Images option let's you see the image for instance for the page you are spellchecking. (See tips on viewer configuration below.) Fourth, with Winguts there is no need to install Perl on Windows computers (for other platforms see below). Other new features are: a "View in Browser" and Hyperlink page numbers buttons on the HTML palette, tearoff of the Unicode menu, listing small caps in the Word Frequency popup, automatic checking for updates (which can be turned off), horizontal rules as css, if nothing is found the cursor returns to the starting point, better ability to find executables automatically, GutWrench scanno files are included, inclusion of rfrank's pphtml and pptxt scripts, a warning to use human readable filenames, option to include goodwords in spellcheck project dictionary, a text processing menu to ease conversion of bold/italics/small caps, the label Image #nnn in Configure Page Labels is clickable, added Find Transliterations and Find Orphaned Markup (before it only searched for unmatched brackets) to Search menu, Adjust Page Markers menu is accessible from the File menu. For developers, there are internal improvements, including refactoring of functionality into perl modules and a unit testing framework. Bug fixes already included: less mangling of HTML page numbers (for instance [Pg 42-44] instead of three overlapping page numbers), superscripts are converted to HTML correctly (Philad^a) without curly brackets, fixed regex editor for scannos, Ctrl-S saves the file. The side by side image viewing works best if the window for the viewer is sized to match the image (in XnView, choose View, Auto Image Size, Fit Image to Window) and only one instance of the viewer is allowed to avoid having one instance for every page viewed (in XnView, choose Tools, Options, General, Only One Instance). To page through images, use the "<" and ">" buttons on the status bar. To Auto Show Page Images, use the "Auto Img" button on the status bar, use the option on the Prefs menu, or checkboxes in the various search/spellcheck dialogs. I would be happy to release Mac and Unix executables which only requires someone to run tkpp following the instructions in COMPILING.txt. A Mac or Unix distribution should also include the OpenJade/OpenSP onsgmls executable for HTML/PGTEI validation. Detailed release notes: Version 0.3.6 There is a much higher likelihood that this version generates valid HTML. Page anchors are no longer placed at the end of the previous paragraph or before the horizontal rule. The default for word search from the Word Frequency menu is now "Whole Word". Fixed Go To Bookmarks. There is only one file guiguts-0.3.6.zip which includes the WinGuts.exe. Version 0.3.5 Fixed multiple page markers at a single location so they do not overlap but stack vertically like [Pg 32]<br />[Pg 33]. Fixed problem with moving mark left (entry for initial page number was blank) or up (code was garbled). Removed splash screen. Fixed monthly update check to give user the option. Fixed openpng so if the image viewer is not set the user is prompted to set it. Version 0.3.3 Fixed problem in WinGuts.exe with conflicting Tk library. Version 0.3.2 Fixed pathname problem for non-Windows system for splash screen.

1 0

these grapes are sweet -- lesson #08
by Bowerbird＠aol.com 14 Sep '11

14 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** the original o.c.r. text-file, copied from archive.org, is here: > http://zenmarkuplanguage/grapes000.txt the file, after its first round of editing by me, is here: > http://zenmarkuplanguage/grapes001.txt here are the changes i made in my text-editor: eliminate: front-matter (for now) eliminate: back-matter eliminate: blank-lines in front of numbers-in-column-1 auto-change: delete trailing spaces auto-change: 3-blank-line sequences with === line auto-change: double-quotes at line-start auto-change: double-quotes at line-end auto-change: floating semicolons auto-change: floating colons search: floating commas search: floating periods search: period at line-start search: comma at line-start search: single-quotes at line-start search: single-quotes at line-end search: letter-number combo search: number-letter combo search: number-space-number combo search: one-letter-lines search: two-letter-lines search: three-letter-lines search: four-letter-lines search: one-character-lines search: two-character-lines search: two-number-lines (line #s from 10 to 99) search: chapter-headers & subheads & dropcaps/smallcaps search: singlequote-singlequote search: floating doublequotes search: floating singlequotes search: opening singlequotes search: closing singlequotes search: lowercase-uppercase search: uppercase-uppercase (roman numerals) search: 11 ii vv search: lowercase-linebreak-linebreak-lowercase search: comma-linebreak-linebreak search: [@#$%^&*()_+{}\|] auto-change: utf8 emdashes fixes: various fixes of stuff i encountered along the way they should all be pretty obvious, but if you have any questions, don't hesitate to ask me. this is the important "first step" here. *** your mission, if you choose to accept it, is to edit the text-file, exactly as i described above, and compare your results to mine. a good file-comparison utility is an essential when you digitize, so that you can compare subsequent versions of your text-file, to ensure that you made (1) the changes that you intended, and (2) no _unintended_ changes. find a good file-comparison tool. i use and like diffmerge, which is cross-platform and cost-free: > http://www.sourcegear.com *** i get backchannels. i'd much rather talk frontchannel, because dialog is one of the objects here, but i do get backchannels too. like this: > Why not put your project documentations in a blog/tumblr? > It seems it has far wider general interest than the PG project. i might put some stuff out in a location that's a bit more public. or i might not. it doesn't seem to me that there are many people looking for tools to help them digitize books. i wish there were. but mainly it's project gutenberg with distributed proofreaders... for a while, i posted in the forums at distributed proofreaders... i carefully documented many problems with their workflow, and i volunteered to write software for them, software they _needed_ -- desperately -- way back then, and ever since, to this very day. but they didn't like my constructive criticism, so they banned me. that's right, they _banned_ me from posting... can you imagine? merely because i was telling them they were doing it _wrong_, they decided to declare me persona non grata, stick their heads "in the sand" (if you know what i mean), and simply pretend that i hadn't offered to help them -- a fact they continue to ignore... meanwhile, their volunteers have struggled with primitive tools, again _to_this_very_day_. indeed, here's a recent forum thread: > http://www.pgdp.net/phpBB2/viewtopic.php?p=788130 as background, you should know that "guiguts" is the name of the software they currently use to do "post-processing", which is shortened to "pp", so "a new pp-er" is a person who wants to learn post-processing, and has thus set out to install "guiguts". the thread begins like this: > As a new PP-er who has just struggled, cursing all the while, > through installing Guiguts under MacOSX, could I endorse > the observation that: > > "It would be of more benefit to find a middle ground. > We should not be so dependent upon difficult or > buggy software that it interferes with our production." > > And dare I look forward to the development of a Mac version > of Guiguts that is as simple to install as (say) TextWrangler? > If prohibitively difficult to encompass all of Guiguts' > bells and whistles could a subset be useful and feasible? this trouble installing guiguts (and then using it as well) has been experienced by tons of people over the past many years. and thus d.p. volunteers have been crying out for better tools. the "powers that be" at d.p. -- that's what they call themselves, it's not (just) a disparaging term that i have applied to them -- keep saying "we don't have programmers to code better tools." but that's a bald-faced lie, and they know it. they chased me from their midst, and they did it intentionally (and with malice). and meanwhile, their volunteers suffer with outdated tools... so one reason i post here, and not there, is because i'm banned over there... but i can still rub their nose in it by posting here... their people moan, the "powers that be" respond by saying that they have no programmers, and i come here and say "i program, and i code apps that execute on the full range of offline systems, and i've written clean-up programs, and you can have them _if_ you ask me for them", and of course nobody ever asks for them. it goes deeper than that, though. i have a respect for project gutenberg, and for michael hart. when i was being attacked by the wolf-pack here on this list -- for a period of _years_ -- michael defended my presence. the idiots here tried to get me banned here, but _michael_ wouldn't let them get their way. he knew i had something to offer to the cause, and he refused to let them bully him into banning me. so, in my mind, he _deserves_ my posts. he _earned_ 'em. (it's hard to speak of him in past tense.) but it goes even deeper than that. the cyber-archeologists of the future will want to study the evolution of thinking about e-books, and this listserve will be one of their major watering holes. so i want to be here, and i want to have my posts be here. (of course, one of the main reasons why this list's archives will be so valuable to these future scholars will be _because_ my posts are here... so there's a certain degree of circularity going on with this.) and then, of course, we have the dynamics of the past... if i put my stuff on an open blog, it would mostly be ignored. of course, i'm ignored here as well. but at least when i post to this list, i _know_ the actual _people_ who're ignoring me. and i know they're ignoring me at odds to their own interest. which amuses me, making the whole thing that much sweeter. so i just gradually notch it up, bit by bit, as the years go by... you know, kinda like turning up the heat on a lobster in a pot. plus it makes it all the more difficult for d.p. "powers that be" to continue to lie to their volunteers about having no options. they know they're lying, and at some point in the future, the entire sordid mess will be exposed, and those d.p volunteers will realize that their "managers" _intentionally_ lied to them, and karma will do what it always does -- punish the guilty... so about every 6 months, i put forth yet another initiative, aimed at helping d.p. -- if the "powers that be" are simply smart enough to put aside their stupid stubborn "pride" -- or -- when they are not -- showcasing their bad decision... if i have time sometime, i might document the full history of these initiatives, but most recently, i made an attempt in september of last year, to do a "collaboration" that was designed to program an online post-processing system, named "babelfish", except nobody else did any work on it, so the so-called "collaboration" collapsed because of that. (which is pretty much exactly what i figured would happen, but i wanted to demonstrate that _i_ was willing to do it.) then, in spring of this year, i displayed how an online app could be coded that would convert a clean text to an .epub, without using any markup sludge except for my "zen magic". (d.p. has the proofers and formatters put in a bunch'o'crap which the post-processers then have to take out; so dumb.) a year before that -- in the spring of 2010 -- i used perl to code an online text-cleanup system that could've been used in a collaborative fashion, just to prove that it could be done. but, as usual, the "powers that be" over at d.p. just ignored it, all the while claiming there was no one to code such a thing. not that they just ignore me, either. years back, when i was arguing that they could code an offline app for cleaning text -- because i had already programmed such a thing -- one of the d.p. volunteers (dkretz) went to work on his own version of such an app. he did a good job of showing the potential, but the "powers that be" hate him, too, so they ignored that. what was really astounding, though, was when rfrank -- yet another d.p. volunteer -- built a distributed proofing system that worked online. his system challenged a lot of the myths of the d.p. system, and it produced some very good results... but you know the story -- the "powers that be" just ignored it. the thing is, rfrank had been one of the _darlings_ over at d.p. he was (and remains) a very popular member of the community, and was even voted on to the board of directors for the site, but that evidently wasn't enough to get the attention that he needed in order to propagate change within the system as it now exists. so, you know, if rfrank can't even get the system fixed over there, there's no chance that _i_ can. but i come here anyway, and post, and show proof-of-concept, and then come back later and show a working prototype, and then come back even later with a finished product, all because i know those future cyber-archeologists will be digging here... so this is when i pause and say "hi, future cyber-archeologists!" *** here's another piece of an e-mail from the backchannel: > I haven't specifically looked around for such, but > I have a vague idea of starting some sort of site > where folks can submit ebooks for the general public > (haven't decided if it should allow selling of them or not) > and having a way to convert uploaded text > directly to epub sure would be useful. lots of similar sites already exist: > http://www.smashwords.com > http://www.pressbooks.com > http://www.pandamian.com > http://www.leanpub.com > http://www.gluejar.com there are more, too, but that's off the top of my head... > I have my doubts that any system can effectively > produce an error-free html version, you are mistaken. look up "light markup". it's easy. > I have my doubts that any system can effectively > produce an error-free html version, > but hand-editing html isn't really all that hard to do, > so I'm not worried about that one myself. > On the other hand, epub is something loads of folks ask for, > and I don't think it'd be much of a site if it didn't allow such. take a look at what's already out there. if you decide you can add something, do it... :+) -bowerbird

1 0