September 2011 - gutvol-d - lists.pglaf.org

these grapes are sweet -- lesson #07
by Bowerbird＠aol.com 13 Sep '11

13 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** benjamin said: > I use groovy. cool... ;+) feel free to port any of my python code over to that platform. also, benjamin, since you might be the only "new" guy here, i will be open to any feedback you'd have on a web interface for proofing and editing o.c.r. text alongside its pagescan... you saw one possibility, in the links i gave recently: > http://z-m-l.com/go/mabie/mabiep123.html > http://z-m-l.com/go/myant/myantp123.html > http://z-m-l.com/go/sgfhb/sgfhbp123.html i have also floated others, but i'm interested in any designs that you might've generated and would be willing to share... *** ok, so here we go... since we're using o.c.r. text from archive.org, it'll be good to review some of the basic information about their typical files. 1. you can get to the files for a book from its "details" page: > http://www.archive.org/details/booksculture00mabiuoft click the "all files: http" link you find in the leftmost column. this will take you to an "index" page that lists all of the files. the file with "djvu.txt" at the end of its name holds the o.c.r. this is the file we'll be working on, so take a good look at it. does the text look like o.c.r. did a good job of recognition? in some of its early books, archive.org "lost" the em-dashes, so that output is totally worthless; don't even bother with it! some files are also missing quotemarks or end-line hyphens. archive.org also scanned some files where library patrons had scrawled notes in the margins, or underlined some passages, which -- as you might imagine -- wreaks havoc with the o.c.r. before using any archive.org text, ensure it's not fatally flawed; your time is much better spent seeking a _worthwhile_ copy... (archive.org itself often has multiple copies of a specific book.) if you can't find any better o.c.r., do some other book instead! or, if you have to do _that_ book, just re-do the o.c.r. yourself. life is too short to waste your time correcting a bad o.c.r. file. thank goodness, most of the recent archive.org files are good. 2. we also need to be able to review the actual page-scans, to resolve questions. archive.org hosts the scans, both in a high-resolution form (which is far too bulky to bother with) and a lower-resolution (which is more than good enough)... they're in a .zip file there on the "index" page for the book. because i have a tool that shows me each scan alongside the o.c.r. text for that page, i usually download the actual scans. but in addition to the scans, archive.org offers .djvu and .pdf _output_ formats that collect the scan-set so you can view it, and a one-file format is often better than hundreds of scans. .djvu is pretty good, because it's smaller than the .pdf, plus _sometimes_ the o.c.r. text is included too, so it's searchable. (realistically, though, all we usually need is the pagenumber.) .pdf often comes in either black-and-white or "color", which means a sepia background was inserted (for some reason)... 3. so now let's concentrate on the text in the actual o.c.r. file. after you've downloaded it, there is some basic cleaning to do. there's a space at the end of every line, including "blank" ones, so we must to do a global search-and-replace to delete them. i usually do this first, just to get it out of the way right away... 4. pagebreaks are indicated by 3 blank lines. unfortunately, sometimes a 3-blank-line sequence occurs elsewhere as well. which means that we can't _depend_ on it being a pagebreak. which is quite sad, yes. still, for the most part, it usually will. we'll often need to look at a scan to resolve an o.c.r. glitch, so pagenumbers will be important to us, so pagebreaks are too. i replace these 3-blank-line sequences with a distinctive line. (a line full of tildes, for instance, if there are none in the file.) this way i know that's where the pagebreaks (probably) were. 5. the runheads (at the top of a page) _are_ retained, as are pagenumbers when they got printed at the bottom of a page. this is good news. (d.p. always decapitates the runheads and pagenumbers; thus their scans are devoid of identifying info; just one of several idiotic things at distributed proofreaders.) the runheads and pagenumbers keep us grounded in the file; otherwise it can be too easy to get lost in the vast sea of text. but the runheads and pagenumbers often have garbled o.c.r. (this might be because their fonts often differ from the body.) luckily, there is enough redundancy that we can control this; runheads almost always exhibit heavily-repetitive elements, and pagenumbers have an extremely-predictable sequence. one of the first things that i generally do is fix pagenumbers; it speeds up later changes that require any review of a scan... a few easy reg-ex searches can do wonders on pagenumbers. the best situation is when pagenumbers were on the bottom of the p-book pages, because then they are followed by the pagebreak, which is in turn followed by the runhead, so we have an extremely predictable sequence of lines going on... (and luckily for us, that's the situation we have in this book.) 6. front-matter is almost always recognized badly by o.c.r. i've spent a lot of time analyzing o.c.r. output to determine how it can be corrected automatically. but sometimes you just have to throw in the towel and admit you are defeated. such is the case with front-matter. you must fix it by hand. this is not necessarily a bad thing, since the contents page gives you an overview of the book that's very good to have. for example, for this thread, it informed me that our book had _24_ chapters, so when a search for "chapter" ended at chapter 23, i knew i had a problem that i needed to check. and sure enough, i found it, right there on "chapler xxiv". which reminds me that i always check the chapter-headers, to make sure they were recognized correctly, but also that they're on the same page as listed in the table of contents. (this is one of the redundancy checks that it's good to do.) when i check the chapter-headers, i give 'em proper z.m.l., which means 4 blank lines above them, and 2 below them. i also give them a pseudo-runhead and pagenumber, since those were typically eliminated from chapter-header pages, whereas our process will count on 'em being on every page. also when i do this job, i often fix the drop-caps/small-caps (which are a _staple_ of the chapter-header pages), because o.c.r. always fails, in a spectacular fashion, to get 'em right. 7. "scene-breaks" from the print-book are _not_ recorded as empty lines in the o.c.r. output... so they are essentially lost, we'll need to do a page-by-page look-through to restore 'em before we're finished. italics are also lost, so ditto for them. and the third leg of this final look-through on the full book is to make sure that all the paragraphing was done correctly, with particular focus on those that start at the top of a page. happily, most paragraphs _do_ get separated by blank lines... (but it's also the case that tight leading on the original page -- a typographical tactic often employed on block-quotes -- sometimes results in mid-paragraph intervening blank lines.) again, this 3-pronged check is done at the end of the process, so this information is merely a heads-up at this point in time. 8. i forget what 8 was for. 9. archive.org generally scans _all_ of the pages in a book, including the blank pages at the end, which often have stuff on them, such as the library form showing the date on which the book had to be returned... this extraneous back-matter is scrubbed, of course, but _do_ look at the scans, and enjoy! let your imagination roam, about the people who borrowed that book, and read it, and what they might've thought of it. it can make you quite nostalgic... 10. i'm gonna return for a minute to discuss scan-viewing... as i said above, both the .djvu and .pdf options are very good. and you can always download the .zip file of all of the scans; loading them into a file-viewer makes them quite accessible, by choosing the one you want to view based on its filename. one glaring problem with all of these approaches, however, is that archive.org never adopted a _smart_ policy regarding their filenames, meaning the number you see in the filename usually is _not_ the pagenumber for that scan. instead, it is the sequential number of that scan within the full scan-set... this stupid filenaming-policy means that you have know the "offset" for each pagenumber in order to know its filename. if you're lucky -- and we were, in this book -- the "offset" is an unchanging number through the entire book. in this book, the offset is "6", meaning that if you want to see the scan for page 123, for example, you need to view the scan for "129". likewise with the .djvu and .pdf files. however, on the mac, at least, the "preview.app" system graphics-viewer-program can _insert_and_delete_ pages from a .pdf, meaning that you can delete some of those unnecessary front-matter pages so the pagenumbers of the .pdf will match the real pagenumbers. this makes your life easier. finally, there is one more option. for every book, archive.org offers a streaming viewer-program. for instance, for the mabie book we're using here, the u.r.l. is: > http://www.archive.org/stream/booksculture00mabiuoft#page/123 that "page 123" part of the u.r.l. there actually shows page 123, and _not_ page 117, as you might expect, given the "6" offset... also, the finger-pointer slider at the bottom of the display is based upon the "real" pagenumbers too, so you can use that. so this is a way of getting around the pesky "offset" nonsense. but this app is only available if you have an internet connection, and it downloads each pagescan as needed, so it can be _slow_, especially if/when your internet connection isn't all too speedy. plus every time you view a page again, it has to be downloaded yet again (unless it happens to still be in your browser cache)... and realize that eventually we're going to "step through" each and every page in the whole book, and probably more than once. so i certainly can't unequivocally recommend this option to you. i believe it's better to download the scans once, and work offline. but, for the sake of being complete, i've informed you about it... 11. so much for the overview on the files you'll encounter... so let's review the process we'll go through. we will first do the first-pass cleaning of the o.c.r. then we'll check that the pagenumbers and runheads are correct throughout the text. after that, we'll do the rest of the "obvious" cleaning needed, referring back to the scans whenever it is required to do so... next, we'll do a spellcheck and fix the warts. then, after that, we'll smooth any rough edges, and do our last-minute check. the final step will be to convert into various output formats... *** our second program, a slight mod from the first, is appended. you can run -- and view -- this program here: > http://zenmarkuplanguage.com/grapes102.py > http://zenmarkuplanguage.com/grapes102.txt this second script auto-fixes the floating punctuation and skips over the unnecessary front-matter and back-matter. -bowerbird p.s. here's the python source-code for our second program... #!/usr/bin/python import urllib import re f=urllib.urlopen("http://ia700300.us.archive.org/1/items/booksculture00mabiu oft/booksculture00mabiuoft_djvu.txt") s = f.read() f.close() s = re.sub("\r\n","\n",s) s = re.sub("\r\n","\n",s) s = re.sub("\r","\n",s) s = re.sub("\r","\n",s) s = re.sub(" \n","\n",s) s = re.sub(" \n","\n",s) s="\n"+s s=s+"\n" s = re.sub(" ,",",",s) s = re.sub(" ;",";",s) s = re.sub(" \:",":",s) s = re.sub(" !","!",s) s = re.sub(" \?","?",s) s = re.sub(" \!","!",s) print "Content-type: text/html\n\n" print "<html><head><title>" print "grapes102.py" print "</title></head><body><pre>" pg = s.split("\n") maxpg=len(pg) startat=236 endat=7764 for i in range(startat,endat): if i < 10: print "000"+str(i)+" "+pg[i] elif i < 100: print "00"+str(i)+" "+pg[i] elif i < 1000: print "0"+str(i)+" "+pg[i] elif i > 999: print ""+str(i)+" "+pg[i] print "</pre><hr></body></html>"

1 0

Obituary for Michael Hart
by Greg Newby 11 Sep '11

11 Sep '11

Sorry for the bad news. I am in touch with the family, and arrangements are being made for a funeral in Urbana, Illinois. http://www.gutenberg.org/wiki/Main_Page Michael Stern Hart was born in Tacoma, Washington on March 8, 1947. He died on September 6, 2011 in his home in Urbana, Illinois, at the age of 64. His is survived by his mother, Alice, and brother, Bennett. Michael was an Eagle Scout (Urbana Troop 6 and Explorer Post 12), and served in the Army in Korea during the Vietnam era. Hart was best known for his 1971 invention of electronic books, or eBooks. He founded Project Gutenberg, which is recognized as one of the earliest and longest-lasting online literary projects. He often told this story of how he had the idea for eBooks. He had been granted access to significant computing power at the University of Illinois at Urbana-Champaign. On July 4 1971, after being inspired by a free printed copy of the U.S. Declaration of Independence, he decided to type the text into a computer, and to transmit it to other users on the computer network. From this beginning, the digitization and distribution of literature was to be Hart's life's work, spanning over 40 years. Hart was an ardent technologist and futurist. A lifetime tinkerer, he acquired hands-on expertise with the technologies of the day: radio, hi-fi stereo, video equipment, and of course computers. He constantly looked into the future, to anticipate technological advances. One of his favorite speculations was that someday, everyone would be able to have their own copy of the Project Gutenberg collection or whatever subset desired. This vision came true, thanks to the advent of large inexpensive computer disk drives, and to the ubiquity of portable mobile devices, such as cell phones. Hart also predicted the enhancement of automatic translation, which would provide all of the world's literature in over a hundred languages. While this goal has not yet been reached, by the time of his death Project Gutenberg hosted eBooks in 60 different languages, and was frequently highlighted as one of the best Internet-based resources. A lifetime intellectual, Hart was inspired by his parents, both professors at the University of Illinois, to seek truth and to question authority. One of his favorite recent quotes, credited to George Bernard Shaw, is characteristic of his approach to life: "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people." Michael prided himself on being unreasonable, and only in the later years of life did he mellow sufficiently to occasionally refrain from debate. Yet, his passion for life, and all the things in it, never abated. Frugal to a fault, Michael glided through life with many possessions and friends, but very few expenses. He used home remedies rather than seeing doctors. He fixed his own house and car. He built many computers, stereos, and other gear, often from discarded components. Michael S. Hart left a major mark on the world. The invention of eBooks was not simply a technological innovation or precursor to the modern information environment. A more correct understanding is that eBooks are an efficient and effective way of unlimited free distribution of literature. Access to eBooks can thus provide opportunity for increased literacy. Literacy, the ideas contained in literature, creates opportunity. In July 2011, Michael wrote these words, which summarize his goals and his lasting legacy: âOne thing about eBooks that most people haven't thought much is that eBooks are the very first thing that we're all able to have as much as we want other than air. Think about that for a moment and you realize we are in the right job." He had this advice for those seeking to make literature available to all people, especially children: "Learning is its own reward. Nothing I can say is better than that." Michael is remembered as a dear friend, who sacrificed personal luxury to fight for literacy, and for preservation of public domain rights and resources, towards the greater good. This obituary is granted to the public domain by its author, Dr. Gregory B. Newby.

8 10

these grapes are sweet -- lesson #06
by Bowerbird＠aol.com 10 Sep '11

10 Sep '11

this thread explains how to quickly and easily digitize a book. i did the book "books and culture", by mabie, which you can find by searching archive.org for "booksculture00mabiuoft". and i will share my secrets about how to do the job _fast_... *** i've done this lotsa times in the past, but the difference now is that i pledge to do most of the changes with just a text-editor, whereas usually i custom-program tools to help myself out... this time, if i do need a custom tool, i pledge that i will code it and furnish you the code so that you can run the tool yourself. don't fret if you're not a programmer, or not a very good one. i'm not a very good one either, but i can get the job done, and remember -- i'll be coding the tool, and handing you the code. moreover, to force myself to keep things as simple as possible, i will code any tools in _python_, which i've never used before... so since i'm learning it from scratch, i can teach it from scratch, and answer any just-a-beginner questions that you might have. *** so, to get our toes wet, i've appended our first program... it grabs the mabie text that we're using from archive.org and lists it inside your browser, with handy-dandy line-numbers... you can run this program here: > http://zenmarkuplanguage.com/grapes101.py you an also get the source-code at a similar address: > http://zenmarkuplanguage.com/grapes101.txt (i'll follow this naming convention throughout this thread.) as you can see when you run this program, the o.c.r. had quite a severe problem with floating semicolons. there are _444_ of them. but who's counting? ;+) -bowerbird #!/usr/bin/python import urllib import re f = urllib.urlopen("http://ia700300.us.archive.org /1/items/booksculture00mabiuoft/booksculture00mabiuoft_djvu.txt") s = f.read() f.close() s = re.sub("\r\n","\n",s) s = re.sub("\r\n","\n",s) s = re.sub("\r","\n",s) s = re.sub("\r","\n",s) s = re.sub(" \n","\n",s) s = re.sub(" \n","\n",s) s="\n"+s s=s+"\n" print "Content-type: text/html\n\n" print "<html><head><title>" print "grapes101.py" print "</title></head><body><pre>" pg = s.split("\n") maxpg=len(pg) startat=1 # will be 236 endat=maxpg # will be 7764 for i in range(startat,endat): if i < 10: print "000"+str(i)+" "+pg[i] elif i < 100: print "00"+str(i)+" "+pg[i] elif i < 1000: print "0"+str(i)+" "+pg[i] elif i > 999: print ""+str(i)+" "+pg[i] print "</pre><hr></pre></body></html>"

2 1

Re: Obituary for Michael Hart
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

jon richfield has some points, but since he is a person who "contributes quite actively" to wikipedia, he might not be able to see the border of the circle he is inside... :+) so it might not be as constructive as he makes it sound. (from "the office" t.v. show: "there is no inner-circle; i can say that for sure because i'm in the inner-circle. ...which does not exist.") you might remember that i explicitly asked if there were any wikipedia insiders here. because that's what it takes if you want to edit a page and you want the edit to stick. if jon is such a person, and he is offering to do an edit, then by all means, we should take him up on his offer... *** jon said: > Edit war is a 4-letter matter in those circles well, let's go take a look at how wikipedia defines it, ok? > An edit war or revert war is a situation that sometimes > arises on websites which are run on wiki principles, > such as Wikipedia, where users repeatedly > re-edit or undo or reverse the prior user's edits that doesn't really sound like "a 4-letter matter" to me. of course, some edit wars escalated to heights that are truly ridiculous (far worse than a mere "4-letter matter"), such as the ones on hot-button issues like abortion, but i certainly didn't imply that such heat would happen here. at the time of my first post, we just had one single edit... at the time of my second post, we had only the one revert. still, if theo10011 had reverted the revert _and_ it would have prompted _another_ counter, that'd be "an edit war", as defined by wikipedia itself, in the passage i just quoted. but... i have since gone back and looked at the edit-histories of the two parties, so i now suspect no "edit war" will occur... that's because one person has fewer edits than the other. and the one with more edits (50+ since just september 5) is also an employee of the wikimedia foundation, meaning that if the other editor doesn't want to get into a fight that he can't win, he'd better just smile and accept that revert... because yes, that _is_ the kind of political power-struggle which wikipedia editing has devolved into, over the years... luckily for us, and michael, the editor with more power is the one who calls michael "the inventor of the e-book", and not the one who called him "an inventor of e-book". but even if it was the other way around, and the page ultimately settles on "an inventor of e-books", so what? who cares? if anyone bothered to ask who the "other" inventors of e-books were, what would the answer be? vannevar bush? so, how many books did he put online? jeff bezos? oh really?, so what year did he "invent" 'em? won't be no "war" because there's nothing to fight over. but -- for the record -- the person with fewer edits has 50 edits since june 4th, so s/he isn't an outsider either. if you haven't made any edits in the last year, or _ever_, you darn well better not go try and edit michael's page. > Perfect? Not a hope. Important? Hugely. > As important as PG et al? In the long run certainly; > at present? Dunno. You tell me. wikipedia is not perfect; by far. but it is hugely important. indeed, it's probably even more important than "hugely"... (although then we'd just get into "an adjective war".) ;+) is wikipedia as important as project gutenberg? certainly. it's far more important, already, and the gap grows daily... wikipedia grows in stature more and more every moment. project gutenberg's importance has largely been eclipsed, by both google and the internet archive scanning efforts. michael's contribution is that project gutenberg was the _impetus_ that convinced the world e-books were viable, thus becoming the _spur_ for those big scanning projects. kickstarting that contribution was an important sub-plot, namely that -- if the powers-that-be refused to take on the task -- _we'd_do_the_job_ourselves,_as_individuals_. on the one hand, it's silly to call michael "the" inventor. thousands of people -- including me -- had the idea, and we got it all by ourselves, not from michael hart... heck, thousands of people _heard_ the idea _from_me_, originally, because i was busy telling everyone i knew... the thing is, i considered this to be a _big_ task, one for the librarian of congress, or an entity at a level like that. i'm guessing that michael would have been _delighted_ if the librarian of congress would've taken on the task, but what made michael different -- the thing that finally made me willing to grant him the honorific exclusively -- is that michael was willing to do the job _all_by_himself_, if he had to, a mentality going all the way back to 1971. michael grabbed the power himself, and refused to allow any of the big entities to tell him that they wouldn't do it. which is why, on the other hand, it is _not_ silly to give him the distinction and the title of being "the" inventor. michael refused to let the universe say "no" to his dream. he broke it down to the "one-book-at-a-time" level, and that made it possible for him to do it, and then he did it. and, by doing it himself, he became the model for others to do the job too -- one-book-at-a-time -- and after a period of many years, the effort cumulated to a _library_. it became an accomplishment that was worth celebrating. thus manifests the third wrinkle in michael's contribution: the fact that his individual effort attracted other people, such that the combination of their individual efforts soon cumulated to the clear reality of a dedicated community... and the emergence of distributed proofreaders is merely icing on the cake, in the sense that "one-book-at-a-time" naturally extends itself to the "one-page-at-a-time" idea. (this doesn't discount at all the brilliance of charlz franks in bringing about the _tech_ for distributed proofreaders; it merely acknowledges the genesis of the idea behind it.) i've just described them in their reverse chronology, but it's important that we understand the causal sequence... 1. michael says "i'll do this job all by myself, if i have to." 2. other people say "i'll help too", forming a community. 3. the resultant library proves e-book viability to google. (in case it rings a bell, yes, this "we'll do it ourselves" is precisely why napster was so important to digital music; it said "you can't pretend this capability does not exist"; of course, it's much easier to rip a c.d. than scan a book.) jimbo did the exact same thing with wikipedia, of course, except he had project gutenberg as a "proof-of-concept". still, the idea that individual efforts can cumulate into a synergistic whole creating an unbelievably large product that is extremely useful to society at large is _important_. of course, the end-note on both stories is a touch sour... project gutenberg has now become almost totally reliant on distributed proofreaders -- it lost its own community. and, of course, its library has been completely dwarfed... and distributed proofreaders is trapped in a distant past; it's now being controlled by a pack of technocrats which is entirely incapable of changing its obsolete workflow, so it has stagnated at a book-digitization of 2500/year; it'd take d.p. 1000 years to do google's 25 million books. the scenario at wikipedia is a little different, in that it has been taken over by the politics of trivial power struggles, rather than technocrats, but the end-result is the same: the vibrant community of volunteers which it once was has degenerated into a relatively small group of people. perhaps future visionaries will sidestep these problems, and be able to prevent the capture of their communities. we'll have to see. on the other hand, facebook might instead be showing how a clever puppeteer can manipulate a community for the purpose of renting them out to the highest bidders. -bowerbird

2 1

Re: Obituary for Michael Hart
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

I said: > now _perhaps_ kaldari will come back and > revert the change that theo10011 made, so > michael will again be called "the" inventor... and sure enough, while i was in the very process of writing my last post, that's exactly what kaldari did. now maybe theo10011 will concede, or perhaps it will become what wikipedia calls "an edit war", but the best advice to outsiders remains to _stay_out_. -bowerbird

2 1

Re: Obituary for Michael Hart
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

greg said: > In short, I think anyone who wants to > may edit the main Wikipedia page > (it's not locked or otherwise protected). the page is not locked, or otherwise protected. but the idea that "anyone who wants to" can edit a wikipedia page has been a myth since... oh, i dunno, about 2005 or so. you can _change_ a page, at least temporarily, but if one of the wikipedia insiders doesn't care for what you've written, they'll revert it. and there's not too much you can do about it. unless there are some wikipedia insiders here?... otherwise, let's look and see what they _have_, because that's likely to be what they will keep... first, they were very prompt entering his death. the timezones are a bit confusing, but it looks like they had it entered before greg's post here. (that's not surprising; there's competition over who will be the first person to enter any news.) and, as you'd expect, the attention meant that a flurry of edits have been made in the interim, most of them are rather inconsequential, but at 23:04 on the 8th, kaldari did an edit noting michael as "best known as the inventor of the e-book and the founder of project gutenberg", which i believe that michael would consider as the most important phrase describing himself. (and face it, "the most important phrase" is all that history and the great masses recall.) however, at 12:10 on the 9th, theo10011 did an edit (which s/he labeled a "minor rewrite") that changed "the inventor of the e-book" to "an inventor of e-book". this strikes me as a cop-out (and in addition the grammar's bad.) the washed-down version might be a bone that's being thrown just to avoid contention, because a lot of wiki editors shy away from statements that might be "controversial"... certainly there were people who wanted to bestow that "inventor" title to someone else; a common suggestion was vannevar bush, but i got the firm impression that some of michael's detractors would have been happy to have _anyone_ be awarded that little title, as long as it wasn't michael. petty bastards. now _perhaps_ kaldari will come back and revert the change that theo10011 made, so michael will again be called "the" inventor... or maybe it will just stick as "an inventor"... in the long run, it might not matter much. it's certainly true that michael is laughing about such trivia now that he has passed. he never got enough credit when he was alive and kicking, so why would he expect to get any more credit now that he's dead? but if the page _does_ get changed, it must be a change from one of the insider editors, who are all quite familiar with outsiders who want to barge in and "spruce up a page" as a way to deal with their grief from a death. -bowerbird

1 0

these grapes are sweet -- lesson #05
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

we set aside today's prepared lesson, in order to wish godspeed to our dear friend michael hart... we will attempt to continue tomorrow, because that's what michael would have wanted us to do. -bowerbird

1 0

oh man that hits hard
by Bowerbird＠aol.com 07 Sep '11

07 Sep '11

oh man, that news really hits me very hard. michael lived a very good life, and he got to see his dreams come true, so i am sure that he's happy upstairs. but it's gonna take me some time to get used to being without his spunkiness down here on the mortal plane. -bowerbird

2 1

these grapes are sweet -- lesson #04
by Bowerbird＠aol.com 07 Sep '11

07 Sep '11

the thread explains how to quickly and easily digitize a book... i did the book "books and culture", by mabie, which you can find by searching archive.org for "booksculture00mabiuoft". *** first off, a little note for the future lessons in this thread... in order to do your digitization, you're going to need to use a text-editor or word-processor that can do reg-ex searches. on the mac, i use a program called "tex-edit" (not "text-edit", but "tex-edit", where "tex" stands for texas) by tom bender... if you've been confused by regular expressions in the past, and are still gun-shy, don't worry about it. i promise that we won't become obsessed in the overly-complex details... but you need to be able to search for specific simple patterns, such as "lowercase followed by space followed by uppercase". i'll specify that combo as "lowercase-space-uppercase", and you'll need to know how to translate it into a reg-ex search. for text-edit, just to give you an example, that would be this: > [a-z] [A-Z] and you would specify that the search is to be case-sensitive. (tex-edit has a button for that, but your system might differ.) you also need to learn how to search for "whitespace", which might be a regular space, but might also be a linebreak or tab. (it's very important that you be able to search for a linebreak; some word-processors, even advanced ones, will not do that.) finally, another handy thing about a reg-ex search is that you can specify a search for multiple items, which is a time-saver. so you could do one search that would find a forward-slash and a backward-slash and a left-bracket and a right-bracket and a cent-sign and a tilda and a downward-sloping accent and an @-sign and a #-sign and a $-sign and a %-sign and an ^-sign and a & and an asterisk and "=" and an underbar. all in one search. which is faster than doing 15 searches... my pledge, in this series, is to do all of the digitization with _only_ my word-processor. if i do need a specialized tool, i will program it, and make it available online for you to see, and use, if you would like to replicate the work i am doing. (which would be a great way to learn how to do it yourself!) *** in the last lesson, we discussed "auto-cleaning" -- changes that are made to the o.c.r. file globally, without monitoring. auto-cleaning can fix a huge number of errors -- at times literally _thousands_ of them -- so it's incredibly efficient, and its great usefulness should never be underappreciated. for instance, in our mabie book, auto-cleaning made fixes to 543 lines, out of the 5700 in the body-text, right at 10%. those are fixes that cost you zero time or energy. right on! but that's just the first type of cleaning you will want to do... *** so now let's discuss the second type of cleaning you can do. after having done auto-cleaning, i turn to "high-probability" searches that quickly locate o.c.r. errors which are common. over the years, i've posted messages to educate this list of an amazing number of routines that help find o.c.r. errors. once again, here i am, doing the same thing another time... right about now, you're probably starting to get pissed off with my pedagogical tone... you might be saying "boy, that bowerbird certainly thinks he smart, doesn't he?" or maybe you'd like to take me down a few notches, knock me off my high horse. that's why the people on this list _rarely_learn._ they're too busy getting all riled up because of my "tone"... if you woulda been paying attention, instead, you would've learned quite a lot over the years, and you'd be thanking me. lucky for you, though, someone once created a collection of my tips, translating them into regular-expression searches, and posting them to the web somewhere... i'm not going to tell you where -- mostly because i never wrote that down -- but maybe someone did, and is willing to share it with you. for now, as usual, i'll tell you the routines that would make the "minimal set" for finding the unique errors in this book. *** there is a great assortment of cleaning routines that i do which are ones where i manually approve whether a change will be made or not. a very good example of this is the search for "11" (eleven). the replacement-term will be "ll" (lowercase letter "l", twice). here are the lines this search locates in the mabie book: > Chapter 11. > 112 > 8 113 > 114 > 117 > 119 > "They *11 take me there some day," > 211 as i expect whenever i run this search, there are a number of "false-alarms" that turn up, where the number 11 is correct... nonetheless, this is still a very productive search, because it helps to locate some problems. first of all, that "chapter 11" isn't actually chapter 11, it's chapter 2, in roman numerals, so that's an o.c.r. error that needs to be two uppercase "i" letters. next, the "8 113" line should really just be "113", because it's the pagenumber for page 113. but the "8" _was_ on the page, yes indeed, as the signature-number, so this wasn't "an error". but since we don't need the signature-number, we'll delete it. (books are usually printed as a collection of 16-page sections, which are called "signatures"... if you look closely at the spine of a book, along the top edge, you'll see these signatures and how they were sewn or glued together. back in the olden days, printers numbered the signatures, so as to keep 'em straight.) next, in the "take me there" line, there _is_ a misrecognition of "11" (eleven) for "ll" (two lowercase "l"), which is _exactly_ what we were seeking with this search. but there is _also_ an error with the apostrophe plus a floating-space before it, so this line needs hand-editing, not just a search-and-replace... finally, there are some "dogs that did not bark" in this search. where is the pagenumber for page 11? it shoulda turned up. likewise with the pagenumbers for 111, 115, 116, and 118... the fact that they are missing means we should go look there. so this is an instance of a search yielding a rather high payoff, both in the lines that it found (3 of the 8 of them had errors) and also (once you grasp it) in the lines that it did _not_ find, since it indirectly pointed to 5 other errors in the book's text. *** another neat example is the _mid-line_ floating-doublequote. take a look at one of the sets of lines from our last lesson -- where we fixed floating-doublequotes at the start of the line. we see that most also have mid-line floating-doublequotes: > "thrift of time," which brings ripe- > "Chronicles " and North's trans- > "Still studying Dante ? " said the > "Divine Comedy " or in Goethe's > "Faust " for the first time discovers > "Master and Man," is one of those > "Odyssey " are of more importance > "In any museum," says Mr. La > "Anna Karenina " leaves no reader if we do a search for this space-doublequote-space combo, there will be two very-common replacement-terms, namely _either_ space-doublequote _or_ doublequote-space. your first instinct might be just to find the next occurrence, and then edit it by hand. but that's not the right approach... if at all possible, you want to resist the urge to hand-edit. it's much smarter, faster, and easier to search-and-replace. so, in this case, you do the search-and-replace _two_times_. first, you specify doublequote-space as the replacement-term, and you approve all cases where that is the correct replacement (this is the right fix for the lines given above), and skip the rest. then the second time, you specify space-doublequote instead, and approve all cases where _that_ is the correct replacement. (since you already fixed the other cases, it should be all cases, but there might be some exceptions, so you still need to look.) some people here will undoubtedly recall that i have a program that can correctly fix such floating-doublequotes automatically. but, as i said above, i'm trying to work only in a word-processor. since there were only 59 cases of these floating-doublequotes, the task was manageable enough that i could handle it manually. *** a lot of times, people ask me to provide a "full set of searches"; they want to have a list that can find every error in every book... you _can_ get something that comes close. but you pay a price... almost every search-routine will have some "false-alarms"... (remember that we had a couple when we searched for "11"?) if a routine finds enough errors, the false-alarms are "worth it". but still, it takes time and energy to process those false-alarms. so you only want to run the routines that will locate errors... i know what you're thinking: "i'm running the routines so i can _find_ the errors, because i don't know what errors are there, so how do i know which routines will be the ones to find errors?" it _does_ seem to be a bit of a knot, but the escape is simple: if you listen to the text, you will hear it telling you the errors. let me explain... when you're correcting one error, you'll notice other errors... when you are doing your non-automatic cleaning, you need to have a certain mindset. on the one hand, you must be focused. when you start a specific change-routine, you have to finish it; don't let yourself be distracted by other errors that will pop up. on the other hand, don't put blinders on yourself. you must be open and attentive to the text. when you're correcting an error, be mindful of other text around the error, and any errors in it... whether you correct the other errors -- or leave 'em for later -- is immaterial. what is important is that (1) you retain your focus on the process of continuing what you had been doing _and_that_ (2) you make a note of the nature of the other error you spotted, so that you can later do a specific search for that _type_ of error. because if an error exists, it's likely that other such errors exist. and if you "listen" to the text, it'll tell you about all of its errors... another thing that people have asked is about any "order" they should perform the search-routines in. it doesn't matter much. because each search-routine will lead to other search-routines, if you're listening to the text. so eventually you'll find all errors. in this regard, however, i will say that a simple _spellcheck_ is quite likely to find the greatest number of errors most quickly. one thing about spellcheck is that it isn't much of a challenge... i like to clean up text because it's something akin to a _puzzle_, and i enjoy the game. a brain-dead spellcheck ruins all my fun... but if you just want to speed up the job, do a spellcheck. be aware, though, that spellcheck will _not_ find everything, so just because you spellchecked doesn't mean you're "done". there are lots of digitization errors that can survive spellcheck, and i'll discuss one of them right after this. before concluding with spellcheck, though, i should tell you that the secret of spellcheck is to add all the "correct" words to your dictionary, so subsequent occurrences are not flagged. this means proper names, of course, but it also means things like archaic spellings (if you're retaining them in your output), and slang, and dialect contractions, and all of that other crap. this means that it's best if your spellcheck program can use a _customized_dictionary_. (because you don't wanna be adding archaic spellings to the dictionary you use for other documents.) but once you've added all the "unique" words to your dictionary, spellcheck is a breeze, because -- unless you screw stuff up -- spellcheck will run on the whole document without a single stop. which means you can do it often, to ensure you didn't screw up. but remember -- a clean spellcheck doesn't mean a clean text. *** there was an interesting situation in this book. if you want to follow along, you can pull up the o.c.r. file: > http://ia700300.us.archive.org/1/items/booksculture00mabiuoft/booksculture0… (that u.r.l. will probably be split, so you'll need to rejoin it.) i'll draw your attention to page 9, where we find these lines: > thoughts, acts, dispositions, and pas- > sions of humanity. There is no > getting to the bottom of Shake- > speare, for Instance, or to the end of > his possibilities of enriching and In- > teresting us, because he deals habit- > ually with that primary substance > of human life which remains sub- > stantially unchanged through all the > mutations of racial, national, and > personal condition, and which Is al- > ways, and for all men, the object of > supreme interest. Time, which is > the relentless enemy of all that Is > partial and provisional. Is the friend > of Shakespeare, because It continu- > ally brings to the student of his more specifically, look at these lines: > his possibilities of enriching and In- ... > the relentless enemy of all that Is > partial and provisional. Is the friend > of Shakespeare, because It continu- you can see that we have a problem with the o.c.r. program turning small "i" words into capitalized ones. this is a glitch i've seen before, and i've even discussed it on this list before. it's easy to understand how the dot on the lowercase-i could mislead the o.c.r. that it is the solid line of an uppercase-i... (and, of course, just to follow up on the point made above, spellcheck doesn't flag this as an error -- its spelled write!) so we'll need to fix these glitches. specifically, we need to change lowercase-space-capital-i to lowercase-space-lowercase-i, replacing the mistaken capital. in most books, this replacement won't introduce many errors, if any at all, so i have auto-changed this before (albeit rarely). so, since it was a common mistake in the o.c.r. for this book, with a count of 69 such instances, should we auto-change it? i've appended the list of instances. take a look at it, and see whether this change would be good to do as an auto-change, or whether we'd need to approve each as a monitored change. go ahead and make your assessment now, i'll wait... so, did you think it would work as an auto-change? starting at the top, we see auto-correction would apparently work just fine, including for all those in the upper two-thirds. so you might've got tired of looking, and said "auto-change". however, toward the bottom, we see we may have a problem. and yes, a glance at one of the scans tells us we do indeed... specifically, this author has given "honorific capitalization" to the terms "idealism", "idealist", "idealists", and "ideal"... (elsewhere in the book, he contrasts idealism with realism, so "realism", "realist", and so on, also receive capitalization. and, for the record, so does "nature"; mabie's a tree-hugger.) so, in this book, this replacement as an auto-change would have meant more than a typical number of introduced errors. and not just auto-change. sometimes even when you are "monitoring" a search-and-replace, you go on auto-pilot, and just start pressing "replace" without actually _looking_ to consciously confirm that the change is really necessary. so be aware of that, and guard against it. keep your focus, and remember to _also_ be aware of the surrounding text... *** so this capital-i thing was an interesting twist in this book. but it gets even better! take a close look at the middle line: > the relentless enemy of all that Is > partial and provisional. Is the friend > of Shakespeare, because It continu- this error would _not_ have been isolated by our find routine, because the capitalized-word "is" was preceded by a _period_. it just so happens that -- in the reality of the print-book -- that "is" word was preceded by a _comma_ -- not a period -- but the o.c.r. _also_ misrecognized that comma as a period... so my error-finding routine wouldn't have isolated this line, because it's looking for cases of lowercase-space-uppercase. we just happened to see the error in this line because we're looking at the identical error in the lines which bordered it. such are the intricacies of correcting an o.c.r. file... :+) *** in our next lesson, we'll start getting down to the nitty-gritty of finding and fixing the errors in the text, so stay tuned... -bowerbird 69 instances of lowercase-space-capital-i > speare, for Instance, or to the end of > his possibilities of enriching and In- > personal condition, and which Is al- > the relentless enemy of all that Is > of Shakespeare, because It continu- > is to Invite an intimacy with It which > is to Invite an intimacy with It which > The man who Is bent on getting > is by no means Irresponsible ; It may > is by no means Irresponsible ; It may > of giving Itself up to idle reverie ; > "Spectator" and turned them Into > The man who is thrown Into constant > association with Inferior work either > of the genius of a race or an ageo It > to enter Into the thought of the un- > Idea and feeling so deftly woven to- > gether, and follow each back to Its > man of culture shares In no small > Demeter this Is precisely what Mr. > " TT Is undeniable," says Matthew > teachers Impart the breath of life by > giving us inspiration and Impulse. > paintings and of a gallery of Italian > he receives it into himself, and In the > and he is sufficient unto himself In > 'X'HERE IS a general agreement > tion, " Home for Incurable Children." > other words, culture had wrought Its > tinct from himself. There Is no easy > because It Is a matter of growth. > because It Is a matter of growth. > and possessing them in the Imagina- > education ; but genius Is not only in- > the Immense majority of men any ef- > must not be Inferred that a man Is > must not be Inferred that a man Is > laws, habits, or Institutions. It is the > itself out In material and social rela- > a main or central movement In > prominence, Imbedded themselves > These events are described In narra- > tive form, with episodes, Incidents, > be found In the pages of Homer, > Idealism, that, like other spiritual > The true Idealist has his feet firmly > as much as Idealism from weak practi- > The essence of Idealism is the ap- > he defined the Ideal as the comple- > rational Idealism sees all these things, > that the Idealist asks is that life shall > This is true Idealism ; but it is > acterises true Idealism, and which > makes all the greater writers Idealists > of the representative Idealists of his > bits of true Idealism which has been > Idealists in the breadth of their vision > Idealism which not only discovers the > reality of the Ideal as Plato saw it is > A rational Idealism is, therefore, > of the Ideal in personal and social life > stroying a false Idealism, convention- > true Idealism which the world has yet > Arden and on Prosperous Island there > Idealism which grows out of the vision > such an Idealism, and no spiritual > great Idealists is one of the greatest > ter *'As You Like It" must give > of "As You Like It" is not only

1 0

these grapes are sweet -- lesson #03
by Bowerbird＠aol.com 06 Sep '11

06 Sep '11

the thrust of this thread, once again, is how to digitize a book. i went through all the steps, and will regurgitate them for you. i used the book "books and culture", by mabie, which you can locate by searching archive.org for "booksculture00mabiuoft". as i said in my last post, the o.c.r. was clean, which is typical, with around 97% of the body-text lines recognized correctly, at least _after_ i'd run my automatic cleaning-routines on it... people who've been here for a long time will recognize that this is a finding that i have obtained over and over and over. most of the time, you'll get errors in roughly 3% of the lines. that depends, of course, on the quality of the printed book, how much care you take in the scanning process, and so on. but another big variable is how much _cleaning_ you can do. *** the first type of cleaning is something i call "auto-cleaning". when i say "auto-cleaning", i'm talking about global changes that can be made to the text _without_any_ manual approval. ("manual approval" is where you accept/reject each change.) an example of such auto-cleaning would be a global change of all instances of a space-comma (" ,") to just a comma (","). in old books, o.c.r. often puts a space in front of punctuation. the punctuation characters were actually _printed_ with some extra space in front of 'em, at least according to the standard of modern-day typesetting, so it isn't really an o.c.r. "error"... but it's something we'll want to replace in our digitized text, so it's a global change we can make without manual approval. if you can do a lot of auto-cleaning, you can drop the errors down to the 1% range,,. if you can't, it might float up to 5%... *** one of the risks that you take with auto-cleaning is that you aren't actually looking at the changes that you're making, so _might_ change something that's right to something wrong. is it possible that auto-changes might _introduce_ an error? sure it's _possible_. but with some changes, it's so unlikely we don't need to worry about it. i certainly don't feel i must "manually" approve every change of space-comma to comma. if such an auto-change does introduce an error, i'll just trust that a reader from the public will find the error and report it. now, obviously, if the auto-cleaning fixes 998 errors but also _introduces_ 2 new errors, that's probably "worth it" anyway, as it's easier to clean the 2 new errors than the 998 old ones. but it helps to know what kind of errors you're introducing, if/when you are, so i always like to review the results of any new routine that i might be considering for auto-cleaning... *** another example of an auto-clean routine is to correct the "floating-doublequote" when it happens at the start of a line. o.c.r. often has trouble with doublequotes in older books, putting an extra space in front of them or behind them... (in real life, with modern-day conventions, a doublequote has a space _either_ in front of it _or_ behind, or _neither_ if parentheses or brackets are involved, but _never_ both.) a floating-doublequote in the middle of a line is ambiguous, since the extra space might've been introduced in front of it, or might've been introduced behind it. but at the beginning of a line, the extra space was obviously introduced behind it, so can be removed with a no-look global search-and-replace. some line-starting floating-doublequotes in the mabie o.c.r.: > " thrift of time," which brings ripe- > " Chronicles " and North's trans- > " Still studying Dante ? " said the > " Divine Comedy " or in Goethe's > " Faust " for the first time discovers > " Master and Man," is one of those > " Odyssey " are of more importance > " In any museum," says Mr. La > " Anna Karenina " leaves no reader likewise, floating-doublequotes at the _end_ of a line are also candidates for a no-look auto-cleaning search-and-replace... again, the 3 examples from the current book: > guidance ? " > they make. The student of" Faust " > No man can read "In Memoriam " as that first example shows us, question-mark characters are also often "floating", and a good candidate for auto-cleaning. (likewise with exclamation-points, semicolons, and colons...) *** auto-cleaning also brings up some "philosophical" questions about the digitization process... for instance, i close up any spacey-ellipses (". . .") so they are tight ("..."), _plus_ i globally change all 4-dot ellipses to 3-dotters (wanna fight about it?); i also merge each ellipse with the word that it follows (usually) or that it precedes (when the ellipse is at the start of the line). i always change "to-day", "to-night", and "to-morrow" so that they lose their unnecessary dash. these are "in-house styles", so i don't expect you to agree, and don't care if you disagree, so it really won't be productive to discuss this kind of issue... you don't have to do all my auto-changes if you don't want to. *** so that takes care of auto-cleaning... there are other types of cleaning, and i'll talk about them in our next lesson... -bowerbird

2 1