- gutvol-d - lists.pglaf.org

these grapes are sweet -- lesson #10
by Bowerbird＠aol.com 15 Sep '11

15 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** in the last lesson, we culminated with a program that gave us a search capability on the original o.c.r. text-file of our book. we can make that program work on our edited text as well. the newest version of our edited text is here: > http://zenmarkuplanguage.com/grapes002.txt you can use your comparison tool to see the changes made from the previous version... but i'll tell you that one set of changes was to ensure all of the pagenumbers are correct. and since the pagenumbers are correct, we can show them, (rather than the linenumber in the file) for lines we display. so we'll do that. we'll show the pagenumber for each line, and we'll include the linenumber _on_that_specific_page_... (so we kinda know if the line is up top, middle, or bottom.) plus, as we said earlier, having this information be correct means that -- once we pull out any particular line of text -- we can show (or link to) the _scan_ of the page containing it. so let's incorporate these additions into our search program. the search program, now working on grapes002.txt, is here: > http://zenmarkuplanguage.com/grapes109.py > http://zenmarkuplanguage.com/grapes109.txt as you can see, once you run a search, each located line is preceded by the pagenumber:linenumber -- plus the line (in its entirety) has been made into a link to its page-scan. click on a line to refer to its scan to resolve any questions. for an example of this, consider that -- when i am doing edits on the text -- if there is something about which i am uncertain, i'll mark it with a "??", so i can return to it later. when i was doing the "first-pass" edits to this file, i didn't bother to look at pagescans, but if something was unclear, and i couldn't make the decision without seeing the scan, i just used the "??" notation. so, now that we have links to view the pagescan for any line found by the search routine, you can play along with me and search for those "??" lines... the tool will show you these: > 052:10 _____ the 'Myth of the Soul' ?? I discovered > 092:18 _____ spell which they all possess, ?? -- the > 100:06 _____ sonalities ?? the atmosphere of such re- > 105:22 _____ Kourotrophos ?? the mother of corn and > 138:18 _____ ?? climate; intensity of heat and light; > 146:22 _____ characterise ?? the man of culture will be > 266:18 _____ us. They disclose wide ?? diversity of > 275:11 _____ in the educational ?? of the indi- it's simple enough to click the line, so as to view its scan. for instance, clicking the top line will display the scan for page 52, and we know the line is #10, so midway down... and yes, the scan tells me that there _was_ a comma there (which is what i had been unsure about). so i will fix that. indeed, i can now fix all of those lines in similar fashion... *** another thing i've added is the ability to list all of the lines on a particular page. just enter "p#123" in the search-box, for instance, and all the lines for page 123 will be shown... and all of the lines are links to the scan for page 123, yes, but in this case, we obviously know we wanna see the scan, so the program just displays it, without us having to tell it. *** if you are paying attention, all this should give you lots of ideas about what could be done next, so think about that... -bowerbird

1 0

these grapes are sweet -- lesson #09
by Bowerbird＠aol.com 14 Sep '11

14 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** i showed you the list of edits which i made in my "first pass", all of which i found using various searches in my text-editor. the auto-change that stripped the trailing space off every line touched... well... every line, quite obviously, so in that sense, every single line of the original o.c.r. text-file was changed, albeit in a terribly uninteresting way. if we look at non-trivial auto-changes, there were far fewer... we auto-changed floating punctuation, but that was merely a couple lines for most of the marks, which the exception of (1) linebreak doublequotes, where 19 lines were fixed, and (2) semicolons, where some 444+ instances were corrected, which is a big number, but it remains an uninteresting edit. mid-line floating doublequotes got changed manually, with roughly 62 of those corrected. in most books, i would _not_ correct those glitches by hand, but since there were so few... but again, all of these are boring in the extreme... probably the most interesting set of edits were those which involved "weird" characters. using reg-ex, it's easy enough to locate such characters in a single search, if you choose... which means that it's easy to write a program to list the lines that contain any such characters. that's what we'll do today, so you can actually see the lines located with that routine... this is a useful program that you'll want to run on any file when you begin to work on it, to see what problems it has. it helps you zero in on the most common o.c.r. problems... and again at several points in your workflow, you'll want to run it again, just to make sure that you haven't introduced any of these weird characters into your text by accident... this program is a modification of the last one, adding in an array of characters that we want to flag, then looping through the lines of the file and listing any of those lines which contain a flag characters. you can run this program, or grab the code, here: > http://zenmarkuplanguage.com/grapes103.py > http://zenmarkuplanguage.com/grapes103.txt *** again, that code lists the lines by their order in the file... (it displays the linenumber for your convenience as well, since many text-editors will let you jump to a certain line. if you don't know of such a feature, you should look for it.) now, we might also want to group the lines according to the flag character, so we see all the lines that have a dollarsign (for instance) listed together, so we'll rework the code a bit. you can run this new program, or view the code, here: > http://zenmarkuplanguage.com/grapes104.py > http://zenmarkuplanguage.com/grapes104.txt this produces a nice grouped list, plus it also shows us the flagged terms which produced _no_ instances from the file. as you can see, most of the listed lines _are_ o.c.r. errors. so, for the most part, once we've cleaned up these glitches, whenever we run this program again later in our workflow, it should produce very few lines. it _will_, however, produce _some_ lines. at least it should. for instance, we're flagging "11", because in a lot of books, o.c.r. misrecognizes the end of words like _they'll_ or _we'll_ as an _11_... (you'd think it'd be smarter than that, but no.) indeed, as you can see from the output of our little program, there was one such misrecognition in this file, on line #4938. it's also the case, however, that there will be a lot of lines where "11" is the correct recognition, like the pagenumber for page 11, for instance. and for 111 and 211 too. d'uh. don't forget 112, 113, 114, 115, 116, 117, 118, and 119... but we see no "111" in this file, as the pagenumber for 111 was misrecognized as "iii". we also have no 115, and we see (because we searched for "ii" as well) that 116 and 118 were misrecognized, with the "11" seen as a pair of lowercase i's. also, pagenumber 113 has a signature number we'll erase... (this informs us we need a search to find that phenomenon, which will be a reg-ex looking for number-space-number.) likewise with that "ii". few english words have an "ii" pair, but imagine if you were digitizing a book about "skiing"... the goal is to catch o.c.r. errors but minimize false-alarms. our grouped listing also shows, in this text, that asterisks were usually a misrecognized doublequote (or singlequote). and we see we had a few glitches where "w" was seen as "v/". also note that we had problems involving the insertioncaret. of specific import is that some of these caret problems were on pagenumbers, which will be particularly important to us... *** the linenumbers of the file, which we've been listing, are good. but there's better positional information inside the text itself, i.e., pagenumbers (plus, by extension, per-page linenumbers). the pagenumbers give us the vital ability to _display_the_scan_ for any particular line of text, so we wanna correct 'em _now_, so we start benefiting immediately from that great capability. so let us write some code to help us correct the pagenumbers... that's rather simple to do, because our expectation is that they will march along for us in a sequence that's entirely predictable. so we'll just look closely for any departures from that sequence. *** the routine that helps clean up pagenumbers is found here: > http://zenmarkuplanguage.com/grapes105.py > http://zenmarkuplanguage.com/grapes105.txt you can puzzle through the code later if you like, but for now just look at the output. what you'll see is a line of "xxxxxxxx" wherever we get a departure from the next-expected-number. the program also shows us any short lines that it encounters, which helps us immensely to see where the problems might lie. it also lists the actual contents of the line, not just the number which it expected to find. sometimes a line "reduces down to" the expected number, but it has some garbage characters in it, and we're gonna wanna eliminate those garbage characters... and as before, this is a program that we're gonna wanna re-run every so often, to make sure we didn't mess up any pagenumbers. indeed, we might want to leave this routine in all our programs, and have it be silent as long as everything checks out fine, but have it bring notice to our attention if it suddenly finds an error. so, let's run this code against the file that i edited, to check it. > http://zenmarkuplanguage.com/grapes106.py > http://zenmarkuplanguage.com/grapes106.txt oops! as we can see, i fixed most of the pagenumber problems, but not _all_ of them. there's still a bug on pagenumber 61, and pagenumber 183 is still missing-in-action, and pagenumber 228 was misrecognized by the o.c.r. as "328", so we have to fix that... we also have a listing for linenumber 5957 and linenumber 5959, where a footnote number is confusing our routine. we can fix the code so that it ignores such footnote numbers in the future. *** so far, we've taken the general approach of _searching_for_ the particular character(s) which might be an o.c.r. misrecognition. that's an ok approach if we know what we should look for, but sometimes we don't. so we can take the _opposite_ approach, and say "these are _good_ characters, so ignore all of them, but show me any characters which are _not_ these good characters". that's the code we've got here: > http://zenmarkuplanguage.com/grapes107.py > http://zenmarkuplanguage.com/grapes107.txt what we find when we run that is some high-bit characters, which we were not expecting to find in 7-bit o.c.r. output... remember how i told you that archive.org was "losing" their em-dashes, and i had to badger them for years to fix that? well, when they "fixed" it, they did so by introducing a utf8 character into their otherwise-all-7-bit-files. what idiots! these are technocrats of the very worst kind, ones who will introduce unnecessary complexity into the simplest of files. oh well, it's just another global change, so it's fairly easy to ignore this particular obstacle the technocrats threw at us... besides em-dashes, there are 2 other lines that show up... those were also high-bit characters that got put in the file. but we've isolated them, so it's easy enough to delete 'em... *** now that we've got search-related routines in the program, might as well code first pass at a general search capability. that's here: > http://zenmarkuplanguage.com/grapes108.py > http://zenmarkuplanguage.com/grapes108.txt now we can search the file for any word we want... this is pretty simplistic, but it's much farther along than we were before we coded it, so let's enjoy our progress... -bowerbird

1 0

Re: these grapes are sweet -- lesson #08
by Bowerbird＠aol.com 14 Sep '11

14 Sep '11

hunter said: > i am the lead developer for guiguts well, hunter... given the poor probability d.p. will change its antiquated workflow, it's great that a person like you is willing to try to smooth and improve the tools... thank you for sharing. -bowerbird

1 0

these grapes are sweet -- lesson#08
by hmonroe.pglaf＠huntermonroe.com 14 Sep '11

14 Sep '11

I am the lead developer for guiguts and have made some progress in cleaning up its many bugs. I would be happy to include any useful scripts from Bowerbird or anyone else. Any volunteers with perl experience to help with clean up bugs and adding new features would be appreciated. We are providing a Windows executable already and can provide a Mac or Linux executable if someone volunteers to build it. See the latest release notes below. Hunter ------------- This is a major version which will become 1.0 once the bugs are ironed out (see this discussion http://www.pgdp.net/phpBB2/viewtopic.php?t=48584) Note: upgrading to this version requires a complete reinstall (save your header.txt and setting.rc files). Relative to version 0.2.10 there are several major new features. First, all the HTML checks can be run with a single click, and the output is clickable in most cases. Second, HTML and CSS validation can now be done on your own computer (and PGTEI as well) and there are checks for unused CSS and image issues (using rfrank's pphtml, ppimage, and pptxt scripts). Third, there is now an option to view text and images side-by-side without having to click on "See Image" for each page. For instance you move forward or back one page for both text and image with the "<" and ">" buttons on the status bar. Also, the Auto Show Images option let's you see the image for instance for the page you are spellchecking. (See tips on viewer configuration below.) Fourth, with Winguts there is no need to install Perl on Windows computers (for other platforms see below). Other new features are: a "View in Browser" and Hyperlink page numbers buttons on the HTML palette, tearoff of the Unicode menu, listing small caps in the Word Frequency popup, automatic checking for updates (which can be turned off), horizontal rules as css, if nothing is found the cursor returns to the starting point, better ability to find executables automatically, GutWrench scanno files are included, inclusion of rfrank's pphtml and pptxt scripts, a warning to use human readable filenames, option to include goodwords in spellcheck project dictionary, a text processing menu to ease conversion of bold/italics/small caps, the label Image #nnn in Configure Page Labels is clickable, added Find Transliterations and Find Orphaned Markup (before it only searched for unmatched brackets) to Search menu, Adjust Page Markers menu is accessible from the File menu. For developers, there are internal improvements, including refactoring of functionality into perl modules and a unit testing framework. Bug fixes already included: less mangling of HTML page numbers (for instance [Pg 42-44] instead of three overlapping page numbers), superscripts are converted to HTML correctly (Philad^a) without curly brackets, fixed regex editor for scannos, Ctrl-S saves the file. The side by side image viewing works best if the window for the viewer is sized to match the image (in XnView, choose View, Auto Image Size, Fit Image to Window) and only one instance of the viewer is allowed to avoid having one instance for every page viewed (in XnView, choose Tools, Options, General, Only One Instance). To page through images, use the "<" and ">" buttons on the status bar. To Auto Show Page Images, use the "Auto Img" button on the status bar, use the option on the Prefs menu, or checkboxes in the various search/spellcheck dialogs. I would be happy to release Mac and Unix executables which only requires someone to run tkpp following the instructions in COMPILING.txt. A Mac or Unix distribution should also include the OpenJade/OpenSP onsgmls executable for HTML/PGTEI validation. Detailed release notes: Version 0.3.6 There is a much higher likelihood that this version generates valid HTML. Page anchors are no longer placed at the end of the previous paragraph or before the horizontal rule. The default for word search from the Word Frequency menu is now "Whole Word". Fixed Go To Bookmarks. There is only one file guiguts-0.3.6.zip which includes the WinGuts.exe. Version 0.3.5 Fixed multiple page markers at a single location so they do not overlap but stack vertically like [Pg 32]<br />[Pg 33]. Fixed problem with moving mark left (entry for initial page number was blank) or up (code was garbled). Removed splash screen. Fixed monthly update check to give user the option. Fixed openpng so if the image viewer is not set the user is prompted to set it. Version 0.3.3 Fixed problem in WinGuts.exe with conflicting Tk library. Version 0.3.2 Fixed pathname problem for non-Windows system for splash screen.

1 0

these grapes are sweet -- lesson #08
by Bowerbird＠aol.com 14 Sep '11

14 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** the original o.c.r. text-file, copied from archive.org, is here: > http://zenmarkuplanguage/grapes000.txt the file, after its first round of editing by me, is here: > http://zenmarkuplanguage/grapes001.txt here are the changes i made in my text-editor: eliminate: front-matter (for now) eliminate: back-matter eliminate: blank-lines in front of numbers-in-column-1 auto-change: delete trailing spaces auto-change: 3-blank-line sequences with === line auto-change: double-quotes at line-start auto-change: double-quotes at line-end auto-change: floating semicolons auto-change: floating colons search: floating commas search: floating periods search: period at line-start search: comma at line-start search: single-quotes at line-start search: single-quotes at line-end search: letter-number combo search: number-letter combo search: number-space-number combo search: one-letter-lines search: two-letter-lines search: three-letter-lines search: four-letter-lines search: one-character-lines search: two-character-lines search: two-number-lines (line #s from 10 to 99) search: chapter-headers & subheads & dropcaps/smallcaps search: singlequote-singlequote search: floating doublequotes search: floating singlequotes search: opening singlequotes search: closing singlequotes search: lowercase-uppercase search: uppercase-uppercase (roman numerals) search: 11 ii vv search: lowercase-linebreak-linebreak-lowercase search: comma-linebreak-linebreak search: [@#$%^&*()_+{}\|] auto-change: utf8 emdashes fixes: various fixes of stuff i encountered along the way they should all be pretty obvious, but if you have any questions, don't hesitate to ask me. this is the important "first step" here. *** your mission, if you choose to accept it, is to edit the text-file, exactly as i described above, and compare your results to mine. a good file-comparison utility is an essential when you digitize, so that you can compare subsequent versions of your text-file, to ensure that you made (1) the changes that you intended, and (2) no _unintended_ changes. find a good file-comparison tool. i use and like diffmerge, which is cross-platform and cost-free: > http://www.sourcegear.com *** i get backchannels. i'd much rather talk frontchannel, because dialog is one of the objects here, but i do get backchannels too. like this: > Why not put your project documentations in a blog/tumblr? > It seems it has far wider general interest than the PG project. i might put some stuff out in a location that's a bit more public. or i might not. it doesn't seem to me that there are many people looking for tools to help them digitize books. i wish there were. but mainly it's project gutenberg with distributed proofreaders... for a while, i posted in the forums at distributed proofreaders... i carefully documented many problems with their workflow, and i volunteered to write software for them, software they _needed_ -- desperately -- way back then, and ever since, to this very day. but they didn't like my constructive criticism, so they banned me. that's right, they _banned_ me from posting... can you imagine? merely because i was telling them they were doing it _wrong_, they decided to declare me persona non grata, stick their heads "in the sand" (if you know what i mean), and simply pretend that i hadn't offered to help them -- a fact they continue to ignore... meanwhile, their volunteers have struggled with primitive tools, again _to_this_very_day_. indeed, here's a recent forum thread: > http://www.pgdp.net/phpBB2/viewtopic.php?p=788130 as background, you should know that "guiguts" is the name of the software they currently use to do "post-processing", which is shortened to "pp", so "a new pp-er" is a person who wants to learn post-processing, and has thus set out to install "guiguts". the thread begins like this: > As a new PP-er who has just struggled, cursing all the while, > through installing Guiguts under MacOSX, could I endorse > the observation that: > > "It would be of more benefit to find a middle ground. > We should not be so dependent upon difficult or > buggy software that it interferes with our production." > > And dare I look forward to the development of a Mac version > of Guiguts that is as simple to install as (say) TextWrangler? > If prohibitively difficult to encompass all of Guiguts' > bells and whistles could a subset be useful and feasible? this trouble installing guiguts (and then using it as well) has been experienced by tons of people over the past many years. and thus d.p. volunteers have been crying out for better tools. the "powers that be" at d.p. -- that's what they call themselves, it's not (just) a disparaging term that i have applied to them -- keep saying "we don't have programmers to code better tools." but that's a bald-faced lie, and they know it. they chased me from their midst, and they did it intentionally (and with malice). and meanwhile, their volunteers suffer with outdated tools... so one reason i post here, and not there, is because i'm banned over there... but i can still rub their nose in it by posting here... their people moan, the "powers that be" respond by saying that they have no programmers, and i come here and say "i program, and i code apps that execute on the full range of offline systems, and i've written clean-up programs, and you can have them _if_ you ask me for them", and of course nobody ever asks for them. it goes deeper than that, though. i have a respect for project gutenberg, and for michael hart. when i was being attacked by the wolf-pack here on this list -- for a period of _years_ -- michael defended my presence. the idiots here tried to get me banned here, but _michael_ wouldn't let them get their way. he knew i had something to offer to the cause, and he refused to let them bully him into banning me. so, in my mind, he _deserves_ my posts. he _earned_ 'em. (it's hard to speak of him in past tense.) but it goes even deeper than that. the cyber-archeologists of the future will want to study the evolution of thinking about e-books, and this listserve will be one of their major watering holes. so i want to be here, and i want to have my posts be here. (of course, one of the main reasons why this list's archives will be so valuable to these future scholars will be _because_ my posts are here... so there's a certain degree of circularity going on with this.) and then, of course, we have the dynamics of the past... if i put my stuff on an open blog, it would mostly be ignored. of course, i'm ignored here as well. but at least when i post to this list, i _know_ the actual _people_ who're ignoring me. and i know they're ignoring me at odds to their own interest. which amuses me, making the whole thing that much sweeter. so i just gradually notch it up, bit by bit, as the years go by... you know, kinda like turning up the heat on a lobster in a pot. plus it makes it all the more difficult for d.p. "powers that be" to continue to lie to their volunteers about having no options. they know they're lying, and at some point in the future, the entire sordid mess will be exposed, and those d.p volunteers will realize that their "managers" _intentionally_ lied to them, and karma will do what it always does -- punish the guilty... so about every 6 months, i put forth yet another initiative, aimed at helping d.p. -- if the "powers that be" are simply smart enough to put aside their stupid stubborn "pride" -- or -- when they are not -- showcasing their bad decision... if i have time sometime, i might document the full history of these initiatives, but most recently, i made an attempt in september of last year, to do a "collaboration" that was designed to program an online post-processing system, named "babelfish", except nobody else did any work on it, so the so-called "collaboration" collapsed because of that. (which is pretty much exactly what i figured would happen, but i wanted to demonstrate that _i_ was willing to do it.) then, in spring of this year, i displayed how an online app could be coded that would convert a clean text to an .epub, without using any markup sludge except for my "zen magic". (d.p. has the proofers and formatters put in a bunch'o'crap which the post-processers then have to take out; so dumb.) a year before that -- in the spring of 2010 -- i used perl to code an online text-cleanup system that could've been used in a collaborative fashion, just to prove that it could be done. but, as usual, the "powers that be" over at d.p. just ignored it, all the while claiming there was no one to code such a thing. not that they just ignore me, either. years back, when i was arguing that they could code an offline app for cleaning text -- because i had already programmed such a thing -- one of the d.p. volunteers (dkretz) went to work on his own version of such an app. he did a good job of showing the potential, but the "powers that be" hate him, too, so they ignored that. what was really astounding, though, was when rfrank -- yet another d.p. volunteer -- built a distributed proofing system that worked online. his system challenged a lot of the myths of the d.p. system, and it produced some very good results... but you know the story -- the "powers that be" just ignored it. the thing is, rfrank had been one of the _darlings_ over at d.p. he was (and remains) a very popular member of the community, and was even voted on to the board of directors for the site, but that evidently wasn't enough to get the attention that he needed in order to propagate change within the system as it now exists. so, you know, if rfrank can't even get the system fixed over there, there's no chance that _i_ can. but i come here anyway, and post, and show proof-of-concept, and then come back later and show a working prototype, and then come back even later with a finished product, all because i know those future cyber-archeologists will be digging here... so this is when i pause and say "hi, future cyber-archeologists!" *** here's another piece of an e-mail from the backchannel: > I haven't specifically looked around for such, but > I have a vague idea of starting some sort of site > where folks can submit ebooks for the general public > (haven't decided if it should allow selling of them or not) > and having a way to convert uploaded text > directly to epub sure would be useful. lots of similar sites already exist: > http://www.smashwords.com > http://www.pressbooks.com > http://www.pandamian.com > http://www.leanpub.com > http://www.gluejar.com there are more, too, but that's off the top of my head... > I have my doubts that any system can effectively > produce an error-free html version, you are mistaken. look up "light markup". it's easy. > I have my doubts that any system can effectively > produce an error-free html version, > but hand-editing html isn't really all that hard to do, > so I'm not worried about that one myself. > On the other hand, epub is something loads of folks ask for, > and I don't think it'd be much of a site if it didn't allow such. take a look at what's already out there. if you decide you can add something, do it... :+) -bowerbird

1 0

these grapes are sweet -- lesson #07
by Bowerbird＠aol.com 13 Sep '11

13 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** benjamin said: > I use groovy. cool... ;+) feel free to port any of my python code over to that platform. also, benjamin, since you might be the only "new" guy here, i will be open to any feedback you'd have on a web interface for proofing and editing o.c.r. text alongside its pagescan... you saw one possibility, in the links i gave recently: > http://z-m-l.com/go/mabie/mabiep123.html > http://z-m-l.com/go/myant/myantp123.html > http://z-m-l.com/go/sgfhb/sgfhbp123.html i have also floated others, but i'm interested in any designs that you might've generated and would be willing to share... *** ok, so here we go... since we're using o.c.r. text from archive.org, it'll be good to review some of the basic information about their typical files. 1. you can get to the files for a book from its "details" page: > http://www.archive.org/details/booksculture00mabiuoft click the "all files: http" link you find in the leftmost column. this will take you to an "index" page that lists all of the files. the file with "djvu.txt" at the end of its name holds the o.c.r. this is the file we'll be working on, so take a good look at it. does the text look like o.c.r. did a good job of recognition? in some of its early books, archive.org "lost" the em-dashes, so that output is totally worthless; don't even bother with it! some files are also missing quotemarks or end-line hyphens. archive.org also scanned some files where library patrons had scrawled notes in the margins, or underlined some passages, which -- as you might imagine -- wreaks havoc with the o.c.r. before using any archive.org text, ensure it's not fatally flawed; your time is much better spent seeking a _worthwhile_ copy... (archive.org itself often has multiple copies of a specific book.) if you can't find any better o.c.r., do some other book instead! or, if you have to do _that_ book, just re-do the o.c.r. yourself. life is too short to waste your time correcting a bad o.c.r. file. thank goodness, most of the recent archive.org files are good. 2. we also need to be able to review the actual page-scans, to resolve questions. archive.org hosts the scans, both in a high-resolution form (which is far too bulky to bother with) and a lower-resolution (which is more than good enough)... they're in a .zip file there on the "index" page for the book. because i have a tool that shows me each scan alongside the o.c.r. text for that page, i usually download the actual scans. but in addition to the scans, archive.org offers .djvu and .pdf _output_ formats that collect the scan-set so you can view it, and a one-file format is often better than hundreds of scans. .djvu is pretty good, because it's smaller than the .pdf, plus _sometimes_ the o.c.r. text is included too, so it's searchable. (realistically, though, all we usually need is the pagenumber.) .pdf often comes in either black-and-white or "color", which means a sepia background was inserted (for some reason)... 3. so now let's concentrate on the text in the actual o.c.r. file. after you've downloaded it, there is some basic cleaning to do. there's a space at the end of every line, including "blank" ones, so we must to do a global search-and-replace to delete them. i usually do this first, just to get it out of the way right away... 4. pagebreaks are indicated by 3 blank lines. unfortunately, sometimes a 3-blank-line sequence occurs elsewhere as well. which means that we can't _depend_ on it being a pagebreak. which is quite sad, yes. still, for the most part, it usually will. we'll often need to look at a scan to resolve an o.c.r. glitch, so pagenumbers will be important to us, so pagebreaks are too. i replace these 3-blank-line sequences with a distinctive line. (a line full of tildes, for instance, if there are none in the file.) this way i know that's where the pagebreaks (probably) were. 5. the runheads (at the top of a page) _are_ retained, as are pagenumbers when they got printed at the bottom of a page. this is good news. (d.p. always decapitates the runheads and pagenumbers; thus their scans are devoid of identifying info; just one of several idiotic things at distributed proofreaders.) the runheads and pagenumbers keep us grounded in the file; otherwise it can be too easy to get lost in the vast sea of text. but the runheads and pagenumbers often have garbled o.c.r. (this might be because their fonts often differ from the body.) luckily, there is enough redundancy that we can control this; runheads almost always exhibit heavily-repetitive elements, and pagenumbers have an extremely-predictable sequence. one of the first things that i generally do is fix pagenumbers; it speeds up later changes that require any review of a scan... a few easy reg-ex searches can do wonders on pagenumbers. the best situation is when pagenumbers were on the bottom of the p-book pages, because then they are followed by the pagebreak, which is in turn followed by the runhead, so we have an extremely predictable sequence of lines going on... (and luckily for us, that's the situation we have in this book.) 6. front-matter is almost always recognized badly by o.c.r. i've spent a lot of time analyzing o.c.r. output to determine how it can be corrected automatically. but sometimes you just have to throw in the towel and admit you are defeated. such is the case with front-matter. you must fix it by hand. this is not necessarily a bad thing, since the contents page gives you an overview of the book that's very good to have. for example, for this thread, it informed me that our book had _24_ chapters, so when a search for "chapter" ended at chapter 23, i knew i had a problem that i needed to check. and sure enough, i found it, right there on "chapler xxiv". which reminds me that i always check the chapter-headers, to make sure they were recognized correctly, but also that they're on the same page as listed in the table of contents. (this is one of the redundancy checks that it's good to do.) when i check the chapter-headers, i give 'em proper z.m.l., which means 4 blank lines above them, and 2 below them. i also give them a pseudo-runhead and pagenumber, since those were typically eliminated from chapter-header pages, whereas our process will count on 'em being on every page. also when i do this job, i often fix the drop-caps/small-caps (which are a _staple_ of the chapter-header pages), because o.c.r. always fails, in a spectacular fashion, to get 'em right. 7. "scene-breaks" from the print-book are _not_ recorded as empty lines in the o.c.r. output... so they are essentially lost, we'll need to do a page-by-page look-through to restore 'em before we're finished. italics are also lost, so ditto for them. and the third leg of this final look-through on the full book is to make sure that all the paragraphing was done correctly, with particular focus on those that start at the top of a page. happily, most paragraphs _do_ get separated by blank lines... (but it's also the case that tight leading on the original page -- a typographical tactic often employed on block-quotes -- sometimes results in mid-paragraph intervening blank lines.) again, this 3-pronged check is done at the end of the process, so this information is merely a heads-up at this point in time. 8. i forget what 8 was for. 9. archive.org generally scans _all_ of the pages in a book, including the blank pages at the end, which often have stuff on them, such as the library form showing the date on which the book had to be returned... this extraneous back-matter is scrubbed, of course, but _do_ look at the scans, and enjoy! let your imagination roam, about the people who borrowed that book, and read it, and what they might've thought of it. it can make you quite nostalgic... 10. i'm gonna return for a minute to discuss scan-viewing... as i said above, both the .djvu and .pdf options are very good. and you can always download the .zip file of all of the scans; loading them into a file-viewer makes them quite accessible, by choosing the one you want to view based on its filename. one glaring problem with all of these approaches, however, is that archive.org never adopted a _smart_ policy regarding their filenames, meaning the number you see in the filename usually is _not_ the pagenumber for that scan. instead, it is the sequential number of that scan within the full scan-set... this stupid filenaming-policy means that you have know the "offset" for each pagenumber in order to know its filename. if you're lucky -- and we were, in this book -- the "offset" is an unchanging number through the entire book. in this book, the offset is "6", meaning that if you want to see the scan for page 123, for example, you need to view the scan for "129". likewise with the .djvu and .pdf files. however, on the mac, at least, the "preview.app" system graphics-viewer-program can _insert_and_delete_ pages from a .pdf, meaning that you can delete some of those unnecessary front-matter pages so the pagenumbers of the .pdf will match the real pagenumbers. this makes your life easier. finally, there is one more option. for every book, archive.org offers a streaming viewer-program. for instance, for the mabie book we're using here, the u.r.l. is: > http://www.archive.org/stream/booksculture00mabiuoft#page/123 that "page 123" part of the u.r.l. there actually shows page 123, and _not_ page 117, as you might expect, given the "6" offset... also, the finger-pointer slider at the bottom of the display is based upon the "real" pagenumbers too, so you can use that. so this is a way of getting around the pesky "offset" nonsense. but this app is only available if you have an internet connection, and it downloads each pagescan as needed, so it can be _slow_, especially if/when your internet connection isn't all too speedy. plus every time you view a page again, it has to be downloaded yet again (unless it happens to still be in your browser cache)... and realize that eventually we're going to "step through" each and every page in the whole book, and probably more than once. so i certainly can't unequivocally recommend this option to you. i believe it's better to download the scans once, and work offline. but, for the sake of being complete, i've informed you about it... 11. so much for the overview on the files you'll encounter... so let's review the process we'll go through. we will first do the first-pass cleaning of the o.c.r. then we'll check that the pagenumbers and runheads are correct throughout the text. after that, we'll do the rest of the "obvious" cleaning needed, referring back to the scans whenever it is required to do so... next, we'll do a spellcheck and fix the warts. then, after that, we'll smooth any rough edges, and do our last-minute check. the final step will be to convert into various output formats... *** our second program, a slight mod from the first, is appended. you can run -- and view -- this program here: > http://zenmarkuplanguage.com/grapes102.py > http://zenmarkuplanguage.com/grapes102.txt this second script auto-fixes the floating punctuation and skips over the unnecessary front-matter and back-matter. -bowerbird p.s. here's the python source-code for our second program... #!/usr/bin/python import urllib import re f=urllib.urlopen("http://ia700300.us.archive.org/1/items/booksculture00mabiu oft/booksculture00mabiuoft_djvu.txt") s = f.read() f.close() s = re.sub("\r\n","\n",s) s = re.sub("\r\n","\n",s) s = re.sub("\r","\n",s) s = re.sub("\r","\n",s) s = re.sub(" \n","\n",s) s = re.sub(" \n","\n",s) s="\n"+s s=s+"\n" s = re.sub(" ,",",",s) s = re.sub(" ;",";",s) s = re.sub(" \:",":",s) s = re.sub(" !","!",s) s = re.sub(" \?","?",s) s = re.sub(" \!","!",s) print "Content-type: text/html\n\n" print "<html><head><title>" print "grapes102.py" print "</title></head><body><pre>" pg = s.split("\n") maxpg=len(pg) startat=236 endat=7764 for i in range(startat,endat): if i < 10: print "000"+str(i)+" "+pg[i] elif i < 100: print "00"+str(i)+" "+pg[i] elif i < 1000: print "0"+str(i)+" "+pg[i] elif i > 999: print ""+str(i)+" "+pg[i] print "</pre><hr></body></html>"

1 0

Obituary for Michael Hart
by Greg Newby 11 Sep '11

11 Sep '11

Sorry for the bad news. I am in touch with the family, and arrangements are being made for a funeral in Urbana, Illinois. http://www.gutenberg.org/wiki/Main_Page Michael Stern Hart was born in Tacoma, Washington on March 8, 1947. He died on September 6, 2011 in his home in Urbana, Illinois, at the age of 64. His is survived by his mother, Alice, and brother, Bennett. Michael was an Eagle Scout (Urbana Troop 6 and Explorer Post 12), and served in the Army in Korea during the Vietnam era. Hart was best known for his 1971 invention of electronic books, or eBooks. He founded Project Gutenberg, which is recognized as one of the earliest and longest-lasting online literary projects. He often told this story of how he had the idea for eBooks. He had been granted access to significant computing power at the University of Illinois at Urbana-Champaign. On July 4 1971, after being inspired by a free printed copy of the U.S. Declaration of Independence, he decided to type the text into a computer, and to transmit it to other users on the computer network. From this beginning, the digitization and distribution of literature was to be Hart's life's work, spanning over 40 years. Hart was an ardent technologist and futurist. A lifetime tinkerer, he acquired hands-on expertise with the technologies of the day: radio, hi-fi stereo, video equipment, and of course computers. He constantly looked into the future, to anticipate technological advances. One of his favorite speculations was that someday, everyone would be able to have their own copy of the Project Gutenberg collection or whatever subset desired. This vision came true, thanks to the advent of large inexpensive computer disk drives, and to the ubiquity of portable mobile devices, such as cell phones. Hart also predicted the enhancement of automatic translation, which would provide all of the world's literature in over a hundred languages. While this goal has not yet been reached, by the time of his death Project Gutenberg hosted eBooks in 60 different languages, and was frequently highlighted as one of the best Internet-based resources. A lifetime intellectual, Hart was inspired by his parents, both professors at the University of Illinois, to seek truth and to question authority. One of his favorite recent quotes, credited to George Bernard Shaw, is characteristic of his approach to life: "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people." Michael prided himself on being unreasonable, and only in the later years of life did he mellow sufficiently to occasionally refrain from debate. Yet, his passion for life, and all the things in it, never abated. Frugal to a fault, Michael glided through life with many possessions and friends, but very few expenses. He used home remedies rather than seeing doctors. He fixed his own house and car. He built many computers, stereos, and other gear, often from discarded components. Michael S. Hart left a major mark on the world. The invention of eBooks was not simply a technological innovation or precursor to the modern information environment. A more correct understanding is that eBooks are an efficient and effective way of unlimited free distribution of literature. Access to eBooks can thus provide opportunity for increased literacy. Literacy, the ideas contained in literature, creates opportunity. In July 2011, Michael wrote these words, which summarize his goals and his lasting legacy: âOne thing about eBooks that most people haven't thought much is that eBooks are the very first thing that we're all able to have as much as we want other than air. Think about that for a moment and you realize we are in the right job." He had this advice for those seeking to make literature available to all people, especially children: "Learning is its own reward. Nothing I can say is better than that." Michael is remembered as a dear friend, who sacrificed personal luxury to fight for literacy, and for preservation of public domain rights and resources, towards the greater good. This obituary is granted to the public domain by its author, Dr. Gregory B. Newby.

8 10

these grapes are sweet -- lesson #06
by Bowerbird＠aol.com 10 Sep '11

10 Sep '11

this thread explains how to quickly and easily digitize a book. i did the book "books and culture", by mabie, which you can find by searching archive.org for "booksculture00mabiuoft". and i will share my secrets about how to do the job _fast_... *** i've done this lotsa times in the past, but the difference now is that i pledge to do most of the changes with just a text-editor, whereas usually i custom-program tools to help myself out... this time, if i do need a custom tool, i pledge that i will code it and furnish you the code so that you can run the tool yourself. don't fret if you're not a programmer, or not a very good one. i'm not a very good one either, but i can get the job done, and remember -- i'll be coding the tool, and handing you the code. moreover, to force myself to keep things as simple as possible, i will code any tools in _python_, which i've never used before... so since i'm learning it from scratch, i can teach it from scratch, and answer any just-a-beginner questions that you might have. *** so, to get our toes wet, i've appended our first program... it grabs the mabie text that we're using from archive.org and lists it inside your browser, with handy-dandy line-numbers... you can run this program here: > http://zenmarkuplanguage.com/grapes101.py you an also get the source-code at a similar address: > http://zenmarkuplanguage.com/grapes101.txt (i'll follow this naming convention throughout this thread.) as you can see when you run this program, the o.c.r. had quite a severe problem with floating semicolons. there are _444_ of them. but who's counting? ;+) -bowerbird #!/usr/bin/python import urllib import re f = urllib.urlopen("http://ia700300.us.archive.org /1/items/booksculture00mabiuoft/booksculture00mabiuoft_djvu.txt") s = f.read() f.close() s = re.sub("\r\n","\n",s) s = re.sub("\r\n","\n",s) s = re.sub("\r","\n",s) s = re.sub("\r","\n",s) s = re.sub(" \n","\n",s) s = re.sub(" \n","\n",s) s="\n"+s s=s+"\n" print "Content-type: text/html\n\n" print "<html><head><title>" print "grapes101.py" print "</title></head><body><pre>" pg = s.split("\n") maxpg=len(pg) startat=1 # will be 236 endat=maxpg # will be 7764 for i in range(startat,endat): if i < 10: print "000"+str(i)+" "+pg[i] elif i < 100: print "00"+str(i)+" "+pg[i] elif i < 1000: print "0"+str(i)+" "+pg[i] elif i > 999: print ""+str(i)+" "+pg[i] print "</pre><hr></pre></body></html>"

2 1

Re: Obituary for Michael Hart
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

jon richfield has some points, but since he is a person who "contributes quite actively" to wikipedia, he might not be able to see the border of the circle he is inside... :+) so it might not be as constructive as he makes it sound. (from "the office" t.v. show: "there is no inner-circle; i can say that for sure because i'm in the inner-circle. ...which does not exist.") you might remember that i explicitly asked if there were any wikipedia insiders here. because that's what it takes if you want to edit a page and you want the edit to stick. if jon is such a person, and he is offering to do an edit, then by all means, we should take him up on his offer... *** jon said: > Edit war is a 4-letter matter in those circles well, let's go take a look at how wikipedia defines it, ok? > An edit war or revert war is a situation that sometimes > arises on websites which are run on wiki principles, > such as Wikipedia, where users repeatedly > re-edit or undo or reverse the prior user's edits that doesn't really sound like "a 4-letter matter" to me. of course, some edit wars escalated to heights that are truly ridiculous (far worse than a mere "4-letter matter"), such as the ones on hot-button issues like abortion, but i certainly didn't imply that such heat would happen here. at the time of my first post, we just had one single edit... at the time of my second post, we had only the one revert. still, if theo10011 had reverted the revert _and_ it would have prompted _another_ counter, that'd be "an edit war", as defined by wikipedia itself, in the passage i just quoted. but... i have since gone back and looked at the edit-histories of the two parties, so i now suspect no "edit war" will occur... that's because one person has fewer edits than the other. and the one with more edits (50+ since just september 5) is also an employee of the wikimedia foundation, meaning that if the other editor doesn't want to get into a fight that he can't win, he'd better just smile and accept that revert... because yes, that _is_ the kind of political power-struggle which wikipedia editing has devolved into, over the years... luckily for us, and michael, the editor with more power is the one who calls michael "the inventor of the e-book", and not the one who called him "an inventor of e-book". but even if it was the other way around, and the page ultimately settles on "an inventor of e-books", so what? who cares? if anyone bothered to ask who the "other" inventors of e-books were, what would the answer be? vannevar bush? so, how many books did he put online? jeff bezos? oh really?, so what year did he "invent" 'em? won't be no "war" because there's nothing to fight over. but -- for the record -- the person with fewer edits has 50 edits since june 4th, so s/he isn't an outsider either. if you haven't made any edits in the last year, or _ever_, you darn well better not go try and edit michael's page. > Perfect? Not a hope. Important? Hugely. > As important as PG et al? In the long run certainly; > at present? Dunno. You tell me. wikipedia is not perfect; by far. but it is hugely important. indeed, it's probably even more important than "hugely"... (although then we'd just get into "an adjective war".) ;+) is wikipedia as important as project gutenberg? certainly. it's far more important, already, and the gap grows daily... wikipedia grows in stature more and more every moment. project gutenberg's importance has largely been eclipsed, by both google and the internet archive scanning efforts. michael's contribution is that project gutenberg was the _impetus_ that convinced the world e-books were viable, thus becoming the _spur_ for those big scanning projects. kickstarting that contribution was an important sub-plot, namely that -- if the powers-that-be refused to take on the task -- _we'd_do_the_job_ourselves,_as_individuals_. on the one hand, it's silly to call michael "the" inventor. thousands of people -- including me -- had the idea, and we got it all by ourselves, not from michael hart... heck, thousands of people _heard_ the idea _from_me_, originally, because i was busy telling everyone i knew... the thing is, i considered this to be a _big_ task, one for the librarian of congress, or an entity at a level like that. i'm guessing that michael would have been _delighted_ if the librarian of congress would've taken on the task, but what made michael different -- the thing that finally made me willing to grant him the honorific exclusively -- is that michael was willing to do the job _all_by_himself_, if he had to, a mentality going all the way back to 1971. michael grabbed the power himself, and refused to allow any of the big entities to tell him that they wouldn't do it. which is why, on the other hand, it is _not_ silly to give him the distinction and the title of being "the" inventor. michael refused to let the universe say "no" to his dream. he broke it down to the "one-book-at-a-time" level, and that made it possible for him to do it, and then he did it. and, by doing it himself, he became the model for others to do the job too -- one-book-at-a-time -- and after a period of many years, the effort cumulated to a _library_. it became an accomplishment that was worth celebrating. thus manifests the third wrinkle in michael's contribution: the fact that his individual effort attracted other people, such that the combination of their individual efforts soon cumulated to the clear reality of a dedicated community... and the emergence of distributed proofreaders is merely icing on the cake, in the sense that "one-book-at-a-time" naturally extends itself to the "one-page-at-a-time" idea. (this doesn't discount at all the brilliance of charlz franks in bringing about the _tech_ for distributed proofreaders; it merely acknowledges the genesis of the idea behind it.) i've just described them in their reverse chronology, but it's important that we understand the causal sequence... 1. michael says "i'll do this job all by myself, if i have to." 2. other people say "i'll help too", forming a community. 3. the resultant library proves e-book viability to google. (in case it rings a bell, yes, this "we'll do it ourselves" is precisely why napster was so important to digital music; it said "you can't pretend this capability does not exist"; of course, it's much easier to rip a c.d. than scan a book.) jimbo did the exact same thing with wikipedia, of course, except he had project gutenberg as a "proof-of-concept". still, the idea that individual efforts can cumulate into a synergistic whole creating an unbelievably large product that is extremely useful to society at large is _important_. of course, the end-note on both stories is a touch sour... project gutenberg has now become almost totally reliant on distributed proofreaders -- it lost its own community. and, of course, its library has been completely dwarfed... and distributed proofreaders is trapped in a distant past; it's now being controlled by a pack of technocrats which is entirely incapable of changing its obsolete workflow, so it has stagnated at a book-digitization of 2500/year; it'd take d.p. 1000 years to do google's 25 million books. the scenario at wikipedia is a little different, in that it has been taken over by the politics of trivial power struggles, rather than technocrats, but the end-result is the same: the vibrant community of volunteers which it once was has degenerated into a relatively small group of people. perhaps future visionaries will sidestep these problems, and be able to prevent the capture of their communities. we'll have to see. on the other hand, facebook might instead be showing how a clever puppeteer can manipulate a community for the purpose of renting them out to the highest bidders. -bowerbird

2 1

Re: Obituary for Michael Hart
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

I said: > now _perhaps_ kaldari will come back and > revert the change that theo10011 made, so > michael will again be called "the" inventor... and sure enough, while i was in the very process of writing my last post, that's exactly what kaldari did. now maybe theo10011 will concede, or perhaps it will become what wikipedia calls "an edit war", but the best advice to outsiders remains to _stay_out_. -bowerbird

2 1