- gutvol-d - lists.pglaf.org

a review of some digitization tools -- 005
by Bowerbird＠aol.com 28 Nov '11

28 Nov '11

ok, today we'll look at another book that was digitized by jim adcock. thankfully, this one has more meat on its bones than the previous two. this one is pg#31103, a.k.a., "new juveniles for 1864", by james miller. as before, i've appended the tags that jim used. we have new ones! first, there's the standard [p] tags for paragraphs, and [br] for breaks. we also have the usual [i] tags jim uses for italics. strangely, though, we also have a few [em] emphasis tags as well, also meaning italics... in addition, we have a [strong] tag, to indicate text rendered in bold. both the [em] and the [strong] tags have a _class_ attribute as well, and later on we see there are some [div] tags used too, which makes me very suspicious that this file was generated by using _calibre_... calibre is a tool that does conversions among many e-book formats, shuttling them all into its own intermediate format which uses these "class" attributes extensively -- usually prefixing them with "calibre". sure enough, jim has (atypically) used a .css stylesheet on this book, and the stylesheet looks just like the one that is created by calibre... so i suspect that jim just did a global change of "calibre" into "cal"... i was going to review calibre later, but i suppose we might as well get it out of the way right now, so i'll give a quick-and-dirty on it. calibre works fine for people who don't want to "look at the source", and who'll likely never have any need to ever _change_ that tagging. but if you have any need, or even a desire, to _understand_ the file, calibre is not a good solution, because its tagging is _very_ obtuse. (a good class-name will explain its purpose; "calibre-14" does not.) i don't know if calibre's tagging is any good, as it's so obtuse that i cannot even force myself to look at it long enough to evaluate it. i think it's probably _not_ very good, because obtuse stuff rarely is, but in most cases, on most machines, its output is "good enough". it does the job, and it does it fairly well, and that's "good enough". because of that, i think calibre might be _good_ for readers who want to convert from one format to another, for their private use. but it's _not_ the kind of product we could build a system around. *** meanwhile, let's look at the additional tags required by this book. first, we have [blockquote] tags. blockquotes are a structure that is used often in e-books, so we'll need to add it to our repertoire. the blockquote tag has some similarity to the [pre] tag we've used, as [pre] preformatted material is often indented on both sides, just like we'll want our blockquotes to be indented, but blockquotes -- unlike poetry, for instance -- can often be rewrapped if necessary. jim also used some [hr] horizontal rules in this book. fair enough. we also see various structures -- [div] and [p] and [span] -- that're used with the "class" attributes, to perform various types of styling. this gets perilously close to "presentational" markup, which is often contrasted with "semantic" markup, but i won't get in that cesspool. we also see that a half-dozen images were scattered in this book, so we will need [img] image markup to indicate their presentation. finally, as with the earlier books, we have the expected _headers_. jim uses header-levels 1 through 5, mostly for the forward-matter, probably only to set the various elements to an "appropriate" size. there's nothing "semantic" about the many levels, except [h1] title, so again, this tagging is "presentational" and not "structural", but... while i'm at it, for the record, i'll note that some so-called "purists" might make an objection about the fact the table-of-contents lines are tagged as "paragraphs", saying they aren't _really_ "paragraphs". they'll call this "tag abuse", and probably get all indignant about it... i don't think we need to even think twice about such minor matters, let alone pay attention if people get all huffed-up about them, but i thought i'd mention it, so you know the crap that we're up against. and that about sums up today's book... *** so, once we ignore the unnecessary complexity introduced by calibre, we see that this book has introduced three new structures to handle: a. blockquotes b. horizontal rules c. images thanks, once again, to jim adcock for digitizing this book, and indeed, thanks to _all_ the people who work hard to digitize all the p.g. books. -bowerbird p.s. here are the .html tags that jim used to digitize pg#31103... > [p][/p] > [br] > [i][/i] > > [em class="cal3"][/em] > [strong class="cal8"][/strong] > > [blockquote class="sgc-12"][/blockquote] > > [hr width="25%"] > [hr width="50%"] > > [div][/div] > [div class="sgc-15"][/div] > [div][span class="sgc-11"][/span][/div] > > [p class="sgc-2 sgc-4"][/p] > [p class="sgc-4"][/p] > [p class="sgc-18"][/p] > > [span class="sgc-8"][/span] > [span class="sgc-11"][/span] > [span class="sgc-13"][/span] > > [img src="31103-h_files/img0002.png" alt="The Dream of Little Tuk."] > [img src="31103-h_files/img0003.png" alt="Children Dancing."] > [img src="31103-h_files/img0004.png" alt="Man Carrying Firewood."] > [img src="31103-h_files/img0005.png" alt="Mother Praying with Angel Overhead."] > [img src="31103-h_files/img0006.png" alt="The Little Match Girl."] > [img src="31103-h_files/img0007.png" alt="Mother Holding Mistletoe Above Infant."] > > [h1 align="center"]New Juveniles for 1864[/h1] > [h2 align="center"]JAMES MILLER,[/h2] > [h3 align="center"]MAGNET STORIES,[/h3] > [h4 align="center"]For Summer Days and Winter Nights.[/h4] > [h5 align="center"]SECOND SERIES.[/h5] > > [p align="center"][a name="ch_TheOldHouse" href="#a_TheOldHouse"]I. The Old House[/a][/p] > [p align="center"][a name="ch_TheDropOfWater" href="#a_TheDropOfWater"]II. The Drop of Water[/a][/p] > [p align="center"][a name="ch_TheHappyFamily" href="#a_TheHappyFamily"]III. The Happy Family[/a][/p] > [p align="center"][a name="ch_TheStoryOfAMother" href="#a_TheStoryOfAMother"]IV. The Story of a Mother[/a][/p] > [p align="center"][a name="ch_TheFalseCollar" href="#a_TheFalseCollar"]V. The False Collar[/a][/p] > [p align="center"][a name="ch_TheShadow" href="#a_TheShadow"]VI. The Shadow[/a][/p] > [p align="center"][a name="ch_TheOldStreetLamp" href="#a_TheOldStreetLamp"]VII. The Old Street-Lamp[/a][/p] > [p align="center"][a name="ch_TheDreamOfLittleTuk" href="#a_TheDreamOfLittleTuk"]VIII. The Dream of Little Tuk[/a][/p] > [p align="center"][a name="ch_TheNaughtyBoy" href="#a_TheNaughtyBoy"]IX. The Naughty Boy[/a][/p] > [p align="center"][a name="ch_TheTwoNeighboringFamilies" href="#a_TheTwoNeighboringFamilies"]X. The Two Neighboring Families[/a][/p] > [p align="center"][a name="ch_TheDarningNeedle" href="#a_TheDarningNeedle"]XI. The Darning Needle[/a][/p] > [p align="center"][a name="ch_TheLittleMatchGirl" href="#a_TheLittleMatchGirl"]XII. The Little Match-Girl[/a][/p] > [p align="center"][a name="ch_TheRedShoes" href="#a_TheRedShoes"]XIII. The Red Shoes[/a][/p] > [p align="center"][a name="ch_ToTheYoungReaders" href="#a_ToTheYoungReaders"]XIV. To The Young Readers[/a][/p] > > [a name="a_TheOldHouse"][/a][h2][span class="sgc-11"]THE OLD HOUSE.[/span][/h2] > [a name="a_TheDropOfWater"][/a][h2][span class="sgc-11"]THE DROP OF WATER.[/span][/h2] > [a name="a_TheHappyFamily"][/a][h2][span class="sgc-11"]THE HAPPY FAMILY.[/span][/h2] > [a name="a_TheStoryOfAMother"][/a][h2][span class="sgc-11"]THE STORY OF A MOTHER[/span][/h2] > [a name="a_TheFalseCollar"][/a][h2][span class="sgc-11"]THE FALSE COLLAR.[/span][/h2] > [a name="a_TheShadow"][/a][h2][span class="sgc-11"]THE SHADOW.[/span][/h2] > [a name="a_TheOldStreetLamp"][/a][h2][span class="sgc-11"]THE OLD STREET-LAMP.[/span][/h2] > [a name="a_TheDreamOfLittleTuk"][/a][h2][span class="sgc-11"]THE DREAM OF LITTLE TUK.[/span][/h2] > [a name="a_TheNaughtyBoy"][/a][h2][span class="sgc-11"]THE NAUGHTY BOY.[/span][/h2] > [a name="a_TheTwoNeighboringFamilies"][/a][h2][span class="sgc-11"]THE TWO NEIGHBORING FAMILIES.[/span][/h2] > [a name="a_TheDarningNeedle"][/a][h2][span class="sgc-11"]THE DARNING-NEEDLE.[/span][/h2] > [a name="a_TheLittleMatchGirl"][/a][h2][span class="sgc-11"]THE LITTLE MATCH GIRL.[/span][/h2] > [a name="a_TheRedShoes"][/a][h2][span class="sgc-11"]THE RED SHOES.[/span][/h2] > [a name="a_ToTheYoungReaders"][/a][h2][span class="sgc-11"]TO THE YOUNG READERS[/span][/h2]

2 1

what to do about those abbyy output files
by Bowerbird＠aol.com 21 Nov '11

21 Nov '11

alex said: > The script uses only the gzipped > abbyy output to generate the html > (ie, what's available here alex, i'll let you sort out lee's questions with him. good luck with that... but... from the standpoint of what can be "generated" from the abbyy files, you _could_ make an .rtf... and when i say that "you" could do it, what i mean is that you could simply instruct abbyy to output it. no coding, no nothing. just tell abby to output .rtf. and that .rtf file would be enough to _build_ from... i have detailed the problems with your .txt files: 1. pagebreaks have been, for the most part, lost. 2. styling is lost (like italics, bold, and fontsize). 3. in some of your books, em-dashes were lost. it's worth noting that _all_ of these problems were introduced by the archive.org workflow. _none_ of 'em are endemic to abbyy, or its output. indeed, the abbyy file you've just referenced above -- like all abbyy output -- contains _all_ this stuff. i can write a program (already wrote it, long ago) which pulls all of the relevant data out of that file. the problem is that "the relevant data" is buried in ~60megs of x.m.l. crud. it only takes a few minutes to pull out "the gold" from this one file, but when we multiply those "few minutes" by two _million_ books, the task becomes undoable. that's the problem, alex. the file that contains the data also contains _crud_. and when you output the data, you eliminated the crud, yes, but you also eliminated important data. that is, you threw the baby out with the bath water. also, generating .html from the abbyy output is _not_ a good solution. because that .html is _not_ going to be "semantically accurate". for the obvious example, the headers are _not_ gonna be _tagged_ as headers. they're just going to be marked as 24-point bold text. so you're _still_ gonna have to do some analysis of it. don't get sucked into a wild goose chase... -bowerbird

1 0

Re: [gutvol-d] a review of some digitization tools -- 002
by Bowerbird＠aol.com 21 Nov '11

21 Nov '11

keith said: > The smiley was directed to you rants towards > Lee and Don! Just a small poke in the side. yeah, i got that. except what i said wasn't a "rant"... it's a straightforward fair evaluation of the situation. open-office is an admitted effort to clone ms-word, with most liabilities of ms-word, and very little else, and thus there is almost no reason to recommend it. and as for ms-word itself, even its biggest fans today consider the latest versions to be grotesque monsters, bloated by featureitis and burdened by complexity, its everyday functionalities constantly being hidden under some new and totally confusing interface-of-the-day. the _only_ reason it continues to have any presence is because workplace administrators are afraid of change, so keep using what they've used for the last 2 decades. and then because people are forced to use ms-word at work, they know it, and "choose" to use it at home too. but yes, absolutely, if ms-word is your wordprocessor, your crescent-wrench, then i'll recommend you use it for your own personal workflow, with no reservations. you might as well be comfortable, use what you know. heck, i still talk to people who use wordperfect. fine! but if you suggest that _other_ people use ms-word, or that it be made integral to a _digitization_system_, i will ridicule your suggestion, because it deserves it. -bowerbird

3 2

Re: [gutvol-d] A novel suggestion!
by Bowerbird＠aol.com 21 Nov '11

21 Nov '11

keith- that "suggestion" isn't "novel" -- it's downright bizarre! but hey, since "tools for doing the initial proofing" are _simple_ to make -- as i've demonstrated many times -- how about _you_ make some, and then get together with those other three individuals to make your neato system? -bowerbird p.s. i'm tempted to give your post one of my $2 prizes for "a positive contribution" by being such a stirring example of being bizarre, but i'd be afraid of setting a precedent...

1 0

Re: [gutvol-d] a review of some digitization tools -- 003
by Bowerbird＠aol.com 21 Nov '11

21 Nov '11

ok, you'll remember our three overarching requirements: 1. obtain the text (via scanners, and o.c.r. software) 2. clean the text (the first set of tasks for our tools) 3. turn the text into e-books (the second set of tasks) under the second of those requirements, we covered: 2a. do a spellcheck 2b. fix spacey punctuation 2c. restore styling, e.g., italics now we'll work on the list of tasks for the third one: 3a. tagging the structural aspects of the text 3b. converting the text into .html (intermediate and final) 3c. converting intermediate .html into e-book output-files our biggest task here is 3a, tagging the structural aspects... so let's talk about what it takes to "tag" the elements of a book, where the tags describe the _structure_ of the specific element. the first thing you need to know is that -- for many books -- the number of types of tags needed is embarrassingly small... indeed, here are the big 5... 1. paragraphs you need a paragraph tag -- [p] -- at the beginning of paragraphs. (i'll use square brackets in this post, rather than angle-brackets, just so nobody gets confused, as i'm speaking in general terms about marking the elements, and not about .html coding per se.) depending on the flavor of the markup, you might need to have _closing_ tags, which might be a [/p] at the end of a paragraph. similar in function is the [br] break command, to force a linebreak. 2. blocks which should not be rewrapped some blocks -- such as poetry -- should not be rewrapped, so you need to mark them, so they'll retain their formatting. a quick-and-dirty way is a [pre] tag before, and [/pre] after. 3. headers you need to mark headers, usually indicating the _level_ as well. so [h1] might start a level-one header, with [/h1] ending it, and [h2] and [/h2] surrounding a level-two header, and continuing... 4. table-of-contents links the table of contents in an e-book should link to the proper spots. in .html, these links consist of an [a] anchor with a [href] argument. 5. italics words/phrases which are to be italicized must be marked as such. *** and really, that's all you need for a _surprising_ number of books! i know, just 5 easy things, but believe me, that's often _sufficient_! *** for instance, since nobody has suggested an example book to use, let us examine "the rainbow", by d.h. lawrence, which was digitized for project gutenberg by jim adcock; this book is known as #28948. > http://www.gutenberg.org/files/28948 first of all, jim, thanks for digitizing this important book for p.g. i've taken a cursory look at it, and found a few errors for you to fix. in the .html version, chapter 9 is unfortunately called "chapter ixix". and chapter 11 is called "chapter xii"; that's true in the .txt files too. no need for thanks, jim, i'm happy to help, and won't see it anyway. oh yeah, jim, by the way, your .txt files are badly in error, because you forgot to indicate all of the italicized words with underscores! that's over 160 errors throughout the book, just on those italics... luckily, the italics _are_ indicated in your .html file, so you can just use that file to find where to make corrections to your .txt versions. because i'm sure you don't want to do a disservice to d.h. lawrence. anyway, back to the topic of the thread... i've appended a list of the .html tags that jim used. as you can see, the entire list consists of only the 5 types of tags i mentioned above. of course, we'll want to be able to handle more than just the simple basic structures of a straightforward book with any system we build, but it's nice to know you can _start_ with something simple and still be able to digitize an important book like this one by d.h. lawrence. -bowerbird > [p][/p] > [pre][/pre] > [i][/i] > [br] > > [h1 align="center"][/h1] > [h2 align="center"][/h2] > [h3 align="center"][/h3] > [h4 align="center"][/h4] > [h5 align="center"][/h5] > > [a name="ChI" href="#CI"][/a] > [a name="ChII" href="#CII"][/a] > [a name="ChIII" href="#CIII"][/a] > [a name="ChIV" href="#CIV"][/a] > [a name="ChV" href="#CV"][/a] > [a name="ChVI" href="#CVI"][/a] > [a name="ChVII" href="#CVII"][/a] > [a name="ChVIII" href="#CVIII"][/a] > [a name="ChIX" href="#CIX"][/a] > [a name="ChX" href="#CX"][/a] > [a name="ChXI" href="#CXI"][/a] > [a name="ChXII" href="#CXII"][/a] > [a name="ChXIII" href="#CXIII"][/a] > [a name="ChXIV" href="#CXIV"][/a] > [a name="ChXV" href="#CXV"][/a] > [a name="ChXVI" href="#CXVI"][/a] > > [h3 align="center"][a name="CI"][/a][/h3] > [h3 align="center"][a name="CII"][/a][/h3] > [h3 align="center"][a name="CIII"][/a][/h3] > [h3 align="center"][a name="CIV"][/a][/h3] > [h3 align="center"][a name="CV"][/a][/h3] > [h3 align="center"][a name="CVI"][/a][/h3] > [h3 align="center"][a name="CVII"][/a][/h3] > [h3 align="center"][a name="CVIII"][/a][/h3] > [h3 align="center"][a name="CIX"][/a][/h3] > [h3 align="center"][a name="CX"][/a][/h3] > [h3 align="center"][a name="CXI"][/a][/h3] > [h3 align="center"][a name="CXII"][/a][/h3] > [h3 align="center"][a name="CXIII"][/a][/h3] > [h3 align="center"][a name="CXIV"][/a][/h3] > [h3 align="center"][a name="CXV"][/a][/h3] > [h3 align="center"][a name="CXVI"][/a][/h3]

2 1

looking at the most meaningful indicator
by Bowerbird＠aol.com 21 Nov '11

21 Nov '11

keith said: > It is true that publishers have to > change towards the digital to survive. > Yet, the complete demise of the printed book > is still far off. people conflate "publishers" with "the printed book". what a huge mistake. is there a printer attached to your computer? if there is, and you have enough paper on hand, you can print a book. you could print a few of them, every day, each and every day, for a year or two, enough printed books to overflow your house. and speaking of your house, is there a copy-shop down the way? if there is, it could print enough books in a week or so to overflow every house on your block, and then the next block over, and so on. in the future, every book written has a chance to be "a printed book", not just those which publishers think they can sell for profit, meaning we will have _far_more_ "printed books" than we've _ever_ had before... -bowerbird

4 7

a review of some digitization tools -- 002
by Bowerbird＠aol.com 20 Nov '11

20 Nov '11

keith- most of the points you made in your posts this morning are ones which i feel do not need a response immediately, if at all, so if i get to them this weekend, i'll write a reply... and if not... not... the one exception is this... i said: > > so the first tool you'll use for digitization is > > your old friend, your trusty wordprocessor... keith said: > Are you really, advising people to > use bloat ware, MS and OO. > ;-)))) first, yes, i see the smiley, and i respond under that light... this series consists of reviews of e-book digitization tools. your wordprocessor is the first tool in your toolbox, so yes, indeed, i am really advising that people consider it as such. the wordprocessor serves a number of different functions. it's like a crescent-wrench -- it can fit a nut of any size... for that reason, the crescent-wrench is extremely valuable, undoubtedly the single most-important wrench in the box. > http://en.wikipedia.org/wiki/Adjustable_spanner nonetheless, most mechanics almost always use a wrench that's more specialized for the exact task at hand, such as a socket-wrench with a head that's the same size as the nut. > http://en.wikipedia.org/wiki/Wrench likewise, even though we _could_ be using a wordprocessor -- my "sweet grapes" series in october took that approach -- this series looks at tools _specialized_ for the tasks at hand. that's why much of the focus here is on digitization _tasks_. if you wanna review _tools_, you need to have a good handle on what you want those tools _to_do_ -- what to accomplish. the _overarching_ requirements here are simple to list: 1. obtain the text (via scanners, and o.c.r. software) 2. clean the text (the first set of tasks for our tools) 3. turn the text into e-books (the second set of tasks) i dispensed with the first requirement rather quickly, since that's not a job most of us need to do these days, thanks to the big scanning projects from google and internet archive. the second requirement is also fairly straightforward, in that you _could_ do the job by using your favorite wordprocessor. more to the point, though, was to list the tasks there: 2a. do a spellcheck 2b. fix spacey punctuation 2c. restore styling, e.g., italics once you're aware of the exact tasks, you can evaluate tools. for instance, the proofing rounds at distributed proofreaders exist to complete tasks 2a and 2b. and the formatting rounds have the job of doing 2c, and _some_ tasks in requirement 3. it is this backdrop of knowing the tasks that need to be done which allows me to rail loudly about the inefficiencies of d.p. they're wasting far too many human resources to accomplish these relatively-simple tasks; they could be doing much more. anyway... once you know tasks, you can evaluate the tools. in that regard, let's list the tasks for the third requirement. 3a. tagging the structural aspects of the text 3b. converting the text into .html (intermediate and final) 3c. converting intermediate .html into e-book output-files i'm gonna cut directly to the chase, and tell you that there are scripts to accomplish the 3b and 3c conversion steps, and thus the main focus from here on will be _tagging_the_structure_... at distributed proofreaders, which is important to examine since it _is_ an existing system of the type we want to make -- because the way to test this stuff is in _the_real_world_, and _not_ in the realm of some "theoretical discussions" -- some of 3a is done during formatting rounds, with the rest accomplished by offline, non-distributed "post-processing". a sizable minority of post-processors, including _lucy24_, do their work in a wordprocessor. lucy uses "subethaedit", a collaborative text-editor that truly is lean and powerful. > http://www.codingmonkeys.de/subethaedit/ but most d.p. post-processors use an app called "guiguts". guiguts was stagnant for a long time, but hunter monroe has recently taken on the burden of updating it again, so i'll give him a chance to step in now and tell you about it... hunter, are you here? -bowerbird

3 2

now keith wants me to jump in his tar-pit
by Bowerbird＠aol.com 20 Nov '11

20 Nov '11

keith, you seem to be having "theoretical" discussions on 3 or 4 vague phantom topics, while i'm talking about stuff which either actually _exists_ or has been solidly proposed. and there's no sense continuing when we're so badly out of sync. so let us summarize and finish. 1. i have systems, online and off, which are simple and work nicely. 2. keith is too busy to code stuff, but seems to have many opinions. 3. jim likes using bloatware, but is too cheap to buy commercial-grade. 4. lee wants someone else to do all his .html markup for him first. 5. don knows databases, but hasn't generalized from his d.p. experience the conclusion that any system which requires administrators will become corrupt because of the power-seekers. 6. tar-pits are sticky places. 7. sometimes listserves can be too. 8. i forget what 8 was 4. 9. number 9, number 9... 10. you need 10 items for a top-10. 11. is my favorite number. -bowerbird

3 3

ok let's take stock
by Bowerbird＠aol.com 20 Nov '11

20 Nov '11

given all of the recent topic-shattering, it's only natural to feel a little scattered. so let's take stock, and regroup. first of all, it's _cool_ that people feel free to share our individual e-book workflows. other individuals here might learn from it. and more importantly, we might come to learn something that could be of use to us in building a system for collaborative work. it seems to me that a collaborative system is more in tune with aims of this listserve... and if your workflow depends in a vital way on your own individual experience or talent, it's not gonna be of use to that, not really... likewise, if it depends on a software program (such as ms-word or open-office) which has grown to be a gnarly complex monster, then odds are people aren't gonna want to use it. fine for your use, sure, just not for our use... also, hand-in-glove here, if the app requires a good deal of hand-holding, that's not good. what we want, then, is simple software that takes care of a simple workflow, and which can run either offline or online, invoking a system for collaborative e-book digitization. in this vein, we find distributed proofreaders. they have such a system, and such software... and, for the most part, although they will say that they are not completely satisfied with it and have lots of features they'd like to add, they often turn down coders who volunteer... (when those coders will not be subservient.) so my impression is d.p. is happy with itself. and the same can be said of the p.g. machine. i don't get a feeling they want any help either. they don't want you to meddle with their stuff. in a way, that's pretty liberating, since it means we don't have to support any lingering baggage. we are free to design a system any way we like... i've built the engine of such a system. if people would like me to put it up again, for evaluation, i could do that. (it was the old "sitka" project.) roger frank also coded the engine for a system. he has also taken his project down, but i bet if enough people expressed interest in seeing it, he'd also remount it, so you could evaluate it... *** all these systems are pretty "standard", in that the interface has a text-field and shows a scan. if a review of the scan reveals the text is wrong, you edit the text. it's all very straightforward... other interfaces have been suggested as well. i made one where each text-line is displayed next to a checkbox that should be checked if the line contains any errors, so that that line will later be corrected by the administrators. i made another one where questionable lines get shown in various forms, and the person is asked to indicate the one which is correct. (that's for when multiple files get compared.) we also saw an interface from edward betts on this list recently, where lines are shown under the relevant slice from the scan, and the person clicks any incorrect word to fix it. so the interface can vary as well as the tool... *** it's easy to code an e-book digitization system. since the d.p. code has grown into a monster, people familiar with that code might laugh at such an assertion. but i make it with certainty: it's easy to code an e-book digitization system. i've already done my autumn project this year, so my plan was to wait until spring to do this, but if people want to go to work on it _now_, just let me know, and we can start a project to collaboratively code such a digitization system as an open-source project. since, as i just said, i've already done this, i could serve as a guide. but i'm not going to just hand over my code, so if you want it to be open-source, you'll have to do the vast majority of the coding of it yourself. and that's good, because it means that you can make the choice of the language by yourselves. let me know. if nobody's interested in tackling such a project, i'll do another how-to series, on digitizing tools. in fact, i might do that either way. yes, yes i will. so, by all means, if you want to share aspects of your personal e-book digitization workflow, do! most _especially_ if they might have relevance to a _collaborative_system_ for digitizing e-books... but if your individual workflow is unique to you, you must recognize that it's not of any use to us. -bowerbird

3 2

a review of some digitization tools -- 001
by Bowerbird＠aol.com 17 Nov '11

17 Nov '11

so let's review some digitization tools, shall we? the task of digitizing a book is now fairly simple, thanks to cheap scanners and good o.c.r. today... scanners have been incorporated into printers for a while now, and even the cheaper ones are able to do a satisfactory job on most p-books... also, the best-of-breed o.c.r. app -- abbyy -- is now cross-platform _and_ fairly affordable... but the best news is that you don't even need to do the scanning or o.c.r. yourself nowadays -- if you want to do a public-domain book -- because the chances are very good that google or internet archive has already done it for you. so search around to see if scans already exist. (if you want a book in a non-english language, you might find it at google or internet archive, but if not, there are other sites you can search, and you likely know more about 'em than i do.) you might even discover that a book has been scanned multiple times. that's a good thing... what you should do is look at all of the copies, and find the one with the best-looking scans and (more important) the most-accurate o.c.r. download the scans and the o.c.r., as well as a djvu copy (if available) and/or the .pdf copy. the first thing to do is to spellcheck the o.c.r. when doing this, you must look at the scans! keep in mind that you aren't "correcting" the spelling, per se, but rather matching the book. now, do not take this to a ridiculous extreme. if there was clearly an _error_ in the p-book, you will want to fix it. you might even wanna update some anachronisms, such as changing "to-day" to the version that we all use "today". (i typically delete periods from chapter-heads.) and there might be some changes you'll make in order to make the text work as an _e-book_. (i make sure that the table-of-contents entries _exactly_match_ the chapter-heads themselves, so my automatic link-creator works correctly.) but be conservative in making these changes, because you might change the book's flavor... for instance, in the early days, p.g. would strip out the pagenumbers from an index, "because an e-book doesn't have pages"... well, that's stupid, because those numbers didn't _hurt_ anything, and they might have proven to be quite useful down the line, so they _should've_ been left in, "just in case". don't make that kind of mistake. while you are doing your spellcheck, it is very important that you _add_new_words_ to your spellcheck dictionary -- such as character names, place names, and so on. not only does this mean you only have to deal with them _once_, it also means that when a similar-looking variant comes up, it's probable that it's gonna be a "scanno". adding words also means you can re-do the spellcheck after any/all future edits, with a result that's both quick and clean. that helps catch any errors that you have inadvertently introduced during editing, something which happens to everybody... _after_ spellcheck, depending on the text you have downloaded, you might have to _restore_any_text_styling_, such as italics. if o.c.r. output gets saved as "plain-text", it loses its formatting, including _italics_. so you need to restore it. in most books, there aren't many italicized words, so this is a manageable task, albeit very mundane. if your book _does_ have lots of italics -- and you should check this at the outset -- you might want to re-do the o.c.r. for it... specify that the output be saved as .rtf... and then use your wordprocessor to place _markers_ around all the italicized words. you can do this by using a global change. you can use underscores as markers, or the bracket-i/bracket-slash-i html tags, or another convention you might invent. you'll also want to mark any _bold_ too, but most books contain very little bold, except in their headers, which we will be marking separately in a later process, so you don't need to worry about them now. in this vein, note that what you're marking right now are _individual_ italicized words or phrases. if a whole paragraph is italics, or bold, make a note about it for the future. the other thing you'll have to do, again in your wordprocessor, is to "close up" any "spacey punctuation". for the most part, these are double-quotemarks (and periods and commas) with a space on _both_ sides, so it's rather easy to do a search for them. so the first tool you'll use for digitization is your old friend, your trusty wordprocessor... -bowerbird p.s. if you have a book you'd like me to use as an example during this series, suggest it.

2 1