November 2011 - gutvol-d - lists.pglaf.org

a review of some digitization tools -- 002
by Bowerbird＠aol.com 20 Nov '11

20 Nov '11

keith- most of the points you made in your posts this morning are ones which i feel do not need a response immediately, if at all, so if i get to them this weekend, i'll write a reply... and if not... not... the one exception is this... i said: > > so the first tool you'll use for digitization is > > your old friend, your trusty wordprocessor... keith said: > Are you really, advising people to > use bloat ware, MS and OO. > ;-)))) first, yes, i see the smiley, and i respond under that light... this series consists of reviews of e-book digitization tools. your wordprocessor is the first tool in your toolbox, so yes, indeed, i am really advising that people consider it as such. the wordprocessor serves a number of different functions. it's like a crescent-wrench -- it can fit a nut of any size... for that reason, the crescent-wrench is extremely valuable, undoubtedly the single most-important wrench in the box. > http://en.wikipedia.org/wiki/Adjustable_spanner nonetheless, most mechanics almost always use a wrench that's more specialized for the exact task at hand, such as a socket-wrench with a head that's the same size as the nut. > http://en.wikipedia.org/wiki/Wrench likewise, even though we _could_ be using a wordprocessor -- my "sweet grapes" series in october took that approach -- this series looks at tools _specialized_ for the tasks at hand. that's why much of the focus here is on digitization _tasks_. if you wanna review _tools_, you need to have a good handle on what you want those tools _to_do_ -- what to accomplish. the _overarching_ requirements here are simple to list: 1. obtain the text (via scanners, and o.c.r. software) 2. clean the text (the first set of tasks for our tools) 3. turn the text into e-books (the second set of tasks) i dispensed with the first requirement rather quickly, since that's not a job most of us need to do these days, thanks to the big scanning projects from google and internet archive. the second requirement is also fairly straightforward, in that you _could_ do the job by using your favorite wordprocessor. more to the point, though, was to list the tasks there: 2a. do a spellcheck 2b. fix spacey punctuation 2c. restore styling, e.g., italics once you're aware of the exact tasks, you can evaluate tools. for instance, the proofing rounds at distributed proofreaders exist to complete tasks 2a and 2b. and the formatting rounds have the job of doing 2c, and _some_ tasks in requirement 3. it is this backdrop of knowing the tasks that need to be done which allows me to rail loudly about the inefficiencies of d.p. they're wasting far too many human resources to accomplish these relatively-simple tasks; they could be doing much more. anyway... once you know tasks, you can evaluate the tools. in that regard, let's list the tasks for the third requirement. 3a. tagging the structural aspects of the text 3b. converting the text into .html (intermediate and final) 3c. converting intermediate .html into e-book output-files i'm gonna cut directly to the chase, and tell you that there are scripts to accomplish the 3b and 3c conversion steps, and thus the main focus from here on will be _tagging_the_structure_... at distributed proofreaders, which is important to examine since it _is_ an existing system of the type we want to make -- because the way to test this stuff is in _the_real_world_, and _not_ in the realm of some "theoretical discussions" -- some of 3a is done during formatting rounds, with the rest accomplished by offline, non-distributed "post-processing". a sizable minority of post-processors, including _lucy24_, do their work in a wordprocessor. lucy uses "subethaedit", a collaborative text-editor that truly is lean and powerful. > http://www.codingmonkeys.de/subethaedit/ but most d.p. post-processors use an app called "guiguts". guiguts was stagnant for a long time, but hunter monroe has recently taken on the burden of updating it again, so i'll give him a chance to step in now and tell you about it... hunter, are you here? -bowerbird

3 2

now keith wants me to jump in his tar-pit
by Bowerbird＠aol.com 20 Nov '11

20 Nov '11

keith, you seem to be having "theoretical" discussions on 3 or 4 vague phantom topics, while i'm talking about stuff which either actually _exists_ or has been solidly proposed. and there's no sense continuing when we're so badly out of sync. so let us summarize and finish. 1. i have systems, online and off, which are simple and work nicely. 2. keith is too busy to code stuff, but seems to have many opinions. 3. jim likes using bloatware, but is too cheap to buy commercial-grade. 4. lee wants someone else to do all his .html markup for him first. 5. don knows databases, but hasn't generalized from his d.p. experience the conclusion that any system which requires administrators will become corrupt because of the power-seekers. 6. tar-pits are sticky places. 7. sometimes listserves can be too. 8. i forget what 8 was 4. 9. number 9, number 9... 10. you need 10 items for a top-10. 11. is my favorite number. -bowerbird

3 3

ok let's take stock
by Bowerbird＠aol.com 20 Nov '11

20 Nov '11

given all of the recent topic-shattering, it's only natural to feel a little scattered. so let's take stock, and regroup. first of all, it's _cool_ that people feel free to share our individual e-book workflows. other individuals here might learn from it. and more importantly, we might come to learn something that could be of use to us in building a system for collaborative work. it seems to me that a collaborative system is more in tune with aims of this listserve... and if your workflow depends in a vital way on your own individual experience or talent, it's not gonna be of use to that, not really... likewise, if it depends on a software program (such as ms-word or open-office) which has grown to be a gnarly complex monster, then odds are people aren't gonna want to use it. fine for your use, sure, just not for our use... also, hand-in-glove here, if the app requires a good deal of hand-holding, that's not good. what we want, then, is simple software that takes care of a simple workflow, and which can run either offline or online, invoking a system for collaborative e-book digitization. in this vein, we find distributed proofreaders. they have such a system, and such software... and, for the most part, although they will say that they are not completely satisfied with it and have lots of features they'd like to add, they often turn down coders who volunteer... (when those coders will not be subservient.) so my impression is d.p. is happy with itself. and the same can be said of the p.g. machine. i don't get a feeling they want any help either. they don't want you to meddle with their stuff. in a way, that's pretty liberating, since it means we don't have to support any lingering baggage. we are free to design a system any way we like... i've built the engine of such a system. if people would like me to put it up again, for evaluation, i could do that. (it was the old "sitka" project.) roger frank also coded the engine for a system. he has also taken his project down, but i bet if enough people expressed interest in seeing it, he'd also remount it, so you could evaluate it... *** all these systems are pretty "standard", in that the interface has a text-field and shows a scan. if a review of the scan reveals the text is wrong, you edit the text. it's all very straightforward... other interfaces have been suggested as well. i made one where each text-line is displayed next to a checkbox that should be checked if the line contains any errors, so that that line will later be corrected by the administrators. i made another one where questionable lines get shown in various forms, and the person is asked to indicate the one which is correct. (that's for when multiple files get compared.) we also saw an interface from edward betts on this list recently, where lines are shown under the relevant slice from the scan, and the person clicks any incorrect word to fix it. so the interface can vary as well as the tool... *** it's easy to code an e-book digitization system. since the d.p. code has grown into a monster, people familiar with that code might laugh at such an assertion. but i make it with certainty: it's easy to code an e-book digitization system. i've already done my autumn project this year, so my plan was to wait until spring to do this, but if people want to go to work on it _now_, just let me know, and we can start a project to collaboratively code such a digitization system as an open-source project. since, as i just said, i've already done this, i could serve as a guide. but i'm not going to just hand over my code, so if you want it to be open-source, you'll have to do the vast majority of the coding of it yourself. and that's good, because it means that you can make the choice of the language by yourselves. let me know. if nobody's interested in tackling such a project, i'll do another how-to series, on digitizing tools. in fact, i might do that either way. yes, yes i will. so, by all means, if you want to share aspects of your personal e-book digitization workflow, do! most _especially_ if they might have relevance to a _collaborative_system_ for digitizing e-books... but if your individual workflow is unique to you, you must recognize that it's not of any use to us. -bowerbird

3 2

a review of some digitization tools -- 001
by Bowerbird＠aol.com 17 Nov '11

17 Nov '11

so let's review some digitization tools, shall we? the task of digitizing a book is now fairly simple, thanks to cheap scanners and good o.c.r. today... scanners have been incorporated into printers for a while now, and even the cheaper ones are able to do a satisfactory job on most p-books... also, the best-of-breed o.c.r. app -- abbyy -- is now cross-platform _and_ fairly affordable... but the best news is that you don't even need to do the scanning or o.c.r. yourself nowadays -- if you want to do a public-domain book -- because the chances are very good that google or internet archive has already done it for you. so search around to see if scans already exist. (if you want a book in a non-english language, you might find it at google or internet archive, but if not, there are other sites you can search, and you likely know more about 'em than i do.) you might even discover that a book has been scanned multiple times. that's a good thing... what you should do is look at all of the copies, and find the one with the best-looking scans and (more important) the most-accurate o.c.r. download the scans and the o.c.r., as well as a djvu copy (if available) and/or the .pdf copy. the first thing to do is to spellcheck the o.c.r. when doing this, you must look at the scans! keep in mind that you aren't "correcting" the spelling, per se, but rather matching the book. now, do not take this to a ridiculous extreme. if there was clearly an _error_ in the p-book, you will want to fix it. you might even wanna update some anachronisms, such as changing "to-day" to the version that we all use "today". (i typically delete periods from chapter-heads.) and there might be some changes you'll make in order to make the text work as an _e-book_. (i make sure that the table-of-contents entries _exactly_match_ the chapter-heads themselves, so my automatic link-creator works correctly.) but be conservative in making these changes, because you might change the book's flavor... for instance, in the early days, p.g. would strip out the pagenumbers from an index, "because an e-book doesn't have pages"... well, that's stupid, because those numbers didn't _hurt_ anything, and they might have proven to be quite useful down the line, so they _should've_ been left in, "just in case". don't make that kind of mistake. while you are doing your spellcheck, it is very important that you _add_new_words_ to your spellcheck dictionary -- such as character names, place names, and so on. not only does this mean you only have to deal with them _once_, it also means that when a similar-looking variant comes up, it's probable that it's gonna be a "scanno". adding words also means you can re-do the spellcheck after any/all future edits, with a result that's both quick and clean. that helps catch any errors that you have inadvertently introduced during editing, something which happens to everybody... _after_ spellcheck, depending on the text you have downloaded, you might have to _restore_any_text_styling_, such as italics. if o.c.r. output gets saved as "plain-text", it loses its formatting, including _italics_. so you need to restore it. in most books, there aren't many italicized words, so this is a manageable task, albeit very mundane. if your book _does_ have lots of italics -- and you should check this at the outset -- you might want to re-do the o.c.r. for it... specify that the output be saved as .rtf... and then use your wordprocessor to place _markers_ around all the italicized words. you can do this by using a global change. you can use underscores as markers, or the bracket-i/bracket-slash-i html tags, or another convention you might invent. you'll also want to mark any _bold_ too, but most books contain very little bold, except in their headers, which we will be marking separately in a later process, so you don't need to worry about them now. in this vein, note that what you're marking right now are _individual_ italicized words or phrases. if a whole paragraph is italics, or bold, make a note about it for the future. the other thing you'll have to do, again in your wordprocessor, is to "close up" any "spacey punctuation". for the most part, these are double-quotemarks (and periods and commas) with a space on _both_ sides, so it's rather easy to do a search for them. so the first tool you'll use for digitization is your old friend, your trusty wordprocessor... -bowerbird p.s. if you have a book you'd like me to use as an example during this series, suggest it.

2 1

we don't need corporate publishers to have physical books
by Bowerbird＠aol.com 17 Nov '11

17 Nov '11

keith said: > You call that what comes out of a consumer printer a book! for the most part, yeah. use "half-letter" or a5, for a good size. plus you need to "bind" the pages, maybe put a nice cover on it. but a heavy-duty stapler works well enough to do the "binding", and a wrap-around adhesive slick cover is simple to engineer... the internet bookmobile was doing this a decade ago already... > Nahhh! just a bunch of loose sheets of paper. wrong. > Even if I had a binding machine (true there are copy shops), > and the high quality paper for those nice color spreads > (costs to much for normal consumers) you don't need "a binding machine". unless you want one. and people are already printing full-color stuff regularly, most often missing-dog flyers and garage-sale posters... > why print it? well, there might be any number of good reasons, or at least reasons which the person who is doing the printing considers to be "good enough" to spend the resources to do the printing. but i'd guess that the most typical reason would be because the book means enough to the person that they want to have a tangible -- i.e., _physical_ -- copy of it, as a sort of talisman. you know, the kind of book that was extremely meaningful? the kind of book you _love_? the kind that changes your life? so you want something representing it that you can _touch_. in much the same way that a ticket-stub serves as a "reminder" -- or a souvenir -- of that concert that you and your sweetheart went to for your first date, the night you made love the first time. that's what i would suspect will be the most compelling reason. but i suppose every person will have their own unique reasons... and who are we to tell them that their reasons aren't any good? > I am very happy with sitting in bed, on the couch, > or where ever reading on my 17" lap top. The earth > disapating from its bottom is, almost as good as a warm > fire place(I do not have one). that's cool. (or warm, as the case may be.) and that's _you_. and i am most certainly not gonna tell you that you're wrong. whatever you want to do is fine with me. but likewise i'm not gonna tell anyone else that they're wrong to print out a book. if that's what they want to do, i'm gonna tell 'em to "go for it". > Why would anyone waste all that paper. except that they obviously don't consider it to be a "waste". they consider it to be a good use of resources they paid for. > As for me I do not print anything any more > unless I have to. It is just a waste. i'm very much like you. haven't had a working printer for years. maybe a few times a year i ask my girlfriend to print something on her machines at work, but for the most part, i don't miss it... but that's me. other people do things differently. c'est la vie. > I had a friend (has died), he use to print his emails > and read them from paper. perhaps god struck him down -- for wasting paper. :+( (oops. sometimes i do cross the line. but i always admit it.) > He simply could not get himself accostumed > to reading off a computer screen/monitor. well, see, there you go, another very good reason to print stuff. > Sorry, I think the future should be digital. the future _will_ be digital. correction: the future _is_ digital even as we speak... and it will only become even more so... but we are also still physical beings. who value physical stuff. that has always been true, and will undoubtedly always be true. still, we don't need corporate publishers to have physical books. -bowerbird

2 1

KFires arriving
by Jim Adcock 17 Nov '11

17 Nov '11

FWIW, I received my KFire this morning... ... but still no KF8 documentation nor support tools!

1 1

responding to threads i initiated
by Bowerbird＠aol.com 16 Nov '11

16 Nov '11

ok, i see that lee too is replying to threads i initiated... i don't know if he's responding to me, or to others who have posted in those threads, because -- as i've said -- i have reinstated my policy not to read lee's messages... i have also found that -- unlike keith, who i believe is often "genuinely" confused about the topic at hand -- lee is prone to intentional attempts to confuse things. (amusingly, jim falls into _both_ groups.) so it might not be apparent if lee is responding to _me_ or to _someone_else_. but whatever the case might be, the situation stands. if you would like to see me respond to anything that lee has stated, or questions he's asked -- whether to me personally or to anyone else as well -- or if you think i will be _stumped_ by what lee said and you wanna see me stumped, rephrase in your own words. i will respond, carefully and thoroughly, just like always... otherwise, i'll assume that y'all wish to see these issues die a graceful death, if that is indeed still a possibility... thank you for your time and attention. :+) -bowerbird p.s. some of you might wonder why my antagonists even put out their stuff, since i immediately refute it. the reason is pretty simple... they know that _many_ of the people in their clan have me kill-filtered, thus their target-audience never hears my refutation, and thus believes whatever falsehoods they have planted. but of course the misimpressions inside those heads will die when their bodies die, whereas the _archives_ of this list will persist for a _long_ time in cyberspace, a permanent record of all the recurring dishonesties...

4 3

now keith wants me to "slow down"
by Bowerbird＠aol.com 15 Nov '11

15 Nov '11

keith said: > Slow down a bit BB, oh dear sweet lord... > You are misunderstanding something here. > I leave out the html part because > that is a different matter which > you can have out with don! yeah, well, gee, keith, that's very unfortunate. because "the html part" was the on-topic part, the relevant part, the part we were talking about. so don was the one who was "misunderstanding". because -- as he admitted in his last message -- he had shifted the topic to his use of wordpress as a _storage_mechanism_ for existing html/css. just to remind people... i criticized lee's tool because it kicked in only _after_ the text had been marked up in .html, as if the markup process happened magically, when -- as i said -- markup is "the hard part". lee responded that word-processors can do a conversion to .html. (which i had granted already, but noted their markup is terrible.) then jim replied "you could use open office". open office, the software which proves that an open-source effort can create a program as bloated and complex as microsoft makes. yeah, great suggestion, jim. bzzzt. next!... and then don stepped in to suggest wordpress. and yes, wordpress will turn text into .html... but the output is a mess. so i called him on it. and then he responded that he wasn't actually using wordpress to mark up the text into .html. in other words, he admitted he was off-topic... (not directly, of course. but it was clear enough that i didn't even feel the need to rephrase it -- until now.) > It is the database part you do not understand. i understood "the database part" fine, both in the part of its mechanics, and the part it was off-topic. and, as i said, as long as don is only talking about his _own_ use of wordpress, for his _own_ project, as a content-management-system, i'm fine with it. > The user need not know anything of the database > to extract, use, and save files, in what ever format! that's true, keith, but only on a very superficial basis. what if i want to "use" one of the files by submitting it to an exterior spell-check app? or grammar-check? what if i want to merge all of them into a single file? what if i want to do a specific search-and-replace on any files that contain graphics, but a different one on any files that do not contain graphics, except for the ones which contain front-matter, plus i need to do a separate search-and-replace on the reference section? > You see the database is first and firmly just > a filing system for the files! Just like the OS is! except those systems aren't "just like" each other by any stretch of the imagination, since the file system is understood implicitly by every average user because it is utilized every day in the course of normal practice, whereas only a tiny percentage of them know databases, and only a few of _those_ people can manipulate mysql. and it shouldn't even be necessary for me to explain that, because if you can't see a gulf that wide, you must be blind, or have your eyes closed, or have your head stuck somewhere. > So, if don wants to use a database (system) as a file(filing) > system what is wrong with that. i answered that question up above. for his own stuff, don can use any system he likes. why would i care, in the slightest? but if don wants to suggest a database approach for a system that has thousands of users who are digitizing thousands of books, all with one database, then i'm going to suggest that he rethink that model. which was a _different_ thread we had going here once. (but your comment here, keith, if off-topic to both.) > But, the people administering the site > know it is there and glad they have it. ok, and here is the crux. you guys are all happy to invent a site with "administrators". my aim is to create a workflow that average people grok, and can manipulate, if necessary, _without_ administrators. because, as d.p. has shown so clearly to those who can see, once you have a need for "administrators", the system gets trapped by the power-hungry and becomes nonresponsive... > Basically, Don will have a front end to his database, > that will give you all you need and it will be just as easy > (probably easier) than using your script based system. that's what the "administrator" types always _promise_. "i'll make things easy for you, give you all you need..." but once you turn over your soul to them, it's too late. > On the other side I would have to learn > how to use your scripts and how to find them, > how to reprogram them to work > for a particular book or text! my word, keith, you seem to know _a_lot_ about something you know nothing about. > To quote Steve Jobs: "It just works!" it's easy to quote steve jobs. any blooming idiot can do it! it's a heckuva lot harder to make something that just works. *** so, um, no, keith, i'm not going to "slow down". sorry, dude. however, i will kindly suggest that you speed up. to keep up. -bowerbird

6 7

don wants me to "slow down"
by Bowerbird＠aol.com 15 Nov '11

15 Nov '11

don said: > Slow down let me come back to that... > read the *second* sentence. i did read it. i wasn't impressed. > It's pure html in the mysql database. precisely. that's my objection. and i stated it, as such, quite clearly, when i said: > so on top of the obfuscation of the text by the .html, > it's now buried inside of a mysql database, meaning > that anyone who wants to understand the system well > needs to acquire an additional hairy set of experience. in other words, you take clean text and gunk it up with .html, and then you also bury that gunked-up text in a database. and you want us to think that is a good system? that's like taking an art-book with nice pictures and drawing mustaches and beards on all the pretty women (that would be the .html coding), then putting the book into a bag (the database), and then expecting us to "appreciate" the artistry. um, no thanks. just leave the thing unmolested. > And if it's "impure", then blame me, because > it's precisely what I imported from the DP project > - for better or worse. Same for the CSS. well, one of us is confused, that's for sure... _i_ thought we were talking about a system that could be used by an entity like d.p. to do book-digitization, with a suggestion wordpress would do the .html/.css. in other words, we put in .text, and get out .html/.css. but now _you_ seem to be discussing your own system, where you "imported" the .html (and the .css) from d.p. you seem to have already _started_ with the .html/.css. so one of us is confused about the topic of discussion. i don't really care what you use for your own projects. whatever makes you happy is absolutely fine with me. but if you're going to propose a system for volunteers, i'd say it must take their skill-sets into consideration, and attempt to make things as simple as they can be. and for book-digitization, text-files are all you need, a fact which i have demonstrated time and time again. text-files have the advantage of magical simplicity... the weird thing is that most geeks know this very well, and practice it near-religiously in their own workflows. a very big part of the unix philosophy is "piping" text... but all of a sudden, in dealing with _books_, which are (and always have been) predominately text (with some pictures thrown in), all of a sudden we need _markup_. what's up with that? especially since, in their own _manuals_, geeks such as the python community use -- ta-da! -- light-markup... i guess angle-bracket-tags are only for the suckers who are too stupid to realize that they don't really need 'em... *** don said: > Slow down um, again, no thanks, don. i have a much better suggestion -- flip your script. let's instead have _you_ "speed up" a bit -- a lot! -- along with the rest of this listserve, if it's possible... you've become accustomed to the glacial pace of distributed proofreaders and project gutenberg and this listserve, where the topics of conversation haven't changed all that much in a decade when -- all around us -- e-books have finally "caught on". 8 years ago, this list was hung up on .xml, acting as if that was "the important thing" for discussion... history proved that was one big stinking dead-end. and now you're back on .html as "the big thing", but that's a dead-end too. do you really believe that in 2025, people will still be fussing with angle-brackets? get off the merry-go-round and shoot for the moon. do you want to know the most exciting development in e-books in recent months? it's amazon's "x-ray". > Amazon invented X-Ray, a new feature that > lets customers explore the "bones of the book." > With a single tap, readers can see all the passages > across a book that mention ideas, fictional characters, > historical figures, places or topics that interest them, > as well as more detailed descriptions from Wikipedia > and Shelfari, Amazon's community-powered encyclopedia > for book lovers. Amazon built X-Ray using its > expertise in language processing and machine learning, > access to significant storage and computing resources > with Amazon S3 and EC2, and a deep library of > book and character information. The vision is > to have every important phrase in every book. if amazon pulls off "x-ray", it could be phenomenal. this is the kind of thing that p.g. could've "invented", since it once had the biggest public-domain corpus. but you guys here were caught up in _file-formats_, among the most tedious and irrelevant e-book trivia. so while you've focused on markup, other entities are doing stuff that's far more interesting and important. it's as if you were at the grand canyon, but instead of paying any attention to the marvelous natural features, you're all still at the tram arguing about what kind of upholstery will be best to use for the seats in the tram. and when i roll my eyes, you think i'm picking a fight. you're missing out on the real action, and you don't have the slightest clue that that's what's happening. so no, don, i _ain't_ gonna "slow down", because i'm _tired_ of going nowhere and nowhere and nowhere. and i firmly believe that you, don, should also decide not to let your good imagination be held back by the foot-draggers over at d.p. you're smarter than them. so demonstrate it, don, by letting your creativity soar. and believe me, a suggestion to "use wordpress" does _not_ accomplish that objective; it doesn't come close. -bowerbird

4 3

now keith wants me to "slow down"
by Bowerbird＠aol.com 14 Nov '11

14 Nov '11

and again, i'm no longer reading jim adcock's posts, so if he makes any points you'd like me to address, you will need to rephrase them in your own words... also, if you think anyone else in my kill-filter, like marcello or lee passey or josh whatshisname or david starner or the bad alex, makes a post which i should consider for my $2 "positive contribution", you should rephrase that, and attribute accordingly. thank you. have a nice day. :+) -bowerbird

1 0