- gutvol-d - lists.pglaf.org

What Could Have Entered the Public Domain on January 1, 2012
by Jim Adcock 02 Jan '12

02 Jan '12

http://www.the-digital-reader.com/2012/01/01/what-could-have-entered-the-pub lic-domain-on-january-1-2012/

1 0

pontifications from mount high horse -- #1500
by Bowerbird＠aol.com 02 Jan '12

02 Jan '12

remember i said that, in the early days of d.p., many people doubted that it would even work, because too many cooks would spoil the soup? the same scenario is now playing again at d.p., as some people there believe postprocessing can't be distributed, giving similar reasoning. meanwhile, other people -- including don -- think it'd be wise to test to see if distribution will work well enough to clear a bad backlog. plus, as i said, i believe that digitization will increasingly be done "for the love of a book", by people who'll do that book "end-to-end". while some of them -- like roger frank, or nick hodson -- will do hundreds of books, others might only do one or two, or a few... whatever the case, it would be good if they had a website that guides them through it, and helps 'em attain a good level of quality. this project, the one i started christmas day, might eventually grow into that kind of site: > http://zenmagiclove.com/bkdig.py because although some people might get a sense of satisfaction by finishing a page, as roger recently argued, i think more will seek that feeling by finishing a whole book, especially a book that's meaningful to them. as one aspect of "book-dig", i built an editor: > http://zenmagiclove.com/bettyedit.py the name is because i used the "betty lee" book that we've been using for sample content lately. like the rest of the "book-dig" system, bettyedit is still "under construction", and thus a bit raw (e.g., edits aren't "sticky" yet, but will be soon). but as an example of the type of help it'll give, i've currently coded it to render italic highlights (in magenta) and doublequote-checks (in blue), with due apologies to the color-blind out there. cleaning the text of books is a lot of fun for me. and coding apps to do that job is even more fun. *** have a nice year in 2012. i will. -bowerbird

2 1

pontifications from mount high horse -- #1498
by Bowerbird＠aol.com 02 Jan '12

02 Jan '12

ok, i went back and did a better job on "betty little", and then compared the product to the original o.c.r. well, what i _acutally_ have is what i scraped from his editor demo, and some of that text had been edited... but i got to it pretty early, so i think i minimized that. (it would be nice if roger put out his _actual_ o.c.r. it would also be great if roger put out his .rtf copy. but i'm not sure how interested he is in this stuff.) at any rate, i will shy away from hard numbers, and just report the pattern of results, which is very clear. in general, i found this digitization looks exactly like the dozens of other ones that i have reported on here. as usual, the o.c.r. was good. quite surprisingly good. (i guess it's time that we should no longer be surprised. still, these scans were _murky_; but no, it didn't matter.) there were right around 256 errors, on a 256-page book, on the raw o.c.r., which is pretty much what you'd expect. considering the number of lines -- over 7000 of them -- those 256 errors constitute an accuracy rate of over 95%. roger hasn't released his text yet, so i can do a compare, which will probably reveal more errors that i missed, but even if i missed twice as many as i found (quite unlikely), the accuracy of the raw o.c.r. is still gonna be about 90%. *** i spent more time finding and fixing errors in this book than i wanted to, more time than i would have otherwise, because i was using it as the content for a new program. so i can't give a good estimate of the cost-benefit ratio of the time i spent, but i can say that i did indeed catch a pretty good percentage of the errors on my first pass. i made errors, lots of them! -- many more than the 22 which roger reported a while back -- but i can also say that i woulda caught almost all of the errors originally _if_ i had done a careful job, and done it more fully... i did a rush job, because i didn't know how fast roger was going to act, and i wanted to get my stuff out first, so roger and everyone else would know that i _hadn't_ used his results to create mine, i did mine on my own. and i wasn't thorough, because i didn't know if people would care. heck, i didn't even know if _i_ would care. but the project ended up being fun. it was _nostalgic_, coming in at the end of the year, plus i hadn't done an analysis of a digitization in a long time. and i'm rarely able to assess my own performance, so that's a blast... maybe i'll make a list of the errors later, for you to see. or maybe not. either way, the results are unmistakable. the o.c.r. was good. many of the "errors" were due to scan-spots -- which o.c.r. is duty-bound to report -- or outright errors in the p-book. it was full of errors! this is one of those e-books that is _more_accurate_ -- out of the chute -- than the p-book it came from. and, to repeat, this result is _the_typical_finding_... across the board, i have demonstrated, over and over, that the o.c.r. is good, and the vast majority of errors can be fixed by using extremely simple preprocessing, the type that you can do in one hour for a simple book. correcting the o.c.r. -- and even doing the formatting -- for a book is easy. it doesn't take rounds and rounds of volunteers wasting time and energy poring over a book. all it takes is one or two people using a good tool, and a couple of smoothreaders to catch the stealthy stuff... if you want to split up the job, you can have 10 or 20 people using that same "good tool" to do the job, and 10 or 20 smoothreaders, probably giving better results. but even one person and one smoothreader can do fine. and if you solicit error-reports and act on 'em diligently, you can execute a very smart march toward perfection... *** the things i just said apply to d.p. and p.g., obviously, but they also apply to some points roger has made... for instance, roger said this: > I've heard from some people that solo process > that actually like to go through the book > page at a time because they enjoy > following the story as they go, which > doesn't happen when someone is > in production mode > at the book level. now, first of all, of course, this is another case where roger exhibits fundamental misunderstanding about the essence of "production mode at the book level"... rather, it is page-oriented systems like the one at d.p. which make it difficult for people to "follow the story". in a system like the ones i make, the entire book is available to a person at all times, so they can surely "follow the story" if they choose to read it in order. the main difference is that, in a book-oriented system, you will _begin_ by cleaning up the big errors first -- the ones that are simple for the system to auto-detect -- so that you can then "settle in" to read each page, during which process you can look for _subtle_ errors, without being distracted by a need to fix any big ones, as that does indeed detract from "following the story"... in a page-oriented system, an absence of preprocessing means you might need to fix a bug on nearly every page, and that hurts both your accuracy _and_ comprehension. so roger has not just "failed to get things right" here... he has actually gotten it _completely_backward_, sadly. and, like i said, he's one of the smarter guys here. sadly. *** if anyone wants to see my analysis of my performance on "betty lee", let me know. or view the product online. > http://z-m-l.com/go/betle/betle.zml > http://z-m-l.com/go/betle/betlep123.html oh yeah, i almost forgot to tell you... i've programmed yet another book-digitization editor. once again, it's in python, like the one i built recently. but it's rather full-fledged, like the one i built in perl, back in 2010, when i was working on roger's "sitka"... it's not all finished yet, but you can look at it here: > http://zenmagiclove.com/bettyedit.py that is targeted at the ipad right now, but i can also make it work on an iphone by sizing the text smaller. it's _increasingly_ important to offer people the chance to contribute to your digitization project when they are using a mobile form-factor, like the ipad or the iphone. *** have a nice day. -bowerbird

5 7

Happy Public Domain Day!
by Wallace J.McLean 01 Jan '12

01 Jan '12

http://t.co/As075vYh

1 0

pontifications from mount high horse -- #1501
by Bowerbird＠aol.com 01 Jan '12

01 Jan '12

it's time for e-books to leave their incunabula period, and nobody else is better suited to removing their swaddling... tomorrow... tonight, i'm quaffing scotch through a straw... *** have a happy new year. -bowerbird

1 0

pontifications from mount high horse -- #1498
by Bowerbird＠aol.com 01 Jan '12

01 Jan '12

don said: > Al, Roger, Bird, anyone else, > For your list of scannos, > do you examine every instance > of every word on the list? > Many of those words > would be more frequently correct > than a misscan. i recommend you not even consider testing for a specific scanno unless you suspect that it will return more hits than false-alarms. then i would collect data from every test, to _ensure_ that it does, and stop using any that do not. that's a very high bar to clear, but if a test returns more false-alarms than hits, it's wasting my time. it is more important and valuable to build an infrastructure which acts quickly and efficiently on error-reports from your community than to waste the time and energy of your digitization volunteers. sadly, p.g. never gave itself the opportunity to learn that lesson... -bowerbird

1 0

Re: [gutvol-d] a review of some digitization tools -- 022
by Bowerbird＠aol.com 31 Dec '11

31 Dec '11

keith said: > Why do you not ask john if pandoc can do it > without human intervention! "without human intervention"? whatever in the world could that possibly mean? -bowerbird

5 6

pontifications from mount high horse -- #1499
by Bowerbird＠aol.com 31 Dec '11

31 Dec '11

i don't think anyone will build another digitization community like distributed proofreaders. and d.p. will wither away soon... the digitizers of tomorrow will be people like paul flo williams, and james simmons, who take on the task as "a labor of love"... they'll do books that they love -- _because_ they love them -- and not just take part in some abstract "digitization" project... they'll be far more likely to do a half-dozen books end-to-end, than to take on some piece of dozens (or hundreds) of books... they might get a few people to help 'em out, or they might not. it won't matter, not much, because either way they'll trudge on. and they'll use tools that will fit their small, personal workflow. *** this brings up something roger said a while back: > I also feel it fits comfortably with > users that want to just get one page right. > There's a sense of accomplishment > resolving the warnings on one page > and knowing it is "done." first of all, let me give roger full credit for what is one of his most significant contributions over the past few years, and that's saying a lot because he's made plenty of them... but one of the most significant is that his actual tests showed that if a person _says_ they consider a page to be "finished", then the odds are that it really _is_ finished, i.e., error-free... i had always argued, based on my gut, that it took _2_ people to confirm a page as error-free before we could "trust" it was, _and_ that anyone who made a change to a page could _not_ be counted as one of them. in other words, 2 _confirmations._ roger's research showed that that's not necessarily the case... at least it _hinted_ at that... but there was one troubling confound in his test, which is that he didn't factor the _initial_state_ of the page. and we _know_ that -- typically in o.c.r. -- _many_ pages start out error-free. indeed, in some o.c.r. files, after a decent preprocessing run, the majority of the pages will be error-free, so a person could certify _all_ of them as "finished", without even looking at 'em, and be "correct" far more often than they were "wrong". but there's another aspect to this, a psychological aspect, and one that involves the _motivation_ to do digitization, especially in a large project where you're a cog in a wheel. i fully recognize that people get satisfaction from the feeling of "finishing" a page, and that we can use that for motivation. but at what point does that person begin to feel manipulated, because the vast majority of the pages were "finished" before the person even looked at them, and we neglected to tell 'em, or -- worse yet -- we implied that the page _did_ have errors. i mean, it's one thing to say, "we think this book is error-free, so go ahead and read it, and let us know if you catch anything", which is the way that i would pitch beta-reading to my people. but it's another thing to say, "proofread this word-by-word", when you darn well know _most_ of the pages are error-free. there is a dishonesty about that which causes me discomfort. or, in other words, what kind of satisfaction does a person get from certifying a page as "finished" if the page was "finished" before they even got it, but you led them to believe it wasn't? *** there's a flip side to this, too. what kind of satisfaction did roger get from calling his pages "done" when he later found that he had missed many of the italics? this cuts both ways... anyway, just giving you something to think about in 2012... *** have a nice year. -bowerbird

1 0

pontifications from mount high horse -- #1498
by Bowerbird＠aol.com 31 Dec '11

31 Dec '11

roger said: > my sampling of the kind of errors > left behind by BB's process as far as it went. thanks for the fuller exposition, roger... just as a note, before we begin, i'd like to issue one minor note of protest that these errors are labeled as "bb:", when, actually, they were errors that were in _the_o.c.r._... it's not as if i _introduced_ these errors... it's just that my preprocessing failed to find and fix them. there's a difference... i don't have a suggestion for an alternative, and i see the reason for that nomenclature, just want people to grok the bias in the label. *** on a continued note... it's also the case that my preprocessing _did_ find-and-fix some 250 errors in the o.c.r., so i think the fact that it missed 21 is acceptable, as a first-pass. it can probably be improved, too, but i believe a 92% reduction in errors is nothing to sneeze at. even if i missed _100_, that would be a 70% reduction in o.c.r. errors. *** now that we're done with the minor protest... for the folks who aren't paying full attention, the whole point behind my argument is that a minimum of effort can fix a _lot_ of errors. i have never claimed it can find _all_ of them -- indeed, that would be a form of magic -- or that it's a _sufficient_ mechanism, by itself. this "minimum of effort" absolutely _needs_ to be followed by a beta-reading component. although i have never failed to stress that need, sometimes people seem to forget it completely. please remember to consider the full picture... once you do, you'll understand my rejoinder: my beta-readers woulda caught these errors. (just like roger's smoothreader caught them.) having said that, it is of some concern to me that this seems to change the error-rate so it becomes worse than 1-error-per-10-pages. gonna have to see what i can do about that... > With the addition of a good smoothreader, > many of these diffs would have disappeared right. (well, i'd say _all_ of them, but i'll settle with "many of them" if you insist that it's true, as we're talking about imaginary beta-readers, in my case, so i can't be certain of their skills, but if you say they're not perfect, i won't argue.) these results are from a midpoint in the workflow. while all the errors you reported are very real -- or almost all of them, anyway -- they are from classes that i have oft-acknowledged as ones that are missed in my _preprocessing_... there's a good reason i use that particular word -- _preprocessing_ -- because it _emphasizes_ the point that this doesn't create a final product. let's look at those acknowledged weakspots: > stealth scannos > missed italics > splotches > spelling discrepancies "stealth scannos" are something my preprocessing will never catch. that's why it needs beta-readers. "missed italics" will depend on abbyy's .rtf quality. if you don't use abbyy, you'll do all of it manually. and if you suck at it, like me, the results will suck. it's not that i cannot imagine coding a routine that could check for stealth scannos, or italics. i could develop some ideas and take a stab at it. i just wouldn't expect that i'd be very successful, in cost-benefit terms. you can do lotsa checks for stealth scannos, but you get too many false alarms. and my whole point is that you utilize cost-benefit. it's easy -- and efficient -- for beta-readers to catch stealth scannos, and even missed italics... so let them do that job! that's the wisest course. "splotches" are something else that's hard to catch, sometimes even impossible. i'll look at these errors, and see if they are the type of thing that _could_ be detected with some programmed routines. but still, it's obvious that some can't, which is -- yet again -- why you need to have beta-readers in the workflow. as for "spelling discrepancies", i do a spellcheck, but if both forms pass, then i let both of them go. and i didn't do consistency checking on this book. i _will_ make it part of my process, when i finally formalize it, but i didn't do it on this book, nope. *** so this wasn't a fair fight. roger used a smoothreader, and i didn't. so _of_course_ his results will be better... roger might've even done a word-by-word proofing, similar to the kind done at distributed proofreading. that can give even _better_ results. sometimes not, but sometimes it _will_, there's no question about it. where the question comes in is whether it's worth it, in terms of the time and energy that it takes people. go take a look at a project page over at d.p., and see how much time it takes the proofers to step through the pages of a round. you can see a page being saved every minute, or two, or three, so a 256-page book -- like "betty lee" -- can take about 600 minutes to go through a single round. that's 10 hours of work! for one round! is this a wise use of time and energy? i sure don't think so. when a beta-reader does a book, at least they get the pleasure of actually _reading_ it... those proofers over at d.p. don't even get that benefit. i don't think the time and energy that they _volunteer_ is being used in the best possible cost-benefit manner. and _that_ is what this discussion should be about... *** all in all, these results don't surprise me one bit. this o.c.r. had more scannos and splotches than i usually find, but the scans weren't all that hot, as you'll readily see just by paging through 'em. maybe i'll run 'em through scan-tailor, just to see if i can then get better o.c.r. out of them... but if roger will promise to make available his data -- i.e., scans, r.t.f. -- for his future books, i will formalize my preprocessing into a system, so he could run regular tests of its efficiency... -bowerbird p.s. now, notes on each of the individual errors... stealth scannos -- i leave these for the beta-readers... > RF: some of them, and give Ramon's message, > BB: some of them, and give Earn on's message, shoulda caught this, as an unexpected mid-sentence cap. > RF: Betty and next to Peggy Pollard, who, it > BB: Betty and nest to Peggy Pollard, who, it crap. i did catch one "nest/next" scanno, but forgot to check the rest of the file for another. > RF: a thing to work for that being president > BB: a tiling to work for that being president i'd guess a check for "tiling" would be a worthy one. > RF: the back. Mary Emma could not go with > BB: the back. Mary Emma could hot go with "could hot" and "can hot" checks will surely be worthy. > RF: problems. From Lucia's manner, she > BB: problems. From Lucia's manlier, she a "manlier" check might well be worthy. > RF: of the page and below was a brief resume > BB: of the page and below war; a brief resume a check for "war" might be good, or might give false alarms. > RF: I'm the crossest girl you ever saw, so far as mere looks > BB: I'm the Grossest girl you ever saw, so far as mere looks shoulda caught this, as an unexpected mid-sentence cap. *** spelling discrepancies -- these are just because i made mistakes > RF: of those still, quiet stiletto exchanges > BB: of those still, quiet stilletto i probably gave this the o.k. because i don't know how to spell it. > RF: tonsillitis. Betty saw her and overheard > BB: tonsilitis. Betty saw her and overheard i probably gave this the o.k. because i don't know how to spell it. *** splotches -- not much i can do about these > RF: packed a thin chiffon dress, while > BB: packed, a thin chiffon dress, while can't see how i could ever devise a test to catch that. > RF: this, Miss Betty Lee!" > BB: this,' Miss Betty Lee!" didn't do a check of the balancing of single-quotemarks. *** missed italics -- not much i can do about these > RF: wouldn't do _one thing_. She is sweet > BB: wouldn't do one thing. She is sweet i don't know how to code a test for italics. > RF: other times too, but _always_ then, > BB: other times too, but always then, before i suck at finding italics. i missed 40 cases, not? *** missing quote marks -- my tests should've caught these > RF: won't you?" > BB: won't you? shoulda found this, unless both quotemarks were missing. > RF: who sat down. "How is your mother > BB: who sat down. How is your mother shoulda found this, unless both quotemarks were missing. *** extra quote marks -- an ironic coincidence of splotches > RF: little habit of dropping in when > BB: little habit of 'dropping in' when two splotches happened to do something that made sense. *** levenshtein check -- i should include this in my process > RF: are the Sevillas and where do they live? > BB: are the Savillas and where do they live? i got so confused on this name, but thought i checked 'em all. *** guiguts-catchable errors -- i just plain forgot these tests > RF: sometimes! I can't study! Come over here > BB: sometimes! I can't study I Come over shoulda caught this, as an unexpected mid-sentence cap. > RF: who reads the sport page." > BB: who reads the! sport page." shoulda caught this, as an unexpected sentence-starting lower. > RF: know." > BB: know," (at end of paragraph) shoulda caught this, as an improper paragraph-termination. *** a bug in BB's generator -- not "a bug"; indicates a continued quote > RF: like my residence here. > BB: like my residence here." " not a bug. will be deleted before the product goes "final".

1 0

Seven Million+ New Reader Devices activated on Xmas Day Alone
by James Adcock 30 Dec '11

30 Dec '11

http://www.the-digital-reader.com/2011/12/27/nearly-7-million-ios-and-androi d-devices-were-activated-on-christmas/#more-28013

7 8