September 2011 - gutvol-d - lists.pglaf.org

these grapes are sweet -- lesson #02
by Bowerbird＠aol.com 04 Sep '11

04 Sep '11

hello. i hope you're having an enjoyable holiday weekend. :+) as i said, the thrust of the thread is how to digitize a book. if anyone had suggested a book, i woulda used their choice; but since they didn't, i decided to do an old standby again... "books and culture", by hamilton mabie, is a pleasant book. it's 280 pages, with about 220k of text, so it's average size. i used a version scanned by the internet archive last november. find it by searching archive.org for "booksculture00mabiuoft". as many of you will remember, we've looked at this book a lot. in fact, it was one of the examples which i just gave benjamin. this means that i can test my efforts in this current digitization by comparing my results to a very-well-proofed criterion-file... i'll share all my in-process versions with you later in the week, but for now i can post some preliminary results. as i expected, the o.c.r. results were very good. ignoring front-matter pages, the text in the body of the book contained ~5700 lines of text. i did a few standard cleanup routines on the o.c.r. file, such as closing up spacey punctuation and converting utf8 em-dashes. after that cleaning, of the ~5700 lines, over 97% were correct... specifically, about 5,555 were correct, and about 123 were not. that, in a nutshell, is the problem with the workflow at pgdp.net. d.p. has volunteers examine every line in a file, multiple times!, rather than laser-focusing on the 3% which are "problem lines". as i have shown before, and will show once again, it's rather easy to find those "problem lines". heck, most show up in spellcheck! after the cleanup, it's a breeze to turn the file into a nice e-book. -bowerbird

1 0

Re: awfully quiet -- 2555 chances at redemption
by Bowerbird＠aol.com 03 Sep '11

03 Sep '11

i said: > how the finally situation will resolve itself... mixed got i words there up some. *** rickywong said: > I know the guys at bookglutton. They aren't alone. i'm not sure what "alone" means there... > I think the term "social reading" is > quite broad and for different purpose. ok. > I'm looking more at enhanced content. then i'm not sure you picked the best examples. because i'd say it's gonna be pretty difficult to "enhance" things like "alice in wonderland" or "art of war" or the bible. or maybe not, i dunno. at any rate, i'd love to see you take your stab at it. i hope other people here give you feedback too... -bowerbird

2 1

these grapes are sweet -- lesson #01
by Bowerbird＠aol.com 03 Sep '11

03 Sep '11

ok, well, i'll be popping in and out over this 3-day weekend, and indeed for the next couple weeks, but let me now show how i can create and maintain a productive thread on this list _all_by_myself_, if need be. of course, you're invited to join, if you'd like, and can force yourself to be _constructive_, but -- lacking that -- i'm capable of carrying the thread myself... the thrust of the thread will be how to digitize a book. i've covered this many times already on this list, of course, so there will be pointers to previous posts, here and elsewhere, seasoned with updates gained from newly-won experience... so let's review the basics first. o.c.r. is pretty good nowadays, even at google and archive.org. with some elbow-grease, the right tools, and the proper focus, a person can digitize a typical book in about an hour these days. that's not a new development, either. i documented it years ago. in september of 2006, to be exact, so we're talking 5 years back. here are the posts: > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-26,1 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-27,6 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-28,5 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-29,5 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-10-02,6 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-10-03,2 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-12-22,1 i did the digitization with some tools that i programmed myself, which were unavailable to the general public, but you can achieve similar results today with "standard" apps and tools i'll give to you. and i'll show you exactly how to achieve those results... so if you want to play along, go and grab a book at archive.org. (if you like, tell me which book it is, and i will provide guidance.) -bowerbird p.s. when i was googling for those posts, i came across this: > http://www.mobileread.com/forums/archive/index.php/t-15544.html unbelievably, in january of 2008, i had to argue _against_ people to tell them that the iphone would be a device people would use to read. this on a site called "mobileread.com". i'd guess they're smarter now.

1 0

Re: precisely why i called the bluff
by Bowerbird＠aol.com 01 Sep '11

01 Sep '11

jim adcock said: > This is silly word gaming. > Anything programmatic "fixes" which > can be applied without human input can be > considered effectively part of the OCR algorithm. you know, jim, there were reasons i stopped even bothering to have discussions with you, namely that your arguments lacked real weight, and that you'd constantly flop from side to side. first you say that google's o.c.r. is improving -- ("towards 'reasonable' 100% human-free o.c.r.") -- then you say that o.c.r. will never be perfect... both those statements are true, yet lightweight, because their truth-value is quite self-evident, plus their valence points in opposite directions. and besides, they are beside the point, because -- as i noted just yesterday -- i have _proven_ here a great many times that it is quite easy to improve on the results of o.c.r. programmatically, such that the text is perfect even if the o.c.r. is not. i'd say that's precisely what google has been doing. which is why their _text_ is now close to error-free, even though their ever-improving _o.c.r._ is still not -- and never will be -- perfect. this is most assuredly _not_ "silly word gaming", and if you think that it _is,_ then there's no use in even discussing it any further... if you want to define "o.c.r." as something that happens _after_ the "optical character recognition" process, fine; but don't be so disingenuous when you have done that to say that _i_ am the one who is doing "silly word gaming". and the simple fact that i need to _remind_ you exactly what "o.c.r." stands for shows how stupid your dialog is. and here's the bottom-line: it doesn't matter one whit whether google's o.c.r. improves from this point on, because the post-o.c.r. fixes it already knows how to do will be sufficient to take their text close to perfection... > After having looked at these examples > I stand by what I said earlier about > the difficulty moving from demoware to product > which "real world customers" actually accept. where, precisely, is "the difficulty"? i see none at all. > Insulting the customers is indicative of the problem, > not the solution. i assume by "customers" you mean "volunteers", but i haven't the faintest clue what you mean by "insulting". and i sincerely doubt that any "explanation" you give will make any more sense than you have made thus far. so, jim, how about this? if someone else understands what you mean, they can explain it to me. and if nobody else understands what the heck you're talking about, but they remain curious, they can ask you to explain, to them, what you mean, so that then they can -- in turn -- explain it to me... and if nobody else knows what you're talking about, and nobody _cares_ to know, either, they just shut up, and you stop talking to me, and i stop talking to you. and if benjamin has any questions or comments for me, he and i can talk to each other, without your interruption. how does that sound, jim? -bowerbird

1 0

Re: precisely why i called the bluff
by Bowerbird＠aol.com 01 Sep '11

01 Sep '11

jim said: > As one who has actually worked professionally > on a number of different recognition system > I respectfully disagree with your prediction of > the future. No OCR system will ever be "error free." i didn't say the o.c.r. would be error-free, jim... i said their _text_ would eventually be error-free. as i have proven here, exhaustively, there are _plenty_ of post-o.c.r. fixes that are programmatically applied, first and foremost among these many measures being comparison of separate digitizations of the same book. (and google sometimes has half-a-dozen digitizations.) if i had their corpus, i could create hundreds of fixes... plus i expect google has invented a good number more of these effective techniques than i could ever dream of. > "Demos" often remain "demos" forever, because > turning "demos" into "real world products > accepted by real world users" is so danged hard. well, yes, sometimes that charge has merit... and sometimes it's nothing but a cheap shot. which is it in this case? well, jim, you might have wanted to check the subject-header here. check out the rest of this reply, to benjamin... *** benjamin said: > And would your lordship stoop to provide cut the cutesy crap, ok? :+) i've done far more than enough grunt work here to prove that i don't consider myself as anything more than a grunt. no matter how loudly i yell... the only people i am superior to here are the stupid, who refuse to learn, who stick their head in the sand. (unfortunately, there are quite a few of 'em here, but the good news is most have put me in their kill-files, so they don't know i told the truth about them again.) > And would your lordship stoop to provide > the location (as in URI) of these demos to > a relative newcomer with the best of intentions? here's the standard set of things that i point to: > http://z-m-l.com/go/myant/myantp123.html > http://z-m-l.com/go/mabie/mabiep123.html > http://z-m-l.com/go/sgfhb/sgfhbp123.html these are all page 123 from respective books, nestled in the set of all pages from each book, shown with text on one side, scan on the other. here are the master files that generated the sets: > http://z-m-l.com/go/myant/myant.zml > http://z-m-l.com/go/mabie/mabie.zml > http://z-m-l.com/go/sgfhb/sgfhb.zml each of the pages in the set contains a form at the bottom where errors can be reported... go ahead and make a sample report on a page; you'll find the report is appended to that page, and also collected on a separate page with all the reports that have been made for that book. as i said, though, error-correction is quite easy. the hard part -- or so people here seem to think -- is turning the text into a respectable e-book, but i've got that base covered quite thoroughly as well, as you see if you go to the main page at z-m-l.com and follow the link for examples. text-to-html, and the next step (.mobi and .epub) follows from there... > I always thought that that project was for > independent producers producing ebooks > on their own for PG, as opposed to the > general public proposing fixes to PG volunteers. well, that whole sentence shows you're very confused, but that's to be expected in "a relative newcomer", eh? i have suggested all kinds of methodologies, ranging from independent producers to collaborative methods (e.g., d.p.) to encouraging and using feedback from the general public. but slicing up the world that way isn't particularly productive. the tools i've created can be used by independent producers, or in collaborative workflows, or after-the-fact by the public. they can exist offline or online, and the behavior is the same. the tools don't know (or care) how the humans split the tasks. if you want more demos, i've got a ton. i'm also willing to program new ones, quite specific to exactly what you want, providing you agree to host them online for people to use... same offer goes for you, jim. note that i'm calling your bluff. -bowerbird

3 2

Re: precisely why i called the bluff
by Bowerbird＠aol.com 01 Sep '11

01 Sep '11

benjamin said: > I use “your lordship” frequently, > often to refer to persons whose > actual lordship I totally do not respect. > So I did not mean by it what you thought I meant. did you mean to put the word "not" in "do not respect"? whatever the case, i am proud to work in the trenches... and i'm proud to have you working alongside me there. i'm also proud of people who work with good intentions, even if their fear of a new workflow causes inefficiencies. but i'm still gonna try to help get them straightened out. > “Their” is plural; “head” is singular. :P / :-) there are a lot of them. each with only one head... :+) which is why it's a shame if they stick it in the sand. > I am more than merely “very” confused. I am > _totally_ confused. I am confused in the extreme. > (I even question whether I am quite conscious > of my own confusion, so confused am I.) ok, maybe you're not as confused as i thought you were. ;+) > Up until now I only knew of one tool, and that is > the one that was being discussed on this list back > in February, of which new demos were being uploaded that was just the latest demo in a series of very many... > I never knew that any such demo allowed feedback > “after-the-fact by the public.” many of the demos i have created were geared to that aim. but i've also had lots of other demos for lots of other aims. all in all, i have covered every single part of the tool-chain. -bowerbird

1 0