May 2010 - gutvol-d - lists.pglaf.org

Re: Typesetting ("gods and fighting men")
by Bowerbird＠aol.com 15 May '10

15 May '10

jim said: > my understanding was that you were trying to work from > the PG txt files – which do not retain the original hyphenations. right. so those end-of-line hyphenates need to be restored. > Recovering original hyphenations should be in theory possible > too, but not work that I have looked at yet. "in theory"? you just take them from the o.c.r. at archive.org, which is what you used to restore the linebreaks anyway, right? > The linebreak recovery algorithm I worked on was intended > to allow people at DP, for example, if they want to, to resubmit > some of the early PG works and run them through DP again. ok, fair enough. but in that case, your routines should have done what d.p. requires of its proofers, which is to move the second part of the end-of-line hyphenate to the previous line. i _believe_ that in some cases, you moved the first part down... (but i could be wrong on that, so do please let me know if i am.) > Without automatic recovery of linebreaks one has several days > of extremely tedious work reintroducing the original linebreaks. as i said above, i use the o.c.r. text from archive.org to restore them. it's pretty straightforward, and automatic. it didn't take long to code, and it runs very fast. but you're right; doing it manually is painful... > The other alternative for you is to leave healthy right margins > and leave your PDF’s “ragged right” [*very* ragged right!] well, one object is to clone the pages of the printed book itself, so ragged-right isn't really an option. -bowerbird

2 1

Re: spill baby spill
by Bowerbird＠aol.com 14 May '10

14 May '10

none of the "containment" methods are working. and gobs of goo are starting to come up on-shore. and now we learn that the gusher is releasing some _ten_times_ as much oil as we were told previously. with no end in sight, heaven help us... -bowerbird

1 0

the problem with the e-books from the internet archive -- 04 of 32
by Bowerbird＠aol.com 14 May '10

14 May '10

for 32 days, i am showing samples of the problems with the text in e-books from the internet archive... *** today's sample is "alice's adventures in wonderland", the book that -- more than any other -- captured the imagination of people, from martin gardner on down. in the e-book arena, "alice" has been "the prototype", in that it's the overwhelming choice to use in demos. it's the text which every new system cuts its teeth on. one warning: don't use any archive.org o.c.r. version! here's the page-scan: > http://www.archive.org/stream/alicesadventur00carr#page/20 here's the o.c.r. for the full book: > http://ia311325.us.archive.org/3/items/alicesadventur00carr/alicesadventur0… and here's the o.c.r. for that page (which is page 20, not "n35"): > > 20 THE POOL > > and four times six is thirteen, and four times > seven is â€” oh dear! I shall never get to twenty > at that rate ! However, the Multiplication Table > doesn't signify : let's try Geography. London is > the capital of Paris, and Pciris is the capital of > Eome, and Eome â€” no, that 's all wrong, I'm > certain ! I must have been changed for Mabel ! > I'll try and say ' Hoiv doth the little â€” '" and she > crossed her hands on her lap as if she were > saying lessons, and began to repeat it, but her > voice sounded hoarse and strange, and the words > did not come the same as they used to do : â€” > > " Hoi'j doth the little crocodile > Improve Iris shining tail, > And 2^our the v:aters of the Site > On every golden sccde ! > > " Hoir cheerfully he seems to grin, > Hoio neatly spreads his elavjs, > And welcomes little fishes in > With gently smiling jaws/" gosh, what a litany of errors and glitches on just this one page. first, that funny character after "second is" in the second line is an em-dash. it's a utf8 em-dash. that's right. to "solve" the problem of the em-dashes that their workflow had misplaced, they stick a _utf8_ em-dash into what is otherwise 7-bit ascii... i think that's _stupid_, but i guess it's what they call "progress". but let's go on. numerous examples of "floating" punctuation dot the page, which is not all that unusual for o.c.r., and i note that it _can_ be corrected automatically, except it _hasn't_ been. we also have misrecognitions on "paris" and "rome" (twice)... moreover, the _italics_ were lost, both in the text for emphasis, and on the poem. the _centering_ of the poem was lost as well. and there were more misrecognitions on the italicized text, on "how" (4), "his", "pour", "waters", "nile", "scale", and "claws", plus an exclamation-point that was misrecognized as a slash. all in all, a pretty dismal performance, all on this single page... a classic book like this deserves much better... -bowerbird

1 0

the problem with the e-books from the internet archive -- 03 of 32
by Bowerbird＠aol.com 13 May '10

13 May '10

for 32 days, i am showing samples of the problems with the text in e-books from the internet archive... *** today's example is from "books and culture", the book google offered as its very first public-domain scan-set. here's the pagescan for page 14: > http://www.archive.org/stream/booksandculture00mabirich#page/14 here's the o.c.r. for the book: > http://ia341039.us.archive.org/2/items/booksandculture00mabirich/booksandcu… and here's the o.c.r. for page 14: > > Material and Method. > > the wisest of modern readers has said > that the most important character- > istio of the real critic the man who > penetrates the secret of a work of > art is the ability to admire greatly; > and there is but a short step between > admiration and love. And as if to > emphasise the value of a quality so > rare among critics, the same wise > reader, who was also the greatest > writer of modern times, says also > that fc where keen perception unites > with good will and love, it gets at > the heart of man and the world ; > nay, it may hope to reach the high > est goal of all." To get at the heart > of that knowledge, life, and beauty > which are stored in books is surely > one way of reaching the highest > goal. > > That goal, in Goethe s thought, > was the complete development of > 14 if you look at that text, you'll see a few o.c.r. errors... specifically, "charateristio" instead of "characteristic", and a double-quotemark that's misrecognized as "fc". the bigger problems, however, are what you do not see. comparing the text against its scan, you'll discover that one end-line-hyphenate -- "highest' -- lost its hyphen. also notice that the apostrophe in "goethe's" was lost... both of these problems happened frequently in this book. they're serious flaws, to be sure, but they lend themselves to fixes that can be applied more-or-less automatically... although that fact gives little solace once we consider the offsetting fact that the auto-fixes have not been applied. but there's yet another problem here, even more serious. as you see, em-dashes were lost in the o.c.r. of this page, review of other pages shows this problem is _book-wide_, and a look at other books reveals it's a common problem. although it might not seem like that big of a deal, the loss of em-dashes can make some passages weird and illogical. as a solution, i programmed a tool that will let you easily insert em-dashes into a text. it works fine, but the job is still time-consuming. indeed, i found it much quicker to simply re-do the o.c.r., a route that's clearly unacceptable. the problem got more interesting, however, when i learned -- after considerable sleuthing -- that this is a problem in the _workflow_ of the internet archive, not the o.c.r. per se. the o.c.r. records em-dashes, but somewhere along the line, due to some bad coding, they are being lost from the file... in other words, the employees at archive.org screwed it up. this is human error, from people being paid to know better. this problem is especially galling to me, because i thought -- after i'd gone through some severe agitation to get it -- i had extracted a promise from the archive.org people that they would go back and _fix_ this glitch on all the old files. but as can be seen clearly, they reneged on their promise... i guess they thought that once they banned me from their listserve, they could forget the promises they made to me. but i can't figure out why they don't care about their books. -bowerbird

1 0

Re: the problem with the e-books from the internet archive -- 02 of 32
by Bowerbird＠aol.com 12 May '10

12 May '10

if some of those stimulus funds are used to _clean_ the o.c.r., that would be an extremely excellent use of them, to be sure... if they're used to scan more books and create more crap o.c.r., however, that would be sad. to put this in perspective, it takes about _one_hour_ per book to clean up the o.c.r. and turn that digital text into an e-book. just one hour. (people from distributed proofreaders will try and tell you it takes longer than that; that's because they're doing it wrong.) but "just one hour" for a million books takes a million hours. so you need to _budget_ for that. brewster worked himself into a hole when he promised that he could digitize a book for $30 (roughly 10 cents per page): > http://www.opencontentalliance.org/2009/03/22/economics-of-book-digitizatio… you can _scan_ it and handle the other associated overhead, but you need an hour of crafty human labor _after_ scanning in order to make the digital text worthy of human exposure... -bowerbird

1 0

the problem with the e-books from the internet archive -- 02 of 32
by Bowerbird＠aol.com 12 May '10

12 May '10

for 32 days, i am showing samples of the problems with the text in e-books from the internet archive... *** once again, i love internet archive and its mission. and when someone you love has a booger hanging from their nose, you _tell_ them that's the situation, so they won't go embarrassing themselves publicly. that's what i'm doing here... *** today's example comes from "pride and prejudice", which i'm sure you know is one of the most popular downloads of all time for public-domain e-books... here's the pagescan: > http://www.archive.org/stream/prideprejudicewi00austuoft#page/34 and here's the o.c.r. from that page: > > 34 PRIDE AND PREJUDICE > > *Ur.>' Iv,' replied Darcj", (o whom tliis remark was chiefly > > addrcucu. [;.cic is meanness it " â€¢' - arts whicii ladies sometimes > condescend to rturl,-'. (.-r I.- Wlutrvrt Ivars atfimiy > > (O cunning is i > > Mitj Binglc)- wii no( lo entirely utist;ed with this reply as to > continue the suhjca. > > ' ' â€¢ â€¢ â– â– - i th.cm again only to uy that her sivter was > > ,, . : could niH lca\c l-.cr. Binglc)' urged Mr. Jones's > > being sent for immediately; uh.ilc hit utters, convinced that ik> > country adncc could be of any service, recommended an express > to f ' one of the most eminent physiciaru. Thii the would > > :â€¢ â– ' " â€¢ -'x was not so unwilling to comply with their > > , and It was settled that Mr. lones should l-'c > tent for eaily in the morning, if Miss Bennet u ere not decidedly > bcncr. Bingley was quite uixomfortable; his sisten declared > that the)- were misaable. The)- solaced their uteti " > howc\a, by duets after suppa, while he could find i.. . ^.vt > rcUefto his feelings than by giving his houickccjxrr direaioni that > c\cry possible attention might be paid to the tick lady and > her sisiCT. no comment necessary, as that o.c.r. speaks for itself. -bowerbird

2 1

in other news
by Bowerbird＠aol.com 11 May '10

11 May '10

now in other news, rfrank's "fadedpage" proofing site has taken a big step forward, transitioning to a "match-the-page" mode. what's funny is that he thinks he's in "uncharted territory" now. um, no, roger, i've already mapped that land quite extensively. but hey man, if you wanna learn it all by yourself, be my guest. roger is also doing a good job of listening to his users, who keep asking him to provide the types of functionality that i demonstrated with my proofing site. if he only just keeps responding to their requests, sooner or later his site will be working very similar to mine. convergence can be beautiful. one down-side is that roger went to a database operation... that's sad, because when you box your text into a database, it's not nearly as open and convenient as when it's on the web. which means that other people cannot build utility around it... what makes this doubly sad is that it seems that he went to the database to solve the problems he was having with sync, like replacing the text of one page with that of another page. the thing is, synch problems are typical of new e-book code, particularly when it is being written by a new e-book coder, and it really has _nothing_ to do with the underlying storage, whether that be in a database or in your ordinary file-system. so he didn't really _need_ to switch to a database operation... oh well; as long as it's only one step back for two steps forward, he'll keep making progress. oh, and by the way, even at the very start of "match-the-page", the proofers seem to like it a lot. and that will grow over time. -bowerbird

1 0

the problem with the e-books from the internet archive -- 01 of 32
by Bowerbird＠aol.com 11 May '10

11 May '10

for 32 days, i am showing samples of the problems with the text in e-books from the internet archive... *** let me begin by saying that this is guided by _love_... there is much that i admire about the internet archive. right at the top is the man at the top -- brewster kahle. he crafted, and is being guided by, the right philosophy: "universal access to knowledge". at a time when _many_ people adapt a "pragmatic" approach to our cyberlibrary, it's useful and vital to offset it with a philosophical tone. because the unique opportunity has presented itself now, creating a cyberlibrary is the _right_ thing for us to do... to _fail_ to do so would be a massive failure on our part. so i believe strongly in the mission of the internet archive. but in practice, there is much improvement to be desired... specifically, what good is a library when its text is garbled? *** and now some background... as is often the case, the problem's origin is deep in the past. archive.org took a fork in the road a long time back that has ramified to its present, in a way that is not particularly good. specifically, a decision was made to focus on _scans_ of books, a decision which made digitization of the text less important... moreover, it made digitization seem less costly than it truly is, because it ignored the costs associated with cleaning the o.c.r. but mostly, it was that the _attention_ got focused incorrectly; text was a second-class citizen, thus o.c.r. was largely ignored. in some ways, this decision was "understandable" at the time... early efforts proved that correcting o.c.r. was time-consuming; plus then creating a "book-like" version of that corrected o.c.r. was a layout task that seemed too hard for archive.org people. meanwhile, brewster was impressed by a page-flipping demo by the british library. and the scans didn't require the "layout" that seemed so difficult. so archive.org headed in that direction. as you might expect, a good many people (notably michael hart, with his firm insistence that "a picture of a book is _not_ a book") questioned this decision, and advocated on behalf of digital text. but the decision stuck to focus on scanning. as a consequence, correction of the digital text has never been a priority, a decision that has come back to haunt archive.org... end-users have found that archive.org's huge scan-sets are just too inconvenient. scan downloads are bulky, and take up space, and the absence of any searchable text is a distinct disadvantage, as is a total inability to reflow the book to differing form-factors. moreover, our shift to _mobile_ machines magnifies the need for e-books to be nimble, and archive.org scans will never be nimble. even the low-resolution scansets can run to dozens of megabytes. (and the high-res scansets are so darn big that they're ridiculous.) so, the demand is for digital text, now and into the future too. which means archive.org has been caught with its pants down, and is now paying the price for a bad decision made years ago. they find that their end-users are _demanding_ digital text, primarily in the form of .epub downloads and accessible text for blind and dyslexic users (which must be available by law), but the only digital text archive.org can provide is riddled with embarrassing o.c.r. errors making the text unusable practically. *** archive.org hasn't made _any_ efforts to clean up their o.c.r. consequently, it's quite easy to find examples of terrible o.c.r. in the archive.org library. indeed, it's so easy that i wouldn't ordinarily even bother to point out such _common_ examples. i'd say that 95% of the archive.org e-books exhibit problems in their digital text. finding e-books _without_ any problems is the difficult task here, not finding ones that have problems. indeed, the problem is so pervasive that if _i_ were making the decision to release this text -- as it is -- to the general public, i would refuse to do so... that's how bad some of this text is... but not only is archive.org _releasing_ the text, it's _promoting_ the release of its flawed text, as if it were _proud_ of the release. i don't really understand why they'd _do_ this. (it's almost as if they expect that nobody will actually even _look_ at their books, even though they're promoted with high-profile press releases.) but -- after having tried for years and years and years to bring this problem to their attention and have them do something to correct it -- i can no longer sit by without making a public fuss. and let there be no mistake about this. i am _willing_ to help archive.org correct their o.c.r. i've done lots of work on this, and i _have_ offered my assistance. i was largely ignored, so i persisted, and -- to my amazement -- i was _banned_ from their mailing listserves. talk about shooting the messenger! *** so here's today's example. for this one, i will pick a book that archive.org itself once trumpeted as an example of its work -- the adventures of tom sawyer. in keeping with its focus on its page-flipping scan-set viewer, archive.org showed one of the chapter-header pages, complete with an illustration that overlapped with the opening text there. here's where they promoted the page: > http://openlibrary.org/dev/docs/ui#anchor5 and here's the actual page: > http://www.archive.org/stream/adventuresoftoms00twaiiala#page/26 the reason (the scan of) this page was so impressive was because our current e-book formats are largely incapable of integrating text with pictures in such a sophisticated and pleasing mesh... but that intertwining of text and graphic has a deleterious effect on the o.c.r. so if you pull up the o.c.r. text from this very page, you'll see that it shows this: > morning was > > come, and all the summer world was > bright and fresh, and brimming with > life. There was a song in every heart ;. > and- if the heart was young the music > issued at the lips. There was cheer > in every face and a spring in every > step. The locust trees were in bloom > and the fragrance of the blossoms > filled the air. Cardiff Hill, beyond > the Village and above it, was green > with vegetation, and it lay just far > enough away to seem a Delectable > Land, dreamy, reposeful, and inviting. > Tom appeared on the sidewalk with > a bucket of whitewash and a long- > handled brush. He surveyed the > fence, and all gladness left him and > a deep melancholy settled down upon his spirit. Thirty yards of board fence > nine feet high. Life to him seemed hollow, and existence but a burden. Sighing. > > 26 as you can see, the chapter-header has been lost, and the word "saturday" at the beginning of the chapter too. so a person who is "reading" this o.c.r. text won't know that this is a new chapter, or that the day is a saturday. other than that, aside from a few other o.c.r. glitches, the text is ok. but that's a serious problem for a reader, not knowing that this is the start of a new chapter. and this is on a page that was actively promoted by archive.org! -bowerbird

1 0

cory's got a new book out today
by Bowerbird＠aol.com 11 May '10

11 May '10

cory's got a new book out today -- "for the win"... > http://craphound.com/ftw/download -bowerbird

1 0

Re: Internet Archive makes more than 1 milllion books available online to blind, dyslexic
by Bowerbird＠aol.com 10 May '10

10 May '10

kimo said: > Thanks, I live in SF I will spread the word in the tech community right. you do that, kimo. tell them i'm here waiting at bowerbird at aol dot com to talk to them, any time they're ready. in fact, you can tell them that, for 32 days (32=2**5) -- from may 11 through june 11 -- i'll be posting a brief example every day showing a snippet of the terrible o.c.r. that archive.org foists off on the blind. and at the end of that time, i'll package all of those examples and send 'em off to places like boing boing, slashdot, reddit, and digg, and maybe to some techblogs like techdirt, techcrunch, and so on. -bowerbird

1 0