- gutvol-d - lists.pglaf.org

the problem with the e-books from the internet archive -- 01 of 32
by Bowerbird＠aol.com 11 May '10

11 May '10

for 32 days, i am showing samples of the problems with the text in e-books from the internet archive... *** let me begin by saying that this is guided by _love_... there is much that i admire about the internet archive. right at the top is the man at the top -- brewster kahle. he crafted, and is being guided by, the right philosophy: "universal access to knowledge". at a time when _many_ people adapt a "pragmatic" approach to our cyberlibrary, it's useful and vital to offset it with a philosophical tone. because the unique opportunity has presented itself now, creating a cyberlibrary is the _right_ thing for us to do... to _fail_ to do so would be a massive failure on our part. so i believe strongly in the mission of the internet archive. but in practice, there is much improvement to be desired... specifically, what good is a library when its text is garbled? *** and now some background... as is often the case, the problem's origin is deep in the past. archive.org took a fork in the road a long time back that has ramified to its present, in a way that is not particularly good. specifically, a decision was made to focus on _scans_ of books, a decision which made digitization of the text less important... moreover, it made digitization seem less costly than it truly is, because it ignored the costs associated with cleaning the o.c.r. but mostly, it was that the _attention_ got focused incorrectly; text was a second-class citizen, thus o.c.r. was largely ignored. in some ways, this decision was "understandable" at the time... early efforts proved that correcting o.c.r. was time-consuming; plus then creating a "book-like" version of that corrected o.c.r. was a layout task that seemed too hard for archive.org people. meanwhile, brewster was impressed by a page-flipping demo by the british library. and the scans didn't require the "layout" that seemed so difficult. so archive.org headed in that direction. as you might expect, a good many people (notably michael hart, with his firm insistence that "a picture of a book is _not_ a book") questioned this decision, and advocated on behalf of digital text. but the decision stuck to focus on scanning. as a consequence, correction of the digital text has never been a priority, a decision that has come back to haunt archive.org... end-users have found that archive.org's huge scan-sets are just too inconvenient. scan downloads are bulky, and take up space, and the absence of any searchable text is a distinct disadvantage, as is a total inability to reflow the book to differing form-factors. moreover, our shift to _mobile_ machines magnifies the need for e-books to be nimble, and archive.org scans will never be nimble. even the low-resolution scansets can run to dozens of megabytes. (and the high-res scansets are so darn big that they're ridiculous.) so, the demand is for digital text, now and into the future too. which means archive.org has been caught with its pants down, and is now paying the price for a bad decision made years ago. they find that their end-users are _demanding_ digital text, primarily in the form of .epub downloads and accessible text for blind and dyslexic users (which must be available by law), but the only digital text archive.org can provide is riddled with embarrassing o.c.r. errors making the text unusable practically. *** archive.org hasn't made _any_ efforts to clean up their o.c.r. consequently, it's quite easy to find examples of terrible o.c.r. in the archive.org library. indeed, it's so easy that i wouldn't ordinarily even bother to point out such _common_ examples. i'd say that 95% of the archive.org e-books exhibit problems in their digital text. finding e-books _without_ any problems is the difficult task here, not finding ones that have problems. indeed, the problem is so pervasive that if _i_ were making the decision to release this text -- as it is -- to the general public, i would refuse to do so... that's how bad some of this text is... but not only is archive.org _releasing_ the text, it's _promoting_ the release of its flawed text, as if it were _proud_ of the release. i don't really understand why they'd _do_ this. (it's almost as if they expect that nobody will actually even _look_ at their books, even though they're promoted with high-profile press releases.) but -- after having tried for years and years and years to bring this problem to their attention and have them do something to correct it -- i can no longer sit by without making a public fuss. and let there be no mistake about this. i am _willing_ to help archive.org correct their o.c.r. i've done lots of work on this, and i _have_ offered my assistance. i was largely ignored, so i persisted, and -- to my amazement -- i was _banned_ from their mailing listserves. talk about shooting the messenger! *** so here's today's example. for this one, i will pick a book that archive.org itself once trumpeted as an example of its work -- the adventures of tom sawyer. in keeping with its focus on its page-flipping scan-set viewer, archive.org showed one of the chapter-header pages, complete with an illustration that overlapped with the opening text there. here's where they promoted the page: > http://openlibrary.org/dev/docs/ui#anchor5 and here's the actual page: > http://www.archive.org/stream/adventuresoftoms00twaiiala#page/26 the reason (the scan of) this page was so impressive was because our current e-book formats are largely incapable of integrating text with pictures in such a sophisticated and pleasing mesh... but that intertwining of text and graphic has a deleterious effect on the o.c.r. so if you pull up the o.c.r. text from this very page, you'll see that it shows this: > morning was > > come, and all the summer world was > bright and fresh, and brimming with > life. There was a song in every heart ;. > and- if the heart was young the music > issued at the lips. There was cheer > in every face and a spring in every > step. The locust trees were in bloom > and the fragrance of the blossoms > filled the air. Cardiff Hill, beyond > the Village and above it, was green > with vegetation, and it lay just far > enough away to seem a Delectable > Land, dreamy, reposeful, and inviting. > Tom appeared on the sidewalk with > a bucket of whitewash and a long- > handled brush. He surveyed the > fence, and all gladness left him and > a deep melancholy settled down upon his spirit. Thirty yards of board fence > nine feet high. Life to him seemed hollow, and existence but a burden. Sighing. > > 26 as you can see, the chapter-header has been lost, and the word "saturday" at the beginning of the chapter too. so a person who is "reading" this o.c.r. text won't know that this is a new chapter, or that the day is a saturday. other than that, aside from a few other o.c.r. glitches, the text is ok. but that's a serious problem for a reader, not knowing that this is the start of a new chapter. and this is on a page that was actively promoted by archive.org! -bowerbird

1 0

cory's got a new book out today
by Bowerbird＠aol.com 11 May '10

11 May '10

cory's got a new book out today -- "for the win"... > http://craphound.com/ftw/download -bowerbird

1 0

Re: Internet Archive makes more than 1 milllion books available online to blind, dyslexic
by Bowerbird＠aol.com 10 May '10

10 May '10

kimo said: > Thanks, I live in SF I will spread the word in the tech community right. you do that, kimo. tell them i'm here waiting at bowerbird at aol dot com to talk to them, any time they're ready. in fact, you can tell them that, for 32 days (32=2**5) -- from may 11 through june 11 -- i'll be posting a brief example every day showing a snippet of the terrible o.c.r. that archive.org foists off on the blind. and at the end of that time, i'll package all of those examples and send 'em off to places like boing boing, slashdot, reddit, and digg, and maybe to some techblogs like techdirt, techcrunch, and so on. -bowerbird

1 0

Re: Typesetting ("gods and fighting men")
by Bowerbird＠aol.com 10 May '10

10 May '10

jim said: > if you guys ever start listening to me. listening? to you? oh jim, don't be naive. if you really wanted people to _listen_ to you, the first thing you'd have to do is start making sense. and stop saying things that are demonstrably false. but even then, nobody would probably listen... that's just not something that people _do_ here. but at least they might start _reading_ your posts a bit less infrequently, and stop shaking their heads when they _do_ read your posts, which is a good start. -bowerbird

2 1

Re: Missing apostrophes in the generated HTML / ePub versions of Madame Bovary
by Bowerbird＠aol.com 09 May '10

09 May '10

andrew said: > Well, everyone is entitled to their own opinions. > My own opinion is still that > the way microsoft introduced > and have used their character sets > has created unnecessary difficulty > for many people. that's funny, because i have the same opinion about the way the unicode process has gone... but at some point, we need to just let it all go. > I'm sorry if you perceive that as making an excuse. well, yeah, if you're blaming microsoft (on something that they did a few decades back) for a mistake that was made much later, yeah, then i "perceive" that as making an excuse, yes. moreover, i "perceive" it as laughably abysmal... why not just say, "oh, hey, the encoding was wrong", fix it, and move on? why the need to point a finger? then -- as indicated in my previous message -- i would go on to _improve_the_workflow_ so that such problems wouldn't crop up again in the future. so you either make your converter impervious to the incorrect encoding declarations in the library, or you scrub the library to remove the incorrect declarations. i would probably do both, just to be on the safe side... what i would _not_ do is point 30 years into the past to try to deflect responsibility from where it belongs. -bowerbird

1 0

happy mother's day!
by Bowerbird＠aol.com 09 May '10

09 May '10

moms are special. -bowerbird

1 0

Missing apostrophes in the generated HTML / ePub versions of Madame Bovary
by Joaquin Cuenca Abela 09 May '10

09 May '10

Hi, the ebook http://www.gutenberg.org/etext/14155 is missing all the apostrophes in the generated versions (at least HTML and ePub). The two hand-crafted files (plain text and rtf) contain the apostrophes, for instance one of the very first lines in the plain text file is: Permettez-moi d’inscrire This has been converted in the HTML version to: <p id="id00015">Permettez-moi dinscrire Is this due to a bug in the epub-maker used to convert the file, or is there something buggy in the original text? Cheers, -- Joaquin Cuenca Abela

3 5

Re: Missing apostrophes in the generated HTML / ePub versions of Madame Bovary
by Bowerbird＠aol.com 09 May '10

09 May '10

andrew said: > If you want some history, basically you can blame microsoft. what kind of "history" is that? c'mon, let's go _back_. let's blame the serpent in the garden of eden who tempted eve to take a bite of the forbidden fruit... get real. and stop making excuses. now that marcello knows that the encoding declaration can't be trusted, he can _improve_ his converter script by actually doing some reality-checks on the text itself, to make sure it actually is the encoding it claims to be. and apostrophes would be an _excellent_ way to start... or better yet, run such a checker-program against the entire library, and ferret out the incorrect declarations, so the converter script _can_ know the encoding for sure. -bowerbird

2 1

Re: one million units moved in 28 days
by Bowerbird＠aol.com 08 May '10

08 May '10

> projecting Kindle to be at 7 Million "projections" of future numbers are pure bunk. even estimates of the current number are suspect, since amazon resolutely refuses to announce it, but at least we have auxiliary measures that will provide some kind of realistic bounding restraints, one of the most important being e-book _sales_... in this regard, the best observer to pay attention to is david moynihan -- over at blackmask/munsey's -- who has been paying very close attention for years... david was the one who rang the bell when the kindle started moving substantial units. (and pooh-poohed everything else as mere hype and marketing bullcrap.) he has his finger on the pulse; he's the only one i trust. of course, i'm interested in the ipad _regardless_ of the numbers it rings up, because it presents opportunities in a number of new arenas, so i think it will springboard a lot of creativity that the kindle was too crude to free up. current estimates of the kindle weigh in at about 3 million. i don't see them selling more in the next 5 months than they have sold up to this point in time. but i can see them selling a lot of units nonetheless. and i can definitely say that i absolutely love the t.v. commercials they're running. one of 'em has a tune that knocks my socks off every time. i love it. it makes me feel _joy_. i replay it over and over. > in the early innings of ebook reader development. oh please. we've had worthwhile "e-book readers" for a decade, in the realm of both hardware and software. so these are hardly "early innings". that was long ago. and the development of a mobile computer has been a widespread vision ever since the early days of star trek. yes, things _will_ move a lot faster now, thanks to apple showing the world the direction in which it should move. and we'll be amazed by how far we go in the next 5 years. but let's not pretend we couldn't have taken these steps 5 years earlier, if the hardware businesses didn't have a policy of rolling out innovation as _slowly_ as they can, and soak up as much profit as possible from each step... previous tablets were crude, now that we've seen apple do them _correctly_. but if those crude tablets would've cost $500, instead of $1500, people woulda bought 'em, and the world could have started moving forward earlier. but those hardware guys are greedy greedy greedy boys... > I'm wondering why we don't see them more often. . . ? it's purely a coincidence thing. i noticed i didn't see many iphones until their number got to about 5 million or so. and it was only when it got up to 15 million that i started seeing them on a "regular" basis... but kindles _are_ out there... my girlfriend belongs to a women's philanthropic education organization, and there are three women in her chapter (of 45) who own a kindle. -bowerbird

10 32

Re: Typesetting ("gods and fighting men")
by Bowerbird＠aol.com 08 May '10

08 May '10

jim- your rewrapping of the p.g. e-text to the original linebreaks was pretty good, but i should have specified more carefully... it would've been best to retain the _end-line_hyphenates_ too (or else justification is unworkable), as well as _pagebreaks_ (because one objective is to compare the text with the scans.) i'll handle this one from here... but note that for next time... :+) -bowerbird

2 1