- gutvol-d - lists.pglaf.org

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 03 Jun '06

03 Jun '06

michael said: > Perhaps the way to think about this is to consider > just how many more or less readers we would get if > the file sizes were that much larger or smaller. there are something like 100,000 books available at google. d.p. digitizes about 2,000 books a year. they can't keep up. > In the end, I think we should provide both. in the end, users will turn exclusively to "digital reprints" -- digital text that mimics the scans so accurately that there's really no good reason to consult the scans at all. after 10 or 20 years of nobody downloading the scans, we'll be able to feel comfortable taking them offline... > Some operations deliberately do not put their high > resolution scans online for downloading, rather an > automated process reduces the resolution, so these > scans are no longer suitable for OCRing. yeah, that's sad. but what are you gonna do about it? > The odds of being able to create a complete eBook, > using those scans that are usually made available, > perhaps about 1/4 to 1/3, based on the reports you > have probably already seen. yeah, that's sad too. but that's a quality-control issue that i suspect the scanning operations will solve soon... > Once you go through the effort of scanning missing > pages, rescanning the pages that did not work with > your OCR programs, etc., it often might seem worth > the effort simply to scan the entire book with the > higher resolution scans that you can then post for > others to use. i don't think -- for most books -- that will be the case. but perhaps that's because i don't see much use for high-resolution scans. i am _not_ in love with scans. like i said above, they will eventually be left behind. the important point _today_, though, is that we have a shitload of scan-sets, more than we can process now, and it's silly to ignore them when we _could_ offer them for people to _read_ now, even if they aren't digitized... > Do raw scans qualify as eBooks? does it matter? they are what they are. no more, no less. and almost everyone sees them for exactly what they are. > This is the "quick and dirty approach" and doesn't > cost much in terms of time, effort or money um, scanning does indeed take time, effort, and money, at least if you're doing it on a scale of millions of books... > I suppose the real question comes down to > purposes for making eBooks. i'm not sure of that. we make e-books for people to read, and so their text can be searched and easily repurposed... scans get us part of the way. digital text gets us the rest... > The various university projects still seem to be a > great deal concerned with keep their eBooks out of > the hands of the public, as has Google, though the > Google philosophy may be in the process of change. the michigan librarian pledged that all public-domain books scanned from their library will be made available to the public. i assume he meant the scan-sets. but from them, we will soon be able to automatically get digital text, so there's no difference. > Right now it's hard to tell what Google has chosen > as their goal; will they really try to do millions > of books in the next 54 months after perhaps stats > of .1 million in the first 18 months? they most certainly will. > Will Google change their philosophy per downloading scans, if we open up negotiations with them, _maybe_. we can hope. > and or downloading their full text searching database? they'll never make their text-database public, as that's the competitive edge for which they are paying many millions... do you really think they're gonna hand it over to microsoft? > Until Google decides to actually proofread eBooks, if you mean "ensure that their digital text is highly accurate" -- which can be completely orthogonal to "proofreading" -- then you can be certain that they will "decide" to take that step. inaccurate text gives bad search results; google won't tolerate that. > My own goal has always been for the public to have their own > home eLibraries, just as they have their own home computers. that's the goal for a lot of us. -bowerbird

5 5

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 03 Jun '06

03 Jun '06

maitri said: > Page scans are not eBooks. They are not universally searchable, > readable and editable, and have full ability to become proprietary, > i.e. owned or copy-protected. thanks for joining the thread. where ya been? -bowerbird

1 0

Re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries (fwd)
by Michael Hart 02 Jun '06

02 Jun '06

>From one of my French friends much more familiar with Gallica. On Thu, Jun 01, 2006 at 09:16:03AM -0700, Michael Hart wrote: > Each .pdf file seemed to just hold a .gif file. . .or is there > something else going on there that was missed? Gallica just has _pictures_. It is hardly more than a scanning bank. They can either serve them as PDF or TIFF files. When they did the word to have a TXT file (which they did for about 1% of their books), they give that too. (ex: _L'=EEle des pingouins_, both on Gallica as text and on PG). > And this is supposed to prepare the book as a single .pdf file? Yes. > Searchable? No, just a bunch of pictures. The document is just like a document with a picture of a different painting, or a photograph of a different landscape, on each page. It's like HTML: HTML can have text (-> searchable) or display a sequence of pictures (-> non searchable, even if they are pictures of pages with text). PDF is more confusing because the layout depends less on the viewer than with HTML (one can define custom colors/sizes/margins in HTML with CSS and the like; not so with PDF). Make the experiment with the ZIP file I pointed to. It contains a PDF file, small & light & searchable. The PDF file produced by Gallica (take the example given by the other person) is heavy and not searchable. I did the tests with xpdf but it should be the same with Acrobat Reader. To know whether a PDF file is text or a picture, I'm not sure what to do. Here are hints: . pdftotext will only work with text-PDF . searching too . if the letters look dirty, with noise, or the lines are not quite horizontal, it is a picture.

2 1

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 02 Jun '06

02 Jun '06

michael said: > We work with all possible sources to get eBooks. all of your sources, combined, won't be able to keep up. not until the digitization becomes nearly-automatic. which, as i've said, is not that far down the line anyway. > I seem to get plenty of messages from scholarly types > who think source scans will always be in high demands, > at the ivory tower level, at least. i'm not sure they know what they want. in fact, i'm almost sure that they don't... > after 10-20 years the actual hardware requirements will > appear so drastically reduced that the load will be nil. it won't be the resources required (or not required) that makes us take down the scans, it will be the lack of demand. digital reprints will do the same job, better, with fewer resources. > Once you provide a better alternative, you force those > who should have done it originally to do it better too. if you can scare up the $250-million budget, please do. as i've said elsewhere, that's what we spend in _two_days_ on the war in iraq, so you wouldn't _think_ it's all that hard to find the same amount for such a culturally important task. but i don't see anyone except google stepping up to the plate. > I was under the impression that much of this low-quality > was intentional, so I don't think those will be improving, > at least until someone provides a better mousetrap. that's exactly why i asked "what can we do about it?" > All depends on how much effort it is for the particular person > in question. . .if it's a lot of effort to get the materials, > but low effort to do the scanning, you may as well replace the > entire file with your better examples of what should be done. i agree. and in the cases where we can't use google's scans, that's what we'll have to do. let's just hope, though, that that won't be the case for the bulk of those 10-million unique titles. > 1. Makes for better OCR i'm rooting for better o.c.r. i sincerely hope that it happens, and i suspect the abbyy folks still have tricks up their sleeves. and, just to remind everybody here, they have _already_ made a version of their software that's specially-adapted for old books, a version that nobody here, to my knowledge, has even _tried_, so y'all will need to do some convincing in order to convince me that you're really as concerned with the o.c.r. thing as you claim. but as for me, i'm not counting on the o.c.r. much at all; i'll take what is currently available in regard to o.c.r. tech. my aim is to jack up the post-o.c.r. correction routines, using a wide array of automagic. > 2. The scholarly types, as above. let 'em use their scholar dollars to create whatever they need. i can't be bothered with their esotericism. i just love the books. > Yes, and we should. well, i'm glad i finally got _that_ tooth pulled! ;+) > It matters to the integrity of the eBook world. my integrity does not turn on semantics. > _I_ have no intention of quitting > until I can give away a million books, that's what i love about you, big boy, your dedication. > It will be interesting to see who can put a million eBooks > online first, and how good they are. i agree. > Yep. . .scans are just one step, I say it's the easiest. depends on how many you do. > I can only hope he meant something more worthwhile > to the masses than what most of the current scan-sets provide > and that he will be able to find some way to keep the ball rolling. i'll be happy to help him out, just like i'm happy to help you out. > We'll see, and I am taking bets. pizza. loser buys in the winner's city. you can fly out to santa monica and spend some time with me sometime when it's wintry cold there in illinois. > I think we should work as though it all depends on us, > and hope that Google will get somewhere. you're not the best person to do that negotiation anyway. > They claim all those million are spent on scaning, not OCR. it is. and that's why it will take a _negotiation_ to get them to release the public-domain scans. they won't do it "just because". but i think there _are_ some things we can offer in negotiation. one would be the quality-control that we're willing to do for 'em. although i think they'll realize soon they need to do this themselves, at the time when they've still got the book right there by the scanner, it never hurts to have another entity take a look at your work later... another would be an offer to serve as their "reading room", which would mean we'd dish the pages to people for reading, so google could instead concentrate completely on being "the search engine". (this might mean we'd have to agree not to furnish our own search capability, but as long as their engine is nicely integrated into our presentation regime, i don't think that would be a problem at all; many websites use google as their search-engine even at present.) and perhaps most importantly, what we could offer is huge help in the form of friend-of-the-court briefs that would be supportive of google's scanning project in facing their various legal challenges. public opinion will be very important when this comes to judgment, and a good-faith effort like turning their public-domain scans loose could go a _long_ way in drumming up public support for their work. on the other hand, a selfish attitude on google's part would make 'em look bad, and that appearance could be quite devastating to their case. i assume all of these points are reasonably apparent to google already, so the "negotiation" wouldn't have to be antagonistic in nature. indeed, it might be very short and very sweet, and we could find ourselves with 100,000 scan-sets on our machines before we knew it. that possibility sounds too good to me to pass up without giving it serious consideration. > Actually, you have it backwards there. . .think about it. . . . > Google's monster speciality is SEARCH ENGINES!!! > They are MUCH more interested in writing a search engine that will > read fuzzy OCR text than in increasing the accuracy of the text. if you can search fuzzy text, you can correct fuzzy text. that's the point. if google lets its text remain fuzzy, it will be because they _decided_to_. and there are a couple reasons why they might well decide to do that, but i'd rather not take a chance of making them real by discussing them. still, as i've said, i myself will show the world how to correct fuzzy text, within 5 years, assuming that abbyy hasn't already solved the problem. -bowerbird

1 0

roger frank, breakout programming star over at distributed proofreaders
by Bowerbird＠aol.com 01 Jun '06

01 Jun '06

recently i mentioned 3 programmers over at d.p. lately, roger frank is doing some excellent work too. some involves the task of creating a project-specific dictionary, which is a very powerful tool d.p. has mostly ignored up to now. another excellent arena roger is working on involves the flagging of suspicious words. a brief overview is at: > http://pgdp.rfrank.net/ruby/dp-view.html text-to-html conversion routines are another thing that roger has been working on. all of these ideas are _fine_. -bowerbird

1 0

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 01 Jun '06

01 Jun '06

yahoo has enough money to generate press any time they want. but you have to take that money out of your pocket to do it... so the answer is that they haven't wanted it enough. indeed, the only contribution to o.c.a. that i've seen acknowledged is a $5-million one from microsoft. which is pretty much peanuts, coming from a company as rich as microsoft, and we all know it... i get the impression everyone at o.c.a. -- except maybe brewster -- is trying to go the cheap route. might work that way, or might not, but o.c.a. certainly ain't gonna get much publicity without buyin' it. -bowerbird

6 8

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 01 Jun '06

01 Jun '06

michael said: > Please stop quoting from what I said about illustrations > when bandwidth was a serious issue. . .not that everyone > has broadband these days. . .or wants pictures; however, > the point was made long ago when making an eBook larger, > often many times larger, would stop people from reading. i too have changed my position on this only recently, when the penetration of broadband in homes in the u.s. passed over 50%. but like you, i am still cognizant that not everyone has broadband, and that those who don't are on the poor side of the digital divide and thus must be given priority in our thinking. since this has been a cornerstone of my thinking all along, i'm sure it will continue to be. i know -- personally -- many people with hand-me-down machinery, far too many for me to have this vital issue slip from my radar-screen... -bowerbird p.s. i myself just moved to an ibook g4 with o.s.x. a little over a year ago...

1 0

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 01 Jun '06

01 Jun '06

michael said: > as I have had multiple comments from our readers that they > LIKE not having such boringly justified right margination, > as it helps them better keep track of what line is next. and the beauty of a good viewer-app is it lets each user decide. > As for the footnotes, I still agree with those who want an > appendix containing all of them, rather than having breaks > between pages contain them. it doesn't have to be "either/or" with e-books, it can be "both". > I like this with paper books, and even more with eBooks, as it is > trivial to switch from the text to the footnote and back again. i think it's even better to have both displayed at the same time; no switching required. > I'll leave the pagination and margination issues to reader > choice as their own personal decisions, along with fonts. i agree. -bowerbird

1 0

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 01 Jun '06

01 Jun '06

sebastien said: > Most of the time the original typesetting does not matter much. different people can disagree on that. > I believe you are missing the point. > Michael doesn't care as much about collections of pictures > as he does about digitalized text. different people disagree with michael. > As long as scans and/or OCR technologies are so disappointing, > we'll have to rely on higher-level humain brains with initiatives > such as PGDP or ebooksgratuits.com or methodologies which are better. > Of course having easy access to pictures is useful and > much better than nothing and serves you well, but > that's not what PG and ebooks are about. different people can disagree on that too. > ebooks are much more than photographs of regular analog books. yes, but photographs of regular analog books _might_ qualify as e-books, for _some_ people. different people can disagree on that too. > 3. is the top we are heading for. 2. is just a step on the way. but #2 might serve the needs of person x just fine. > I did that and got > 20845628 bytes for 604 pages. scans are resource hogs. nobody disagrees about that. one argument is that since these resources are now plentiful, it doesn't matter that scans are resource hogs. different people can disagree on that too. as long as we can easily move scan-sets to digitized text, i don't see much purpose in continuing to debate these two as if they were competitors. they're not. they're complimentary. -bowerbird

3 3

re: !@!Re: [gutvol-d] Kevin Kelly in NYT on future of digital libraries
by Bowerbird＠aol.com 01 Jun '06

01 Jun '06

karl said: > Sure, these ASCII files are also useful for special purposes, > but telling us again and again that's the best solution > for all books and all times, is highly arguable. to my mind, the only problem with the ascii files is the absence of book typography -- bold headings, justified lines, bottom-balanced pages, pagination!, properly rendered footnotes, all the looks-nice stuff, leading to a display that is so boring it becomes tedium. fortunately, i've found with my z.m.l. that an intelligent viewer-program can figure out all of that stuff itself, given a reasonable amount of easily-done "hinting", and apply it, which removes this "only problem" and makes the ascii files "the best solution" for most books, most of the time. -bowerbird

2 1