July 2005 - gutvol-d - lists.pglaf.org

[gutvol-d] Some specific thoughts on the scan repository (directory, format, etc.)
by Jon Noring 18 Jul '05

18 Jul '05

Thanks to Branko (public message) and Jon Niehof (private message, I believe) for their quick reply to my initial sets of questions/ /concerns/thoughts regarding the scan repository which was posted earlier today. It is interesting that the answers differed in several ways, which I won't reply to in the usual manner. But it does show that there will not be uniformity in thought in all the details of the scan repository. But filtering through both messages -- and despite the angst I expressed in an earlier message today agreeing with what Bowerbird had to say on this topic ("what should be"), it is clear that we can and should go ahead with a "first generation" scan repository focused primarily on centralizing the preservation of the scans submitted to DP, and to make them publicly available pretty much "as is". (Expansion to allow submissions of scan sets outside of DP activities can be considered once the DP scan repository is up and running, and mostly debugged. We will certainly learn a lot from it. Thus my go-slow preference.) Now, regarding Branko's comment that Brewster has disk space to burn (implying he doesn't care about the details of what we intend to do), this is NOT true. I've had several personal discussions with Brewster regarding other matters, and I know he will be concerned about the breadth/depth/future of what we have in mind, the public access issues, the format, etc. He may also be concerned how it will relate to his current book scanning project in Canada (not in a competitive sense, but in a compatibility sort-of-way). Some of his questions and concerns we cannot predict in advance. But the more we understand what we want to do (and what we don't want to do), the easier it will be to discuss this with him, hopefully resulting in his offer to host the PG/DP scan repository. Probably the #1 problem as I see it is to assure that scan sets don't become "dissassociated" from PG/DP. That is, there needs to be sufficient metadata so each scan set can be attached to a particular DP produced digital text, and from that, attached to the associated PG text. So, does DP keep an internal identifier and associated metadata for each project it works on? And will it be pretty easy to associate a scan set with that DP identifier, or will it require some human intervention to make the association? And does the DP identifier correlate to a PG text identifier? Another issue concerns the directory structure of the repository which will hold the scan sets. It has to take into account that there may be not only the original scan set which was submitted to DP, but some derivative sets produced by some processing (of course there's the 120 dpi used for the DP proofing interface.) I tend to view that for the DP scans, that the DP identifier be used as part of the directory name holding the scan sets associated with that identifier. Another issue is "zip vs. individual files". My view is leaning towards keeping each page scan image a separate file, so end-users need not download a whole gigabyte-size (or more) ZIP file to just look at one page (some books may have scan sets this large.) Another issue, which affects directory design, metadata and access, is serialization. Some Works were serialized (such as multiple parts in periodicals), which are combined into one digital text. How do we handle/organize the scan sets for this? There are no doubt other odd "exceptions" we will have to handle. Format is another issue. Banko mentioned that DjVu is proprietary (or essentially so). Yet Brewster was, last I talked with him last year about scan formats, enamored with it and using it for his scanning projects to greatly conserve disk space (he *is* concerned about the cost of storing book scans, thus his interest in DjVu.) If Brewster is still using DjVu, I can *guarantee* he will mention it to us -- we need an answer if we don't want to use DjVu compression. The "proprietary" issue will not sway him if he is using it. We need to maybe look at alternative "lossy" formats (I detest JPEG for compressing scanned images because of the significant artifacts it creates -- there must be something better which is also an open standard?) Obviously, I'd prefer to store the scan sets (and derivatives) in a format which preserves, in a lossless sense, the scans as they were submitted to DP. But we need not be wedded to keeping the identical format as they were submitted in so long as they are losslessly the same. Another issue regards end-user compatibility. Web browsers to view scanned pages are usually not TIFF- (nor DjVu-) ready without a plugin, while all contemporary browsers will handle PNG and JPG. When it comes to lossless compression, some of the CCITT protocols (as used in a TIFF encapsulation) for bitonal images are significantly better than PNG at lossless compression (PNG is designed for general lossless compression of all kinds of images.) Of course, there is the option that if a "third party" interface is developed to access the repository, image conversion on the fly could be employed to translate the format in the repository to most any other end-user can handle, even if the source is in some odd format. Another possible issue regards illustrations. Am I right that some scan projects scan the whole book at one resolution, then return and do the pages with illustrations at a higher resolution and crop to the illustrations? This may have an impact on directory structure and metadata fields. Anyway, that's enough for now. Comments? Jon

4 7

re: [gutvol-d] The future of the PG collection...
by Bowerbird＠aol.com 17 Jul '05

17 Jul '05

jon said: > It does beg the question, though, of what features, functions, user groups, etc., > that we'd like the PG collection of, say, 2020, to support (or integrate with). i have some ideas about that. so do you. and so do lots of other people. we can discuss it if you like. or not. might be nice. but it doesn't matter all that much to me, because i'm busy working to make my ideas a reality. the discipline of making them come alive while living under the constraint of creating source-code that a compiler turns into a working program gives me real-world feedback about whether my vision holds water. pudding is proof. as far as project gutenberg itself, that's up to the people steering that boat. i would hope they would listen to the founder. his vision brought them here. i know i can live with whatever michael does with his baby. > If anyone wants to know why I take various hard-nosed positions > regarding PG collection development, they are largely based on > meeting the requirements, as I understand them, for the above listed items. > I want to see the PG collection, and any digital text collection, be able to > meet the needs for most human endeavors, rather than being limited in scope. except you're working in a vacuum. so you think a file-format is an answer. you will get a lot smarter once you start getting feedback from programmers about which of your ideas will really work, and which ones are just pipedream s. until then, you don't know what you don't know. unfortunately for you, i do... -bowerbird

1 0

Fwd: [ElectroBooks] FYI Hand held Reading Machine
by Aaron Cannon 17 Jul '05

17 Jul '05

>A little OT, but it may have future implications for PGDP. > >Date: Sun, 17 Jul 2005 16:08:40 -0400 >Subject: [ElectroBooks] FYI Hand held Reading Machine > > >This is a forward of a forward. >Rick >>> > >>>Kurzweil Technologies and National Federation of the Blind premiere the > >>> > >>>world's first portable reading machine for the blind KurzweilAI.net, July > >>>11, 2005 Kurzweil Technologies , Inc. ( KTI ) and the National Federation > >>>of > >>>the Blind ( NFB ) introduced the world's first portable reading machine > >>>for > >>>the blind last week at the NFB's > >>> > >>>2005 National Convention in Louisville, KY. > >>> > >>>The portable reading machine , which can fit in a user's shirt pocket, > >>>can > >>>read print and text materials as users go through their normal daily > >>>routine. It converts print into human > >>> > >>>-sounding speech and can read handouts at meetings, signs on a wall, text > >>>on > >>>packages, and electronic displays. The hardware consists of a consumer > >>>digital camera with a standard PocketPC, so the hardware cost is expected > >>>to > >>>benefit from the rapid improvement of price-performance of consumer > >>>electronic s. The camera and pocket computer are held together by a > >>>snap-in > >>>case. > >>> > >>>The technology was developed by Ray Kurzweil and his colleagues at KTI in > >>>close collaboration with the NFB, which is organizing a comprehensive > >>>testing program with blind users. Ray Kurzweil was the principal > >>>developer > >>>of the first print-to-speech reading machine for the blind, also > >>>developed > >>>in close collaboration with the NFB. Ray Kurzweil noted that "The new > >>>unit > >>>is 10,000 times smaller than the original 1976 reading machine , yet the > >>>computer it's using is 2,000 times more powerful." This new portable unit > >>>is > >>>expected to be available for sale in 2006. > >>> > >>>NFB Director of Strategic Projects Jim Gashel demonstrated the reading > >>>machine to an enthusiastic audience of more than 2,000 blind delegates. > >>>"There was cheering for several minutes while the machine read the > >>>document," said Kurzweil, who followed the demo with a talk explaining > >>>the > >>>machine , KTI's collaboration with NFB on the project, and the future of > >>>this technology . > >>> > >>>The portable reader provides feedback to the user on what it sees, > >>>guiding > >>>the user to properly frame reading material. It can report, for example, > >>>if > >>>one of the edges of a document is out of view. It is also capable of > >>>stitching together portions of a document from multiple pictures taken by > >>>the camera. It can detect and correct any degree of document rotation and > >>>is > >>>insensitive to three degrees of freedom of image tilt or rotation. The > >>>software also includes image-enhancement techniques to compensate for > >>>uneven > >>>illumination and the low quality optics of inexpensive consumer cameras. > >>> > >>>Future version will include scene-recognition capabilities to locate > >>>objects > >>>such as chairs, lamps and people. > >>> > > > > > > > > > >Yahoo! Groups Links > ><*> To visit your group on the web, go to: > http://groups.yahoo.com/group/ElectroBooks/ > ><*> To unsubscribe from this group, send an email to: > ElectroBooks-unsubscribe(a)yahoogroups.com > ><*> Your use of Yahoo! Groups is subject to: > http://docs.yahoo.com/info/terms/ > -- E-mail: cannona(a)fireantproductions.com Skype: cannona MSN Messenger: cannona(a)hotmail.com (Do not send E-mail to the hotmail address.)

1 0

how long does it take to digitize a book?
by Bowerbird＠aol.com 17 Jul '05

17 Jul '05

some reports tell us that the new harry potter is out in digital form already, thanks to "hackers" who digitized it within 24 hours of its release on paper. unknown is whether it has already been converted to a dozen different formats, in the same manner as the books of cory doctorow, who releases his text freely; his books are commonly converted -- by his fans -- in a matter of a week or so... (his sharp-eyed readers also do a great job of proofreading for him, so good in fact that if i were tor -- his publisher -- i would hold off on the first printing until _after_ they had found the few dozen mistakes that were present in cory's latest book...) -bowerbird

2 1

re: [gutvol-d] RFC: Posting Page Scans in DJVU Format
by Bowerbird＠aol.com 17 Jul '05

17 Jul '05

sebastien said: > Now suppose that in a few years time > 1/ this _Study of ancient Greece_ gets into PG, number 78967 > (in the same edition John Smith used) you've swept under the rug what could be a very important disqualifier. > 2/ some text-crawling program detects its fuzzy quotation > in the Smith's essay > 3/ some robot and/or human reworks Smith's essay to include > hyperlinks to PG text 78967, at the correct page number > (which would land at the right paragraph in the HTML) > and checks them one by one why require those changes to be made after-the-fact? why not build a program that can do all of that _on-the-fly_? without requiring any heavy markup? that's what i'm aiming at. (i've got aspects of this approach working already in my stuff.) the ability of the machine to find links like this automatically -- i.e., without us specifying them -- is extremely important! we want an intelligent viewer-program that knows what we want, _without_ us having to spell out every aspect of it in great detail. > That would be a killer ultimate library! well, it's one step _toward_ that killer ultimate library, anyway. there are a lot more steps that need to be taken on that journey. -bowerbird

1 0

DRM and paper books
by Marcello Perathoner 17 Jul '05

17 Jul '05

"A grocery store in Coquitlam, British Columbia, Canada accidentally sold several copies of the sixth Harry Potter book before the authorised release date. The Canadian publisher, Raincoast Books, obtained an injunction from the Supreme Court of British Columbia prohibiting the purchasers from reading the books in their possession." http://en.wikipedia.org/wiki/Harry_Potter_and_the_Half-Blood_Prince http://upload.wikimedia.org/wikipedia/en/8/86/Harry_Potter_Injunction.pdf -- Marcello Perathoner webmaster(a)gutenberg.org

1 0

re: [gutvol-d] RFC: Posting Page Scans in DJVU Format
by Bowerbird＠aol.com 17 Jul '05

17 Jul '05

people who are building the programs that will use the pagescans need to be the ones that specify the conventions in making them, in consultation with the people who are actually doing the scanning. marcello, i don't see that you have much expertise on either end... that's not to say that your set of recommendations is all that bad. it's not that much different from the set i laid out some time back. (although in the places that it is different, mine is the better plan, reflecting experience in scanning and developing scan-using apps.) in direct response to david, just keep scanning the way you want, the way that works best for you. we can deal with what you create after the fact, if necessary. we'll have some feedback on workflow that we think will be mutually beneficial, but whatever you do, know for now that we can deal with your output, so just keep on scanning. -bowerbird p.s. as i feared, the current examples of pagescans in the library are case-studies in how _not_ to do it, for reasons i will list later...

1 0

re: [gutvol-d] RFC: Posting Page Scans in DJVU Format
by Bowerbird＠aol.com 17 Jul '05

17 Jul '05

marcello said: > Suppose we post a multi-page djvu, where > the filenames of the single-page djvu files it contains > are not in sync with the real page numbers. you mean like the one you created and pointed us to yesterday? -bowerbird

1 0

re: [gutvol-d] Initial thoughts on a PG/DP scan repository
by Bowerbird＠aol.com 17 Jul '05

17 Jul '05

marcello said: > I don't want to unload hundreds of image files on the user. except that's precisely what happens when you bundle them all together. the user who only wants one of the pagescans has to download them all. there are a raft of good reasons why a person might want a single scan. lets think in the pluralistic mentality for which project gutenberg is known. there's no reason you can't have one place that has the individual scans, one that has them zipped up, another that has them as a djvu (both forms), one that has them as a .pdf, and even one where they are a quicktime movie. per lockss, you want multiple copies scattered throughout cyberspace anyway. speaking of pluralism, any all-image format that can't play on a playstation, or on machines like that new nokia770 that's coming out, is a non-starter. > Can you imagine reading Ulysses from 1.000 tiff files? why not? the person is only looking at one or two pages at a time, so the viewer-program only has to open one or two files at a time, who cares how many there are in the whole book? > Is there even any picture gallery software that can handle that gracefully? i'm sure there is. but why use that? instead, use e-book viewer software that's geared for scans. my viewer-program can use scans _or_ text, or even _both_, in combination, for people who want to view them side-by-side (e.g., to do proofing). *** this is a good demonstration of one problem with project gutenberg: you don't have any application programmers helping you lead the way. so you are forced to settle for the least worst program at your disposal, instead of grabbing the bull by the horns and creating the best program. i would suggest that the next time a programmer like me comes along, you be nice to him or her, instead of spitting in their face. -bowerbird

2 1

re: [gutvol-d] XML: Interpublication and deep linking
by Bowerbird＠aol.com 16 Jul '05

16 Jul '05

i'm off the merry-go-round for this round. let me know when the library is marked up. -bowerbird

1 0