- gutvol-d - lists.pglaf.org

re: [gutvol-d] Re: 600 dpi vs. 300 dpi for text (a quickie visual experiment)
by Bowerbird＠aol.com 21 Jul '05

21 Jul '05

i said: > you've got too books under your belt now. two cute... -bowerbird

1 0

re: [gutvol-d] 600 dpi vs. 300 dpi for text (a quickie visual experiment)
by Bowerbird＠aol.com 20 Jul '05

20 Jul '05

jon said: > If the purpose of scans is for multiple use cases (and not only OCR), > then it appears wise to scan text at 600 dpi, preferably 24-bit full color except over at d.p., their reason for scanning is to do o.c.r., for proofing. but perhaps other projects, like million-books or internet-archive, would find your position persuasive. or maybe not. > Of course, the huge downside to higher-rez, > higher-color-depth images are much greater file sizes. > This causes difficulty with online archival storage and transport. that's only one huge downside. another huge downside is that scanning takes _much_ longer. indeed, i would say that _that_ would be their primary objection. most d.p. scanning people are processing lots and lots of books, and do not have the time to spare to create scans at a resolution that is far higher than what they require to do the job that they set out to do. as always, it has everything to do with the cost-benefit ratio. you can rave all you want about the benefit, but if people don't want to pay the cost... -bowerbird

1 0

re: You often don't really understand the problem
by Bowerbird＠aol.com 20 Jul '05

20 Jul '05

eric raymond said: > "You often don't really understand the problem > until after the first time you implement a solution." > -- The Cathedral and the Bazaar by Eric S. Raymond bingo. and it is only after you have implemented a dozen solutions -- each of them bearing on a different facet of the problem -- that you understand that the problem all along was an elephant, not a rope, or a tree, or anything else, and you are a blind man... -bowerbird

1 0

Re: gutvol-d Digest, Vol 12, Issue 25
by Lee Passey 19 Jul '05

19 Jul '05

Marcello Perathoner <marcello(a)perathoner.de> wrote: > Robert Shimmin wrote: > >> I know of one fairly prominent commercial digital library (Eighteenth >> Century Online) that made the decision that the issues with using >> page numbers as unique identifiers are sufficiently hairy that they >> went with sequential *image* numbers (which are unique), and the >> database maintains as metadata the page number that goes with each >> image number (which may not be unique, sequential, numerical, or even >> present). > > > They may have a completely different usage case than we have. > > Putting all pages of a book into a container and numbering them in one > sequence may work well if you don't have external references to the > sequence number. It may also work well if you are sure, *really* sure > that you won't ever go back and add or remove some pages from the > collection. > > But if you want people to link to the page images *and* be able to > insert or delete pages from the collection after the first publication > you cannot use the single sequence numbering technique. > > Suppose we have published page images for "Alice in Wonderland". > Suppose the first edition just contains the pages and does not contain > separate images for the illustrations. Later somebody goes back and > scans the illustrations in a better resolution and naturally wants > them to be inserted in the right position. Using sequential numbering > you are dead in the water. > > Suppose you have collected a lot of page images but thrown away the > images of the cover pages if there was no text on them. Later you > decide to add those images too. Again, using single-sequence numbering > you are dead in the water. > > But if you used the true page numbers none of these changes poses the > slightest problem because you never have to change the urls of the > pages you published before. This is an extremely important point that I feel has been woefully neglected up to this point. Just what are the use case scenarios this solution is intended to solve? You cannot effectively answer the question of 'How?' until you have answered the question of 'Why?' If you can come up with a number of reasonable scenarios as to how these page scans could be used, the naming mechanism almost defines itself. I'm very skeptical that more than an extremely small number of people will be interested in downloading a full set of page scans. As I see it, the most likely use case is that someone is reading a Gutentext offline and encounters a passage that doesn't seem right. This person would say to himself, "Hmm, that doesn't seem right. I wonder if that's really what was in the book." He (or she) would then go online, look up the image of the page in question and conclude, "I guess that's right", or "wow, what a glaring mistake; I'd better tell someone", or "ohh, the emphasis is missing, _now_ it makes sense." Of course, it is impossible to search for a word in an image, so in this scenario it is important that the name of the page scan be embedded in the resulting text. The actual name used is not important, so long as it is immutable and unique across the namespace of all literature. Also, in this scenario it is important that the page scans be available individually on the 'net, and not solely as part of an encapsulation. Thus, there must exist some mechanism to identify which scans belong to which edition of which work. Another possible use case is when an individual or company says, "The production values of Gutentexts are extremely poor, I would like to create a high-quality version of a public domain work that I could then sell for 5 bucks." (http://www.hidden-knowledge.com/) In this case, it is probably important that the page scans be available in a format that can be easily used by an OCR program (because in most cases it is easier to re-OCR to retain formatting than it is to try and clean up a Gutentext), and it is probably important that the page scans be named in some way that preserves their ordering. Of course there is the use case Mr. Perathoner postulated where a scan is simply incomplete and it needs to be augmented in the future. It may be that the scanner was simply so short-sighted as to discard otherwise important data, thinking that it wasn't really important, it may be that the scanner was forced to excise material that was still covered by copyright at the time of scanning, or it may be that it is a scan of the "Kama Sutra", and the only available copy had been razor-bladed by sophomores from the local high school to get at "the good parts." In this case it is important that the scans be maintained as individual pages, and that the naming convention permits new images to be "inserted" between two already existing scans. While I can't come up with a good use case scenario off the top of my head, it seems to me that it will be important that the naming convention will need to support a work flow which is 100% machine based; that is, once the scans and OCRs are completed any number of other processes can occur which do not require _any_ human intervention. At first blush, Mr. Perathoner's proposal for a naming convention seems to satisfy the requirements of all these use cases (although there are some abiguities and uncertainties that may need to be tightened up). But it seems to me that there should be a bit more discussion about the use cases before discussing implementation. Now I need to digress for a minute. David Starner <prosfilaes(a)gmail.com> wrote: >On 7/17/05, Marcello Perathoner <marcello(a)perathoner.de> wrote: > > >>And rightfully so. Deep linking to images is not permitted. This is >>documented at: >> >> > >So what's the point of worrying about deep linking to images if we're >not permitting deep linking to images? > I think there may be some misunderstanding of what Jon Noring is talking about when he uses the term "deep-linking". Consider the case of reported United States legal decisions. A typical legal decision is cited as, e.g., "MATTHEW BENDER & CO. v. WEST PUBLISHING CO., 158 F.3d 674 (2nd Cir. 1998)." This citation contains some metadata (the names of the parties, the appeals court which rendered the decision [the Second Circuit Court of appeals], and the year of the decision). It also contains a reference to how this case may be found: look in the 158th volume of the Federal Reporter, 3rd series at page number 674 (the official Federal Reporter is published by West Publishing Company). Now suppose I wanted to point a person to that point in the case where Judge Jacobs first cites the U.S. Supreme Court case of "FEIST PUBLICATIONS, INC. v. RURAL TELEPHONE SERV. CO., 499 U.S. 340, 111 S.Ct. 1282 (1991)". I would do this by citing '158 F.3d 674 at 679'. As you can see from this last example, the citation actually contains two parts: the shallow link, '158 F.3d 674', and the deep link, 'at 679.' The shallow link and the deep link are actually two orthogonal concepts; related but independant. The shallow link tells you how you can find the document, the deep link tells you how you find the point you are looking for inside the document once you have found it. Now in the online version of the _Bender v. West case_ (http://www.law.cornell.edu/copyright/cases/158_F3d_674.htm) deep linking is acheived through "star pagination"; scattered throughout the text you will find strings of the form "[p*679]" which tell you where the page number was in the official West publication. (If you want to know more about "star pagination", read the case; it was an attempt by West Publishing to get a ruling that star pagination, or deep linking via page numbers, was a violation of their copyright. West lost.) Deep linking in XHTML can be acheived in a couple of different ways. On way is to include id attributes on tags throughout the file. Thus, the aforementioned star pagination could be encoded as '<a id="p679">[p*679]</a>', which would allow a person to cite the location as '158_F3d_674.htm#p679.' On the other hand, using XPATH that citation could be cited as ''158_F3d_674.htm//p[13]' which identifies the 13th <p> element in the document. The XPATH mechanism gives you greater granularity (the citation to _Feist_ is actually in paragraph 13, which is the third full paragraph after the page marker) and doesn't require _any_ fragment identifiers to be imbedded in the document, but it is not yet well supported in user agents, and it _does_ require the document to be in a canonical form. The point of this little digression is that funtional requirements for shallow linking and deep linking are very different, as are their solutions. Shallow linking has to solve the problem of "How do I find a copy of a particular document in the great digital library that is the web", and deep linking has to solve the problem of "How do I indicate a specific point in a document that I have already found." This brings me to the last use case I will present (I'm sure there are many others, as well). Many, if not most, lower court cases are not yet digitized, yet the cases citing them are. I want to be able to present the digitized text with deep links (but perhaps not shallow links) that will allow me to get to an as yet undigitized cited case _when it appears in the future_. Page scans which are named according to existing page numbers will allow be to do that in the future, as will embedding page number links into structured digital texts as they are produced. Now the inevitable counter to my desire to be forward-looking is the argument "we don't need it right now, so why should we do it now." My response is much more philosopical than it is empirical: If we don't have the time to do it right, how will we find the time to do it over?

2 1

Re: Scans and Texts
by Juhana Sadeharju 19 Jul '05

19 Jul '05

I would like that we archive the scans. * The original print is more pleasant to read than the ascii or html text. * Text with figures is better in its original layout. * Math text is better in its original print than in the TeX or math-html equivalent. * Many old books are simply too beautiful to be replaced with an ascii copy. However you archive the scans, please avoid 1-bit scans. 16 grey levels with 200 dpi would be better than 1-bit with 600 dpi in my experience. Juhana -- http://music.columbia.edu/mailman/listinfo/linux-graphics-dev for developers of open source graphics software

7 7

Re: [gutvol-d] Copyright Verification?
by Joshua Hutchinson 19 Jul '05

19 Jul '05

----- Original Message ----- From: "Jon Noring" <jon(a)noring.name> > > However, the scans themselves have value, including for direct reading. > Scanning at higher resolution gives more to play with respect to image > restoration for direct reading uses. But for our purposes, getting text into PG, they don't have any other purpose. I realize that scans are important to you, but they simply are NOT to most of us, beyond their use as OCR sources. I don't have a problem with archiving the images but I will not change the work flow I do (or recommend the change in DP) to get "archival quality" scans. It simply is not something I deem important to the work we do here. Josh

18 69

re: [gutvol-d] RFC: Posting Page Scans in DJVU Format
by Bowerbird＠aol.com 19 Jul '05

19 Jul '05

jon said: > One can come up with all kinds of cool and simple systems to do a job, > and 9 times out of 10 none of them will work in the real world, > at least without some specific exceptions handling capability. you obviously didn't understand what i said about the act of binding creating a specific linearity which we can duplicate by a judicious naming of the files, thereby rendering all of your "exceptions" moot. if you think there is a situation that i cannot handle, describe it here. otherwise, your ignorance is showing. my system will "work in the real world", and without imposing undue difficulty on volunteers. indeed, it will make their work go much easier... > So it does pay to identify and discuss the exceptions > before deciding on any design which will be employed > in a system that will handle thousands and eventually > millions of texts. again, show me _one_ "exception" that i cannot handle. > If one "does it right" at the start, it makes things > easier down the road, even if mid-course corrections are > later required due to things that just could not be foreseen. i'm saying you don't know how to "do it right", and that the "system" you are proposing is unnecessarily complex. > The more exceptions that are handled today, > the fewer that will need to be handled tomorrow -- > that sounds good when are talking about one million etexts. i'll say it one more time: show me one exception. just one. i've been thinking about this part of the workflow for _years_, years which include experience at every step of the process, including ones that you won't get around to for a good while. if you think your off-the-top-of-the-head musings trump that, you're obviously deluded. and given your lack of experience, it must be particularly galling to the regular scanners at d.p. that _you_ now want to make the rules about how they do it. -bowerbird

1 0

re: [gutvol-d] Unknown Mexico by Carl Lumholtz, and copyright
by Bowerbird＠aol.com 19 Jul '05

19 Jul '05

jeroen said: > The curious thing is that it is public domain everywhere, except for Mexico, > which extended copyrights to a ridiculous life+100 years. wow. that certainly is ridiculous. but the next thing the copyright cartel in this country will be wanting to do, if they start winning some of the legal fights they are picking, like with google, is to "harmonize" to that, dragging all the other countries in the world along. -bowerbird

1 0

re: [gutvol-d] RFC: Posting Page Scans in DJVU Format
by Bowerbird＠aol.com 18 Jul '05

18 Jul '05

robert said: > I know of one fairly prominent commercial digital library > (Eighteenth Century Online) that made the decision that > the issues with using page numbers as unique identifiers > are sufficiently hairy that they went with sequential > *image* numbers (which are unique), and the database > maintains as metadata the page number that goes with > each image number (which may not be > unique, sequential, numerical, or even present). and that's a perfectly reasonable fall-back position that will work in any situation where the normal system won't. (actually, most normal cases are a subset of this, where the page-number and the image-number happen to match.) after all, these are books, comprised of sheets of paper that were bound together, with a natural consequence of _being_ bound being the enforcing of a specific linearity on those sheets. as long as the filenames -- when sorted by our various tool apps -- result in that _same_ specific linearity, there will be no uncertainty. (and, since we have the power to name them, we can ensure that.) the focus on exceptions here is making too much ado about nothing. it means you're talking about the wrong things -- there are questions which _do_ need to be discussed , but this is frankly not one of them. and it also means that you're making things unnecessarily complicated, which is not surprising, since that's the jack-in-trade forte of jon noring. the right approach is _a_simple_flexibility_ that gives the right outcome. -bowerbird

2 1

[gutvol-d] Some specific thoughts on the scan repository (directory, format, etc.)
by Jon Noring 18 Jul '05

18 Jul '05

Thanks to Branko (public message) and Jon Niehof (private message, I believe) for their quick reply to my initial sets of questions/ /concerns/thoughts regarding the scan repository which was posted earlier today. It is interesting that the answers differed in several ways, which I won't reply to in the usual manner. But it does show that there will not be uniformity in thought in all the details of the scan repository. But filtering through both messages -- and despite the angst I expressed in an earlier message today agreeing with what Bowerbird had to say on this topic ("what should be"), it is clear that we can and should go ahead with a "first generation" scan repository focused primarily on centralizing the preservation of the scans submitted to DP, and to make them publicly available pretty much "as is". (Expansion to allow submissions of scan sets outside of DP activities can be considered once the DP scan repository is up and running, and mostly debugged. We will certainly learn a lot from it. Thus my go-slow preference.) Now, regarding Branko's comment that Brewster has disk space to burn (implying he doesn't care about the details of what we intend to do), this is NOT true. I've had several personal discussions with Brewster regarding other matters, and I know he will be concerned about the breadth/depth/future of what we have in mind, the public access issues, the format, etc. He may also be concerned how it will relate to his current book scanning project in Canada (not in a competitive sense, but in a compatibility sort-of-way). Some of his questions and concerns we cannot predict in advance. But the more we understand what we want to do (and what we don't want to do), the easier it will be to discuss this with him, hopefully resulting in his offer to host the PG/DP scan repository. Probably the #1 problem as I see it is to assure that scan sets don't become "dissassociated" from PG/DP. That is, there needs to be sufficient metadata so each scan set can be attached to a particular DP produced digital text, and from that, attached to the associated PG text. So, does DP keep an internal identifier and associated metadata for each project it works on? And will it be pretty easy to associate a scan set with that DP identifier, or will it require some human intervention to make the association? And does the DP identifier correlate to a PG text identifier? Another issue concerns the directory structure of the repository which will hold the scan sets. It has to take into account that there may be not only the original scan set which was submitted to DP, but some derivative sets produced by some processing (of course there's the 120 dpi used for the DP proofing interface.) I tend to view that for the DP scans, that the DP identifier be used as part of the directory name holding the scan sets associated with that identifier. Another issue is "zip vs. individual files". My view is leaning towards keeping each page scan image a separate file, so end-users need not download a whole gigabyte-size (or more) ZIP file to just look at one page (some books may have scan sets this large.) Another issue, which affects directory design, metadata and access, is serialization. Some Works were serialized (such as multiple parts in periodicals), which are combined into one digital text. How do we handle/organize the scan sets for this? There are no doubt other odd "exceptions" we will have to handle. Format is another issue. Banko mentioned that DjVu is proprietary (or essentially so). Yet Brewster was, last I talked with him last year about scan formats, enamored with it and using it for his scanning projects to greatly conserve disk space (he *is* concerned about the cost of storing book scans, thus his interest in DjVu.) If Brewster is still using DjVu, I can *guarantee* he will mention it to us -- we need an answer if we don't want to use DjVu compression. The "proprietary" issue will not sway him if he is using it. We need to maybe look at alternative "lossy" formats (I detest JPEG for compressing scanned images because of the significant artifacts it creates -- there must be something better which is also an open standard?) Obviously, I'd prefer to store the scan sets (and derivatives) in a format which preserves, in a lossless sense, the scans as they were submitted to DP. But we need not be wedded to keeping the identical format as they were submitted in so long as they are losslessly the same. Another issue regards end-user compatibility. Web browsers to view scanned pages are usually not TIFF- (nor DjVu-) ready without a plugin, while all contemporary browsers will handle PNG and JPG. When it comes to lossless compression, some of the CCITT protocols (as used in a TIFF encapsulation) for bitonal images are significantly better than PNG at lossless compression (PNG is designed for general lossless compression of all kinds of images.) Of course, there is the option that if a "third party" interface is developed to access the repository, image conversion on the fly could be employed to translate the format in the repository to most any other end-user can handle, even if the source is in some odd format. Another possible issue regards illustrations. Am I right that some scan projects scan the whole book at one resolution, then return and do the pages with illustrations at a higher resolution and crop to the illustrations? This may have an impact on directory structure and metadata fields. Anyway, that's enough for now. Comments? Jon

4 7