- gutvol-d - lists.pglaf.org

Scan file naming -- another comment
by Jon Noring 25 Jul '05

25 Jul '05

Bowerbird's thoughts on scanning is a good summary of some of the issues. And despite his view that we've gotten off-track on the discussion, his points about filenaming, image processing (deskewing), etc., align pretty well with the ongoing discussion. Regarding scan filenaming, he rightfully notes that a source book (or Work) identifier be prepended to the filename. This is what I have also proposed. Where we differ in filename convention is that I believe right after the source ID be a sequential number which describes where the page side is in the linear order of all the page sides in the book (totally independent of how the publisher may have paginated the book.) This way one will unambiguously and immediately know the position of every page scan in the bound book (starting with the inside of the front cover, which can be "side 1", and end with the inside of the back cover -- alternatively we can start with the front cover as "side 1" which has some advantages with respect to the dominant recto/verso page numbering convention.) All blank pages will be included. Now, this sequential number will not correlate at all with whatever pagination the publisher uses to 'id' the pages. So, after the sequential number we have a third field in the filename which gives the actual publisher supplied page number (if any; can be implied). This way we decouple the publisher pagination with the page sequence in the book, thereby simplifying the system and making it more flexible. It will be able to handle *any* bizarre pagination system the publisher/author dreamed up (the publisher could number the pages backwards for all we care, and this system will handle it without any complications -- yet we preserve the publisher-supplied page "number" in the filename which is important for referencing/citation.) Example: DP0000239-00125-106.png "DP0000239" is the source book identifier, here a DP identifier. If the scan project is independent of DP, it could be 'PG0014239' to associate the scan set with PG text number 14239. "00125" says this is the 125th "side" in the full sequence of sides in the book, starting from the front cover or wherever else is considered the starting point. "106" is the string (which can be more complicated like "A2", "5-4", "ix", "ABCD" whatever), which the publisher printed on that page to identify it (that's really what a publisher-supplied page "number" is: a page identifier.) (My proposed system has a couple more fields after these three, dealing with exceptions and generation of the scan set from the original, which aid in keeping tracking of multiple derivative scan sets and a few other oddities. The details are described in previous messages.) Jon Noring [Note: In the "My Antonia" project, it is interesting that there is no "Page 1" and "Page 2". The book starts (after the Roman numbered foreword section) with Page 3! Now imagine getting a scan set of "My Antonia" where we have defined page scan sequencing using the page numbering the publisher used (which are the systems proposed by Marcello and Bowerbird.) The first question I will have is "where are pages 1 and 2? Are they missing from the set?" However, if the scans are sequentially numbered based on their position in the book (with the knowledge the book passed QC checking), then I would know that at least the project saw this, too, and likely there were no missing pages, thus it is likely Page 1 and 2 never existed. And for those who will ask, before scanning the book I took it apart to determine if there was a page which got ripped out there, but there definitely was no ripped out page. There might have been an inserted/ glued plate which "fell out", but checking with the "My Antonia" experts there definitely was no such insert in the First Edition. How many other books start pagination of the body with something other than 1?]

7 25

re: [gutvol-d] re: greg offers diskspace for real live examples
by Bowerbird＠aol.com 25 Jul '05

25 Jul '05

greg said: > If you'd like to just make smaller stuff > available to me via email, I can upload it there. > Or, I can set up a username for you to ftp/scp > or otherwise get content to. thanks. probably better if i can do it myself, so as not to trouble you. thank you very much. -bowerbird

1 0

re: greg offers diskspace for real live examples
by Bowerbird＠aol.com 25 Jul '05

25 Jul '05

greg said: > The "say" is that we're all anxiously awaiting any completed eBooks > demonstrating any or all ways of doing things. ok, i will take you up on that. i've prepared some material, and programmed a demo app. where can i upload it? > There will not be any decision about the official way of doing things > (if there ever is) until some people create demonstrations. well, i've created one demo already, and will make more. but... how about if you correct the examples you've already got up, on the basis of the constructive criticism i've already offered? until you've done that, you're not really ready for many more... > It's clear that light programming at gutenberg.org > to make a given demo work is available. i don't need any programming help, thank you, just some server-space to upload my examples. and, so that i can make examples using your content, i need for all of your scans to be individually available, rather than bundled up into a .zip file, or a djvu file... > So is server space there or elsewhere. just give me a u.r.l. where scans will be located. one thing that hasn't been discussed thus far is the generation of a "scan-set" for the e-texts that are a "new" project-gutenberg publication, i.e., an amalgamation of _several_ source-texts. (jon, try to control that sputtering that is about to well up in your throat right now; it will soon pass...) ;+) for the sake of completion across the library, they should have an associated "scan-set" too. *** nonetheless, greg, i think you're getting a bit ahead of yourself here, aren't you?, and falling behind at the same time. or at least doing things in the wrong order. there's lots of scan-sets sitting around. it won't do much good to put more up until you know what you want to _do_ with them once they've been put up... i know what you _should_ want to do with them, but i don't get the feeling that _you_ do. (or they'd _be_ up.) so why don't we all discuss _that_? once you've got that figured out, then i'll deliver to you the app that you need, and the decision about how to structure your scan-sets will be a lot easier to make. it's been rather entertaining to watch the silliness of people making design decisions without much awareness at all of the tasks for which those decisions are being made. and no, i'm not going to "just tell you". (well, i already have, several times now, but i guess that's beside the point, since nobody appears to be listening very well.) you don't deserve to have the answers handed to you. further, if you have to tell me what you want before i deliver it, you won't be able to deny that it is right once i have delivered it to you... so c'mon, show me you're smart enough to figure out on your own what you need here... -bowerbird

2 1

here are the suggestions that should be accepted
by Bowerbird＠aol.com 25 Jul '05

25 Jul '05

jon said: > Well, we've both had our say, > so let the few others who > are following this topic > decide for themselves. how about instead we have people forget about what you said so they can pay attention to what i said? :+) the system you want to lay on people is so convoluted from the standpoint of their _actual_ workflow right now that they reject what you say out of hand, and end up unwilling to discuss the topic. and that makes it all that more difficult to get them to accept reasonable ideas. as david put it: > That is true. That will continue to be true > what ever standard is agreed upon, since > people aren't going to suddenly change > their scanning patterns based on what the > final product, a long ways down the line, looks like. now, the reality of the situation is that a change in their "scanning patterns" might well _improve_ their work flow, make them more efficient, and so on. but since you don't understand that flow -- and since they know that you don't -- you can't really hope to talk at that level. i'm sorry to have to keep on harping that you don't know what you're talking about. i'm aware that some people hear that and think i'm some mean person dumping on you. the truth of the matter is i don't like saying it. i don't like that you make it necessary to do so. but _somebody_ has to level with you, jon. take another example, from your latest post: > But your system requires that each image, > when it is first saved, needs a human being > to eyeball the page, determine the publisher > supplied page number (if any; may be implied), > and then manually save the page > using the publisher number. so i tell you again that you need to look at the auto-naming feature in your scanning program. it saves the page automatically, then increments the page-counter in anticipation of the next file. (if you don't have such a capability, you really need to get yourself a more modern program. but since it was common even a decade ago, i assume that your program has this feature.) auto-naming is what most all d.p. scanners use. they don't enter a name for each individual file, as that would waste energy and slow them down. they get into a dream routine where their body is flying in a ritual of page-turning and positioning. some of 'em can scan hundreds of pages an hour. if you had scanned in more than a few books, and scanned 'em in the manner that d.p. scanners do, you would know this, and you wouldn't say things so inconsistent with the reality of their processes. *** here's what _really_ needs to be done to take page-image scan-sets public... 1. scanning should be done such that the auto-naming of the scanned images fits the outcome we want them to have. however, if the scanner for some reason cannot or did not do this, it's no problem, as our tools can rename the files _easily_. i developed such a renamer using an e-text that had unnumbered plates and blank pages (some of which were scanned and some not), plus a whole raft of missing pages (part of an english/hawaiian facing-translations section), and so on. and the renaming tool worked fine. 2. quality-control needs to be done _during_ the process of scanning, so errors (like missing pages or duplicates) are caught when they can be fixed easily. i believe d.p. scanners have learned this. you do a visual inspection of each page, to make sure it is acceptable quality, and then a check where you step through all of the images checking their page-numbers to confirm you got every single one of 'em. again, regular d.p. scanners do this already, because they've learned that it's worthwhile. finding bad pages after fact = pain in the ass. 3. all of the scans need to be _cleaned_. they need to be deskewed and regularized, both in terms of size, and placement of text. since this job can be done fairly automatically, without much need for human intervention, and since it will help the o.c.r. accuracy, it should be done _before_ the o.c.r. the original files can be discarded; the foibles of scanning are unimportant. i have not lobbied on behalf of such cleanup before because i figured d.p. would learn it sooner or later. (now if only the large scanning projects would too!) but if you take scans public, they must be cleaned. but again, if this wasn't done originally, it can easily be done later, since it is generally rather automatic. 4. the o.c.r. process should output individual files, so as to retain the linkage between text and its scan. at this point, any weirdnesses in the page-numbering can be addressed, and blank scans inserted if needed. i won't get into the reasons why now, but these files should have all styling (margins, italics, etc.) saved! folks, do _not_ strip your o.c.r. files to plain-ascii! 5. the posted e-text-files -- ascii, html, all of 'em -- need to have the page-breaks indicated in some way, so that end-users and their tools can easily correlate the text with the exact page-scan from which it came. so pagebreaks must be retained every step of the way! stop discarding this useful information from the files! if you need input on how to represent them, ask me! 6. all the way through the cleaning-up of the text, a person should be able to deal with one page of text _or_ with the entire book, whichever is convenient for the particular clean-up task at hand at the time. this is, of course, a function of the _tools_ they use, rather than the scan-set per se, but i mention it now since the implication is you'll need to be receptive to feedback and requests you get from your toolmakers; an example of that is my input on naming of the files. 7. every page-scan should be accessible separately. bundles -- such as djvu and .zip -- can also be made, but the world must be able to grab each page-scan _individually_ when one is all that they want to see; it's unreasonable to expect 'em to download 'em all. 8. a system of "continuous proofreading" should be initiated to make the best use of these public scans, as it will leverage the immense power of user eyeballs to march the e-texts toward an error-free perfection. 9. any all-image-format should be easily ported to the current and future platforms that can utilize it, like the sony playstation portable and the nokia770. it would be great if you offered such porting online. i'll be offering it as a feature in my viewer-program, with the text-file as input, but it would be nice if you could offer it using the original page-scans as well... 10. whenever possible, the converted e-text should accompany the scans, to overcome the weaknesses of an image-format (e.g., the lack of searchability, the inability to copy-and-paste text, and so forth...). -bowerbird

1 0

re: [gutvol-d] why is michael hart being censored?
by Bowerbird＠aol.com 24 Jul '05

24 Jul '05

not good enough, jon. not even close. i see no long-timers on your listserve being moderated _except_ michael. you are not wise enough to judge him; his visionary status is too well-proven, while your crystal ball is badly cracked. you've got 21 hours left to right the wrong. -bowerbird

1 0

why is michael hart being censored?
by Bowerbird＠aol.com 24 Jul '05

24 Jul '05

a rather alarming thing has come to my attention. it seems that michael hart is being censored by jon noring over on jon's listserve. messages from michael appear to have to be "approved" before they are posted to the list. the evidence is sitting right there in the "headers" to see. jon, is this actually the case? i demand you confirm or deny. if so, i find it quite preposterous. love him or hate him, michael hart _invented_ e-books. when a lot of people were just flapping their mouths, michael sat down and started _typing_ various books into the computer and making them available globally, for _free_. and that's as in beer _and_ as in speech. if any person has earned the right to speak freely about electronic-books, michael has. (and let us remember that no one needs to "earn" this right in the first place; the right to speak freely is an inalienable human right -- one recognized in the charter of the united nations.) unless michael is allowed to post freely on jon's listserve, without having to go through the "mother may i" routine of having posts "approved" by jon or any other moderator, i -- for one -- will cease to converse with noring in any way. you've got until noon on monday to fix this egregious error, jon. -bowerbird

2 1

Who would like to donate page scan sets? (Independent of DP)
by Jon Noring 24 Jul '05

24 Jul '05

Obviously, there's significant interest among many PGers to submit scan sets of books to the PG archive to be linked to (augment) the etext versions. Thus I am doing a preliminary survey of those who work independently (that is, who don't submit their work through DP) who have scan sets of books that they'd be willing to donate to the cause. If you have scan sets to donate, let me know in private email what you have. Include number of books, the approximate total disk space they take up, and the general properties of the scans (resolution, color depth, image format, how they were created, file naming convention, etc.) Of course, Don't Send Me Any Scans At This Time! I'm not prepared to take donations until I get a better idea of the magnitude of what's out there, and its general condition. Then it will be clearer as to how to proceed with the submission stage. Thanks! Jon Noring

1 0

re: [gutvol-d] Scan file naming -- another comment
by Bowerbird＠aol.com 24 Jul '05

24 Jul '05

jon said: > The system I propose for scan file naming > is *simpler* than yours and more flexible no, it isn't. not on either count. the mere act of keeping track of a single name when it contains two -- or more -- variables in it becomes much more difficult than it needs to be. put together a half-dozen books each containing hundreds of scan-files using your verbose names and you'll find yourself drowning in the confusion. your pace would fall to a crawl. try it. you'll see. d.p. scanners can't afford to work at such a pace. my filenaming convention has grown out of my experience through the entire digitizing process. if it would have needed to be more complicated, i would have learned that by now and made it so. you can always make a system more complex. the smart thing -- which experience teaches -- is to realize when the increment in _cost_ will return to you a sufficient increment in _benefit_. and when it will not. you have taken the simple-and-useful principle that it is good to know the contents of a file from its name, and blown it past the point where it is cost-beneficial. there are all kinds of information that you _could_ put into the filename, which _might_ be useful at some point. but if it makes the process of dealing with the filename too unwieldy, it ain't worth it. the trick is to know when to stop. > It also integrates better into the QC system. you don't even have a good idea what a quality-control system might look like, let alone knowledge of problems that might crop up with each particular type of system. you might _think_ you do. but -- as is typical with you -- you don't know what you don't know. your knowledge has not been tempered by the big face-slap of the real-world. nor have you programmed the _apps_ that could implement such a quality-control system. when you get to that stage, then come back and we can have this discussion again, jon. > 2) During the next stage where a human being is > looking at each scan, they append the *actual* > publisher supplied page number (or string) > to the filename from (1). No need to add any > letter prefixes or anything -- they use the *actual* > string "as it is". and here's a good illustration of your lack of knowledge because of an absence of experience, combined with an ignorance of the kinds of tasks that the machine can do. (your willingness to _discuss_ an issue is quite admirable, jon, really. but that alone can only take a person so far.) in a properly designed system, the human being should _not_ have to mess with filenames at all, or only on the rare occasions of mondo weirdness. that's why i told david and all the other scanners to keep on doing whatever they are now doing, because i can deal with their stuff after-the-fact. (well, there are _some_ things i wish they would do; but it has nothing to do with the type of useless tripe you want them to have to deal with, that's for sure.) specifically, it's easy to write a routine that looks in the o.c.r. results to find the page-number of the page. (in general, i don't say something is "easy" unless i've already done it myself. because i have learned that many things that seem like they should be easy are not. i've already written this routine. it was easy to write.) if you're writing a tool to clean up a scan=set, as i am, it's pretty much _required_ that you write this routine, because you need to delete that number from the text. but before you delete it (as that will be among the last things that your tool does), you can use it to _rename_ both the o.c.r. file and the scan-file. indeed, i basically recommend that this file-renaming be one of the _first_ things you do during clean-up, because it will usually be _so_ much easier to deal with the other clean-up tasks when the filenames and the page-numbers match up. of course, what you should have done in the first place is ensured that the scans were _auto-named_ correctly, setting the auto-name counter at 1 when scanning page 1, and then next scanning each numbered page in sequence, only going back afterwards to scan out-of-sequence pages. but accidents do happen, so i programmed this routine that renames your o.c.r. files and scan-files if needed. assuming you've got a clean set of scans and the page-numbering doesn't have many anomalies, it won't take you more than a minute of two to review the new names and approve a mass change. so, once again, jon, you've made a mountain out of a molehill, and then put together a baroque "plan" on how to scale it... -bowerbird

7 8

re: [gutvol-d] Scan file naming -- another comment
by Bowerbird＠aol.com 23 Jul '05

23 Jul '05

marcello said: > If you have a sheet feeder > just extract inserted illustrations > and such out-of-sequence stuff > (making a note of the page number > on the back). Then feed the whole pile > starting with page arabic "1" > and go drink a coffee. i'm sure juliet will get a chuckle out of that... :+) -bowerbird

1 0

re: [gutvol-d] Scan file naming -- another comment
by Bowerbird＠aol.com 22 Jul '05

22 Jul '05

jon, go back and read my post in 5 years. you will understand it much better then... the overarching rule is to make the system as complex as it needs to be, and no more. -bowerbird

3 2