- gutvol-d - lists.pglaf.org

re: [gutvol-d] Re: DPF images archives [Was: Re: Kevin Kelly ...]
by Bowerbird＠aol.com 27 May '06

27 May '06

robert said: > For one thing, the DP pages # rarely correspond to the physical page #s, > and each book will require manual intervention to determine page numbers, > short of some kind of re-ocr and automatic page # extraction from the headers. > Personally, I'm betting on manual intervention. :) actually, i have a program that aids greatly in that regard. it'll display each scan so you can get the page-number from it. if there's a mismatch between file-name and page-number, you just enter the correct page-number, so it'll be renamed. the time-saver is an auto-increment button, which you click if the page-number is the next one in sequence, which both fills in the rename-info and steps to the next page, meaning you use that for most of the files, so the task goes fairly fast. the actual renaming takes place as a batch process at the end. however... i've also used a variation on the other process you mentioned. if for some reason you screwed up the file-name/page-number relationship when you scanned the book, you can fix it quickly. since the text-file from the o.c.r. process has the _same_name_ as the image-file from which it came, you can have the program read the text-file and do an "extraction" of the page-number -- it's almost always the number one higher than the previous, and mostly, it'll be in the running-head, so in the first line of the file, which is why you must always make sure to scan running-heads, and if it's an even-number it will be on the left, odd on the right, except on pages where a new chapter starts, when it's often gone entirely, or (equally often) centered at the _bottom_ of the page, which does mean, yes sir, that that is one of those 30 variables that are an indication that this specific page starts a new chapter, in which case the first line(s) of that page constitute the header, and i do apologize for the run-on digression this has become -- thus you can use that information to rename both the image-file and the text-file so that they reflect the correct page-number... unlike the former process, which is mostly manual, this one is mostly automatic, you just have to give the file-renaming list a quick once-over to make sure that it got everything correct. again, the renaming takes place as a batch process at the end. even a book with unnumbered plate-pages intermixed can be fixed in fairly short order with this program. it's pretty sweet... it's best to run this correction _immediately_, before you even start to do any other processing on the files, because it's simply a waste of time to work with inaccurately-named files. however, you can even use it after-the-fact on completely-processed books. (unless you've discarded or combined all the individual text-files.) i use this app most frequently, however, after scraping a scan-set at google or distributed proofreaders. after displaying all the pages, i copy all of those files from the browser-cache into a separate folder; the program can sort the files according to their modification-time, which puts them in the correct order (since their alphanumeric names from the cache are often not in alphabetic order), and i get the first name in sync, and then just keep on clicking the auto-increment button. slick. anyway, i'm sure you scripting cats can whip up a version of this program on your own, now that i've described how it works. but if you do need this program i've written, let me know what operating system you want it for, and i'll compile you out a copy... -bowerbird

1 0

re: [gutvol-d] Re: DPF images archives [Was: Re: Kevin Kelly ...]
by Bowerbird＠aol.com 26 May '06

26 May '06

donovan said: > This is contingent on getting the archived DP files in > order, a task which I have yet to ascertain the size of. um, that will be a big task. a very big task. so it will take a while. a good while. but hey, we knew that anyway. right? yes we did. it's sure nice to know you're taking the first step! :+) -bowerbird

1 0

FAQ item on page scans
by Greg Newby 26 May '06

26 May '06

David Widger reminded me that we actually have an FAQ posted about page scans: http://www.gutenberg.org/faq/S-21 It's comparable to the note I forwarded from Jim, with some variation in point #4. Marcello's method is certainly fine by me. Ditto for BB's. Other variations welcome. To get this into the workflow (from DP & elsewhere), we'll want to standardize a bit more, but right now I'll be happier getting some more hands-on experience with the workflow, and feedback. At this point, let's try to get a few dozen done, in a few different ways, and take a look at accessibility/utility. There was lots and lots of discussion about this back in 2004, but we've only done a few. Here are the eBook #s for items I've found with a "page-images" file, for your consideration: 10463 http://www.gutenberg.org/etext/10463 12973 http://www.gutenberg.org/etext/12973 13254 http://www.gutenberg.org/etext/13254 14040 http://www.gutenberg.org/etext/14040 14100 http://www.gutenberg.org/etext/14100 14101 http://www.gutenberg.org/etext/14101 14116 http://www.gutenberg.org/etext/14116 15874 http://www.gutenberg.org/etext/15874 16122 http://www.gutenberg.org/etext/16122 16413 http://www.gutenberg.org/etext/16413 17061 http://www.gutenberg.org/etext/17061 17068 http://www.gutenberg.org/etext/17068 17226 http://www.gutenberg.org/etext/17226 17469 http://www.gutenberg.org/etext/17469 17472 http://www.gutenberg.org/etext/17472 17799 http://www.gutenberg.org/etext/17799 -- Greg

2 1

Repost: RFC: Posting Page Scans in DJVU Format]
by Marcello Perathoner 26 May '06

26 May '06

-------- Original Message -------- Date: Sun, 17 Jul 2005 00:30:44 +0200 To: Project Gutenberg Volunteer Discussion PREFACE PG allows the posting of page images along an existing ebook. The only currently accepted format is tiff files collected in a zip archive. This format is cumbersome for the reader, wasteful on storage space and doesn't allow live deep linking to the page images. SCOPE This RFC specifies an alternative format that is less cumbersome for the reader, less wasteful of storage space and allows live deep linking to the page images. This new format will not replace the old format. Page images for any book can be posted in whichever of the 2 formats the poster chooses. The format specified here will permit online viewing of the page images (with djvu plugin). It is not required to download the file and unpack it as with the old format, although it is still possible to do so. The new format also allows for linking from an html document to an arbitrary page image, so that a click on a link will open the right page image in the djvu browser plugin. The new format compresses text pages 2-3 times better than the old format with no loss of readability or ocr-ability. The new format has GPLed compressors and decompressors. Even if the format should happen to become encumbered by patents or other licensing issues, PG will be able to easily and automatically convert the format into a legally unencumbered format. FORMAT The page images for a book will be posted into the main ebook directory in one multi-page djvu file. The file will be named #####.djvu (replace ##### with the ebook number). The multi-page djvu file contains a collection of single-page djvu files. Each single-page djvu file will contain one single-sided page of the book (cover, back and spine also count as single-sided pages) or an illustration scanned in a different resolution or color depth. Numbering / Naming of page files A book usually contains 2 page number sequences, a roman one followed by an arabic one. We considered the cover pages as yet another sequence. A filename for a single-page djvu file MUST follow this pattern: <prefix><page number>.djvu The prefix for the cover pages is: "c". The prefix for the roman pages is: "f". The prefix for the arabic pages is: "p". If there are more page number sequences in the book, they MUST be handled in a similar fashion, using an arbitrary free letter. The <page number> is the true page number as seen on the physical page (or inferred from the previous / next pages) expressed in arabic numerals and left-padded with zeroes to a length of 4 digits. For blank pages there should be no file and the page number should be skipped. Optionally an image saying: "This page is blank in the original." may be inserted. Missing pages MUST be replaced by an image saying: "This page is missing." A filename for a single-page djvu file containing an illustration scanned in a different resolution or color depth MUST follow this pattern: <prefix><page number>-<image position on the page>.djvu The <image position on the page> is "1" for the first image, "2" for the second, etc. If present, front cover, back cover and spine MUST be named as follows: front cover outside: c0001.djvu front cover inside: c0002.djvu back cover inside: c0003.djvu back cover outside: c0004.djvu spine: c0005.djvu Example of file naming: front cover c0001.djvu back cover c0004.djvu spine c0005.djvu i title page f0001.djvu ii title verso f0002.djvu iii dedication f0003.djvu iv is blank v contents f0005.djvu page 1 p0001.djvu page 2 p0002.djvu image on page 2 p0002-1.djvu image on page 2 p0002-2.djvu page 3 p0003.djvu page 4 is blank page 5 p0005.djvu ... ... page 9999 p9999.djvu Compression To produce the single-page djvu files you should use the most appropriate compressor: lossy jb2 for bitonal text, iw44 for continuous-tone images. Assemblage All single-page djvu files MUST be assembled into a multi-page djvu file. Only the multi-page djvu file will be posted. All "roman" and "arabic" pages MUST appear in the same order in the multi-page file as in the book. All cover pages and the book spine should appear at the front. The naming scheme was chosen so that saying: djvm -c 12345.djvu *djvu in a directory containing all single-page djvu files will assemble the multi-page djvu file in the correct sequence. APPENDIX Open Source compressors and browser plugins for Linux and Windows are available here: http://djvulibre.djvuzone.org/ Windows users also have to get and install Cygwin from here: http://www.cygwin.com/ -- Marcello Perathoner webmaster(a)gutenberg.org

1 0

re: [gutvol-d] Page scans draft policy
by Bowerbird＠aol.com 26 May '06

26 May '06

someone said: > 4. Each page image should be a separate file > and named with the page number within the set; > e.g. 001.tif, 002.tif, etc. Separate, non-page images, > such as covers or color images scanned separately > from the pages, should have suitable names, > such as "cover.jpg" or "072-image.tif" this is a bad policy. a wonky naming convention screws everything up. (and an inconsistent convention is a wonky one.) also, it's absolutely imperative that a library have _unique_filenames_ for every single file within it. naming files "001.tif" and expecting their _folder_ to differentiate them is a disaster-in-the-making. for a better way of doing things, check any of the links to the .html files i gave in an earlier message. all of them incorporate the image-scans. you want an alphabetical sort of the filenames to give you the _exact_ order in which the p-book pages were bound... > All page images for the book will be zipped into one file this is a bad policy too. you need to have each image individually accessible. it is _tremendously_ important that this be the case... > we will not post separate page images in that directory, > since that would double the space used, and we believe that > people who want to consult the images will probably want them all. that's a bad assumption. there are all kinds of situations where people want a single scan. (ponder your own experience...) and these scans _need_ to be accessible via the web, not just as a download package. again, very important! if it comes down to a choice between saving them in a zip file and saving them individually, toss the zip file _immediately_... *** your policy on the way in which the scans are saved simply must reflect the realities about how those scans will eventually be used. spend some time analyzing those uses so you make good decisions. -bowerbird p.s. here are those links again, straight into their subdirectories: > http://snowy.arsc.alaska.edu/bowerbird/mabie/ > http://snowy.arsc.alaska.edu/bowerbird/myant/ > http://snowy.arsc.alaska.edu/bowerbird/tolbk/ > http://snowy.arsc.alaska.edu/bowerbird/sgfhb/ > http://snowy.arsc.alaska.edu/bowerbird/henty/ > http://snowy.arsc.alaska.edu/bowerbird/bachwm/ > http://snowy.arsc.alaska.edu/bowerbird/ahmmw/ the .html files are on another machine; you can't see them as a group, but you can view-source on any one of them to see how they operate. in each case, the .html files were auto-generated from the .zml file, and the image-file is spliced in to facilitate "continuous proofreading". > http://www.greatamericannovel.com/ahmmw/ahmmw.zml > http://www.greatamericannovel.com/ahmmw/ahmmw.html > http://www.greatamericannovel.com/mabie/mabie.zml > http://www.greatamericannovel.com/mabie/mabiep001.html > http://www.greatamericannovel.com/myant/myant.zml > http://www.greatamericannovel.com/myant/myantc001.html > http://www.greatamericannovel.com/sgfhb/sgfhb.zml > http://www.greatamericannovel.com/sgfhb/sgfhbc001.html

1 0

re: [gutvol-d] DPF images archives [Was: Re: Kevin Kelly ...]
by Bowerbird＠aol.com 26 May '06

26 May '06

donovan said: > It's an interesting idea, but initially I'd like to focus > on getting the existing projects in order. :) the volunteer who does the work sets the agenda. :+) but putting up existing scan-sets, so people could read them while they are standing in line waiting to be digitized, seems to me to be the best possible way to put them to use _now_... and if anyone needs another good idea, i think it would be a good idea to round up the scan-sets from google that represent books that are already in the p.g. library... a list of such books has been compiled on the d.p. forums. the focus of that list is now "...so don't bother with these...", but i think they could instead represent a rare opportunity... -bowerbird

4 5

Page scans draft policy
by Greg Newby 26 May '06

26 May '06

As I said, there was subsequent discussion about the details of formatting...Here's some info about page scan formats. I note that point 4 is somewhat different than what I just typed in my other message, and seems a whole lot smarter. -- Greg ----- Forwarded message from Jim Tinsley <jtinsley(a)pobox.com> ----- From: "Jim Tinsley" <jtinsley(a)pobox.com> To: "Posted Etexts for Project Gutenberg" <posted(a)listserv.unc.edu> Subject: [posted] Posted (#12973, Butler) ! Date: Tue, 20 Jul 2004 20:24:32 -0700 (PDT) Personal Recollections of Pardee Butler, by Pardee Butler 12973 [Editor: Mrs. Rosetta B. Hastings] [Contributor: Mrs. Rosetta B. Hastings] [Contributor: Elder John Boggs] [Contributor: Elder J. B. McCleery] [Link: http://www.gutenberg.net/1/2/9/7/12973 ] [Files: 12973.txt; 12973-h.htm; 12973-page-images] Thanks to Roger for finding and scanning this book. This is the first PG book to be posted with page images. We are now beginning to accept page images along with the regular postings. Of course, DP has always preserved its page images, and those will eventually be uploaded in a big batch, or series of batches, but non-DP contributions may now begin adding page images. For now, we're setting the following guidelines for page image postings: 1. PG is now accepting page images of books posted. Page images will be posted _only_ as an addition to an etext posted in the normal way -- we will not post page images without plain text. 2. Page images are an option; they are not and will not be required for the posting of a text. 3. All page images should be good enough to work reasonably well with OCR packages, up to 600 dpi, and should be stored as black-and-white TIFFs with CCITT-4 (aka ITU-G4 or Fax Group 4) compression. This is important, so that we keep the overall file size down to a sustainable level. With this compression, a typical 600dpi page can be stored for about 40KB. Our ability to post these images depends on the file sizes staying fairly reasonable. Pages such as color pictures or greyscale photos that cannot reasonably be stored as black-and-white only should be stored as TIFF or JPEG with the best compression you can get for that image. (Note: Irfanview for Windows does this nicely individually or in batch. ImageMagick v 6.x: convert myimage.png -compress group4 myimage.tif ) 4. Each page image should be a separate file and named with the page number within the set; e.g. 001.tif, 002.tif, etc. Separate, non-page images, such as covers or color images scanned separately from the pages, should have suitable names, such as "cover.jpg" or "072-image.tif" All page images for the book will be zipped into one file, to be called FILENUMBER-page-images, e.g. 12345-page-images.piz (reverse the extension) for etext #12345, and stored in the main directory for that etext. It will unzip to a subdirectory ./page-images, but we will not post separate page images in that directory, since that would double the space used, and we believe that people who want to consult the images will probably want them all. So, for now at least, if you want the images, you download the PIZ (backwards again) file. jim ----- End forwarded message -----

1 0

re: [gutvol-d] re: accurately converted to any possible format
by Bowerbird＠aol.com 26 May '06

26 May '06

greg said: > I don't understand. Where is the URL that > does conversion on the fly from that file > to arbitrary formats? PDF, HTML...others... > with user-specified settings. there is no u.r.l. conversions are done by my viewer-app. you don't need to be involved, since your users can do it all by themselves. > There's enough space on /data1, > where your home directory is, > to make your own copy of > as much the collection as you want > in your own directory. great! i'll figure it out. thank you! -bowerbird

3 2

re: [gutvol-d] re: accurately converted to any possible format
by Bowerbird＠aol.com 25 May '06

25 May '06

sorry, i screwed up that last post, and forgot to include this: greg said: > Your earlier message said you could offer > reformatting into any reasonable format, > as long as the input was sufficiently > well-formed to your standards. actually, as the header shows, i said "any possible format", which is kind of ludicrous in retrospect, isn't it? so let's be precise about exactly _what_ formats we mean, in the future, and let us further provide _samples_ so that people can evaluate the _quality_ of the conversions we do. otherwise, it's just a hype war, and that does no one any good. > Sounds like you don't actually have any such thing. it's not released yet, no. but i have pointed to samples. > Put it up for free public download, > and I'll change my tune in a heartbeat. would that you were so demanding of the .tei folks. *** now, let me restate, just to remind everybody again, i have no objection to the .tei folks, or the .xml folks. i don't even have an objection that .tei is the "official" position on how project gutenberg moves to the future. i merely wish to assert _my_ opinion, which i will back up with solid evidence, that a much simpler methodology will give substantially similar (if not better) benefits, at a cost (both initial and maintenance) that is _significantly_ lower. even then, if people want to stick with the .tei/.xml method, that's fine with me, as it is no skin off my nose. comprenez? -bowerbird

1 0

re: accurately converted to any possible format
by Bowerbird＠aol.com 25 May '06

25 May '06

greg said: > So you're against storing (or creating) WAP versions, > Braille versions and MP3 versions on our server? is that a serious question? i doubt it. but just in case, my answer is no, i'm not "against" that. indeed, i'd like to support all of that. in my offline app. i already have the option of text-to-speech saved as mp3, but you might be talking about human audio versions... (if you do mean computerized text-to-speech, it seems to me that that could be done offline too, but if you want to create, store, and serve such files, i have no objection.) as for braille, i don't have the slightest idea how to do that. and i'm almost as clueless when it comes to wap, sorry... but i'm sure z.m.l. is simple and straightforward enough that anyone who does know about these formats could write a conversion program that would do the job well... and i would retain my preference for offline converters. but i think it'd be _great_ to have online converters too! it's just that you'll need a web-programmer, i can't help, at least for now, because i've just done a little perl stuff. still, even my brief experimentation has demonstrated perl can do such conversions from z.m.l. quite easily. and i'd assume that python and ruby are just as facile. > That's cutting out a whole lot of > potential readers I would like to support. i don't think you should cut anyone out. > Nobody's stopping anyone from running a program > on their own computer to do whatever conversion they like. i should hope not. besides, it's unlikely they could prevent it. but all that is beside the point, isn't it? the point is to _enable_ people to do these conversions. right? > Why are you trying to stop me from enabling various conversions on > the server, for people who want or need to get conversion done there? i'm not trying to stop you. i'm cheering you on. go greg! but if you are telling me .tei/.xml is the best way (let alone the only way!) to do what you want to do, then we really should have a long discussion. -bowerbird

1 0