- gutvol-d - lists.pglaf.org

go fun, or go home
by Bowerbird＠aol.com 27 Sep '12

27 Sep '12

jon said: > We can improve the extant text through the errata system, > we can provide something better as a buried project > for those in the know and > we can provide excellent final formatted versions externally > based on these buried versions. > Maybe we can even give Amazon the nod about > the new versions -- that really would be consequential. ok, let's take those one-by-one... > We can improve the extant text through the errata system, nice thought. i wish it could happen. i wish it _would_ happen. my observation, though, is that it does not. not dependably. the _only_ time an extant text gets improved is if/when a whitewasher decides that that's what's gonna happen. and a long list of errata doesn't necessarily give them that kind of motivation. it _might_. or it might _not_. you might be surprised at some of the solidly-documented error-reports which whitewashers have completely ignored. i'm just sayin'... (and i would love for you to have success in this regard!) > we can provide something better as a buried project > for those in the know and that you can certainly do. and it's not a bad thing to do. but you had earlier ruled that out flatly, jon, said you were completely uninterested in that. it's ok to change your mind, though. :+) > we can provide excellent final formatted versions externally > based on these buried versions. and you can certainly do that as well. but you'll probably get even less traction for those files than the "buried" updates located at project gutenberg. > Maybe we can even give Amazon the nod about > the new versions -- that really would be consequential. well, it might or might not be "consequential", but i'd guess you won't get any attention from amazon. it doesn't really care much about the quality-level of the public-domain e-books which it gives away... > Unfortunately I do think DP is necessary. that will prove to be a tremendous stumbling block. greg has offered to send volunteers your way, and _that_ is most definitely the move you should make. d.p. will be nothing but a tar-baby for you. avoid! > I would love for Bowerbird to prove me wrong, > but the response to his P&P "pilot" didn't > fill me with confidence. i'm not trying to "recruit" anyone. i work alone, dude. :+) i actively reject followers. they just create drag. > there are 1000 odd people who _want_ to do the > fastiduous painstaking work that, if we're honest, > we want to avoid. if you don't make this fun, you are doing it wrong. at the very least, you need to give people tools that make the job _easy_. if you think only masochists can do digitization, your attitude has defeated you. -bowerbird

1 0

let's step through this process.002
by Bowerbird＠aol.com 27 Sep '12

27 Sep '12

so, while you guys have yet another discussion, i'm doing some work on "pride and prejudice"... *** here's the version i'm using, from archive.org: > http://archive.org/details/harvardclassicss03elio *** you'll find my work in this folder on my site: > http://zenmagiclove.com/prhpr *** the original o.c.r., from archive.org, is here: > http://zenmagiclove.com/prhpr/prhpr-000.zml subsequent edits of that file bump up the filename: > http://zenmagiclove.com/prhpr/prhpr-001.zml > http://zenmagiclove.com/prhpr/prhpr-002.zml > http://zenmagiclove.com/prhpr/prhpr-003.zml the first two of those were just to get the text into my standard page-separated skeleton, which marks the beginning of each page of o.c.r. text with a line that has _double-braces_ with the scan-name inside. the ending of each page is signified by a line with _brackets_ that hold the page-number of the page. *** "prhpr-003.zml" was the first file where i did edits. these consisted primarily of global-type changes... > http://zenmagiclove.com/prhpr/prhpr-003.zml here is a program which documents the changes that were made from the "002" to the "003" files. > http://zenmagiclove.com/prhpr/docomp23.py as you can see, if you run that script, there were 7,607 lines edited by these global-type changes. *** i've now done more edits, creating another version: > http://zenmagiclove.com/prhpr/prhpr-004.zml most of these changes were of a "global" style, but i also fixed idiosyncratic errors when i spotted 'em. as before, you can view all lines that were changed: > http://zenmagiclove.com/prhpr/docomp34.py as shown, 515 lines changed from "003" to "004". *** also, like yesterday, we can convert the .zml to .html: > http://zenmagiclove.com/prhpr/prhpr-004h.html this "entire-book-on-one-page" is in the style of the way most of the books from pg/dp are made, manifesting the "scroll" methodology of the web... the fact that you can get a rather-nicely formatted e-book using barely-reworked o.c.r. output would -- i believe -- be somewhat surprising to many of the post-processors from d.p., who seem to believe that this is work which requires a modicum of effort. (but note that the text remains unrefined o.c.r., and all the inline text-styling still has yet to be applied.) *** the new twist for today is that we have also spit out files that show each page of text alongside its scan: > http://zenmagiclove.com/prhpr/prhprp222.html you will have to adjust the zoom-level and the size of your browser-window for the best presentation. (advanced versions of this will give better control, using javascript to customize the elements better.) and of course, these pages connect with each other, via the "prev" and "next" links on each one. (you can also advance to the next page by clicking the scan.) this is the "design" you're used to seeing from me... a paginated display like this one lends itself more to smoothreading or, as i call it, "continuous proofing". but we are not ready for smoothreading yet, because we still have bad o.c.r. text that needs to be corrected. more later... -bowerbird p.s. did anyone notice the thing that i hinted about?

1 0

procedures for contributions, forum for questions and ideas
by Steve White 27 Sep '12

27 Sep '12

Hi, I have several questions. I got a nice e-book reader, and became interested in the Gutenberg collection. The collection itself is great! Some of the e-books are almost unusable on my reader, mostly due to unfortunate automatic formatting. So...I took the e-books apart, and re-formatted them. Then I tested them in several e-reader environments. I think they're very nice now. And, I would like to share these improvements, preferably through the Gutenberg project. ** Does the project accept these kinds of improvements to text? I see some talk of old and new editions in the books. ** Would such changes constitute new editions? ** What are the preferred procedures? ** Is this a good forum for general technical and policy questions regarding Gutenberg e-books? Thanks!

14 36

Readable epubs
by Robert Gibbins 27 Sep '12

27 Sep '12

I read Stevan White's recent appends about improving PG epubs with a certain amount of wry amusement. Not at Stevan's append, which I thoroughly agree with, but at the fact that I wrote a fairly similar one about 18 months ago, which was more or less completely ignored by the PG decision makers. (And I am sure I wasn't the first person to do this, as BB kindly pointed out to me at the time.) Since that time I've come to realise that PG basically isn't interested in technology, but in books, and preserving them into the digital future. The key PG technology policy, which seems to have originated from the late and much-lamented Michael S. Hart, is that the best way to do this is to require that all books should wherever possible be made available in a 'lowest common denominator' plain text format so that the books would be as universally accessible as possible. Remembering back to the 1960's, when I started programming, I can understand why that was a pretty sensible decision at the time, although as someone who started out writing in EBCDIC, I probably have less faith in US ASCII as a lingua franca than many others do. What I find a lot harder to understand is why the rule still stands today, when there is a much more universally known lingua franca called (X)HTML in which, with a certain amount of standardisation, it is possible to encode important parts of the book that can't easily be done with plain text. Basically PG has failed to provide leadership in setting standards for HTML encoding of books, and that the result is that though, as Stevan said, the books are great, in reality anyone wanting to create an epub for their reading device cannot rely either on the PG HTML, or the generated epubs. It seems to me that this is a major strategic failure of PG, which if it were a commercial organisation would by now have put it out of businesss. How did this happen? Did PG really DECIDE to limit itself to being largely an ASCII text mine/museum, or did it fail to change? Bob Gibbins

5 5

Re: [gutvol-d] procedures for contributions, forum for questions and ideas
by James Adcock 27 Sep '12

27 Sep '12

>How about trying your experiment with more recent books? The current crop are up in the 40000+ range. OK, as a "sanity check" I went back and double-checked more recent submissions to see if they are no being formatted "reasonably correctly." IE, if this was an e-book that I had bought commercially would I say "yes, this formatted reasonably" or "no, this book has corrupted formatting." Not a high standard, just: does this book "work" or not? 40000 no 40001 no 40002 no 40003 no 40004 no 40005 no 40006 no 40007 no 40008 no 40009 no The two most common and glaring problems are: 1) Paragraphs are not formatted "reasonably" corresponding to any known standard of formatting at any point in time of mankind. And 2) No TOC in an e-book where one would reasonably expect and need a TOC.

5 8

let's step through this process
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

don said: > Trac i've explained to y'all many times how to do this job simply. yet you continue to flock to the most complicated processes. so go ahead and do it that way; you will get what you deserve. *** for the people who wanna actually get something accomplished, though, i'll step through a process to show how it can be done... *** here's a copy of "pride and prejudice" at archive.org: > http://archive.org/details/harvardclassicss03elio you want to use a scan-set from archive.org or google, because those are canonical copies available to the public. no one has to "take your work" that this is an edition that is true to the author, because it'd been sitting in a library. and no one can accuse you of "doctoring" the scans, since the scan-set is being housed by an entity external to you, and can be fully accessed independently of your influence. a scan-set from a book you bought at barnes&noble has _neither_ of those two necessary and sufficient qualities. *** to work on that edition, i set up this folder on my site: > http://zenmagiclove.com/prhpr first off, i copied over all of the scans from archive.org. "pride and prejudice" was reprinted with another book, meaning that it didn't actually even begin until page 141, so the scans start with page 141, and continue up to 512. > http://zenmagiclove.com/prhpr/prhprp141.jpg > http://zenmagiclove.com/prhpr/prhprp512.jpg (512 is actually a blank page, but we need an even number.) you will find the original .djvu o.c.r. text-file in that folder: > http://zenmagiclove.com/prhpr/harvardclassicss03elio_djvu.txt i also re-saved that file with a new name, befitting its new folder: > http://zenmagiclove.com/prhpr/prhpr-000.zml i did some reworking of the file, mostly to show pagebreaks: > http://zenmagiclove.com/prhpr/prhpr-001.zml i think you can pretty clearly see this is just a text-evolution. then i did more work, for correct paging, and some niceties: > http://zenmagiclove.com/prhpr/prhpr-002.zml i think you can still see that this is just more text-evolution. then i wrote a program to turn that .zml file into rough .html: > http://zenmagiclove.com/prhpr/prhpr-002h.html and again, the relationship to the earlier .zml file is very direct. then i did a few global-type changes to start cleaning the o.c.r.: > http://zenmagiclove.com/prhpr/prhpr-003.zml then, to show how simple it can be to use a diffing procedure, i wrote a python script that compares 002.zml against 003.zml: > http://zenmagiclove.com/prhpr/docomp.py if you run that script -- by just clicking on that link there -- you'll see the lines that were edited within the first 2000 lines. the top line's from prhpr-002.zml, bottom from prhpr-003.zml. there is also a "change-line" below the other 2 lines, which shows where the lines differ (as sometimes it's hard to see the change). for instance, the change-lines of the diffs on lines 176 and 183 reveal that i globally deleted the space in front of a semicolon... the leftmost part of the change-line is a link that opens the scan for the relevant page in another window. (it will continue to open the scan in a named window, so you should make sure that it _is_ a window, not a tab, so you'll see both windows at the same time.) i have coded compare programs which are much more powerful, but this one serves its purpose as a little app to demo comparison. more tomorrow. but for now, the takeaway: none of this is hard. -bowerbird p.s. there's something very revealing in those .html versions, so take a little look at them and see if you can tell me what that is...

1 0

diffing as a superior methodogy to word-by-word proofing
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

robert said: > These appends seem to me to imply that > diffing parallel/independently produced texts > is a faster and more efficient way > to correct the OCR process > than sequential manual checking. much faster. much more efficient. > Has anyone (possibly including DP) > ever tried any of the above, and > documented the results in a scientifically valid way? well, yeah. just one week from today, it will be 7 full years since i posted this: > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-03,3 > Does DP work that way anyway? nope. and they didn't want to learn, either. because, at the end of 2006, i posted this in a d.p. forum: > http://www.pgdp.net/phpBB2/viewtopic.php?t=24008 *** much faster. much more efficient. absolutely no comparison. it's a complete waste of time to proof by reading word-by-word. *** by now it should be clear to everyone why i am getting so tired of repeating myself. -bowerbird

1 0

straight quotes
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

carlo said: > For example, one cannot assume that > a ' between letters is an apostrophe, hence > rendered with a right single quotation mark > (do you know the standard exception in English?) no, carlo, i don't know the exception. what is it? -bowerbird

2 1

might have a suitable dead trees edition
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

greg said: > There ARE master scans available for a number of eBooks. not for "classics" that were posted early and are downloaded often. > I'm going to visit B&N today to see whether they > might have a suitable dead trees edition for this purpose. it'd be a _very_ bad idea to use such a version as a "master". the good part about that idea is that a modern printing will give very clean o.c.r. if that o.c.r. was then proofed against a _worthy_ scan-set -- one that was published "back then" -- it _might_ (emphasis on "might") save a bunch of proofing... (depending on the source used by the current-day publisher). it's worth an experiment, but i would be very sensitive to whether it was actually working well, and quit if it wasn't. -bowerbird

1 0

blahblahblahblah
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

jon said: > 1. Develop a methodology for nominating master scans > where no record of provenance if extant text exists. > > 2. Source master scans and upload to PG. > > 3. Run master scans through DP until P2 > using LOTE non-clothing exception. > Capture P2 output and diff it against extant text > to produce the RTT. Note that if P2 output is perfect > the RTT _is_ P2 output. > > 4. Diff RTT against extant text > to produce comprehensive errata list, > and deliver to WWs so they can update the e-texts. > > And that, for the moment, is the limit of my ambition. > As Bowerbird points out, I have more than likely > just described an impossible scenario, even though > from a technical point of view it is completely trivial. well, yes, it's "impossible" because the people who would be necessary to make it work have no interest in doing it. but even if they did, this isn't really "completely trivial"... the twist is your assumption that there _must_ be a solid and usable provenance for the existing e-texts. for some of the classic e-texts, though, the "source" was an amalgamation of several different editions... so you're not gonna find a scan-set for "provenance". and for others, the "source" document was a _reprint_ done by the "classics" imprint of a corporate publisher. so you're not going to find a scan-set for those, either. (and even if you did, you wouldn't want to use it, since those imprints often "took liberties" with the real text.) therefore, steps 3 and 4 in your list simply won't work, not in the way that you've described them. furthermore, the whitewashers aren't gonna accept any "error-reports" that might be based on your work, because your scan-set wasn't the actual source of the currently-existing e-text. (the fact that the currently-existing file _has_no_source_ is of little concern to them, or to anybody else, it'd seem, except to some of us obsessive-compulsive fuss-budgets, although i've predicted, for some time, that will change.) the simple solution, of course, would be to replace the unsourced file with the one that has a solid provenance. but, as we see, p.g. doesn't seem to want to do that... -bowerbird p.s. it's also worth stating that even a replacement will _not_ be a cure-all. many end-users get their files from "republishers" -- manybooks.com, feedbooks.com, etc. -- and i'm not sure if those republishers go back and re-do the low-numbered "classics", even when they're replaced.

1 0