September 2012 - gutvol-d - lists.pglaf.org

RST as a master format, readable PG books for Kindle
by James Simmons 27 Sep '12

27 Sep '12

I think the important issue for PG is to create adequately formatted books for the Kindle and Nook. My most recent contribution to PG was this: http://www.gutenberg.org/ebooks/39442 This was really a book from hell. It had tons of family tree tables which I reproduced as ASCII art in the text version and both images and ASCII art in the HTML version. I used RST as the master format. The EPUB and Kindle versions of this book do not handle the family tree table (either version) well at all. HOWEVER, with just two changes to what PG is doing with RST submissions the EPUB and Kindle versions could have been quite adequate. Those changes are: 1). Allow pictures which are rotated in the printed book to use the same rotation in the HTML version, instead of requiring it to be rotated "right way up". Looking a pictures in a Kindle or a Nook you have the same constraints a printed book has, and sometimes you just have to rotate a picture so the width of the picture corresponds to the height of the page (or screen). If you leave the orientation of the picture alone the Kindle or Nook reader has a better experience and the web page reader has a tolerable experience. If we consider e-book readers more important that web page viewers (and I think we really should) then we should eliminate the requirement to rotate pictures. 2). Put some kind of directive in the RST file which says "this content belongs in the text file ONLY". This would have dealt with my family tree table problem nicely. Maybe have directives to replace tables wider than three columns with image files in EPUB and Kindle formats, as these readers do not handle large tables well. I found RST easy to work with and plan to use it on any future submissions I do. James Simmons

1 0

backlog blah blah blah
by Bowerbird＠aol.com 27 Sep '12

27 Sep '12

i said: > you might be surprised at some of the solidly-documented > error-reports which whitewashers have completely ignored. now some whitewasher will feel the need to come here and say, "we don't ignore your error-reports; but we do have a backlog". they might also add: "and half of the error-reports we receive are faulty, in that the so-called error is not, in fact, an error." let me grant you both of those points. (even though i have some suspicions about the second one.) my point is that the mere submission of an error-report does _not_ mean the "errors" will be fixed immediately... ...or at all... jim can tell you more about this, if you want to ask him... -bowerbird p.s. although maybe i should have let a whitewasher appear. wouldn't it be nice, every once in a while, to hear from one?

1 0

The fatal flaw
by Jon Hurst 27 Sep '12

27 Sep '12

OK, I just got round to ranting at Bowerbird for ponting out that RTT was a fatal flaw. He's an easy person to start ranting at. I hit the send button too soon perhaps. He considers RTT a fatal flaw. Roger feels pretty much the same. Actually, I don't think I have sold anyone on it. But here's a thing. I'm not wrong in stating that suggesting any master format is a worse flaw unless you happen to belong to that particular camp. So I have acheived my goal. I have found the fatal flaw that keeps this job undone. It turned out not to be PG policy, which could be worked around and actually made a certain amount of sense. It turned out not to be DP intransigence, although that might have come into play. It turns out to be simply that everyone thinks that there must be a master format, and that it has to be the one they prefer. So, I guess that is it. Thanks to everyone for the discussion, and apologies to Bowerbird for the rant... I really tried hard not to. Regards Jon

2 1

go fun, or go home
by Bowerbird＠aol.com 27 Sep '12

27 Sep '12

jon said: > We can improve the extant text through the errata system, > we can provide something better as a buried project > for those in the know and > we can provide excellent final formatted versions externally > based on these buried versions. > Maybe we can even give Amazon the nod about > the new versions -- that really would be consequential. ok, let's take those one-by-one... > We can improve the extant text through the errata system, nice thought. i wish it could happen. i wish it _would_ happen. my observation, though, is that it does not. not dependably. the _only_ time an extant text gets improved is if/when a whitewasher decides that that's what's gonna happen. and a long list of errata doesn't necessarily give them that kind of motivation. it _might_. or it might _not_. you might be surprised at some of the solidly-documented error-reports which whitewashers have completely ignored. i'm just sayin'... (and i would love for you to have success in this regard!) > we can provide something better as a buried project > for those in the know and that you can certainly do. and it's not a bad thing to do. but you had earlier ruled that out flatly, jon, said you were completely uninterested in that. it's ok to change your mind, though. :+) > we can provide excellent final formatted versions externally > based on these buried versions. and you can certainly do that as well. but you'll probably get even less traction for those files than the "buried" updates located at project gutenberg. > Maybe we can even give Amazon the nod about > the new versions -- that really would be consequential. well, it might or might not be "consequential", but i'd guess you won't get any attention from amazon. it doesn't really care much about the quality-level of the public-domain e-books which it gives away... > Unfortunately I do think DP is necessary. that will prove to be a tremendous stumbling block. greg has offered to send volunteers your way, and _that_ is most definitely the move you should make. d.p. will be nothing but a tar-baby for you. avoid! > I would love for Bowerbird to prove me wrong, > but the response to his P&P "pilot" didn't > fill me with confidence. i'm not trying to "recruit" anyone. i work alone, dude. :+) i actively reject followers. they just create drag. > there are 1000 odd people who _want_ to do the > fastiduous painstaking work that, if we're honest, > we want to avoid. if you don't make this fun, you are doing it wrong. at the very least, you need to give people tools that make the job _easy_. if you think only masochists can do digitization, your attitude has defeated you. -bowerbird

1 0

let's step through this process.002
by Bowerbird＠aol.com 27 Sep '12

27 Sep '12

so, while you guys have yet another discussion, i'm doing some work on "pride and prejudice"... *** here's the version i'm using, from archive.org: > http://archive.org/details/harvardclassicss03elio *** you'll find my work in this folder on my site: > http://zenmagiclove.com/prhpr *** the original o.c.r., from archive.org, is here: > http://zenmagiclove.com/prhpr/prhpr-000.zml subsequent edits of that file bump up the filename: > http://zenmagiclove.com/prhpr/prhpr-001.zml > http://zenmagiclove.com/prhpr/prhpr-002.zml > http://zenmagiclove.com/prhpr/prhpr-003.zml the first two of those were just to get the text into my standard page-separated skeleton, which marks the beginning of each page of o.c.r. text with a line that has _double-braces_ with the scan-name inside. the ending of each page is signified by a line with _brackets_ that hold the page-number of the page. *** "prhpr-003.zml" was the first file where i did edits. these consisted primarily of global-type changes... > http://zenmagiclove.com/prhpr/prhpr-003.zml here is a program which documents the changes that were made from the "002" to the "003" files. > http://zenmagiclove.com/prhpr/docomp23.py as you can see, if you run that script, there were 7,607 lines edited by these global-type changes. *** i've now done more edits, creating another version: > http://zenmagiclove.com/prhpr/prhpr-004.zml most of these changes were of a "global" style, but i also fixed idiosyncratic errors when i spotted 'em. as before, you can view all lines that were changed: > http://zenmagiclove.com/prhpr/docomp34.py as shown, 515 lines changed from "003" to "004". *** also, like yesterday, we can convert the .zml to .html: > http://zenmagiclove.com/prhpr/prhpr-004h.html this "entire-book-on-one-page" is in the style of the way most of the books from pg/dp are made, manifesting the "scroll" methodology of the web... the fact that you can get a rather-nicely formatted e-book using barely-reworked o.c.r. output would -- i believe -- be somewhat surprising to many of the post-processors from d.p., who seem to believe that this is work which requires a modicum of effort. (but note that the text remains unrefined o.c.r., and all the inline text-styling still has yet to be applied.) *** the new twist for today is that we have also spit out files that show each page of text alongside its scan: > http://zenmagiclove.com/prhpr/prhprp222.html you will have to adjust the zoom-level and the size of your browser-window for the best presentation. (advanced versions of this will give better control, using javascript to customize the elements better.) and of course, these pages connect with each other, via the "prev" and "next" links on each one. (you can also advance to the next page by clicking the scan.) this is the "design" you're used to seeing from me... a paginated display like this one lends itself more to smoothreading or, as i call it, "continuous proofing". but we are not ready for smoothreading yet, because we still have bad o.c.r. text that needs to be corrected. more later... -bowerbird p.s. did anyone notice the thing that i hinted about?

1 0

procedures for contributions, forum for questions and ideas
by Steve White 27 Sep '12

27 Sep '12

Hi, I have several questions. I got a nice e-book reader, and became interested in the Gutenberg collection. The collection itself is great! Some of the e-books are almost unusable on my reader, mostly due to unfortunate automatic formatting. So...I took the e-books apart, and re-formatted them. Then I tested them in several e-reader environments. I think they're very nice now. And, I would like to share these improvements, preferably through the Gutenberg project. ** Does the project accept these kinds of improvements to text? I see some talk of old and new editions in the books. ** Would such changes constitute new editions? ** What are the preferred procedures? ** Is this a good forum for general technical and policy questions regarding Gutenberg e-books? Thanks!

14 36

Readable epubs
by Robert Gibbins 27 Sep '12

27 Sep '12

I read Stevan White's recent appends about improving PG epubs with a certain amount of wry amusement. Not at Stevan's append, which I thoroughly agree with, but at the fact that I wrote a fairly similar one about 18 months ago, which was more or less completely ignored by the PG decision makers. (And I am sure I wasn't the first person to do this, as BB kindly pointed out to me at the time.) Since that time I've come to realise that PG basically isn't interested in technology, but in books, and preserving them into the digital future. The key PG technology policy, which seems to have originated from the late and much-lamented Michael S. Hart, is that the best way to do this is to require that all books should wherever possible be made available in a 'lowest common denominator' plain text format so that the books would be as universally accessible as possible. Remembering back to the 1960's, when I started programming, I can understand why that was a pretty sensible decision at the time, although as someone who started out writing in EBCDIC, I probably have less faith in US ASCII as a lingua franca than many others do. What I find a lot harder to understand is why the rule still stands today, when there is a much more universally known lingua franca called (X)HTML in which, with a certain amount of standardisation, it is possible to encode important parts of the book that can't easily be done with plain text. Basically PG has failed to provide leadership in setting standards for HTML encoding of books, and that the result is that though, as Stevan said, the books are great, in reality anyone wanting to create an epub for their reading device cannot rely either on the PG HTML, or the generated epubs. It seems to me that this is a major strategic failure of PG, which if it were a commercial organisation would by now have put it out of businesss. How did this happen? Did PG really DECIDE to limit itself to being largely an ASCII text mine/museum, or did it fail to change? Bob Gibbins

5 5

Re: [gutvol-d] procedures for contributions, forum for questions and ideas
by James Adcock 27 Sep '12

27 Sep '12

>How about trying your experiment with more recent books? The current crop are up in the 40000+ range. OK, as a "sanity check" I went back and double-checked more recent submissions to see if they are no being formatted "reasonably correctly." IE, if this was an e-book that I had bought commercially would I say "yes, this formatted reasonably" or "no, this book has corrupted formatting." Not a high standard, just: does this book "work" or not? 40000 no 40001 no 40002 no 40003 no 40004 no 40005 no 40006 no 40007 no 40008 no 40009 no The two most common and glaring problems are: 1) Paragraphs are not formatted "reasonably" corresponding to any known standard of formatting at any point in time of mankind. And 2) No TOC in an e-book where one would reasonably expect and need a TOC.

5 8

let's step through this process
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

don said: > Trac i've explained to y'all many times how to do this job simply. yet you continue to flock to the most complicated processes. so go ahead and do it that way; you will get what you deserve. *** for the people who wanna actually get something accomplished, though, i'll step through a process to show how it can be done... *** here's a copy of "pride and prejudice" at archive.org: > http://archive.org/details/harvardclassicss03elio you want to use a scan-set from archive.org or google, because those are canonical copies available to the public. no one has to "take your work" that this is an edition that is true to the author, because it'd been sitting in a library. and no one can accuse you of "doctoring" the scans, since the scan-set is being housed by an entity external to you, and can be fully accessed independently of your influence. a scan-set from a book you bought at barnes&noble has _neither_ of those two necessary and sufficient qualities. *** to work on that edition, i set up this folder on my site: > http://zenmagiclove.com/prhpr first off, i copied over all of the scans from archive.org. "pride and prejudice" was reprinted with another book, meaning that it didn't actually even begin until page 141, so the scans start with page 141, and continue up to 512. > http://zenmagiclove.com/prhpr/prhprp141.jpg > http://zenmagiclove.com/prhpr/prhprp512.jpg (512 is actually a blank page, but we need an even number.) you will find the original .djvu o.c.r. text-file in that folder: > http://zenmagiclove.com/prhpr/harvardclassicss03elio_djvu.txt i also re-saved that file with a new name, befitting its new folder: > http://zenmagiclove.com/prhpr/prhpr-000.zml i did some reworking of the file, mostly to show pagebreaks: > http://zenmagiclove.com/prhpr/prhpr-001.zml i think you can pretty clearly see this is just a text-evolution. then i did more work, for correct paging, and some niceties: > http://zenmagiclove.com/prhpr/prhpr-002.zml i think you can still see that this is just more text-evolution. then i wrote a program to turn that .zml file into rough .html: > http://zenmagiclove.com/prhpr/prhpr-002h.html and again, the relationship to the earlier .zml file is very direct. then i did a few global-type changes to start cleaning the o.c.r.: > http://zenmagiclove.com/prhpr/prhpr-003.zml then, to show how simple it can be to use a diffing procedure, i wrote a python script that compares 002.zml against 003.zml: > http://zenmagiclove.com/prhpr/docomp.py if you run that script -- by just clicking on that link there -- you'll see the lines that were edited within the first 2000 lines. the top line's from prhpr-002.zml, bottom from prhpr-003.zml. there is also a "change-line" below the other 2 lines, which shows where the lines differ (as sometimes it's hard to see the change). for instance, the change-lines of the diffs on lines 176 and 183 reveal that i globally deleted the space in front of a semicolon... the leftmost part of the change-line is a link that opens the scan for the relevant page in another window. (it will continue to open the scan in a named window, so you should make sure that it _is_ a window, not a tab, so you'll see both windows at the same time.) i have coded compare programs which are much more powerful, but this one serves its purpose as a little app to demo comparison. more tomorrow. but for now, the takeaway: none of this is hard. -bowerbird p.s. there's something very revealing in those .html versions, so take a little look at them and see if you can tell me what that is...

1 0

diffing as a superior methodogy to word-by-word proofing
by Bowerbird＠aol.com 26 Sep '12

26 Sep '12

robert said: > These appends seem to me to imply that > diffing parallel/independently produced texts > is a faster and more efficient way > to correct the OCR process > than sequential manual checking. much faster. much more efficient. > Has anyone (possibly including DP) > ever tried any of the above, and > documented the results in a scientifically valid way? well, yeah. just one week from today, it will be 7 full years since i posted this: > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-03,3 > Does DP work that way anyway? nope. and they didn't want to learn, either. because, at the end of 2006, i posted this in a d.p. forum: > http://www.pgdp.net/phpBB2/viewtopic.php?t=24008 *** much faster. much more efficient. absolutely no comparison. it's a complete waste of time to proof by reading word-by-word. *** by now it should be clear to everyone why i am getting so tired of repeating myself. -bowerbird

1 0