October 2012 - gutvol-d - lists.pglaf.org

Dumbed-down outline of PG process
by don kretz 05 Oct '12

05 Oct '12

This is absolutely without regard to method, technoiogy, markup, standards, or responsible parties. Those can come later, but this is to enumerate the steps in the PG workflow (although "flow" is a stretch in some cases) in order to provide scope for any discussion of PG workflow per Greg Newby's post. These have to be accomplished one way or another in order for a book to become an ebook in the PG catalog, and support multiple format output and ongoing refinement and error correction. Some can be simultaneous and there's overlap. Some may be provided by external sources (e.g. images.) Participants include volunteers, PG, and customers. 1. Choose a book to digitize. 2. Collect and record metadata, and confirm Public Domain status. 2. Obtain images of all the pages. 3. Produce text with OCR. 4. Refine the text to match the OCR. 5. Locate and mark significant artifacts in the text. 6. Submit package of text(s) and images to PG. 7. PG review and post to catalog. 9. Select layout and format for desired device or application. 10. Select or create layout and format for text and artifacts. 11. Issue ebook per spec (possibly including production.) 12. Identify errors or other revisions to text. 13. Locate and describe revisions. 14. Submit revision to PG. 15. Back to step 7 to validate and apply revision. There are obvious definitions to be provided, including metadata, refine, location and mark, significant artifacts, package, etc. etc. Don

1 0

top 10 reasons why this current thread will go nowhere -- #10
by Bowerbird＠aol.com 05 Oct '12

05 Oct '12

#10. that's where these threads always go, and nothing has changed. -bowerbird

1 0

blah blah blah terminology blah blah
by Bowerbird＠aol.com 05 Oct '12

05 Oct '12

don said: > I've never seen the term "vanity html" before. i've used the term "snowflake" _many_ times before. i'm quite sure jim is using "vanity" in a similar way... > The only effect I see for using it is > to add to the hostility level and > decrease the likelihood you will get cooperation. i am the one who uses tactical means to increase the odds that y'all won't act on my advice, not jim. i do that because i am _amused_ by your resistance to the message due to your dislike of the messenger. i find it quit entertaining that you guys would rather thrash around, back and forth, for _years_ (literally!), instead of adopting a solution i have proven to work. in other words, i don't need any of your "cooperation". i'm doing just fine, _laughing_ here on the sidelines... as for jim, i think the term "vanity" probably represents the fact that he sees the "snowflake" .html as selfish... and i think he's probably right about that. even though i use a different term (one that might be less evaluative), i say jim's entitled to use "vanity" to express his feelings. that doesn't mean he's using it to decrease cooperation. > Same for "dumb-downed" but not as aggressively. i disagree that this term is being used for "spin" either. i think it accurately depicts a viewpoint which is common. i also believe, and feel i have proven, that that viewpoint is not based in fact. you definitely have to change stuff, to make it work on today's somewhat primitive e-readers, but the resultant product still works sufficiently properly to avoid the label of "dumbed-down". indeed, at times, the "fancy" .html used for desktops can get "too cute" and trip over itself, and ends up giving a degraded experience. so i strongly _disagree_ with jeroen, and the other people who use this term, because i believe it reflects a mindset that doesn't have to be held to describe reality accurately. (it's certainly true that one _can_ "dumb-down" the .html. but i say that it's not _necessary_ to do so to make it work. i've been able to make all common book-elements work.) but even though i disagree with the term "dumbed-down", i understand that it reflects the mentality of some people, a mentality which is not accurate, but still one they hold... and thus, i certainly don't think it's being used for "spin", or to increase the hostility, and most certainly not for the purpose of sabotaging any possibilities for "cooperation". indeed, "dumbed-down" doesn't have any ad hominem in it. (and even "vanity" is only ad hominem if you interpret it so.) > I think better terms might be "complex" vs. "simple" > and "portable" vs. "non-portable." those poles are no better. in some ways, they are worse. "conversion-capable" versus "snowflake" is the essence. this stresses that _conversion_ is the main objective... "snowflake" is the obvious word to use for the one pole, because everyone knows that every snowflake is _unique_. there are lots of ways to accomplish something in .html. if you multiply all the possibilities, they become infinite. you need to have common coding conventions in order to chop down that infinity into a number that can be handled. to sum up, it must be _standardized_to_a_limited_palette_. (where the word "limited" pertains to numbers, not quality.) this means that if there are 89 ways to do something, you might have to chop that down to a mere 23. or 7. or, if you can formulate code to handle all 89, then there's no reason from that standpoint to set a limit, although you might impose a limit anyway, so as to make it easy on the humans maintaining the library. for the purpose of computer conversion, then, you'll need to actually _write_the_code_ so you know _how_many_ and _which_ of those 89 different ways your code can handle... avoiding snowflakes is a _necessary_ condition for conversion. but it's not enough, in and of itself. in order to convert properly, you need to understand the structural nature of every single element within the book. now, it's possible to have a complex scheme which is still _incapable_ of conversion because it doesn't communicate the necessary information about the structures in the book. on the other hand, it's possible to have a scheme which is _simple_, yet still manages to communicate the information. sure, there's a possibility for a complex scheme that works. (although then there arises another question about whether that would be a good solution for a volunteer organization. but for this reply, we're concentrating on terminology use.) and there is, of course, the simple scheme that doesn't work. but it's just wrong to think that any simple scheme will fail, and any complex scheme will succeed. so that label is bogus. as for portable/non-portable, i'm not sure what you mean. i suppose you could say you meant "convertable/snowflake", but if that's what you really meant, then don't be ambiguous, say it in a way that doesn't require the internal interpretation. -bowerbird

1 0

Blah**5 - Virtualising layout
by Robert Gibbins 04 Oct '12

04 Oct '12

There is an interesting question not quite getting discussed in this debate about what exactly IS epub-friendly HTML. Greg's last append, and this isn't the first one, seems to imply that if only contributors would use only structural HTML, then all would be well. I spent much of the 1980's writing my technical reports in GML (a predecessor of HTML (it's where the <p>, <ol>, <li>, <h1>, <h2> etc. tags in HTML came from)) in which you could specify little MORE than document structure. For technical reports conforming to a well-understood corporate layout policy, this worked pretty well. When I started looking at how to improve PG epubs a year or two ago, I assumed initially that declaring document structure in this way was indeed the main objective. But it now seems to me that just declaring document structure in HTML is inadequate, because there is meaning in the layout of books, which isn't easily or obviously coded in simple HTML structural tags. One might object that HTML extended with CSS styles is much more flexible and customisable than plain old HTML - you can use the facilities of CSS to effectively create your own tags (e.g. lots of <div>'s and <span>'s of specifically declared classes). This is perfectly true, but the danger if you do this is that by doing so you have superimposed your own conceptual model of the structure of the book contents onto the author's original work. And what you may well actually be doing is recording your misunderstanding. To give an example, if you look at many 19th century novels there are quite often many letters given in the text (Austen, Brontes, Trollope etc). You could come up with a standard set of styles for letter headings, openings and closings which would then standardise how to format them. Initially I thought this was what I should do (and from a brief look at TEI that seems to be what it does), but looking a little closer, the particular style and layout of the headings, etc. often conveys something about the formality or intimacy of the relationship between writer and correspondent. To satisfactorily encode this into a discrete set of CSS styles one would first have to have a profound understanding of letter writing conventions when the book was written, and how they changed over time, and secondly a way to encode those conventions into HTML/CSS in a sufficiently expressive way. My current thinking is that although much of the coarse-grained structure of a literary work (the division into front matter, main body and back matter, the division into volumes, chapters and sections, the division of much of the text into paragraphs) CAN be satisfactorily encoded using standard HTML structural tags, for the finer-grained structure it is more realistic, and probably more likely to convey the author's original meaning, to try instead to simply 'virtualise' the original layout (using CSS styles to document the virtual layout). What I mean by virtualise, is to do the markup to preserve layout in as scalable, reflowable and device-independent a way as HTML allows. For example: - not quoting fixed font sizes - specifying margins in scalable units like ems - using the hanging-indent technique quoted by Jim Adcock a few months ago where appropriate I'm sure other contributors to this forum could produce many more such techniques. Whenever in the past I have suggested that a little standardisation of HTML might be a good idea this has been rejected by Greg, more or less out of hand on grounds like: - PG is not interested in individuals who want us to admire their pretty layouts - PG doesn't want to place artificial restrictions on submitter's HTML Actually I agree with both of these points, but my belief is that if PG wants to produce readable epubs (something which many people agree it isn't currently doing), there isn't a simple one-line answer that achieves this. PG is going to have to step up to the plate and think about standardising its HTML contributions. And (ahem) from what I can see of their contributions, it's not totally clear to me that the current PG inner circle necessarily have all the experience and skills to do this on their own. Bob Gibbins

3 2

let's step through this process.011
by Bowerbird＠aol.com 04 Oct '12

04 Oct '12

now for the debriefing on our update walk-through... *** we re-digitized "pride and prejudice", using this source: > http://archive.org/details/harvardclassicss03elio all of our work-files are here: > http://zenmagiclove.com/prhpr the final "updated" master z.m.l. file is here: > http://zenmagiclove.com/prhpr/prhpr.zml the derivative versions are found here: > http://zenmagiclove.com/prhpr/prhpr.html > http://zenmagiclove.com/prhpr/prhprp222.html *** ok, let's start this debriefing with the main headline: digitization isn't as hard as some of you seem to think. you might remember that i did this project because jon hurst wanted to re-do the top-10 p.g. downloads. he offered up some grand plan that would've involved d.p. in some complicated workflow, and it was obvious to me that he was overestimating the work required to re-do one of these classics... thanks to plentiful scans and a variety of digitizations out in the wild, it can be relatively quick and easy to do books from start to end. so i decided to prove it with a bit of my own pudding. now, one week later, we have a newly-done "update", including an extensive documentation of the process, from little old me working a small number of hours on a handful of days. (it took more time to write it all up.) by doing it myself, i have shown that you can do it too. but if you _do_ choose to take action, take my advice: it is _not_ the smartest way to just jump in like i did, and start it up, not if you don't have the _experience_ that i have, working with those archive.org materials... so here's some advice on choosing your material... this is about archive.org, since i'm stuck with them, being that i gave up on google a while back because google seemed to be making it harder and harder to scrape their material, whereas archive.org is _open_. but archive.org stuff can suck, so you must take care. particularly avoid the stuff that archive.org scraped from google, because then you get the worst quality of both institutions, and that can be a lethal combo. (it might also be the case that now that google has brought out their commercial arm, the quality of the work they offer up to the public has improved. i haven't checked, to know, but it _is_ a possibility.) but back to internet archive... my first piece of advice about archive.org material is to choose it very carefully. some of their scans are not-good. if a book has bad scans, _skip_the_book_. life is too short to spend time staring at lousy scans. if you can't find good scans, move on to another title. you should particularly avoid scan-sets archive.org has "liberated" from google, because they are awful. sometimes the _scans_ can look good enough to use, but the _o.c.r._ turned out lousy. skip those as well... the o.c.r. which i used in this project was _terrible._ the letter "e" was frequently misrecognized as "c", and as you might know, the letter "e" is _common._ and there were lots of other misrecognitions too... i was able to deal with it because i know the ropes, but until you clean a dozen archive.org o.c.r. files, you won't, so you should avoid the hassle entirely... (and once you _have_ cleaned a dozen such messes, you will know quite well you don't want to do more.) you should especially watch out for some o.c.r. files from archive.org which have _lost_ their em-dashes! restoring those em-dashes is a huge pain in the ass. speaking of the o.c.r., the easy thing to do is to grab the djvu.txt file that's listed right on the "files" page. that's what i did, but it is _not_ the right thing to do, not if you want to minimize the time it will take you. it's much better to pull the text out of the x.m.l. file that's saved by abbyy finereader in the o.c.r. process. that's because _styling_ information gets saved there, and in many cases, it might well take a ton of time to reconstitute it. so i recommend you extract it initially. the problem with my recommendation is that you will have to write yourself an app to grab that information. (yes, i wrote such an app, but you may not use mine.) if you can't write an app, or use somebody else's, then i guess you'll just have to use the djvu.txt file instead. it won't be the end of the world. heck, i did it myself, for this demo project, because i knew i already _have_ all of the styling information for "pride and prejudice". but if you can write an app, then go ahead and do that. another benefit of using the abbyy.xml file is that you will see that it contains those _em-dashes_ which the incompetent technocrats over there lost in the deja-vu. also, if you will be working in a web-based application, you can squirt the scans from archive.org to your site, without doing all the dirty-work of downloading them to your machine and then uploading them to your site. those are the big points. if anyone has any questions about smaller-grained issues, i'll be happy to answer, to the best of my ability. most glitches have bitten me. as far as the step-by-step workflow of doing the edits, i have belabored that many, many times in the past, so your best bet in that regard will be to hit the archives. *** just a couple more notes. i programmed all my tools for this "pride and prejudice" project _from_scratch_... so even if you have to start from scratch, it ain't hard. i don't think i hit hard enough that this last round of page-by-page .html files have an error-report form, right on each page, so it makes it _very_convenient_ for someone to report an error, unlike the p.g. way... and yes, the text i created does needs smoothreading. i'm not vouching for its full accuracy, by _any_ means. i miss the good old days, when jose menendez would appear every time i posted something, and do a reveal of the 197 errors he found in it. that kept me humble. so go ahead and find the errors that i could not find! it'll make me happy to have my human-ness exposed. because, and wraps up my summary, i think that i did a pretty good job. i think this text is amazingly clean. if there is a cleaner "pride and prejudice" _anywhere_, i would very much like to see it, so do please tell me... and not only that, but i did the entire job in one week. *** so, once again, the executive summary: digitization isn't as hard as some of you seem to think. -bowerbird

1 0

the proof is in the pudding
by Bowerbird＠aol.com 04 Oct '12

04 Oct '12

i have proven, with numerous examples, that it is possible to generate a quality .html file _and_ a quality .epub _and_ a quality .mobi -- should i mention the high-quality .pdf as well? -- all from a plain-text light-markup .zml file. proven it! with examples! numerous times! the longer you guys jabber back and forth and ignore this fact that's right in front of your eyes, the more _stupid_ you make yourselves look to the lurkers who are laughing at your expense... -bowerbird

1 0

Does anyone know what's going on with PG Canada and Faded Pages?
by James Simmons 04 Oct '12

04 Oct '12

I donated page scans from several books to DP Canada many months ago. A couple of them have been finished and the books are on fadedpage.com but NOT on the PG Canada site: http://www.fadedpage.com/showbook.php?pid=20120626 http://www.fadedpage.com/showbook.php?pid=20120707 What puzzles me is that new books have been added to PG Canada recently, but not these. PG Canada has EPUBs for the books, and fadedpage.com only has text and HTML. I ran the HTML through Kindlegen so I could read the books on my Kindle and found that both had page numbers that the Kindle didn't know what to do with. In the Benchley book the page numbers were often as not stuck inside words that may have been hyphenated in the original. Does anyone know what's going on here? If it was up to me all the books would be on PG Canada's site only. James Simmons

4 5

just because i had aroused my own curiosity
by Bowerbird＠aol.com 04 Oct '12

04 Oct '12

just because i had aroused my own curiosity, i actually went and looked at a couple books, even though i desperately want to avoid this entire conversation of unnecessary duplication. but i looked. sure enough, the reality is as bad as i thought. in fact, on digging deeper, yes, it's even worse. *** let's start with huck finn... the current file, #76!, was done by ron burkey, from an original done by an anonymous person, and david widger is top-listed as the maintainer. *** let's detour a bit, to talk about david widger... he's a whitewasher, and he's one of a handful of volunteers who has given _years_and_years_ of good and solid service to project gutenberg. if anyone can feel proud of their contribution to project gutenberg, it would be david widger. he has handled literally _thousands_ of e-texts as a whitewasher, and digitized many of those by himself. david is one of the _heroes_ of p.g. indeed, although such things are impossible to judge with any kind of certainty, it would _not_ be a stretch of the imagination to assert that david widger has given more than anyone else -- _anybody_ -- to project gutenberg. period. that's how far up on the totem pole david is... and any course of action has to _respect_ that... which is not to put a halo around the man's work, because i will soon describe that work objectively, but it _is_ to remind everyone that we still need to respect the man behind that work, and the depth of the long-time service he's given to the project. we can criticize the work and still respect the man. not that my intent is to set out to "criticize", per se. again, i am just describing his work _objectively_... so if i get something wrong, feel free to correct me. *** so, let's look at "huck finn" in project gutenberg... the original is credited to "dell(a)wiretap.spies.com". its post-date says 29-jan-2002, but the file itself says it was "released to the public july 1993", so... your guess is as good as mine. that's hfinn10.txt. hfinn11.txt has a post-date of 13-aug-2002, and this is the one which was prepared by ron burkey. his version dehyphenated the end-line-hyphenates, and also reformatted the paragraphs, owing to that. hfinn12.txt was posted on 15-may-2004, and this is the version where david widger came on-board. his major contribution was to add an .html version, which was an accomplishment back in those days. with updates coming so frequently, it didn't matter that this e-text was actually still rather primitive... for instance, it was still using uppercase for italics. (believe it or not, it was only at the celebration of the 10,000th e-text -- held in december of 2003 -- when p.g. _committed_ to actually marking italics, and it took a year or two for the promise to kick in, just so you have the historical knowledge necessary to grok that such primitiveness was common then.) but that's when the problem set in. because the frequent updates to this book stopped. the 2004 file sat on the shelf, and sat on, and on. tons and tons of downloads. tons. but no updates. and as time went on, its primitive nature became more and more obvious, more and more grating. *** finally, jim decided to update our good old huck, and it was posted with a date of 10-may-2010. and, to reiterate this part of the story, the update was posted _not_ as a replacement for number 76, but as a completely new e-text, numbered 32325. and, as of just now, according to the p.g. website, the old file has 16,537 downloads, the new 528.... (but who knows what those numbers really mean?) it certainly appears that david widger didn't want to share slot #76 with work that overwrote his own... i'm _not_ saying that i _blame_ him for that action. i'm just paying attention to the download numbers. the "new improved" file isn't getting any traction... so that's the problem, in a nutshell... *** but that's not the end of the story. unfortunately. because, on 31-aug-2012, #76 was _updated_ too! that's right, it was updated just over a month ago! maybe there _is_ hope! :+) (there might have also been some kind of action in february of this year, perhaps revolving around that ill-fated mess greg was proposing back then, but we'll just pretend that none of that happened.) so, an update to huck finn #76 is good news, right? well, you might think so, but if you look, actually, no. because, believe it or not, #76 still has _no_italics._ that's right, a p.g. e-book updated just a month ago, is _still_ using uppercase to indicate italics. bullshit! even worse, if you take a look at the .html source, you'll see that the headers are tagged as paragraphs! so much for "tagging the structure of the document". and this is from a whitewasher, for crying out loud! on a classic book, with a number inside the first 100, which is in the top-10 list of downloads, getting tons. something is most definitely wrong with this picture. plus, as i said, i seem to remember that jim said that he had found _hundreds_ of mistakes in #76, and even that he'd actually posted a change-log... but i couldn't even bear to look to see if the errors had been corrected in this "updated" file, because i had a very strong feeling i would find they hadn't. at any rate, the message needs to be repeated: something is most definitely wrong with this picture. *** i also looked at "pride and prejudice", obviously, since i had just completed doing my own update. the history for that e-text is remarkably similar... various versions (09-12) and dates (2002-2006), but then an updated version(s) appearing recently, with dates of 28-nov-2011 and 12-may-2012. and a closer look reveals the same sad situation... the one positive exception is that this book _does_ have the chapter-headers tagged as headers! yay! with the italics properly marked ever since 2006! chalk up victories, no matter how slight they seem! it's not a triumph for structural tagging, however. there are several letters in this book, and they are tagged as plain paragraphs. the only distinction they receive is that the salutations and sign-offs get a "center" tag, hardly a very significant tip-off. so i was tempted to just write it off with that, but for this book, however, i simply had to look to see what changes were made the most-recent "update", given that i just spent a week poring over that text. besides, maybe david had spotted an error i hadn't, and thus my text would improve by analyzing his. so i decided to look at the changes in the update. not that it was all that easy to "look", mind you... because this revised e-text had been _rewrapped_. thus, a simple comparison couldn't be done easily. but i expected that if a rewrapping was required, then it meant that many changes had been made, and i was eager to see exactly what they'd been... so i jumped through hoops to set up a comparison. which means that you can probably understand that i was quite disappointed to find a mere 53 changes. moreover, most of 'em were extremely minor ones, nothing that would necessitate a rewrap, which led me to wonder if the rewrap had been done so as to _hide_ the fact that so few changes had been made. i can't make the charge, of course, but i did wonder. but the news gets even worse. because when i stepped through those 53 changes, i discovered that some of 'em were actually incorrect! indeed, around 20 of the 53 were _improper_edits_... (a full 16 incorrectly revised "upstairs" to "up stairs".) so, as with huck, this is bad. it is embarrassingly bad. so, um, i don't think i can look at any more "updates"; it's just too depressing. i am at a loss for anything else to say, so i will close... -bowerbird

2 1

let's cut to the chase
by Bowerbird＠aol.com 03 Oct '12

03 Oct '12

having this conversation _again_, for the 83rd time, is just lunacy. let's just cut to the f**king chase, ok? *** do we all agree that p.g. has to offer .epub versions? and .mobis? and the standard .html? and .txt as well? do we all agree that maintaining multiple versions of every book (.epub, .mobi, .html, .txt) is not practical, at least not if/when we expect whitewashers to do it? do we all agree that p.g. absolutely _needs_ to have a dependable tool that can do all of its conversions? do we all agree that marcello's tool is badly inferior, because it puts undue constraints on the input files but nonetheless makes massively crappy derivatives? do we all agree that p.g. cannot allow "snowflakes", idiosyncratic files that will break the conversion tool? do we all agree that if the individual contributor will bear the burden to keep files updated reasonably well, they can submit their own versions for all derivatives? do we all agree that if an individual contributor wants p.g. to bear the update responsibility, then _one_ file -- and _only_ the one -- will be used as the "master" which generates the derivative versions, and _tough_ if that individual contributor doesn't like the quality? do we all agree that every individual contributor then _must_ submit the p.g.-defined "master", in the event that the individual contributor is judged to be derelict fulfilling the duty of doing timely update maintenance, even if the contributor considers this "master" file to be a "dumbed-down" insult to their other _superior_ files? *** if we can all -- or maybe just _most_ of us -- _agree_ on that set-up, maybe (just maybe) the whitewashers will agree to loosen their grip on the reins of power... otherwise, if you expect _them_ to do all of the work, you can expect that they will continue to call the shots. -bowerbird

1 0

let's step through this process.010
by Bowerbird＠aol.com 03 Oct '12

03 Oct '12

ok, we're finished! call in the smoothreaders! we re-digitized "pride and prejudice", using this: > http://archive.org/details/harvardclassicss03elio our files, from start to finish, are found here: http://zenmagiclove.com/prhpr *** for the _final_ set of changes to our text, i checked the balancing of dialog quotes, i downcased all decorative small-cap text, and i marked end-line hyphenates to show which end-line hyphens to retain on rewrap. this book is as close to perfect as i can make, so i turn it over to the world to move it closer. do please let me know of any errors you find... *** the final "updated" version is located at: > http://zenmagiclove.com/prhpr/prhpr.zml the derivative versions are here: > http://zenmagiclove.com/prhpr/docomp9f.py > http://zenmagiclove.com/prhpr/prhpr.html > http://zenmagiclove.com/prhpr/prhprp222.html *** i'll eventually churn out an .epub and a .mobi, but i'll give y'all a chance to offer your critiques first... :+) *** i'll be back tomorrow with a complete wrap-up... but the executive summary only requires one sentence: digitization isn't as hard as some of you seem to think. -bowerbird

1 0