- gutvol-d - lists.pglaf.org

Blah**5 - Virtualising layout
by Robert Gibbins 04 Oct '12

04 Oct '12

There is an interesting question not quite getting discussed in this debate about what exactly IS epub-friendly HTML. Greg's last append, and this isn't the first one, seems to imply that if only contributors would use only structural HTML, then all would be well. I spent much of the 1980's writing my technical reports in GML (a predecessor of HTML (it's where the <p>, <ol>, <li>, <h1>, <h2> etc. tags in HTML came from)) in which you could specify little MORE than document structure. For technical reports conforming to a well-understood corporate layout policy, this worked pretty well. When I started looking at how to improve PG epubs a year or two ago, I assumed initially that declaring document structure in this way was indeed the main objective. But it now seems to me that just declaring document structure in HTML is inadequate, because there is meaning in the layout of books, which isn't easily or obviously coded in simple HTML structural tags. One might object that HTML extended with CSS styles is much more flexible and customisable than plain old HTML - you can use the facilities of CSS to effectively create your own tags (e.g. lots of <div>'s and <span>'s of specifically declared classes). This is perfectly true, but the danger if you do this is that by doing so you have superimposed your own conceptual model of the structure of the book contents onto the author's original work. And what you may well actually be doing is recording your misunderstanding. To give an example, if you look at many 19th century novels there are quite often many letters given in the text (Austen, Brontes, Trollope etc). You could come up with a standard set of styles for letter headings, openings and closings which would then standardise how to format them. Initially I thought this was what I should do (and from a brief look at TEI that seems to be what it does), but looking a little closer, the particular style and layout of the headings, etc. often conveys something about the formality or intimacy of the relationship between writer and correspondent. To satisfactorily encode this into a discrete set of CSS styles one would first have to have a profound understanding of letter writing conventions when the book was written, and how they changed over time, and secondly a way to encode those conventions into HTML/CSS in a sufficiently expressive way. My current thinking is that although much of the coarse-grained structure of a literary work (the division into front matter, main body and back matter, the division into volumes, chapters and sections, the division of much of the text into paragraphs) CAN be satisfactorily encoded using standard HTML structural tags, for the finer-grained structure it is more realistic, and probably more likely to convey the author's original meaning, to try instead to simply 'virtualise' the original layout (using CSS styles to document the virtual layout). What I mean by virtualise, is to do the markup to preserve layout in as scalable, reflowable and device-independent a way as HTML allows. For example: - not quoting fixed font sizes - specifying margins in scalable units like ems - using the hanging-indent technique quoted by Jim Adcock a few months ago where appropriate I'm sure other contributors to this forum could produce many more such techniques. Whenever in the past I have suggested that a little standardisation of HTML might be a good idea this has been rejected by Greg, more or less out of hand on grounds like: - PG is not interested in individuals who want us to admire their pretty layouts - PG doesn't want to place artificial restrictions on submitter's HTML Actually I agree with both of these points, but my belief is that if PG wants to produce readable epubs (something which many people agree it isn't currently doing), there isn't a simple one-line answer that achieves this. PG is going to have to step up to the plate and think about standardising its HTML contributions. And (ahem) from what I can see of their contributions, it's not totally clear to me that the current PG inner circle necessarily have all the experience and skills to do this on their own. Bob Gibbins

3 2

let's step through this process.011
by Bowerbird＠aol.com 04 Oct '12

04 Oct '12

now for the debriefing on our update walk-through... *** we re-digitized "pride and prejudice", using this source: > http://archive.org/details/harvardclassicss03elio all of our work-files are here: > http://zenmagiclove.com/prhpr the final "updated" master z.m.l. file is here: > http://zenmagiclove.com/prhpr/prhpr.zml the derivative versions are found here: > http://zenmagiclove.com/prhpr/prhpr.html > http://zenmagiclove.com/prhpr/prhprp222.html *** ok, let's start this debriefing with the main headline: digitization isn't as hard as some of you seem to think. you might remember that i did this project because jon hurst wanted to re-do the top-10 p.g. downloads. he offered up some grand plan that would've involved d.p. in some complicated workflow, and it was obvious to me that he was overestimating the work required to re-do one of these classics... thanks to plentiful scans and a variety of digitizations out in the wild, it can be relatively quick and easy to do books from start to end. so i decided to prove it with a bit of my own pudding. now, one week later, we have a newly-done "update", including an extensive documentation of the process, from little old me working a small number of hours on a handful of days. (it took more time to write it all up.) by doing it myself, i have shown that you can do it too. but if you _do_ choose to take action, take my advice: it is _not_ the smartest way to just jump in like i did, and start it up, not if you don't have the _experience_ that i have, working with those archive.org materials... so here's some advice on choosing your material... this is about archive.org, since i'm stuck with them, being that i gave up on google a while back because google seemed to be making it harder and harder to scrape their material, whereas archive.org is _open_. but archive.org stuff can suck, so you must take care. particularly avoid the stuff that archive.org scraped from google, because then you get the worst quality of both institutions, and that can be a lethal combo. (it might also be the case that now that google has brought out their commercial arm, the quality of the work they offer up to the public has improved. i haven't checked, to know, but it _is_ a possibility.) but back to internet archive... my first piece of advice about archive.org material is to choose it very carefully. some of their scans are not-good. if a book has bad scans, _skip_the_book_. life is too short to spend time staring at lousy scans. if you can't find good scans, move on to another title. you should particularly avoid scan-sets archive.org has "liberated" from google, because they are awful. sometimes the _scans_ can look good enough to use, but the _o.c.r._ turned out lousy. skip those as well... the o.c.r. which i used in this project was _terrible._ the letter "e" was frequently misrecognized as "c", and as you might know, the letter "e" is _common._ and there were lots of other misrecognitions too... i was able to deal with it because i know the ropes, but until you clean a dozen archive.org o.c.r. files, you won't, so you should avoid the hassle entirely... (and once you _have_ cleaned a dozen such messes, you will know quite well you don't want to do more.) you should especially watch out for some o.c.r. files from archive.org which have _lost_ their em-dashes! restoring those em-dashes is a huge pain in the ass. speaking of the o.c.r., the easy thing to do is to grab the djvu.txt file that's listed right on the "files" page. that's what i did, but it is _not_ the right thing to do, not if you want to minimize the time it will take you. it's much better to pull the text out of the x.m.l. file that's saved by abbyy finereader in the o.c.r. process. that's because _styling_ information gets saved there, and in many cases, it might well take a ton of time to reconstitute it. so i recommend you extract it initially. the problem with my recommendation is that you will have to write yourself an app to grab that information. (yes, i wrote such an app, but you may not use mine.) if you can't write an app, or use somebody else's, then i guess you'll just have to use the djvu.txt file instead. it won't be the end of the world. heck, i did it myself, for this demo project, because i knew i already _have_ all of the styling information for "pride and prejudice". but if you can write an app, then go ahead and do that. another benefit of using the abbyy.xml file is that you will see that it contains those _em-dashes_ which the incompetent technocrats over there lost in the deja-vu. also, if you will be working in a web-based application, you can squirt the scans from archive.org to your site, without doing all the dirty-work of downloading them to your machine and then uploading them to your site. those are the big points. if anyone has any questions about smaller-grained issues, i'll be happy to answer, to the best of my ability. most glitches have bitten me. as far as the step-by-step workflow of doing the edits, i have belabored that many, many times in the past, so your best bet in that regard will be to hit the archives. *** just a couple more notes. i programmed all my tools for this "pride and prejudice" project _from_scratch_... so even if you have to start from scratch, it ain't hard. i don't think i hit hard enough that this last round of page-by-page .html files have an error-report form, right on each page, so it makes it _very_convenient_ for someone to report an error, unlike the p.g. way... and yes, the text i created does needs smoothreading. i'm not vouching for its full accuracy, by _any_ means. i miss the good old days, when jose menendez would appear every time i posted something, and do a reveal of the 197 errors he found in it. that kept me humble. so go ahead and find the errors that i could not find! it'll make me happy to have my human-ness exposed. because, and wraps up my summary, i think that i did a pretty good job. i think this text is amazingly clean. if there is a cleaner "pride and prejudice" _anywhere_, i would very much like to see it, so do please tell me... and not only that, but i did the entire job in one week. *** so, once again, the executive summary: digitization isn't as hard as some of you seem to think. -bowerbird

1 0

the proof is in the pudding
by Bowerbird＠aol.com 04 Oct '12

04 Oct '12

i have proven, with numerous examples, that it is possible to generate a quality .html file _and_ a quality .epub _and_ a quality .mobi -- should i mention the high-quality .pdf as well? -- all from a plain-text light-markup .zml file. proven it! with examples! numerous times! the longer you guys jabber back and forth and ignore this fact that's right in front of your eyes, the more _stupid_ you make yourselves look to the lurkers who are laughing at your expense... -bowerbird

1 0

Does anyone know what's going on with PG Canada and Faded Pages?
by James Simmons 04 Oct '12

04 Oct '12

I donated page scans from several books to DP Canada many months ago. A couple of them have been finished and the books are on fadedpage.com but NOT on the PG Canada site: http://www.fadedpage.com/showbook.php?pid=20120626 http://www.fadedpage.com/showbook.php?pid=20120707 What puzzles me is that new books have been added to PG Canada recently, but not these. PG Canada has EPUBs for the books, and fadedpage.com only has text and HTML. I ran the HTML through Kindlegen so I could read the books on my Kindle and found that both had page numbers that the Kindle didn't know what to do with. In the Benchley book the page numbers were often as not stuck inside words that may have been hyphenated in the original. Does anyone know what's going on here? If it was up to me all the books would be on PG Canada's site only. James Simmons

4 5

just because i had aroused my own curiosity
by Bowerbird＠aol.com 04 Oct '12

04 Oct '12

just because i had aroused my own curiosity, i actually went and looked at a couple books, even though i desperately want to avoid this entire conversation of unnecessary duplication. but i looked. sure enough, the reality is as bad as i thought. in fact, on digging deeper, yes, it's even worse. *** let's start with huck finn... the current file, #76!, was done by ron burkey, from an original done by an anonymous person, and david widger is top-listed as the maintainer. *** let's detour a bit, to talk about david widger... he's a whitewasher, and he's one of a handful of volunteers who has given _years_and_years_ of good and solid service to project gutenberg. if anyone can feel proud of their contribution to project gutenberg, it would be david widger. he has handled literally _thousands_ of e-texts as a whitewasher, and digitized many of those by himself. david is one of the _heroes_ of p.g. indeed, although such things are impossible to judge with any kind of certainty, it would _not_ be a stretch of the imagination to assert that david widger has given more than anyone else -- _anybody_ -- to project gutenberg. period. that's how far up on the totem pole david is... and any course of action has to _respect_ that... which is not to put a halo around the man's work, because i will soon describe that work objectively, but it _is_ to remind everyone that we still need to respect the man behind that work, and the depth of the long-time service he's given to the project. we can criticize the work and still respect the man. not that my intent is to set out to "criticize", per se. again, i am just describing his work _objectively_... so if i get something wrong, feel free to correct me. *** so, let's look at "huck finn" in project gutenberg... the original is credited to "dell(a)wiretap.spies.com". its post-date says 29-jan-2002, but the file itself says it was "released to the public july 1993", so... your guess is as good as mine. that's hfinn10.txt. hfinn11.txt has a post-date of 13-aug-2002, and this is the one which was prepared by ron burkey. his version dehyphenated the end-line-hyphenates, and also reformatted the paragraphs, owing to that. hfinn12.txt was posted on 15-may-2004, and this is the version where david widger came on-board. his major contribution was to add an .html version, which was an accomplishment back in those days. with updates coming so frequently, it didn't matter that this e-text was actually still rather primitive... for instance, it was still using uppercase for italics. (believe it or not, it was only at the celebration of the 10,000th e-text -- held in december of 2003 -- when p.g. _committed_ to actually marking italics, and it took a year or two for the promise to kick in, just so you have the historical knowledge necessary to grok that such primitiveness was common then.) but that's when the problem set in. because the frequent updates to this book stopped. the 2004 file sat on the shelf, and sat on, and on. tons and tons of downloads. tons. but no updates. and as time went on, its primitive nature became more and more obvious, more and more grating. *** finally, jim decided to update our good old huck, and it was posted with a date of 10-may-2010. and, to reiterate this part of the story, the update was posted _not_ as a replacement for number 76, but as a completely new e-text, numbered 32325. and, as of just now, according to the p.g. website, the old file has 16,537 downloads, the new 528.... (but who knows what those numbers really mean?) it certainly appears that david widger didn't want to share slot #76 with work that overwrote his own... i'm _not_ saying that i _blame_ him for that action. i'm just paying attention to the download numbers. the "new improved" file isn't getting any traction... so that's the problem, in a nutshell... *** but that's not the end of the story. unfortunately. because, on 31-aug-2012, #76 was _updated_ too! that's right, it was updated just over a month ago! maybe there _is_ hope! :+) (there might have also been some kind of action in february of this year, perhaps revolving around that ill-fated mess greg was proposing back then, but we'll just pretend that none of that happened.) so, an update to huck finn #76 is good news, right? well, you might think so, but if you look, actually, no. because, believe it or not, #76 still has _no_italics._ that's right, a p.g. e-book updated just a month ago, is _still_ using uppercase to indicate italics. bullshit! even worse, if you take a look at the .html source, you'll see that the headers are tagged as paragraphs! so much for "tagging the structure of the document". and this is from a whitewasher, for crying out loud! on a classic book, with a number inside the first 100, which is in the top-10 list of downloads, getting tons. something is most definitely wrong with this picture. plus, as i said, i seem to remember that jim said that he had found _hundreds_ of mistakes in #76, and even that he'd actually posted a change-log... but i couldn't even bear to look to see if the errors had been corrected in this "updated" file, because i had a very strong feeling i would find they hadn't. at any rate, the message needs to be repeated: something is most definitely wrong with this picture. *** i also looked at "pride and prejudice", obviously, since i had just completed doing my own update. the history for that e-text is remarkably similar... various versions (09-12) and dates (2002-2006), but then an updated version(s) appearing recently, with dates of 28-nov-2011 and 12-may-2012. and a closer look reveals the same sad situation... the one positive exception is that this book _does_ have the chapter-headers tagged as headers! yay! with the italics properly marked ever since 2006! chalk up victories, no matter how slight they seem! it's not a triumph for structural tagging, however. there are several letters in this book, and they are tagged as plain paragraphs. the only distinction they receive is that the salutations and sign-offs get a "center" tag, hardly a very significant tip-off. so i was tempted to just write it off with that, but for this book, however, i simply had to look to see what changes were made the most-recent "update", given that i just spent a week poring over that text. besides, maybe david had spotted an error i hadn't, and thus my text would improve by analyzing his. so i decided to look at the changes in the update. not that it was all that easy to "look", mind you... because this revised e-text had been _rewrapped_. thus, a simple comparison couldn't be done easily. but i expected that if a rewrapping was required, then it meant that many changes had been made, and i was eager to see exactly what they'd been... so i jumped through hoops to set up a comparison. which means that you can probably understand that i was quite disappointed to find a mere 53 changes. moreover, most of 'em were extremely minor ones, nothing that would necessitate a rewrap, which led me to wonder if the rewrap had been done so as to _hide_ the fact that so few changes had been made. i can't make the charge, of course, but i did wonder. but the news gets even worse. because when i stepped through those 53 changes, i discovered that some of 'em were actually incorrect! indeed, around 20 of the 53 were _improper_edits_... (a full 16 incorrectly revised "upstairs" to "up stairs".) so, as with huck, this is bad. it is embarrassingly bad. so, um, i don't think i can look at any more "updates"; it's just too depressing. i am at a loss for anything else to say, so i will close... -bowerbird

2 1

let's cut to the chase
by Bowerbird＠aol.com 03 Oct '12

03 Oct '12

having this conversation _again_, for the 83rd time, is just lunacy. let's just cut to the f**king chase, ok? *** do we all agree that p.g. has to offer .epub versions? and .mobis? and the standard .html? and .txt as well? do we all agree that maintaining multiple versions of every book (.epub, .mobi, .html, .txt) is not practical, at least not if/when we expect whitewashers to do it? do we all agree that p.g. absolutely _needs_ to have a dependable tool that can do all of its conversions? do we all agree that marcello's tool is badly inferior, because it puts undue constraints on the input files but nonetheless makes massively crappy derivatives? do we all agree that p.g. cannot allow "snowflakes", idiosyncratic files that will break the conversion tool? do we all agree that if the individual contributor will bear the burden to keep files updated reasonably well, they can submit their own versions for all derivatives? do we all agree that if an individual contributor wants p.g. to bear the update responsibility, then _one_ file -- and _only_ the one -- will be used as the "master" which generates the derivative versions, and _tough_ if that individual contributor doesn't like the quality? do we all agree that every individual contributor then _must_ submit the p.g.-defined "master", in the event that the individual contributor is judged to be derelict fulfilling the duty of doing timely update maintenance, even if the contributor considers this "master" file to be a "dumbed-down" insult to their other _superior_ files? *** if we can all -- or maybe just _most_ of us -- _agree_ on that set-up, maybe (just maybe) the whitewashers will agree to loosen their grip on the reins of power... otherwise, if you expect _them_ to do all of the work, you can expect that they will continue to call the shots. -bowerbird

1 0

let's step through this process.010
by Bowerbird＠aol.com 03 Oct '12

03 Oct '12

ok, we're finished! call in the smoothreaders! we re-digitized "pride and prejudice", using this: > http://archive.org/details/harvardclassicss03elio our files, from start to finish, are found here: http://zenmagiclove.com/prhpr *** for the _final_ set of changes to our text, i checked the balancing of dialog quotes, i downcased all decorative small-cap text, and i marked end-line hyphenates to show which end-line hyphens to retain on rewrap. this book is as close to perfect as i can make, so i turn it over to the world to move it closer. do please let me know of any errors you find... *** the final "updated" version is located at: > http://zenmagiclove.com/prhpr/prhpr.zml the derivative versions are here: > http://zenmagiclove.com/prhpr/docomp9f.py > http://zenmagiclove.com/prhpr/prhpr.html > http://zenmagiclove.com/prhpr/prhprp222.html *** i'll eventually churn out an .epub and a .mobi, but i'll give y'all a chance to offer your critiques first... :+) *** i'll be back tomorrow with a complete wrap-up... but the executive summary only requires one sentence: digitization isn't as hard as some of you seem to think. -bowerbird

1 0

Blah**5 - Virtualising layout
by Bowerbird＠aol.com 03 Oct '12

03 Oct '12

wow! an intelligent post in the merry-go-round of blah. thank you, robert gibbins. i wish i could buy you a beer! -bowerbird

1 0

twisting pride into the mix
by Bowerbird＠aol.com 03 Oct '12

03 Oct '12

pride. ownership. hard work, running to _years_, not just "weeks" or "months". being replaced. after you were told you would be immortal. these are all thorny issues, to be sure. good luck in getting them sorted out, and i mean that with all sincerity... but c'mon. certainly we don't have to twist all of that into the message that jim is trying to send here, and it strikes me as disingenuous to try to make it sound so simplistic, especially because things _are_ thorny. and heck, i don't even read jim's posts any more, because he never seemed to be able to decide -- at least when he was having any dialog with _me_ -- which side of the fence that he wanted to be on. but even _i_ can figure out the simple dynamic here, because jim has told us what it is, many times now... "huck finn" sat around from the earliest of p.g. days. wouldn't surprise me if it was in the first 100 titles... it probably wasn't a _bad_ version, but it certainly wasn't an exemplar of what would be a _good_ one. if memory serves, ron burkey did that one, and he always did a decent job, but that was the early time. yet -- despite the popularity of that book, and the ever-present never-ending threads here on updates -- the book never did get improved. it just sat there. (like almost all of the early ones have just sat there.) so finally jim did it. he said he had a change-log that showed literally hundreds of changes, i seem to recall. never looked at it, but i have no reason to doubt that. jim provided p.g. the update, but instead of having it _replace_ the early version, jim's update was shuffled back to a huge number that all the new texts receive... now, again, i'm not taking a stand about whether that is a good thing to do, or a bad thing, right, or wrong, but the fact of the matter is that that means that the "new and improved version" is going to suffer because it doesn't have any downloads, whereas the "old" one has a ton of 'em, and people search by download-count. again, thorny issues, no easy answers, all acknowledged. but we can still surely admit that this is a problem too, when we have a _better_ text that's being ignored while a _less-good_ text continues to be downloaded by users. and lest there be any doubt, i'll bet dollars to doughnuts that jim's "new" file is indeed superior to the "old" one... so let's talk about the problem, and not try to _distort_ motivations of the other people in this discussion, ok? especially you, don, because you have complained (and _rightly_so_, i'll add) when people do this to you on d.p. i don't really care how you guys work it out, _especially_ since even if you _can_ come to some kind of conclusion, it won't matter, because the whitewashers have the power, and they've already made it clear how they're gonna do it, no matter what you might "decide", so you're just wasting your time. but if you're going to put stuff in front of me, and expect me to read it, you will have to up your game... -bowerbird

1 0

let's step through this process.009
by Bowerbird＠aol.com 02 Oct '12

02 Oct '12

we're almost done creating our updated version of the book "pride and prejudice", using this archive.org book as our source: > http://archive.org/details/harvardclassicss03elio *** here's our itinerary for today... 1. now the words are correct. we hope. 2. resolve all of the outstanding issues. 3. apply the styling and the formatting. 4. that's good for now, so go upload it. 5. prepare to cross the finish-line next. ** ok, since we have finished our comparison with a "clean" text, we'll assume the words are as correct as we can make 'em for now. (and by "words", i mean "punctuation" too. the scans for this book made it difficult to discern semicolons from colons, so i might have made more than a few judgment calls when it came to them; c'est la vie, c'est vrai.) *** now it's time to resolve any nagging issues. there's always some; a dozen in this book... you must be cautious in deciding to correct a publisher's error, but you also have to be able to pull that trigger when it's necessary. *** next is to apply styling and do formatting... text-styling -- italics, etc. -- can be a bear. in principle, it's easy; but in the cruel world, it can be painfully easy to miss some styling. not just on the first pass, but the second too. and the third, and the fourth, and the fifth... and italics are _a_necessary_component_ to grok what the author intended to emphasize. (whereas, for example, a difference between a semicolon and a colon is often negligible.) formatting is less of a chore, in the sense that it's usually much simpler to "get it right", and because it's fun to make the text "look right". but it's still a thing that eats time and energy. luckily, however, for _this_ book, many clean texts that we have available to us already did the grunt-work of the styling and formatting. so it was just a matter of slapping that in... (using a first-draft app, so no guarantees!) *** so the new version is now up: > http://zenmagiclove.com/prhpr/prhpr-009.zml and of course the derivative versions as well: > http://zenmagiclove.com/prhpr/docomp89.py > http://zenmagiclove.com/prhpr/prhpr-009h.html > http://zenmagiclove.com/prhpr/prhprp222.html *** we're closing in on the finish-line now... which means we need to run some last-minute tests, -- e.g., to make sure our quotemarks all balance -- and prepare the text for "continuous proofreading", which means putting an error-report form on each of the pages that show the scan alongside the text. see you tomorrow! :+) -bowerbird p.s. if you have any questions, now is the time to ask! p.p.s. hahahaha. i crack myself up sometimes. ;+)

1 0