February 2012 - gutvol-d - lists.pglaf.org

Producing epub ready HTML
by Robert Gibbins 01 Feb '12

01 Feb '12

Almost exactly a year ago I put an append on this forum about what was wrong with PG-generated epubs, offering my own hand-crafted ones to replace them. The response at the time could probably be summarised as "yes, we know about that, go away" Over the last year I've probably re-mastered (for my own consumption) about 50 of the PG-generated epubs, many of them Anthony Trollope's novels produced by Joseph Loewenstein and others. Mostly what I am doing to the actual works is, as Greg Newby described in a recent append, removing stuff to do with formatting/layout and instead tagging with structural information that relates it to layout specified in a css file. I also have standard templates for the PG front and back matter wrappers and the xml files that the epub v2 standard requires. The css file I use is not particularly complicated, it contains remarkably few styles, those that are there being largely derived from existing PG stylesheets. I make no claims for the excellence of my markup or stylesheet, I'm still learning, no doubt you chaps could do better, but I achieve results that are satisfactory to me, and without using any more complicated tools than emacs and winzip. My main point is that to do this sort of thing you don't need to be an ace compiler writer or rocket scientist. A year later I look again at this forum and you are still arguing about it, but nothing seems to have been achieved! Some of you want to produce a grandiloquent change management/source code control systems, that gives fine-grained cooperative shared access to pages in the work. (Personally I found that the only significant productivity improvement I got was to take the generated epub and stick all the individual xhtml files back together so I could do emacs global changes on them (regular expressions are wonderful!)and then split them back up into individual chapter files afterwards.) Greg Newby seems to want to kind of shunt everyone's epub contributions into a siding where they will wither and die from not being updated. Nothing is sacrosanct except no one else is allowed to touch David Price or David Widger's work. It seems to me that really all that is required is to recognise that a lot of the html in PG's archive is pretty antiquated by modern standards and needs to be updated. And basically this is a labour-intensive task which can't be completely automated because it wasn't originally done to consistent standards, because it happened over a decade or two during which html and ideas about how to use it evolved. So really what needs to be done is to call for volunteers to do the clean up. And since Greg seems to imply that he is currently deluged with offers of documents from people who think they have already done this, it seems to me that there is plenty of enthusiasm out there (including me!). And before that happens, some PG group (contributors to this forum, Wwers, whatever) needs to get together and write a 'How To' describing at the nuts and bolts level how to do the clean up. This should include a recommended set of styles (and the css that implements them) lots of advice about what not to do for people that don't already know, also including access to templates, css style sheets, etc. As my ex-employer used to say, 'there are no technical answers to management problems, but sometimes there are management answers to technical problems'. Bob Gibbins

5 7

"challenging" error-report challenge
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

al said: > Here's the error report on PG#3441. great. > I'm assuming the Locations and Pages are peculiar > to the reporter's ereader device, whatever it is. since s/he mentions a kindle, and because the kindle uses "locations" as its milestone term, i would guess it's a kindle. as for "pages", i'd assume that's in the archive.org material, namely from that version of the book, which might (or not) be the version of the book on which this e-text was based... > It's quite possible that some of the reporter's > proposed corrections may themselves be incorrect. of course. that is always the case. that's why you have to compare each proposed correction against the page-scan. > Don Kretz is my preferred supplier of the corrected version, > since he offered first. "preferred supplier"? ok... > bowerbird can work on any or all of the other 15 volumes. do you have error-reports for them? > Or, the two can collaborate on all 16, > if they choose to do that. Up to them. you might have misunderstood what i'm offering. i didn't offer to _do_ the corrections for you; i offered to build a system to show you how you could make it _fun_ for people to process the error-reports in a speedy way... and, lest it was not clear, i will code and mount a system, but i do not intend to share any code with you, or with p.g. as i said not too long ago, the ship on that sailed long ago. at this point in time, you couldn't _pay_ me to code for you. i'm willing to code a demonstration, for you and this list, but that's as far as it goes. just so we understand each other, al. > I refuse to deal with revised versions of the same etext > from two different people. you should be able -- and willing!, and thankful! -- to evaluate corrections coming in from _twenty_ people, or two _hundred_, assuming those corrections aren't overloaded with false-alarms. the cost of evaluating a correction-already-made should be nil... *** i had anticipated that i would have to tell you that you need to _instruct_ people on the best way to submit an error-report, but this person -- bless their little heart -- did an _excellent_ job... indeed, they did more than they needed to... for instance, they said this: > Location 170 (Page 4) > day like this; and now I bid thee enter among Gharib s host and, > "Gharib's" should have an apostrophe. > day like this; and now I bid thee enter among Gharib's host and, with an untrained naive error-reporter, you often just get that line with the "should have an apostrophe" version of the correction in it, and that line is the least useful of all of them -- totally unnecessary. also, since p.g. threw away all the pagenumbers, it's great that this wise person included the pagenumber, so we know what scan it is... otherwise, all they needed to say was this: > day like this; and now I bid thee enter among Gharib s host and, > day like this; and now I bid thee enter among Gharib's host and, the computer can tell the difference between the lines, and _highlight_ it for the person who is doing the checking, and find that line in the file, and show the page-scan for review, and then record the judgment of the reviewer to edit or stet, and then _make_ the necessary edits and save all the changes. it's all of that grunt work which my demo-program will handle. -bowerbird

1 0

i don't know -- third base!
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

greg said: > I'm entirely in favor of > automatic conversion > to & from different formats. > Of master formats. the problem is you don't _have_ a master format... except for maybe r.s.t., and its uptake is very slim. and you don't have any adequate converters either. your .rst-to-.html converter makes people unhappy. your .rst-to-.epub converter makes people unhappy. your .rst-to-.mobi converter makes people unhappy. your .html-to-.mobi converter makes people unhappy. your .html-to-.epub converter makes people unhappy. your .txt-to-.mobi converter makes people unhappy. your .txt-to-.epub converter makes people unhappy. your .txt-to-.html converter isn't as bad as the others, but it's still not anything near to what it _could_ be... > I was personally deeply involved in a whole > lot of our current setup for those things that's nothing to brag about, as clearly indicated above. > and am hugely optimistic about > improvements in the future. ok, there's something we can all agree would be nice... until you have a master-format, however, i'd think that your "optimism" might be a tad bit unfounded. but, hey, i'll be more than happy to admit it if you prove me wrong. the point is this: yes, your converters definitely need work. but the problem lies much deeper, i.e., in your input-files... until you have some consistency in your input-files, you will constantly be battling against nature's ability to make fools. and yes, greg, yes, yes, a thousand times yes, this is _exactly_ the pro-consistency argument which i've been making forever. and now you see why. and my constant prediction that you would have to face this very demon, sooner or later, has now also been vindicated... > Meanwhile, we have an immediate and useful way > to improve many reader's experiences, by > providing access to hand-crafted alternatives. except you're just setting yourself up for future complications. _inconsistency_ is the problem. you can't solve it with "more". > It should be obvious that I'm constantly seeking > the long-term view of Project Gutenberg. it's not obvious, because your "solutions" cause more problems. that's not a way to take "a long-term view". if you said instead that you were "seeking a short-term, temporary fix", then yeah, i might buy what you're selling. but even then, it is a bad idea. > That is why I'm thinking of solutions that > involve keeping all derived files or changes > we create/publish/distribute, forever. keeping crap around forever is not a long-term solution either. in fact, it's the exact opposite: a big albatross around your neck. > But that doesn't mean I'm against providing > some files that will only have a few days, months or years > of useful life before they are superseded. except you don't have a mechanism for deciding to supersede. and once those files are out in the wild, you can't "expire" them. the more you talk, the worse this idea sounds. especially since "the problem" which you're attempting to solve has _already_ been solved. we know, for the most part, how to mark up .html that will give us a good .mobi and a good .epub. (at least once we give up the ghost that .html can be a "master", such that the exact same .html gives good .mobi _and_ .epub.) the hard part, then, is getting d.p. postprocessors to _use_ that. they've each developed their snowflake workflow, and are loathe to give it up. especially since -- and this is the _hardest_ part -- they refuse to settle for a "dumbed-down" .html file that doesn't "look like" the paper-book. but... if you can give 'em "the look", they _will_ standardize their .html, for the greater good, because they actually _do_ want the work to become useful in more places. and, of course, if you would have listened to me _six_years_ago_, when i was telling you the same thing, project gutenberg could've been the role-model for google, and internet archive, and amazon. but you were off lost in your xml/tei haze of confusion instead... and even to this very day, you are _still_ overcomplicating things! -bowerbird

2 1

now we're finally getting "deep"
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

greg said: > I put huge backing into addressing this challenge, > and so did other people on this list and elsewhere. > The answer was: TEI. you're out of touch, greg. in a massively spectacular way. the learning curve on .tei is too steep for your volunteers. not just a little too steep, either... far and away too steep. which was obvious to even the most oblivious observers... and furthermore, your workflow never worked correctly... even the handful of volunteers who _tried_ to make it work ended up banging their heads against the wall repeatedly... that's no way to treat your workers. even if you paid them. and you don't. you expect them to undergo torture for free. you would know all this if you merely read the d.p. forums. so your first answer was a non-answer. indeed, a non-starter. > Then, a couple of years later, ditto. The answer was: RST. oh please. none of the proper groundwork was done on .rst... it's a proven methodology, one that is in use all over the place, one that's used to handle documents that are very complicated. but even now, you guys cannot get it to work correctly even for the comparatively simple types of documents that you prepare. and you'd know _that_, too, if you just read the d.p. forums... further, you didn't try to address the desires of the volunteers. you just dumped .rst in the middle of the street, and hoped that they would eat the dogfood. they sniffed it, but it smelled bad, so they turned their noses up at it, and that was the end of that. and just as with .tei, a handful of d.p. people have tried .rst, but -- just as with .tei -- it doesn't really make them all that happy. the converter is a buggy "work-in-progress", and marcello is... well, marcello is marcello, so of course shit's never _his_ fault, it's always the fault of someone else. so of course that's a fail. was there ever any reason to expect anything else? nope. none. > There are scarce few things that TEI or RST are not suitable for neither of them seems capable of making d.p. volunteers happy. but maybe that's not an important function in _your_ world, greg. > The WWers have procedures to process such files - bring it on. you've never reminded me so much of george w. bush as right now. let's go place a "mission accomplished" banner on gutenberg.org... > Remaining problems include: > - how to convince contributors to make > new submissions in these master formats that shoulda been your _second_ order of business. (your first, which you still haven't done, by the way, is to iron out the bugs in your converter workflow.) > - consideration of alternate master formats, as desired the more the merrier! > Solved problems include: > - automatically generating all derived formats, with > a far higher level of integrity than other master formats bullshit! > - applying fixes to the master and > then regenerating derived formats no, you don't know how to do that either. > - having an easily editable master format .rst is one of the heavier light-markup formats around... but as you're desperate, i'll grant you this one, for now... > But the unfortunate fact is that the "better mousetrap" > (or, at least, one that purports to have many of the > improvements over HTML+text as master formats), > was built and delivered years ago. if you really believe that, greg, that's sad. just plain sad. because that is so removed from reality that it's pathetic. > But not too many folks have taken the bait. that's because your bait tastes like crap. > I can already hear a few voices calling (or writing), > "that's because your bait sucks!" it doesn't surprise me that you could already hear us saying that your bait tastes like crap, because it does. > and "but your bait cannot do X" and so forth. that too. most of what it cannot do is give the people who do postprocessing over at d.p. what they require. which is why it's a wonder you ever thought it'd "work". just exactly who do you think is _doing_ the work? > It would be contrary to past experience > for anyone to think they can, indeed, come up > with a solution set that will be above criticism. bullshit, bullshit, bullshit. i demonstrated "a solution set" right here, last week, from test-suite through .html to .mobi and .epub... your not looking at it doesn't negate its existence... -bowerbird

1 0

volunteers are a litmus test on whether your workflow is lame
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

al said: > I've got an errata report on my hands > that's 3400 lines long, that I haven't > had the courage to plow through yet. > The reporter lists something wrong on > almost every one of the book's nearly 400 pages. that's a big job, to be sure. but it's not difficult. and if it seems like it is, you are doing it wrong. put up the files, and the scans, and i'll show you how to do it right. so it's _fun_ for people to do. > On top of that, he'd like an HTML version > created with the footnotes cross-linked and > from what I can tell, the page numbers inserted > because there are internal references to them. sounds totally reasonable to me, and furthermore, how the thing shoulda been done in the first place. > Question: where's an Errata Team when you want one? > Answer: as happened a year or so ago, they find out > what they're up against, and vanish. <g>) it wasn't the problem that frightened them away, al. it was the monster you proposed as the "solution"... :+) but hey, i'm sure it doesn't compare in the slightest to the years of volunteers that d.p. has chased away. -bowerbird

1 0