- gutvol-d - lists.pglaf.org

how to improve the p.g. book files
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

don said: > OK, I'm willing to listen to how I can improve any > "book files" (whatever they are) PG provides today. are you kidding? have you ever even _looked_ at any of the output -- .mobi or .epub -- that is being generated by p.g.? don't you feel any need to inform yourself before you speak? > What great secret do you share with them > that the rest of us aren't privy to? um, it's no "great secret" what the obvious flaws are, don... because -- d'uh -- the obvious flaws are terribly _obvious_. but, since you don't seem to be able to _see_ the obvious, i'll give you a few examples to sharpen your observation... first, the e-books start with legalese. you know that, right? and maybe you're accustomed to it, and you've "accepted" it, but if you want an improved product, move it _all_ to the end. what usually comes next is the title-page gunk of the p-book. that's typically too much information, too fast, and undesired. you should open with the equivalent of the cover of a p-book; that is, the title (and subtitle, if any) and the author (and the co-creators, if there were any), contact info, creation date of this latest version, and the e-book's canonical online address. that's all! all of this info should present itself compactly, on one screen, even if the end-user has their fontsize cranked up fairly high... p.g. e-texts _often_ get this "cover-page" screen badly wrong. *** next should come the table-of-contents. _without_question_. people should know -- with certainty -- where the t.o.c. is, and it should be in a convenient location, so let's make it standard. nor should this be one of those sprawling 10-screens things... indeed, _this_ table of contents should fit _on_a_single_screen_. even if, once again, the person sets their font-size fairly large... if you've got one of those sprawling table-of-contents monsters, make that the _next_ section, and call it "the extended contents". but the second screen should be a compact overview of the book, and a quick way to jump even to the sections at the very end of it. another thing which you can consider is to split up the monster. let's say the book has 5 sections, with 10 chapters in each one... the table-of-contents in the front could just list the _sections_. then, at the start of each section, you have another linked list, which gives the person all the chapter names for that section, plus links to the previous and next section-headers. this allows the reader to gracefully page through the sections of the book, getting a good idea of the chapters contained within each one... this is far preferable to a sprawling table-of-contents up front, where the screen-breaks will never coincide with each section... there are other ways you can make this work, too. for instance, p.g. has a book along the lines of "365 recipes"... yep, there's a recipe for every day of the year. pretty nifty, eh? so in that book, i turned each month into its own section, with the appropriate number of entries for the days of that month... depending on the book, there might be other ways that you can separate the table of contents into chunks that are meaningful. again, the idea is to make something that works as an _e-book_. that's the ideal, anyway. but, getting back to our reality... project gutenberg derivative formats often botch the t.o.c. badly. in many books, the items get joined together in one big jumble... in the worst cases, not al that atypical, the links don't even work. this sets a bad impression even before the person starts reading, not to mention the worse fact that it hurts the t.o.c. functionality. on the one hand, i know that the problem largely revolves around the fact that it's difficult to anticipate all of the t.o.c. snowflakes... so i'm tempted, for half a second, to excuse marcello's poor work. on the other hand, it shouldn't be that difficult to detect the t.o.c., and do whatever work is necessary to make it work in a useful way. (sorry, marcello, i tried to be nice, because i know you're a big fan.) *** a list of illustrations might come next, if the book has any... even if there wasn't one in the p-book, there should be one in the e-book. there's no law against improving the product. *** after that, you can put the obtuse crap from the title-and-verso. but... as far as most front-matter goes, it's not a bad idea to _move_it_ to the end of the book. an introduction should stay in the front, obviously, and a dedication doesn't take much effort to skip over, but a lot of the other stuff that's often presented as front-matter could -- quite easily -- be read at the end, and be just as useful. if you _do_ leave this stuff up front, though, you absolutely must give the person a link to skip it, and jump right into the book... do p.g. derivative formats do any of this? no, sadly, they don't. so these are ways that those formats could easily be improved. *** this is just to get you started, don. if you need more input, on matters involving the interior, i can give that to you as well. but you really should look for yourself. -bowerbird p.s. oh, and i know that a _lot_ of you other people are just as uninformed as don is, but -- unlike don -- you're too stupid to even ask any questions to get the knowledge you are lacking...

4 5

that error-report
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

al- i see that that error-report is on a burton book that jon noring used to talk about all the time... so i am highly suspicious that this error-report is just him yanking your chain, and that he has a clean copy of the text that he's holding back, because he wants to put you on a gerbil wheel. so i'm not gonna do any more work on this one. cheer up, though, because there are only 333 actual places with errors, and it looks like it is a solid report, not peppered with false-alarms. so if it is from noring, he's not being dishonest, just being "much less helpful than he could be"... either way, though, i don't feel like helping him, even in a roundabout indirect way like this one... -bowerbird

1 0

Producing epub ready HTML
by Robert Gibbins 01 Feb '12

01 Feb '12

Almost exactly a year ago I put an append on this forum about what was wrong with PG-generated epubs, offering my own hand-crafted ones to replace them. The response at the time could probably be summarised as "yes, we know about that, go away" Over the last year I've probably re-mastered (for my own consumption) about 50 of the PG-generated epubs, many of them Anthony Trollope's novels produced by Joseph Loewenstein and others. Mostly what I am doing to the actual works is, as Greg Newby described in a recent append, removing stuff to do with formatting/layout and instead tagging with structural information that relates it to layout specified in a css file. I also have standard templates for the PG front and back matter wrappers and the xml files that the epub v2 standard requires. The css file I use is not particularly complicated, it contains remarkably few styles, those that are there being largely derived from existing PG stylesheets. I make no claims for the excellence of my markup or stylesheet, I'm still learning, no doubt you chaps could do better, but I achieve results that are satisfactory to me, and without using any more complicated tools than emacs and winzip. My main point is that to do this sort of thing you don't need to be an ace compiler writer or rocket scientist. A year later I look again at this forum and you are still arguing about it, but nothing seems to have been achieved! Some of you want to produce a grandiloquent change management/source code control systems, that gives fine-grained cooperative shared access to pages in the work. (Personally I found that the only significant productivity improvement I got was to take the generated epub and stick all the individual xhtml files back together so I could do emacs global changes on them (regular expressions are wonderful!)and then split them back up into individual chapter files afterwards.) Greg Newby seems to want to kind of shunt everyone's epub contributions into a siding where they will wither and die from not being updated. Nothing is sacrosanct except no one else is allowed to touch David Price or David Widger's work. It seems to me that really all that is required is to recognise that a lot of the html in PG's archive is pretty antiquated by modern standards and needs to be updated. And basically this is a labour-intensive task which can't be completely automated because it wasn't originally done to consistent standards, because it happened over a decade or two during which html and ideas about how to use it evolved. So really what needs to be done is to call for volunteers to do the clean up. And since Greg seems to imply that he is currently deluged with offers of documents from people who think they have already done this, it seems to me that there is plenty of enthusiasm out there (including me!). And before that happens, some PG group (contributors to this forum, Wwers, whatever) needs to get together and write a 'How To' describing at the nuts and bolts level how to do the clean up. This should include a recommended set of styles (and the css that implements them) lots of advice about what not to do for people that don't already know, also including access to templates, css style sheets, etc. As my ex-employer used to say, 'there are no technical answers to management problems, but sometimes there are management answers to technical problems'. Bob Gibbins

5 7

"challenging" error-report challenge
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

al said: > Here's the error report on PG#3441. great. > I'm assuming the Locations and Pages are peculiar > to the reporter's ereader device, whatever it is. since s/he mentions a kindle, and because the kindle uses "locations" as its milestone term, i would guess it's a kindle. as for "pages", i'd assume that's in the archive.org material, namely from that version of the book, which might (or not) be the version of the book on which this e-text was based... > It's quite possible that some of the reporter's > proposed corrections may themselves be incorrect. of course. that is always the case. that's why you have to compare each proposed correction against the page-scan. > Don Kretz is my preferred supplier of the corrected version, > since he offered first. "preferred supplier"? ok... > bowerbird can work on any or all of the other 15 volumes. do you have error-reports for them? > Or, the two can collaborate on all 16, > if they choose to do that. Up to them. you might have misunderstood what i'm offering. i didn't offer to _do_ the corrections for you; i offered to build a system to show you how you could make it _fun_ for people to process the error-reports in a speedy way... and, lest it was not clear, i will code and mount a system, but i do not intend to share any code with you, or with p.g. as i said not too long ago, the ship on that sailed long ago. at this point in time, you couldn't _pay_ me to code for you. i'm willing to code a demonstration, for you and this list, but that's as far as it goes. just so we understand each other, al. > I refuse to deal with revised versions of the same etext > from two different people. you should be able -- and willing!, and thankful! -- to evaluate corrections coming in from _twenty_ people, or two _hundred_, assuming those corrections aren't overloaded with false-alarms. the cost of evaluating a correction-already-made should be nil... *** i had anticipated that i would have to tell you that you need to _instruct_ people on the best way to submit an error-report, but this person -- bless their little heart -- did an _excellent_ job... indeed, they did more than they needed to... for instance, they said this: > Location 170 (Page 4) > day like this; and now I bid thee enter among Gharib s host and, > "Gharib's" should have an apostrophe. > day like this; and now I bid thee enter among Gharib's host and, with an untrained naive error-reporter, you often just get that line with the "should have an apostrophe" version of the correction in it, and that line is the least useful of all of them -- totally unnecessary. also, since p.g. threw away all the pagenumbers, it's great that this wise person included the pagenumber, so we know what scan it is... otherwise, all they needed to say was this: > day like this; and now I bid thee enter among Gharib s host and, > day like this; and now I bid thee enter among Gharib's host and, the computer can tell the difference between the lines, and _highlight_ it for the person who is doing the checking, and find that line in the file, and show the page-scan for review, and then record the judgment of the reviewer to edit or stet, and then _make_ the necessary edits and save all the changes. it's all of that grunt work which my demo-program will handle. -bowerbird

1 0

i don't know -- third base!
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

greg said: > I'm entirely in favor of > automatic conversion > to & from different formats. > Of master formats. the problem is you don't _have_ a master format... except for maybe r.s.t., and its uptake is very slim. and you don't have any adequate converters either. your .rst-to-.html converter makes people unhappy. your .rst-to-.epub converter makes people unhappy. your .rst-to-.mobi converter makes people unhappy. your .html-to-.mobi converter makes people unhappy. your .html-to-.epub converter makes people unhappy. your .txt-to-.mobi converter makes people unhappy. your .txt-to-.epub converter makes people unhappy. your .txt-to-.html converter isn't as bad as the others, but it's still not anything near to what it _could_ be... > I was personally deeply involved in a whole > lot of our current setup for those things that's nothing to brag about, as clearly indicated above. > and am hugely optimistic about > improvements in the future. ok, there's something we can all agree would be nice... until you have a master-format, however, i'd think that your "optimism" might be a tad bit unfounded. but, hey, i'll be more than happy to admit it if you prove me wrong. the point is this: yes, your converters definitely need work. but the problem lies much deeper, i.e., in your input-files... until you have some consistency in your input-files, you will constantly be battling against nature's ability to make fools. and yes, greg, yes, yes, a thousand times yes, this is _exactly_ the pro-consistency argument which i've been making forever. and now you see why. and my constant prediction that you would have to face this very demon, sooner or later, has now also been vindicated... > Meanwhile, we have an immediate and useful way > to improve many reader's experiences, by > providing access to hand-crafted alternatives. except you're just setting yourself up for future complications. _inconsistency_ is the problem. you can't solve it with "more". > It should be obvious that I'm constantly seeking > the long-term view of Project Gutenberg. it's not obvious, because your "solutions" cause more problems. that's not a way to take "a long-term view". if you said instead that you were "seeking a short-term, temporary fix", then yeah, i might buy what you're selling. but even then, it is a bad idea. > That is why I'm thinking of solutions that > involve keeping all derived files or changes > we create/publish/distribute, forever. keeping crap around forever is not a long-term solution either. in fact, it's the exact opposite: a big albatross around your neck. > But that doesn't mean I'm against providing > some files that will only have a few days, months or years > of useful life before they are superseded. except you don't have a mechanism for deciding to supersede. and once those files are out in the wild, you can't "expire" them. the more you talk, the worse this idea sounds. especially since "the problem" which you're attempting to solve has _already_ been solved. we know, for the most part, how to mark up .html that will give us a good .mobi and a good .epub. (at least once we give up the ghost that .html can be a "master", such that the exact same .html gives good .mobi _and_ .epub.) the hard part, then, is getting d.p. postprocessors to _use_ that. they've each developed their snowflake workflow, and are loathe to give it up. especially since -- and this is the _hardest_ part -- they refuse to settle for a "dumbed-down" .html file that doesn't "look like" the paper-book. but... if you can give 'em "the look", they _will_ standardize their .html, for the greater good, because they actually _do_ want the work to become useful in more places. and, of course, if you would have listened to me _six_years_ago_, when i was telling you the same thing, project gutenberg could've been the role-model for google, and internet archive, and amazon. but you were off lost in your xml/tei haze of confusion instead... and even to this very day, you are _still_ overcomplicating things! -bowerbird

2 1

now we're finally getting "deep"
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

greg said: > I put huge backing into addressing this challenge, > and so did other people on this list and elsewhere. > The answer was: TEI. you're out of touch, greg. in a massively spectacular way. the learning curve on .tei is too steep for your volunteers. not just a little too steep, either... far and away too steep. which was obvious to even the most oblivious observers... and furthermore, your workflow never worked correctly... even the handful of volunteers who _tried_ to make it work ended up banging their heads against the wall repeatedly... that's no way to treat your workers. even if you paid them. and you don't. you expect them to undergo torture for free. you would know all this if you merely read the d.p. forums. so your first answer was a non-answer. indeed, a non-starter. > Then, a couple of years later, ditto. The answer was: RST. oh please. none of the proper groundwork was done on .rst... it's a proven methodology, one that is in use all over the place, one that's used to handle documents that are very complicated. but even now, you guys cannot get it to work correctly even for the comparatively simple types of documents that you prepare. and you'd know _that_, too, if you just read the d.p. forums... further, you didn't try to address the desires of the volunteers. you just dumped .rst in the middle of the street, and hoped that they would eat the dogfood. they sniffed it, but it smelled bad, so they turned their noses up at it, and that was the end of that. and just as with .tei, a handful of d.p. people have tried .rst, but -- just as with .tei -- it doesn't really make them all that happy. the converter is a buggy "work-in-progress", and marcello is... well, marcello is marcello, so of course shit's never _his_ fault, it's always the fault of someone else. so of course that's a fail. was there ever any reason to expect anything else? nope. none. > There are scarce few things that TEI or RST are not suitable for neither of them seems capable of making d.p. volunteers happy. but maybe that's not an important function in _your_ world, greg. > The WWers have procedures to process such files - bring it on. you've never reminded me so much of george w. bush as right now. let's go place a "mission accomplished" banner on gutenberg.org... > Remaining problems include: > - how to convince contributors to make > new submissions in these master formats that shoulda been your _second_ order of business. (your first, which you still haven't done, by the way, is to iron out the bugs in your converter workflow.) > - consideration of alternate master formats, as desired the more the merrier! > Solved problems include: > - automatically generating all derived formats, with > a far higher level of integrity than other master formats bullshit! > - applying fixes to the master and > then regenerating derived formats no, you don't know how to do that either. > - having an easily editable master format .rst is one of the heavier light-markup formats around... but as you're desperate, i'll grant you this one, for now... > But the unfortunate fact is that the "better mousetrap" > (or, at least, one that purports to have many of the > improvements over HTML+text as master formats), > was built and delivered years ago. if you really believe that, greg, that's sad. just plain sad. because that is so removed from reality that it's pathetic. > But not too many folks have taken the bait. that's because your bait tastes like crap. > I can already hear a few voices calling (or writing), > "that's because your bait sucks!" it doesn't surprise me that you could already hear us saying that your bait tastes like crap, because it does. > and "but your bait cannot do X" and so forth. that too. most of what it cannot do is give the people who do postprocessing over at d.p. what they require. which is why it's a wonder you ever thought it'd "work". just exactly who do you think is _doing_ the work? > It would be contrary to past experience > for anyone to think they can, indeed, come up > with a solution set that will be above criticism. bullshit, bullshit, bullshit. i demonstrated "a solution set" right here, last week, from test-suite through .html to .mobi and .epub... your not looking at it doesn't negate its existence... -bowerbird

1 0

volunteers are a litmus test on whether your workflow is lame
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

al said: > I've got an errata report on my hands > that's 3400 lines long, that I haven't > had the courage to plow through yet. > The reporter lists something wrong on > almost every one of the book's nearly 400 pages. that's a big job, to be sure. but it's not difficult. and if it seems like it is, you are doing it wrong. put up the files, and the scans, and i'll show you how to do it right. so it's _fun_ for people to do. > On top of that, he'd like an HTML version > created with the footnotes cross-linked and > from what I can tell, the page numbers inserted > because there are internal references to them. sounds totally reasonable to me, and furthermore, how the thing shoulda been done in the first place. > Question: where's an Errata Team when you want one? > Answer: as happened a year or so ago, they find out > what they're up against, and vanish. <g>) it wasn't the problem that frightened them away, al. it was the monster you proposed as the "solution"... :+) but hey, i'm sure it doesn't compare in the slightest to the years of volunteers that d.p. has chased away. -bowerbird

1 0

Re: [gutvol-d] Producing epub ready HTML
by Lee Passey 31 Jan '12

31 Jan '12

On Tue, January 31, 2012 2:52 pm, Greg Newby wrote: > On Tue, Jan 31, 2012 at 12:53:06PM -0700, Lee Passey wrote: >> On Tue, January 31, 2012 12:48 pm, Robert Gibbins wrote: >> >> [snip] >> >> > And before that happens, some PG group (contributors to this forum, Wwers, whatever) needs to get together and write a 'How To' describing at the >> nuts >> > and bolts level how to do the clean up. This should include a recommended set of styles (and the css that implements them) lots of advice about what not to do for people that don't already know, also including access to templates, css style sheets, etc. >> >> I, for one, would like to see the style sheets you have created. Mr. Newby, can you provide us with some sort of repository where we can share this kind of stuff? > > I already did, and you're using it: > http://lists.pglaf.org/mailman/private/gutvol-d/ Well, this isn't very persistent, but if it's the best you can do... Robert, if you can zip up your style sheets and attache them to a message, I'll try to post them on my web site so they are available for discussion. I'll post my own reformulation of the PG CSS there as well.

1 0

Re: [gutvol-d] it's monday, after noon, let's get the crowdsourcing started
by Bowerbird＠aol.com 31 Jan '12

31 Jan '12

al said: > My suggestion, start with etext03 (etexts 3601-4800). that's a consideration. let's see what greg says... and what the other teams in the tournament say. looks like so far it's: > 1. bowerbird. > 2. marcello. > 3. lee with dkretz second in line right behind him. > 4. jim "j-mad" adcock. > 5. john "limpid" redmond. > 6. keith. > 7. karl eichwalder. > 8. hunter "don't forget guiguts" monroe. > 9. carlo "garweyne" traverso, and his students. no offense intended if i left anyone out, just let us know. and if you're listed, but don't want to play, please say so. 3601-4800 would be 1200 books... if we want to cut that number down -- at least for the opening round -- i suggest 4321-4567 -- about 250... that's enough to weed non-contenders. it strikes me that, with e-text numbers this low, this'll be a clean-up job rather than an exercise in how to generate better derivative file-formats, which is what i think greg was aiming at, but i'm ready to go with the consensus of the tournament. al, you'll need to point us to _canonical_versions_ of these books -- preferably at internet archive, or someplace where you can furnish us with scans -- so we're all working toward the same end-product. -bowerbird p.s. oh, and all you other teams, please take all of your preparation plans elsewhere; we don't need to hear 'em. the proof will be in the pudding -- where it _always_ is... unless, that is, you want to amuse us with your fumbling.

3 2

PG governing body
by Karen Lofstrom 31 Jan '12

31 Jan '12

On Tue, Jan 31, 2012 at 11:32 AM, Greg Newby <gbnewby(a)pglaf.org> wrote: > It should be obvious that I'm constantly seeking the long-term view of Project Gutenberg. That is why I'm thinking of solutions that involve keeping all derived files or changes we create/publish/distribute, forever. But that doesn't mean I'm against providing some files that will only have a few days, months or years of useful life before they are superseded. I'm starting a new discussion here. I hope that I can present this topic in such a way as to avoid an immediate descent into acrimony and personal attacks. I have the impression that when Michael Hart was alive, he *owned* PG. It was his baby, he had kept it going for years, he owned the name, etc. There was a board of directors, but it was not elected by PG volunteers. Greg Newby was MH's deputy, and over the years, took on much of the workload. Greg has served selflessly, silently, devotedly. (Correct me if I'm wrong in my impressions, please.) However, I note with some disquiet that Greg seems to have stepped into MH's shoes, as the man who runs PG and gets to make all the decisions. See the quote above. I am not comfortable with this as anything more than an interim measure, until MH's estate is settled and the copyrighted name problem is sorted out. I hope that PG can make a transition from a monarchy to an elected board, just as DP did. I am not sure what form this board should take, and how elections might be run. Who counts as a volunteer with voting rights? That was easier with DP, which is more organized. It might also be a good idea to have ex-officio members from other digital preservation organizations, such as Internet Archive and Hathi Trust. Perhaps a certain number of seats could be set aside for such organizations, and the choice of organizations NOT be set into stone. Old groups may falter and new ones form. I do hope that Greg will see this not as a personal attack against HIM, but a move towards an organization, and a library, that is not dependent on any one person's efforts. I do not see this as deposing Greg, who will continue to be essential. I just would like to see him as the CEO, not the king. With some trepidation ... -- Karen Lofstrom

1 0