- gutvol-d - lists.pglaf.org

... and then there are conversion tools
by D Garcia 15 Dec '11

15 Dec '11

I'm surprised that no one has yet mentioned pandoc as a useful piece of kit. http://johnmacfarlane.net/pandoc/

2 1

a review of some digitization tools -- 020
by Bowerbird＠aol.com 15 Dec '11

15 Dec '11

so, we have compared the .html version from my converter with the one from p.g., and learned they look virtually identical, although the p.g. version has "tag abuse" in its coding, while my version does not... (you are _welcome_ to give full criticism on the .html being generated by my converter... give me feedback now before it goes public.) then yesterday, we compared the _.mobi_ i created with the p.g. auto-generated one, and found the p.g. version had many flaws, most of them stemming from the tag abuse. so today we're gonna compare the .epub i made with the p.g. auto-generated one. an .epub file is mostly a "wrapper" around the .html, so i'd say we can now guess that the .epub is going to have some problems, especially since we have been experienced. here's where to download the p.g. .epub: > http://www.gutenberg.org/ebooks/16736 and here's where you can find mine: > http://zenmagiclove.com/misc/ukagl-epub.epub > http://zenmagiclove.com/misc/ukagl-epub.zip *** the results are exactly as we would fear. so i'm not even going to step through the drill. here are some screenshots; you'll figure it out. > http://zenmagiclove.com/misc/bac-epub001.jpg > http://zenmagiclove.com/misc/bac-epub002.jpg > http://zenmagiclove.com/misc/bac-epub003.jpg > http://zenmagiclove.com/misc/bac-epub004.jpg > http://zenmagiclove.com/misc/bac-epub005.jpg > http://zenmagiclove.com/misc/bac-epub006.jpg > http://zenmagiclove.com/misc/bac-epub007.jpg > http://zenmagiclove.com/misc/bac-epub008.jpg > http://zenmagiclove.com/misc/bac-epub009.jpg > http://zenmagiclove.com/misc/bac-epub010.jpg > http://zenmagiclove.com/misc/bac-epub011.jpg > http://zenmagiclove.com/misc/bac-epub012.jpg garbage in, garbage out. sorry, no way around that. and again, one more time, because it must be said, this was a _simple_ book! headers and paragraphs! oy vey... *** to repeat, i welcome feedback on my .html coding! z.m.l. input file: > http://zenmagiclove.com/grapes009.txt python script to create .html output: > http://zenmagiclove.com/cgi-bin/tday2011.py once this is released publicly, i'll likely be deluged with requests to fiddle with stuff, so i will probably deny all of them, with little or no consideration, so this is your _only_ chance to get it how you want it. *** tomorrow, i sum up the comparisons from this week. next week we get back to building that python script. -bowerbird

1 0

Re: [gutvol-d] Let's keep the facts in prespective.
by Bowerbird＠aol.com 15 Dec '11

15 Dec '11

keith said: > Though HTML5 seems to be the future, > it is so to say futuristic. I mean > its final is expected first in a decade! .epub2 still doesn't have one viewer-program that supports the spec totally and completely. not a single one... not after an entire decade... and as the attention has now shifted to .epub3, we can confidently guarantee that _it_never_will_. of course, this means that even by the year 2020, .epub3 will not have a single compatible viewer... but since we'll be on .epub6 by then, who cares? i advise offering stuff in every format you can, and hope like heck your audience can read one. just one, please, that's all i need, one good one. > Let's keep the facts in prespective. good idea. here's the reality. for the next few years, very little is going to happen to move e-book software ahead. adobe has been doing as little as it can... in 2011, it made one update to adobe digital editions. one! and adobe is the lead software company for i.d.p.f. apple recently came out with an update for ibooks. atop the list of new features was "night-time mode". that's right, they finally did a white-on-black view! imagine the programmer-years _that_ feature took! good thing they're a rich company, and can afford it. but wait, there's more! they _also_ now give you the option to remove the print-book background-image. their programmers worked overtime to deliver that! what powerful capabilities will they dream up next? google? readers are _stampeding_ to google-books. and amazon has finally decided to drop mobi! yay! so we no longer have to deal with a format designed in 1995. unless, that is, we want to still move books to all the kindle hardware that's out there right now, because amazon isn't going to update their firmware. so now we'll have to support mobi _and_ kf8... yay! so tomorrow will be the same as yesterday, buddy... .pdf, .html(5), .epub(2), .mobi(1995), .kf(8), .txt(70). can you feel the _excitement_, keith? it's _palpable_! -bowerbird

2 1

Re: [gutvol-d] let's get some tutorials from the xhtml crowd
by Bowerbird＠aol.com 15 Dec '11

15 Dec '11

paul flo said: > Please tell me you didn't infer all that > from the discussion that ensued from Jim's thoughts on > fixing paragraph separation on the Kindle? heavens no. i didn't infer _any_ of that, from _anything_ jim has said in the past month or so, when he went into my kill-file... i have an 8-year history on this listserve. so i have _plenty_ of backlog to "infer" things from, paul. in fact, this recent engagement made me realize something. i originally started battling the xhtml people when they were feeling heady, back in 2003-2005. the world was their oyster. back then, almost everyone said xml/xhtml was "the future"... but in the ensuing years, everything ground to a slow crawl, until finally, people realized xml/xhtml had very little future, and only then as a format for machines to swap information... meanwhile, i stayed tenacious in my stance for light-markup. i started doling out more and more examples, until all of my antagonists came to the conclusion they were being defeated. so the arguments became less heated, and then less and less, until finally we were going months on end without having one. but lately, there's been this "rear-guard" action. i know that it's just the dying gasp of some hardcore xhtml enthusiasts. the whole thing makes me feel nostalgic in a way, though... we've gone through the entire cycle. i started as underdog, climbing the mountain, as xhtml slid down the other side... it's like the gandhi quote that ends with "and then you win". > Along with a straw man, apparently. actually, my request for elucidation of your workflow is an honest and sincere one. i really wanna know how you do it. because what i keep hearing is "i have this xhtml file," and i think people would like to know where that _came_ from. > For me the tools are Vim, Perl, xsltproc, Calibre, > kindlegen, git, xmllint and elbow grease. i don't see how any of those tools help you code your xhtml. did you use abbyy? did you have it save its output as xhtml? do you type in your angle-brackets and xhtml tags manually? > Thank goodness I'm not claiming to have produced > a magic button that can produce nineteen formats > before breakfast from an input language used by no one. paul, really. you're making the exact type of mistake that my early adversaries on this listserve made, using terms like "claiming" and so on. have you actually went and _run_ the python script that i'm writing, which does those conversions? have you compared the input file and the output files to see just exactly how the script works. were you able to grok it? was there anything difficult about the reverse-engineering? because if you have some questions, i _can_ answer them... last year i did this demo in perl. wanna see those scripts? (no, of course you can't have them. but you can _run_ em.) and for goodness sake, paul, i understand fully well that i am not showing you all of my cards. i am doing that on purpose. but if you wanna convince yourself that light markup _works_, go look at markdown. look at restructured text. or asciidoc. there are lots of light-markup formats that are all worked out, that have been proving their worth for year after year already, in heavy-duty environments. restructured text has been the format of choice for the python community's documentation since, i dunno, something like 2004... when you pooh-pooh something that already exists, it just makes you look stupid... do just a little bit of homework to spare yourself from that, ok? -bowerbird

2 1

The problems with paragraph formatting at PG
by Jim Adcock 15 Dec '11

15 Dec '11

One of the things that make books ugly at PG is the problem of simple paragraph formatting, which sounds like a simple issue under HTML, but which in practice becomes a mess particularly in EPUB and even more so in MOBI (Kindle). For people who don't understand or don't believe this yet, example files will be provided at the bottom of this posting. People who believe they know HTML believe they know how this issue works, namely one can specify top and bottom margins for one's paragraphs using units one chooses, and similarly choose to indent the first line, or not, and then the HTML browser goes off and does what you ask. And if you specify both top and bottom margins then the browser "merges" those margins so that a paragraph followed by a paragraph doesn't end up with a double-wide margin between those paragraphs. And that is how it works, more or less, in HTML browsers, except, not really, not necessarily, not even there, because even in HTML browsers style specifications are *hints*, not firm rendering commands, and need not be followed the way you specified them. Even in HTML browser land there are still people running HTML browsers old-enough not to merge top and bottom margins, for example, or who have overridden these specifications. But, the situation becomes much worse in EPUB, and especially MOBI land. In EPUB land there is common EPUB rendering software out there that simply ignores paragraph formatting specifications, and substitutes their own. Like Aldiko, for example. [Adobe Digital Editions and its derivative such as Sony and Nook seem to get this right] In MOBI land -- read Kindle and kindlegen (which is pretty much how everyone including PG is force to make MOBI files) there are two big problems: 1) Top and bottom margins are NOT merged. When a paragraph follows a paragraph then the vertical whitespace between those two paragraphs is added to each other. 2) Top and bottom margins are ROUNDED to the closest 1em. So, say for example you take a "split the baby" approach of specifying a top-margin of 0.5em (or the equivalent in one or another of the measures) and a bottom-margin of 0.5em. What do you get displayed on a Kindle? You get two full vertical "blank lines" between every two adjacent paragraph. Now if this doesn't sound to bad to you, consider what happens when a "dialog" happens between two characters in your book: You get two full vertical "blank lines" between each line of dialog. And what happens when you specify poetry using the "one paragraph per line" approach? You get two full vertical "blank lines" between each line of poetry. Does this look attractive? No it does not. It looks d*mned ugly. Barely readable, in fact. How then does one work around these problems? Once one recognizes that there really is a problem, then three (partial) work-around solutions come to mind: 1) Do not even specify paragraph formatting, but rather allow the built-in paragraph formatting in each HTML, EPUB, or MOBI device to do its job. 2) Specify (say) a 1.0em top margin and a 0.0em bottom margin. 3) *Almost* "Split the Baby" by specifying a 0.51 top margin and a 0.49 bottom margin. Finally, understand that all paragraph style specifications are *hints* and if you are relying on details of paragraph formatting to make your book "work" then you are going to have disappointed readers on one or another device. IE: use the <p> tag to specify paragraphs, not things which are not paragraphs. Example test files demonstrating the problem: http://freekindlebooks.org/Dev/marginstestpg.mobi http://freekindlebooks.org/Dev/marginstestpg.epub -- not sure why, but my browser likes to rename marginstestpg.epub into marginstestpg.zip on downloads. Rename it back to marginstestpg.epub to make most epub display software happy playing with it again. PS: I have verified that the problems discusses are not problems introduced by epubmaker per say.

7 14

a review of some digitization tools -- 018
by Bowerbird＠aol.com 15 Dec '11

15 Dec '11

keith said: > I can not tell you why BB does it thanks for declining to speak on my behalf, as i am no longer answering jim's questions, not unless someone else rephrases them first. this ensures that the question is a worthy one, and not simply a bad waste of everyone's time. if you, keith, _intended_ to refer this question to me simply by quoting it in your reply to it, let me reiterate that you must _rephrase_it_... that ensures that the question is not _vague_, or _misleading_. or at least it minimizes that. sadly, jim has something of a mush-mouth... for example, just this one time, to elucidate, i will pick up jim's question. jim said: > What I am not clear about is why BB insists > that what one starts from must be > an "an impoverished text-file" first, if someone is "not clear about" something i've said -- or not said, as the case might be -- the best thing to do is ask me for my reasoning. i can assure you that i've thought about it, and i do indeed have good reasons for my decision. so -- just like they require you on jeopardy -- your question must be phrased _as_a_question_. jim has instead phrased this as an _assertion_. moreover, jim said that i "insist" on something. the problem is, jim's assertion is simply incorrect. meaning, of course, his extension that i "insist" it is doubly incorrect, compounding things further. this is typical of jim. wrong piled on top of wrong. i do not "insist" that people start with plain-text, not if they realistically have a choice in the matter. look at a post i made as recently as november 16th: > http://lists.pglaf.org/mailman/private/gutvol-d/2011-Nove mber/008360.html you will clearly see that i explicitly recommended that a digitizer have finereader output an .rtf file, because .rtf retains some vital styling information. indeed, this was post #001 in my current series, the very thread under which this post was made! so jim is trying to commandeer _my_ point, as if he was using it as counterargument against me! once again, he shows up on _my_ side of the net, hits the ball back to his court, and calls it a winner. it's a very strange way of agreeing with me... :+) i also repeated the .rtf suggest on november 21st: > http://lists.pglaf.org/mailman/private/gutvol-d/2011-November/008385.html this was a suggestion aimed at alex and archive.org. again, the intent is to retain vital styling information, which you will otherwise have to reinsert once again... so that's the best way to "start". if you have the choice. but in a lot of situations, people do not have a choice. if you want to use the o.c.r. from an archive.org book, you are stuck with plain-text, absent the styling info... or if you're a volunteer at d.p., postprocessing a book, you are stuck with plain-text, albeit with styling info... (even if a lot of it is rendered in d.p. pseudo-markup.) so i don't "insist" that a digitizer "start with" plain-text. i just recognize it as a common reality of the situation. now, i _do_ recommend that a person take the .rtf and convert it to an ascii file, retaining the vital styling info, so maybe jim was "really" asking why i recommend that. but i gave up on trying to interpret what's in jim's mind. still, if anyone wants me to answer that question, ask it! -bowerbird

3 2

Re: [gutvol-d] The problems with paragraph formatting at PG
by Bowerbird＠aol.com 15 Dec '11

15 Dec '11

paul flo said: > There is no ASCII version; > that was showing > how the poem looks > in the original book. well, i certainly hope you will forgive me for being pedantic when i point out that that most certainly is an ascii version, as you "digitized it" in the process of typing it into an e-mail. and it's somewhat ironic, when you think about it, isn't it?, that it's so easy -- using ascii -- to "show what it looks like". to indent, you press the spacebar the right number of times. (or maybe you used the tab key, i dunno -- it's all the same.) but i grok what you are saying. you make no text version. which is fine, except the discussion here revolves around a workflow project gutenberg might use, and p.g. requires a text version, so you can understand what i was thinking... (see, there's even a "pg" in the subject of your e-mail.) your personal workflow will find interest here, on this list, but _more_ so if it could be adopted by a large cyberlibrary, of the type exemplified by project gutenberg or archive.org. it is difficult to find volunteers to staff such a cyberlibrary, and it would be even more so if we had to expect them to be able to work with text whilst it is embedded in (x)html, so your workflow -- "i produce an xhtml master version" -- won't ultimately find much relevance in such a cyberlibrary. your point -- that one can modify xhtml for kindlegen -- is a good one. but only provided one already _has_ xhtml. if, on the other hand, one turns light-markup into (x)html, using a script, then one might just as well use that script to create something directly from light-markup for kindlegen. which is what i do. i fly directly, instead of doing a layover. i see the value of your point in the context of your workflow. so, do you see the value of my point in the context of mine? if so, then we have reached a mutual understanding. :+) > I edit the XHTML and push a button to convert, > just as you'd edit your zml, well, if you mean to imply that you can edit your xhtml just as _easily_ as i can edit my .zml, i'd need evidence. because i have a lot of skill zooming around a text-file. as we say in the programming world, i'm at bare metal. but you and i are not the real parties of interest, are we? the people who will be doing the work are the volunteers who donate their time and energy to our cyberlibrary, and i am firmly convinced that _they_ can edit my .zml _much_ more easily than they could ever hope to edit your xhtml... if you want to say otherwise, you'll need a _lot_ of evidence. > I'd be interested to see how you markup > and transform poetry in a way that copes with > different font sizes and screen with zml and your tool. the markup that my routine ends up sending to kindlegen probably looks very similar to the markup that you send it. indeed, i could make it be _exactly_ identical if i wanted to. i did some work on poetry in the early months of this year, when i showcased an online app to do these conversions... were you subscribed then? if not, i can dig all of that out... but it basically involves pushing in the overall left-indent for a line (in the case that it will need to get wrapped), and then giving the line an outdent (so it starts where it should). there's really not much rocket science to it... but if you can come up with something to do the job better, i'll "borrow" it. by the way, as a poet, it warms my heart to see that so many of you people out there are digitizing poetry. good for you! -bowerbird

2 1

a review of some digitization tools -- 018
by Bowerbird＠aol.com 14 Dec '11

14 Dec '11

first, let's just cut directly to the chase today... we're going to compare my converted .html to the .html version of "books and culture" which project gutenberg has posted as their version... > http://www.gutenberg.org/files/16736/16736-h/16736-h.htm here are a bunch of head-to-head screenshots: > http://zenmagiclove.com/misc/bac-html001.jpg > http://zenmagiclove.com/misc/bac-html002.jpg > http://zenmagiclove.com/misc/bac-html003.jpg > http://zenmagiclove.com/misc/bac-html004.jpg > http://zenmagiclove.com/misc/bac-html005.jpg > http://zenmagiclove.com/misc/bac-html006.jpg > http://zenmagiclove.com/misc/bac-html007.jpg > http://zenmagiclove.com/misc/bac-html008.jpg > http://zenmagiclove.com/misc/bac-html009.jpg > http://zenmagiclove.com/misc/bac-html010.jpg > http://zenmagiclove.com/misc/bac-html011.jpg > http://zenmagiclove.com/misc/bac-html012.jpg across the board, we see that the two versions are very much the same, with only occasional differences. and most of the differences are "fixed" now, which you can verify, if you like, by running the program. > http://zenmagiclove.com/cgi-bin/tday2011.py it was a simple matter of tweaking my script a bit. for instance, i had to revert my curly-quotes back to straight-quotes, repair some typos, delete the spaces i'd put around my em-dashes, and so on. so now the two look almost completely identical... this makes it abundantly clear that -- at least for this simple book -- my script works just as well as the "hand-formatting" performed by this volunteer. so that's the main takeaway of today's lesson... that's unsurprising. if/when a book consists mostly of headers and plain paragraphs, then all you have to do to make different versions look the same is to ensure their basic setup of those two structures is equivalent. *** so now let's circle back around for the exposition... *** we'll start with a short refresher on the workflow. we whipped the text into shape, and now we're ready to convert it, into .html -- "and beyond", as buzz lightyear has been known to proclaim. you can see the text file here: > http://zenmagiclove.com/grapes009.txt it looks very much like an o.c.r. output file, or a "concatenated text file" (c.t.f.) from d.p. it's also quite reminiscent of a p.g. e-text... you know, the kind that jim labels "txt70", and lee calls "an impoverished text-file"... it's also a .zml file. it follows the z.m.l. rules. my thanksgiving-day python script has been converting that .zml file into our .html output. lots of lots of people -- including jim and lee -- have suggested that p.g. should mandate .html. or that _any_ cyberlibrary should use .html as its "master-format" -- the one it actively maintains. what these guys are never clear about is exactly _how_ volunteers are supposed to transform the _text_ files (of the type we mentioned up above, the type you end up with after your digitization) _into_ .html. they're strangely mum about that. but _i_ have an answer -- i.e., a script like mine! are there other converters which could be used? well, of course there are. but the usual situation is that the other guy's converter doesn't convert the same way that _you_ would prefer to convert.. (we see that keith said that very thing just today... indeed, he just assumes that that'll be the case.) can't you do text-markup "by hand"? using just a text-editor? that's what these guys usually say. and yes, you certainly _can_ do markup by hand. and not everyone can, or wants to. but "you" can. it's just not a lot of fun. a lot of it is grunt-work. so you find yourself using "regular expressions" -- if you know what they do and how they work. and then you find yourself with _lots_ of reg-ex, and the next thing you know, you write a script... so you might as well _start_off_ writing a script... especially since i am showing that it's so easy... *** now, you couldn't be blamed if you thought that the appeal of a script is that _it_makes_life_easy_ regarding the work involved in coding the .html. but that's only the half of it. the other benefit from a light-markup approach is that the output will be _dependably_consistent_. from the perspectives of _re-use_ and _re-mix_, an ability to _know_ that the output is _consistent_ is extremely valuable, because you can depend on the output structure, and treat it programmatically. and that gives us _immense_power_ down the line. this contrasts _significantly_ with the reality of p.g., as it is currently construed, which is _inconsistency_. very little p.g. .html has "dependable consistency"... even files from the same producer can -- and do! -- take different approaches, sometimes varying widely. and, as you'd imagine, files from different producers often have almost nothing in common between them. so these .html files can't be treated programmatically. you can try, and you might experience some success, more or less, depending on how wild it is what you do, but you can't go in _expecting_ to have 100% success. or even 70%. on some things, you might get only 20%. essentially, every .html version is a "snowflake", in that it is different from every other .html file in the library. and we see this demonstrated on this very book that we've been using in this series -- "books and culture", so let's discuss this specific example next... *** as we have seen, "books and culture" is a simple book, from the standpoint of digitization, since it consists of plain old paragraphs and chapter-headers, containing only a couple blockquotes to break up the monotony... so you'd think "the markup" on it would be predictable. i certainly thought so; but as it turned out, i was wrong. in order to make my version "look like" the p.g. version, i was originally going to simply copy in their .css file... but as soon as i did that, i realized there'd be trouble... i've appended the .css, so you can see for yourself... for a straightforward book such as this one, we'd expect that the .css file would be sparse, if there was one at all. about the only thing that _routinely_ needs to be altered for the look-and-feel of old books is to center headings. (and it's often better to achieve that by using a "style" command inside the header tag, most especially since the kindle is infamous for not even using the .css file.) but the .css for this book from p.g.? it's _not_ "sparse". it defines some styles, a few of 'em quite questionable... the most troubling, in that vein, is called ".chaptertitle". but others also of concern are ".chapter", and ".quote". the reason the "chapter" classes are a red-flag is that you generally want the chapters to get [h] header tags. structurally, that is what they "are", and a number of tools are geared toward this "semantic" interpretation. sure enough, when i looked at the .html tags within, there was indeed trouble. as feared, chapter headers were not marked with [h] header tags, but with a [p]. that's right. the headers were declared as paragraphs. oops! i will let someone else give you the "tag abuse" lecture, and dole out the appropriate flogging here... but you don't have to be compulsive to see that's bad. we also find the blockquotes were marked with a [p], with the class defined as ".quote", more "tag abuse". ditto with the footnote, although that'll be less clear. so, you ask, was anything actually tagged as a header? well, as a matter of fact, yes. here's the full list of them: > [h1] BOOKS AND CULTURE > [h4] By > [h2] HAMILTON WRIGHT MABIE > [h4] NEW YORK: PUBLISHED BY > [h4] MDCCCCVII > [h4] [i] Copyright, 1896 > [h4] [span class="sc"] By Dodd, Mead and Company > [h4] University Press: > [h4] To > [h3] EDMUND CLARENCE STEDMAN > [h3] CONTENTS oops! they did it again. aside from the [h1] and maybe the [h2] -- or should i say "mabie" the [h2], hahaha! -- that's pretty much presentational tagging throughout... meaning more lecturing and some additional flogging. all in all, even on this very "simple" book, sorry to say, the tagging would have to be assigned a failing grade. i guess if you wanted to be kind, you could give it a "d". *** now some people might offer up as an explanation that this book was done a long time ago -- october of 2005. but if you think about it, that "excuse" really backfires... because the problem is _not_ that the postprocessor was incapable of handling a relatively difficult task... on the contrary, the postprocessor tried to "get fancy", and basically tripped over themselves on a simple task. it's like the person who misses the very first question on "who wants to be a millionaire?". what were they thinking? and believe me, in the ensuing years, the postprocessors over at d.p. have gotten even _more_ fancy with their code. they've accumulated lots of crud and a ton of tricks as well, and they wanna throw _all_ of it into every project they do. so the snowflake problem is getting _worse_, not better... *** it is also worth noting, before we leave this book behind, that this was a book which had _extremely_high_visibility_ -- an excruciating amount -- at the time it was posted... it has the distinction of being the first public-domain book which was released by google as part of its scanning project. so it was a major milestone when d.p. picked it up to do it. that -- all by itself -- would've made it a memorable book. but this book also marked another historical turning-point. people who were around back then might remember that the distributed proofreading site had come into its own... the volunteers over there were feeling very smug about it, claiming loudly that their accuracy trumped everyone else. they were superior. and their arrogance knew few bounds. pride goeth before destruction, a haughty spirit before the fall. the truth is that they _were_ "superior", at least if compared to the rather flawed e-texts that comprised the p.g. library... but their accuracy was far short of the perfection they claimed. nonetheless, they proclaimed far and wide that their method of _collaboration_ meant that more eyes checked the text and -- ipso facto -- that meant that the output was more accurate. it's a neat argument, in the sense that it has intuitive appeal... the thing was that it pissed off some people, who thought that they did a darn good job all by themselves, working all alone... one of those people was a guy by the name of jose menendez. so, when google released "books and culture", jose proofed it. and then he waited for p.g. to proof it. and waited and waited. and when d.p. _finally_ released its version, and p.g. posted it, some 6 months later, jose pounced on it. and he found juice... it ends up that d.p. didn't do a "perfect" job on its digitization. it didn't even come close, with 48+ errors -- yes, over 48! -- on a 279-page book (i.e., 1 every 6 pages) that is under 210k. and jose detailed all of those errors. painstakingly. ouch. but that number wasn't the most embarrassing thing, not at all. the most embarrassing was that d.p. _dropped_an_entire_page_! mighta been google's scanning error originally, can't remember, but it was still true that d.p. failed to notice that missing page... but jose hadn't. nor did he miss the 49 errors that d.p. missed. d.p. was aghast. all of their claims of superiority were dashed. their vaunted collaborative system had been out-performed by a single individual, working all alone, with just his own two eyes. d.p. rushed to fix its errors -- jose helped them very graciously. and p.g. posted the corrected version just as soon as it could. if you look at the "old" folder, you'll see that the update came within a week, which -- as most of you know -- is astounding. and even with that, there's some history that's been rewritten, since not even the "old" file is missing that big chunk of text. (it ended up that p.g. got the missing page of text right away, but then it neglected to fix the other 48+ errors jose found. when that was pointed out, they did an update in one week.) the history of all this was documented online at the time... the main "battlefield" for it was the "bookpeople" listserve: > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-09-30,3 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-05,4 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-06,7 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-11,3 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-21,4 right around the time of this disillusionment, d.p. decided that it needed to have more than two rounds in its system, splitting the task into "proofing" and "formatting" rounds. and -- at least for a short while, anyway -- d.p. stopped making big claims about how "superior" their system was, and acknowledged that "some" individuals did good work. gradually, however, that d.p. overconfidence has returned. once again, it got smug, claiming a degree of "perfection" which -- at least the last i checked -- was not warranted... the standard line at d.p. is that "our product has improved", and they conveniently split their history in half, admitting "the first half" of books they've done contain flaws, whereas "the last half" reflect the better results that they now attain. this slippery scale is very convenient. back when there were 30,000 books in the library, #16736 was in "the good half". now, however, it has been safely relegated to "the bad half", and any flaws in it -- flaws that've been there since 2005 -- have now been "explained away" as mere historical artifact. but i am sure that lee will be all too happy to tell you that he's been giving his "tag abuse" lecture since before 2005 -- heck, i bet you can find it in the archives of this list -- and that this .html was just as faulty back then as it is now. my point is a bit different. i stress that this was a high-profile e-text, even back then... so the questionable .html is _not_ due to "a lack of attention". you can bet that the whitewashers looked closely at this one. especially when they had to re-do it twice within two weeks... nevertheless, it _still_ ended up being badly flawed... for a drop-dead simple book like this one, that's shocking. and like i said, things have only gotten "fancier" since then... so if you wanted an illustration of the "snowflake" problem, you'd be hard-pressed to find a better example than this... *** as i noted, the body of the book looked almost identical between the two versions, which is as we would expect... but there were visual differences in the _frontmatter_, so i will show them to you for the sake of completion... for instance, here is the very top of the file: > http://zenmagiclove.com/misc/bac-html013.jpg the p.g. legalese is really a turnoff. when will p.g. get smart, and condense the stuff at the very top to one line, embedded in the middle of the cover-page, which just points to the license to the end of the file? after the legalese, the p.g. titlepage is quite nice: > http://zenmagiclove.com/misc/bac-html014.jpg it's nicer than mine. (i didn't put effort into mine.) especially because it has that wee flower doodad... i don't like the fact that the p.g. titlepage bleeds into the following pages, which also bleed into each other, with no separation at all, but maybe that's just me... i also like the look of their table-of-contents: > http://zenmagiclove.com/misc/bac-html15.jpg again, i might need to put some work in... or not... as noted above, the only wrinkle in the book's body was the presence of a couple blockquote paragraphs. the flow varies, since i retained the p-book linebreaks: > http://zenmagiclove.com/misc/bac-html016.jpg > http://zenmagiclove.com/misc/bac-html017.jpg pedants will also note that my blockquote does _not_ have the indent present in the p-book. (yes, i looked.) in return, i will smile at them, mocking their pettiness. which brings us to the close of today's post... *** in sum, let's repeat the takeaway of this experiment: for this simple book, my conversion worked as well as the "hand-formatting" performed by a p.g. volunteer. indeed, from the standpoint of _re-mix_consistency_, the product of my conversion is seen as far superior... -bowerbird p.s. here is the .css file from "books and culture" at p.g. > http://www.gutenberg.org/files/16736/16736-h/16736-h.htm [style type="text/css"] [!-- body {margin-left: 3em; margin-right: 3em;} p {text-indent: 1em; text-align: justify;} .ctr {text-align: center; text-indent: 0em;} .noindent {text-indent: 0em;} .sc {font-variant: small-caps;} .chapter {text-indent: 0em; margin-top: 2.5em; margin-bottom: 1em; text-align: center; font-size: 115%; font-weight: bold;} .chaptertitle {text-indent: 0em; margin-top: .5em; margin-bottom: 1.5em; text-align: center; font-size: 115%; font-weight: bold;} .quote {text-align: justify; margin-left: 2.5em; margin-right: 2em; text-indent: .75em;} h1,h2,h3,h4,h5,h6 {text-align: center; margin-top: 1.5em; margin-bottom: 1em;} hr.long {text-align: center; width: 95%; margin-top: 2em; margin-bottom: 2em;} hr.med {text-align: center; width: 60%; margin-top: 1.5em; margin-bottom: 2em;} .footnote {font-size: 96%; text-indent: 3em;} ul.nameofchapter {list-style-type: none; position: relative; width: 75%; margin-left: 4em; line-height: 150%; font-size: 76%;} ul.TOC {list-style-type: upper-roman; position: relative; width: 75%; margin-left: 4em; line-height: 150%; font-size: 96%;} .ralign {position: absolute; right: 0;} --] [/style]

4 4

Re: [gutvol-d] The problems with paragraph formatting at PG
by Bowerbird＠aol.com 14 Dec '11

14 Dec '11

paul flo said: > I'm suggesting doing some pre-processing > to the HTML that you give to Kindlegen ... > Here's a concrete example: thanks for the example. it makes it much simpler for me to make my point. for convenience, i will label these three sections... *** this is "version 'a', the ascii version"... > The first line of my poem > The second line of my poem > A longer line comes next, and goes on a bit > but still does not rhyme > The last line ends with a flourish! *** this is "version 'h', the .html version"... > I've decided that a flexible way of marking this up in HTML is this: > > [html] > [head] > [style] > .poem { margin-left: 2em } > .line, .iline { display: block; text-indent: -2em } > .line { } > .iline { margin-left: 1em } > [/style] > [body] > [div class="poem"] > [p class="verse"] [span class="line"] The first line of my poem[/span] > [span class="iline"] The second line of my poem[/span] > [span class="line"] A longer line comes next, and goes on a bit but still does not rhyme[/span] > [span class="iline"] The last line ends with a flourish![/span] [/p] > [p class="verse"] [span class="line"] The first line of my poem[/span] > [span class="iline"] The second line of my poem[/span] > [span class="line"] A longer line comes next, and goes on a bit, but still does not rhyme[/span] > [span class="iline"] The last line ends with a flourish![/span] [/p] > [/div] > [/body] > [/html] > > You'll note that this needs CSS to show up as I intended, > but the display in a modern browser works well. > However, kindlegen does an awful job at converting this. > > So I decide to preprocess the HTML that I feed to kindlegen. > I can strip the CSS entirely and use the classes to do > some substitutions, so that I feed kindlegen this: *** and this is "version 'k', the kindlegen version"... > [html] > [head] > [/head] > [body] > [p height="1em"] The first line of my poem[/p] > [p height="0" width="-2em"] The second line of my poem[/p] > [p height="0"] A longer line comes next, and goes on a bit, but still does not rhyme[/p] > [p height="0" width="-2em"] The last line ends with a flourish![/p] > [p height="1em"] The first line of my poem[/p] > [p height="0" width="-2em"] The second line of my poem[/p] > [p height="0"] A longer line comes next, and goes on a bit, but still does not rhyme[/p] > [p height="0" width="-2em"] The last line ends with a flourish![/p] > [/body] > [/html] *** ok, so if i understand you correctly, paul, you take version "a" and (somehow-or-other, i'll call it "a2h"), you turn it into version "h". then you run a script ("h2k") which massages version "h" into version "k". then you zip version "h" into .epub "e" -- "h2e". (how do you create the required auxiliary files?) and of course you then use kindlegen to turn version "k" into .mobi file "m" ("k2m")... (again, how do you create the auxiliary files?) am i correct? *** ok, that's a fairly good workflow... any changes in the "somehow-or-other" "a2h" conversion _could_ impact the later "h2k" one. and a typo-edit to version "a" would require either rerunning the entire cycle, _or_ mirror changes made to "h" and "k", and to the auxiliary files if necessary, followed by rerunning "h2e" and "k2m"... which is a bit bulky. but other than that, it's a fairly solid chain... *** just for the edification of everyone, here is the workflow which i propose. 1. start with .zml text "z". 2a. script converts "z" into "h" .html e-book(s). 2b. script converts "z" into "e" .epub e-book. 2c. script converts "z" into "m" .mobi e-book. 2d. script converts "z" into "p" .pdf e-book. 2e. script converts "z" into "o" offline e-book. 2f. script converts "z" into "w" web-app e-book. 2g. script converts "z" into "g" gutenberg e-text. the click of a button does 2a-2g simultaneously. so a typo-edit to #1 means another button-click. simple. fast. efficient. clean. one input-file and many outputs (all of them disposable as they can be easily regenerated at any time with one click)... -bowerbird p.s. does the following look a bit strange to you? The first line of my poem The second line of my poem A longer line comes next, and goes on a bit but still does not rhyme The last line ends with a flourish! The first line of my poem The second line of my poem A longer line comes next, and goes on a bit but still does not rhyme The last line ends with a flourish! it should. what i have done is duplicated the verse. it's pretty easy to tell that that's what i've done when the text is in a nice clean unobstructed zen format. but it's not quite as obvious when you have all that .html gobbledygook surrounding the text, which is probably why you didn't notice that you seem to have accidentally duplicated the verse in your .html versions.

2 1

a review of some digitization tools -- 019
by Bowerbird＠aol.com 14 Dec '11

14 Dec '11

yesterday, we compared the .html version from my converter with the one up at p.g., and found they were substantially similar. today's mission is to compare the _.mobi_ version that i created with the p.g. version. i once expected to get the same conclusion: that my version was as good as the p.g. one. but that's not the case. my version is better. you can download the p.g. .mobi here: > http://www.gutenberg.org/ebooks/16736 here's my .mobi: > http://zenmagiclove.com/misc/ukagl.mobi > http://zenmagiclove.com/misc/ukagl.mobi.zip > http://zenmagiclove.com/misc/ukagl-folder.zip the p.g. .mobi has problems, mainly caused by that "tag abuse" we discussed yesterday... you can see the glitches in these screenshots: > http://zenmagiclove.com/misc/bac-mobi001.jpg right off the bat, we have that ugly p.g. legalese... the top of a book should be a pleasant coverpage, for a good first impression, not preformatted crap. but there's a bigger problem sitting here as well... this screenshot of the book is on a 23-inch screen. yet the titlepage is not being shown in its entirety. heck, not even the _author_name_ is being shown; we are just being tormented by a teaser "by" line... we have to go to the second screen to get the rest... > http://zenmagiclove.com/misc/bac-mobi002.jpg ...and then we get the bottom half of the titlepage, and the verso, and the dedication, and the first part of the table-of-contents, which appears in a jumble. at the least, we need to have some separator lines... > http://zenmagiclove.com/misc/bac-mobi003.jpg the third screen shows the rest of the t.o.c. jumble... the links _do_ work, so there is a redeeming quality. but obviously, we don't want things looking like that. and then chapter 1 starts right underneath the t.o.c. we really want a new chapter to start on a new screen. but due to the improper tagging of chapter-headers, the viewer-program doesn't know that "chapter 1" is the start of a new chapter, so it simply blunders on... so we have the same problem throughout the book, where a new chapter doesn't start on a fresh screen. > http://zenmagiclove.com/misc/bac-mobi004.jpg but it gets worse. sometimes the chapter-header is located at the _bottom_ of a screen, with the first line of the chapter located on the _next_ screen. not good. > http://zenmagiclove.com/misc/bac-mobi005.jpg even _one_ line from the next chapter looks weird... > http://zenmagiclove.com/misc/bac-mobi006.jpg but what's _even_more_weird_ is when the 2 parts of the chapter-header get split apart by a screen-break... > http://zenmagiclove.com/misc/bac-mobi007.jpg > http://zenmagiclove.com/misc/bac-mobi008.jpg taking a short break from the chapter-header flaws, we see the book's first blockquote on image #9... > http://zenmagiclove.com/misc/bac-mobi009.jpg the kindle doesn't do an indent on the right side, so the indent on the left side probably should be bigger. right now, it's about the same as a first-line indent... > http://zenmagiclove.com/misc/bac-mobi010.jpg image #10 here shows the book's second blockquote, where we again see the left-indent should be bigger. > http://zenmagiclove.com/misc/bac-mobi011.jpg and now we're back to the chapter-header glitches... > http://zenmagiclove.com/misc/bac-mobi012.jpg image #13 shows a stranded line at the screen's top. > http://zenmagiclove.com/misc/bac-mobi013.jpg another header problem... > http://zenmagiclove.com/misc/bac-mobi014.jpg another stranded line, this time at screen-bottom. > http://zenmagiclove.com/misc/bac-mobi015.jpg and another split-in-two chapter-header... > http://zenmagiclove.com/misc/bac-mobi016.jpg > http://zenmagiclove.com/misc/bac-mobi017.jpg and a stranded line -- with one word! -- at the top. > http://zenmagiclove.com/misc/bac-mobi018.jpg and oh gee, yet another split chapter-header... > http://zenmagiclove.com/misc/bac-mobi019.jpg > http://zenmagiclove.com/misc/bac-mobi020.jpg and another isolated one... > http://zenmagiclove.com/misc/bac-mobi021.jpg and one more split one... > http://zenmagiclove.com/misc/bac-mobi022.jpg > http://zenmagiclove.com/misc/bac-mobi023.jpg and, on the last screen of the book, a stranded line. > http://zenmagiclove.com/misc/bac-mobi024.jpg so wow, those are a lot of glitches, unfortunately... my .mobi has occasional glitches too -- crap happens in the kindle reader-app -- but not nearly as many... the kindle app _consistently_ has widow/orphan flaws. but these problems were indeed caused by the fact the converter just didn't know the chapter-headers. most e-book designers start a chapter on a new screen. if the auto-converter had accurately known the headers, it wouldn't have had all those problems with the headers appearing at, or being broken by, the bottom of a screen. but, of course, to begin a new chapter on a new screen, you must be able to _recognize_ that it's a new chapter. and, because of the funky .css and .html tagging used in this "snowflake" edition, the converter failed to see the chapter-headers, thus causing a raft of glitches... my .mobi starts a new chapter on a fresh screen, so it had no glitches of this type. now, some of you will undoubtedly be dismissing these glitches as "cosmetic". and there's something to that argument, i'll grant you. but, on the other hand, shouldn't the books look good? and, furthering that, we certainly do not want the books to have cosmetic flaws because we did something dumb! seriously. this book consists of headers and paragraphs. that's all! and the headers got screwed up. pretty sad... i mean, i guess we can be happy because the paragraphs aren't screwed up too. that would be worse. but _still_... moreover, this header problem is _not_ just skin-deep. you can spot a real red flag if you take a look at the table-of-contents which gets listed under the menu... > http://zenmagiclove.com/misc/bac-mobi025.jpg sure enough, those were the lines _tagged_ as headers -- incorrectly -- so you can't blame the converter-tool. the serious problems in this .mobi are due to the .html. also under that menu, we can summon up the "cover"... > http://zenmagiclove.com/misc/bac-mobi026.jpg oops... that's the generic cover, for all p.g. books, in the kindle viewer-app. kind of embarrassing, eh? what is that, anyway? a picture of some 1999 p.d.a.? and let's view that jumbled in-line t.o.c. once again... > http://zenmagiclove.com/misc/bac-mobi027.jpg again, that's pretty embarrassing. finally, here's what a jump to a chapter looks like... > http://zenmagiclove.com/misc/bac-mobi028.jpg that's caused by putting the anchor in the wrong place. whew! lotta messed up stuff there... which brings us to this restatement of our conclusion: the serious problems in this .mobi are due to the .html. so all in all, when it came to anything that wasn't just a plain old paragraph, this .mobi came out looking bad... that's a pretty sorry situation. and this book was easy! no good reason to screw it up! anyway, if you feel i've been too harsh on this .mobi, feel free to reciprocate by taking it out on _my_ mobi. > http://zenmagiclove.com/misc/ukagl.mobi > http://zenmagiclove.com/misc/ukagl.mobi.zip > http://zenmagiclove.com/misc/ukagl-folder.zip i would love to hear any criticism you might have... -bowerbird

1 0