Hi Jim, Am 13.12.2011 um 14:38 schrieb Jim Adcock:

Keith> 1) ... The space between two paragraphs is just one space. It is not divided between top and bottom! As far as getting the correct space between the different entities of a book it is all in the math.

Agreed that the space between paragraphs is just one space but since not every paragraph is proceeded and/or followed by a a paragraph it became easier for HTML writers to be able to specify top and bottom margins separately, and for HTML browsers to automagically "merge" those margins when a paragraph *was* adjacent to a paragraph. One can "do it all" by just specifying top-margins (and that is what Amazon says you should be doing) it's just somewhat more painful to specify things that way. I do care what Amazon says! It is simple wrong. As for top and bottom margins it is simple wrong to use them! I said it is all in the math. Just because author are to lazy and do not how to do it. It is quite a bit of work to get layout correct. This is not a problem of HTML but all layout done by lay persons using diverse tools and programs. They have not learn how to do it properly nor can they know how to implement proper layout. What most do not understand there is not just one layout/styles for a paragraph. You need multiple layouts. There is no one fits all. Like I have been saying, one has to know how to layout a book in the first place before you can start laying out an ebook with a tool or program.

regards Keith.

2) As you rightly mention most do not understand the semantics of HTML and CSS. Also, in the light the above how do expect them to code the spacing correctly.

This is a chicken-and-the-egg problem. The volunteer transcribers have to have *some* understanding of what they are doing [and why] in order to be able to do it. [And PG, institutionally speaking, is not providing that guidance.] Unfortunately, 99.999% of the HTML documentation and books (and tools) one can find out there have little or nothing to do with the "PG" and ebook tasks at hand, and unfortunately more to do with "fixed layout" HTML "home pages" such as one sees if one for example goes to the home page of amazon.com. Again there are a handful of "ebook" oriented texts out there like Elizabeth Castro, Rufus Deuchler, and Joshua Tallent, but then these tend to swing immediately into issues of Adobe InDesign hacking, which is also not helpful....

3) The inferior implementation of HTML and CCS in the ereaders. This is something which is beyond out control.

My claim is that it is generally really not necessary to exercise all the excesses of HTML nor all the shortcomings in the ereaders in order to create high-quality transcriptions of most of the books that are out there. I think the real problem is the tendency of all of us to try to "prove" we are uber-geeks leads many to try to out-geek all the others when it comes to the complexity of the HTML code being submitted. Now mind you, there *are* legitimately *some* very complicated and technical books being submitted -- such as mathematical texts -- but neither txt70 nor HTML is really suitable for these books in the first place.

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

On Tue, December 13, 2011 3:05 am, Paul Flo Williams wrote:

There is a fourth way: pre-process your (X)HTML to downgrade it to an HTML 3.2 tag soup + Kindle attributes so that the kindlegen step does nothing other than wrapping up your HTML into a MOBI file.

"Tag soup" refers to formatted markup which does not consist of correct HTML syntax or document structure. The expectation for web browsers is that they will not fail when presented with invalid HTML, presenting the content making reasonable heuristic "guesses". "Tag soup" may collectively refer to a large number of common authoring mistakes, such as malformed HTML tags, improperly-nested HTML elements, and unescaped character entities (especially ampersands (&) and less-than signs (<)). See generally, http://en.wikipedia.org/wiki/Tag_soup. I do not know if the MobiPocket parser which is at the heart of the Kindle is a tag soup parser or not, but there is no need to degrade valid (X)HTML to tag soup for the Kindle to use it. That said, it is true that Kindle relies on certain proprietary elements and attributes, which I guess qualifies it generally in the "tag soup" category. But you definitely want to keep the file as well-formed XML. [snip]

So, use xsltproc or Perl plus one of the XML modules to do what kindlegen does, except applying some of your own conversions.

[snip]

Kindlegen does a simple job badly, so bypass the bits that you don't need.

What you are suggesting is essentially to re-write Kindlegen to do things better. This is not a bad idea, but I think it may be a bit more complex than you think (or maybe I'm being unfair in suggesting that you don't grasp the complexity). Using XSLT is a good idea, but it is not a complete solution. It should work fine for inline styles, but doesn't apply CSS styling to each element first. What is needed is a program that will apply CSS to the document tree, and perhaps do other transformations as well, before using XSLT to produce the final output. Clearly perl is an option, although I would favor Java due to its superior performance and the JAXP APIs. Python seem to be the language /du jour/, and it seems to be the one favored by most people involved with e-books, so maybe that would be the right solution. Fast hardware can clearly compensate for Python's poor performance. We all know, more or less, how .mobi files are created, so once you've gotten to the point where the input is Kindle-ready, one could just close the circle and replace Kindlegen all together. Sounds like a fun project.

Paul Flo>No, I'm suggesting doing some pre-processing to the HTML that you give to Kindlegen, converting constructs that it will convert badly into ones that fly straight through from your input to its mobi output. Just to be clear, Marcello is already doing some of this using his epubmaker software in "kindle" mode -- where he moves a bunch of stuff from CSS back to inline coding -- because in theory at least kindlegen handles more stuff correctly inline than it does in CSS. [epubmaker is part of the posting "sausage making" between your submission of HTML to PG and PG posting mobi on their site.] If you install his epubmaker software on your computer, and then, at least if you accidentally on purpose leave off the Path to kindlegen, epubmaker leaves behind kindle versions of the kindle "epub" intermediate files which you can unzip and examine the "HTML" transformations Marcello is already making in an attempt to better accommodate in kindlegen the HTML the volunteers are actually submitting.

.line, .iline { display: block; text-indent: -2em }

Kindle and some other reader devices don't in general like negative indents. Some epub devices allow the end user to adjust (read: reduce) margins from the "end user font choices" system dialog, and then once again negative indents make the devices pretty unhappy. Welcome to the wonders of poetry -- assuming you can discern the original author's intent in the first place.

For those interested in formatting poetry: http://blog.epubbooks.com/898/formatting-poetry-for-small-screens Can't say I agree entirely, but it's always good to get someone else's perspective.

On Wed, December 14, 2011 7:00 am, Paul Flo Williams wrote:

Lee Passey wrote:

[snipped discussion of "tag soup"]

That's an extraordinarily verbose way of agreeing with me :-)

Verbosity is only one of my many faults ;-). The point I was trying to make, perhaps inartfully, is that the term "tag soup" covers a multiple of sins. You might have tag soup where you have valid XML which is not valid HTML because it contains elements or attributes in addition to those allowed by the XHTML DTD. This kind of tag soup can still be parsed by an XML parser, and is mostly harmless. I consider this to be class 3 tag soup. You can also have tag soup which is valid HTML but not valid XML. This is because HTML is derived from SGML, and some things like implicitly closing tags and possibly unnested tags are allowed in SGML but not XML. While valid, this kind of file cannot be parsed by an XML parser, including XSL. To me this is class 2 tag soup. Lastly, you can have tag soup which is simply wrong by any standard: examples would include using block elements inside inline elements, failing to escape ampersands or angle brackets, or using invalid character entities. This is class 1 tag soup. Class 3 tag soup can be fixed by an appropriate XSLT script, but class 2 and class 1 tag soup cannot. Abbyy FineReader produces class 2 tag soup, as does the script at archive.org written by kenh, which means that many of my XML tools cannot work until the file is "fixed." I don't know the tolerance Kindlegen has for these different types of tag soup, but from a practical perspective, I would think you should limit yourself to class 3 tag soup as Kindlegen input. So when you say "downgrade it to an HTML 3.2 tag soup," I simply wanted to caution that it's okay to downgrade to class 3 tag soup, but probably not class 2, and certainly not class 1.

...

What you are suggesting is essentially to re-write Kindlegen to do things better. This is not a bad idea, but I think it may be a bit more complex than you think (or maybe I'm being unfair in suggesting that you don't grasp the complexity).

No, I'm suggesting doing some pre-processing to the HTML that you give to Kindlegen, converting constructs that it will convert badly into ones that fly straight through from your input to its mobi output.

Here's a concrete example:

I'm less interested in concrete examples than I am in abstract examples. It's easy to write a script that converts <div class="iline">...</div> to <div>  ...</div>. It's somewhat more difficult to write a script that converts <div class="iline">...</div> to <div style="margin-left:1em">...</div>, where the style element is derived from the <style> definition, (I can't think of a way to do it with XSLT, which means some other script language must be used) but relatively straight-forward. But what I really want is a script/program that converts <div style="margin-left:[some unpredictable value]"> to <div class="semanitic">[a number of non breaking spaces calculated based on the unpredictable value]...</div>. In other words, a generic transformation rather than a specific transformation. If Kindlegen does /most/ of the transformations correctly, then it makes the most sense to provide a program that performs only those transformations that Kindlegen handles badly, if at all. But if the number of identified transformations that Kindlegen handles badly grows signficantly, and if one has designed a generic transformation engine, then it may make the most sense to simply handle /all/ the required transformations in this new, open-souce transformation engine, and leave Kindlegen only the job of packaging the (X)HTML with additions into the .mobi format. Because we understand the .mobi format, it should be a small step to add the packaging function to the transformation engine and replace Kindlegen entirely. I'm not saying that that's the way it /should/ be done, only that it's an option that should not be dismissed. [snip]

I've decided that a flexible way of marking this up in HTML is this:

<html> <head> <style> .poem { margin-left: 2em } .line, .iline { display: block; text-indent: -2em } .line { } .iline { margin-left: 1em } </style> <body> <div class="poem"> <p class="verse"><span class="line">The first line of my poem</span>

So what will happen if the user agent you're using indents paragraphs 50% of the display? And it's pretty clear to me that a verse is not a paragraph. So why not use <div> for verses instead? The default display mode for <div> is block, and the default display mode for <span> is inline. But in your example you have changed the display mode for <span class="line"> to block. Why not just use <div> in the first place, as its default presentation is exactly what you wanted? My version would have been: <div class="poem"> <div class="verse"> <div class="line">The first line of my poem</div> <div class="iline">  The second line of my poem</div> <div class="line">A longer line comes next, and goes on a bit, but still does not rhyme</div> ... You may note that I have used non-breaking spaces in the "master" version just like in the Kindle version. This is consistent with my view that "master" versions should look acceptable even when the User Agent can't handle CSS.

You'll note that this needs CSS to show up as I intended, but the display in a modern browser works well. However, kindlegen does an awful job at converting this.

So I decide to preprocess the HTML that I feed to kindlegen. I can strip the CSS entirely and use the classes to do some substitutions, so that I feed kindlegen this:

<html> <head> </head> <body> <p height="1em">The first line of my poem</p>

A line is not a paragraph; use <div> instead. You've also lost the association of lines into verses, and verses into poems. Probably not a problem if this markup is /derived/ from a "master" version, but there's really no reason not to preserve the structure moving forward. (It's interesting to note that Kindlegen also preserves styles it cannot convert, and which the Kindle will ignore). I believe Kindle supports the <blockquote> element which provides right/left margin indentation, just like your <div class="poem"> does. Maybe for the Kindle you would want to enclose your entire poem as a <blockquote class="poem">, which may provide some of the display "goodness" you are seeking. [snip]

I don't need to understand the structure of mobi files or throw away kindlegen wholesale.

Absolutely. My fundamental rule is don't do anything that doesn't need to be done (even if it would be fun to do). But I can definitely envision that over time a generic Kindlegen preprocessor might elbow out Kindlegen itself.

On 12/15/2011 02:19 AM, Paul Flo Williams wrote:

Two years ago, I think you'd be right. With KF8 in the wings, I can't say I'm willing to do any more than provide crutches for the vocabulary I've chosen to use for markup.

Relatedly, here's some early info on that: http://www.the-digital-reader.com/2011/12/13/kindle-format-8-demo-now-availa... David