hmonroe>I would appreciate some guidance on what adjustments would need to be made to the HTML produced by Guiguts to make it more amenable to conversion to epub format by epubmaker or otherwise. OK, please see my previous email. I don't really remember how to run guiguts but I installed it new and then I imported my last txt72 that I submitted to PG and autoginned HTML using guiguts and I found most of the classes of problems I described earlier. Putting my previous email even simpler: If you think about it a small say 3" wide machine has the visual space to do exactly ONE thing at a time, but via vertical scolling or page flips it, like all modern display devices it can essentially do an infinite number of things vertically. So if you want to make it "work" you have to stack visual elements vertically, not horizontally. So, on a small display device, and at any moment in time, speaking in terms of horizontal real-estate you only have space to do ONE of the following things on the display: a) Display a picture. b) Display a page number. c) Display left margin whitespace. d) Display right margin whitespace. e) Display a note [a "footnote" or what have you] OR g) Display the actual body text of the book that the customer might actually read. Now I'm not saying that a) though e) might not be interesting things to display on the customer's device. I'm just saying that if you want to display more than one of these things "at any moment of time" you have stack them vertically, not horizontally, or it just ain't going to work. Because the real-estate simply doesn't exist on the customer's device's display. Specific comments looking at the ginned HTML code: * Specifying left and right body margins doesn't work. * Specifying P top and bottom margins of .75em doesn't work. * position absolute doesn't work. * things of 0.5em don't work. * sidenotes don’t work. * smallcaps doesn't work on any machine I know of -- but its typically relatively harmless when they don't work. * float left doesn't work, and float right doesn't work. * margin-rights really don't work. * background colors typically don't work but when they don't work that's typically relatively harmless. * reliance on spans to move things right is probably not a good idea. Etc. I realize what I say will probably make you unhappy and you will probably go away mad without ever changing anything but, the reality remains: What you are trying to do is not going to work on small machines for the simple reason that, well, they are small. Which leaves PG in an interesting position.

Carlo>Jim, are you happy with the epubs and mobi generated through RST, any of these PG numbers: Well, if you ask me "any" of these numbers I will just look at one of each if you don't mind. RST 34605 is a junior reader, which usually doesn't exercise the full set of problems one sees in an adult book, but, let me continue. The HTML version has wide margins, which will tend to make PG customers unhappy when they access this file via a web browser from a small device, like an Android tablet. The HTML version doesn't show page numbers, but the EPUB version does -- so I don't understand how you manage to do that? The EPUB version seems to somehow have the margins trimmed. Did Marcello's software trim the margins for you? Now "magically" there are page numbers in the small right hand margin which didn't exist in the HTML version. I don't understand how this is possible? If PG requires that submissions for EPUB and MOBI version be via HTML, then how is this possible? I'm sure I don't know. The EPUB version is something I could live with, IE I think it is "readable" on most EPUB devices, even though personally I would prefer not ragged right, not having a small right hand margin, and not having page numbers there -- at least not in a junior reader. But still the EPUB is successful enough that I don't consider it embarrassing. I just don't know how it can be "magically" -- if PG requires EPUB and MOBI to be via HTML submission -- how this file ends up with page numbers but the HTML version does not? The MOBI version needs to be looked at in two different ways 1) under a "traditional" Kindle device and 2) in a "Kindle Fire" new generation device. Under Kindle Fire: It seems like you have a way to not use the "Blue PDA" as the cover image. I don't see how you do this, because I don't see how PG allows specification of the cover image to be used with an HTML submission? In any case, on many small devices a cover image like you have which includes title and author is much more useful than the PG default "Blue PDA." Or is it that you don't include a cover at all, and that Amazon is automagically providing me with one? But when I submit an HTML PG gives my EPUB a "Blue PDA" -- whether I want it or not! Title Page images are margin left, in HTML they are centered. Is it possible that image location isn't being specified and is just picking up the web browser defaults? Hitting "Back Link" on a chapter title takes one to a scrambled version of the TOC, which is presumably related to MOBI file lazy evaluation. Is your TOC link target placed *after* the format specification for the TOC? MOBI might work better if the TOC link target is placed *prior* to the TOC format specification, if possible. You are doing "no margins-top, first line indent" on paragraphs, I find it easier one small devices to read "1em margin-top, first line no indent" on paragraphs, but this is mainly a personal preference item. "CHAPTER V: JANET HEARS FROM BETTY" title formatting is not very successful. It would probably format better if you specified this as: CHAPTER V JANET HEARS FROM BETTY IE Chapters with titles in two lines, each line centered. Many of the chapter titles suffer appearance from the choice of implementation. "But when the old woman got home in the dark," etc. This poem formats poorly due to poem line break problems. Not to imply that I personally know a good solution to "HTML" based poetry formatting. But it does seem like you could easily choose to indent poems much less than this, leading to poem linebreak problems less frequently. "Bye little Betty, ..." seems to be indented *way* too much. "THE END" and subsequent PG "advertisements" are formatted poorly. It would be nice if the reader could savor finishing the book for a few moments before being immediately hit by the "pgegalize." Put in a page break before the "pgegalize" ? ----- "Traditional Kindle" : Seems to have the same exact issues as "Kindle Fire" -- which is encouraging, given that presumably PG is still compiling "HTML" to MOBI7 using the previous version of Kindlegen, and not the "KF8" using the current generation of Kindlegen. Picking TEI 34520 "at random" -- just to have a book of a similar vintage to RST 34605 -- I see: The HTML version (reading HTML on a small device): has wide margins, which will tend to make PG customers unhappy when they access this file via a web browser from a small device, like an Android tablet. Page numbers are rendered in the left margin, which probably explains why the margins were set so wide. Page numbers 2 and 3 are rendered literally exactly on top of each other which presumably is a TEI->HTML implementation bug. "PGegalize" is formatted no-wrap mode so it runs ugly off the right hand of the screen. Not clear why the "PGegalize" is placed at the front of some (RST) books and at the end of others (TEI) ? Excess margins in the TOC causes more entries than necessary to wrap unattractively. And in general the TOC is hard to read since it wraps so much (and many times a single entry over multiple lines) which again is due to excessive margins. Images are not set to scale correctly to fit the (small) device page size so that on a small device one only sees a tiny upper left corner nibble of the images. <p> formatting is chosen to be 1em spacing between paragraphs, no paragraph indent, which personally I prefer because I think on small devices it is somewhat easier to read. [pg 014] doesn't fit the margin so it renders: [pg 0 14] And where the "]" somewhat overlaps the first letter of the body text. Same for all margin page numbers, basically. Topic titles are formatted "Margin Left" rather than centered, as I would expect of a traditional book. If this were a computer manual instead of a biblical manual, then I would expect to see topic titles formatted left ala K&R troff style. "Note.-God did this by backing His promises with an oath based upon Himself. Heb. 6:13, 14. By this He pledged and placed at stake His name, or character, for the fulfillment of His word." -- which actually looks even worse on the small web browser than I have represented above since the formatting has been chosen to try to "justify" making the gaps between the above words go all over the place. But the point being when you start using "big" margins on small device the results are necessarily tragic. In this case were the left and right margins needlessly set in ems or something, instead of using % ? O Word of God incarnate, O Wisdom from on high, O Truth unchanged, unchanging, O Light of our dark sky! We praise Thee for the radiance That from the hallowed page, A lamp to guide our footsteps, Shines on from age to age. Shows a "textbook example" of the difficulty of HTML "poem" formatting especially on small devices when large margins and indents are chosen. [The actual poem, should you fail to recognize what it should have been, is:] O Word of God incarnate, O Wisdom from on high, O Truth unchanged, unchanging, O Light of our dark sky! We praise Thee for the radiance That from the hallowed page, A lamp to guide our footsteps, Shines on from age to age. Index seems very long for a small device. Is this really what was in the original book? End "pgegalize" : While generally readable, the http links have been formatted n-wrap which means one cannot read the whole http link making them generally unusable. EPUB: The ngx or whatever its called seems to be way overpopulated. Page numbers now have magically jumped from the left margin to the right margin and what used to read "[PG 014]" now just reads a small grey "14" -- how do they do that? How does one submit HTML which looks so different in an HTML browser than in EPUB, I wish I knew how they do that? Because I was told if one wants to target EPUB or MOBI the ONLY option is to submit an HTML file. Hm, except now I see a line that says: [pg 002][pg 013] ... 7 So now I am confused: Is this page 2, or page 13, or page 7 ??? Margins seem large for the small device Images do not scale to fit device Some images are compressed using a scheme which does not display correctly on the device. Some page numbers [pg 016] are now showing up in the body text, interrupting the reading, which I personally never think represents a reasonable tradeoff. Same kinds of severe poetry formatting issues makes the poems unreadable. Same kinds of severe left and right margin "indenting" on blockquotes makes them unreadable. Etc. EPUB showing basically the same kinds of extreme problems present when trying to read the HTML on a small device. Credits: The credits are formatted so that they show up farther to the right than the center of the display and therefore of course are badly clipped off by the right edge of the physical display. Kindle: No image versions of the files are provided at all? Did a gen tool silently fail? Or no one bothers to fix gen failures? Kindle Fire: Unfortunately these files have been "successfully" provided with the "Blue PDA" default PG cover image, meaning that one has 20 identical appearing "Blue PGA" PG books on one's Kindle, and one does not know which book is which. Author name is apparently not correctly being provided by TEI because it doesn't show up on Kindles, where sorting on author name is an important choice. Content header runs up against previous formatting. In general all the lead-in pages formatting is incorrectly run-together. And left-aligned instead of centered, but left-aligned against an excessive left margin width. TOC items bulleted, which is uncommon in TOCs, I believe, and looks strange to my eyes. The missing images are replace with the left-aligned words: Illustration. The Word of God.... -- except in reality there is no vertical whitespace being used to offset these words. For accessibility reasons I would think if one takes out the images and replaces them with something that something ought to be a description of that which was left out? No vertical space after subject headings. Traditionally one would expect to see some spacing between a subject heading and the following paragraph. Ideally the first paragraph after a subject heading shouldn't have an indentation. Poems are now miraculously left-aligned without an excessive margin. But the even-numbered poem lines *ought* to be indented relative to the odd-numbered poem lines and how somehow they are not. In general something seems to have swallowed ALL the vertical "whitespace" formatting that *ought* to be there. Everything simply "runs together" vertically. End "pgegalize" is now readable, but something has turned what used to be active http links into inactive text. Credits show up "OK" on Kindle, but somewhat excessively indented (but not as bad as on EPUB) Kindle "Traditional:" TOC Entries: the bullets and the TOC entries show up on different lines in the display looking pretty silly. "How Readest Thou?" This section shows up with some kind of "pre" formatting or something which doesn't wrap correctly, and lines break unattractively. In generally beyond this the Kindle "Traditional" showing the same kinds of problems as Kindle Fire. In general, I would say the this particular TEI is no more successful in practice on a Kindle than simply sending a txt72-unwrapped version of the plain text to a Kindle. If then. Sorry -- Jim

Carlo>Jim, none of the submissions that I listed are mine, they have been submitted as PG-RST or PG-TEI master files, from which all the formats have been created by Marcello's software without further manual intervention. Sorry, I didn't mean to imply that you were the actual submitter of the books in question.

In particular, for RST two different HTML are created in the process, one for browsers and one for epub. So it is unfair to view HTML on a small device.

Actually, what I think is unfair is that PG has told submitters, including me, repeatedly, that the ONLY way that PG allows one to submit EPUB or MOBI is by sending in a generic HTML, but then PG "secretly behind the scenes" allows other people other paths to create customized EPUB and MOBI rather than having to rely on "One Generic HTML For All". Re: the general unfairness of reading HTML on a small device: Have you read the numbers of what people out there in the "real world" are actually doing? They have largely stopped buying desktops. They have largely stopped buying laptops. What they ARE buying are tablets. Those tablets now all come with decent HTML browsers. Many of these customers have no idea what an EPUB or MOBI file is. So they end up reading the books in HTML format. Which looks crappy on their small devices since the authors of that HTML really didn't know what they were doing, or didn't care. Now, is it somehow "unfair" that these customers read that which PG provides on that which the customer actually has in hand? Or is it "unfair" that PG takes books written by good authors, transcribed them incorrectly into bad HTML, and then presents them in a totally scrambled manner on the end user's device? Seems like the people being treated "unfairly" are the original authors, and the PG customers, NOT "PG" !!!

PG doesn't redo routinely all the file generation.

IF PG doesn't routinely redo all file generation, then PG has basically been lying to us submitters about their rationale for restricting our contributions to being "HTML ONLY!" Even IF PG insisted on doing their own gen of EPUB and MOBI, PG could still allow us submitters to also submit EPUB-HTML and MOBI-HTML -- which you point out PG is ALREADY allowing in the "special case" of RST! So why one set of rules for some people and other set of rules for all us others ???

Just because you were not aware of RST being used on PG does not meean that it has been "hidden" from you.

You haven't addressed the issue. WHY is "PG" allowing "RST" and "Tei" to bypass the requirement to submit one and only one version of HTML while telling the rest of us submitters that we are not allowed to "submit" alternate versions for "EPUB" nor "MOBI" ? There are no "tool chain" issues here. A submitter could just say to PG "This HTML is for big browsers, this HTML is for EPUB, and this HTML is for MOBI." Or PG could allow us to submit one HTML with conditional compilation statements, even using the CPP for * sake, and then generate *useful* versions of the "big browser HTML", "EPUB HTML", and "MOBI HTML" from that one submitted HTML that includes the conditional compilation statements.

Lee>Perhaps one point is not clear. TEI and RST are both "single source" options. That is, a volunteer submits only *one* file, and all others are generated from it. OK, how about we try this one more time: Let me submit *one* EPUB and PG can generate all the other formats from it -- especially the txt70 requirement -- now *that* would be one giant step forward!

You can submit either *one* HTML or *one* TEI or *one* RST file. In no case has anybody been posting multiple HTML files.

Well we are getting down to Clinton-esc Parsing, but in that case allow me to post *one* HTML which includes CPP conditional processing statements (or the equivalent) such that it is "allowed" to magically split into three Hydra Heads and generate separate targeted "Big HTML", EPUB, and MOBI versions just like RST is allowed to do.

Lee>If you were to follow these rules you would have exactly what you are asking for: a single HTML file that would satisfy all three purposes. If *I* were to follow these rules I would produce the same HTML input to PG as I am already doing 99% of the time. I will admit that on the last day or two before shipping I am willing to compromise and "tweak" locally what goes out to fix any small remaining problems without taking the risk of wholesale damage to what I have already SR'ed two or three times before. The problem is with the great mess of other posters, and with a PG/DP mentality which says every last comma is important, but producing books that real customers can actually read on their real devices is not important. One misplaced comma is important, but literally 1000 formatting errors in one file doesn't count, because "We" don't care. With this caveat: I do not use external CSS files because PG doesn't accept them. And I am happy to put a couple lines into my CSS in order that the same HTML will render "correctly", and more or less consistently, on HTML, EPUB, MOBI, and KF8 devices. Agreed that I personally would probably rather see external CSS files.

Marcello is correct. It was decided in the early discussions of RST that additional custom files, e.g. a custom HTML file or a custom text file, would not be allowed. If custom files were desired/required, then the submission should be a normal text+HTML submission. The first discussion/submission of TEI files predates my time as WWer, but the same principle applies. Al

-----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Lee Passey Sent: Saturday, January 21, 2012 4:52 PM To: gutvol-d@lists.pglaf.org Subject: Re: [gutvol-d] Producing epub ready HTML

On 1/19/2012 4:43 PM, Marcello Perathoner wrote:

...

You can submit either *one* HTML or *one* TEI or *one* RST file. In no case has anybody been posting multiple HTML files.

Is this an exclusive 'or' or an inclusive 'or'?

If I submit an HTML file am I precluded from posting a TEI or an RST

file? Or may I submit one of each?

Mr. Haines or Mr. Newby, would you care to confirm or deny this restriction? _______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

On Sunday, January 22, 2012, Greg Newby <gbnewby@pglaf.org> wrote:

Yes, confirmed.

Having numerous formats derived from a single master is a long-time goal. We've had some success with RST and TEI, and I've encouraged new projects to consider RST. There are still some limitations, though...

On the many, many words on gutvol-d recently about poor results with auto-conversion from HTML to other formats (epub and mobi, among others): this is often due to choices that producers make about using HTML to impact layout, rather than just structure. Enough said. -- Greg

Au contraire. I think there are plenty of producers who don't understand the distinction. Why aren't WWers send back projects that include destructive layout tagging, or don't include important structural tagging? I can think of any number of reasons for rejection that are less disruptive to the reader's satisfaction.

On the many, many words on gutvol-d recently about poor results with auto-conversion from HTML to other formats (epub and mobi, among others): this is often due to choices that producers make about using HTML to impact layout, rather than just structure. Enough said.

Well, I think I work extremely hard to avoid using HTML to "impact layout", AND I work very hard to create HTML that will render "identically" in EPUB and MOBI, AND I check my work using a local copy of epubmaker, AND I check my work on multiple copies of physical EPUB and MOBI devices, YET still I find that I end being disappointed in the EPUB and MOBI that ends up being posted on PG from my well-intentioned efforts. Funny how pushing on one end of a rope doesn't lead to satisfying control over the other end of the rope. Again, if you allow posting in RST or TEI, why not allow posting in EPUB? EPUB comes much closer to being "a real complete book" that HTML ever will. And it's much easier converting EPUB back to HTML than trying to convert HTML to EPUB. And EPUB goes straight into Kindlegen to make MOBI and now KF8, with KF8 being in practice very similar to EPUB.

Evidence that HTML as a master format cannot be satisfying, or at least not universally so. Seems we already knew that, but it's still sad to hear that your toils didn't have the intended outcome.

I have happy results when I process that HTML on my own machines using my own tool chains, in part because I can, and do, easily check my output from those tool chains. In fact it has gotten to the point where if I want to read some book from PG by some other submitter, I find that I can *easily* create a more satisfying read experience locally: 1) Download the PG HTML file(s) 2) Edit the CSS to fix the half dozen formatting "errors" that almost all submissions from DP fall prey to. 3) Compile locally using the latest version of Kindlegen, DIRECTLY submitting the HTML files -- no epubmaker required! 4) Read and enjoy a book that *isn't* scrambled. If I can do this with 5 or 10 minutes effort, then PG could too. Its just a question of whether PG keeps defending the status quo, even though its clearly not working [1], or when or whether PG is willing to acknowledge "you know, the approach we have been trying to take simply isn't working, we need to rethink our approach." [1] How do "we" know that the current approach is not working? Simple 1) Open a PG book in a web browser. 2) Open the same book on an epub device 3) open the same book on a kindle device. Are the results similar, and similarly satisfying? Nope. Are others able to accomplish this same task using PG books as feedstock. Yes. Conclusion: PG is doing something wrong.

On 01/23/2012 05:51 PM, Lee Passey wrote:

I think this is a strong demonstration of why it is important for technical people take more classes in technical writing.

No it is not. It is a textbook example of why technical people should take classes in English. You are the typical nerd that when asked: "Do you want tea or coffee?", answers with: "Yes". To save time I'll just rephrase what I said for nerds who don't quite understand English: You can post exactly one element of HTMLTXT ⋃ TEI ⋃ RST where: HTMLTXT = { (h, t): h ∈ HTML, t ∈ TXT }, TXT = the set of all plain text files that encode the book, HTML = the set of all valid HTML files that encode the book, TEI = the set of all valid TEI files that encode the book, RST = the set of all valid RST files that encode the book. -- Marcello Perathoner webmaster@gutenberg.org

I've seen Marcello's response to this, and to put it into plain English, a submission can be one, and only one, of the following: - one (and only one) TEI file, plus illustrations. No other files are allowed. - one (and only one) RST file, plus illustrations. No other files are allowed. - one (X)HTML file, plus illustrations, plus from one to three text files (UTF8, Latin1, ASCII), depending on the submission's requirements. (This kind of HTML+text submission *may* also include such binary formats as doc, rtf, and pdf, but since they have to be handled manually, and PDF files are difficult, if not impossible, to correct, they're not encouraged.) - Latex - these are rare, and are usually only used for source material having considerable math content, e.g. "Calculus Made Simple" (PG#33283) Al

-----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Lee Passey Sent: Monday, January 23, 2012 8:52 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] Producing epub ready HTML

...

...

...

On 1/19/2012 4:43 PM, Marcello Perathoner wrote:

...

You can submit either *one* HTML or *one* TEI or *one* RST file. In no case has anybody been posting multiple HTML files.

I wrote:

...

...

...

Is this an exclusive 'or' or an inclusive 'or'? If I submit an HTML file am I precluded from posting a TEI or an RST file? Or may I submit one of each?

[snip]

...

On Sat, Jan 21, 2012 at 07:28:30PM -0800, Al Haines wrote:

...

Marcello is correct.

[snip]

On Sun, January 22, 2012 1:21 pm, Greg Newby wrote:

...

Yes, confirmed.

I think this is a strong demonstration of why it is important for technical people take more classes in technical writing.

Mr. Perathoner started this thread making an ambiguous statement. The ambiguity was compounded when he stated "In no case has anybody been posting multiple HTML files," leading one to conclude that the issue here relates to multiple postings of a single markup format.

I then asked for clarification of that statement, to remove the ambiguity. Mr. Haines and Mr. Newby each responded saying, in effect, "yes, Marcello's original ambiguity is the policy of Project Gutenberg."

I am left to conclude that either the Project Gutenberg policy in this regard is still in a state of flux and the ambiguity is intentional, that Mr. Haines or Mr. Newby did not feel comfortable enough with their writing skills to actually respond to the question, or that neither of them gave anything more that cursory attention to the question actually posed.

Given the usual practice on this list of paying little, or no, attention to what other people have to say, I suspect that the last alternative is the correct one.

In any case, the ambiguity remains.

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

On 01/23/2012 10:48 PM, Lee Passey wrote:

On Mon, January 23, 2012 12:43 pm, Al Haines wrote:

...

I've seen Marcello's response to this, and to put it into plain English, a submission can be one, and only one, of the following:

See, I never would have gotten /this/ out of Mr. Perathoner statement.

...

- one (and only one) TEI file, plus illustrations. No other files are allowed.

This seems bizarre to me. What is the rationale for not allowing, let alone requiring, an impoverished text file when the other submission is TEI?

That you can generate the plain txt file out of the TEI. -- Marcello Perathoner webmaster@gutenberg.org

On 01/23/2012 11:21 PM, Lee Passey wrote:

On Mon, January 23, 2012 3:01 pm, Marcello Perathoner wrote:

...

On 01/23/2012 10:48 PM, Lee Passey wrote

...

...

...

- one (and only one) TEI file, plus illustrations. No other files are allowed.

This seems bizarre to me. What is the rationale for not allowing, let alone requiring, an impoverished text file when the other submission is TEI?

That you can generate the plain txt file out of the TEI.

You can generate the impoverished text just as easily out of HTML. So why the distinction?

Please show me how. The generated text must be ready to post, eg. word wrap, pg header, pg footer, lines between chapters, etc. all must be there and adhere to pg standard. There must be no post-generation edits required. -- Marcello Perathoner webmaster@gutenberg.org

On 01/24/2012 12:25 AM, Lee Passey wrote:

On Mon, January 23, 2012 3:48 pm, Marcello Perathoner wrote:

...

Please show me how.

You do it the same way you do TEI, except you map the tags differently. We know, for example, that XHTML can easily be converted to TEI, so from there it seems the process could be the same.

What I did was to load the HTML into an in-memory DOM (best to use a tag-soup parser, because you can't guarantee well-formedness of the input file [unless you've pre-processed with Tidy]). Then walk the tree, spitting out appropriate text as you go. Some tags get translated output both before and after their children (e.g.<i> and<b>) other tags only need something before or after. If you want to be really careful about line lengths, buffer words at a time, and then decide whether a new-line needs to be added before the word.

Ok. Now put that into code, runnable on an ubuntu box, and give it to the WWers to evaluate.

...

The generated text must be ready to post, eg. word wrap, pg header, pg footer, lines between chapters, etc. all must be there and adhere to pg standard. There must be no post-generation edits required.

Do you want me to send you the code (based on the 2005? Tidy code base)? It doesn't spit out the PG garbage text, but that could easily be added. I can't say that it adheres to the PG standard, because I am still unaware that there /is/ any PG standard, but if you were to tell me explicitly what /you/ think the standard is I could tell you whether it satisfies it.

Take a hundred random samples from the archive and pipe the HTML file thru your device and see if something very close to the posted txt file comes out. (You may safely ignore where the lines break, but not the number of empty lines between blocks.) -- Marcello Perathoner webmaster@gutenberg.org

[Same message as sent earlier, but without attachments just in case] On 1/24/2012 1:46 AM, Marcello Perathoner wrote:

On 01/24/2012 12:25 AM, Lee Passey wrote:

...

On Mon, January 23, 2012 3:48 pm, Marcello Perathoner wrote:

...

Please show me how.

You do it the same way you do TEI, except you map the tags differently

Ok. Now put that into code, runnable on an ubuntu box,

I see. You don't just want me to show you how, you actually want me to do all the work. I'm not sure I want to go to that much effort simply to demonstrate to you that it's possible. What I will do for you is send you the C++ code that you can compile and install yourself. Attached are two zip files; one contains an early version of a C++ version of Tidy (circa 2002) the other additional files used to create the html2txt executable. html2txt.zip contains a file named "filelist.txt" which lists the files from each archive necessary to build the program. I neglected at that time to add the "readme.txt" to the zip file, so I am attaching it here separately. I don't know if the gutvol-d list software strips off attachments or not (if it doesn't, it should). If anyone else would like this code and it doesn't come through, contact me directly. The theory of operation of converting HTML to text is really quite simple, and there's plenty of ways to skin this particular cat. If I were doing this again I would probably use Java as it has all the DOM parsing and manipulating functions necessary, if not built in then readily available. With Java it could easily be done in a couple of hundred lines of code and would "run everywhere." The method is so simple and straight-forward that probably even BowerBird could do it in Python, and I'm sure it's doable as an XSL script as well.

and give it to the WWers to evaluate.

LOL! I'm not convinced that any of the white-washers could even spell ubuntu, let alone compile, install and use a Linux program. For them I've attached a MSWindows executable built from the attached code.

Take a hundred random samples from the archive and pipe the HTML file thru your device and see if something very close to the posted txt file comes out. (You may safely ignore where the lines break, but not the number of empty lines between blocks.)

This is an exercise left to the reader. Of course, the real test is not equivalency, but whether the output is something the white-washers would accept; no one can judge whether this requirement is met except the white-washers themselves.

On 1/24/2012 11:03 AM, Marcello Perathoner wrote:

On 01/24/2012 06:25 PM, Lee Passey wrote:

...

I see. You don't just want me to show you how, you actually want me to do all the work. I'm not sure I want to go to that much effort simply to demonstrate to you that it's possible.

I am sure I will *not* do the work for you. You want HTML as master, you write the code.

I did, and gave it to you. All you have to do is compile and install it. Or do you want me to be your system administrator as well?

...

LOL! I'm not convinced that any of the white-washers could even spell ubuntu, let alone compile, install and use a Linux program. For them I've attached a MSWindows executable built from the attached code.

pglaf.org is an ubuntu box.

A classic example of a non-sequitor.

On Tue, January 24, 2012 3:20 pm, Karen Lofstrom wrote:

In addition, reworking the "copyright notice" so that it's a Creative Commons licence, which now has a track record and legal support.

The problem with this idea is that PG's "copyright notice" isn't. It's really a trademark claim. There is (probably) nothing in any PG text that is copyrightable other than the copyright notice itself. The PG notice says, in essence, "the name 'Project Gutenberg' is our trademark. It is intended to assure a certain level of quality (or lack thereof) in documents produced for Project Gutenberg. We can't keep you from modifying the text you get from us, but if you do you can't use our trademark on it because we can't ensure that it satisfies our requirements." This kind of trademark notice is qualitatively different from the copyright notices produced by Creative Commons. Clearly, the PG trademark notice needs to be simplified, probably moved to the end of the document, and in HTML versions included in a classified <div> (e.g. <div class="pgclaims">) so that it can easily be hidden by CSS. But it cannot be replaced by any kind of license from Creative Commons, which all rely on a valid copyright.

On Mon, Jan 30, 2012 at 10:02 AM, Lee Passey <lee@novomail.net> wrote:

So, Ms. Lofstrom, what kind of metadata did you have in mind, and do you have any proposals for storing and accessing it?

NO, because I'm not a computer-savvy librarian. However, my own experience is that that PG (and other online collections, for that matter) fail WOEFULLY at presenting their collections in a manner that is both accessible to the casual browser, and high-powered and accurate enough for researchers looking for specific information. It's not just me; I've certainly read enough online complaints by librarians and academics trying to navigate the e-maze. MARC records would be a start. However, I understand that the software is fussy about the wording of queries, which is a big drawback. Some changes needed there. Also, something like the Amazon and Netflix rating systems (if you liked X, you might like Y) would also be useful. -- Karen Lofstrom

On Mon, Jan 30, 2012 at 10:07 PM, David Starner <prosfilaes@gmail.com> wrote:

Where's the data coming from? Amazon has a huge amount of data in terms of user JoeBob bought these books, claimed to own those books, and put that third group of books on his Wishlist. We don't have that.

The same way Netflix built up its recommendations. Ask people to rate. (And Amazon goes by user ratings as well as purchases.) Sure, it would be slow, and the first iterations would be riddled with errors. But it's not as if it were a new technology at this point. -- Karen Lofstrom

why should PG want to offer ratings or suggestion.

How does a real-world "customer" of PG who doesn't already know a title and an author and who already knows that that book is at PG, find a book they want to read? Currently PG would be ahead even to offer a "I'm feeling lucky" button that simply throws a book at the "customer" at random. Magiccatalog is an example of that kind of "I'm feeling lucky" kind of browsing where "at random" the customer says "H'm that one might look interesting" and clicks on one button to download to their reader device.

"PG is not selling anything."

PG is "selling" their contribution to the world. The price may be free, but that doesn't mean you get anywhere if you are not willing to "sell it." If PG doesn't "advertise" their efforts then would-be volunteers never realize that there is an opportunity to make a contribution to the effort. Others are quite happy to take PG's "advertisements" off the books and substitute their own advertisements so that the end users don't know where the books are actually coming from. It's not *just* that these "competitors" want to use the space to advertise themselves. It is also more profitable to them if PG's efforts fail.

Al>It'll be interesting to see who, if anyone, steps up to this project. Again, I tried "stepping up to the bat" some time ago. As a test of the receptiveness of PG to people actually trying to rework old and crusty versions, I redid 76 as 32325 using a slightly different edition to avoid stepping on anyone's toes and to test PG's response. PG posted it, but continues to "advertise" the old version via download count such that almost no real customer ever finds the new version. Go try and find it without looking at the number above, and see what I mean. Word diff comparisons show that the old version, 76, is in error in about 1500+ places, by today's standards, at least. Where by "in error" I mean 1500 places 76 clearly differs from the text found in the original book. Again, its not clear to me how anyone would, in practice, be allowed to rework an old crusty version such as 76, even if they wanted to. I believe, in practice, the WWers would reject the effort after the volunteer has put in the 40+ hours work.

Carlo>PG 76 (Huckleberry Finn) has been reposted on September 21, 2011. It has an illustrated HTML version. In this case the newest version is advertized first. If you actually take a look inside what is posted there you will find that which is "new" is still old again.

Starting at the most basic level, is there any good reason not to use utf-8 as the basic encoding standard for everything including plain-text? Transforming to and from there into any other encoding is I think a pretty well-trod path by now.

Not to mention the source for all the PG software?

Speaking of sources, I remember writing a utility for DP to rename and upload the project source materials, including images, for DP projects, properly named and assembled to PG's wishes. Having not heard from anyone lately, either it's working perfectly (which is unlikely), or PG isn't staying current on source materials. One of the consequences for DP is that it's not possible for them to recreate plain-text and html files from single sources because everything between the last of the Rounds and what is given to PG is discarded (or never leaves the hands of the post-processor in the first place.) The workflow is pretty broken, isn't it?

Gutcheck/Jeebies/Gutspell - their sources are at http://gutcheck.sourceforge.net/etc.html PG's posting software - no, but since the only person who's modified it since I've been WWing (four years) is Jim Tinsley, its author, version control is probably overkill. PG's server-based stuff - Greg/Marcello will have to answer that. Al -----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of don kretz Sent: Tuesday, January 24, 2012 4:48 PM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] Producing epub ready HTML Not to mention the source for all the PG software?

Don>Starting at the most basic level, is there any good reason not to use utf-8 as the basic encoding standard for everything including plain-text? BOM or no BOM? Unix or Windows style line breaks? Line breaks meaning paragraph separations or line breaks meaning, well, whatever it is that PG means by line breaks? "uft-8" meaning that "we" use the interpretation of the code points as defined by Unicode, or meaning "we" invent our own meanings for those code points?

...

Line breaks meaning paragraph separations or line breaks meaning, well, whatever it is that PG means by line breaks?

...

"uft-8" meaning that "we" use the interpretation of the code points as defined by Unicode, or meaning "we" invent our own meanings for those code

Don>Would we change what we're doing now? Only if PG intended to rejoin the rest of the world. points? Don>Would you seriously consider defining your own? PG already has redefined its own meaning for already-defined utf-8 code points, but PG "lifers" are so ingrained in smelling their own flower garden to even notice this.

Most have argued that UTF-8 requires Unicode. Technically you can UTF-8 encode any set of code points, but for this project it would serve no

Lee>Unicode without composition. purpose. I suggest that PG has historically "reserved" some commonly used Unicode code points for their own special purposes, and it would at least be wise to take the opportunity to choose much less commonly used code points for those special purposes, or alternatively use uncommon code sequences for those special purposes. The way the situation sits right now one cannot reliably write automatic tools to reliably process PG "utf-8" files.

On Thu, January 26, 2012 1:57 am, Keith J. Schultz wrote:

Hi Lee,

Am 25.01.2012 um 20:22 schrieb Lee Passey:

...

Depends on the file. XML files (XHTML, TEI, etc.) are guaranteed to be ASCII in their first line, and that first line declares the encoding, so no BOM is

Just to be picky. But, you err here. The above mentioned files are not guaranteed to be ASCII. only txt. Yet, as you state the first lines can contain encoding information.

XML files are not guaranteed to be "txt." In fact, the term "txt" has virtually no meaning, so you can't say anything is guaranteed to be "txt" unless you first define the meaning of that term. It is true that XML files can be encoded using UTF-16, in which case the first line will /not/ be ASCII, and a BOM should be required (what's the default byte ordering of UTF-16 if there's no BOM, I wonder?) But if the file is not UTF-16 then at least the first line is guaranteed to be ASCII as it must be the "<!xml" signature, and that signature is only composed of ASCII characters. This is possible because ASCII, Latin-1, Windows 1252, and UTF-8 are all identical for values in the ASCII range. So if the first two bytes of an XML file are not a UTF-16 BOM, an automated process can safely read the first line of text and determine the encoding of the entire file from there.

Wrong! Little endian is wrong, too! Am 26.01.2012 um 19:53 schrieb Lee Passey:

On Thu, January 26, 2012 11:29 am, Lee Passey wrote:

...

(what's the default byte ordering of UTF-16 if there's no BOM, I wonder?)

Big endian.

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

On Thu, January 26, 2012 12:16 pm, Marcello Perathoner wrote:

On 01/26/2012 07:29 PM, Lee Passey wrote:

...

But if the file is not UTF-16 then at least the first line is guaranteed to be ASCII

Wrong. It can also be UCS-4, UCS-2, or EBCDIC.

I stand corrected. The first line is guaranteed to start with "<?xml", so what a program needs to do is read the first 5 bytes and then figure out which of the "other encodings" yields that string. Would PG ever consider using one of these "other encodings" or is this discussion just academic?

On Jan 25, 2012, at 3:28 AM, Marcello Perathoner wrote:

The PG website already offers UTF-8 text only.

For a while I was sending up UTF-8 text only along with HTML. I stopped when it seemed that was causing a lot of work for the WWers. As of now, I send ASCII if that's sufficient, Latin-1 if it has characters in the Latin-1 set, and UTF-8 if it has characters not in Latin-1. There are two consequences: (1) everything that goes up in Latin-1 could go up in UTF-8 instead but doesn't and (2) I don't send up plain text with curly quotes at all. I work in UTF-8. It would be easiest for me to send up UTF-8. But the WW tools to check the submission, like gutcheck, struggle with UTF-8. The HTML I send up is UTF-8 and that survives because the WWers don't have to check it. They check the text file, which should be ASCII if it can be, and only UTF-8 instead of Latin-1 if there are characters that absolutely are necessary and not in Latin-1. Curly quotes are not viewed as necessary to the text version. Even an oe ligature isn't strong enough to justify UTF-8. Seems to me it's all about the tools. The WWers always seem overloaded and sending UTF-8 up makes that worse. If they had better tools to do their job, actually getting UTF-8 up on the PG website wouldn't be as problematic. This is an outside view of the problem. I am not a WWer. I'd love to hear from a WWer about Marcello's comment. Though correct as stated, perhaps it would be more accurate to say "The PG website already offers UTF-8 text only, but please don't send it to us unless absolutely necessary." I hope someday that statement will not be true, but I believe it is now. --Roger

On 01/25/2012 08:35 PM, Al Haines wrote:

As Roger mentions, curly quotes and oe-ligatures are insufficient reason for creating a UTF8 file.

As I already said I strongly disagree. You don't need no `reason´ to create an utf8 file. It's the standard encoding all over the world. It is the one file you should create first. You need a reason to create those other encodings that nobody uses any more. But what that reason could be is beyond me. I'd like to hear one, *only one*, good argument in favour of having those other encodings around. (The argument: "Because our tools are too decrepit to handle unicode." is not a good argument.) -- Marcello Perathoner webmaster@gutenberg.org

Hi All, Marcello is right utf8 is pretty much standard in these days. It should be also the standard used by PG. I do disagree that the other formats are not used anymore. Plenty of websites not using utf8. If the tools at PG do not handle utf8 well, the need to be reworked. regards Keith. Am 25.01.2012 um 21:05 schrieb Marcello Perathoner:

On 01/25/2012 08:35 PM, Al Haines wrote:

...

As Roger mentions, curly quotes and oe-ligatures are insufficient reason for creating a UTF8 file.

As I already said I strongly disagree.

You don't need no `reason´ to create an utf8 file. It's the standard encoding all over the world. It is the one file you should create first.

You need a reason to create those other encodings that nobody uses any more. But what that reason could be is beyond me.

I'd like to hear one, *only one*, good argument in favour of having those other encodings around. (The argument: "Because our tools are too decrepit to handle unicode." is not a good argument.)

-- Marcello Perathoner webmaster@gutenberg.org _______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

Hi Al, Am 25.01.2012 um 20:35 schrieb Al Haines:

I should mention that one thing that does slow things down for the WWers are text/HTML files with CR-only line endings. CR/LF is far preferable, since all the WWers use Windows. (I'm aware that there are fix utilities, but CR/LF's make things easier to start with.) You can be serious! For over 10 years I have not needed to worry about the line endings inside of a file. Any decent editor worth its fiddles will identify the line the line endings and internally convert to the native format and should either save in the native format or convert back to the original before saving. Furthermore these files are processed ahead of time and could be automagically convert to CR/LF. If the tools at PG can not do this, it is about time they did.

To my knowledge HTML is supposed to be submitted as a zipped file. Are zippers expected to convert the line endings into the native format!

It's also helpful to the WWers that text files are identified as to their character set. UTF8 files should have "-utf8" embedded in their filename, e.g. "somefile-utf8.txt". Latin1 and ASCII files much the same: "-ltn1", "-lt1", "-latin1", "-iso" or similar for Latin1 files, and "-asc" or "-ascii" for ASCII text files.

Good idea. regards Keith.

Hi David, I believe I mentioned modern editors in the rest of my post. Furthermore, the standard mode of zippers is the convert to native format. I was indirectly raising the question why the line endings pose a problem to the WWs? regards Keith. Am 26.01.2012 um 15:55 schrieb D Garcia:

On 01/26/2012 04:24 AM, Keith J. Schultz wrote:

...

To my knowledge HTML is supposed to be submitted as a zipped file. Are zippers expected to convert the line endings into the native format!

The vast majority of zip utilities are capable of converting the line-endings in the files given as input (or to be output) back-and-forth from CRLF/LF upon request. In practice, this capability is rarely needed, because the vast majority of text editors are capable of detecting line ending style at file open, of optionally converting them, and of saving in any of the typical styles (CRLF/LF/CR).

David _______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

... and primes, double primes, dittos, minutes, seconds, ... On Wed, Jan 25, 2012 at 11:38 AM, Marcello Perathoner < marcello@perathoner.de> wrote:

On 01/25/2012 03:30 PM, Roger Frank wrote:

I don't send up plain text with

...

curly quotes at all.

That is a mistake. There are lots of automated checks you could do in gutcheck et al. if you kept double quotes, single quotes and apostrophes separate.

-- Marcello Perathoner webmaster@gutenberg.org

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

...

...

...

...

"Jim" == Jim Adcock <jimad@msn.com> writes:

>>> I don't send up plain text with curly quotes at all. Marcello> That is a mistake.... Jim> I'm curious if anyone has any idea what percentage of Jim> submitted texts use "curlies" vs. just punting and going with Jim> ASCII-style "straights" ? My assumption has been that the Jim> great majority of submitted texts just use "straights." When I checked 10 consecutive random ones, 9 used straight quotes. IIRC they were 10 different submitters. Some of the DP perfectionists say that they like them better. Carlo

Ideally it would be the reader's choice, not the producer's choice. On Wed, Jan 25, 2012 at 2:35 PM, Carlo Traverso <traverso@posso.dm.unipi.it>wrote:

...

...

...

...

...

"Jim" == Jim Adcock <jimad@msn.com> writes:

...

...

...

I don't send up plain text with curly quotes at all.

Marcello> That is a mistake....

Jim> I'm curious if anyone has any idea what percentage of Jim> submitted texts use "curlies" vs. just punting and going with Jim> ASCII-style "straights" ? My assumption has been that the Jim> great majority of submitted texts just use "straights."

When I checked 10 consecutive random ones, 9 used straight quotes. IIRC they were 10 different submitters.

Some of the DP perfectionists say that they like them better.

Carlo _______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

If,t any point in your process, you decide it's valuable to consider what the text is, as compared to what it looks like from point to point, then it might comprise some amount of additional work. OTOH, if you begin to suspect that the structure of the document will help you construct a better product, then I think the idenfication of sentences and quotations leads to a fair number of validation opportunities that can be automated to provide exceptions to inspect with good chances for improvement. Once you have those, curling the quotes is cake. On Wed, Jan 25, 2012 at 5:08 PM, David Starner <prosfilaes@gmail.com> wrote:

On Wed, Jan 25, 2012 at 2:51 PM, don kretz <dakretz@gmail.com> wrote:

...

Ideally it would be the reader's choice, not the producer's choice.

That doesn't make any sense. They're still curly quotes, even if the font used to render them is straight. And they're not done, because they're not cheap to do; there's always some cost in going the extra step, and I think there's a valid question here about whether it's worth it.

-- Kie ekzistas vivo, ekzistas espero. _______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

On 01/26/2012 04:16 PM, Jim Adcock wrote:

And again there are lots of reader devices out there which are hard-wired to fonts which have really ugly implementations of curlies.

A master format should never be designed with current devices in mind. You are making the same error as Michael when he hard-wired the width of texts to 80 characters. -- Marcello Perathoner webmaster@gutenberg.org

On Wed, January 25, 2012 6:08 pm, David Starner wrote: [snip]

there's always some cost in going the extra step, and I think there's a valid question here about whether it's worth it.

Yes, that is a valid question. Not that many years ago I felt that curly quotes in a document was a silly affectation. Over time that view has moderated, and I now prefer reading curly quotes, so long as I can pick a font that does them fairly well (I can't even imagine using a device or software that doesn't let me select the font and install new ones). I've finally twisted Abbyy's arm far enough that it will preserve curly quotes, and I'm working on a system using Hunspell that will do a spell check on documents that contain them. So far, I think the cost/benefit ratio /for me/ favors preserving (or even creating) curly quotes, but I can definitely appreciate it if others make a different calculation.

Lee>Most people probably don't want to make the effort, particularly when straight quotes are "good enough." Straights were probably *not* good enough in my latest effort, but, oh well. IF hypothetically PG were to want to go curly I'd probably be willing to step up to the bat re fixing the texts I've already submitted.

On Tue, January 24, 2012 5:01 pm, Marcello Perathoner wrote: [snip]

We could start redoing the top 100 list excluding everything that is too hard and everything made by DP.

I could agree to this. I would point out, however, that as you yourself have pointed out the most "important" works are probably those that a volunteer feels most passionate about, whether it's in the top 100 list or not. I would suggest that the top 100 list be considered a suggestion, but tell volunteers that they could re-work /any/ book from the early days that is important to them. As you may recall, a few years back I built a top 100 list based on monthly download lists culled from the Wayback Machine at archive.org. I'd be happy to repost that list if anyone is interested. [snip]

A semi-official branch would be a good occasion to ditch the old WWer workflow in favor of a source repository (git or mercurial) that holds all the masters.

Agreed. I think that both git and mercurial are overkill for this project (this will not be a large project requiring branching and merging, non-linear development, authenticated history, high performance and low bandwidth), but if one of these SCM systems are the only ones you're willing to support I can live with that. From an end-user's standpoint I shouldn't need to worry about learning anything more about these systems that how to get a file, and check in modifications. Having a system like this would allow documents to evolve, and as modifications are made issues encountered and proposed solutions can be retrieved from the history. White-washers should be unnecessary, as no book will every be in a permanently "complete" state, and fixes can always be applied. This would be an implementation of the "continuous proofreading" that BowerBird frequently advocates.

Should we reserve a range of ebook nos. or shadow the existing ones?

Shadow the existing. That way everyone knows what the source was, and what the eventual replacement will be.

On Tue, January 24, 2012 3:08 pm, Joshua Hutchinson wrote:

So, if someone were to start "refactoring" old PG texts into TEI or RST and working with a WWer to repost them ... is this a workable idea? <div><br /></div>

<div>I'd love to see the PG corpus redone as a "master format" system (and the current filesystem supports "old" format files in a subdirectory, so if someone wanted to get the old original hand-made files, they could).  I'm not particularly wedded to any master format.  Hell, if someone came up with a sufficiently constrained HTML vocabulary that could be easily used to "generate" the additional formats necessary, I'm good with that.</div> <div><br /></div>

<div>But before anyone will start doing this work, there needs to be a concensus from PG (I'm looking at you, Greg!) that the work will be acceptable.  A half-assed "master format" system is no master format system at all.</div> <div><br /></div>

<div>I'm even ok with working up the system as you go (i.e., start with "simple" fiction works and make sure the system handles them before throwing more and more complex works at it, tweaking and fixing in the time honored method of "incremental development").</div> <div><br /></div>

<div>Maybe we start this process on a semi-private mirror of the PG corpus and only when it reaches a critical mass of some sort it gets moved over.  But an official notice that this project has some backing is necessary or we'll just keep seeing everything running around in ten different directions and nothing ever getting done.</div> <div><br /></div> <div>Josh</div>

I'm in. I think. Just to be sure, let me reiterate what I think I'm agreeing to. 1. A semi-official mirror of PG will be created. 2. Texts in the mirror will be refactored into a single file format which can be used to automatically create every delivery format offered by Project Gutenberg. 3. The focus of the project will be to re-work the most popular PG texts. At the outset simple works will be preferred to more complex works. 4. The project will evolve as new knowledge is gained. 5. The controlling principals at PG agree that if this effort is successful the refactored works will eventually replace the original works in the PG repository. Am I wrong in any of these broad points? If so, please clarify.

Lee>> This seems bizarre to me. What is the rationale for not allowing, let

alone requiring, an impoverished text file when the other submission is TEI?

Marcello>That you can generate the plain txt file out of the TEI. Following the PG FAQ on this subject: 1) Open HTML file in web browser. 2) Do a Select All 3) Open a text editor 4) Do a Paste 5) Save under a file name ending in ".txt"

On Mon, Jan 23, 2012 at 09:51:50AM -0700, Lee Passey wrote:

...

...

...

On 1/19/2012 4:43 PM, Marcello Perathoner wrote:

...

You can submit either *one* HTML or *one* TEI or *one* RST file. In no case has anybody been posting multiple HTML files.

I wrote:

...

...

...

Is this an exclusive 'or' or an inclusive 'or'? If I submit an HTML file am I precluded from posting a TEI or an RST file? Or may I submit one of each?

The answer is that you may submit one of each, as Marcello and Al and I have since clarified. And I now say again, "Confirmed."

[snip]

...

On Sat, Jan 21, 2012 at 07:28:30PM -0800, Al Haines wrote:

...

Marcello is correct.

[snip]

On Sun, January 22, 2012 1:21 pm, Greg Newby wrote:

...

Yes, confirmed.

I think this is a strong demonstration of why it is important for technical people take more classes in technical writing.

Mr. Perathoner started this thread making an ambiguous statement. The ambiguity was compounded when he stated "In no case has anybody been posting multiple HTML files," leading one to conclude that the issue here relates to multiple postings of a single markup format.

I then asked for clarification of that statement, to remove the ambiguity. Mr. Haines and Mr. Newby each responded saying, in effect, "yes, Marcello's original ambiguity is the policy of Project Gutenberg."

I am left to conclude that either the Project Gutenberg policy in this regard is still in a state of flux and the ambiguity is intentional, that Mr. Haines or Mr. Newby did not feel comfortable enough with their writing skills to actually respond to the question, or that neither of them gave anything more that cursory attention to the question actually posed.

Thanks for this idea. I will endeavor to give cursory attention to your future messages.

Given the usual practice on this list of paying little, or no, attention to what other people have to say, I suspect that the last alternative is the correct one.

In any case, the ambiguity remains.

You are being pedantic. The full message I responded to follows. The relevant extracts: Marcello:

You can submit either *one* HTML or *one* TEI or *one* RST file.

Al:

Marcello is correct.

It was decided in the early discussions of RST that additional custom files, e.g. a custom HTML file or a custom text file, would not be allowed. If custom files were desired/required, then the submission should be a normal text+HTML submission.

Me:

Yes, confirmed.

-- Greg ** Full extract: Yes, confirmed. Having numerous formats derived from a single master is a long-time goal. We've had some success with RST and TEI, and I've encouraged new projects to consider RST. There are still some limitations, though... On the many, many words on gutvol-d recently about poor results with auto-conversion from HTML to other formats (epub and mobi, among others): this is often due to choices that producers make about using HTML to impact layout, rather than just structure. Enough said. -- Greg On Sat, Jan 21, 2012 at 07:28:30PM -0800, Al Haines wrote:

Marcello is correct.

It was decided in the early discussions of RST that additional custom files, e.g. a custom HTML file or a custom text file, would not be allowed. If custom files were desired/required, then the submission should be a normal text+HTML submission.

The first discussion/submission of TEI files predates my time as WWer, but the same principle applies.

Al

...

-----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Lee Passey Sent: Saturday, January 21, 2012 4:52 PM To: gutvol-d@lists.pglaf.org Subject: Re: [gutvol-d] Producing epub ready HTML

On 1/19/2012 4:43 PM, Marcello Perathoner wrote:

...

You can submit either *one* HTML or *one* TEI or *one* RST file. In no case has anybody been posting multiple HTML files.

Is this an exclusive 'or' or an inclusive 'or'?

If I submit an HTML file am I precluded from posting a TEI or an RST

...

file? Or may I submit one of each?

Mr. Haines or Mr. Newby, would you care to confirm or deny this restriction? _______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

I give up. I can't put it any plainer than I did earlier, when I described the four types of submission. Obviously, the four are mutually exclusive. Duh... Al

-----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Lee Passey Sent: Monday, January 23, 2012 3:28 PM To: gbnewby@pglaf.org; Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] Producing epub ready HTML

On Mon, January 23, 2012 4:27 pm, Greg Newby wrote:

...

On Mon, Jan 23, 2012 at 09:51:50AM -0700, Lee Passey wrote:

...

...

...

...

On 1/19/2012 4:43 PM, Marcello Perathoner wrote:

> You can submit either *one* HTML or *one* TEI or *one* RST file. In no > case has anybody been posting multiple HTML files.

I wrote:

...

...

...

Is this an exclusive 'or' or an inclusive 'or'? If I submit an HTML file am I precluded from posting a TEI or an RST file? Or may I submit one of each?

The answer is that you may submit one of each, as Marcello and Al and I have since clarified. And I now say again, "Confirmed."

Thus illustrating the confusion. You say "one of each" but Mr. Perathoner and Mr. Haines have both said "only one of all". As I understand your position, I could submit a total of 4 files: one HTML, one TEI, one RST and one .txt. As I understand Mr. Perathoner's position if I chose to submit a TEI file I could submit only that one file.

According to Mr. Haines there are special rules for HTML files, but let's not cloud the issue.

Apparently, the ambiguity continues to be confirmed.

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

OK, I won't say anything...

-----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Lee Passey Sent: Monday, January 23, 2012 6:20 PM To: 'Project Gutenberg Volunteer Discussion' Subject: Re: [gutvol-d] Producing epub ready HTML

On 1/23/2012 4:50 PM, Al Haines wrote:

...

I give up. I can't put it any plainer than I did earlier, when I described the four types of submission.

Yes, once you took the time to actually answer the question you were

very clear. Indeed, you were doing really well right up to the point

where you started talking about RTF, PDF and Microsoft Word as part of an (X)HTML submission. Frankly, I don't see how it could be done, so

I'll just pretend it didn't happen; if you don't say anything I won't...

...

Obviously, the four are mutually exclusive. Duh...

How is it obvious? In the very post you're replying to Mr. Newby seemed to think that the answer was "one of each." On the other hand, I'm guessing that Mr. Newby has very little practical impact on or review of the nature of the files that actually get accepted, so I'm going to view yours as the definitive answer.

If and when you change your mind, it would be useful if you could post a message here, as apparently the FAQ is no longer being maintained, and in at least some cases is downright wrong.

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

On 1/24/2012 3:14 AM, Keith J. Schultz wrote:

So the FAQs and HowTos must be updated to mirror the commonly used policies and practices, and be more consistent.

That would be desirable and advisable, but apparently it will not happen as no one is willing to do it (I doubt a consensus could even be reached as to the answers published in such a document). The next best thing would be a disclaimer at the beginning saying that the information contained therein may or may not be the policy of Project Gutenberg, and that questions should be addressed directly to Mr. Haines. As a last resort, the FAQ should simply be withdrawn.

I have one question though. What if one has went through the work of producing a ePub, mobi, HTML and text files that conform to PG requirements, can they be submitted to PG?

According to Mr. Haines, they may not. ePub and .mobi may not be submitted under any circumstance, and HTML may only be submitted if it is accompanied by an associated .txt file that meets his requirements.

The reason I ask is that, as can be seen for the posts on this list production is moving more to the production directly of mobi and epub formats. These formats more or less contain HTML already. If they follow the guidelines for good ebooks. The HTML should fit PG requirments for HTML.

PG has no requirements for HTML, other than that it be HTML. Perhaps it is more accurate to say that PG has no /published/ requirements for HTML. I've little doubt that if you were to submit an HTML document it would be rejected for some reason. I think it would be an interesting project to collect these rejection slips, from which a de facto requirements list could be constructed.

With the above said all PG now needs is a conversion tool to create a plain text version.

Mr. Perathoner is unwilling to consider such a tool. If you would like, you are welcome to use my rather dated tool. Be aware that I think the text file requirement is, as Mr. Adcock pointed out, a requirement to preserve the status quo because it /is/ the status quo, and therefore I am unlikely to put any effort into maintaining the tool. Impoverished text is dead, let the rich text formats flower!

Greg>Having numerous formats derived from a single master is a long-time goal. We've had some success with RST and TEI, and I've encouraged new projects to consider RST. There are still some limitations, though... Greg>On the many, many words on gutvol-d recently about poor results with auto-conversion from HTML to other formats (epub and mobi, among others): this is often due to choices that producers make about using HTML to impact layout, rather than just structure. Enough said. This combination of paragraphs doesn't make sense. I recently posted in this forum an analysis, based on a request from this forum, that indicated that RST in practice *is not* working as a "single master." To whit the EPUB and MOBI being generated from RST is not particularly successful, possibly no more so than the DP "HTML" efforts.

Lee>On the other hand, it may be possible to take a page from PG's book and route around the damage. I've looked at the HTML output you've provided from PG and I haven't seen anything that can't be repaired. It should be possible to build a web interface that sits in front of PG, forwards requests, rewrites the HTML to meet industry standards, then either delivers /that/ HTML or compiles it into ePub or .mobi. In practice if/when people do this kind of things they cache the results so as not to keep hitting the PG website repeatedly unnecessarily. If one does that then consider simply mob getting the books via http://www.gutenberg.org/wiki/Gutenberg:The_CD_and_DVD_Project so as not to have to hit the servers at all. One needs to have more knowledge/love-of DB than I have to make a strong website, but http://freekindlebooks.org still runs as an example of this approach that predates PG willingness to host EPUB and MOBI, and which still draws a couple hundred thousand downloads a month -- even though I've been trying to steer customers back to PG now that PG -- more or less -- hosts EPUB and MOBI file formats. Not clear to me that the PG legalize would allow one to retain the PG tm on the files once one has internally tweaked the formatting problems, though. Bottom line though, is that the books most often downloaded from PG *ought* to be reworked in any case -- whether one wants to read those books in txt70, html, epub or mobi format. It is crazy that in practice PG doesn't have a way to rework that which gets read most often -- and which needs a rework.

On Wed, January 18, 2012 9:41 pm, Carlo Traverso wrote:

Jim, none of the submissions that I listed are mine, they have been submitted as PG-RST or PG-TEI master files, from which all the formats have been created by Marcello's software without further manual intervention.

Hey, /you/ asked.

In particular, for RST two different HTML are created in the process, one for browsers and one for epub. So it is unfair to view HTML on a small device.

It is not unfair, indeed it is desirable. What is unfair is to say "this HTML can only be adequately viewed on devices having these characteristics." In my mind the major flaw in all of the PG HTML files is the inclusion of a <style> block. Individuals who include a <style> block in an HTML file are attempting to create a document that looks "pretty" on their preferred HTML User Agent utterly without regard to the needs or desires of others. It should go without saying that beauty is in the eye of the beholder, and that one man's treasure is another man's trash. The best way to create ePub-ready HTML is to strip out /all/ of the style definitions from the <head> of the HTML files, replacing them with a single link to an external style sheet. The HTML markup should focus on the /structure/ of the document with little or no regard to its presentation; that is the job of the external style sheet. Standard HTML tags should be used, and should be used appropriately: no marking chapter titles with <p> tags! A standard set of classification attributes should be used. If I know that a chapter title is always marked with <h3 class="chapter"> then I know I can change the presentation of chapter titles in /my/ version by simply providing a style sheet file with the selector: h3.chapter { font-size: humongous; text-align: justify; color: stop-light-red; page-break-before: always } without changing it for anyone else (remember, treasure/trash). If a user doesn't have a style sheet and can't figure out how to download one of the standard ones from PG, the book will still look pretty good because the file is free from tag abuse. So the three basic rules for making ePub-ready HTML files: 1. Don't specify styles inside the HTML document, either inline or in the <head>. Do include a link to an external style sheet with the name "pg.css". 2. Use HTML elements only as designed; no tag abuse. 3. Use a standardized set of classification attributes ("class='standard'"). Help contribute to building a standard library of attribute values. Be willing to compromise in the naming of the attributes; standardization is more important than "correctness". The advice at http://www.pgdp.net/wiki/The_Proofreader%27s_Guide_to_EPUB is all good.

On 1/19/2012 8:49 PM, Al Haines wrote:

About 10 years or so ago, I remember reading about an initiative to produce a standard set of CSS classes. It may have been aimed at government/business documentation. I don't remember who the initiators were (W3C or whoever), and don't remember ever hearing that anything came of it. Does anyone remember anything like this?

Are you perhaps thinking of the initiative of the HTML Writers Guild to produce some standards for Project Gutenberg texts? (http://www.hwg.org/opcenter/gutenberg/tutorials.html. See especially, http://www.hwg.org/opcenter/gutenberg/markupXHTML2.html.) IIRC, that initiative was created back in 2000-2001, which coincides with your recollection.

On 1/20/2012 10:48 PM, Jim Adcock wrote:

...

Are you perhaps thinking of the initiative of the HTML Writers Guild to produce some standards for Project Gutenberg texts? (http://www.hwg.org/opcenter/gutenberg/tutorials.html. See especially, http://www.hwg.org/opcenter/gutenberg/markupXHTML2.html.)

An interesting site. Why not try the recommended DTDs and see where it gets you?

Because I'm not interested in DTDs. A DTD can tell you that you can't put a <p>aragraph inside a <span>, or that a <body> or <blockquote> no longer is considered a block element when it comes to <a> or <br> tags. But it can't tell you that <p> elements shouldn't be used for titles (use <h?>), or that lists shouldn't use a collection of nested <div> tags, but should use <ol> or <ul> or <dl> and <li> instead. When it comes to building "good" HTML, DTDs are of limited value (some, but not much). What is needed is good old-fashioned plain language, like that found at http://www.pgdp.net/wiki/The_Proofreader%27s_Guide_to_EPUB (but more detailed), a list of recommended classes and why you should use them, and respect and cooperation among the people willing to follow the guidelines. The list of tags at the HTML Writers Guild is a good starting point, as is the Wiki page a DP. Respect, cooperation and acceptance is nowhere to be seen...

Jana>There are several ways of specifying a cover in your HTML version that epubmaker will use for the epub (and Kindle?) version(s) of your book. See Marcello's advice here: http://www.pgdp.net/wiki/The_Proofreader%27s_Guide_to_EPUB#Cover_Page OK thanks -- it didn't occur to me to look in DP for advice on how to target a PG tool. Hopefully people who are providing an alt image for the "Blue PDA" realize that this image needs to include Title and Author. Which it will if people say just provide a jpg of the title page. But if they use the "first pretty picture" in the book, such as a frontal plate, then they need to use some kind of picture editing software to place title and author over that image. Thanks again.

Custom epub covers were discussed among the WWers and Greg Newby several weeks ago. (The question was raised by DP-Canada.) With Greg to confirm/deny/clarify as needed, the general guidelines decided on were: - the book's original cover is preferred, even if has only the title and author. (Obviously, not all covers have any graphical content, with some having a completely blank cover. I've even seen one with a full-cover illustration, with no title or author on it, those being only on the spine.) - they must be a reasonable size--50K-150K, 300dpi is plenty, most likely in jpg or png format. A 1M, 600dpi image, for example, if not rejected outright, will probably elicit a request for something reasonable, which if not forthcoming, the original would probably be reduced considerably by the WWers (and perhaps not used at all). - a custom cover should be provided only when a submission is first uploaded. They will not, repeat, NOT, be applied retroactively, i.e. to already-posted submissions. The WWers haven't time for a bunch of vanity reposts. - there must be a mention in the Notes to WWers field, and in a Transcriber's Note in the HTML file, that the image was produced by the submitter, and is being placed into the public domain. Obviously, PG wants no hassle because a submitter helped themselves, from some other website, to something *not* in the public domain. Al -----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of James Simmons Sent: Thursday, January 19, 2012 10:10 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] Producing epub ready HTML Strangely enough, I made some nice cover images for my three PG submissions so I could put somewhat better formatted versions in the Kindle Store. I would have been happy to include these with my PG submissions if I had known it was possible! https://plus.google.com/photos/114689250113657220289/albums/5681329822 764797041?authkey=COCMmsfqwNDOqgE https://plus.google.com/photos/114689250113657220289/albums/5676125941 560802081?authkey=CPDP0daf1ee9aw http://www.amazon.com/Ancient-Manners-Illustrated-ebook/dp/B0055OMGN0/ ref=sr_1_6?s=digital-text&ie=UTF8&qid=1326996235&sr=1-6 It would be good if this was better known. Maybe the White Washers could ask if someone wanted to include a cover image with his submission. They are not hard to make. A few minutes with The GIMP is all it takes. James Simmons On Thu, Jan 19, 2012 at 11:30 AM, Jim Adcock <jimad@msn.com> wrote: Jana>There are several ways of specifying a cover in your HTML version that epubmaker will use for the epub (and Kindle?) version(s) of your book. See Marcello's advice here: http://www.pgdp.net/wiki/The_Proofreader%27s_Guide_to_EPUB#Cover_Page OK thanks -- it didn't occur to me to look in DP for advice on how to target a PG tool. Hopefully people who are providing an alt image for the "Blue PDA" realize that this image needs to include Title and Author. Which it will if people say just provide a jpg of the title page. But if they use the "first pretty picture" in the book, such as a frontal plate, then they need to use some kind of picture editing software to place title and author over that image. Thanks again. _______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

Al>- the book's original cover is preferred, even if has only the title and author. I would be concerned that on many recent small devices the cover image is used in the form of a small thumbnail as the sole identifier for the book -- at least in many situations -- and the historical covers [as one can see demonstrated in the "Recent Books" section of the PG website] often offer, when scanned, negligible contrast between the main color of the cover and the (often) gilded title+author.

On 1/18/2012 8:15 PM, Jim Adcock wrote:

EPUB:

The ngx or whatever its called seems to be way overpopulated.

NCX. Navigation Control file for Xml. Designed by Daisy to support Digital Talking Books. Rammed through the IDPF with little discussion by Adobe, who is the IDPF's 800-lb Gorilla now that Microsoft has stopped caring. Adobe preferred the NCX to the existing "tours" element, probably as a result of the NIH syndrome. NCX is a rather chatty format, due in large part to the fact that it was designed as a navigation aid for audio books for the blind rather than as a simple table of contents. Fortunately, end users never need look at it, and it can easily be built using automated tools rather than by hand.

Page numbers now have magically jumped from the left margin to the right margin and what used to read "[PG 014]" now just reads a small grey "14" -- how do they do that?

Magic. Seriously, I think one needs to be cautious when trying to judge the quality of an ePub file by how it looks in any particular User Agent. Due to the influence of Adobe on the ePub specification, I believe that the Adobe Digital Edition reader is probably the most compliant of the ePub readers available, but even that software is not fully compliant with the specification. If you find problems with the way an ePub looks using any particular software it is as likely that the problem is with the software as it is that the problem is in the file. I'm guessing from your description that ADE is what you were using to look at the ePub. The developers of ADE seem to be committed to making ePubs as much like PDF as possible. Thus, ADE automatically puts a page number in the right hand margin of the display. If page numbers are not actually included in the ePub (more on this later) ADE will make some up. The only relation these made-up numbers have to the underlying text is that they are sequential, i.e. 2 is guaranteed to come after 1. BTW, ADE is not the only User Agent that carries on this charade. I know that Aldiko does it, and it's possible that the Nook does it as well. In Aldiko I've learned how to turn it off; so far, I haven't been able to figure out how to do that in ADE. [snip]

Hm, except now I see a line that says:

[pg 002][pg 013] ... 7

So now I am confused: Is this page 2, or page 13, or page 7 ???

The NCX file that is part of ePub is used to create the only Table of Contents that ADE recognizes (remember, NCX was Adobe's idea in the first place). But that is not it's only purpose. In addition to the <navMap> section, which defines that which is approximately a Table of Contents, an NCX file can also contain a <pageList> section which is intended to contain a list of page anchors that point to the beginning of each page in the document (because, after all, PDF is page oriented). In ADE, if this page list is present in the NCX file it is used instead of the made-up numbers. I think that what you're seeing is page numbers which were unadvisedly placed into the source HTML file and not suppressed by your User Agent (many ePub User Agents don't do a good job with the { display:none } style), plus the page markers auto-generated either from the NCX file or made up just because ADE (or equivalent) seemed to think you would want them (I don't, but YMMV). The HTML ePub maker used by PG does a pretty good job at constructing an NCX file from the structure of an HTML file based upon the ordering and levels of <h?> headers. It could probably be enhanced to build a page list in the NCX file as well, by finding the embedded page anchors in the HTML, adding references to those anchors to the NCX, and removing any visible component of the anchor before packaging the ePub. Anyone who is evaluating the quality of the ePubs produced by the PG tool would be well advised to learn the ePub and XHTML specifications, unzip the subject ePub and examine the markup by hand. There's simply too much variation among User Agents to be able to come to any valid conclusion any other way.

On 1/20/2012 10:32 PM, Jim Adcock wrote:

...

Anyone who is evaluating the quality of the ePubs produced by the PG tool would be well advised to learn the ePub and XHTML specifications, unzip the subject ePub and examine the markup by hand. There's simply too much variation among User Agents to be able to come to any valid conclusion any other way.

Let us remember that you are defending a pile of weirdness that PG is generating from one and/or another PG tool.

I can't see anything in my post that could possibly be construed as "defending" this stinky, steaming pile of ... What I /said/ was that when you're evaluating the quality of an ePub you need to look at its contents, not its behavior in any given User Agent. Which is apparently just what you did this second time.