February 2010 - gutvol-d - lists.pglaf.org

Re: ok, let's take a look at gardner's book, just for the exercise
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

gardner said: > Interesting. The colours go a bit wonky at um, actually, i have the colors totally reversed, all the way through. and the "bad" line was on the bottom, when i usually put it on top... i confused myself by looking at some diffs where the o.c.r. was right and your version was wrong. they are few and far between, but i still managed to convince myself that up was down and port was starboard. sorry! > substituted the correct spelling "Sainte Hélène" where the that's a different problem. my text-editor assumes mac-roman, so i canceled out the few places where there was high-bit text. > But there are places where "my" text contains errors > that I do not believe were in my text. again, that's because i mixed up the colors. your text was in red. i got it straightened out, and will be uploading a new version soon, so wait until i do that before you spend any more time checking it. (and actually, you don't have to do anything; i will do a full report.) i'm sorry, i apologize for the confusion. i ain't perfect... :+) -bowerbird

1 0

Re: ok, let's take a look at gardner's book, just for the exercise
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

i did some more work on gardner's book, "the advocate". i continued to do preprocessing on the archive.org o.c.r., and posted the results on my website. here's a sample url: > http://z-m-l.com/go/gardn/gardnp123.html the number of lines that differ between the cleaned o.c.r. and gardner's file has dropped to under 300, which is great, since the book contains about 3,900 lines in it. so it's <8%. so again we show the stupidity of a word-by-word proofing. i'm gonna go through one final round of preprocessing, and then we'll set out to resolve the differences. already, however, i can tell that gardner did a very good job on it. not perfect! but certainly something to be quite proud of. the (presumably incorrect) lines from the o.c.r. are in red, while the lines from gardner's proofed copy are in blue... if you prefer to view all the edits on one web-page, see: > http://z-m-l.com/go/gardn/gardn-hybrid5.html i also included the preprocessing edits i made on page 126, so you can get a feel for how little time i spent doing them... -bowerbird

2 1

Re: Bowerbird's software projects
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

gardner said: > http://unixcomputer.net:81/new-photo/cd/Advocate/ i scraped them from canadiana.org myself... :+) but hey, do you still have your original o.c.r.? or a latest version that has the original linebreaks intact? *** > I don't believe that a single pass is feasible, ok, i should elaborate. multiple passes, to check different aspects, will be required. but multiple passes to check the _same_ aspect are inefficient. > I don't believe that a single pass is feasible, > in particular for mismatched quotes and spaced quotes. i believe you're wrong, and that i can show you. > in particular for mismatched quotes and spaced quotes. > You fix the open quote, or in my case close more often than not, > then that reveals another quote problem further down/up. i have already demonstrated that you can fix spacey quotes, and -- in the vast majority of cases -- fix 'em automatically. leading and trailing spacey quotes are easy to fix, of course. from there, it's a simple matter of segmenting the text into _paragraphs_, and counting quotemarks in each paragraph, making sure that the odd ones are open, and the even closed. then when you come upon a spacey quote, fix it to be open if it is an odd one, and fix it to be close if it is an even one. if you come up against a case where there is an odd number of quotes in a paragraph, and the next paragraph does not start with a quote, then you have a case you need to look at. similarly, if any of the quotemarks come up as the wrong type (an odd that's close, or an even that's open), you need to look. you can test this for yourself. you'll find that it's very robust. usually there's no need to spend much time on spacey quotes. > In any event I am not troubled by multiple passes. ok. > Well it *is* a Gutenberg text after all. right. that point wasn't directed at you, as you correctly realized. > Thanks. well, the fact that you haven't wasted your time is only _part_ of the equation. the fact that you won't get much credit down the line (because _your_ text will be discarded because you threw away info that people will want) is yet another (bigger) part of the equation. > Sure. The book *is* public domain after all. Do what you like. i think you missed the point. you can mount a version of your work that doesn't throw away the important information, and then no one will have to re-do it, in which case they will be happy to continue to give you the credit. -bowerbird

7 7

Markup and Pagination (was RE: Re: !@!!@!!@!Re: Re: so what is so important about pagination?)
by cmiske＠ashzfall.com 23 Feb '10

23 Feb '10

The only justification that _I_ can see for the PG 'books' to contain some form of pagination would be in a markup version with a hidden reference to the source page (scan image - if one exists). This would facilitate referring to the original book pages for simple corrections in the released versions. If it is _your_ intention to preserve the book, then the retention of pagination and other formatting would make perfectly good sense as well. I have followed this project (and this list) for around a decade. I have always been of the understanding that Project Gutenberg was about making available the contents of public domain books to as many people as possible. It has never seemed to me that a goal of the project was to create a re-printable facsimile of an existing edition, but rather to create the Project Gutenberg edition. I’ve always thought of us as the ancient scribes who spent their lives copying the contents of aging editions of books onto new materials so as to preserve the content for another few generations of time. The wonderful thing about the digital editions at Project Gutenberg is that they will last ‘forever.’ I have wondered for quite some time why the primary format for archiving (not necessarily submission or release) at PG does not include markup. It would be amazingly easy to generate the PG plain text version (for release) to exacting standards from a rather simply marked up original, but it is rather complex and potentially error prone to produce any format other than plain text from the plain text document. I have done the final plain-text formatting stage before and it is a slow and potentially human-error riddled process to manually format to PG standards (many of the standards I refer to are actually from DP rather than PG). I believe volunteers could be found to markup texts submitted (current and past) that lack such enhancements and to work towards getting the backend of PG more standardized so as to facilitate greater accessibility and distribution of the content. I realize that DP is already producing HTML versions, but that is not precisely what I am referring to and does not address the multitude of files without HTML versions and those that follow individual markup standards. And, I would never suggest that submission of a book be blocked or release be delayed by a contributor not having the ability or desire to markup their document. I am only suggesting the PG use a more re-usable formatting method for archiving and on-the-fly content generation than the current standard of somewhat-standard plain-text (and somewhat PG/DP standard HTML). There has been a lot of talk on here over the years of markup and preservation of information from the original book. Markup would enable the retention of as much (or as little) information beyond the content as the volunteers wished to provide without interfering in the creation of a Plain Vanilla text file. It also would enable the creation of one document with multiple editions nested within (or at least the difference between editions). The later being what I believe Michael just indicated was his ‘dream.’ Also, markup, when done with that purpose in mind, is human readable. <chapter title=”Chapter I” subtitle=”The quest for solitude”> <paragraph>Helen was <emphasis display=”italics” edition=”1845, Dover”>not</emphasis> in the mood for company and she resented the <errata edition=”1845,Dover” type=”replace;typographical error” details=” nkock”>knock< /errata> at the door….</paragraph> <paragraph><quote>Just go away!</quote> she yelled and then went back to reading her copy of <title type=”magazine” display=”italics;underline” edition=”1845, Dover;1865,NY Press”>The Mystery Guild Weekly</title>.</paragraph> …. <page number=”1” edition=”1845, Dover” /> <paragraph>She really had no idea why there were so many people coming <page number=”1” edition=”1865,NY Press” />to her house on such a day as <errata edition=”1845, Dover” type=”replace” details=”to-day”>today< /errata>. One would <errata edition=”1865,NY Press” type=”addition” details=”almost” /> think it was her birthday….</paragraph> …. </chapter> This is a rather poorly thought out and quickly done markup schema, but I believe it serves the point in that it is human readable. It would not be an enjoyable read, but it would be readable. And, long after all of us are gone, if this were the only copy of this document left on earth, someone could make sense of it enough to convert it to plain text (or another more readable format). They could also create a reproduction of the 1845 book or the 1865 book. Or, create an edition of one year or the other with footnotes or sidenotes/marginalia of the differences in the other edition(s). In the meantime, while we are still around, PG could publish the Plain Vanilla version (and any other versions they choose to make available). I proposed something similar to this many years ago over at DP, but there was little interest in it at the time. I’m not sure there is much interest in it now…. In any case, thank you to all the volunteers for all of the hard work over the years and all the books you have provided for my pleasure and for the pleasure of so many others! I’ve never felt as though you get thanks often enough from all of us users of the end results of your dedication. Thank you! Carel

1 0

Re: !@!!@!!@!Re: Re: so what is so important about pagination?
by Bowerbird＠aol.com 22 Feb '10

22 Feb '10

michael, i wish you would've taken the time to read what i actually wrote, instead of just giving your kneejerk reaction. because your response doesn't address the point that i made. i am loathe to get into this argument, because it won't mean a thing in the long run... the future will have its own issues, and it will need to deal with them, and i laid it all out clearly. but let me just address a few things, to provide some clarity. > he fails to talk about the wide variety of Jane Austen's > p-books, and that paginations run rampant among them, > not to mention margination, spelling, etc. there are different editions of many books, to be sure... and i count each edition as a separate book. your e-book will have to mimic _one_ of the editions in a faithful manner, or it will be discarded... notice that when i say "mimic", i do _not_ mean that it has to match it _exactly_. so, for instance, if you wanna close up spacey contractions, or correct spelling, or make other kinds of changes, they might (or might not) be totally acceptable to any one specific end-user in the future... but you _will_ have to make it easy for that specific end-user to _compare_ your e-book with a p-book, in order to spot changes. i've shown how this comparison is done, by mounting a web-page which has the text on one side of the screen, the scan on the other. but if your e-book is a formless blob, that's not gonna cut it... and, for the record, i'm most certainly _not_ recommending that we create some "scholarly" version of our books. i laugh at that. the _only_ thing we know about the scholars of the future is that we do _not_ know what they will want and it'd be foolish to guess. put yourself in the shoes of the future. you have a dozen different e-book files, all purporting to be copies of "sense and sensibility". you know that some of them have been doctored, and others have been bowdlerized, and you _hope_ that some of them are accurate. you can, with some work, find the differences between them, but you'd prefer not to have to go through that exercise if you could, because you'd have to then do more work to find the _right_ copy. so, how do you proceed? well, i can tell you that if _one_ of those copies made it _simple_ for you to verify its accuracy by assuming the form of the p-book, that will be your obvious first choice. think about it. you'll agree. so, michael, if you want to respond to this, answer that question. -bowerbird

2 1

Re: so what is so important about pagination?
by Bowerbird＠aol.com 22 Feb '10

22 Feb '10

as i said earlier, there's no real need to provide "justification" for pagination. the fact that _some_ people want it is enough to make us decide that we shouldn't toss out that information. however, since nobody has mentioned the _best_ justification for including pagination information, i might as well tell you... here at the end of the paper-book half-millenium, we have roughly 10 million different books out there in the world... (this is according to my memory of recent figures, which may be off, perhaps even by a large amount, but that's immaterial.) if we figure there are an average of 1,000 copies of each book, that means we've got about 10 billion copies of paper-books... that's a lot of paper-books out there in the world. a whole lot. those paper copies are the _originals_, and they always will be. in the future -- even right now, thanks to google -- we have a virtually unlimited number of digital copies of those originals. but again, those digital versions will _always_ be "the copies"... and the paper-books will _always_ be "the originals"... forever. (even books that're "born digital" often become physical quickly, and that will continue into the far future with print-on-demand; and paper-books, due to their _physical_and_material_ nature, will always be the "real" books, while digital versions will always be the "copies", especially since they can be manipulated at will, while physical books have the virtue/liability of being "frozen".) "real" doesn't mean "more valuable" or "more important", it means _physical_ and _tangible_ and _visible_ and _made_out_of_atoms_. you really have to ground yourself in this thinking to understand -- _physical_ books are the "real" ones; digital books are "copies". that's our first important factor... and our second important factor is that e-books are manipulatable. and just as the frozen nature of p-books is both virtue and liability, so too is this manipulability. on the one hand, it's easy to fix errors, provide updates, and so on and so forth... but, on the other hand, it's also easy to alter the book in a way the author did not intend... and if you don't think people _will_ try to rewrite history, you're nuts. plus there's just sheer incompetence, which has already resulted in a number of very shoddy digitizations of books, full of inaccuracies. just try and find all the copies of "pride and prejudice" out there, and then do a determination on which ones are "accurate" and which not. you will find this task to be overwhelming, and nearly impossible, and that's just one book out of our 10 million books. that is the problem. so there's little question that people in the future will be _skeptical_ about each and every e-book which they are handed, and rightly so... for reasons from accidental to quite intentional, it might be inaccurate. so we have a state where there are some "known" p-book "originals", and a ton of digital "copies" that might or might not be "trustworthy". (i believe jon noring has been absent from here for long enough that it's once again safe to use that word without all his derogatory spin.) now, there's only one solution to this state. any specific digital copy will have to be able to _prove_ its correspondence to a paper-copy... the easiest way to provide such proof is to assume the same form as the paper-copy; that is, it must adopt the linebreaks and pagination, so that each and every page can be subjected to visual confirmation... of course, in order to have value as a digital book, the file must be able to drop the linebreaks/pagination, and assume another form, one that reflows to the current set of desires of the end-user, _but_ it _must_ be able to mimic the look-and-feel of the paper-book too. if it cannot, it's simply going to be discarded as being untrustworthy. your e-book cannot afford to be nothing more than a formless blob. it _must_ be able to "snap to" a form that exactly imitates the p-book. and for it to be able to do that, you must keep linebreaks/pagination. it's really that simple. -bowerbird

2 1

hallelujah
by Bowerbird＠aol.com 22 Feb '10

22 Feb '10

hallelujah! some people are finally talking some sense into rfrank's head. one person suggested some reg-ex tests be shown to proofers. all by itself, this is an improvement, but not that big, because these tests should be done _before_ the text goes to proofers. however, what it did was it jolted roger out of his thinking that such tests are done in _postprocessing_, a huge ideological shift. roger admitted as much. thank you lord! another person pointed out that a global search-and-replace would be a real asset. d'uh, who's been saying that for _years_? one person said: > I've noticed "Pem" scanned as "Pern" a few times. roger responded with: > Done. Fifty-seven replacements. yes! see how easy this can be? so roger said, "tell me what kind of global changes you'd make". so one person came back and said, "how about things like these:" > change did n't to didn't > change could n't to couldn't kinda hard to believe that those fixes weren't already being made in preprocessing, isn't it? but hey, let's be glad for the progress... sure enough, roger got the hint, and changed all of the floating contractions, and then promised he'd do that in _preprocessing_ next time. hallelujah! now is the time to give roger that list of 30 tests that i highlighted back in that month-long series i did... now the _next_ thing would be for someone to _volunteer_ to do all of this preprocessing for roger, using the tool dkretz coded... hallelujah! -bowerbird

1 0

Re: "The Inheritance" by Susan Edmondstone Ferrier
by Bowerbird＠aol.com 22 Feb '10

22 Feb '10

al said: > Hmmm... maybe bowerbird is up for them? what's my motivation, al? the one edition i looked at -- in english, i speak no german -- is a simple book, quite straightforward, so wouldn't prove anything. i mean, i'm totally willing to run through the exercise-wheel, but what do i get when i come out the other side? how about this, a win-win-win-win situation for everyone... for a while, the p.g. website has been directing newbies over to distributed proofreaders if they want to help out with the cause. and sure enough, d.p. gets a ton of volunteers as a result of that. unfortunately, d.p. doesn't appreciate the newbies, because they just contribute more stuff to the plethora of p1-proofed backlog. so how about, if i were to do this book for you, p.g. would start sending all the new volunteers to rfrank's roundless site instead? d.p. happy, rfrank happy, the bird happy, al happy, and p.g. happy. so, do we have a deal? -bowerbird

3 2

Re: so what is so important about pagination?
by Bowerbird＠aol.com 22 Feb '10

22 Feb '10

keith said: > I am not talking markdown or Restructured. > I am talking about a true markup langauge. it appears you don't really know what you're talking about, as both markdown and restructured _are_ "true markup languages". you want to invent a new one. fine. go ahead. i did it myself... -bowerbird p.s. don, restructured text is older than markdown, and textile too, as far as i know... it's a reworking of "structured text", which is the granddaddy of all the light markup languages. and -- by the way -- z.m.l. is older than markdown. indeed, one of the few appearances of charlz on this listserve was when he came to announce markdown as a z.m.l. clone...

2 1

[SPAM] re: Re: so what is so important about pagination?
by Bowerbird＠aol.com 21 Feb '10

21 Feb '10

jim said: > I dislike anything that prevents reflow, > which I think is necessary for the enjoyment of most users. there is absolutely nothing about retaining pagination (or linebreaks, or end-line hyphenates) that "prevents reflow", jim, and i wish you would stop repeating that nonsense. you need to pay better attention. -bowerbird

5 8