October 2012 - gutvol-d - lists.pglaf.org

list archive
by don kretz 24 Oct '12

24 Oct '12

Lee, I found reference to a post you made identified by From: Lee Passey <lee(a)novomail.net> To: Project Gutenberg Volunteer Discussion <gutvol-d(a)lists.pglaf.org> Cc: Date: Tue, 29 Jul 2008 18:18:31 -0600 Subject: Re: [gutvol-d] getting my wikisource bearings John Vandenberg wrote: that gave a long narrative of the markup rules you used and are apparently still using. Do you have a copy? The pglaf archives seem to have biig holes in them, one of which includes this time period. Also, bird, I can't find the list of elements you provided that need to be identified. Do you have a date for that? Don

2 2

powering through the adventures
by Bowerbird＠aol.com 23 Oct '12

23 Oct '12

well... "the adventures of huck finn" has certainly given me some "adventures" in creating my new, accurate, digitization... and i'll document them all in a summary. *** but for today, it's refreshing to know that i am finally _powering_through_ the book. you might remember that, as of yesterday, i'd reduced the comparison diffs to ~2000. and i was contemplating a shift away from my current "clean" text, since it's based on _a_different_edition_ than what i am using, one that has heavily edited the punctuation. that's why there are so many paired diffs... usually, with a book this clean, there might be 1% to 3% of the lines that would be diffs. in this book, completely due to punctuation, it's almost _20%_ -- ~2,000 out of ~11,000. and, in about ~92% of those diffs, the o.c.r. is correct, so the diff is being caused by the _edit_, rather than an o.c.r. misrecognition. the thing is, the comparison helps to catch the _other_ ~8%, which are real o.c.r. errors. so even though it's tedious to step through ~2,000 diffs, to find some ~160 o.c.r. errors, it is _still_ a rather efficient way of doing it... so i'm gonna stick with this different edition. but i adopted some "tricks" to power through. and those are going very well. i'm now down to ~1,000 diffs, so i'm sure i will finish soon. so i thought i'd tell you about these "tricks"... *** i didn't mention one of these "tricks" earlier, but it was the one where i focused in on the _words_and_letters_ instead of punctuation. i did that using the following trick... after finding a diff -- a pair of relative lines from the two files that were not the same -- i culled out the non-letters from each line... if, after culling, the 2 lines were the same, then that meant the letters were the same, so the diff was entirely due to punctuation, and -- since in that pass -- i was _focused_ on resolving differences in the words/letters, i could skip over such punctuation-only diffs. but if the lines were _different_ after culling, then i _would_ examine and resolve the diff. in the first round of this pass, a refinement is that i'd ignore the diff if the culled punctution was different as well, so i was only looking at diffs where _the-letters-and-only-the-letters_ were different, allowing a very-specific focus. after all those diffs were resolved, i looked at lines where letters _and_ punctuation differed. thus, when all of _those_ were resolved too, i knew that i'd resolved all letter-based diffs. (a further refinement, which retains spaces when the culled-lines are created, helped to find words where the letters were the same, and in the same order, but _words_ differed, such as when words were improperly joined, which can happen on rare occasions in o.c.r.) *** this methodology of narrowing the focus of the _type_ of diff being displayed, to assist your attentional focus in the resolving of it -- by letting you know what to "expect" -- can lend itself to a variety of these "tricks". for instance, earlier work on the files had shown me that one of them _hyphenated_ the expression "by-and-by", but the other did not. so i did a search just for that diff. it was quite easy to see that every instance (but one, which i considered "printer error") was hyphenated in my edition, meaning that i did a global-change to the "clean" text so those instances would not be flagged again. with my mind on the one track, it went fast. *** another trick was to present only the diffs where the diff lines were the same length. this usually meant that a punctuation mark had been misrecognized as another mark... e.g., a semi-colon misrecognized as a colon. this trick was very good at providing focus, so i expanded it, and displayed only those diffs where the line-length was off by one. then by two, and so on, until all were done. for instance, if the line-lengths were off by one character, that often meant that one of the two lines had a comma the other lacked. once you know that this is the nature of the diffs you are looking at, the resulting focus in your attention can speed up the process... some of you might see that this variable of line-length serves as a _proxy_, of sorts, for what is termed "edit distance" -- defined as the number of edits that needs to be made to one string to change it to another string. so using "edit distance" might be a further refinement that i could use the next time... and after that, i might examine the diffs, to categorize them, and then present only one specific type of diff, like when one line has a comma while the second line doesn't. another difficult situation is when the lines differ at one point, and then at a later point, and one line is correct at the first difference, while the second line is right on the second. in these cases, it would be good to _split_ the diff lines into the two respective parts. it will take a bit of coding, but it's doable... anyway... all of this new experience indicates to me that there are many things to do that will allow me to make the comparison process easier, speedier, and much more efficient. at any rate, we're close to finishing _huck_. -bowerbird

1 0

apple puts all the other companies in check
by Bowerbird＠aol.com 23 Oct '12

23 Oct '12

with its announcements today on "ipad mini", as well as its other advances, apple has now placed all of its chess competitors _in_check_. and now that the form-factor gap between the iphone and the ipad has been plugged, apple can look to the horizon to go bigger. because once apple creates the "ipad giant"" -- with a form-factor of 9*12, meaning that we can finally map our 8.5*11 paper-world onto it in a full-size, non-compromised way, for "all-in" on our paper-to-digital switch -- then it will be _check-mate_ and game-over. -bowerbird p.s. which doesn't mean that it'll all stop... the next step is for apple to put a _handle_ on the imac, so you carry it like a briefcase.

1 0

T. E. Lawrence "Seven Pillars" 1922 Oxford Text Edition published in the USA ???
by James Adcock 22 Oct '12

22 Oct '12

People who have been following this issue probably know the Lawrence "Seven Pillars" [["Lawrence of Arabia"]] as being perhaps the best known screwed up copyright issue around -- well, perhaps excepting Mickey Mouse. I have in my hands, to my surprise, a copy of T. E. Lawrence "Seven Pillars of Wisdom: A Triumph" purporting to be a copy of the 1922 Oxford Text - that which famously was "printed but not published but rather privately distributed" - and yet here it is being "published" in America. ISBN 9781463587376 Final pages stating: Made in the USA, Lexington, KT, 15 October 2012 And what would normally be the copyright page stating simply: ===== First paperback edition. All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, except in the case of excerpts by a reviewer, who may quote brief passages in a review. Printed and manufactured in the United States of America ===== Where a publisher isn't even identified in the book, but is given on Amazon as being "CreateSpace Independent Publishing Platform" My question is simply whether anyone knows anything about this? Is this just a "pirate" edition that Amazon has picked up unknowingly? Or has some newer copyright analysis been done by someone somewhere?

2 3

meanwhile, right here
by Bowerbird＠aol.com 22 Oct '12

22 Oct '12

meanwhile, right here... *** jim has pointed out numerous times that most of the e-books which he examines in his occasional random title selections are flawed, and he gives lists of the flaws. have any of these flaws been addressed? i once gave an extensive analysis of the problems with several different p.g. titles of "alice in wonderland", both old and new. have any of those flaws been addressed? i recently pointed out that even _headers_ in the "just-updated" low-number e-text for "pride and prejudice" were tagged as plain paragraphs, and displayed as such, along with some other problems with it... have any of those flaws been addressed? does any progress ever get made? _ever?_ or is everyone posting here just yapping to hear themselves make noise? i mean, me, heck, i'm just a poet/slacker, so i don't really have anything "important" to do, so i like to hear myself make noise. but i would expect that you other people do have more important stuff to be doing. so why are you wasting your time _here_? -bowerbird

1 0

here's your huck finn update for 10/22/2012
by Bowerbird＠aol.com 22 Oct '12

22 Oct '12

here's your huck finn update for today... *** i reduced the diffs from ~3,000 to ~2,000. so now all of the words/letters are "correct". (the job of standardizing word-forms over the entire book becomes very problematic when dialect words get phonetic spelling.) so now the ~2,000 diffs are on punctuation. based on my memory of punctuation in resolving ~1,000 diffs so far, i'd say that approximately 95% was right in the o.c.r. if that were to hold up, it'd mean we now have roughly 100 o.c.r. punctuation errors. i'm not sure it is worth it to resolve those ~2,000 diffs to fix 100 punctuation errors. (although, to restate the obvious again, that would be much more efficient than checking _all_ the punctuation manually, which is the workflow used over at d.p.) so this experiment confirms my earlier experience that different editions _can_ make comparison less worthwhile to do. (depends, of course, on how many edits were done to the text in the new edition.) it's especially true if you have the option to use an independent digitization of the _same_ edition, even if it'd be some work. as i _do_ have an independent digitization of the same edition, i will change course to execute "the final push" for this book. -bowerbird

1 0

blah blah blah more blah blah blah
by Bowerbird＠aol.com 22 Oct '12

22 Oct '12

don said: > That's why I want to challenge any proposers to > provide at least a prototype of what is being proposed, > in the form of examples collected in one place > so they can be remembered and compared. ... > It would also help to avoid reactions that > categorize proposals as categorically bad and > make general statements about > the proposer's character or intelligence. ... > Or fail to provide a better alternative, > with real examples we can compare > with each other. ... > Yes, bowerbird, I know you're providing the last item. > Not so much on the others. by "the others", do you mean the other people here? or my performance on the "other" items? (whatever those other items might be.) and what, specifically, does "the last item" mean? you could probably write a little more clearly, if you worked on it, for better communication. -bowerbird

1 0

blah blah blah times 10
by Bowerbird＠aol.com 21 Oct '12

21 Oct '12

don? are you there? if so, please listen up and consider this... first of all, most of these jokers are now in my kill-filter. jim, passey, starner, marcello, you know, the whole crew. all because they failed to add value to the conversation. (or actively subtracted value.) so i really don't want to put you in that kill-filter too, don. but you're seriously going to have to step up your game... because you're not adding much value these days. just sayin'... *** don said: > There's a legitimate argument that, if we hope > many people are going to be involved proofing a text, > the instructions need to be clear, simple, and few; that's pretty hard to argue with... but if someone wants to try and defend a position for instructions that are vague, difficult, and numerous, well, gee, i dunno, but that might be... entertaining... > and they should be matching the image, that needs to be heavily caveated. i know that's the common mantra, but there are tons of "exceptions" and breaches that y'all are evidently just no longer capable of seeing... there are many reasons not to match a scan. > plus as little markup as possible > to capture the essential semantics i _agree_ with that. but i suspect that our "agreement" might be only temporary, due to yet-unspoken criteria. > to capture the essential semantics > (to be defined.) well, _some_ of you might still need to "define" the "essential semantics". but i gave you a list a rather long time ago, so my status is just fine. > Few would expect DP proofers to be > applying complex formal markup > like TEI, ReST, or latex. you are extremely unclear if you're talking about d.p. as it currently exists, or as you think it should change, or whether you mean "p.g. times 10", or something new. so let me be perfectly clear when i say that i am _not_ interested in d.p. as it currently exists, or might change. i am interested in a "best practice" theoretical way only. having said that... we can't expect an ordinary person to know t.e.i. or latex. or to learn it. on the other hand, r.s.t. isn't all that complicated. and neither is markdown, or z.m.l., or other light systems... > ZML is the only markup that anyone has suggested > might have a chance, but it has other issues. _you_ might have "other issues" with z.m.l. but z.m.l. itself has no "other issues". speak clearly, and stop trying to spin, because you're no good at it. besides, the point i have always been making is that a light-markup system will work. you don't have to use z.m.l. -- there are other light-markup systems. and you can even invent your own, if you prefer that. but there's another, deeper, point still lurking here, the "unspoken" assumption that i mentioned above. the way you have put things just now informs us that you have already chopped the digitization process into a "proofing" stage, where it is expected that people are rather witless creatures only capable of basic decisions. then brighter people come along at some later stage and apply their brains to make the text ever-smarter. i reject that model. wholeheartedly. i say normal people can take pages from start to finish. maybe not every person, maybe not every page, but almost every person can _finish_ almost every page. and no, i'm not the only person who believes that. roger frank built a system on that very premise, and his experience was that it worked very well. if you treat people like they are dumb proofers, all you'll get from them will be dumb proofing... but if you encourage them to _finish_ the page, what you'll get from 'em is lotsa finished pages. the reason you don't see that kind of performance is because you haven't empowered your people to give it to you, thanks to the tools that are at hand. the first tool you have to give them is light-markup. they need a simple system that does the job _well_. the second tool you must give them is a _viewer_... once they have applied the light-markup, they need to be shown their html-rendered page, so they can compare it to the scan to make sure it "looks right". since your light-markup is designed in such a way that it is impossible to do "presentational" markup with it, if the display "looks right", it's because it is coded correctly from a _semantic_ perspective... if it doesn't "look right", it means it's coded wrong, and must be repaired. this is how you get accuracy. so you give your "proofers" a system (with tools) that allows them to _finish_ the page right away. *** > On the other hand, once the handwork is done, > a version of the text that is exhaustively constructed > with all the essential semantics (to be defined) > clearly identified and located would, it is supposed, > make it possible to write mapping modules to > generate formatted texts that convey the structure > and meaning of texts clearly and attractively. that's a lot of words that don't really sum to much, at least not that couldn't be phrased more simply. the irony is that all of this confusion is introduced simply because you're _talking_ instead of _doing_. once you toss off all of the superficial aspects of applying a certain "type" of markup, you'll see that there is little disagreement on the essential base... all you guys are gonna apply _very_ similar .html... further -- according to your own rhetoric -- the .css is user-swappable, so nothing crucial can reside there. so once you get down to the doing of actual examples, you'll find there's nothing left to argue about in words. of course, then you'll probably fail to realize that you were wasting your time all along. indeed, you'll likely think you "accomplished" something, and proceed to slapping yourselves on the back and giving high-fives. but seriously, you're just wasting your time. > We have few examples of such a "master text" > that has proven capable of providing the source > for clear and attractive ebooks in various formats. bullshit. bullshit bullshit bullshit. i have given numerous examples. and can provide more on demand. i'm not sure what it is that you think you gain by telling such a bald-faced lie, right in public. > I think no one is advocating any form of text > that serves both purposes simultaneously, > so that apparently leaves another step in the process - > converting proofed text into some master text. i'm not even going to go back and try to sort out what those "purposes" were, but i assure you that it is not necessary to "convert" proofed .zml into "some master text", because it already _is_ that... > Not many have offered to discuss how this would be done. i've been discussing it for 10 years here. > Which leaves the suspicion that some might expect > the DP "post-proofer" volunteers to fill this role. i don't know what you're talking about. and i suspect that you don't know either. > That seems an unlikely scenario to me. whatever... > But however it is accomplished, it would probably > (it seems to me) to depend heavily on the path > provided by the essential semantics (to be defined.) there you go again. > I mentioned the McGuffey text as a possible test case > for a discussion with real examples of alternatives. you still haven't provided accurate text. so, do you want "real" or not? seriously. > I thank Lee Passey for picking up the cue. an e-book with inaccurate text is worse than none at all. > I find it tedious to endure advocates who only discuss > their own ideas, so this will have at least two of us > for a while, anyway. i hope the two of you have a nice long happy relationship. > I'm currently using the McGuffey text, > and learning from how Lee has applied XHTML as markup oh, how sweet is that. you're _learning_ from lee. darling. what a good relationship. > I'm building a list of candidates for "essential semantics". i can hardly wait. for something i did 8 years ago. > It will be interesting to me to see how Lee > views his construction from the same perspective. isn't it wonderful to fall in love! you want to hear everything the other person has to say, about _everything_. it's so tender. > In the meantime, I'm using the resources of > the WordPress installation on readingroo.ms > see how much of the semantic content of McGuffey > is available (as bowerbird would like it to be) > inherently in the plain text, don't be an idiot, don. it's _not_ that i would "like it to be" the case that the "semantic content" of a book can be gleaned from the type that is set on the page. it _is_ true. and it always has been. that's how human beings soak up the knowledge that's printed on the page. there's no one sitting next to 'em pointing out that "this is a chapter-header, and this here is a list..." the _words_and_typography_ do all of that implicitly. you technocrats are so caught up in your "separation" of _content_ versus _presentation_ that you _ignore_ the fact that it is presentation which signals structure. so those two things aren't independent in the slightest. they are woven together inextricably into a potent mix. so first you toss out the presentation as unimportant, and thus you need to reintroduce the structural info... you're like a food-processor who boils out all of the natural sugars, and then puts in artificial sweeteners. so, in the midst of you thinking you are oh so smart, you're really being just about as stupid as you can be. > and how much needs to be added, > by constructing views that allow > different versions of text to be > viewed in parallel, and edited. this is a nice vague chunk of text that sounds like it is a _teaser_ for some kind of demonstration. let's see it. > The host has the essential capabilities of a linux server, > including utilities for I think all the formats > commonly mentioned (except ZML), screw you, don. i've had sites up that handle z.m.l. for ages now. something doesn't cease to exist just because you won't look at it. so anyone who wants to try out z.m.l. can do it in my sandbox. they don't need to go to yours. *** seriously, step up your game, dude. -bowerbird

1 0

top 10 reasons why this current thread will go nowhere -- #6
by Bowerbird＠aol.com 19 Oct '12

19 Oct '12

#6. you want to treat the exceptions, while you ignore the main rule. #7. d.p. post-processors were recruited with "you can do it your way." #8. if your workflow has no supporting tools, you have no workflow. #9. this list has proven it can ignore a solution put right in front of it. #10. that is where these threads always go, and nothing has changed. -bowerbird

1 0

huck finn doan' be kickin mah ass no mo'
by Bowerbird＠aol.com 19 Oct '12

19 Oct '12

so huck finn, mah frien'... yer _dun_ kickin mah ass! *** i've finally overcome the huckleberry monster. that's the good news. the bad news is that the book isn't finished yet. the ugly? roughly 3,000 diffs in the line-pairs between my version and a "clean" comparison. the reason is one that i've known all along, and which has multiplied my difficulty-factor greatly, but i chose to bear it, for precisely that difficulty. the reason? my "clean" copy is a different edition. you can tell from the first sentence in the book: my version: > You don’t know about me, > without you have read a book > by the name of > “The Adventures of Tom Sawyer,” > but that ain’t no matter. the "clean" version: > You don’t know about me > without you have read a book > by the name of > _The Adventures of Tom Sawyer,_ > but that ain’t no matter. notice the comma after "about me", and the quotemarks around the book-title... (the other one did it in _italics_ instead.) that was a big tip-off to massive differences in punctuation throughout the two editions... so i knew about the issue, but i persisted... because, again, i wanted to see how big of a problem this would pose for the comparison. first, even _preparations_ for the comparison (mainly restoring linebreaks and pagebreaks) were hindered quite a bit by these differences. indeed, the prep may be the biggest problem. maybe i'll be able to ease the prep tangles, now that i have a better idea what they are. we'll see... but even then, 3000 diffs is a lot to resolve... of course, it's a lot easier to resolve the diffs than to check all the punctuation "manually", which is what the proofers over at d.p. do... (just so's we keep it in perspective, mind ya.) but still, that's a heaping bunch o' diffs. now, the next thing to note is that _a_tool_ that's custom-built to do such comparisons, and resolve diffs, might well ease this task... i'll have some comments on that in a minute. but even with a good tool to help the process, 3000 diffs is a lotta diffs to check, no kidding. so it just might be the case that _eventually_, i will change over to another "clean" copy that _is_ from the same edition, such as this one: > http://www.marktwainproject.org/xtf/view?docId=works/MTDP10000.xml;chunk.id… as you see, it's from "the mark twain project", so we can expect that they took some care... just as a sidenote, it's interesting that _most_ of the copies of this text floating in cyberspace are _not_ from the edition that i am using and which the "mark twain project" also is using... i'd guess that this is the case because _most_ of those copies floating around were _based_ on the project gutenberg e-text to begin with. i haven't done the forensics research required to bolster that claim, but that is my suspicion. *** so let's talk about a tool for doing comparisons. the input for the tool will be the two text files: my in-progress o.c.r. file, and the "clean" text. the prep got these files synchronized into lines which match those that we'll find on the scans. (between the slanguage and the punctuation, this synchronization was tremendously thorny.) so our tool now walks through the line-pairs, stopping when it finds a diff to show the lines _plus_ the scan for the relevant page, so that we can see which of the lines (if either) is right. so you know, you'll find a copy of this post here: > http://zenmagiclove.com/misc/comptool0.txt and here's a graphic of the basic framework: > http://zenmagiclove.com/misc/comptool1.png you'll also notice, in that screenshot, that i create a "diff line" underneath the pair, to flag the diff(s). this "diff line" is a hugely important functionality, because it spots the diff(s) _accurately_and_fast_, so i don't spend my valuable attention doing that. (that task can be time-consuming, and draining.) so i see what the diff is, then i examine the scan to resolve the diff by seeing which line is correct. it's a straightforward task. but let's speed it up. first, it is troublesome to locate the appropriate _place_ where the line resides on the page-scan. but it would help some if we were to know that "the line is at the top of the page" (or the middle, or the bottom). even better, say the line-number! so the first innovation is to give the line-number: > http://zenmagiclove.com/misc/comptool2.png you'll see the big number "8" next to the text-pair. it indicates the diff is in the 8th line on the page... this is a chapter-opener, so there are "blank" lines; and line "8" means "toward the top, but down a bit." we can do better -- let's signal where the lines are: > http://zenmagiclove.com/misc/comptool3.png oh, this helps a lot. in the margins, we show circles where each line will fall. (i did put in line-numbers, but that started to make the display a little "busy"; so i compromised by coloring the actual "hit" line.) scans aren't always done uniformly, so the circles might be off, and aren't to be taken too seriously -- the other good reason i deleted the numbers -- but they're excellent at helping to focus the search. but we can even do better. since we have an idea about _where_ the appropriate line resides, we can screenshot a portion of the scan around that point, and display it right at the top, under our line-pair. voila! > http://zenmagiclove.com/misc/comptool4.png wow. that makes a really nice and useful display... my eyes don't have to travel anywhere at all. great! so, the next part of this job is indicating to the tool which of the lines is correct -- or if both are wrong. so here's the functionality for that: > http://zenmagiclove.com/misc/comptool4.png you'll see we now have 3 buttons at upper-right... the "1" button means the top line is the right one. the "2" button indicates the bottom line is correct. the "3" button says that we're _skipping_ this pair. each of the buttons, once selected, stores the info, and then instructs the tool to locate the _next_ diff. this minimizes the interaction for each comparison to a single button-click, which makes the task easy. and yes, you can keypress "1" or "2" or "3" instead. in addition, the cursor-up key says "top line is right", while cursor-down says "bottom line is right", and cursor-right key says "let's skip this pair for now..." (cursor-left will take you back to the previous pair, if you realize that you indicated the wrong choice.) also note that the text-fields are _live_, so you can _edit_ them, in case neither one of them is correct. pick the one that's easiest to fix, repair it, and then click its button, and your edited line will get saved. (there are times, however, where you simply cannot make a decision; that's why there's a "skip" option.) you can also do a "find" operation on a search-term, or jump to a specific line in the file, if you need to... other abilities which can be periodically useful are the ability to skim along chapter-heads, or pages. all of this is very straightforward vis-a-vis coding; nothing here challenges a competent programmer. there are some additional tricks that i will describe, once i've started in on those 3000 diffs, but _this_ is already more than enough for you to think about. *** i haven't said as much, but i'm not trying to "hide" the fact that my clean copy is the one jim created, which is up at project gutenberg as e-text #32325. i initially picked it because i thought a comparison between his copy and mine would be beneficial to both of us. but since i've had to bend his so much, to facilitate the comparison, that's probably not true at this time, at least not as it applies from his view... so now, if jim wants to do it, he will have to "bend" my version so as to facilitate a comparison with his. -bowerbird p.s. here's some more info to detect the editions. > http://zenmagiclove.com/huckf/wisdom.py?whatpage=26 > "Oh, she'll do, she'll do. That's all right. Huck can come in." > the "clean" copy, #32325 from project gutenberg. > "Oh, she'll do. That's all right. Huck can come in." *** > http://zenmagiclove.com/huckf/wisdom.py?whatpage=189 > They swarmed up the street towards Sher- > burn's house, a-whooping and yelling > and raging like Injuns, and every- > thing had to clear the way or get run > the "clean" copy, #32325 from project gutenberg. > They swarmed up towards Sherburn's house, > a-whooping and raging like Injuns, and > everything had to clear the way or get run and as the final note here, pg#76 is a little embarrassing, because it includes graphics from the edition _i'm_ using, like the one on page 189, from chapter 22, meaning that words in the graphic _contradict_ the digital text below it.

1 0