February 2010 - gutvol-d - lists.pglaf.org

Re: rfrank reports in
by Bowerbird＠aol.com 10 Feb '10

10 Feb '10

jim said: > I just checked my previous claims about the problems with P3 > (for example) against a pretty straight-forward text. > I reviewed 200 pages from that text. > P3's made 38 changes on those pages. > Of these changes 7 change represented a positive > contribution towards making the txt correct. > Of those positive changes about a half could > easily be found by a simple tool like guiguts. > 10 of the changes introduced by the P3's were negative changes > -- changes that moved the text to a less perfect state. jim, and others, if you're going to continue to discuss the problems in the current system at distributed proofreaders, please do it in your own thread, and not in my threads, ok? as for your findings, jim, it helps to report the actual lines. but it's probably not necessary. in the past, i have documented the same things you report, in great detail, in book after book after book, with evidence. in comparison, your anecdotal reports are relatively flimsy. i'm not saying you're wrong. indeed, you're absolutely correct. i'm just saying your reports are not going to convince anyone. heck, there are people here who refused to believe what i said, in spite of the fact i piled up enough evidence to choke a horse. (nor did i _create_ the evidence; i used data taken directly from various experiments, performed over at d.p. by other people... the truth is out there, and easy to find, if you just care to look. this is what is so silly about all these "experiments". i've told everyone here the simple correct answers, so all that's needed is to test these simple hypotheses and see they _are_ correct. but instead people are testing overly complicated stuff in ways that are not definitive, leading them to become more confused.) *** perhaps the most impressive findings of my results were these: 1. the best way to know a page is "finished" is when proofers stop making changes to it... up to that point, it's not finished! it's the "best" way to know because a no-diff is easy to measure. 2. 3 rounds of p1 were as effective as a series of p1-p2-p3. (simple solution to queue problems? run the text through p1 until every page comes out repeatedly as a no-diff page.) 3. the third round of p1 found as many additional errors as the p3 round, but _neither_ route found _all_ the errors. the p1(3) proofers found errors that the p3 proofers missed, and the p3 proofers found errors the p1(3) proofers missed, plus there were other errors that both p1(3) and p3 missed. (takeaway: the p1 proofers are not inferior to the p3 proofers, and a "make the page better" philosophy will eventually work to "create a perfect page", without all the attendant pressure.) 4. "parallel" p1 was _not_ useful at turning up any more errors, but it might have value to determine a page is "done", although more research would need to be done to test that hypothesis... -bowerbird

1 0

Re: how to make roundlessness work, in one brief post
by Bowerbird＠aol.com 10 Feb '10

10 Feb '10

jim said: > many motivated "early readers" love a particular author, > and would be happy to get early access to the text via > some kind of tool that allowed them to fix or at least mark > the bugs they find as a part of their reading. "Marking" bugs > as a part of reading could be as simple as asking them to > read on a notepad or what have you and put a > Q-mark in the text where they think they see a bug. while you seem to be talking about smoothreading here, the text you quoted from me was about preprocessing... preprocessing happens before the text goes to any proofer -- it's scheduled immediately after o.c.r. has been done -- and it doesn't require reading of _any_ kind at all, which is why it is about fourteen times more efficient than proofing. a preprocessing tool finds glitches that are almost certainly errors, and takes you to them directly in the text-file while displaying the appropriate scan for referral, and often even gives you buttons that will perform the desired correction... some glitches (like spacey quotes) can even be auto-fixed. i have demonstrated here that, for several books i tested, this preprocessing would take the errors down to a rate of less-than-one-error-every-10-pages, which makes the word-by-word proofing rounds almost like smoothreading. check out dkretz's "twisted" tool to see an example of this... -bowerbird

1 0

Re: rfrank reports in
by Bowerbird＠aol.com 09 Feb '10

09 Feb '10

jim said: > Under the *current* DP system i'm steering clear of discussing the d.p. system right now; there's so much cruft over there it's not worth the trouble. i'm either discussing rfrank's experiment, or my own system. -bowerbird

1 0

how to make roundlessness work, in one brief post
by Bowerbird＠aol.com 09 Feb '10

09 Feb '10

here's how to make a roundless system work, in a nutshell... 1. do aggressive preprocessing. 2. use nonintrusive zen markup. 3. submit the page to a p1 proofer. 4. repeat #3 until no change is made. 5. submit the page to a p1 proofer again. 6. if a change is made, go back to step #3. 7. if there is no change in #5, page is done. if you want it even briefer, do aggressive preprocessing, and then repeat processing through p1, until you obtain 2 consecutive rounds of no change, and the page is done. for greater accuracy, or if you have proofers in abundance, repeat until you get 3 consecutive rounds without change. (but the increased accuracy isn't worth the increased work.) for lesser accuracy, stop after 1 round that sees no change, but the decreased accuracy here is too high a price to pay... aggressive preprocessing is the secret, because most errors can be located automatically, so the pages are clean before they even get to "proofers", who are really "smoothreaders". this, of course, is the same formula i've suggested for years. (once you've hit upon the right answer, no reason to change.) you can easily assure yourselves that this is the right answer; track how many errors persist through 2 rounds of no-change. (versus how many persist through 1 round, 3 rounds, 4, etc.) no need to collect any messy stats. just 2 rounds of no-diff. the time you spend exploring other stuff is just wasted time. just watch. this is the formula that will prove to be the best. and when you get around to admitting it, i'll say "i told you so". -bowerbird

2 1

Formats and gripes
by Greg M. Johnson 08 Feb '10

08 Feb '10

I observed a bunch of things in the PG collection that were somewhere between maddening deficiencies in quality and simply how I wouldn't have done things. I could have made inflammatory demands that the whole system be rewrought according to my word. Instead I came up with my own custom DVD. And I'll testify that MH in discussing the idea was annoyingly non-paternalistic in not telling me how it had to be. ;) So, when someone complains, it is great when they can just go fix something themselves. There's the story of the king who demanded the whole world be covered in soft leather to keep his foot from being struck on a stone. Maybe someone should come up with a custom DVD with all the good works in EPUB format. I would respect PG if it were to refuse to host any and all of these bleeding edge formats. But I would love to be given the authority to create a raw text format without the 80-char line breaks. I agree that having these breaks (in the only TXT format available ) makes them unusable. The first-time visitor to PG probably won't be impressed that tools may be available to fix it.

9 26

roundlessness -- 003
by Bowerbird＠aol.com 07 Feb '10

07 Feb '10

we're looking at rfrank's "roundless" experiment at fadedpage.com... *** ok, it's been a while since i went through this little drill, so i'm gonna give you a little refresher-course on where i stand on a workflow that distribution proofreaders could be using, particularly from the viewpoint of rfrank's roundless research. *** first off, i would substantially re-ramp the "preprocessing"... this is the first step, where you're doing the scanning and o.c.r., or fetching the results from one of the major scanning projects. we'll start at the very beginning... *** i've advocated -- strongly -- a filenaming system that indicates, for each file, the contents of the file, so the name is meaningful. specifically, the _page-number_ of the file should be included in a consistent way in the name of a file. thus, for instance: > http://z-m-l.com/go/myant/myantp123.jpg the image-file for page #123 in the book "my antonia" is named myantp123.jpg. the "myant" prefix is common to files for the book, and the "p123" part refers to page 123. this is so straightforward that you couldn't be faulted for assuming that it's taken for granted. but it's not. many of the content-providers over at d.p. name their files differently, so the filename is _not_ unambiguously tied to the page-numbered contents inside; and a price is paid for this unclarity. yes, rfrank is one of these ill-advised content-providers, and he has carried over this bad habit to this experiment on roundless proofing. this shortcoming is even worse in a roundless system, because there is a need to refer to specific pages in such a system, and the absence of a sensible naming-scheme therefore becomes a bigger problem... (in a round-based system, where all the pages are treated as a "batch", the problem isn't as bad, but it's still something that should be fixed.) *** next, there are a number of things that are done in "preprocessing", at distributed proofreaders and by rfrank in his roundless experiment. some of these things should _not_ be done, in my considered opinion. on the other hand, there are other things that _should_ be being done. these are some things that are done which should _not_ be done: 1. run-heads and page-numbers are eliminated. 2. end-line hyphenates are being joined. 3. end-line em-dashes are being "clothed". these are some things that _should_ be done, which are not: 1. obvious and easily-located problems should be fixed. 2. spacey-quotes should be fixed. 3. ellipses should be standardized on a 3-dot ellipse. we can engage in debate on all of these suggestions, but it will be instructive for us to see some of the o.c.r. errors i'm talking about. appended to this post is a list of bad words or lines that were pulled from a book currently being proofed at rfrank's roundless experiment. these are almost all errors that will certainly need to be fixed. aggressive preprocessing can find these errors without looking for them. that's very important, in a roundless system, because if you can find and fix all the errors _before_ a page is subjected to a word-by-word review, then that word-by-word review can become the first "no-diff" in a chain of "no-diff" reviews, and the shorter that chain is, the more efficient you are. in the other scenario, if the page is dirty, you might have to have 1 or 2 (or even 3) proofings before the page is clean enough to receive a no-diff, meaning your efficiency has plummeted. there's no reason to make a human _search_ for errors when those errors can be located quickly and efficiently by a computerized search routine... -bowerbird p.s. sorry i've been sluggish with this series. i thought i'd been away from this stuff long enough that it wouldn't bore me to do it all again, but so far it has been a drag... i can only repeat this so many times... i'll try to get the motivation back again, but no promises if i cannot... p.p.s. here's that list of "probable errors" pulled out of rfrank's book: of'more Fd say pver t'other Enew a'slipped pn curtiss-robih somethin'to ght asighin' buncft whick mother'Ship ground^igood hefteaw iny wonderf ^Devolutions punkins outen tpreviously bustl gun'ls thaft ag^ny sud-v denly ij J>ack jumpin'his blame'em Jiack apture twise Weil numba chorteled pvounded Wowl oaly ^{w}Oh! haulded uae vre knovt apeak gink givin* stretch.'* J'ack wheen clost you'l althought eyefull weuns I"<cr> oa pHot etchin' jest'magine iframe ha.nd you'fe valk a'been outen morc'n MCGrath tc Unc' wuss'n pizen fresfi hirn orr hinsel| But you never can tell just what may hap]7^{en sion of weird noises springing up from the g<9^{a}^ 74-* /' 161 17S "Working over a bird with red feathers,'^{1 fall for such a decent game 93 taxidecentry or 18? was the work of a few second3. Hardly had the THE COMEBACK 227 HUNT OF THE S-18 belonged to the Hun pHot, Oscar Gleeb. must be pesU,@Jn' you like all get-out, so I made cruel to keep me a'guessing any longCi. so anxious to get started oA their way Porter Press disappears from an airpnQp .Watch my smoke, that's all." .until finally it died away completely. This gave .keep your eyes fastened on him. Whatever . A fortune hangs in the balance when young Dan Tierney, press , "Gripes! that was worth somthin'to glimpse, ; for Perk valued a few words of praise ;whooped the delighted Perk as he squatted pn you to hold my own. That's j^t how it should say,^{r} she bust out o' that little fog cloud right such as would tell ^ business being put through rendezvous and^ it's our game to chase after them, light, whether ^{r}'and or gulf, the chances were holding ground^igood--a heavily laden sailing path, and going through the most wonderf ^Devolutions. and be ready to pounce down on their inten<J^{e}d prey after the fashion of a hungry eagle striki ^{n} a fat duck that had been selected out of the flc^{c}k One thing he did do was to cut his intend^ wide circle short and again head toward ^{ne }scene of action, a move that certainly afford^ the eager Perk more or less satisfaction, he bel^{n} thrilled with the expectation of breaking into th^{e }game without much more loss of time. But you never can tell just what may hap]7^{en }when rival forces are striving against one ^^{n}' other. The best laid plans often go wrong a^{n}^ there was always a chance of the unexpected happening. Hardly had the airship whipped around ag^^{n }so as to head into the north than Perk beca<"^{e }aware of the fact that there was a sudden acc^{es}' sion of weird noises springing up from the g<9^{a}^ toward which they were now aiming. Jack, t^ must have caught the increased volume, for ^{ne }sheered off as if to hold back a bit so as to gr^^{s}P the meaning of the new racket. Men were no longer simply talking or laug^h" ing as they so cheerfully labored in transferring some of the contraband from the sloop to ^{ne }deck of the speedboat--their voices were rai^^{e}^ to shouts in which surprise, even the element ^ the frenzied sufferers in their ag^ny had been ^{w}Oh! That can be put through without muck said that name exactly three tin^s, like it meant operator as Oswald Kearns pick oui^an ordinary would rather have Jack praise him than ^ny one ^{r}fully five feet long and as thick through the body him, no matter where he goes-^-sorter dude, I'd will y^{u}> boy--two--three fellers jest swarmed "Working over a bird with red feathers,'^{1?? M^ans our gent has a raft o' ships comin' an' through fire or some similar means of destruc^ our man ditto. Mebbe now I'll soon^{x}get a chance "Gosh, amighty, we're flyin'\Mgh, buddy!"

1 0

roundlessness -- 002
by Bowerbird＠aol.com 07 Feb '10

07 Feb '10

we're looking at rfrank's "roundless" experiment at fadedpage.com... as i said yesterday, this test is a very very very very very good thing, because distributed proofreaders has been bogged down in a morass of "rounds" for many years now. their standard workflow now calls for _three_ rounds of proofing, followed by _two_ rounds of formatting... throw in a "preprocessing" round, and their "postprocessing", which is following by "postprocessing verification", and you've got 8 rounds. i don't know about you, but to me, that seems like a lot... but that's not the worst of it. the worst is the resultant backlogs... the problem arises because d.p. has thousands of proofers doing p1 (the first round of proofing), but d.p. only has hundreds that do p2 (the second round), and mere _dozens_ doing p3 ("final" proofing)... needless to say, the large number of proofers doing p1 can proof more than the smaller number doing p2, or the tiny number in p3. the backlog created is (understandably) frustrating and demoralizing for the proofers trying to keep up in p2, and is killing the p3 proofers. there is also the gnawing feeling that not all pages _need_ 3 rounds. indeed, _most_ pages in _most_ books are simple enough that they can be finished in one round, two at the most. so the _inefficiency_ of the 3-round proofing is rather striking as well. the thought is that each page should be proofed only as many times as that page needs; this has been labeled as a "roundless" system. aside from the backlogs of partially-done material, the other sign of a problem with the dp.p. workflow is that production has flattened... even though d.p. enjoys a constant stream of incoming volunteers, thanks to all of the good-will that project gutenberg's free e-books have generated over the years, d.p. output has leveled out at under 250 books per month, which works out to less than 3,000 per year. against the backdrop of the _millions_ of books google has scanned, this is a mere drop in the bucket. a small drop in a very large bucket. rfrank doesn't go into all of this on his site. perhaps he didn't need to, since the d.p. people he's recruited are well-acquainted with the issues. but rfrank is also unclear on many of the details of his little experiment, which is a more worrying matter. specifically, i don't see a lot of experimental rigor here. it seems to me that roger is unfamiliar with the mechanics of the scientific method and its applicability to human social experiments. i see no evidence of any stated hypotheses, nor any way such hypotheses can be disconfirmed... the reason people developed the scientific method was because we found that when we just fooled around "to see how things turn out", we often ended up fooling ourselves about what we had seen, and what it meant. we learned that we had to actually specify our hypotheses, and devise tests (experiments) specifically designed to disconfirm our hypotheses. otherwise, our brains are only too willing to accommodate what we find as being "supportive" of our initial impressions. ("experimenter bias" is the term by which this insidious phenomenon is most well-known.) if i'm correct, this problem will surface in rfrank's future results, and surface repeatedly, so there's no need for me to labor the point now. but i wanted to frame this particular issue, here and now, in advance. that's enough for today. see you tomorrow... -bowerbird

7 10

Re: Psychology of interacting with (Google's) ebooks.
by Bowerbird＠aol.com 06 Feb '10

06 Feb '10

greg said: > So what on earth are these big-pipe servers doing? collecting information, so as to assemble it, and reassemble it, thereby producing new information that might change the world. or make money. lots of money. lots and lots and lots of money... -bowerbird

1 0

Re: psychology of interactimg with ebooks
by Greg M. Johnson 06 Feb '10

06 Feb '10

<<discussion of poetry auto-generated files messed up>> IMNSHO, another fruit of making the TXT-80 format the default standard. The programming will ne a headache if one can't just delete *all*the CR's.

2 1

Re: psychology of interactimg with ebooks
by Bowerbird＠aol.com 06 Feb '10

06 Feb '10

greg said: > The programming will ne a headache > if one can't just delete *all*the CR's. nah. programming an unwrap routine is quite easy... > http://z-m-l.com/go/unwrap.pl -bowerbird

1 0