ok, rfrank has made a report over in the d.p, forums on
the latest set of results from his "roundless" experiment.

so let's see what he says, and what i think in reaction...

***

rfrank said:
>   So far in testing the roundless system as it stands,
>   I've left it to the proofer to say when they thought a page
>   was done. Turns out, that is reliable for only a very few proofers.
>   Those who wish to say "I told you so" can chime in now, rightfully.

ok.


>   the post-processing clearable errors were
>   caused mostly by four proofers, and
>   each of those made several different kinds of mistakes.
>   These mistakes were found almost entirely
>   on pages that were one-and-done, that is, proofed once.
>   So what is to be done?

inform those proofers they are making mistakes, and how,
and that they not doing nearly as well as they think they are.

and to put this into perspective, there are about a dozen
committed proofreaders taking part in this experiment,
with another 5 dozen people contributing fewer pages...

so four "bad" proofers constitutes about 1/3 of the lot...

in other words, even though "4 proofers" sounds _rare_,
the actuality is the percentage of "bad" proofers is high.

this fact should _not_ be surprising.  when you fail to give
people any feedback on their performance, many will think
they're doing a fine job, even if they're doing a terrible job.

(this is a big problem over at the d.p. mothership, but we
probably shouldn't be getting into that can of worms now.)

after all, if they didn't think they were doing fine, they would
change what they were doing, so they _could_ be doing fine.
so you absolutely need to give them good and fast feedback.


>   One solution is to have every page
>   looked at by at least two proofers.

that seems straightforward, but it has some gotchas.


>   That seems straightforward but it has some gotchas.

right.     :+)


>   If every proofer knows that every page
>   is going to be looked at by someone else,
>   will they proof that page differently
>   than if they intended it to be one-and-done?

it's likely.  so you'd need to assume it, and work from there.


>   I think they might. Knowing the underlying mechanism
>   can undermine the process.

well, you must assume people "know the underlying mechanism",
because you want to be open and transparent about it with them.
there's really no other option when you're working with volunteers.


>   Also, what if the second proofer is one of the four mentioned earlier?

or what if they both were?


>   There is a good chance that many of the errors would slip through.

right.


>   It's easy for me to change the site code to force two looks at every page,
>   and I'll probably do that, perhaps even with a project in progress.

doesn't matter.  even after two forced looks, some errors will remain.


>   A down side to the "every page looked at by at least two proofers"
>   approach is specific to fadedpage: that there are only a dozen or so
>   active proofers of the 60 or so registered users. The double-look
>   algorithm adds about 35% to the number of page looks on a project.

doesn't matter.  there's no need for any haste on the books coming out...


>   A better solution that just a double-look is
>   to actually instantiate Confidence in Proofer (CiP).

i was afraid you were gonna say that.  and it's absolutely the wrong approach.


>   For these four proofers, the system could schedule a second look at
>   their pages even if they check the "this is done" box when done proofing.
>   It would give them plenty of diffs to look at, and they would be
>   expected to look at those diffs that show some correction was made.

well, it'd be better just to inform them and educate them in the first place,
rather than impose an "expectation" on them that informs them (indirectly)
and forces them to educate themselves (again, in a very indirect fashion)...


>   If diffs were not checked, then their access to new pages
>   would be reduced. The kind of proofer who checks diffs,
>   learns, and continues to contribute is exactly what is needed.

well, yeah, maybe...  but you're assuming a real luxury of
an overabundance of volunteers, and a willingness to throw
a good number of them away as "not exactly what is needed".

it's better to figure out how to find a use for _all_ volunteers.


>   I believe for a roundless system to work, there has to be
>   a reliable mechanism for stopping a page as done.

d'oh.  there has been complete agreement that that is the issue from day 1.


>   I also believe that to have a reliable way to make that determination,
>   some form of Confidence in Proofer needs to be in place.

some people have held that belief, yes.

i think it's unobtainable, and wrongheaded, and basically a dead end.
even if you get a rudimentary version, it won't turn out to be useful...


>   Therefore, CiP, which is important, and page tweets, which are
>   useful and fun, are currently my main coding efforts at fadedpage.

yeah, well, you'll be coming back sometime down the line and saying
"those who wish to say 'i told you so' can chime in now, rightfully"...

the thread has more, on confidence-in-proofer, but i'm not gonna
waste any more of my time dealing with that flawed concept...

-bowerbird