- gutvol-d - lists.pglaf.org

Re: roundlessness -- 013
by Bowerbird＠aol.com 14 Mar '10

14 Mar '10

i said: > even then, i just don't think "confidence in proofer" will actually work. > > or, to be more accurate, i think it'll work just well enough that rfrank will > put lots of energy into it before he finds out it doesn't work well enough. > > or, worst case scenario, he'll convince himself that it really _is_ working, > and other people believe him, and we all end up with non-perfect pages. good thing i posted that yesterday. because today rfrank posted his first "informal analysis". and it looks like i was right... rfrank did his analysis on 32 pages that were marked as "done" but then subsequently proofed again, as is done for a random sample... he admits this is a small number of pages, and that there are also "many factors at play", but then goes on to draw conclusions anyway. of the 32 pages, two had added proofer notes, and 1 error was fixed. he doesn't tell us if either (or both) of the proofer notes were good, in the sense that they pointed out something of value, so we'll have to assume that they were meaningless and just added noise to the text. but even then, we have 1 error missed in 32 pages. on the face of it, that means that 3% of the "done" pages had an error. so, for a 200-page book, that would cumulate to a total of 6 mistakes. again, by my 1-error-every-10-pages criterion, that's fully acceptable. but by the (unrealistic) standards of _most_ of the volunteers, it's not. rfrank concludes that "this seems to say that making sure every page is seen by two proofers is not warranted"... so that's his take on this. *** partly the decision rests on the abundance of proofers. if you have lots and lots of proofers, like d.p., then you can afford to send a page through them 2 times or 3 times, even 4 or 5 times. but if your proofers are scarce, like they are over at fadedpage.com, then you might be reluctant to have them view a page even twice... i think i'm pretty good about making sure proofers are used _wisely_. i don't think i abuse their contribution, or that i take 'em for granted; neither am i afraid to use their resources if it is responsible to do so. and i think having 2 people verify a page as clean is responsible use. *** the other thing, though, in evaluating all these experiments, is that you need to know how many errors there _really_ were on each page. only _then_ can you accurately access the accuracy of the proofers... remember that there are lots of pages in these books that have _no_ errors on them, none at all. is it any surprise, then, that they were _actually_ "done" when they were _marked_ as "done"? not hardly... likewise, it isn't really a surprise when a page with _one_ error on it has that error fixed, is then marked as "done", and is _really_ done. what you have to pay attention to, in such cases, are the pages where an error is _not_ found by the first person, who marks it "done", but is then found by the second person. rfrank isn't making nearly enough information available that we can analyze the results in a reasonable way. so i guess we just have to "trust" him. i just wish i had more faith in his reasoning. -bowerbird

7 8

Re: New Tool "pgdiff"
by Bowerbird＠aol.com 14 Mar '10

14 Mar '10

jim said: > Sorry – I guess I assumed people know how to use a regex editor! um, you just shot yourself in the foot, jim. you assume that people know how to use a reg-ex editor, but you also assume they need to have wdiff wrapped in an .exe? not much logic there, i'm afraid... but hey, for the sake of completeness of the thread, how about you quickly run through how i'd "use a reg-ex editor" for this? because i honestly don't know. (and i _do_ know how to wdiff.) -bowerbird

2 1

[SPAM] re: New Tool "pgdiff"
by Bowerbird＠aol.com 13 Mar '10

13 Mar '10

jim said: > To help clarify what I am talking about > I enclose below an except of the output of this tool personally, i find that output to be obtuse and hard to read. and editing it would be very problematic, and error-ridden. giving people a tool that would present the choices to them, and let them click a button for the correct one, would make this far more easier to work with. *** jim said: > I have created a new command line tool “pgdiff” good for you, jim! i assume you wrapped the "wdiff" routine in an .exe? that'll make it easier to use for normal windows users. > In this regards it is similar to “worddiff”, as opposed > to “diff” which is the approach BB has been > talking about, which compares on a per-line basis. well, i use "diff" as a generic term. whether you use "diff" or "wdiff" depends largely on whether the lines are broken in a similar way, or have been rewrapped. i usually find it's worthwhile to fix the linebreaks so they are identical in the files, and match the p-book. that's because to resolve many of these differences, you have to look at the actual page, and that job is infinitely easier if your linebreaks match the page... > But my new tool has several tricks > that haven’t been seen before: um, ok... > It can be used with two different versions or editions of > the text as long as there are not really long differences ok, but that's something that's been "seen before"... > This means it can also be used for “versioning” – > for example using a copy of a PG text from one version > or edition of a text to help fix and create a text > from a different version or edition of the text. i'm not sure i understand what you're talking about here. if there are differences, how do you know if the differences are edition differences or o.c.r. differences? you'd have to refer to the page-scans for one version or the other, right? > It can also be used to recover linebreak information, > where linebreak information has been lost, for example > to take an older PG text and recover linebreak information > in order to allow, for example, the resubmission of that > PG text back to DP for a clean-up pass. again, not something that hasn't been seen before... but i'd love to see this in action. carlo has _posted_ that people could use wdiff to do this chore automatically, but when asked to explain the procedure, he failed to follow up. > In normal mode when if finds an mismatch it outputs > the mismatch like this { it’ll | it’11 } within the body of > the text so that given a regex compatible editor it is > very quick to search for and fix the errors found. i'd really like to learn the reg-ex that makes this "very quick". i assume you'd search for the first half of the pair, and erase it if it's incorrect. then you'd do the same for the second half. then you'd go back and globally remove the excess characters. but i'd sure like to see that in action. and i don't think it would be very fast. or feel very easy. especially when -- for an error like '11 -- a global change within each of the files would end up being more efficient. it's also the case that, as i mentioned up above, you _need_ to have the scan available for viewing to resolve some diffs, so the ability of the tool to present those scans is _crucial_. > I find that finding differences on a word basis rather than > a line basis makes it quicker and easier to fix the errors if you've looked at the diffs i've presented, the _indicator_line_ narrows your focus down to a single word (if that's the diff), or even a single _character_ (like a comma, if that's the diff). it's just showing you the entire line so you have the _context_, and so you can _find_that_line_ more easily on the page-scan. > Source and a compiled windows version at i'll take a look, as soon as i happen to be around a windows box. in the meantime, congratulations for programming a tool! :+) -bowerbird

3 2

any arguments against automatic good-word listing?
by Bowerbird＠aol.com 12 Mar '10

12 Mar '10

i recently praised rfrank for simplifying the procedure of adding a word to the "good-words list", compared to d.p. when a word which _should_ be on that list is missing, proofers have to struggle through the false flagging of that word, and that decreases the efficiency of flagging. for instance, when the name of a character in the book is flagged every time it appears, it can cause "flag fatigue", by virtue of its frequent nature. moreover, it can cause you to miss the cases where the name was misrecognized, because you've grown accustomed to skipping that flag... (if the name is in the good-words list, only an _incorrect_ version of the name gets flagged, which is what we want.) and once your good-words list is _complete_, you can do spellcheck on the book and have it come out _clean_. this is extremely valuable, because it means that you can repeat that spellcheck after any major editing operation (or at any milestone that you decide during the workflow) to make sure that your processing didn't introduce errors. so it's in everyone's best interest to have a good-words list which actually contains all "good words" in the book. which is why rfrank's simple-and-immediate procedure is far superior to the d.p. way, which is hard and slow... but there are methods even better than rfrank's... one of the most useful tools in my arsenal is one that takes text as its input -- up to the entirety of a book -- and quickly spits out a list of words not in its dictionary. thus it gives me a list of words that i'll need to check... but _many_ of these words, primarily _names_, but also words that appear a relatively large number of times, are ones that will go onto the good-words list for the book. so this tool can be used in _preprocessing_ to create a good-words list that is actually compellingly complete. but let's say you did not use this tool in preprocessing, and your good-words list is still missing lots of words... now let's look at a case where a proofer has just finished a page that had a number of words flagged on it, because they were not in the dictionary or on the good-words list. even if the proofer didn't take the time to add the flagged words to the good-words list, should they be auto-added? because, if they're ok on this page, they should be added! in other words, why make the proofer go to _any_ trouble to add a word to the good-words list? just analyze the page they've saved, as "good", finding all words that are not on the good-words list, and adding them automatically? as far as i can see, there are 2 problems that might result. the first is that the proofer made an error, and failed to catch a flagged word that was incorrect. in such a case, that would mean that any other occurrences of that word would not be flagged. that's unfortunate, of course, but is it a great tragedy? i think not. proofers need to know that an unflagged word _might_ be incorrect, n'est pas? of course they do. that's the essence of a stealth scanno. the second problem is that the word might be correct on _this_ page, but is _incorrect_ if it appeared elsewhere, and would need to be flagged there. again, i do not take a failure to flag every bad word as being necessarily bad. but directly to the point of this second possible problem, i simply don't think this situation occurs all that often... so i would like to issue a challenge, to the people who look at more books-in-progress than i do, to _locate_ this situation, where a non-dictionary word is _correct_ on _one_ page, yet _incorrect_ on some _other_ page... after all, this is the raison d'etre for the d.p. procedure, which insists that a word nominated for the good-words list must be inspected/approved by the project manager. so -- if this situation is more common than i believe -- project managers should have _lots_ of examples for me. so let's hear them. anyway, that's the suggestion, that when a page is proofed, all the words which had been flagged are _automatically_ added to the good-words list. and yes, i will also add that i believe such additions should be _screened_ by someone, but that's in keeping with my overall plan that any and all changes that are made will be doublechecked by someone. -bowerbird p.s. if you were quick enough to grok a flip-side suggestion that any word on the good-words list which was _changed_ on any page should automatically be _removed_ from the good-words list, give yourself a gold star for a sharp mind.

1 0

Re: a question for dkretz about twisted
by Bowerbird＠aol.com 12 Mar '10

12 Mar '10

gardner said: > an AIR application does not translate easily into > a "just a browser" server-hosted application. so, gardner, i think you're telling me that "it can't be done", in regard to running "twister" in a browser. if i'm mistaken, and it can be done -- i.e., _you_ think that _you_ can do it -- do please let me know... thanks. -bowerbird

1 0

a question for dkretz about twisted
by Bowerbird＠aol.com 11 Mar '10

11 Mar '10

ok, so i'm coding a proofing system. and wow, i'm impressed with myself and how far i've gotten in just 2 days. i've got a solid engine going already... in programming, the saying goes that the first 90% of a project takes 90% of the time, and the remaining 10% takes the other 90% of the time. and it's true. but still, to have a solid engine after just 2 days means i think i can have a pretty smooth system in 2 weeks... but before i go reinvent the wheel... a question for dkretz on "twisted"... it was coded in "air", so in _theory_ anyway, it will run on a web-server. so, don, can you make it do that? is there any place where it _is_ running on a web-server now? if someone (like me) wanted to run it on their server, would you make the app available to them? i'd love to see it run in a browser. -bowerbird

3 3

Re: a question for dkretz about twisted
by Morasch＠aol.com 11 Mar '10

11 Mar '10

don said: > Close, but I don't think close enough. ok, cool. thank you. just thought i'd ask. i'd be interested in playing with it, though, if you've got it up and available somewhere, or if i can install it on my own site, just to see exactly how "close" it comes... > I appreciate what Steve Jobs said recently i've hated adobe for a long, long, long time... but yeah, at one time, i did respect their work. now it just seems shoddy. bloated and shoddy. > What I'd recommend you consider is > building on the WordPress blog engine. sounds like too much overhead cruft to me... i like to be close to the metal. > Another reasonable alternative > might te Google Docs/Google Apps. sounds like more cruft. i'll stick with perl... thanks again. -bowerbird

2 1

the comparison method does indeed work, and work well
by Bowerbird＠aol.com 09 Mar '10

09 Mar '10

a little while back, i pointed out that a person can compare two independent digitizations to find errors in both of them, and that this method works very well. carel said: > That depends on a lot of factors including the assumption > that two OCR programs would not make the same mistake that's a good point. if the two digitizations have errors in common, then the comparison method won't be able to find them, and thus its effectiveness will be lessened somewhat. there's no argument with that. what's surprising to me, however, is how many people are completely defeated by this _possible_ shortcoming. upon learning that there _might_ be a problem with the comparison method, they dismiss it with no other thought. not me. i set out to actually _test_ the assumption. i documented the results in a thread in the d.p. forums. you can search for "revolutionary o.c.r. proofing". it's at: > http://www.pgdp.net/phpBB2/viewtopic.php?t=24008 as i note there, i presented the data earlier elsewhere, at: > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2005& post=2005-10-03,3 so yeah, that's right, my findings are over 4 years old now. *** what my research found is that there were virtually _no_ errors-found-in-common between the two digitizations. and this finding was replicated, and replicated once again. in other words, the effectiveness of the comparison method is _not_ lessened by this possible shortcoming. no indeed, the evidence says it is not even affected in the slightest way. the clarity of the results was striking; they are unforgettable. if you doubt the data, i encourage you to repeat the research. because repeating the possible problem, without any data, won't get anyone very far in the future, not if i'm listening... *** here's a quick-and-dirty experiment, for anyone willing... i just used the comparison method on gardner's e-text, and found 159 differences between his work and mine... i then resolved the differences by consulting the scans... 79 differences were due to errors in his work. 77 were due to errors in mine. 3 were due to errors in _both_ his and mine. now, of course, any errors-in-common will still reside in both his and mine. why don't you see if you can find any? > http://z-m-l.com/go/gardn/gardn.zml > http://z-m-l.com/go/gardn/gardnp123.html i'll be waiting. but i won't be holding my breath... *** carel said: > I feel that a human looking at > a smaller subset of a large document > is a good thing in the error finding process. > You apparently do not think it is. if the comparison method has already found all the errors, why waste the time and energy of a human rechecking that? > Neither of us is right or wrong: > It is a matter of perspective and opinion. unless i get a good answer to the question that i just asked, my opinion will continue to be that i am _absolutely_right_, and you're wrong because you're wasting human resources. that's my perspective, and i'm not changing it... :+) -bowerbird

1 0

roundlessness -- 013
by Bowerbird＠aol.com 09 Mar '10

09 Mar '10

ok, i talked about the positive parts of rfrank's roundless experiment. now it's time to review the "bad news" -- the not-so-positive parts... many of the _negative_ parts are quibbles about the implementation of the positive aspects, so i will discuss them at the end of this post... there are, however, a few points that are almost fully negative, still. *** the very first thing i do with a set of files that i start working with is to make sure that they're named _correctly_and_intelligently_... that is, a filename must explain, all by itself, the file's _contents_, and the filename must contain the pagenumber from the p-book. moreover, every file must have a _unique_ filename, and every file associated with the same page should share a similar name (e.g., the name will be the same, but with a different extension.) i've done extensive work with datasets that follow these rules and with datasets that do not, and i can say without any hesitation that the datasets which do not follow these rules are much more clumsy, and waste bits of my time that are small but cumulate to significance. and that's why i now no longer will even work with badly-named files. it's just unnecessary frustration. people who use badly-named files will tell you they have "adapted" to the naming. that's pure and simple crap. they don't know better because they haven't worked long and hard with both kinds of data. they are handicapped, and they just don't know they are handicapped. rfrank names his files incorrectly. maybe someday he will learn... *** rfrank does a lot of things right. his scans are extremely well-done, which indicates that he is very careful and meticulous when scanning. it's quite likely he also does some refinements on the scans, such as straightening them, centering them, and perhaps despeckling them. they look quite nice, and they are generally a pleasure to work with... however, all this care seems to be dropped once he's done the o.c.r. his preprocessing routines used to be abysmal. they're better now, but they still have considerable room for improvement. i'm hopeful that he's learned the lesson. he has pushed many of his checks back, from postprocessing to the proofing stage. so now he just needs to push them back from the proofing stage to the preprocessing stage. it would perhaps be very helpful in this regard if _somebody_ who is working at the fadedpage.com site would _volunteer_ to do the step of nondistributed preprocessing, thus freeing rfrank from doing it... he's probably feeling very overwhelmed at the moment, so an offer like that would probably be something that he would accept readily, and it would make a remarkable difference in evolving his progress. *** we've already discussed recently that rfrank should submit his scans along with his postings to p.g. alas, he's picked up the bad habit of failing to do that from his distributed proofreader upbringing. pity. *** it would also be good if rfrank kept the linebreaks and pagebreaks of the original p-book when he submitted the book to project gutenberg. but hey, that's unlikely, isn't it? what _is_ more likely, however, is that he would keep the linebreaks _consistent_ between the various versions that he submits to p.g. but on one file i checked, the 7-bit version was wrapped differently than the 8-bit version, which was wrapped differently than the .html. this is madness, if/when it comes to doing long-term version-control. *** rfrank also picked up the bad habit from d.p. of "clothing" em-dashes. of all the stupid things d.p. does, this is among the most stupid of all. and yet rfrank, who showed the ability to rethink proofing/formatting and roundlessness per se, failed to grasp the basic stupidity of this... *** ditto with unhyphenating the end-of-line hyphenates. i take it all back about "clothing hyphens" being the most stupid thing... dehyphenating has to be the _most_ stupid, because when the proofers do this, they actually destroy the evidence that a computerized routine would use to do the job _properly_, which is _on_a_book-wide_basis_... again, the failure of rfrank to rethink such an obvious stupidity is sad... (kudos, however, to one of his members, for spelling it out in a post. let's just hope that that reasoning will soak in to rfrank's busy brain.) *** again, repeating a d.p. flaw, rfrank strips runheads and pagenumbers from his o.c.r. and perhaps fate is trying to teach him a lesson on this, because he has had several problems where text on a page was deleted, or replaced with text from some other page. these types of problems can be detected and prevented when each page contains its pagenumber. in general, you want to _retain_ this information because it "earmarks" each page of text, making it clear what book it comes from, and where. it also serves as the "suspenders" in a "belt and suspenders approach" along with the filename, which will contain the very same information, and thus the two make it very easy to crosscheck and confirm each other. the silliness of naming all your scansets "001.png" through "999.png" and expecting their subdirectory name to distinguish them is _stark_... (and it has caused all kinds of grief for people in the past, i assure you.) *** rfrank hasn't really installed any instructions of his own, just letting his members rely on their d.p. training, so he has no policy of his own on ellipses, at least that i've been able to detect. but it would be refreshing if he decided to avoid the merry-go-round of never-ending changes that sometimes happens at d.p., and went _exclusively_ with the 3-dot ellipse. (it's funny, because many of his books don't even seem to _have_ ellipses!) *** rfrank is putting a lot of stock in "c.i.p." except, to confuse _everyone_, to him, "c.i.p." means "confidence in proofer", not "confidence in page", which is how everyone else defined the term, up to this point in time... now, me?, i don't think you can put much stock in "confidence in proofer". even the best proofers miss errors, and they don't know when they miss, so i don't think that you can trust their judgment and get perfect pages. rfrank's big mistake here is that he's not necessarily looking for "perfect", since he sees himself, as the postprocessor, as the last line of correction, and he's willing to take a non-perfect page if he can get it a little faster... even if that's fine for him, i don't think it's a good way to build a system. but even then, i just don't think "confidence in proofer" will actually work. or, to be more accurate, i think it'll work just well enough that rfrank will put lots of energy into it before he finds out it doesn't work well enough. or, worst case scenario, he'll convince himself that it really _is_ working, and other people believe him, and we all end up with non-perfect pages. on the other hand, rfrank has shown in some cases that he _can_ learn from the data, and change his mind on something he held dearly, so... *** ok, now we're down to the implementation quibbles... *** first, i'll repeat that it's sad that rfrank is "archiving" his finished projects. it would help all of us learn more about roundlessness if he left them up. i offer webspace if rfrank needs it. and project gutenberg has offered too. *** it's still "in-process", so i expect that it might improve, but the spellcheck display that rfrank is offering would benefit by retaining the linebreaks, so the search for "unresolved words" on the pagescan was much easier to do. i'd also like to see each unresolved word in _clickable-button_ form, for both the good-word and bad-word lists, so a button-push would do that. (in the current form, a person has to copy-and-paste each of the words.) i must add, though, that the ability to include these words immediately is a _tremendous_improvement_ over the d.p. method, one that shows its value to the proofer right away, and is thus very robust and valuable. empowering the proofers to benefit themselves is a remarkable asset... *** in this regard, an ability for a _proofer_ to execute a global change would be a mind-blowing step, and thus a very brave thing to try... of course, bear in mind that i believe that all global changes should have every occurrence verified, so take the suggestion appropriately. and i believe that any global changes that might be required _should_ be sussed out during the preprocessing, before proofers even see text. but nonetheless, putting such a powerful tool in the hands of proofers would speak _volumes_ on the responsibilities you entrust them with, and thus make a tremendous statement that would _embolden_ them... even if they never ever used it... *** as it is now, though, rfrank does the global changes himself, and he has been a little reluctant to do the job in the way that he really "should"... at least he was in one case -- where he declined to fix a contraction -- but perhaps that was not representative of his feelings more generally, so i'll let it go for now... *** as i said before, i don't know whether the d.p. separation between proofing and formatting is a good thing or not. i see the arguments in favor of it, and they seem compelling to some degree, but i also know that the vast majority of pages have little or no formatting, so i'm reluctant to lay another step on the overall process for no benefit. so, in cases like this where the answer is unclear, i'd do an experiment. luckily, rfrank is doing an experiment. it's not a well-controlled one, and we're not really privy to all of the data, so it's far from being ideal, but at least we're engaged in the active questioning of an unknown... still, it would be nicer if we were doing the experiment _properly_... *** rfrank does a pretty good job of showing proofers their diffs, _except_ that you must visit each project-page to see your diffs for that book... it would be far better if you were presented with all of 'em on one page. (and i would emphasize that page by presenting it to the user _first_, when they return for more proofing, so they'll realize its importance.) there's also the slightly troubling aspect that if you mark a page "done", the odds are lowered that it will be proofed again, so you don't obtain the satisfaction of getting a "no-diff" result on that page. i do believe that's counterproductive, and i'd like to see every page reproofed once, even after the page was marked "done", even by a high-c.i.p. proofer... and, of course, having the page reproofed, and having the "done" status confirmed by a "no-diff" by the subsequent proofer, would also raise the "confidence-in-page" for that page, and thereby serve a double benefit... (conversely, if the next proofer finds an error, they rescued a false done.) *** the "page tweet" idea is a good one. (the astute observer might realize that this is the same idea i always use on the bottom of my web-pages, where a person can leave a comment about that specific p-book page.) however, a way to _consolidate_ the tweets for a book would be useful. (and easy to code.) that way, a proofer could look at all of the "tweets" and perhaps answer some of the questions being posed, or fix some of the problems being reported, or take some other kind of positive action (such as finding a person who _can_ fix the problem if you cannot do it). also, it would be good if there were some dedicated buttons on the page, so it would be easy to say things like "difficult formatting, please check" or "foreign language specialist needed on this page", or stuff like that... again, that way a person perusing all of the tweets for a book will know exactly what needs to be done among this list of possible specific tasks, *** and -- just to finish up this post by taking it back to the beginning -- i note with amusement that rfrank uses the term "page" throughout his system. he lists the "pages" that you've done, and calls the notes you attach "page tweets", and the diffs are listed by "page", and so on. so it is ironic that when he talks about "page 123", he's not _really_ talking about _page_123_ at all! he's really talking about _.png_ 123! and the file named "123.png" probably isn't about page 123 at all! indeed, let's review the 7 files named "123.png" rfrank has up now: > http://fadedpage.com/p/201002140505/d/123.png > http://fadedpage.com/p/201002140533/d/123.png > http://fadedpage.com/p/201002270757/d/123.png > http://fadedpage.com/p/201002280257/d/123.png > http://fadedpage.com/p/201003020840/d/123.png > http://fadedpage.com/p/201003040537/d/123.png > http://fadedpage.com/p/201003070309/d/123.png what we actually find are pages 120, 112, 118, 106, 82, 124!, and 118, respectively. that's quite a range of pages, but alas, none are page 123. so any reference to a "page" number on the faded.com website is gonna frustrate anyone who wants to know what _page_ was being talked about, once rfrank has gone and deleted all of those files. which is a real pity... but alas, here i am talking about filenaming conventions again. help me! time to draw this to a close... *** while i'm letting myself discuss the negatives without feeling any guilt, i might add that it'd be nice if rfrank shared data from his experiments. of particular interest are all the intermediate files, such as the various pages as saved by individuals, and the concatenated text file at various "checkpoints" along the way, notably before and after postprocessing... without such data, we really have no way of evaluating the experiment! rfrank comes from a world of engineers working in private companies, where data is closely guarded, and he doesn't seem to have the attitude that is prevalent in the scientific world that data belongs to the public, and that sunshine is the best disinfectant, and open data is a positive... in this regard, i read and liked this article: > http://flowingdata.com/2010/03/04/think-like-a-statistician-without-the-mat… i sure could learn a lot more if rfrank were open with sharing his data, and my guess is that lots of other people could learn lots more as well. -bowerbird

1 0

roundlessness -- 012
by Bowerbird＠aol.com 08 Mar '10

08 Mar '10

rfrank's roundless experiment is proving to be _very_ interesting... and, as you might expect, there is good news and there is bad news. let's talk about the good news here in post #12, the bad in post #13. *** first of all, rfrank is showing that it's not all that difficult to set up a proofing site. in just 2 months, he's put together a critical mass, and that's quite an achievement. if he shares his code with others, they'll be able to move even faster. (if he doesn't, i've got a little code that'll do the trick for people who want a bit of a head-start.) it's another matter to pull workers to the site, of course. however, if project gutenberg chose to steer people to these _other_ sites, instead of funneling all the volunteers to distributed proofreaders (who -- truth be told -- don't even _want_ new people nowadays), it wouldn't be hard at all for these sites to get enough volunteers. but even with his low numbers of volunteers, what rfrank is doing is _head_and_shoulders_ more interesting than anything d.p. is doing. his site is dynamic, while d.p. has been too moribund for too long... *** in the last week, rfrank installed a spellcheck capability to his site. after a mere 2 months. d.p. went about 5 or 6 _years_ without it. *** moreover, when d.p. finally got a programmer to code spellcheck, the process was plagued by a forum discussion that ran 30 pages. at 15 messages per page, that's 450 messages. and most of 'em were from people who didn't know what they were talking about, and thus just added a buncha noise and confusion to the process. which is why it's probably not surprising that it was coded wrong. well, "wrong" is perhaps a bit strong. but the decision was made to do spellcheck using "aspell", because "it's open-source code". which would be fine, if you needed a full-fledged spellcheck... but that's not what a proofing site needs, because the object is _not_ to have another word "suggested" (which is the hard part about coding spellcheck), but merely to _flag_suspicious_words_ (a ridiculously easy task consisting of searching a dictionary to ascertain whether the word you're checking is included therein) so that all the suspicious words can be compared to the scan... i'm guessing rfrank did his spellcheck the simple way. *** rfrank also installed a capacity for a "good" and "bad" wordlist, necessary since that customizes the dictionary for each book, and -- like d.p. -- lets the proofers suggest words to include. unlike d.p., however, under rfrank's system, whenever a person "suggests" a word, it's _automatically_ included _immediately_. at d.p., a suggestion must be considered by a superior, who might or might not agree, and might or might not be timely. this is a signal of the disgust with which d.p. treats proofers. it also means that rfrank's system throws far fewer false flags, which means it provides much greater value to the proofers... i worked very hard, in the confines of that 30-page thread, to have d.p. give the proofers an automatic capability to add words to the good and bad lists, but they just wouldn't do it. rfrank did. good for rfrank. he's smarter than the d.p. crowd. *** rfrank has also included reg-ex checks, and scanno checks, so his list of helpful tools is already very impressive, 2 months in. *** rfrank has also shown he's willing to do global changes to text, which is one of those things that d.p. has been unwilling to do, in spite of the fact that i've pointed out the utility of it for years. d.p. would rather have individual proofers correct every instance of a global error -- one by one by one, painstakingly -- instead of fixing 'em all immediately, with one global change. shameful. *** rfrank also showed considerable independence when he decided he would have his people do proofing and formatting together... it's unclear to me whether the d.p. split between those two tasks is effective or not, but the _religion_ at d.p. is that it has been... so it is quite courageous of rfrank to test that accepted "wisdom". *** rfrank also seems committed to using diffs to train up volunteers. this, of course, is one of the benefits a roundless system offers, so it's natural that he'd take advantage, but it's still a good thing. *** rfrank has given workers a way to make comments _about_ a page without actually putting them _inside_ the text, which is fantastic. (he calls this feature "page tweets".) at d.p., they have a "project thread" in the forums (as does rfrank), but the only way to make comments about a page is to put them _inside_ the text. but of course then someone later down the line has to _remove_ them from there. that's a sign of a bad workflow, when someone later on must undo something that was done earlier. *** in looking at some of the projects, it seems that rfrank has finally started doing more aggressive preprocessing of the o.c.r. itself... for instance, the number of spacey quotes has dropped remarkably. there are still some, but nowhere near the number he had before... since this is an area that i know to be _so_ important, any progress toward enlightenment at all is the sign of a very good development. *** so, all in all, there's lots of positive aspects to rfrank's experiment. -bowerbird

1 0