- gutvol-d - lists.pglaf.org

let's step through this process.008
by Bowerbird＠aol.com 02 Oct '12

02 Oct '12

welcome back to the work-week. so let's get back to work! :+) or is this fun? :+) are we having fun yet? we should be having fun. *** today i'm comparing against a "clean" text. of course, we never really know if one text is totally "clean" or not. if we did, we could just use _that_ text, and throw away the rest. but we _can_ say that two texts are the same, and as far as we know, neither has any errors. or, alternatively, we can say they're different, and list the differences. we might also then say that one version conforms to edition "x" of the book, and the other matches edition "y". but we always have to leave ourselves open to the possibility that errors remain in the text, and that someone might eventually spot them, and hopefully tell us about 'em, so we fix 'em. to that end, we can make it _easy_ for people to do that text/scan comparison, so they _will_ be more motivated to do it, and locate errors. but let's not get too far ahead of ourselves... for now, i've just done the comparison, and i must say it went rather well. we've located and fixed lots of little hard-to-locate bugs... as you'd expect, the new version is located here: > http://zenmagiclove.com/prhpr/prhpr-008.zml and of course the derivatives from that master: > http://zenmagiclove.com/prhpr/docomp78.py > http://zenmagiclove.com/prhpr/prhpr-008h.html > http://zenmagiclove.com/prhpr/prhprp222.html now we're rounded the final corner for the stretch... -bowerbird

1 0

Re: [gutvol-d] blah blah blah blah blah
by don kretz 01 Oct '12

01 Oct '12

Further to Roger's post- Instead of DP resources, Jon/PG may be willing to direct other volunteer resources our way. As for product priority, perhaps something equivalent to the PG self-publishing process could be involved - call it Gutenberg Gold Edition. Don From: Roger Frank Sent: 9/26/2012 4:41 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] blah blah blah blah blah Jon, I feel overwhelmed by the RTT discussion at this point. I wonder if this paragraph summarizes the basics of what has been proposed: The main task is that PG would host a master scan particular to one preferred edition. Along with that master scan would be a text-based version, the RTT, marked up in some currently undefined way, that represents a starting point for anyone who wants to produce a book in any media using any further tools or markup they choose. A related task is that DP would be involved in making the RTT. Also, tools would be developed to take the RTT to various output formats. If that's the heart of it, then at least I understand the goal. If I have it wrong, please correct me. I'm still unclear on the RTT and how it can hold all the information of the original such as italics, superscripts, etc. but this seems less important than the logistics questions: 1. Some have warned that there are at least two miracles in the critical path for this project: one is buy-in from DP/Louise that DP would generate the RTT. There is a sense that DP would not join this work, which is why there is a discussion, I think, of other proofing tools that presumably would be hosted elsewhere and used by other volunteers. 2. The other unlikely critical event is buy-in from PG/Marcello to host/feature the RTT and well-formatted final versions based on that RTT. There is a sense that PG would not join this work, which is why there is a discussion, I think, of having the final versions, derived from the RTT, hosted on other sites. 3. Assuming PG would at least host the RTT, I think the plan included the WWers only having to approve the RTT as it came to them. Are they prepared to do that and relinquish control of the final formatted versions? Is this what they want? Have they been asked? It seems to me that the likely result, if 1-3 were to be taken to their probable conclusions, would be not using DP to do the work and a not using PG to host the RTT-derived final versions. Is that the direction this proposal is going? --Roger _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

4 6

let's step through this process.007
by Bowerbird＠aol.com 30 Sep '12

30 Sep '12

ok, so we had water: > http://zenmagiclove.com/prhpr/water.py and then we had wine: > http://zenmagiclove.com/prhpr/wine.py and now we have whiskey: > http://zenmagiclove.com/prhpr/whiskey.py as you can see, we've now cleaned up all of the _red_ words -- the ones not in the dictionary -- so now our text is looking extremely clean, it is. so you know, the new text is located here: > http://zenmagiclove.com/prhpr/prhpr-007.zml and of course the derivatives from that master: > http://zenmagiclove.com/prhpr/docomp67.py > http://zenmagiclove.com/prhpr/prhpr-007h.html > http://zenmagiclove.com/prhpr/prhprp222.html but don't be fooled by all that non-red text there. because it just means we fixed the obvious errors. there might still be some stealth scannos lurking. and, without question, we have a lot of errors in punctuation. we can run tests to find a lot of 'em, but there will still be some that remain impervious. fortunately, in addition to those tests we can run, we still have two more solid lines of potent power. the first is a comparison with another digitization. i've done research, and documented it with proof in the form of reports (both here and over at d.p.) which give us firm evidence that this is worthwhile. you can't look at my data and deny the value of this. more to the point, if you've got other digitizations, why not use 'em? yes, it takes some effort to set up a comparison, but the right tools help significantly. and _results_ from a comparison are very gratifying. errors that are surfaced by the diffs are _often_ the type that is difficult to find otherwise, true stinkers. so you'll feel the value of this method in your bones. and of course our final "line of defense" are readers. first, "smoothreaders" or "continuous proofers" or "beta-readers" or whatever you want to call them, are valuable because they take first crack at stuff... but also worth mentioning are the final customers, the end-users, who -- if properly encouraged -- can be a tremendous resource for the error-reports that move our e-books steadily toward _perfection_. -bowerbird p.s. for those keeping track of the version changes: there were 7,607 lines changed from "002" to "003". and there were 515 lines edited from "003" to "004". and 275 lines were revised between "004" and "005". and 258 more lines were revised from "005" to "006". and now, 327 lines were changed from "006" to "007".

1 0

let's step through this process.006
by Bowerbird＠aol.com 29 Sep '12

29 Sep '12

ok, like i said, our text is getting relatively clean. as a reminder, here's how it presently stacks up: > http://zenmagiclove.com/prhpr/prhpr-006.zml we're gonna stick with that version for right now, because i'm gonna show you a spell-checker thing. should be pretty self-explanatory, so here it is: > http://zenmagiclove.com/prhpr/water.py that display works nicely on my cinema-monitor. sorry if it doesn't work for you; you might need to resize your browser-window, or zoom the display. or maybe just go buy a big monitor; they're cheap, and you'll love the real estate, i can guarantee it... as you'll see, words not found in the dictionary are displayed in red, and marked with a "*****". (step through them nicely by find-nexting that.) in addition, whenever there's a highlighted word on a page, that _page-scan_ also gets displayed, so you can easily check on the "error" right there. *** considering that it was raw o.c.r. not long ago, there is a lot of black (i.e., clean) text now, plus it's also clear that lots of the red words are ones which are correct, but idiosyncratic to this book; most noticeable are the names of the characters. so the next step is to add 'em to the dictionary, so they won't be flagged. i _could_ ruminate on the process of how best to accomplish this task, but i don't care to take that side-trip right now, because none of you seem to care that much, so i will just cut to the chase and do it how i do it... for your information, though, the "water" script (up above) displays an _alphabetized_list_ of the colorized words; you will find it at the bottom... such a list might give you ideas on how you can repurpose it for use as your auxiliary dictionary. behold, once i'd made a book-specific dictionary, our situation becomes remarkably more focused: > http://zenmagiclove.com/prhpr/wine.py now _this_ is a useful display. let's correct errors! -bowerbird

1 0

let's step through this process.005
by Bowerbird＠aol.com 28 Sep '12

28 Sep '12

ok, so i am in the process of taking a book from o.c.r. to finished product in a few days, all by my lonesome, and documenting it, and none of you has a single word to say about it? why does this not surprise me? you'll jabber back and forth all day and night if you think it's all "a matter of opinion", but once you are confronted with cold hard facts, you clam up just like... well, just like a clam... no wonder i don't take most of you seriously. have a nice weekend. :+) hey, maybe i'll see you at the beach! ;+) -bowerbird p.s. oh and, if my pace is too leisurely for you, i did a book a few years back in just _one_hour_, and documented that, in a series of posts to the "bookpeople" listserve that you'll still find online. in my old age, i've decided to slow down a little, to smell the roses... and make fun of the clams...

1 0

Double blind OCRing?
by Robert Gibbins 28 Sep '12

28 Sep '12

Somewhere in the recent spate of discussion, several contributors, including BB and/or Jim Adcock I think, described diffing two different (OCRed) editions of the same work, and also finding many scannos. These appends seem to me to imply that diffing parallel/independently produced texts is a faster and more efficient way to correct the OCR process than sequential manual checking. I wonder what happens if, starting from the same physical text, you do one or more of: - scanning with two different scanners - OCRing with two different algorithms/programs - manual correction (PPing?) independently by two different people And then diffing the results. Has anyone (possibly including DP) ever tried any of the above, and documented the results in a scientifically valid way? Does DP work that way anyway? It sort of makes sense to me that if the above processes are basically 95%-99% accurate, then automatically comparing two independent results to find errors might be a lot more reliable than manually refining something that's already so good that humans can't see the difference. It's not quite the same, but I've spent enough of my life trying to see the errors in program code, i.e. read what the code actually says, rather than what I think it says, to know that humans have amazingly good subconscious error correction algorithms which it's impossible to turn off. Bob Gibbins

5 5

let's step through this process.004
by Bowerbird＠aol.com 28 Sep '12

28 Sep '12

extra update for you, as a special friday surprise! *** "prhpr-006.zml" is up: > http://zenmagiclove.com/prhpr/prhpr-006.zml there were 7,607 lines changed from "002" to "003". and there were 515 lines edited from "003" to "004". and 275 lines were revised between "004" and "005". and 258 more lines were revised from "005" to "006". as always, you can view lines that were changed, via: > http://zenmagiclove.com/prhpr/docomp56.py and of course the .html file, converted from .zml: > http://zenmagiclove.com/prhpr/prhpr-006h.html adjust your zoom and window-size for best display of all of the page-by-page files, now newly updated: > http://zenmagiclove.com/prhpr/prhprp222.html *** especially since i haven't run a full spellcheck yet, the o.c.r. has now become quite clean, relatively, which might surprise you if you had never been paying attention to all of my previous research. if you _have_ listened, then you already _know_ that most o.c.r. can be cleaned this quickly and easily by employing mechanized means to find and fix errors. and doing a full spellcheck will take it up a notch. after that, i will then compare this text to a known "clean" version (or, really, even a not-so-clean one), which will take this text close to proofed perfection. (formatting is still another beast that must be tamed, but knowing you have the words correct is a boost.) we have several clean texts for "pride and prejudice", so we're in luck, because we can do many comparisons. (we really only need one, but let's do two, just for fun.) i might not get to those comparisons until next week -- busy weekend ahead -- so if you would like to do it, feel free to go ahead and give it a shot. catch you later! have a good weekend! :+) -bowerbird

1 0

Re: [gutvol-d] blah blah blah blah blah
by don kretz 28 Sep '12

28 Sep '12

One helpful step would be to assist the user to make a more informed decision. It should be possible to request the most recent eBook, or eBooks that have been submitted with a tuned version for a given platform, rather than one synthesized from another format. Since so many copies are acquired from secondary sources, it would be interesting to explore whether any of them would be interested in being directed to improved versions as they are posted, in a format congenial to their publication process. Don Sent from my Phone From: James Adcock Sent: 9/28/2012 9:49 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] blah blah blah blah blah >Why should PG nominate one particular instance of a text as THE canonical version? PG has no particular expertise in textual criticism or literary studies. Whether PG should or shouldn't, brief usage of the PG website will demonstrate that they do. Not sure how you display results on a web page without "nominating" one over the other? Randomize? _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

1 0

Readable epubs
by Robert Gibbins 28 Sep '12

28 Sep '12

I wonder if I could drag the tone of this debate down a level or two to pragmatics. A few appends ago I may have used the phrase 'HTML master version' or something similar. This was never intended to imply that HTML could or should be used as a true master version from which any other format could be generated, merely that for a large group of basically simple PG works, with a certain amount of standardisation of markup it might be possible to generate acceptable text and epub files from a single HTML source, and thus that it might be possible to fairly easily improve the current rather unsatisfactory situation where the generated epubs make less than happy reading, without causing vast amounts of ongoing maintenance/update work by keeping multiple copies of the same work in step. This is essentially a piece of pragmatics, and is only relevant because currently PG has rather a lot of works which exist both in plain text and let's say unconstrained HTML formats and which, as someone else remarked and I have also experienced myself, it's not that difficult to re-work the HTML to generate much better epubs. The only reason I keep gently nagging away about this is that I am lazy and would like not to have to do this myself every time I want to read a PG book on one of my e-readers, and whilst adding them to MobileRead achieves a certain amount of sharing/access to reformatting done by others, it seems to me that if PG took this up a lot more sharing would be achieved. Just to repeat myself, I do not maintain that HTML is a better master format than TEI, RST, LateX, MathML, etc., merely that PG seems to me to have a lot more HTML than the other formats, and also that epub (aka zipped HTML) seems to be a more important format for the increasingly large group of people who want to read PG works on their e-readers. I don't expect The Powers That Be to experience a Damascene conversion to my way of thinking by reading this append, but having read recent arguments from others that images plus plain text is the way to standardise because there is no universally acceptable master format to standardise on, I thought I should make it clear that my advocacy of somewhat standardising the current large PG catalogue of HTML is not intended to imply anything general about the excellence or otherwise of HTML as a master document format. Bob Gibbins

2 1

Re: [gutvol-d] DP protocols
by don kretz 28 Sep '12

28 Sep '12

Jon/Roger This appears functionally equivalent to the effort undertaken by several of us 3 or 4 years ago to extract the text from several projects and post them to readingroo.ms as non-PG certified "preview" texts. Advance approval was given by Jon Noring on behalf of PG, and was required from each of the Project Managers. Louise was aware but functionally silent. The transfer mechanism was established and put into operation by volunteers using existing DP interfaces entirely and disrupting no existing DP operations or personnel. No one in the DP hierarchy showed any positive interest or volunteered assistance. As I recall PM participation was mixed; but this was advertised as being for a few willing participants, not the general run of projects. After a short period of operation several dozen projects were transferred. At that point Louise declared the whole operation illegal and demanded it be terminated and any existing project texts be removed without exception from readingroo.ms, which was done. The "rule" that was established thereby was that the only texts allowed to escape into the wild would be those which had undergone the entire DP process from P1 through PP/PPV. Except, as I said before, by going through Louise first. Sent from my Phone From: Roger Frank Sent: 9/27/2012 3:05 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] DP protocols On Sep 26, 2012, at 2:03 PM, Jon Hurst <jon.a(a)hursts.eclipse.co.uk> wrote: > Quick question to Don/Roger/Anyone who PM's at DP. > > Assume Greg has got us a scan that of P&P. What DP rule are we breaking > by skipping P3, F1, F2 and doing a simplified PP consisting of diffing > against extant text and converting to UTF-8? I don't think there is a "rule" for this at DP. If you want to attempt it, get DP management to buy off on it first and then be very clear in the project instructions that the users' work is not going to be taken to a finished book on the DP site. I believe some users would choose not to work on a project, especially with LOTE rules, that is only doing P1 and P2. That's not the DP workflow. If you were to get that done, I understand that the P2 output is the RTT in your scheme. That means that anyone who derives a final version (HTML, epub, kf8 etc. file) from it has to find and handle all the markup like intalics and superscripts on their own. It's not coded in the RTT if it's coming out of P2, since formatting is not present in the proofing stages, even inline formatting. That is a serious shortcoming to me. --Roger _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

2 1