September 2012 - gutvol-d - lists.pglaf.org

Re: [gutvol-d] blah blah blah blah blah
by don kretz 09 Oct '12

09 Oct '12

One wonders about how wwers distinguish between calling a submission a "correction" (and presumably altering the existing distribution), and calling it a "new version" and giving it a new number with zero distribution credibility. If it's derived from the old version with the intent to only make corrections, is there a threshold number of corrections beyond which it becomes a new version with all the attending discredit? If I also add markup in addition to corrections, does that make it a new book number, or a new format for the old one? Seems like the original contributor could be presumed to have implicitly authorized, indeed welcomed, corrected versions as replacements, regardless of the magnitude of the resulting change. Don From: Jon Hurst Sent: 9/25/2012 4:27 AM To: gutvol-d(a)lists.pglaf.org Subject: Re: [gutvol-d] blah blah blah blah blah On 2012-09-25, Bowerbird(a)aol.com wrote: > > I would also like to do a LaTeX version, so will > > need a stage before you add your formatting -- > > it doesn't have to be my RTT, > > but the lines need to correspond. > > i'm just mounting the projects. > > somebody else will have to clean the text. > > and add the formatting. > > i mean, i _might_ do some of that stuff, but > i'm not making any guarantees that i _will_... > > but i doubt i will. i have bigger fish to fry... > > also, i think it's silly not to add the formatting, > so you'll have to _remove_ any that i've added. > (but that would strike me as immensely stupid, > because then you'll just have to add it back in.) So here we have a demonstration of the exact set of problems that I was trying to solve. You don't want to clean the text, due to your big fish frying activities. I don't particularly want to spend my (admittedly fishless) limited time cleaning the text either. Pretty much everyone on this list doesn't want to clean text that is not for their own solo project. There are 1000 odd people who will be glad to clean text for you and know what they are doing, but they live at PG and if you want to play there, you have to play by their rules. Greg has a hose full of people that will clean text for you, but if he turns it your way, you will have to train them first, and in your case you will not just have to train them to check that the word in the text is the same as the word in the image, you will have to train them in all things ZML as well. I don't want to start from your ZML. I'm sure ZML is lovely, but I want to work from the corrected text and images where the number of introduced errors is lowest. That is my preferred workflow, and I'm not prepared to compromise on it. Consider it a religious thing, that I am going to be utterly unreasonable about, and that nothing you say will ever change my mind on it. A workflow can go from RTT to ZML very easily, but the reverse is not necessarily the case. That was the point of the RTT. If you mandate that what comes out of your system is ZML and only ZML, then I'm not interested in working within your system. Finally, unless I am reasonably convinced that something useful will come of it, I am not going to proofread word 1. Useful means default PG download and that, at the very least, means preferred PG edition, and the existence of some sort of default download protocol. Posting a small selection of books on your own site while laudable means pretty much nothing at all in the grand scheme of things. So... we agree on the necessity of master scans, we agree that a default download protocol is a requirement before it is worth commencing any work. Let's narrow our focus to these two seemingly simple items. With these in place, people can roll up their sleeves. Maybe some will do redos as solo projects, maybe some will do them through your system, maybe one of the DP's can help make some RTTs. Without them, it is simply not worth starting anything... which you may recall was what I was trying to determine right at the start of all this dialogue, and more or less the first piece of advice you gave me. Cheers Jon _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

12 34

Does anyone know what's going on with PG Canada and Faded Pages?
by James Simmons 04 Oct '12

04 Oct '12

I donated page scans from several books to DP Canada many months ago. A couple of them have been finished and the books are on fadedpage.com but NOT on the PG Canada site: http://www.fadedpage.com/showbook.php?pid=20120626 http://www.fadedpage.com/showbook.php?pid=20120707 What puzzles me is that new books have been added to PG Canada recently, but not these. PG Canada has EPUBs for the books, and fadedpage.com only has text and HTML. I ran the HTML through Kindlegen so I could read the books on my Kindle and found that both had page numbers that the Kindle didn't know what to do with. In the Benchley book the page numbers were often as not stuck inside words that may have been hyphenated in the original. Does anyone know what's going on here? If it was up to me all the books would be on PG Canada's site only. James Simmons

4 5

Re: [gutvol-d] blah blah blah blah blah
by don kretz 01 Oct '12

01 Oct '12

Further to Roger's post- Instead of DP resources, Jon/PG may be willing to direct other volunteer resources our way. As for product priority, perhaps something equivalent to the PG self-publishing process could be involved - call it Gutenberg Gold Edition. Don From: Roger Frank Sent: 9/26/2012 4:41 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] blah blah blah blah blah Jon, I feel overwhelmed by the RTT discussion at this point. I wonder if this paragraph summarizes the basics of what has been proposed: The main task is that PG would host a master scan particular to one preferred edition. Along with that master scan would be a text-based version, the RTT, marked up in some currently undefined way, that represents a starting point for anyone who wants to produce a book in any media using any further tools or markup they choose. A related task is that DP would be involved in making the RTT. Also, tools would be developed to take the RTT to various output formats. If that's the heart of it, then at least I understand the goal. If I have it wrong, please correct me. I'm still unclear on the RTT and how it can hold all the information of the original such as italics, superscripts, etc. but this seems less important than the logistics questions: 1. Some have warned that there are at least two miracles in the critical path for this project: one is buy-in from DP/Louise that DP would generate the RTT. There is a sense that DP would not join this work, which is why there is a discussion, I think, of other proofing tools that presumably would be hosted elsewhere and used by other volunteers. 2. The other unlikely critical event is buy-in from PG/Marcello to host/feature the RTT and well-formatted final versions based on that RTT. There is a sense that PG would not join this work, which is why there is a discussion, I think, of having the final versions, derived from the RTT, hosted on other sites. 3. Assuming PG would at least host the RTT, I think the plan included the WWers only having to approve the RTT as it came to them. Are they prepared to do that and relinquish control of the final formatted versions? Is this what they want? Have they been asked? It seems to me that the likely result, if 1-3 were to be taken to their probable conclusions, would be not using DP to do the work and a not using PG to host the RTT-derived final versions. Is that the direction this proposal is going? --Roger _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

4 6

let's step through this process.007
by Bowerbird＠aol.com 30 Sep '12

30 Sep '12

ok, so we had water: > http://zenmagiclove.com/prhpr/water.py and then we had wine: > http://zenmagiclove.com/prhpr/wine.py and now we have whiskey: > http://zenmagiclove.com/prhpr/whiskey.py as you can see, we've now cleaned up all of the _red_ words -- the ones not in the dictionary -- so now our text is looking extremely clean, it is. so you know, the new text is located here: > http://zenmagiclove.com/prhpr/prhpr-007.zml and of course the derivatives from that master: > http://zenmagiclove.com/prhpr/docomp67.py > http://zenmagiclove.com/prhpr/prhpr-007h.html > http://zenmagiclove.com/prhpr/prhprp222.html but don't be fooled by all that non-red text there. because it just means we fixed the obvious errors. there might still be some stealth scannos lurking. and, without question, we have a lot of errors in punctuation. we can run tests to find a lot of 'em, but there will still be some that remain impervious. fortunately, in addition to those tests we can run, we still have two more solid lines of potent power. the first is a comparison with another digitization. i've done research, and documented it with proof in the form of reports (both here and over at d.p.) which give us firm evidence that this is worthwhile. you can't look at my data and deny the value of this. more to the point, if you've got other digitizations, why not use 'em? yes, it takes some effort to set up a comparison, but the right tools help significantly. and _results_ from a comparison are very gratifying. errors that are surfaced by the diffs are _often_ the type that is difficult to find otherwise, true stinkers. so you'll feel the value of this method in your bones. and of course our final "line of defense" are readers. first, "smoothreaders" or "continuous proofers" or "beta-readers" or whatever you want to call them, are valuable because they take first crack at stuff... but also worth mentioning are the final customers, the end-users, who -- if properly encouraged -- can be a tremendous resource for the error-reports that move our e-books steadily toward _perfection_. -bowerbird p.s. for those keeping track of the version changes: there were 7,607 lines changed from "002" to "003". and there were 515 lines edited from "003" to "004". and 275 lines were revised between "004" and "005". and 258 more lines were revised from "005" to "006". and now, 327 lines were changed from "006" to "007".

1 0

let's step through this process.006
by Bowerbird＠aol.com 29 Sep '12

29 Sep '12

ok, like i said, our text is getting relatively clean. as a reminder, here's how it presently stacks up: > http://zenmagiclove.com/prhpr/prhpr-006.zml we're gonna stick with that version for right now, because i'm gonna show you a spell-checker thing. should be pretty self-explanatory, so here it is: > http://zenmagiclove.com/prhpr/water.py that display works nicely on my cinema-monitor. sorry if it doesn't work for you; you might need to resize your browser-window, or zoom the display. or maybe just go buy a big monitor; they're cheap, and you'll love the real estate, i can guarantee it... as you'll see, words not found in the dictionary are displayed in red, and marked with a "*****". (step through them nicely by find-nexting that.) in addition, whenever there's a highlighted word on a page, that _page-scan_ also gets displayed, so you can easily check on the "error" right there. *** considering that it was raw o.c.r. not long ago, there is a lot of black (i.e., clean) text now, plus it's also clear that lots of the red words are ones which are correct, but idiosyncratic to this book; most noticeable are the names of the characters. so the next step is to add 'em to the dictionary, so they won't be flagged. i _could_ ruminate on the process of how best to accomplish this task, but i don't care to take that side-trip right now, because none of you seem to care that much, so i will just cut to the chase and do it how i do it... for your information, though, the "water" script (up above) displays an _alphabetized_list_ of the colorized words; you will find it at the bottom... such a list might give you ideas on how you can repurpose it for use as your auxiliary dictionary. behold, once i'd made a book-specific dictionary, our situation becomes remarkably more focused: > http://zenmagiclove.com/prhpr/wine.py now _this_ is a useful display. let's correct errors! -bowerbird

1 0

let's step through this process.005
by Bowerbird＠aol.com 28 Sep '12

28 Sep '12

ok, so i am in the process of taking a book from o.c.r. to finished product in a few days, all by my lonesome, and documenting it, and none of you has a single word to say about it? why does this not surprise me? you'll jabber back and forth all day and night if you think it's all "a matter of opinion", but once you are confronted with cold hard facts, you clam up just like... well, just like a clam... no wonder i don't take most of you seriously. have a nice weekend. :+) hey, maybe i'll see you at the beach! ;+) -bowerbird p.s. oh and, if my pace is too leisurely for you, i did a book a few years back in just _one_hour_, and documented that, in a series of posts to the "bookpeople" listserve that you'll still find online. in my old age, i've decided to slow down a little, to smell the roses... and make fun of the clams...

1 0

Double blind OCRing?
by Robert Gibbins 28 Sep '12

28 Sep '12

Somewhere in the recent spate of discussion, several contributors, including BB and/or Jim Adcock I think, described diffing two different (OCRed) editions of the same work, and also finding many scannos. These appends seem to me to imply that diffing parallel/independently produced texts is a faster and more efficient way to correct the OCR process than sequential manual checking. I wonder what happens if, starting from the same physical text, you do one or more of: - scanning with two different scanners - OCRing with two different algorithms/programs - manual correction (PPing?) independently by two different people And then diffing the results. Has anyone (possibly including DP) ever tried any of the above, and documented the results in a scientifically valid way? Does DP work that way anyway? It sort of makes sense to me that if the above processes are basically 95%-99% accurate, then automatically comparing two independent results to find errors might be a lot more reliable than manually refining something that's already so good that humans can't see the difference. It's not quite the same, but I've spent enough of my life trying to see the errors in program code, i.e. read what the code actually says, rather than what I think it says, to know that humans have amazingly good subconscious error correction algorithms which it's impossible to turn off. Bob Gibbins

5 5

let's step through this process.004
by Bowerbird＠aol.com 28 Sep '12

28 Sep '12

extra update for you, as a special friday surprise! *** "prhpr-006.zml" is up: > http://zenmagiclove.com/prhpr/prhpr-006.zml there were 7,607 lines changed from "002" to "003". and there were 515 lines edited from "003" to "004". and 275 lines were revised between "004" and "005". and 258 more lines were revised from "005" to "006". as always, you can view lines that were changed, via: > http://zenmagiclove.com/prhpr/docomp56.py and of course the .html file, converted from .zml: > http://zenmagiclove.com/prhpr/prhpr-006h.html adjust your zoom and window-size for best display of all of the page-by-page files, now newly updated: > http://zenmagiclove.com/prhpr/prhprp222.html *** especially since i haven't run a full spellcheck yet, the o.c.r. has now become quite clean, relatively, which might surprise you if you had never been paying attention to all of my previous research. if you _have_ listened, then you already _know_ that most o.c.r. can be cleaned this quickly and easily by employing mechanized means to find and fix errors. and doing a full spellcheck will take it up a notch. after that, i will then compare this text to a known "clean" version (or, really, even a not-so-clean one), which will take this text close to proofed perfection. (formatting is still another beast that must be tamed, but knowing you have the words correct is a boost.) we have several clean texts for "pride and prejudice", so we're in luck, because we can do many comparisons. (we really only need one, but let's do two, just for fun.) i might not get to those comparisons until next week -- busy weekend ahead -- so if you would like to do it, feel free to go ahead and give it a shot. catch you later! have a good weekend! :+) -bowerbird

1 0

Re: [gutvol-d] blah blah blah blah blah
by don kretz 28 Sep '12

28 Sep '12

One helpful step would be to assist the user to make a more informed decision. It should be possible to request the most recent eBook, or eBooks that have been submitted with a tuned version for a given platform, rather than one synthesized from another format. Since so many copies are acquired from secondary sources, it would be interesting to explore whether any of them would be interested in being directed to improved versions as they are posted, in a format congenial to their publication process. Don Sent from my Phone From: James Adcock Sent: 9/28/2012 9:49 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] blah blah blah blah blah >Why should PG nominate one particular instance of a text as THE canonical version? PG has no particular expertise in textual criticism or literary studies. Whether PG should or shouldn't, brief usage of the PG website will demonstrate that they do. Not sure how you display results on a web page without "nominating" one over the other? Randomize? _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

1 0

Readable epubs
by Robert Gibbins 28 Sep '12

28 Sep '12

I wonder if I could drag the tone of this debate down a level or two to pragmatics. A few appends ago I may have used the phrase 'HTML master version' or something similar. This was never intended to imply that HTML could or should be used as a true master version from which any other format could be generated, merely that for a large group of basically simple PG works, with a certain amount of standardisation of markup it might be possible to generate acceptable text and epub files from a single HTML source, and thus that it might be possible to fairly easily improve the current rather unsatisfactory situation where the generated epubs make less than happy reading, without causing vast amounts of ongoing maintenance/update work by keeping multiple copies of the same work in step. This is essentially a piece of pragmatics, and is only relevant because currently PG has rather a lot of works which exist both in plain text and let's say unconstrained HTML formats and which, as someone else remarked and I have also experienced myself, it's not that difficult to re-work the HTML to generate much better epubs. The only reason I keep gently nagging away about this is that I am lazy and would like not to have to do this myself every time I want to read a PG book on one of my e-readers, and whilst adding them to MobileRead achieves a certain amount of sharing/access to reformatting done by others, it seems to me that if PG took this up a lot more sharing would be achieved. Just to repeat myself, I do not maintain that HTML is a better master format than TEI, RST, LateX, MathML, etc., merely that PG seems to me to have a lot more HTML than the other formats, and also that epub (aka zipped HTML) seems to be a more important format for the increasingly large group of people who want to read PG works on their e-readers. I don't expect The Powers That Be to experience a Damascene conversion to my way of thinking by reading this append, but having read recent arguments from others that images plus plain text is the way to standardise because there is no universally acceptable master format to standardise on, I thought I should make it clear that my advocacy of somewhat standardising the current large PG catalogue of HTML is not intended to imply anything general about the excellence or otherwise of HTML as a master document format. Bob Gibbins

2 1