May 2005 - gutvol-d - lists.pglaf.org

re: [gutvol-d] Heebee Jeebees on Gutenberg
by Bowerbird＠aol.com 13 May '05

13 May '05

jeff said: > I've been mulling over ideas alright! someone around here is thinking for once! :+) > for applying Natural Language Processing > to catch hard-to-find errors in e-texts. oh-oh. a lapse into acronym-land. not a good sign of quality thinking... ;+) > I have made little practical progress i suggest you ditch the acronym; those things just weigh you down... :+) > but for some reason it occurred to me to > try a few carefully chosen Google searches, > all restricted to site:www.gutenberg.org. good idea. so ok, let's take a look. > "around the comer" returns 17 hits. seems like that's a stealth scanno that should be in the dictionary already, yep. is jim tinsley taking these notes down as error-reports? > "turn the comer" returns no hits. i guess we turned the corner on that one... > "to he" returns 10,700 hits, > a fair number of them not representing typos. false alarms are counterproductive; that's the big problem with gutcheck. every false alarm wastes your time. however -- in contrast to misses, which are invisible, and therefore cannot be instructive -- false alarms _do_ give you information about how to improve your process... that is why you need to treasure them, and to study them. (yes, for the acronym-lovers among us, this is signal-detection theory in action. s.d.t. to you.) > "have clone", 13 hits. have clone, will reproduce. > "will bo", 1 hit. roger wilco. > "to bo", 38, some legit (often using "Bo" as a proper noun) again, you need to examine the false alarms closely, to see if they reveal clues how to make your search more fine-grained. here, for instance, the "b" in the false alarms is capitalized, when it otherwise would not be, indicating it's a proper noun, and gives a reason to weight those instances less heavily, or indeed, if happening repeatedly in one e-text, even ignore 'em. this is where most people go bad. they hit a few false alarms, think "oh no, this isn't working", and give up. that is so wrong! as long as the number of false alarms isn't overwhelming you, it's much more cost-effective to take the hits that you got and then go to work on improving performance on the false alarms... > "went borne" (for "went home"), 5 hits, one of which is legit > and the other being four different editions of the same work, > all with the same error. at least the _errors_ are consistent! ;+) but do you see a pattern emerging here? > "fax away", 1 hit. what should it be? although "fax" _could_ be in a scanno dictionary, for "far", i'd take it out, because very few public-domain books have the word "fax" in them, do they? ;+) so look at every instance of "fax", pure and simple. > "coining to", 23 hits, some legit. any clues as to how to tell the machine to tell the difference? start digging, and dig deep. > "he docs", 7, with some repeat editions. "docs" might be in a list of stealth scannos too. but again, let's be realistic. how often does it come up legitimately? and in the rare case it does come up legitimately... > "it docs", 9, with repeats, but offset by two hits in one work. ...it might come up in that book more than one time, right? and by now, you should definitely see a certain pattern... > "she docs", none. does she? i guess she doesn't... > I don't know what all that proves, > but I found it interesting nonetheless. it proves you're getting smarter about how to find errors in an e-text. (inform jon noring you've started brewing some artificial intelligence!) um, and yes, i know the impetus for this comes from jim's approach. and i could tell you, and jim, how to go about making it even smarter. but my impression is that jim just doesn't care too much to listen... so, meanwhile, can you say -- explicitly and consciously -- what you have done here that has extended jim's notions in a meaningful and important way? and how is that related to the pattern that revealed itself? (and what is that pattern?) -bowerbird

1 0

Heebee Jeebees on Gutenberg
by Geoff Horton 13 May '05

13 May '05

I've been mulling over ideas for applying Natural Language Processing to catch hard-to-find errors in e-texts. I have made little practical progress, but for some reason it occurred to me to try a few carefully chosen Google searches, all restricted to site:www.gutenberg.org. "around the comer" returns 17 hits. "turn the comer" returns no hits. "to he" returns 10,700 hits, a fair number of them not representing typos. "have clone", 13 hits. "will bo", 1 hit. "to bo", 38, some legit (often using "Bo" as a proper noun) "went borne" (for "went home"), 5 hits, one of which is legit and the other being four different editions of the same work, all with the same error. "fax away", 1 hit. "coining to", 23 hits, some legit. "he docs", 7, with some repeat editions. "it docs", 9, with repeats, but offset by two hits in one work. "she docs", none. I don't know what all that proves, but I found it interesting nonetheless.

3 2

Re: [gutvol-d] More on "My Antonia" (was Reply comments on
by Jon Noring 13 May '05

13 May '05

[both a public and private reply] Josh wrote: > Jon Noring wrote: >> Also, anyone here interesting in >> converting the markup to TEI? (It should be relatively straightforward >> since I use a structural/semantic markup which is relatively >> compatible with TEI markup principles.)] > Jon, I'm currently working closely with Marcello on the PGTEI > spec. I'd love to have My Antonia to convert to TEI. > > Either send me a .zip file of the HTML you've got or give me a > link to a download site and I'll put it on my to-do list. I can > probably get to it middle to late next week. Thanks! The valid XHTML 1.1 version in its current state (which is now of pretty low error rate, but could be improved as Bowerbird and I have been "lively" discussing), is found at: http://www.openreader.org/myantonia/ Any of the links to the "XHTML 1.1" versions (using different stylesheets) brings up the same identical XHTML 1.1 document, such as: http://www.openreader.org/myantonia/basic-design-nopagenum/myantonia.html Undoubtedly there are some things I've done in markup which could have been done better, especially with regards to "better mapability" to TEI. I certainly would like feedback on this. We can also chat by phone (on my penny) or on Skype or other free VoIP, if that will be more efficient for you to resolve anything in it which may not be clear (I'd like to know that, too.) Of course, everything in the "front matter" section will probably require more work. I'm not sure how you want to handle things like the Table of Contents. I have planned to revamp and simplify the front matter section (for now I tried to emulate the original layout of the Title Page, for example, but plan something more generic and simple for the final product since most front-matter stuff is metadata rather than content-related.) The encoding itself is pure ASCII (thus UTF-8 conformant) and all the special characters are represented by both numeric and mnemonic character entities, the manifest of which is located at the top of the document. With regards to fixing of the few remaining errors, I'll document that from now on so the TEI version can be simultaneously fixed. Thanks! Jon Noring

1 0

re: still more on "my antonia"
by Bowerbird＠aol.com 13 May '05

13 May '05

jon said: > Anyway, I think you mentioned that "consistency" error before no, this is another one. it's a place where you've deviated from the text on the scan. you _might_ say you've done it for reasons of "consistency"; however, there is no annotation to that effect in the file, as i believe you _would_ put, if that were really the case... > I guess what you really meant to say is that "I will gladly > submit my errata immediately after I am paid." <laugh/> yep, that's it. :+) although, if you haven't found the errors i know about by august -- that's 6 months after you put the file up -- i'll submit my error-free version to project gutenberg. at no cost. :+) (but, since there _is_ a serious point here, let me address it: what you are advocating is a system that is quite well-done, but which is _extremely_ expensive. indeed, it's _so_ costly that it simply _can_not_be_justified_ on a cost-benefit basis, especially since many promised benefits will not materialize. i'm just driving that point home to you. from the standpoint of "every text must be perfect", it might be worth it to _you_ to pay $50 to attain that perfection. but to end-users? no way. they'd rather keep that $50, and live with that one little error -- since no one has even managed to notice it so far anyway. as for source metadata, users wouldn't pay _5_ bucks for that.) > And of course, I won't tell you here > exactly what errors the proofers found do you think i can't pinpoint them myself? (and here's a hint back for you: those weren't the ones.) > What you are implying by this statement is that > *you know* what the error-free text is supposed to be. i do. because i have the scans to refer to. :+) > Interesting that you apparently > missed at least a couple errors. <laugh/> what makes you think i missed those errors? :+) all i said was that i was sitting on 2 errors. i did not say i was sitting on 2 and only 2. i never show all my cards if i don't need to. i only expect to get paid the $50 for the _last_ error. (although if you wait too long, i'll soon be considering a pricetag of $25 for the _next-to-last_ one as well...) > You did bring up a good point about the line break issue. > This leads to my answer to Pauline's question on why I > didn't use DP, and why I'm hesitant now to submit it to DP: just run the scans through o.c.r. again, and keep the linebreaks. that's what i did. that gives you an independent work-product, which is _really_ what you need to have to find all the errors. > it would be a waste to have DP do it all over again > when it is now in very good shape. except it might save you 75 bucks... :+) > If anyone reading this wants to help and proof a few pages using > the "primitive" system I now have (e.g., just print out the pages > from the page scans you would like to proof, and compare them with > the online XHTML version which shows the page numbering and breaks. > There are other ways to do this proofing by comparison, such as > opening up two windows in your browser, one showing the page scan > and the other the online XHTML text.) it is an abuse of volunteers to make them endure processes like that... > As noted above, I would use a DP process for "mass production", > but for this particular project did not for the reasons cited above. well, if you run it through abbyy finereader v7.x, the output is clean. this book would fly through d.p. -- i'd be surprised if it lasted over 3 hours in the queue, because the scans are clear, the text clean, and the book easy and interesting -- and you'd have a separate product, which is worth gold. so you're just not thinking very clearly here... > "My Antonia" is, and has always been, a demonstration project i guess i'm still not sure what it is you're "demonstrating" with it. you say it is a "proof of concept", but what exactly is the concept? this book is dirt-simple -- among the simplest possible 10%, i'd say. and the markup you did pales against that routinely coming from d.p. i don't see anything but ordinary-and-mundane in what you've done. (i sincerely don't mean anything derogatory in saying that, because the job you did is competent, and that is all that is really required. i just don't see anything over and above simple competence here. but if i'm missing something, please do feel free to enlighten me.) > to experiment with new ideas i don't see any. > to get my hands "dirty" with the production process > (although I've transcribed a dozen texts before), but you've just told us that you used a one-time-only process here. (good thing, too, because, as we agree, your process was primitive.) and basically, you borrowed an e-text that someone else produced! that's a good trick, when you can do it. but it won't scale very far. and it can hardly be considered to be "proof of concept". > and to use it for showing to some people interested in this. ok, well i hope it "worked"... there's one born every minute... > This will be a tough test for your "we-don't-need-DP" approach: i never said anything close to that. i think d.p. is swell. many people digitize books individually and independently, so it certainly can be done. given the right tools, it can even be easy. nonetheless, i think cooperation is dandy... but as for your test, bring it on. if you have an old book, with hard-to-o.c.r. text, i highly recommend that you buy the abbyy version specialized for old, hard-to-o.c.r. books. without that, you won't know how successful o.c.r. can be. > Maybe we should do a competition (you seem to love competition!): actually, i believe competition is so 20th-century... but i love to be challenged. and i love to expose the hype people try to spin. so bring on your test. but the way i'll work it is this: you put it through d.p., and then i'll see if i can find any errors they didn't find. if i do, i'll figure out what i'll charge you to reveal them. :+) (if _i_ were you, i'd use the money to buy finereader instead.) > Of course, you will have to promise not to do > late hours hand proofing of the text -- > that'd be cheating -- you have to do it all > auto-magically as you've been advocating. right. like i'm gonna spend a lot of my time volunteering for _you_. i take on your challenges because i can smash them so effortlessly... -bowerbird

2 1

Re: [gutvol-d] More on "My Antonia" (was Reply comments on US issues raised by '
by Joshua Hutchinson 13 May '05

13 May '05

----- Original Message ----- From: "Jon Noring" <jon(a)noring.name> > Also, anyone here interesting in > converting the markup to TEI? (It should be relatively straightforward > since I use a structural/semantic markup which is relatively > compatible with TEI markup principles.)] > Jon, I'm currently working closely with Marcello on the PGTEI spec. I'd love to have My Antonia to convert to TEI. Either send me a .zip file of the HTML you've got or give me a link to a download site and I'll put it on my to-do list. I can probably get to it middle to late next week. Josh

1 0

More on "My Antonia" (was Reply comments on US issues raised by 'orphan works')
by Jon Noring 13 May '05

13 May '05

[Pauline, I explain later on why I have not submitted "My Antonia" to DP. But if someone from DP here says to submit the scans and the current XHTML text for final-stage proofing, then I'll do so. I'll take care of the final XHTML markup. Also, anyone here interesting in converting the markup to TEI? (It should be relatively straightforward since I use a structural/semantic markup which is relatively compatible with TEI markup principles.)] Bowerbird wrote: > Jon wrote: >> I am looking for several proofreaders to finish the proofing of "My >> Antonia" by comparison of the present XHTML to the original page >> scans. It's been partially proofed already (errata has been >> submitted by several people already, including Bowerbird.) > i'm sitting on 2 more errors in "my antonia", jon -- one outright > error, and one consistency error -- waiting until you offer a $50 > bounty for the final error. :+) Don't hold your breath waiting for your moolah. <smile/> Anyway, I think you mentioned that "consistency" error before (it is an interpretation of what the typesetter did regarding spacing with respect to two classes of contractions -- and how they are done in the current XHTML is *correct* based on analyzing the 1926 second edition, which Willa Cather herself was active in its production.) It is interesting in what you've said before about the need for better errata reporting and correction mechanisms, meaning you want errata to be immediately fixed for Public Domain texts as they are found by users. Nice to see you are being consistent to your principles. I guess what you really meant to say is that "I will gladly submit my errata immediately after I am paid." <laugh/> Anyway, the proofers did find a couple errors (and as I noted the text has not been sufficiently proofed yet), which have been corrected in the current online text. I don't know if they corrected the errors you found, but I don't really care. And of course, I won't tell you here exactly what errors the proofers found, but for $100 I'll be happy to tell you (but I'll be nice and give you a clue: "didn't" and "liked".) Otherwise you can download the XHTML document again and run a 'diff' on it -- happy hunting! > i figure at some point, it will become worth that to you to be able > to say with confidence your text is error-free. What you are implying by this statement is that *you know* what the error-free text is supposed to be. Interesting that you apparently missed at least a couple errors. <laugh/> You did bring up a good point about the line break issue. This leads to my answer to Pauline's question on why I didn't use DP, and why I'm hesitant now to submit it to DP: 1) I had a project deadline where I had to get the text reasonably cleaned up and online *in a hurry* (with page scans) for demo purposes. I made that deadline, but it led me to not do it via DP. It was my intention to continue the clean-up process after the deadline -- it's only the right thing to do. However, for actual mass production of texts, DP or similar process would be used -- which I have noted a couple times before. 2) Additionally, because of the deadline, I decided to start with an HTML document which was already pretty well-proofed (it was NOT PG's version), and did various types of analysis and hand-proofing on it (including 'diff' to PG's version, which, though mangled, provided a reference point using the principle that most of the transcribed text in it is accurate -- all discrepencies between PG's and my version were then compared to the original page scans (most of the discrepencies were with PG's text, I might add.) If there were the same errors in both, then those errors would not be caught except by proofing, by running some auto-checks, and maybe by creating a third text using OCR and 'diff'ing that with the others, which is what I assume Bowerbird did.) 3) So, the source I used was already missing the original line breaks, another reason that it was probably too late to submit it DP, since I don't believe the text I've done up to now could be efficiently injected "midway" into DP's workflow; it would be a waste to have DP do it all over again when it is now in very good shape. If anyone reading this wants to help and proof a few pages using the "primitive" system I now have (e.g., just print out the pages from the page scans you would like to proof, and compare them with the online XHTML version which shows the page numbering and breaks. There are other ways to do this proofing by comparison, such as opening up two windows in your browser, one showing the page scan and the other the online XHTML text.) 4) As noted above, I would use a DP process for "mass production", but for this particular project did not for the reasons cited above. "My Antonia" is, and has always been, a demonstration project to experiment with new ideas, to get my hands "dirty" with the production process (although I've transcribed a dozen texts before), and to use it for showing to some people interested in this. Yes, it is just "one" text out of the 5000+ already done by DP, but "My Antonia" was not intended to be competitive to anyone. It is a first small and methodical step towards meeting several goals and proving some principles. It is still "beta", too, and hopefully all the ideas I want to experiment with will be implemented in a couple months (I've just started to implement what I'd like to implement.) What's been done so far has nicely assisted with the LibraryCity business plan, though, and may lead to a parallel business plan to be written by several of us in a couple months that's a little closer to the public domain text area. Btw, Bowerbird, I'm hoping to soon scan my copy of the *original* Burton edition of the Kama Sutra of Vatsyayana, and the scans will be submitted to DP as someone there requested. Like "My Antonia", I'm going to do high quality scans (at 600 dpi) and carefully clean them. This will be a tough test for your "we-don't-need-DP" approach: very small font, poor late nineteenth century typesetting, and some poorly printed pages (and this is one of the better copies of this book!) Maybe we should do a competition (you seem to love competition!): After I complete the scans, I'll submit them to both you and to DP. After the magic is done by you and DP, we'll compare the results (such as by 'diff'ing them) to look for inconsistencies which will then be compared to the original page scans. Of course, you will have to promise not to do late hours hand proofing of the text -- that'd be cheating -- you have to do it all auto-magically as you've been advocating. As some person whose name eludes may says, "The proof is in the pudding." Are you up to the challenge? <smile/> Anyway, I think such an experiment is good to gauge the current state of OCR and auto-processing to clean up texts, so it is a test which will benefit everyone. In addition, the careful screening of this text during the "competition" (such as by comparing the result of the two approaches) should lead to a *very low error*, possibly *zero error* digital text version, which can then be used for quick assessment of the accuracy of new OCR and auto-post-processing algorithms. I believe the Kama Sutra scans plus zero error digital text will make a good addition for a "test suite". Jon Noring

1 0

re: Re[2]: [gutvol-d] Reply comments on US issues raised by 'orphan works'
by Bowerbird＠aol.com 13 May '05

13 May '05

jon said: > I am looking for several proofreaders to finish the proofing of > "My Antonia" by comparison of the present XHTML to the > original page scans. It's been partially proofed already (errata has > been submitted by several people already, including Bowerbird.) i'm sitting on 2 more errors in "my antonia", jon -- one outright error, and one consistency error -- waiting until you offer a $50 bounty for the final error. :+) i figure at some point, it will become worth that to you to be able to say with confidence your text is error-free. by the way, how do you suggest people do the proofing? (it's darn hard to proof text with the linebreaks removed!) if you can guarantee the scans will stay available for a good time to come at their current location, i would be willing to customize my proofing tool to work with 'em. let me know if you are interested... *** pourlean said: > Why not just put the text through DP? there's an idea for you, jon. it might save you 50 bucks... ;+) -bowerbird

1 0

re: [gutvol-d] Reply comments on US issues raised by 'orphan works'
by Bowerbird＠aol.com 13 May '05

13 May '05

jeff said: > Know anybody who needs a proofreader? i hear jon noring is looking... ;+) -bowerbird

3 2

making CD / DVD best of 2005--how and which books?
by Scott Clous 12 May '05

12 May '05

making CD / DVD best of 2005. Which books, and why? I am eager to create an updated CD and DVD from the 2 that already exist, including the corrected versions. My motivation is 2 fold, people for better or worse download things that they think are the most current--using robots to get the most current versions vs. just the CD, etc. and because it does make the site more useable if one download gets them some wonderful content directly--and we all would want it to be the most current (and hopefully correct) versions. Many people have broadband now, and Bittorrent makes many things easier. I know there are some challenges in getting it together, then maintaining it, because the format of presenting the content has changed. Usually, without tagging the content itself, within a field to id that content as going onto a complitation, the only other method of selection is an index of choices for the CD or DVD. To start things off, I'd like to use the existing selections, update them, and make the CD available again. I also will review the new titles, Sherlock is near and dear to all our hearts, I'm sure, and include them if possible and their popularity warrents it. It would be nice to have a method to automatically update it -- fairy tales and all -- so that the site, and this part of it appear to be current. This last goal is additional work, but might be well worth it. Do any of the volunteers or programmers have feedback on these ideas? __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

5 4

Re: gutvol-d Digest, Vol 10, Issue 11
by Scott Clous 12 May '05

12 May '05

The reasons people always will want a "standard" best of, is that it takes time to pick things. Also, that does make it a bit easier to mirror the isos. I want to do a CD first because getting a successful download is key, not how much stuff can be packed onto a DVD that can't be opened. Question--do most of the subscribers/downloaders live in the US, &/or are English speakers? While I would have thought at first to only have English content in a best of, my cousin who is overseas now is enjoying the German portions of the DVD! However, if the goal is to have something easy for the give away, at least on the CD I'll suggest we try for all English; Someone who speaks Spanish, etc. would be in a better place than I to answer the great works on the others, and would have a greater motivation to see it kept up. Greg--"I should have a program available soon (really, really soon - in time for Michael Hart's visit to Fairbanks in June) to "build your own" ISO. Seeding these with lists of eBook #s is the best possible start. I'm looking forward to you program, can you tell us a bit more about it, those of us who "just came in?" Greg-"I think that the current "Best Of" CD is probably your best starting point. While we've done many, many new eBooks since then, the best-known titles were already available for the CD. But many have been updated, and a few have had images or HTML added. Is there any way, other than manually to ID the books from the CD? DVD? __________________________________ Do you Yahoo!? Yahoo! Mail - Helps protect you from nasty viruses. http://promotions.yahoo.com/new_mail

2 1