- gutvol-d - lists.pglaf.org

re: [gutvol-d] HandHeld & E-Books
by Bowerbird＠aol.com 14 May '05

14 May '05

vijay said: > How convenient are handhelds to read E-books ? some people swear by them. -bowerbird

1 0

Challenges for PG.
by rvijay07 14 May '05

14 May '05

What are some of the challenges for PG in your opinion ? I thought of starting this topic after my recent experience. I gave the PG CD to 3 people who expressed interest. One was honest and said that she and her husband saw it for like 10 minutes and wanted to read it more. However, at work they are in front of their PC all day. Hence, to be at home again in front of the PC is strainful for their eyes/posture etc., Also quantitatively, they have limited time and hence even if they did, can read just a few of those books. Vijay _______________________________________________ No banners. No pop-ups. No kidding. Make My Way your home on the Web - http://www.myway.com

1 0

Digital Library of India
by rvijay07 14 May '05

14 May '05

Bowerbird at aol.com wrote: >> i would think that's precisely what michael hart had in mind with that new project of his... >> and yes, it is a great idea... >> the logical extension of his earlier invention of the e-book, collecting them as a library... >> except the extension had to endure a mere 20 months (or so) of flack, not 20 years, like his earlier one... -bowerbird The early 20 years of flak was *FELT* for the most part was due to the fact that very few people had PCs and were on the net. In other words there was nothing against the project then. Vijay _______________________________________________ No banners. No pop-ups. No kidding. Make My Way your home on the Web - http://www.myway.com

1 0

Re: [gutvol-d] Heebee Jeebees on Gutenberg
by Bowerbird＠aol.com 14 May '05

14 May '05

andrew said: > So where to go from here? > With our current set-up, > to merely generate a list of these > and submit them as corrections > would be somewhat overloading our volunteers. sounds like you need some better tools. or, even better yet, a thorough revamping of your error-detection and error-correction processes... you need to have a system that runs a test like this, then automatically summons the scans and the e-text, side-by-side so you can make a decision on the matter -- one small matter here, you need to do the scans -- and then lets you click a button that will automatically make the change, and generate everything necessary, like an updated change-log, new derivative versions, a post message, e-mail to that e-text's listserve, etc. by the way, a subset of this system -- the part that calls up the scan and the text for a comparison -- is what the public can use to verify an error-report. although you'd want to do these tests proactively, it also never hurts to let end-users help you out... in a post to this listserve from maybe 6-9 months ago, i detailed the complete architecture for such a system. the message was ignored. perhaps it should be revisited. -bowerbird

1 0

re: [gutvol-d] Heebee Jeebees on Gutenberg
by Bowerbird＠aol.com 13 May '05

13 May '05

andrew said: > The interesting point is that, apparantly, with a group of > 5-7 people individually examining the same text, > each person usually found at least one error > that none of the others did. that means all the proofers are missing some. :+) someone over on one of the d.p. forums once posted a mathematical formula that you can use in such a situation to predict how many undiscovered errors remain in that e-text... this formula is a valuable thing to be aware of, since that information is usually opaque to you. -bowerbird

1 0

re: [gutvol-d] Heebee Jeebees on Gutenberg
by Bowerbird＠aol.com 13 May '05

13 May '05

jeff said: > I've been mulling over ideas alright! someone around here is thinking for once! :+) > for applying Natural Language Processing > to catch hard-to-find errors in e-texts. oh-oh. a lapse into acronym-land. not a good sign of quality thinking... ;+) > I have made little practical progress i suggest you ditch the acronym; those things just weigh you down... :+) > but for some reason it occurred to me to > try a few carefully chosen Google searches, > all restricted to site:www.gutenberg.org. good idea. so ok, let's take a look. > "around the comer" returns 17 hits. seems like that's a stealth scanno that should be in the dictionary already, yep. is jim tinsley taking these notes down as error-reports? > "turn the comer" returns no hits. i guess we turned the corner on that one... > "to he" returns 10,700 hits, > a fair number of them not representing typos. false alarms are counterproductive; that's the big problem with gutcheck. every false alarm wastes your time. however -- in contrast to misses, which are invisible, and therefore cannot be instructive -- false alarms _do_ give you information about how to improve your process... that is why you need to treasure them, and to study them. (yes, for the acronym-lovers among us, this is signal-detection theory in action. s.d.t. to you.) > "have clone", 13 hits. have clone, will reproduce. > "will bo", 1 hit. roger wilco. > "to bo", 38, some legit (often using "Bo" as a proper noun) again, you need to examine the false alarms closely, to see if they reveal clues how to make your search more fine-grained. here, for instance, the "b" in the false alarms is capitalized, when it otherwise would not be, indicating it's a proper noun, and gives a reason to weight those instances less heavily, or indeed, if happening repeatedly in one e-text, even ignore 'em. this is where most people go bad. they hit a few false alarms, think "oh no, this isn't working", and give up. that is so wrong! as long as the number of false alarms isn't overwhelming you, it's much more cost-effective to take the hits that you got and then go to work on improving performance on the false alarms... > "went borne" (for "went home"), 5 hits, one of which is legit > and the other being four different editions of the same work, > all with the same error. at least the _errors_ are consistent! ;+) but do you see a pattern emerging here? > "fax away", 1 hit. what should it be? although "fax" _could_ be in a scanno dictionary, for "far", i'd take it out, because very few public-domain books have the word "fax" in them, do they? ;+) so look at every instance of "fax", pure and simple. > "coining to", 23 hits, some legit. any clues as to how to tell the machine to tell the difference? start digging, and dig deep. > "he docs", 7, with some repeat editions. "docs" might be in a list of stealth scannos too. but again, let's be realistic. how often does it come up legitimately? and in the rare case it does come up legitimately... > "it docs", 9, with repeats, but offset by two hits in one work. ...it might come up in that book more than one time, right? and by now, you should definitely see a certain pattern... > "she docs", none. does she? i guess she doesn't... > I don't know what all that proves, > but I found it interesting nonetheless. it proves you're getting smarter about how to find errors in an e-text. (inform jon noring you've started brewing some artificial intelligence!) um, and yes, i know the impetus for this comes from jim's approach. and i could tell you, and jim, how to go about making it even smarter. but my impression is that jim just doesn't care too much to listen... so, meanwhile, can you say -- explicitly and consciously -- what you have done here that has extended jim's notions in a meaningful and important way? and how is that related to the pattern that revealed itself? (and what is that pattern?) -bowerbird

1 0

Heebee Jeebees on Gutenberg
by Geoff Horton 13 May '05

13 May '05

I've been mulling over ideas for applying Natural Language Processing to catch hard-to-find errors in e-texts. I have made little practical progress, but for some reason it occurred to me to try a few carefully chosen Google searches, all restricted to site:www.gutenberg.org. "around the comer" returns 17 hits. "turn the comer" returns no hits. "to he" returns 10,700 hits, a fair number of them not representing typos. "have clone", 13 hits. "will bo", 1 hit. "to bo", 38, some legit (often using "Bo" as a proper noun) "went borne" (for "went home"), 5 hits, one of which is legit and the other being four different editions of the same work, all with the same error. "fax away", 1 hit. "coining to", 23 hits, some legit. "he docs", 7, with some repeat editions. "it docs", 9, with repeats, but offset by two hits in one work. "she docs", none. I don't know what all that proves, but I found it interesting nonetheless.

3 2

Re: [gutvol-d] More on "My Antonia" (was Reply comments on
by Jon Noring 13 May '05

13 May '05

[both a public and private reply] Josh wrote: > Jon Noring wrote: >> Also, anyone here interesting in >> converting the markup to TEI? (It should be relatively straightforward >> since I use a structural/semantic markup which is relatively >> compatible with TEI markup principles.)] > Jon, I'm currently working closely with Marcello on the PGTEI > spec. I'd love to have My Antonia to convert to TEI. > > Either send me a .zip file of the HTML you've got or give me a > link to a download site and I'll put it on my to-do list. I can > probably get to it middle to late next week. Thanks! The valid XHTML 1.1 version in its current state (which is now of pretty low error rate, but could be improved as Bowerbird and I have been "lively" discussing), is found at: http://www.openreader.org/myantonia/ Any of the links to the "XHTML 1.1" versions (using different stylesheets) brings up the same identical XHTML 1.1 document, such as: http://www.openreader.org/myantonia/basic-design-nopagenum/myantonia.html Undoubtedly there are some things I've done in markup which could have been done better, especially with regards to "better mapability" to TEI. I certainly would like feedback on this. We can also chat by phone (on my penny) or on Skype or other free VoIP, if that will be more efficient for you to resolve anything in it which may not be clear (I'd like to know that, too.) Of course, everything in the "front matter" section will probably require more work. I'm not sure how you want to handle things like the Table of Contents. I have planned to revamp and simplify the front matter section (for now I tried to emulate the original layout of the Title Page, for example, but plan something more generic and simple for the final product since most front-matter stuff is metadata rather than content-related.) The encoding itself is pure ASCII (thus UTF-8 conformant) and all the special characters are represented by both numeric and mnemonic character entities, the manifest of which is located at the top of the document. With regards to fixing of the few remaining errors, I'll document that from now on so the TEI version can be simultaneously fixed. Thanks! Jon Noring

1 0

re: still more on "my antonia"
by Bowerbird＠aol.com 13 May '05

13 May '05

jon said: > Anyway, I think you mentioned that "consistency" error before no, this is another one. it's a place where you've deviated from the text on the scan. you _might_ say you've done it for reasons of "consistency"; however, there is no annotation to that effect in the file, as i believe you _would_ put, if that were really the case... > I guess what you really meant to say is that "I will gladly > submit my errata immediately after I am paid." <laugh/> yep, that's it. :+) although, if you haven't found the errors i know about by august -- that's 6 months after you put the file up -- i'll submit my error-free version to project gutenberg. at no cost. :+) (but, since there _is_ a serious point here, let me address it: what you are advocating is a system that is quite well-done, but which is _extremely_ expensive. indeed, it's _so_ costly that it simply _can_not_be_justified_ on a cost-benefit basis, especially since many promised benefits will not materialize. i'm just driving that point home to you. from the standpoint of "every text must be perfect", it might be worth it to _you_ to pay $50 to attain that perfection. but to end-users? no way. they'd rather keep that $50, and live with that one little error -- since no one has even managed to notice it so far anyway. as for source metadata, users wouldn't pay _5_ bucks for that.) > And of course, I won't tell you here > exactly what errors the proofers found do you think i can't pinpoint them myself? (and here's a hint back for you: those weren't the ones.) > What you are implying by this statement is that > *you know* what the error-free text is supposed to be. i do. because i have the scans to refer to. :+) > Interesting that you apparently > missed at least a couple errors. <laugh/> what makes you think i missed those errors? :+) all i said was that i was sitting on 2 errors. i did not say i was sitting on 2 and only 2. i never show all my cards if i don't need to. i only expect to get paid the $50 for the _last_ error. (although if you wait too long, i'll soon be considering a pricetag of $25 for the _next-to-last_ one as well...) > You did bring up a good point about the line break issue. > This leads to my answer to Pauline's question on why I > didn't use DP, and why I'm hesitant now to submit it to DP: just run the scans through o.c.r. again, and keep the linebreaks. that's what i did. that gives you an independent work-product, which is _really_ what you need to have to find all the errors. > it would be a waste to have DP do it all over again > when it is now in very good shape. except it might save you 75 bucks... :+) > If anyone reading this wants to help and proof a few pages using > the "primitive" system I now have (e.g., just print out the pages > from the page scans you would like to proof, and compare them with > the online XHTML version which shows the page numbering and breaks. > There are other ways to do this proofing by comparison, such as > opening up two windows in your browser, one showing the page scan > and the other the online XHTML text.) it is an abuse of volunteers to make them endure processes like that... > As noted above, I would use a DP process for "mass production", > but for this particular project did not for the reasons cited above. well, if you run it through abbyy finereader v7.x, the output is clean. this book would fly through d.p. -- i'd be surprised if it lasted over 3 hours in the queue, because the scans are clear, the text clean, and the book easy and interesting -- and you'd have a separate product, which is worth gold. so you're just not thinking very clearly here... > "My Antonia" is, and has always been, a demonstration project i guess i'm still not sure what it is you're "demonstrating" with it. you say it is a "proof of concept", but what exactly is the concept? this book is dirt-simple -- among the simplest possible 10%, i'd say. and the markup you did pales against that routinely coming from d.p. i don't see anything but ordinary-and-mundane in what you've done. (i sincerely don't mean anything derogatory in saying that, because the job you did is competent, and that is all that is really required. i just don't see anything over and above simple competence here. but if i'm missing something, please do feel free to enlighten me.) > to experiment with new ideas i don't see any. > to get my hands "dirty" with the production process > (although I've transcribed a dozen texts before), but you've just told us that you used a one-time-only process here. (good thing, too, because, as we agree, your process was primitive.) and basically, you borrowed an e-text that someone else produced! that's a good trick, when you can do it. but it won't scale very far. and it can hardly be considered to be "proof of concept". > and to use it for showing to some people interested in this. ok, well i hope it "worked"... there's one born every minute... > This will be a tough test for your "we-don't-need-DP" approach: i never said anything close to that. i think d.p. is swell. many people digitize books individually and independently, so it certainly can be done. given the right tools, it can even be easy. nonetheless, i think cooperation is dandy... but as for your test, bring it on. if you have an old book, with hard-to-o.c.r. text, i highly recommend that you buy the abbyy version specialized for old, hard-to-o.c.r. books. without that, you won't know how successful o.c.r. can be. > Maybe we should do a competition (you seem to love competition!): actually, i believe competition is so 20th-century... but i love to be challenged. and i love to expose the hype people try to spin. so bring on your test. but the way i'll work it is this: you put it through d.p., and then i'll see if i can find any errors they didn't find. if i do, i'll figure out what i'll charge you to reveal them. :+) (if _i_ were you, i'd use the money to buy finereader instead.) > Of course, you will have to promise not to do > late hours hand proofing of the text -- > that'd be cheating -- you have to do it all > auto-magically as you've been advocating. right. like i'm gonna spend a lot of my time volunteering for _you_. i take on your challenges because i can smash them so effortlessly... -bowerbird

2 1

Re: [gutvol-d] More on "My Antonia" (was Reply comments on US issues raised by '
by Joshua Hutchinson 13 May '05

13 May '05

----- Original Message ----- From: "Jon Noring" <jon(a)noring.name> > Also, anyone here interesting in > converting the markup to TEI? (It should be relatively straightforward > since I use a structural/semantic markup which is relatively > compatible with TEI markup principles.)] > Jon, I'm currently working closely with Marcello on the PGTEI spec. I'd love to have My Antonia to convert to TEI. Either send me a .zip file of the HTML you've got or give me a link to a download site and I'll put it on my to-do list. I can probably get to it middle to late next week. Josh

1 0