- gutvol-d - lists.pglaf.org

or a watch to take out of it
by Bowerbird＠aol.com 01 Mar '10

01 Mar '10

ok, so there's a new e-book viewer-program in town. liza daly, who is affiliated with the o'reilly folks, has brought out "ibisreader" for your reading enjoyment. so i went to check it out... at ibisreader.com, i clicked "get started" and then -- at "add a book" -- "feedbooks: popular public domain". the 4th book down is "alice's adventures in wonderland", and since the movie is coming out this friday, i got that. feedbooks, as you probably know, is a site that takes e-books from various sources, like project gutenberg, and makes some very nice-looking versions of them... since feedbooks re-works the books, they don't have as many books as some of the other sites, but they are preferred by many people because their books look nice. so i start looking at the text, and i find this: > There was nothing so very remarkable in that; > nor did Alice think it so very much out of the way > to hear the Rabbit say to itself “Oh dear! Oh dear! > I shall be too late!” (when she thought it over > afterwards, it occurred to her that she ought to > have wondered at this, but at the time it all seemed > quite natural); but when the Rabbit actually > took a watch out of its waistcoat-pocket, > and looked at it, and then hurried on, > Alice started to her feet, for it flashed across her mind > that she had never before seen a rabbit with > either a waistcoat-pocket, or a watch to take out of it, > and, burning with curiosity, she ran across the field after it, > and was just in time to see it pop down a large rabbit-hole > under the hedge. well, gee. if you're intimately familiar with this book, you know > took a watch out of its waistcoat-pocket, is a phrase that is _italicized_ in the book. but not on this file... here's where you can see a copy of the original: > http://www.archive.org/stream/alicesadventur00carr#page/2/mode/2up you'll have to take my word for it that it's not italicized in the feedbooks copy that is being used by ibisreader, since i don't see any convenient way for me to link to a specific page there. (but it's book #22 from feedbooks, if you wanna look yourself.) it's pretty clear that what has happened here is that feedbooks has taken pg#11 and used it as its source. what is a sad thing, because -- even after all the years it coulda been "improved" -- pg#11 _still_ doesn't have proper italics in it. it was "updated" in 2005 (leaving no trace behind) and then once again in 2008 (when an .html version was added). but it still has zero italics. > http://www.gutenberg.org/files/11/11-h/11-h.htm > http://www.gutenberg.org/files/11/11.txt the p.g. version _does_ have italics rendered as all-uppercase. so there is _some_ indication of them. but this is ambiguous, since there are places in the book where uppercase is used too: > http://www.archive.org/stream/alicesadventur00carr#page/4/mode/2up (see the reference to "orange marmalade", in all-uppercase.) if i would've been feedbooks, i would've converted _all_ of the uppercased words to italics. but of course then they would've been changing things like chapter headers and first-words too. and of course, feedbooks could've _left_ words in all-uppercase. i don't know why they didn't, but i assume it's because they take _pride_ in their typography, and all-uppercase looks like crap... (but it's true that the tell-tale examples of _actual_ uppercase, namely "orange marmalade", "drink me", and "eat me", are all still rendered in uppercase in feedbooks#22, so i am stumped.) and yes, folks, i know there are other versions of "alice" posted, including some with italics correctly specified. so let us look... pg#19033 _does_ have italics in it, but it does _not_ italicize the phrase about the rabbit, and his watch, and his pocket... whether it's a version-difference -- it _is_ another version -- or whether it's just a digitization mistake, i simply don't know. (and since this is clearly not the explanation for the feedbooks discrepancies, i have no interest in determining the reasons.) pg#928 -- an .html version only -- _does_ have italics. yay! pg#28885 also has the italics, and it has the images as well! although sometimes things don't go right, as shown here: > http://z-m-l.com/misc/alice-glitch.png the feedbooks version has _no_ italics at all, as far as i can see, not even the different set of italics from pg#19033. so, at first, i had thought they'd used pg#11 as their source text, but now, i'm not so sure. (they have some strange contractions, which could indicate that they might've done their own digitization.) at any rate, the point still stands that pg#11 lacks proper italics. all-uppercase is _not_ a substitute. and since the italics _have_ been done -- in pg#928 -- the changes should be incorporated into pg#11. the presence of other versions, at higher numbers, won't change the fact that pg#11 is considered as "the original." so, c'mon, let's see someone who talks about pg/dp "quality" and "incremental improvement" actually _back_up_ the claim. let's get this classic and legendary e-text cleaned up now, ok? -bowerbird

1 0

Re: Processing eTexts
by Bowerbird＠aol.com 27 Feb '10

27 Feb '10

well, it's kinda silly for us to talk about this in generalities, because i'm capable of demonstrating what i mean specifically and exactly... and when i'm done, i'm quite sure that you will agree completely. (and you might say that's what you meant all along, for all i know.) so let's get specific. i've posted a file of text taken from rfrank's roundless experiment: > http://z-m-l.com/go/campf/campf-001.zml as you can see, this book is called "pemrose lorry, camp fire girl"... the book is now in-progress, so it's half-done and half-undone... (it looks like it's been proofed up through about page 80 so far, but since the pages go out multiple times, it's difficult to be certain of it, but after page 80, it's fairly clear that the pages haven't been edited.) that first version had the page-separators fixed, and other basic stuff. the next version has more stuff cleaned up, but it's still rather basic: > http://z-m-l.com/go/campf/campf-002.zml as you can easily see, if you look through either one of the versions, starting after page 80, there are a bunch of paragraphing problems. specifically, there are often blank lines inserted -- incorrectly -- between the lines of a single paragraph. you can find instances of this problem on pages __, __, __, and __. it's a not uncommon glitch. paragraphing is one of the first things i try to correct, because it's necessary to have the paragraphs correct to fix any spacey quotes. fortunately, it's rather easy to locate these bad paragraphs via search. find a blank line followed by a line that starts with a lowercase letter. (in other words, you search for two newlines followed by lowercase.) now, one way of doing the search would be to automatically replace _all_ of these occurrences by simply deleting one of the two newlines. you could click "replace all" on that find-and-replace, and change all of the occurrences without even looking at them. you _could_ do that. the proper changes would almost always outnumber improper ones. whenever anyone talks about "taking the human out of the equation", blind global changes is what immediately springs into my little brain. and i am definitely _not_ a fan of blind global changes. there are some changes you can make blindly, but they are few and far between, and they certainly don't characterize the usual process. and this change -- deleting those excess lines -- i don't do blind... i will step through the changes one-by-one, and look at all of them _against_the_scan_. (even though, in most cases, i wouldn't need to look at the scan, because it's pretty obvious how to fix the problem.) sometimes i even delete the excess line manually, rather than merely approve the find-and-replace, just to keep my grubby fingers busy... but -- here's the very important part -- i do a lot _more_ than that. at every one of these spots to which i'm automatically transported, i look around the immediate problem area, to see if it spilled over. and, if it did, then i will clean up that neighboring area right away. so i'm walking a fine line here between doing a specialized search and a general correction routine. i start with the specialized check, so i have a laser focus on the nature of what the check will show me, so i don't have to waste mental energy figuring out what the glitch is. but then i also use my peripheral vision to see what else needs fixing. and this particular check in this particular text is a very good example. because it ends up that a _lot_ of the surrounding areas had problems. (that's not uncommon with these excess blank lines, because it's often the other glitches that _caused_ the o.c.r. its paragraphing difficulties.) there were a good number of problem-lines caught by this routine -- about 85 if i remember correctly -- but i probably corrected the text on 85 surrounding lines as well, because i saw bugs, so i fixed them. i encourage people to "play along at home" and _do_ this search on this text, so you can really see how you can spot neighboring glitches. what you will _not_ know, however, unless you are also looking at the _scan_ for each page, is that there were often _entire_words_missing_ from the o.c.r. they were cut off entirely from the left-hand margin. this isn't the kind of thing for which you can do a search to find them. they're just missing from the o.c.r., and you don't know they're gone. however, since you are paying attention during this _other_ search, you can become aware of these missing words, via peripheral vision. if you _do_ "play along at home", and actually step through that check, you will also see that there's yet another check that needs to be done to catch _all_ of these paragraphing problems from excess blank lines. this second check is the "flip side" of the first one. the first one was looking for supposedly-first-lines of paragraphs that were lowercase. the second one checks the _termination_ of supposedly-last-lines... so you're checking for a letter or a comma at the _end_ of a line which is followed by a blank line. if it were _really_ the end of a paragraph, as is implied by that blank line, it should be punctuation-terminated. so you're looking now for a letter/comma followed by two newlines... this search turns up roughly 50 instances in this particular text-file. and, once again, when you make these corrections, you look around. roughly half of the 50 instances were cases where the blank line that was deleted was followed itself by a line beginning with a single-dash, which was a misrecognition for an _em-dash_ at the start of that line. (the em-dash is represented by a double-dash.) so those corrections were made at the same time that the excess blank line was eliminated. if you were a party-pooper, and didn't play along at home, i have uploaded a copy of the file where the lines that were flagged by the first page are indicated by an "@" sign, which you can search for: > http://z-m-l.com/go/campf/campf-003.zml and here's one with the lines from the second check tagged with "%": > http://z-m-l.com/go/campf/campf-004.zml you can also view the book page-by-page, as usual. for instance: > http://z-m-l.com/go/campf/campfp123.html (you will see, right there on page 123, an example of a page which has some of the words cut off from the right-margin. when i find this type of problem in a scanset, it just makes me want to scream! when you scan a book, you need to make sure you do it _carefully_. sloppy work such as this is tremendously uncharacteristic of rfrank.) at any rate... it's this "looking around and fixing other errors in the neighborhood" that makes this whole preprocessing thing such a _vibrant_ procedure. ...which brings me squarely back to the major point of this post... *** so yeah, i'm doing a computerized check, and it's darn _efficient_, but i'm surely not "taking humans out of the process". not by a long shot. i am a mentally alert human, who is _actively_ engaged with the text... my cursor is _flying_ all around that document. i move at warp speed. and i am _efficient_ too. i am _tremedously_ efficient. i am rocking! all the other carpenters are using hammers, and i have a _nail-gun_. part of the efficiency is due to that _focus_ that i just talked about... but most is simply because i am being directly transported to errors. i don't waste any of my time _looking_ for them; they're _presented_. i don't waste any time positioning my cursor -- it is _prepositioned_. and the scan is right there, ready for me to look at it, immediately... and it's _fun_. downright _exhilarating_. like driving a sports car. (whereas word-by-word proofing is akin to pushing a baby stroller.) and the results are _better_. not just a little bit better, a lot better. i've shown time after time after time that i can find (lots of) errors proofers _miss_, not just in one round, but two and three and four. and, just to repeat the message yet again, it's _simple_ to do this. a couple dozen _simple_ checks finds a high percentage of errors. so yeah, i know i am explaining all of this with accurate language. but people always seem to misunderstand. on the one hand, they think the tool is making all the changes... on the other hand, they think you need a human to find errors... both of those positions are wrong, and when i argue against one, people jump to the other, and then i have to argue against _that_. i know that i am capable of walking the tight-rope between them, but everyone else seems to get tangled up in the dialog semantics. the tool can find almost _all_ the errors, so you don't need to look. (you're welcome to look, and i'm sure that you'll find a few errors, but at some point, you have to ask whether it was worth your time.) on the other hand, _you_ have to be the active agent that mentally makes the decision whether or not each change should be made... like i said, there are only a few global changes that i make blindly. (to give you a for-instance, i will globally delete any space that follows any doublequote located at the very beginning of a line, or any space preceding a doublequote at the very end of a line. i'll also blindly change spacey-quotes if the open/close pairing within the rest of the paragraph makes the change unambiguous. i'll also close up contractions blindly. maybe a few more things. but for the most part, i really want to look at the changes i make. maybe if i had the luxury of knowing i'd have human volunteers following up with a word-by-word proofing of everything i did, i would be more willing to make blind changes. or maybe not, because that just seems wasteful of donated time and energy.) now, i'm sure that dkretz will confirm what i've been saying here, as his experience with his tool has led him to the same thoughts. and maybe carel will say that this is exactly what she has meant, and we've been doing a semantic dance around the same thought. -bowerbird

1 0

Re: Processing eTexts
by cmiske＠ashzfall.com 27 Feb '10

27 Feb '10

Bowerbird said: >i don't want them to think of this as "the formal proofing process". >i believe we need to put the books in front of them _to_read_... Which is exactly why a book that is ready for release to PG should not force upon an individual the original scan. When the reader spots something they think is an error, they should be able to call up the scan to see if it is an actual error before reporting it as such, etc. Otherwise their reading is distracted by the scan. A proper interface would allow users to select what they want to see. An interface should be about options. In any case, I have no desire to host an archive that duplicates the content of PG and others. The purpose in coming to my project site would be to prepare books for release to PG and to polish released books so that they can be read for leisure and pleasure off of _my_ bandwidth. >> I think calling it a wiki makes it easier for quick comprehension >> rather than explaining the whole process of storage, editing, etc. > >we should never _have_to_ "explain the whole process" to the public. >for them, it should be as easy as clicking a button to say "check this!" >they shouldn't even need to explain the error. (we can find it, right?) Of course, but it is hard to develop it and request assistance with the development of it without explaining the basic concepts, etc. A project must be explained and defined or it cannot be realized. >> Rather than working an editing interface >> into an out-of-the-box wiki system >we don't need to make "an editing interface"... It's rather hard for people to make/submit corrections without having an interface for doing so. >> Has anyone ever just asked google if they could >> use the scans and OCR text for PG? > >michael had a meeting with google when they were just starting out... > >he reported that they treated him rather rudely, hustling them out of >their famous cafeteria before he had even finished his lunch. it's sad, >but i don't think they really appreciate project gutenberg very much... That is too bad. >> XML makes a nice standard for storage and conversion > >it's obtuse, and unnecessary obstructive to the text itself... It's also one of many methods for the storage of 'intelligent' data and is easy to do. Once put in place, it would never be seen again by anyone except someone correcting the markup (which one hopes would be rare). InDesign uses markup, MS Word uses markup, and on and on. I don't think the users of this software find the markup obstructive to the text itself. >> My only protest is that I believe >> a human proofing stage is still a good choice. [snip] >i can't think of one single reason for a word-by-word proofing, >let alone two or three rounds of it. it's simply not efficient, and >anybody who explores the options can learn that for themselves. I don't believe I said anything about multiple rounds of proofing, etc. I believe I discussed the main processing of a document being done through human interaction with scripts and then people actually looking at the results of the process before the output is released (proofing). I feel that a human looking at a smaller subset of a large document is a good thing in the error finding process. You apparently do not think it is. Neither of us is right or wrong: It is a matter of perspective and opinion. >between aggressive clean-up of o.c.r., and the comparison method, >and "smoothreading" by people from the general public, we're good. Yes and no. It depends on your definition of 'good.' >> Otherwise, I would say to have two people >> run the same correction process, diff the results and >> then present the mis-matches to a third set of eyes. >close... but it's unnecessary for "people" to do the digitizations. >two sets of cleaned-up o.c.r., diffed against each other, is enough... That depends on a lot of factors including the assumption that two OCR programs would not make the same mistake and that, if they did, it would be a mistake that could be caught by another process. Etc. >(but yes, it's necessary to have a sentient human doing the clean-up.) Exactly. And, if that human makes a major error, the entire text in now a mess. If two people do the same process it reduces, but does not eliminate, the risk for 'the human error factor.' >o.c.r. clean-up is a fast-paced, active, and energetic task. >which does _not_ mean that it's "harder". just more fun... People actually do enjoy proofing. You do not. Again, neither is wrong in their viewpoint. Volunteers should be involved in a manner that gives them pleasure and satisfaction, etc. I say to give people tools and options and let them be the judge of what is best for them. >i get the impression you think these clean-up tools are just >a bunch of scripts that people turn on, and the results fall out. >but that's not anything like what actually goes on. Since I am planning to write scripts for such an environment, one would hope that I do not "think the results will just fall out." I think much of this will be very difficult to program and that the software must be capable of 'learning' and that it will require a human to interact with it so that the job can be done well and the software upgraded to do the job even better in future. I actually wrote several scripts like this many years ago. They are archived on my CDs, but my manifest was damaged in a fire, so I am not sure where to even begin to retrieve them. It would be faster and easier to just start from scratch than to dig up outdated software anyway, so I do not regard it as a great loss. >> I'm not sure I'm ready to program an AI engine >> capable of "listening to the book." >the software can't listen to the book. _you_ have to do that. I pretty much said that in what you snipped. :) >> Which is why I do not believe in leaving people out of the process >again, you have a profound misunderstanding of the process. I'll just laugh and pretend you didn't say that.... Perhaps I have a profound misunderstanding of _your_ proposed process, but I have a very firm understanding of my own. There is no _the_ process for me to understand. In any case, if you would like to create your own project, feel free. "The road is free to all." And Michael and PG would be happy to recieve the texts submitted through any process. And I would be happy to receive constructive feedback on my efforts as well from any source that cares to give it. I am just interested in creating a method for people to complete initial editions of books rapidly, to avoid putting features in place that 'waste' the time of volunteers in doing tasks that could be done more rapidly and easily through the use of software, and to have an environment that offers the opportunity to correct any errors that were missed in the initial release. Also, it should offer a sense of community for volunteers to interact. The _process_ is not a complex one by any means. As a human, I expect that my project process will not be perfect nor will it agree with the opinions of all others on how it _should_ be done, but it will be a process that produces books. So, it will be part of the solution no matter how many problems it has. :) Nor do I feel that the process at DP is flawed. They are doing the process the way they have chosen to do it and it is a project that acutally exists and is productive. They simply created a bottleneck in their workflow that they need to deal with. I actually hope that some of my tools are of assistance to those who use DP to process books. I am about getting books into PG and care very little how one goes about doing it so long as one _is_ going about doing it. :) Carel

1 0

Re: Processing eTexts
by Bowerbird＠aol.com 26 Feb '10

26 Feb '10

carel said: > Direct comparison to the scans is part of > the formal proofing process and, > although available during the 'quest for perfection,' > should not be a forced issue. i disagree. it's largely a matter of "framing" the issue and its perception. i don't think we should ask the user to help with "proofing" per se. i don't want them to think of this as "the formal proofing process". i believe we need to put the books in front of them _to_read_... _that_ should be our understanding with them, how we "frame" it. they're there because _they_ want to _read_ this particular book. now, as part of the interface by which we present the book to them, we give them both the digital text _and_ the page-scan, together... most interfaces make the user _choose_ one of the other, but i see no reason (especially no _benefit_) in doing it that way... i think it's wise to give them _both_, and have them understand that there may have been some transcription errors during digitization, so we're giving them the page-scan too, for any cases they _suspect_ might be errors. in most cases, they'll see that we matched the scan. or they'll understand why we made the change that we did, if we did. _or_ -- in some rare cases, in which we request their assistance -- they'll find an actual transcription error that we made, and _tell_ us. this is the thing that project gutenberg doesn't do well. it doesn't _invite_ the reader to become a part of the transcription process, to become one of the instruments that marches texts to perfection. indeed, you can _feel_ deep in your gut the _contempt_and_scorn_ that jim tinsley and al haines have for the stupid people who _do_ report errors, because _half_ of those "errors" are not, in fact, errors. well, gee, big surprise. you haven't given them the page-scans, so how in the world are they gonna know if an apparent error _is_ real? > I think calling it a wiki makes it easier for quick comprehension > rather than explaining the whole process of storage, editing, etc. we should never _have_to_ "explain the whole process" to the public. for them, it should be as easy as clicking a button to say "check this!" they shouldn't even need to explain the error. (we can find it, right?) > Rather than working an editing interface > into an out-of-the-box wiki system we don't need to make "an editing interface"... > Has anyone ever just asked google if they could > use the scans and OCR text for PG? michael had a meeting with google when they were just starting out... he reported that they treated him rather rudely, hustling them out of their famous cafeteria before he had even finished his lunch. it's sad, but i don't think they really appreciate project gutenberg very much... > XML makes a nice standard for storage and conversion it's obtuse, and unnecessary obstructive to the text itself... > My only protest is that I believe > a human proofing stage is still a good choice. if you mean a word-by-word reading of the page by a person who has no real interest in reading the entire book, i disagree. a comparison of different digitizations is a much better process, in terms of finding errors, and it takes less work, and is more fun. i can't think of one single reason for a word-by-word proofing, let alone two or three rounds of it. it's simply not efficient, and anybody who explores the options can learn that for themselves. between aggressive clean-up of o.c.r., and the comparison method, and "smoothreading" by people from the general public, we're good. > Otherwise, I would say to have two people > run the same correction process, diff the results and > then present the mis-matches to a third set of eyes. close... but it's unnecessary for "people" to do the digitizations. two sets of cleaned-up o.c.r., diffed against each other, is enough... (but yes, it's necessary to have a sentient human doing the clean-up.) > the existance of a proofing round > creates a sense of belonging to the project and > could assist in recruiting people to try the 'harder' process of > running the 'automation' scripts and submitting books word-by-word proofing is a slow, boring, humdrum task. o.c.r. clean-up is a fast-paced, active, and energetic task. which does _not_ mean that it's "harder". just more fun... doing the former task won't help you at all with the latter. (except perhaps make you _extremely_ grateful that you don't have to do such a boring, miserable job ever again.) > People need a sense that they are not just > assisting tools to create book editions, > but are a fundamental part of the process. is the hammer more important than the carpenter? i doubt it, whether you ask the hammer _or_ the carpenter... i get the impression you think these clean-up tools are just a bunch of scripts that people turn on, and the results fall out. but that's not anything like what actually goes on. > I'm not sure I'm ready to program an AI engine > capable of "listening to the book." the software can't listen to the book. _you_ have to do that. > Which is why I do not believe in leaving people out of the process again, you have a profound misunderstanding of the process. the hammer can't do a thing without the active involvement of the carpenter. even then, the carpenter has to know what s/he is doing, or the hammer won't be able to do one bit of good... the best thing you can do, carel, is to play around with twister... -bowerbird

1 0

Processing eTexts (was RE: Re: d.p.'s undeserved superior attitude about "quality")
by cmiske＠ashzfall.com 26 Feb '10

26 Feb '10

Bowerbird said: >carel said: >> I've read your posts for many years > >so _you're_ the one who's been reading them! ;+) I'm sure many people read them. :) >> I'm not sure the most efficient system on earth >> could keep up with the scan rate. > >sure we can. just have 10 million people do 1 book each. > >piece'a'cake. It sounds easy when you put it that way.... ;) >> And, if you speed things up by skipping the manual proofing >> and use automation completely for the first release then you >> will have text little to no better than what google already has. > >oh, don't kid yourself. google's text is _much_ better than >what they're showing to us members of the general public. Their OCR software requires training, so I agree with you that they always have an at least one step better version on the backburner. They probably have a stepped up version of everything we see.... >i haven't been impressed by the wiki approach, honestly... The standard wiki layout is not a proper environment for editing. Better would be something that allows the vast majority of readers to just read the page and click a button to be presented with an interface for making/submitting corrections if such is required. Direct comparison to the scans is part of the formal proofing process and, although available during the 'quest for perfection,' should not be a forced issue. I think calling it a wiki makes it easier for quick comprehension rather than explaining the whole process of storage, editing, etc. Rather than working an editing interface into an out-of-the-box wiki system, it would be better to write a proprietary system with some of the social networking features of a wiki. That part would take a very firm second place to just getting some tools out there for people to produce decent quality texts in a minimal amount of time. Because of that fact, I haven't really given much thought to the 'wiki' interface as it will evolve by itself from wants/needs. [snip] in re: google: >people on this listserve have been cut off for scraping... >you can still do it, of course, but you have to be careful. Has anyone ever just asked google if they could use the scans and OCR text for PG? In regards to the scans, I believe they just require you to leave the branding. >i would bother to nitpick certain parts of it, like the x.m.l., >but really, my objection is at the more fundamental level. XML makes a nice standard for storage and conversion and can be human readable. I'm not one of those people who believes that XML is the answer to everything, but I do think it works well for this use-case scenerio. Opinions on this will vary, etc. >yes, i believe every page of every book should be online, >on its own webpage with unchanging u.r.l., text and scan, >with an error-report form that the general public can use >to detail any problems with the page, or ask questions, or >even make annotations and have dialog about that page... Agreed. >but i also believe that the text for the vast majority of these >pages should be _perfect_ at the time that text is mounted. We are in agreement on this point as well. The majority of the work towards corrections should be done via automation and human interaction with a software process so as to reduce the error level to a minimm before it is put in the hands of a proofreader (or the general public). >and i've demonstrated, repeatedly, that this is fully possible. It is very possible. [snip] >so here's how i see things going... >when you find a scanset online of a book you want to digitize, >you'll download it, along with the o.c.r. (if it comes from a site >that won't give you the o.c.r., you can upload it to archive.org, >and they'll o.c.r. it for you. providing you have no o.c.r. app.) >you'll use a handy-dandy tool to digitize it in an hour or two, >and then you'll make the results available to everyone online. My only protest is that I believe a human proofing stage is still a good choice. For one thing, it would be a way of catching errors that could be induced by the human assisted automation process. Otherwise, I would say to have two people run the same correction process, diff the results and then present the mis-matches to a third set of eyes. If that doesn't result in a book that is ready for release, I'm not sure what would. Ready for release != perfect. :) On a point in favor of the proofing round that really has nothing to do with the potential for automation, the existance of a proofing round creates a sense of belonging to the project and could assist in recruiting people to try the 'harder' process of running the 'automation' scripts and submitting books, etc. People need a sense that they are not just assisting tools to create book editions, but are a fundamental part of the process. If you try to proof after a release has been done, there will be little interests in it because you lose the motivation to 'get the book out there." >and carel, since you're interested in programming such tools, >we'll have plenty to talk about... Yes. :) It's very much a 'will do' at this point. I just have to survive my mid-terms. I look forward to your feedback and your input on the project. >well, like i told gardner just the other day, i really don't >_have_ any collection of scripts. i use a text-editor and >start looking at the book. i see an error, and devise a >global search-and-replace to deal with that type of error. >when they're all fixed, i look again to find the next error. That brings up the point that pretty much all of what my scripts would be doing could be done in word processing software.... >i think of this process as "listening to the book", in that >the book itself will tell you what kinds of errors it has... I like that. I'm not sure I'm ready to program an AI engine capable of "listening to the book." Which is why I do not believe in leaving people out of the process, but rather giving them tools to assist them with the process. And, the tools can 'learn' from the people who use them. I'll be busy finishing up a group project for my classes, so it'll be late March before I get any real time to work on scripts for this project. I'll work on some of the research, logic, and flow in the meantime so that I am ready to roll when I do have the time. Back to my programming.... Carel

1 0

roundlessness -- 008
by Bowerbird＠aol.com 26 Feb '10

26 Feb '10

by the way, i've coded an offline program that hooks into rfrank's system. if anyone would be interested in using such a beast, take it up with roger. or, if you just want to use it to scrape the books on his site, talk to me... -bowerbird

1 0

roundlessness -- 007
by Bowerbird＠aol.com 26 Feb '10

26 Feb '10

ok now, remember how i said that preprocessing is a _simple_ thing, typically with no need for a large set of checks that need to be done? on the current chunk of text i'm analyzing from rfrank's experiment, it looks like a dozen simple checks would've found 75% of the errors, right off the bat, with no need for a word-by-word proofing round... save the time and energy and _focus_ of your proofers: preprocess. -bowerbird

1 0

Re: the d.p. opinion on "prerelease" of e-texts
by Bowerbird＠aol.com 26 Feb '10

26 Feb '10

donovan said: > I stand by my characterization of PG's pre-press scheme as > both unnecessary and a disturbing shift in attitude towards DP. i don't have a dog in this fight. but i'm glad it's all "amiable" and such. -bowerbird

1 0

Our 400th Portuguese eBook
by Michael S. Hart 26 Feb '10

26 Feb '10

We are up to #398 and I have not received any suggestions for #400. Should we reserve one? Just go with whatever comes along? Other suggestions? Thanks!!! Michael

2 1

Re: the d.p. opinion on "prerelease" of e-texts
by Bowerbird＠aol.com 26 Feb '10

26 Feb '10

david (donovan) said: > All that's required is to convince the 'right' people at DP to either > A) expend limited resources towards that end > _instead_of_ where they're needed, or, > B) stand aside and allow PG to take unreasonable advantage of > what have _so_far_ been amiable terms of relationship. emphasis added... here's an obvious ploy to push things to the ultimatum stage... behold the biggest bottleneck at distributed proofreaders: donovan. -bowerbird

2 1