- gutvol-d - lists.pglaf.org

Re: Processing eTexts
by cmiske＠ashzfall.com 03 Mar '10

03 Mar '10

1 0

Re: Processing eTexts
by cmiske＠ashzfall.com 03 Mar '10

03 Mar '10

1 0

roundlessness -- 009
by Bowerbird＠aol.com 03 Mar '10

03 Mar '10

in our "glass-is-one-quarter-full" news today, i note that rfrank has this to say about using reg-ex tests on his roundless site: > It seems to be a big win to make REs that usually are used > during post-processing available to users during proofing. now if roger would realize those reg-ex checks would be even _more_ useful if they were done in book-wide preprocessing, we could award him the "glass-is-three-quarters-full" prize... but let's be thankful for the huge progress he's made thus far. -bowerbird

1 0

Re: Processing eTexts
by Bowerbird＠aol.com 03 Mar '10

03 Mar '10

so, carel, i hope i didn't scare you away... ;+) -bowerbird

1 0

Re: the d.p. opinion on "prerelease" of e-texts
by Greg M. Johnson 02 Mar '10

02 Mar '10

From: don kretz <dakretz(a)gmail.com> > And what's the message that we send when we use someone > else's work (the book) that someone else scans, and > someone else collects, posts, and manages (TIA) and a > bunch of other people proof and/or format, and then keep > that accumulated and integrated value that's been generously > and freely provided for us to use locked away exclusively > for several years for one Post Processor to work on, > when they get around to it? Is something else happening to the work during this time-- like papers, etc., being written on it? -- Greg M. Johnson http://pterandon.blogspot.com

1 0

Re: the d.p. opinion on "prerelease" of e-texts
by Bowerbird＠aol.com 02 Mar '10

02 Mar '10

jim quoted someone as saying: > The negative reaction you're getting > is to your tone and tactics, not your news flash. now, i didn't see this quote when it came up originally, meaning that it probably came from someone who is in my spam folder, which would mean marcello or zora. i'm betting it's zora, the supreme apologist here for d.p. this is the kind of "blame-the-messenger" crap that they _love_ to do over at d.p. they can't argue with the message, so they talk about your "tone" instead, when it's their own damn fault that you had to adopt that tone in the first place, because they're bound and determined to ignore you totally. and that's because they are incapable of solving anything. and it's interesting to see _why_ d.p. can't solve anything, as the d.p. people here -- right on up to board member newby -- are unable to avoid dragging a thread off-topic. (although we must give marcello credit for a serious detour, by raising a phantom that files are being _stolen_ from d.p.) i mean, seriously, you want to witness something _amazing_, just take a look at recent posts in this thread, were _jim_ is the one who manages to (a) stay on topic! and (b) make sense. jim!, for crying out loud, the same jim who often has difficulty arguing his way out of a wet paper bag, and he's the one here who is doing the _best_, absolutely outshining all of the rest! so, of course, let's attack jim, and his "tone and tactics"... let me break it down to a nutshell... d.p. has thousands of proofers doing p1, the first proof pass. d.p. has hundreds of proofers in p2, the more-careful pass. d.p. has dozens of proofers for p3, the "final final pass" pass. i don't know about you, but i'd expect that a "final pass" will take a closer reading (and thus more time) than a first-pass, but assume p1 and p2 and p3 proofers all take the same time. somehow, however, the fundamental workflow at d.p. expects dozens of p3 people to keep up with thousands of p1 people... this is ridiculous on the face of it. and this is the main problem. or perhaps the _main_ problem is that the d.p. "powers that be" could seriously install such a ridiculous-on-its-face workflow... whichever way we look at it, it's purely and absolutely ludicrous. (quick, somebody please give me more synonyms for "ridiculous.") -bowerbird

1 0

nathan hale -- 001
by Bowerbird＠aol.com 01 Mar '10

01 Mar '10

ok, in discussing one of the books on rfrank's roundless test-site, i remarked that the scans were badly done, uncharacteristic of roger. much more in keeping with his typical level of quality are the scans from a new book on the site, a biography about nathan hale, so i've scraped and remounted those scans on my site, and will do the book. > http://z-m-l.com/go/nhale/nhalep123.html > http://z-m-l.com/go/nhale/nhale.zml as i've been doing lately, the last page of the book shows the changes that i made to the file to clean it up, to show people how simple it is... this book has a lot of correspondence in it (so it has salutations and signatures and stuff like that), and i haven't done such books before, so i'm gonna have to figure out how to handle all of that in z.m.l., which means looking at a bunch of p.g. e-texts to see how people have represented them in the .txt versions up to this point in time... meanwhile, i'm just dealing with them in a presentational manner, either left-justifying or centering or right-justifying, as appropriate. it should be easy to see how i've indicated that by viewing the pages. (i simply used underbars and equal-signs to indicate each of them.) the book still needs some work. but even now, it's a demonstration of how quickly one person can make a book available to the public... -bowerbird

1 0

or a watch to take out of it
by Bowerbird＠aol.com 01 Mar '10

01 Mar '10

ok, so there's a new e-book viewer-program in town. liza daly, who is affiliated with the o'reilly folks, has brought out "ibisreader" for your reading enjoyment. so i went to check it out... at ibisreader.com, i clicked "get started" and then -- at "add a book" -- "feedbooks: popular public domain". the 4th book down is "alice's adventures in wonderland", and since the movie is coming out this friday, i got that. feedbooks, as you probably know, is a site that takes e-books from various sources, like project gutenberg, and makes some very nice-looking versions of them... since feedbooks re-works the books, they don't have as many books as some of the other sites, but they are preferred by many people because their books look nice. so i start looking at the text, and i find this: > There was nothing so very remarkable in that; > nor did Alice think it so very much out of the way > to hear the Rabbit say to itself “Oh dear! Oh dear! > I shall be too late!” (when she thought it over > afterwards, it occurred to her that she ought to > have wondered at this, but at the time it all seemed > quite natural); but when the Rabbit actually > took a watch out of its waistcoat-pocket, > and looked at it, and then hurried on, > Alice started to her feet, for it flashed across her mind > that she had never before seen a rabbit with > either a waistcoat-pocket, or a watch to take out of it, > and, burning with curiosity, she ran across the field after it, > and was just in time to see it pop down a large rabbit-hole > under the hedge. well, gee. if you're intimately familiar with this book, you know > took a watch out of its waistcoat-pocket, is a phrase that is _italicized_ in the book. but not on this file... here's where you can see a copy of the original: > http://www.archive.org/stream/alicesadventur00carr#page/2/mode/2up you'll have to take my word for it that it's not italicized in the feedbooks copy that is being used by ibisreader, since i don't see any convenient way for me to link to a specific page there. (but it's book #22 from feedbooks, if you wanna look yourself.) it's pretty clear that what has happened here is that feedbooks has taken pg#11 and used it as its source. what is a sad thing, because -- even after all the years it coulda been "improved" -- pg#11 _still_ doesn't have proper italics in it. it was "updated" in 2005 (leaving no trace behind) and then once again in 2008 (when an .html version was added). but it still has zero italics. > http://www.gutenberg.org/files/11/11-h/11-h.htm > http://www.gutenberg.org/files/11/11.txt the p.g. version _does_ have italics rendered as all-uppercase. so there is _some_ indication of them. but this is ambiguous, since there are places in the book where uppercase is used too: > http://www.archive.org/stream/alicesadventur00carr#page/4/mode/2up (see the reference to "orange marmalade", in all-uppercase.) if i would've been feedbooks, i would've converted _all_ of the uppercased words to italics. but of course then they would've been changing things like chapter headers and first-words too. and of course, feedbooks could've _left_ words in all-uppercase. i don't know why they didn't, but i assume it's because they take _pride_ in their typography, and all-uppercase looks like crap... (but it's true that the tell-tale examples of _actual_ uppercase, namely "orange marmalade", "drink me", and "eat me", are all still rendered in uppercase in feedbooks#22, so i am stumped.) and yes, folks, i know there are other versions of "alice" posted, including some with italics correctly specified. so let us look... pg#19033 _does_ have italics in it, but it does _not_ italicize the phrase about the rabbit, and his watch, and his pocket... whether it's a version-difference -- it _is_ another version -- or whether it's just a digitization mistake, i simply don't know. (and since this is clearly not the explanation for the feedbooks discrepancies, i have no interest in determining the reasons.) pg#928 -- an .html version only -- _does_ have italics. yay! pg#28885 also has the italics, and it has the images as well! although sometimes things don't go right, as shown here: > http://z-m-l.com/misc/alice-glitch.png the feedbooks version has _no_ italics at all, as far as i can see, not even the different set of italics from pg#19033. so, at first, i had thought they'd used pg#11 as their source text, but now, i'm not so sure. (they have some strange contractions, which could indicate that they might've done their own digitization.) at any rate, the point still stands that pg#11 lacks proper italics. all-uppercase is _not_ a substitute. and since the italics _have_ been done -- in pg#928 -- the changes should be incorporated into pg#11. the presence of other versions, at higher numbers, won't change the fact that pg#11 is considered as "the original." so, c'mon, let's see someone who talks about pg/dp "quality" and "incremental improvement" actually _back_up_ the claim. let's get this classic and legendary e-text cleaned up now, ok? -bowerbird

1 0

Re: Processing eTexts
by Bowerbird＠aol.com 27 Feb '10

27 Feb '10

well, it's kinda silly for us to talk about this in generalities, because i'm capable of demonstrating what i mean specifically and exactly... and when i'm done, i'm quite sure that you will agree completely. (and you might say that's what you meant all along, for all i know.) so let's get specific. i've posted a file of text taken from rfrank's roundless experiment: > http://z-m-l.com/go/campf/campf-001.zml as you can see, this book is called "pemrose lorry, camp fire girl"... the book is now in-progress, so it's half-done and half-undone... (it looks like it's been proofed up through about page 80 so far, but since the pages go out multiple times, it's difficult to be certain of it, but after page 80, it's fairly clear that the pages haven't been edited.) that first version had the page-separators fixed, and other basic stuff. the next version has more stuff cleaned up, but it's still rather basic: > http://z-m-l.com/go/campf/campf-002.zml as you can easily see, if you look through either one of the versions, starting after page 80, there are a bunch of paragraphing problems. specifically, there are often blank lines inserted -- incorrectly -- between the lines of a single paragraph. you can find instances of this problem on pages __, __, __, and __. it's a not uncommon glitch. paragraphing is one of the first things i try to correct, because it's necessary to have the paragraphs correct to fix any spacey quotes. fortunately, it's rather easy to locate these bad paragraphs via search. find a blank line followed by a line that starts with a lowercase letter. (in other words, you search for two newlines followed by lowercase.) now, one way of doing the search would be to automatically replace _all_ of these occurrences by simply deleting one of the two newlines. you could click "replace all" on that find-and-replace, and change all of the occurrences without even looking at them. you _could_ do that. the proper changes would almost always outnumber improper ones. whenever anyone talks about "taking the human out of the equation", blind global changes is what immediately springs into my little brain. and i am definitely _not_ a fan of blind global changes. there are some changes you can make blindly, but they are few and far between, and they certainly don't characterize the usual process. and this change -- deleting those excess lines -- i don't do blind... i will step through the changes one-by-one, and look at all of them _against_the_scan_. (even though, in most cases, i wouldn't need to look at the scan, because it's pretty obvious how to fix the problem.) sometimes i even delete the excess line manually, rather than merely approve the find-and-replace, just to keep my grubby fingers busy... but -- here's the very important part -- i do a lot _more_ than that. at every one of these spots to which i'm automatically transported, i look around the immediate problem area, to see if it spilled over. and, if it did, then i will clean up that neighboring area right away. so i'm walking a fine line here between doing a specialized search and a general correction routine. i start with the specialized check, so i have a laser focus on the nature of what the check will show me, so i don't have to waste mental energy figuring out what the glitch is. but then i also use my peripheral vision to see what else needs fixing. and this particular check in this particular text is a very good example. because it ends up that a _lot_ of the surrounding areas had problems. (that's not uncommon with these excess blank lines, because it's often the other glitches that _caused_ the o.c.r. its paragraphing difficulties.) there were a good number of problem-lines caught by this routine -- about 85 if i remember correctly -- but i probably corrected the text on 85 surrounding lines as well, because i saw bugs, so i fixed them. i encourage people to "play along at home" and _do_ this search on this text, so you can really see how you can spot neighboring glitches. what you will _not_ know, however, unless you are also looking at the _scan_ for each page, is that there were often _entire_words_missing_ from the o.c.r. they were cut off entirely from the left-hand margin. this isn't the kind of thing for which you can do a search to find them. they're just missing from the o.c.r., and you don't know they're gone. however, since you are paying attention during this _other_ search, you can become aware of these missing words, via peripheral vision. if you _do_ "play along at home", and actually step through that check, you will also see that there's yet another check that needs to be done to catch _all_ of these paragraphing problems from excess blank lines. this second check is the "flip side" of the first one. the first one was looking for supposedly-first-lines of paragraphs that were lowercase. the second one checks the _termination_ of supposedly-last-lines... so you're checking for a letter or a comma at the _end_ of a line which is followed by a blank line. if it were _really_ the end of a paragraph, as is implied by that blank line, it should be punctuation-terminated. so you're looking now for a letter/comma followed by two newlines... this search turns up roughly 50 instances in this particular text-file. and, once again, when you make these corrections, you look around. roughly half of the 50 instances were cases where the blank line that was deleted was followed itself by a line beginning with a single-dash, which was a misrecognition for an _em-dash_ at the start of that line. (the em-dash is represented by a double-dash.) so those corrections were made at the same time that the excess blank line was eliminated. if you were a party-pooper, and didn't play along at home, i have uploaded a copy of the file where the lines that were flagged by the first page are indicated by an "@" sign, which you can search for: > http://z-m-l.com/go/campf/campf-003.zml and here's one with the lines from the second check tagged with "%": > http://z-m-l.com/go/campf/campf-004.zml you can also view the book page-by-page, as usual. for instance: > http://z-m-l.com/go/campf/campfp123.html (you will see, right there on page 123, an example of a page which has some of the words cut off from the right-margin. when i find this type of problem in a scanset, it just makes me want to scream! when you scan a book, you need to make sure you do it _carefully_. sloppy work such as this is tremendously uncharacteristic of rfrank.) at any rate... it's this "looking around and fixing other errors in the neighborhood" that makes this whole preprocessing thing such a _vibrant_ procedure. ...which brings me squarely back to the major point of this post... *** so yeah, i'm doing a computerized check, and it's darn _efficient_, but i'm surely not "taking humans out of the process". not by a long shot. i am a mentally alert human, who is _actively_ engaged with the text... my cursor is _flying_ all around that document. i move at warp speed. and i am _efficient_ too. i am _tremedously_ efficient. i am rocking! all the other carpenters are using hammers, and i have a _nail-gun_. part of the efficiency is due to that _focus_ that i just talked about... but most is simply because i am being directly transported to errors. i don't waste any of my time _looking_ for them; they're _presented_. i don't waste any time positioning my cursor -- it is _prepositioned_. and the scan is right there, ready for me to look at it, immediately... and it's _fun_. downright _exhilarating_. like driving a sports car. (whereas word-by-word proofing is akin to pushing a baby stroller.) and the results are _better_. not just a little bit better, a lot better. i've shown time after time after time that i can find (lots of) errors proofers _miss_, not just in one round, but two and three and four. and, just to repeat the message yet again, it's _simple_ to do this. a couple dozen _simple_ checks finds a high percentage of errors. so yeah, i know i am explaining all of this with accurate language. but people always seem to misunderstand. on the one hand, they think the tool is making all the changes... on the other hand, they think you need a human to find errors... both of those positions are wrong, and when i argue against one, people jump to the other, and then i have to argue against _that_. i know that i am capable of walking the tight-rope between them, but everyone else seems to get tangled up in the dialog semantics. the tool can find almost _all_ the errors, so you don't need to look. (you're welcome to look, and i'm sure that you'll find a few errors, but at some point, you have to ask whether it was worth your time.) on the other hand, _you_ have to be the active agent that mentally makes the decision whether or not each change should be made... like i said, there are only a few global changes that i make blindly. (to give you a for-instance, i will globally delete any space that follows any doublequote located at the very beginning of a line, or any space preceding a doublequote at the very end of a line. i'll also blindly change spacey-quotes if the open/close pairing within the rest of the paragraph makes the change unambiguous. i'll also close up contractions blindly. maybe a few more things. but for the most part, i really want to look at the changes i make. maybe if i had the luxury of knowing i'd have human volunteers following up with a word-by-word proofing of everything i did, i would be more willing to make blind changes. or maybe not, because that just seems wasteful of donated time and energy.) now, i'm sure that dkretz will confirm what i've been saying here, as his experience with his tool has led him to the same thoughts. and maybe carel will say that this is exactly what she has meant, and we've been doing a semantic dance around the same thought. -bowerbird

1 0

Re: Processing eTexts
by cmiske＠ashzfall.com 27 Feb '10

27 Feb '10

Bowerbird said: >i don't want them to think of this as "the formal proofing process". >i believe we need to put the books in front of them _to_read_... Which is exactly why a book that is ready for release to PG should not force upon an individual the original scan. When the reader spots something they think is an error, they should be able to call up the scan to see if it is an actual error before reporting it as such, etc. Otherwise their reading is distracted by the scan. A proper interface would allow users to select what they want to see. An interface should be about options. In any case, I have no desire to host an archive that duplicates the content of PG and others. The purpose in coming to my project site would be to prepare books for release to PG and to polish released books so that they can be read for leisure and pleasure off of _my_ bandwidth. >> I think calling it a wiki makes it easier for quick comprehension >> rather than explaining the whole process of storage, editing, etc. > >we should never _have_to_ "explain the whole process" to the public. >for them, it should be as easy as clicking a button to say "check this!" >they shouldn't even need to explain the error. (we can find it, right?) Of course, but it is hard to develop it and request assistance with the development of it without explaining the basic concepts, etc. A project must be explained and defined or it cannot be realized. >> Rather than working an editing interface >> into an out-of-the-box wiki system >we don't need to make "an editing interface"... It's rather hard for people to make/submit corrections without having an interface for doing so. >> Has anyone ever just asked google if they could >> use the scans and OCR text for PG? > >michael had a meeting with google when they were just starting out... > >he reported that they treated him rather rudely, hustling them out of >their famous cafeteria before he had even finished his lunch. it's sad, >but i don't think they really appreciate project gutenberg very much... That is too bad. >> XML makes a nice standard for storage and conversion > >it's obtuse, and unnecessary obstructive to the text itself... It's also one of many methods for the storage of 'intelligent' data and is easy to do. Once put in place, it would never be seen again by anyone except someone correcting the markup (which one hopes would be rare). InDesign uses markup, MS Word uses markup, and on and on. I don't think the users of this software find the markup obstructive to the text itself. >> My only protest is that I believe >> a human proofing stage is still a good choice. [snip] >i can't think of one single reason for a word-by-word proofing, >let alone two or three rounds of it. it's simply not efficient, and >anybody who explores the options can learn that for themselves. I don't believe I said anything about multiple rounds of proofing, etc. I believe I discussed the main processing of a document being done through human interaction with scripts and then people actually looking at the results of the process before the output is released (proofing). I feel that a human looking at a smaller subset of a large document is a good thing in the error finding process. You apparently do not think it is. Neither of us is right or wrong: It is a matter of perspective and opinion. >between aggressive clean-up of o.c.r., and the comparison method, >and "smoothreading" by people from the general public, we're good. Yes and no. It depends on your definition of 'good.' >> Otherwise, I would say to have two people >> run the same correction process, diff the results and >> then present the mis-matches to a third set of eyes. >close... but it's unnecessary for "people" to do the digitizations. >two sets of cleaned-up o.c.r., diffed against each other, is enough... That depends on a lot of factors including the assumption that two OCR programs would not make the same mistake and that, if they did, it would be a mistake that could be caught by another process. Etc. >(but yes, it's necessary to have a sentient human doing the clean-up.) Exactly. And, if that human makes a major error, the entire text in now a mess. If two people do the same process it reduces, but does not eliminate, the risk for 'the human error factor.' >o.c.r. clean-up is a fast-paced, active, and energetic task. >which does _not_ mean that it's "harder". just more fun... People actually do enjoy proofing. You do not. Again, neither is wrong in their viewpoint. Volunteers should be involved in a manner that gives them pleasure and satisfaction, etc. I say to give people tools and options and let them be the judge of what is best for them. >i get the impression you think these clean-up tools are just >a bunch of scripts that people turn on, and the results fall out. >but that's not anything like what actually goes on. Since I am planning to write scripts for such an environment, one would hope that I do not "think the results will just fall out." I think much of this will be very difficult to program and that the software must be capable of 'learning' and that it will require a human to interact with it so that the job can be done well and the software upgraded to do the job even better in future. I actually wrote several scripts like this many years ago. They are archived on my CDs, but my manifest was damaged in a fire, so I am not sure where to even begin to retrieve them. It would be faster and easier to just start from scratch than to dig up outdated software anyway, so I do not regard it as a great loss. >> I'm not sure I'm ready to program an AI engine >> capable of "listening to the book." >the software can't listen to the book. _you_ have to do that. I pretty much said that in what you snipped. :) >> Which is why I do not believe in leaving people out of the process >again, you have a profound misunderstanding of the process. I'll just laugh and pretend you didn't say that.... Perhaps I have a profound misunderstanding of _your_ proposed process, but I have a very firm understanding of my own. There is no _the_ process for me to understand. In any case, if you would like to create your own project, feel free. "The road is free to all." And Michael and PG would be happy to recieve the texts submitted through any process. And I would be happy to receive constructive feedback on my efforts as well from any source that cares to give it. I am just interested in creating a method for people to complete initial editions of books rapidly, to avoid putting features in place that 'waste' the time of volunteers in doing tasks that could be done more rapidly and easily through the use of software, and to have an environment that offers the opportunity to correct any errors that were missed in the initial release. Also, it should offer a sense of community for volunteers to interact. The _process_ is not a complex one by any means. As a human, I expect that my project process will not be perfect nor will it agree with the opinions of all others on how it _should_ be done, but it will be a process that produces books. So, it will be part of the solution no matter how many problems it has. :) Nor do I feel that the process at DP is flawed. They are doing the process the way they have chosen to do it and it is a project that acutally exists and is productive. They simply created a bottleneck in their workflow that they need to deal with. I actually hope that some of my tools are of assistance to those who use DP to process books. I am about getting books into PG and care very little how one goes about doing it so long as one _is_ going about doing it. :) Carel

1 0