February 2010 - gutvol-d - lists.pglaf.org

Re: !@!!@!!@!Re: Re: so what is so important about pagination?
by Bowerbird＠aol.com 17 Apr '10

17 Apr '10

michael said: > I had to have read the whole thing > to get to the part I quoted. . .duh! you can quote something without reading it. but i should have said i wish you would have _understood_ what i was saying, because then you would have understood that your response didn't address the point that i was making. > When you ask people to pay attention, it helps to PAY ATTENTION. i am paying attention, michael. even though i've heard what you're saying lots and lots and lots of times before. and it makes sense when you are addressing the people who are making _other_ points. but they left long ago... > It addresses EXACTLY the point you made that I quoted. . . . no, it doesn't. > Then SAY that!!! Right up front in plain language!!! i _have_ said it, every time i've talked about this issue... including once when marcello brought up this same point you brought up (think about that, michael), about editions. (it was back in september of 2007, if you're curious.) marcello pointed out that the various different editions of "pride and prejudice" had different pagination, and he asked which of the editions should be used to do the pagination... here's the reply i made to marcello's point: > http://z-m-l.com/go/pap/pride%20and%20prejudice(4).txt if you don't care to read all of it, here are the guts of that reply: > the answer to the question as to which set of linebreaks and > pagebreaks to use is this: the ones in the edition you digitize. > > plain old common sense. if you didn't already know the answer, > perhaps you might want to exercise your brain a little bit more... > > if you're digitizing the 1844, use its linebreaks and pagebreaks. > if you're digitizing the 1853, use its linebreaks and pagebreaks. > if you're digitizing the 1870, use its linebreaks and pagebreaks. > if you're digitizing the 1892, use its linebreaks and pagebreaks. and here is a web-page showing the first page of those 4 editions: > http://z-m-l.com/go/pap/pride_and_prejudice(4).html and yes, in case you're wondering, if a p-book was important enough to go through different editions, we should digitize _every_ edition... i'm not going to tell you which of those 4 editions you _should_ use, which one is the "right" one. whichever one you want to use is "right". and you should be able to determine if any specific e-book _does_ or _does_ not match the edition you want to use, or some other edition. *** by the way, there's another web-page of interest in this directory: > http://z-m-l.com/go/pap/jose(4).html this fascinating page shows some work done by jose menendez. jose adopted my suggestion that the e-book be able to mimic the p-book, and he created a series of .pdf books that did just that... shown on this web-page are some screenshots of his .pdf-books, compared to the page-scans from those pages. he did a great job. of course, since a .pdf-book is unable to reflow its text, jose's work doesn't fit the more-important criterion of reflowability, but it does show that the ability to mimic the original can be extremely valuable. > However, that still relegates us to being a Xerox machine, no? no. because a xerox machine can't do reflow. or fix typos. or pull in spacey contractions. or change the font, or size. look, i understand the appeal of digital text _extremely_ well. i've made all of the arguments myself, so there's no need for you to repeat 'em back at me, you're just wasting your breath. but there's a problem looming here, a problem that the future will have to face, and solve, and i'm telling you what you need to do, so that you can _help_ the future _solve_ that problem, such that your e-texts will continue to be used, and not tossed. i'm on _your_ side, michael... i've got your back, good buddy... so you need to get that through your skull and start listening... > I'm never going to get into any of these semantic arguments!!!!!!! > Mimic means to copy as closely as possible. . . . Synonym: copy. it's not a semantic argument, michael. it's protective coloration. if your copy isn't capable of _assuming_the_look_and_feel_ of the thing that it _purports_ to be copying, nobody will trust it. you seem to be forgetting that you are claiming to _be_ a copy. perhaps you are an "improved" copy, but you are _still_ a copy. certainly if you came out and said "we rewrote parts of the middle, because the original was too boring", you would expect that people would throw you away. but what if someone points out a few errors in your work, and says "see, you can't trust this work, it hasn't been faithfully transcribed," then what is your defense? you can say that "it was just a few errors", but what if they then point out a few more, and a few more after that? at what point can you no longer expect the end-user to believe you? > As I have said before, if you would listen, > I am not AGAINST keeping a copy with such pagination > for such purposes well, good, and bully for you, and all that, but the fact of the matter is that project gutenberg is not, at this point in time, actually doing that. > but I draw the lines, pun intended, at keeping every character > in the same page position when there is no need for pages, > in all available PG editions. and, as i have said before, if you would listen, i'm not suggesting, in any way, shape, or form, that pagination and linebreaks need to be kept "in all available p.g. editions". that'd be stupid, absolutely and totally and ridiculously _stupid_, and i don't usually feel a need to rule out stuff that is absolutely, totally, ridiculously stupid... > I want our eBooks to be optimally readable: > Minimal end of line hyphenation. > No page headers or footers. > Just plain reading. i'm 100% in favor of that. and i have demonstrated before, and will be happy to demonstrate once again, any time that you like, how p.g, could save its texts in a format that allows verification of the type that i am talking about, _and_ allows the end-user to have the text exactly like you specify. > Once again, I have no stance AGAINST people who want pagination, > I just don't want for force any such arbitrary formats on anyone > and neither should you or anyone else. > STOP TRYING TO FORCE YOUR OPINIONS ON OTHERS, > MAKE THEM OPTIONS! see, that's precisely why i said you're not listening to me, because there's no way in the world i would try to "force" this on end-users. i've never, ever, said _anything_ even _remotely_ like that, in all the years i've been on this listserve, or the decades i have done e-books, so i don't know who you're having this conversation with, michael, but it's obviously not me. > I CAN tell you that most of the paper editions' page numbers > will fade along with the hyphenation. no, they won't. because our cultural heritage is full of references to page-numbers, and it'll be several orders of magnitude cheaper and more efficient to keep track of those page-numbers than to attempt to re-do all those references using hyperlinks or whatever. > Last time I looked there were still pretty ubiquitous programs to > lay out all such differences. > > IFF you have such deep interests, you can simply put up two editions > side by side when you look at them. . .I do. . . . > > If not, then you aren't really that interested. . .it's all smoke. this is very amusing. i do this kind of work, michael. regularly. and i can tell you that it's not nearly as simple as you make it sound. > "_RIGHT_" copy??? > Now you've contradicted yourself back into the ivory tower. . . . > "_RIGHT_" copy, indeeeeed. . . . you just can't _wait_ to jump to the wrong conclusion, can you? by "right" copy, i mean the one that the person _wants_ to see. > This will ONLY do you any good if you manage to find > that edition, out of all the other paper editions in the world. again, "that edition" is whatever edition the person wants to use. i have a paper copy of "catcher in the rye" on my bookshelf now. let's say, 10 years on, i can find a dozen digital versions online. let's also say that some analysis shows differences between them. i haven't compared all, not in full, but i know there's some diffs. i don't want a dozen different versions. i want the one that matches the paper copy that has been sitting on my bookshelf for 4 decades. how do i determine which one -- _if_any_ -- is the same version as the one that is sitting on my shelf? that's the difficult question. > Sorry, but I anticipated ALL of these questions when I first started, > and have answered, and will continue to answer, at length. no, you didn't answer the question. so i just asked it to you again. > Why can't you just propose your ideas as OPTIONS, > not CARVED IN STONE? stop making the thread ridiculous. no one can carve anything in stone any more. -bowerbird

5 5

PDF-files
by Fernando Maia Jr. 15 Apr '10

15 Apr '10

[Sorry, I forgot to change the subject.] Hello, volunteers! I'm new here and I have a doubt. Would it be interesting for PG if it would have more PDF-files? I've searched for this information everywhere and I haven't found an answer yet, so that I decided to ask about it here. Sorry for possible mistakes (English isn't my native language). Thanks in advance, Fernando

3 2

slightly off topic, first post, scanning
by Sparr 15 Apr '10

15 Apr '10

This is my first post to the list. I come here for advice not directly related to PG. I have a number of physical books that have had their spines cut off and I need to scan them so they can be OCR'd and stored digitally. I am encountering two problems. All of the bulk page-feed scanners I have tried have one or both of the following issues: Duplex scanning happens simultaneously, so extreme contrast changes (such as borders) from one side of the page show through (about 5%) to the other side. Remnants of glue left on the edge of a page (from the removed spine) get stuck to the inside of the scanner or feeder, ruining the scans of subsequent pages until the scanner is cleaned. Can anyone recommend a solution to either of these problems that can hopefully be applied automatically or in a batch operation? Reduction of human intervention/effort is my goal.

5 6

Question about Scanned Books
by Michael McDermott 14 Apr '10

14 Apr '10

Archive.org has many DJVU files of books that have lapsed into the public domain. Would it comply with PG's guidelines to take one of these volumes (the one I was thinking of has a copyright date of 1915 and can be found at http://www.archive.org/details/worksmartinluth00spaegoog)? The important elements here are: 1) I do not have a copy of the paper edition 2) This is a scan of a work that, by all appearances, qualifies having been published in the US before 1923 3) Was digitized by Google Would this work or would its ancestry cause problems? -- Michael McDermott www.mad-computer-scientist.com

3 2

Re: the d.p. opinion on "prerelease" of e-texts
by cmiske＠ashzfall.com 06 Mar '10

06 Mar '10

> some people are quite agitated at the thought of the idea. > they drag out the old rot about how "d.p. means _quality_". > -bowerbird DP does have a standard of quality similar to many current publishers of books. I frankly cannot recall the last time I read a book in any format or from any publisher/producer of books that did not contain errors of some sort or another. Humans are not perfect. Writing, editing, and proofing are human endeavors and, as such, are limited by our imperfections. The fact that they even strive for quality at DP is a mark in their favor. There are many eBook producers of public domain books that hold quantity so far above quality as to make a majority of their books almost unreadable. I would worry that a pre-release version of a text might result in PG getting a reputation for being yet another supplier of illegible eBooks (even if these books are only available in a 'special area'). As Michael asked only for 'serious objections' against a pre-release section being created, I cannot think of any objections that go outside of simply wondering if you are willing to have these books circulated as they stand (people may grab them and distribute them 'as-is'). Would they have a header that specifies that they are 'not ready for prime time' as well as being in a special area? (Many people strip the headers, so the source of origin issue often becomes a moot point.) The more I think about it, the more I feel that it would actually be rather nice to see something along the lines of the 'roundless experiment' produce decent quality 'first release' books and then have people nitpick them to 'perfection' later. The point would be the format of storage so that the rough version could be compared to the original scans and corrections/updates be done with ease. Also, if a book is backlogged at DP, it would make sense to have a method for someone outside of DP to adopt the book and finish it. Carel -------- Original Message -------- Subject: [gutvol-d] the d.p. opinion on "prerelease" of e-texts From: Bowerbird(a)aol.com Date: Tue, February 23, 2010 5:22 pm To: gutvol-d(a)lists.pglaf.org, bowerbird(a)aol.com just in case you were wondering, dkretz _did_ copy michael hart's saturday post over to d.p., giving michael's position in favor of prerelease of texts while they're still "in progress" at d.p. that was on saturday evening, and since then there have been over 100 posts in response: > http://www.pgdp.net/phpBB2/viewtopic.php?t=42940 there was an earlier discussion on the issue too: > http://www.pgdp.net/phpBB2/viewtopic.php?t=42695 the earlier discussion had over 100 posts as well, and had a poll, which got a high number of votes. the poll question? > Making preview texts available on a dedicated site, > with a clear label that they're work in progress. > Good idea/bad idea? the poll results? > 45% -- I'd rather you didn't > 17% -- I couldn't care less > 36% -- Great idea! some people are quite agitated at the thought of the idea. they drag out the old rot about how "d.p. means _quality_". -bowerbird _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

21 76

Re: Processing eTexts
by Bowerbird＠aol.com 27 Feb '10

27 Feb '10

well, it's kinda silly for us to talk about this in generalities, because i'm capable of demonstrating what i mean specifically and exactly... and when i'm done, i'm quite sure that you will agree completely. (and you might say that's what you meant all along, for all i know.) so let's get specific. i've posted a file of text taken from rfrank's roundless experiment: > http://z-m-l.com/go/campf/campf-001.zml as you can see, this book is called "pemrose lorry, camp fire girl"... the book is now in-progress, so it's half-done and half-undone... (it looks like it's been proofed up through about page 80 so far, but since the pages go out multiple times, it's difficult to be certain of it, but after page 80, it's fairly clear that the pages haven't been edited.) that first version had the page-separators fixed, and other basic stuff. the next version has more stuff cleaned up, but it's still rather basic: > http://z-m-l.com/go/campf/campf-002.zml as you can easily see, if you look through either one of the versions, starting after page 80, there are a bunch of paragraphing problems. specifically, there are often blank lines inserted -- incorrectly -- between the lines of a single paragraph. you can find instances of this problem on pages __, __, __, and __. it's a not uncommon glitch. paragraphing is one of the first things i try to correct, because it's necessary to have the paragraphs correct to fix any spacey quotes. fortunately, it's rather easy to locate these bad paragraphs via search. find a blank line followed by a line that starts with a lowercase letter. (in other words, you search for two newlines followed by lowercase.) now, one way of doing the search would be to automatically replace _all_ of these occurrences by simply deleting one of the two newlines. you could click "replace all" on that find-and-replace, and change all of the occurrences without even looking at them. you _could_ do that. the proper changes would almost always outnumber improper ones. whenever anyone talks about "taking the human out of the equation", blind global changes is what immediately springs into my little brain. and i am definitely _not_ a fan of blind global changes. there are some changes you can make blindly, but they are few and far between, and they certainly don't characterize the usual process. and this change -- deleting those excess lines -- i don't do blind... i will step through the changes one-by-one, and look at all of them _against_the_scan_. (even though, in most cases, i wouldn't need to look at the scan, because it's pretty obvious how to fix the problem.) sometimes i even delete the excess line manually, rather than merely approve the find-and-replace, just to keep my grubby fingers busy... but -- here's the very important part -- i do a lot _more_ than that. at every one of these spots to which i'm automatically transported, i look around the immediate problem area, to see if it spilled over. and, if it did, then i will clean up that neighboring area right away. so i'm walking a fine line here between doing a specialized search and a general correction routine. i start with the specialized check, so i have a laser focus on the nature of what the check will show me, so i don't have to waste mental energy figuring out what the glitch is. but then i also use my peripheral vision to see what else needs fixing. and this particular check in this particular text is a very good example. because it ends up that a _lot_ of the surrounding areas had problems. (that's not uncommon with these excess blank lines, because it's often the other glitches that _caused_ the o.c.r. its paragraphing difficulties.) there were a good number of problem-lines caught by this routine -- about 85 if i remember correctly -- but i probably corrected the text on 85 surrounding lines as well, because i saw bugs, so i fixed them. i encourage people to "play along at home" and _do_ this search on this text, so you can really see how you can spot neighboring glitches. what you will _not_ know, however, unless you are also looking at the _scan_ for each page, is that there were often _entire_words_missing_ from the o.c.r. they were cut off entirely from the left-hand margin. this isn't the kind of thing for which you can do a search to find them. they're just missing from the o.c.r., and you don't know they're gone. however, since you are paying attention during this _other_ search, you can become aware of these missing words, via peripheral vision. if you _do_ "play along at home", and actually step through that check, you will also see that there's yet another check that needs to be done to catch _all_ of these paragraphing problems from excess blank lines. this second check is the "flip side" of the first one. the first one was looking for supposedly-first-lines of paragraphs that were lowercase. the second one checks the _termination_ of supposedly-last-lines... so you're checking for a letter or a comma at the _end_ of a line which is followed by a blank line. if it were _really_ the end of a paragraph, as is implied by that blank line, it should be punctuation-terminated. so you're looking now for a letter/comma followed by two newlines... this search turns up roughly 50 instances in this particular text-file. and, once again, when you make these corrections, you look around. roughly half of the 50 instances were cases where the blank line that was deleted was followed itself by a line beginning with a single-dash, which was a misrecognition for an _em-dash_ at the start of that line. (the em-dash is represented by a double-dash.) so those corrections were made at the same time that the excess blank line was eliminated. if you were a party-pooper, and didn't play along at home, i have uploaded a copy of the file where the lines that were flagged by the first page are indicated by an "@" sign, which you can search for: > http://z-m-l.com/go/campf/campf-003.zml and here's one with the lines from the second check tagged with "%": > http://z-m-l.com/go/campf/campf-004.zml you can also view the book page-by-page, as usual. for instance: > http://z-m-l.com/go/campf/campfp123.html (you will see, right there on page 123, an example of a page which has some of the words cut off from the right-margin. when i find this type of problem in a scanset, it just makes me want to scream! when you scan a book, you need to make sure you do it _carefully_. sloppy work such as this is tremendously uncharacteristic of rfrank.) at any rate... it's this "looking around and fixing other errors in the neighborhood" that makes this whole preprocessing thing such a _vibrant_ procedure. ...which brings me squarely back to the major point of this post... *** so yeah, i'm doing a computerized check, and it's darn _efficient_, but i'm surely not "taking humans out of the process". not by a long shot. i am a mentally alert human, who is _actively_ engaged with the text... my cursor is _flying_ all around that document. i move at warp speed. and i am _efficient_ too. i am _tremedously_ efficient. i am rocking! all the other carpenters are using hammers, and i have a _nail-gun_. part of the efficiency is due to that _focus_ that i just talked about... but most is simply because i am being directly transported to errors. i don't waste any of my time _looking_ for them; they're _presented_. i don't waste any time positioning my cursor -- it is _prepositioned_. and the scan is right there, ready for me to look at it, immediately... and it's _fun_. downright _exhilarating_. like driving a sports car. (whereas word-by-word proofing is akin to pushing a baby stroller.) and the results are _better_. not just a little bit better, a lot better. i've shown time after time after time that i can find (lots of) errors proofers _miss_, not just in one round, but two and three and four. and, just to repeat the message yet again, it's _simple_ to do this. a couple dozen _simple_ checks finds a high percentage of errors. so yeah, i know i am explaining all of this with accurate language. but people always seem to misunderstand. on the one hand, they think the tool is making all the changes... on the other hand, they think you need a human to find errors... both of those positions are wrong, and when i argue against one, people jump to the other, and then i have to argue against _that_. i know that i am capable of walking the tight-rope between them, but everyone else seems to get tangled up in the dialog semantics. the tool can find almost _all_ the errors, so you don't need to look. (you're welcome to look, and i'm sure that you'll find a few errors, but at some point, you have to ask whether it was worth your time.) on the other hand, _you_ have to be the active agent that mentally makes the decision whether or not each change should be made... like i said, there are only a few global changes that i make blindly. (to give you a for-instance, i will globally delete any space that follows any doublequote located at the very beginning of a line, or any space preceding a doublequote at the very end of a line. i'll also blindly change spacey-quotes if the open/close pairing within the rest of the paragraph makes the change unambiguous. i'll also close up contractions blindly. maybe a few more things. but for the most part, i really want to look at the changes i make. maybe if i had the luxury of knowing i'd have human volunteers following up with a word-by-word proofing of everything i did, i would be more willing to make blind changes. or maybe not, because that just seems wasteful of donated time and energy.) now, i'm sure that dkretz will confirm what i've been saying here, as his experience with his tool has led him to the same thoughts. and maybe carel will say that this is exactly what she has meant, and we've been doing a semantic dance around the same thought. -bowerbird

1 0

Re: Processing eTexts
by cmiske＠ashzfall.com 27 Feb '10

27 Feb '10

Bowerbird said: >i don't want them to think of this as "the formal proofing process". >i believe we need to put the books in front of them _to_read_... Which is exactly why a book that is ready for release to PG should not force upon an individual the original scan. When the reader spots something they think is an error, they should be able to call up the scan to see if it is an actual error before reporting it as such, etc. Otherwise their reading is distracted by the scan. A proper interface would allow users to select what they want to see. An interface should be about options. In any case, I have no desire to host an archive that duplicates the content of PG and others. The purpose in coming to my project site would be to prepare books for release to PG and to polish released books so that they can be read for leisure and pleasure off of _my_ bandwidth. >> I think calling it a wiki makes it easier for quick comprehension >> rather than explaining the whole process of storage, editing, etc. > >we should never _have_to_ "explain the whole process" to the public. >for them, it should be as easy as clicking a button to say "check this!" >they shouldn't even need to explain the error. (we can find it, right?) Of course, but it is hard to develop it and request assistance with the development of it without explaining the basic concepts, etc. A project must be explained and defined or it cannot be realized. >> Rather than working an editing interface >> into an out-of-the-box wiki system >we don't need to make "an editing interface"... It's rather hard for people to make/submit corrections without having an interface for doing so. >> Has anyone ever just asked google if they could >> use the scans and OCR text for PG? > >michael had a meeting with google when they were just starting out... > >he reported that they treated him rather rudely, hustling them out of >their famous cafeteria before he had even finished his lunch. it's sad, >but i don't think they really appreciate project gutenberg very much... That is too bad. >> XML makes a nice standard for storage and conversion > >it's obtuse, and unnecessary obstructive to the text itself... It's also one of many methods for the storage of 'intelligent' data and is easy to do. Once put in place, it would never be seen again by anyone except someone correcting the markup (which one hopes would be rare). InDesign uses markup, MS Word uses markup, and on and on. I don't think the users of this software find the markup obstructive to the text itself. >> My only protest is that I believe >> a human proofing stage is still a good choice. [snip] >i can't think of one single reason for a word-by-word proofing, >let alone two or three rounds of it. it's simply not efficient, and >anybody who explores the options can learn that for themselves. I don't believe I said anything about multiple rounds of proofing, etc. I believe I discussed the main processing of a document being done through human interaction with scripts and then people actually looking at the results of the process before the output is released (proofing). I feel that a human looking at a smaller subset of a large document is a good thing in the error finding process. You apparently do not think it is. Neither of us is right or wrong: It is a matter of perspective and opinion. >between aggressive clean-up of o.c.r., and the comparison method, >and "smoothreading" by people from the general public, we're good. Yes and no. It depends on your definition of 'good.' >> Otherwise, I would say to have two people >> run the same correction process, diff the results and >> then present the mis-matches to a third set of eyes. >close... but it's unnecessary for "people" to do the digitizations. >two sets of cleaned-up o.c.r., diffed against each other, is enough... That depends on a lot of factors including the assumption that two OCR programs would not make the same mistake and that, if they did, it would be a mistake that could be caught by another process. Etc. >(but yes, it's necessary to have a sentient human doing the clean-up.) Exactly. And, if that human makes a major error, the entire text in now a mess. If two people do the same process it reduces, but does not eliminate, the risk for 'the human error factor.' >o.c.r. clean-up is a fast-paced, active, and energetic task. >which does _not_ mean that it's "harder". just more fun... People actually do enjoy proofing. You do not. Again, neither is wrong in their viewpoint. Volunteers should be involved in a manner that gives them pleasure and satisfaction, etc. I say to give people tools and options and let them be the judge of what is best for them. >i get the impression you think these clean-up tools are just >a bunch of scripts that people turn on, and the results fall out. >but that's not anything like what actually goes on. Since I am planning to write scripts for such an environment, one would hope that I do not "think the results will just fall out." I think much of this will be very difficult to program and that the software must be capable of 'learning' and that it will require a human to interact with it so that the job can be done well and the software upgraded to do the job even better in future. I actually wrote several scripts like this many years ago. They are archived on my CDs, but my manifest was damaged in a fire, so I am not sure where to even begin to retrieve them. It would be faster and easier to just start from scratch than to dig up outdated software anyway, so I do not regard it as a great loss. >> I'm not sure I'm ready to program an AI engine >> capable of "listening to the book." >the software can't listen to the book. _you_ have to do that. I pretty much said that in what you snipped. :) >> Which is why I do not believe in leaving people out of the process >again, you have a profound misunderstanding of the process. I'll just laugh and pretend you didn't say that.... Perhaps I have a profound misunderstanding of _your_ proposed process, but I have a very firm understanding of my own. There is no _the_ process for me to understand. In any case, if you would like to create your own project, feel free. "The road is free to all." And Michael and PG would be happy to recieve the texts submitted through any process. And I would be happy to receive constructive feedback on my efforts as well from any source that cares to give it. I am just interested in creating a method for people to complete initial editions of books rapidly, to avoid putting features in place that 'waste' the time of volunteers in doing tasks that could be done more rapidly and easily through the use of software, and to have an environment that offers the opportunity to correct any errors that were missed in the initial release. Also, it should offer a sense of community for volunteers to interact. The _process_ is not a complex one by any means. As a human, I expect that my project process will not be perfect nor will it agree with the opinions of all others on how it _should_ be done, but it will be a process that produces books. So, it will be part of the solution no matter how many problems it has. :) Nor do I feel that the process at DP is flawed. They are doing the process the way they have chosen to do it and it is a project that acutally exists and is productive. They simply created a bottleneck in their workflow that they need to deal with. I actually hope that some of my tools are of assistance to those who use DP to process books. I am about getting books into PG and care very little how one goes about doing it so long as one _is_ going about doing it. :) Carel

1 0

Re: Processing eTexts
by Bowerbird＠aol.com 26 Feb '10

26 Feb '10

carel said: > Direct comparison to the scans is part of > the formal proofing process and, > although available during the 'quest for perfection,' > should not be a forced issue. i disagree. it's largely a matter of "framing" the issue and its perception. i don't think we should ask the user to help with "proofing" per se. i don't want them to think of this as "the formal proofing process". i believe we need to put the books in front of them _to_read_... _that_ should be our understanding with them, how we "frame" it. they're there because _they_ want to _read_ this particular book. now, as part of the interface by which we present the book to them, we give them both the digital text _and_ the page-scan, together... most interfaces make the user _choose_ one of the other, but i see no reason (especially no _benefit_) in doing it that way... i think it's wise to give them _both_, and have them understand that there may have been some transcription errors during digitization, so we're giving them the page-scan too, for any cases they _suspect_ might be errors. in most cases, they'll see that we matched the scan. or they'll understand why we made the change that we did, if we did. _or_ -- in some rare cases, in which we request their assistance -- they'll find an actual transcription error that we made, and _tell_ us. this is the thing that project gutenberg doesn't do well. it doesn't _invite_ the reader to become a part of the transcription process, to become one of the instruments that marches texts to perfection. indeed, you can _feel_ deep in your gut the _contempt_and_scorn_ that jim tinsley and al haines have for the stupid people who _do_ report errors, because _half_ of those "errors" are not, in fact, errors. well, gee, big surprise. you haven't given them the page-scans, so how in the world are they gonna know if an apparent error _is_ real? > I think calling it a wiki makes it easier for quick comprehension > rather than explaining the whole process of storage, editing, etc. we should never _have_to_ "explain the whole process" to the public. for them, it should be as easy as clicking a button to say "check this!" they shouldn't even need to explain the error. (we can find it, right?) > Rather than working an editing interface > into an out-of-the-box wiki system we don't need to make "an editing interface"... > Has anyone ever just asked google if they could > use the scans and OCR text for PG? michael had a meeting with google when they were just starting out... he reported that they treated him rather rudely, hustling them out of their famous cafeteria before he had even finished his lunch. it's sad, but i don't think they really appreciate project gutenberg very much... > XML makes a nice standard for storage and conversion it's obtuse, and unnecessary obstructive to the text itself... > My only protest is that I believe > a human proofing stage is still a good choice. if you mean a word-by-word reading of the page by a person who has no real interest in reading the entire book, i disagree. a comparison of different digitizations is a much better process, in terms of finding errors, and it takes less work, and is more fun. i can't think of one single reason for a word-by-word proofing, let alone two or three rounds of it. it's simply not efficient, and anybody who explores the options can learn that for themselves. between aggressive clean-up of o.c.r., and the comparison method, and "smoothreading" by people from the general public, we're good. > Otherwise, I would say to have two people > run the same correction process, diff the results and > then present the mis-matches to a third set of eyes. close... but it's unnecessary for "people" to do the digitizations. two sets of cleaned-up o.c.r., diffed against each other, is enough... (but yes, it's necessary to have a sentient human doing the clean-up.) > the existance of a proofing round > creates a sense of belonging to the project and > could assist in recruiting people to try the 'harder' process of > running the 'automation' scripts and submitting books word-by-word proofing is a slow, boring, humdrum task. o.c.r. clean-up is a fast-paced, active, and energetic task. which does _not_ mean that it's "harder". just more fun... doing the former task won't help you at all with the latter. (except perhaps make you _extremely_ grateful that you don't have to do such a boring, miserable job ever again.) > People need a sense that they are not just > assisting tools to create book editions, > but are a fundamental part of the process. is the hammer more important than the carpenter? i doubt it, whether you ask the hammer _or_ the carpenter... i get the impression you think these clean-up tools are just a bunch of scripts that people turn on, and the results fall out. but that's not anything like what actually goes on. > I'm not sure I'm ready to program an AI engine > capable of "listening to the book." the software can't listen to the book. _you_ have to do that. > Which is why I do not believe in leaving people out of the process again, you have a profound misunderstanding of the process. the hammer can't do a thing without the active involvement of the carpenter. even then, the carpenter has to know what s/he is doing, or the hammer won't be able to do one bit of good... the best thing you can do, carel, is to play around with twister... -bowerbird

1 0

Processing eTexts (was RE: Re: d.p.'s undeserved superior attitude about "quality")
by cmiske＠ashzfall.com 26 Feb '10

26 Feb '10

Bowerbird said: >carel said: >> I've read your posts for many years > >so _you're_ the one who's been reading them! ;+) I'm sure many people read them. :) >> I'm not sure the most efficient system on earth >> could keep up with the scan rate. > >sure we can. just have 10 million people do 1 book each. > >piece'a'cake. It sounds easy when you put it that way.... ;) >> And, if you speed things up by skipping the manual proofing >> and use automation completely for the first release then you >> will have text little to no better than what google already has. > >oh, don't kid yourself. google's text is _much_ better than >what they're showing to us members of the general public. Their OCR software requires training, so I agree with you that they always have an at least one step better version on the backburner. They probably have a stepped up version of everything we see.... >i haven't been impressed by the wiki approach, honestly... The standard wiki layout is not a proper environment for editing. Better would be something that allows the vast majority of readers to just read the page and click a button to be presented with an interface for making/submitting corrections if such is required. Direct comparison to the scans is part of the formal proofing process and, although available during the 'quest for perfection,' should not be a forced issue. I think calling it a wiki makes it easier for quick comprehension rather than explaining the whole process of storage, editing, etc. Rather than working an editing interface into an out-of-the-box wiki system, it would be better to write a proprietary system with some of the social networking features of a wiki. That part would take a very firm second place to just getting some tools out there for people to produce decent quality texts in a minimal amount of time. Because of that fact, I haven't really given much thought to the 'wiki' interface as it will evolve by itself from wants/needs. [snip] in re: google: >people on this listserve have been cut off for scraping... >you can still do it, of course, but you have to be careful. Has anyone ever just asked google if they could use the scans and OCR text for PG? In regards to the scans, I believe they just require you to leave the branding. >i would bother to nitpick certain parts of it, like the x.m.l., >but really, my objection is at the more fundamental level. XML makes a nice standard for storage and conversion and can be human readable. I'm not one of those people who believes that XML is the answer to everything, but I do think it works well for this use-case scenerio. Opinions on this will vary, etc. >yes, i believe every page of every book should be online, >on its own webpage with unchanging u.r.l., text and scan, >with an error-report form that the general public can use >to detail any problems with the page, or ask questions, or >even make annotations and have dialog about that page... Agreed. >but i also believe that the text for the vast majority of these >pages should be _perfect_ at the time that text is mounted. We are in agreement on this point as well. The majority of the work towards corrections should be done via automation and human interaction with a software process so as to reduce the error level to a minimm before it is put in the hands of a proofreader (or the general public). >and i've demonstrated, repeatedly, that this is fully possible. It is very possible. [snip] >so here's how i see things going... >when you find a scanset online of a book you want to digitize, >you'll download it, along with the o.c.r. (if it comes from a site >that won't give you the o.c.r., you can upload it to archive.org, >and they'll o.c.r. it for you. providing you have no o.c.r. app.) >you'll use a handy-dandy tool to digitize it in an hour or two, >and then you'll make the results available to everyone online. My only protest is that I believe a human proofing stage is still a good choice. For one thing, it would be a way of catching errors that could be induced by the human assisted automation process. Otherwise, I would say to have two people run the same correction process, diff the results and then present the mis-matches to a third set of eyes. If that doesn't result in a book that is ready for release, I'm not sure what would. Ready for release != perfect. :) On a point in favor of the proofing round that really has nothing to do with the potential for automation, the existance of a proofing round creates a sense of belonging to the project and could assist in recruiting people to try the 'harder' process of running the 'automation' scripts and submitting books, etc. People need a sense that they are not just assisting tools to create book editions, but are a fundamental part of the process. If you try to proof after a release has been done, there will be little interests in it because you lose the motivation to 'get the book out there." >and carel, since you're interested in programming such tools, >we'll have plenty to talk about... Yes. :) It's very much a 'will do' at this point. I just have to survive my mid-terms. I look forward to your feedback and your input on the project. >well, like i told gardner just the other day, i really don't >_have_ any collection of scripts. i use a text-editor and >start looking at the book. i see an error, and devise a >global search-and-replace to deal with that type of error. >when they're all fixed, i look again to find the next error. That brings up the point that pretty much all of what my scripts would be doing could be done in word processing software.... >i think of this process as "listening to the book", in that >the book itself will tell you what kinds of errors it has... I like that. I'm not sure I'm ready to program an AI engine capable of "listening to the book." Which is why I do not believe in leaving people out of the process, but rather giving them tools to assist them with the process. And, the tools can 'learn' from the people who use them. I'll be busy finishing up a group project for my classes, so it'll be late March before I get any real time to work on scripts for this project. I'll work on some of the research, logic, and flow in the meantime so that I am ready to roll when I do have the time. Back to my programming.... Carel

1 0

roundlessness -- 008
by Bowerbird＠aol.com 26 Feb '10

26 Feb '10

by the way, i've coded an offline program that hooks into rfrank's system. if anyone would be interested in using such a beast, take it up with roger. or, if you just want to use it to scrape the books on his site, talk to me... -bowerbird

1 0