- gutvol-d - lists.pglaf.org

Re: d.p.'s undeserved superior attitude about "quality"
by Bowerbird＠aol.com 25 Feb '10

25 Feb '10

carel said: > I've read your posts for many years so _you're_ the one who's been reading them! ;+) > I'm not sure the most efficient system on earth > could keep up with the scan rate. sure we can. just have 10 million people do 1 book each. piece'a'cake. > And, if you speed things up by skipping the manual proofing > and use automation completely for the first release then you > will have text little to no better than what google already has. oh, don't kid yourself. google's text is _much_ better than what they're showing to us members of the general public. > Wikimedia then takes this further by adding the ability > for humans to modify the text in a never-ending cycle. i haven't been impressed by the wiki approach, honestly... > In what manner do you suggest that this would be > differentiated from the current wiki book editing > process at wikimedia that justifies the creation of > the project for the purpose of PG? i'm not suggesting this for p.g. it could be for anybody. including google. as for how it differs, please see below. > it wouldn't be that hard to cull the data from google well, it's not that "hard", but it's not that _easy_ either... first, they've started rewrapping the lines, the dirty rats. second, and more importantly, their terms prohibit it... people on this listserve have been cut off for scraping... you can still do it, of course, but you have to be careful. > Then, someone can yeah, what you've laid out is basically the standard plan. i would bother to nitpick certain parts of it, like the x.m.l., but really, my objection is at the more fundamental level. yes, i believe every page of every book should be online, on its own webpage with unchanging u.r.l., text and scan, with an error-report form that the general public can use to detail any problems with the page, or ask questions, or even make annotations and have dialog about that page... but i also believe that the text for the vast majority of these pages should be _perfect_ at the time that text is mounted. and i've demonstrated, repeatedly, that this is fully possible. one person can take an average book to near-perfection in the space of a couple hours, if they are given a good tool... ("near-perfection" means 1-error-or-less-every-10-pages.) i just showed how you can do this, using gardner's book... so, if 9 out of 10 pages are perfect to begin with, then you simply don't need a full-on wiki format. a wiki is good when you are constantly editing something. but book digitization isn't like that. corrections will be very few and far between. meanwhile, there's no reason to put every page up for grabs. even on a first pass, straight outta o.c.r., assuming good scans and a good o.c.r. app, _half_ the pages have no errors on 'em. in 2001, the idea for distributed proofreaders was brilliant. (i was once a graduate student at u.c.l.a. in social psychology, studying cooperation/competition in the arena of the internet, so i'm probably the best person on the planet to understand just precisely how brilliant the idea was, and it was brilliant.) in 2001, scansets were rare, and o.c.r. stunk (and cost a lot), and it was hard -- very hard -- to digitize a book by yourself. moreover, the primary method of connection was slow-speed. so doling out pages one-at-a-time to people online was a relatively efficient way of doing book digitization in 2001, especially as it allowed us to _cumulate_ the _partial_effort_ of many people. simply put, it made sense to go distributed. a good number of people were willing to "proof a page a day". in 2010, the situation has been turned completely on its head. scansets are no longer rare; indeed, we are swimming in them. o.c.r. no longer stinks, nor does it cost a lot. indeed, many of those scansets now _come_ with the o.c.r. already being done, and -- after some preprocessing, which isn't difficult to do -- the high level of the accuracy of that o.c.r. is rather startling. and perhaps most importantly, broadband is now widespread; we think nothing of throwing tens of megabytes around now. so, in 2010, doling out pages one-at-a-time to people online is just a silly idea. i can download in the background now, but even if i _watch_, i can download a whole scanset in minutes. and a 300k text-file that represents a book? in just seconds. these days, with the right tools at your disposal, it's possible for a person to digitize a book all by yourself in _one_hour_. i proved this -- with a stopwatch -- and documented it fully. so here's how i see things going... when you find a scanset online of a book you want to digitize, you'll download it, along with the o.c.r. (if it comes from a site that won't give you the o.c.r., you can upload it to archive.org, and they'll o.c.r. it for you. providing you have no o.c.r. app.) you'll use a handy-dandy tool to digitize it in an hour or two, and then you'll make the results available to everyone online. or... your alternative is waiting 3 years for d.p. to get it done. and carel, since you're interested in programming such tools, we'll have plenty to talk about... > I am very interested in your text processing scripts > though and always have been. I just haven't been > processing any OCR text for a while. well, like i told gardner just the other day, i really don't _have_ any collection of scripts. i use a text-editor and start looking at the book. i see an error, and devise a global search-and-replace to deal with that type of error. when they're all fixed, i look again to find the next error. i think of this process as "listening to the book", in that the book itself will tell you what kinds of errors it has... and yeah, there are some things that you do to every book, like fix spacey quotemarks, and pull in spacey contractions. and it would be good to write macros or scripts or routines to do these. but it's fairly trivial to type them in each time. it's also the case that a lot of the time you _must_ examine the text in order to approve or disapprove the change, and -- while it is not impossible to code an interface for that -- your typical text-editor _has_ one, which you know and love. > I merely meant that we are not in a position to > have any say in what DP does with itself. > Nor in what PG does with itself. but i love telling them what they should be doing, and why. because once they decide to do it, and they _will_, because i'm never wrong about these things, i can say "i told you so". -bowerbird

1 0

Re: the d.p. opinion on "prerelease" of e-texts
by Bowerbird＠aol.com 25 Feb '10

25 Feb '10

michael said: > We complile notes into computer instructions all the time, > it's not so terribly inefficient. um, ok, if you say so. > How do you think the "posted" list > gets created in the first place? i thought it was in some sensible way. but now i'm just plain afraid to ask... *** anyway, here's how the questions should be answered: 1. programs scrape the d.p. website for info about which books have been posted into which rounds... 2. when a book is posted into a new round, its file is copied to the prerelease site, and the prerelease file posted for the previous round (if there was one) is deleted. 3. when a book is posted to p.g., the latest file on the prerelease site is replaced with a pointer to p.g. 4. all of this happens _automatically_, with no need for any human intervention at all. do you see the difference? because it's a huge difference. of course, this doesn't answer the _political_ questions al asked. but those answers are above my pay grade... -bowerbird

1 0

Re: the d.p. opinion on "prerelease" of e-texts
by cmiske＠ashzfall.com 25 Feb '10

25 Feb '10

>Al said: >Will it contain only pre-release text files, or all working files associated >with a given project (page scans, illustrations, text, HTML, etc, etc)? > >If the latter, what's to stop someone from taking those files, getting their >>own clearance, and submitting them to PG as their own work? Or is DP going >to consider that they're, in effect, abandoned projects, and up for grabs? I actually hope the files are made available for others to finish. If the backlog is deep enough at DP as to warrant the creation of the pre-release area, then it would actually relieve their burden if individuals adopted some of the projects. In any case, I sincerely doubt there would be a mass wave of people grabbing up projects. If someone does finish one of these projects, I assume an attribution to DP would be included in the header to credit the work that they have done. To me, the issues of 'ownership' and 'credit' are secondary to what I believe is the main goal of getting finished works into PG so that people can read them. Carel

1 0

Re: let us not be confused
by cmiske＠ashzfall.com 25 Feb '10

25 Feb '10

>Jim said: >Simply my personal experience finishing a book taking this exact approach. >When you take out the easy stuff, you are left with the stuff that typically >the P2s and P3s find -- hopefully! When you do the machine marking of errors >you double the amount of errors that need to be fixed -- since each input >file contributes its errors. But now many of the "P1" type errors are >marked, which could be a win if the P1 people were presented with a >highlighted version of those errors, similar to what is currently presented >in WordCheck. To me a surprising amount of work goes into fixing >hyphenation/dehyphenation linebreak errors, where it should be possible to >make a more intelligent tool to fix most of these problems -- or does >someone claim to already have such an intelligent tool? I made several such tools many years ago. I'm not sure I can locate them in my backups. In any case, I am in the process of creating some tools to assist in the process of proofing and formating. These would be for the use of 'solo' producers as well as DP users. I do not believe in full automation for anything complex (as it can induce as many errors as it fixes), so many of these tools would primarily prompt the user to make choices about things that 'confuse' the software. Carel _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

1 0

Re: the d.p. opinion on "prerelease" of e-texts
by Bowerbird＠aol.com 25 Feb '10

25 Feb '10

al said: > there would need to be some coordination to remove them from > that environment when they're posted into PG as finished products. michael said: > Greg Newby takes care of deleting from PrePrints, should be > easy enough just to send him notes of what has been completed my gawd, what a remarkable exchange. we're talking about _files_, on the _internet_, with _known_ u.r.l., going through a standard progression of states. and y'all think you need to handle this _manually_, by "sending notes"? you haven't learned one thing about why all of your processes are _so_ inefficient, have you? not a single thing! -bowerbird

2 1

Re: let us not be confused
by Bowerbird＠aol.com 25 Feb '10

25 Feb '10

jim said: > Not that I totally disagree, but > when you take out the easy stuff, > the stuff that's left is harder to find. where's your data that validates that? i think when you clear the page of "the easy stuff", it becomes _easier_ to find "the stuff that's left", because you're not distracted by the easy stuff. it might _feel_ "less effective", since you're making fewer corrections, but the ones you do make are more vital. what usually happens is that it takes one round of proofing to remove the easy stuff, and another for the rest... what i say is to make the first round a tool-aided round, to preserve your human resources, so the humans only have to do one word-by-word round, which is the difficult process. besides, it's not as if zero satisfaction comes in the tool-aided round. myself, i feel _greater_ satisfaction there, since my efficiency is boosted _considerably._ and in a roundless system, anything that moves a page closer to "finished status" is a good thing, because that's the goal. just _offer_ people a good tool to use; you will find they enjoy it immensely... by the way, dkretz has a new version of his tool available now, at the usual place: > http://code.google.com/p/dp50/downloads/list -bowerbird

2 1

let us not be confused
by Bowerbird＠aol.com 25 Feb '10

25 Feb '10

ok, we've got a couple of different topics running around, so let us take a minute to make sure we are not confused... first of all, let's talk about my campaign for preprocessing... i have demonstrated, over and over and over again, that d.p. (and rfrank) should be doing _much_ better preprocessing... i've shown how they can use _very_simple_means_ to do that, and how -- if they did -- they could reduce the error-counts in their books to a ridiculously small amount, even _before_ their text went in front of proofers. i have talked about how it is a huge _waste_ of the generous donations of volunteers (in both time and energy) not to do aggressive preprocessing, which automatically locates errors to make them easy to fix... again, the crux of my argument -- and i have proven it to be absolutely true, again and again -- is that it's _easy_ to do this. indeed, when i have shown the steps taken to locate the errors, it becomes painfully obvious how ridiculously simple they are... they include obvious checks, like a number embedded in a word, or a lowercase letter followed by a capital letter, or two commas in a row, or a period at the beginning of a line. _obvious_ stuff! this isn't rocket science. it's not even _hard_... it's dirt-simple! and yet neither d.p. nor rfrank has instituted such preprocessing. *** let's contrast this with gardner's request, which was to compile a list of reg-ex tests that will locate all possible errors in any random book. this request -- as worthy as it might seem -- is _much_ more difficult to realize. in fact, it's almost impossible. a friend of mine over in england, nick hodson, is a very prolific digitizer. all by himself, he has done some 500 books or more. nick collected an extensive set of checks over the years. i can't remember exactly how many there were, but roughly about 200. however, once nick upgraded his o.c.r. program, he found that about half of his checks were no longer required. they had been necessary essentially as an artifact of an outdated o.c.r. program. the type of books nick was digitizing hadn't changed, and neither had the quality of the scans, or the resolution of the scans, or the digital retouching that he performed on the scans -- none of that. he was the same person, using the same computer and scanner, and he was doing the same things exactly as he had done before. the only thing that changed was the version of his o.c.r. program. yet he found many checks he formerly needed became unnecessary. so, for an operation like d.p., who intakes all kinds of scans and uses a wide variety of o.c.r. programs, operated by users with a huge range of expertise, their results will be all over the board. they're _never_ gonna get a definitive list of checks to be made. it would be _immensely_ difficult, to the point of being impossible. but that's totally beside our other point, about preprocessing... because the fact of the matter is that a few dozen _simple_ tests are all that d.p. needs in order to reduce the number of errors to a level where they can be handled easily by their human proofers. they're never gonna get 100%. but they could find 90% so easily that it's criminal negligence that they aren't doing that already... heck, spell-check by itself will locate 50% of the errors for you... -bowerbird

6 6

Re: Inkmesh
by Carel 25 Feb '10

25 Feb '10

I meant that it would be nice if the site had a version for smartphones. I was failed to realize for a moment that mini has many meanings. Carel Sent from my HTC on the Now Network from Sprint! ----- Reply message ----- From: "James Adcock" <jimad(a)msn.com> Date: Thu, Feb 25, 2010 3:38 PM Subject: [gutvol-d] Re: Inkmesh To: "'Project Gutenberg Volunteer Discussion'" <gutvol-d(a)lists.pglaf.org> >It would be nice if it had a mobi version. Not sure I understand this comment. If the result says “Kindle” and it’s a free version, then it is a ..mobi file, or else is a .prc file -- which in practice is the same as a ..mobi file if you change the file extension.

1 0

Re: Inkmesh
by Carel 25 Feb '10

25 Feb '10

I meant that it would be nice if the site had a version for smartphones. I was failed to realize for a moment that mini has many meanings. Carel Sent from my HTC on the Now Network from Sprint! ----- Reply message ----- From: "James Adcock" <jimad(a)msn.com> Date: Thu, Feb 25, 2010 3:38 PM Subject: [gutvol-d] Re: Inkmesh To: "'Project Gutenberg Volunteer Discussion'" <gutvol-d(a)lists.pglaf.org> >It would be nice if it had a mobi version. Not sure I understand this comment. If the result says “Kindle” and it’s a free version, then it is a ..mobi file, or else is a .prc file -- which in practice is the same as a ..mobi file if you change the file extension.

1 0

Re: Inkmesh
by Carel 25 Feb '10

25 Feb '10

Very nice resource. Thank you! It would be nice if it had a mobi version. Carel Sent from my HTC on the Now Network from Sprint! ----- Reply message ----- From: "James Adcock" <jimad(a)msn.com> Date: Thu, Feb 25, 2010 2:51 PM Subject: [gutvol-d] Inkmesh To: "'Project Gutenberg Volunteer Discussion'" <gutvol-d(a)lists.pglaf.org> Just found about this site which is a cross-site search engine for free ebooks in various formats: http://inkmesh.com _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

1 0