Bowerbird:
carel said:
You don't need to talk to me about having your time wasted. I am the somewhat infamous Carel that left DP nearly a decade ago (after spending around 500 hours creating a front-end and revamping the horrific code that resided on the back-end).
oh, hey, glad to meet you. :+)
i do believe i am the one who has grabbed the "infamous" crown at this point in time, however. i see they still let you post there. myself, i've been banished, so as not to corrupt the youth there.
LOL. It's nice to meet you as well. I've read your posts for many years, but I don't believe we have ever directly 'spoken.'
DP has managed to produce a huge number of etexts
and they could have produced 3 times as many if their workflow were 3 times more efficient, something it could _easily_ attain... according to brewster, internet archive scans 1,000 books a day. (google probably scans twice that many before their lunch break.) d.p. digitizes 2400 books a _year_. i predict they cannot keep up.
I'm not sure the most efficient system on earth could keep up with the scan rate. No matter how much you automate pre- and post-processing and formatting, it comes down to the speed of humans when it comes to proofing. And, if you speed things up by skipping the manual proofing and use automation completely for the first release then you will have text little to no better than what google already has. I realize that is not what you are proposing, but what google has done is the most efficient with little regard to the potential for the quality added by human interaction. Wikimedia then takes this further by adding the ability for humans to modify the text in a never-ending cycle.
Their etexts also have a decent level of quality.
i'm with michael hart on this. (even if he refuses to see that.)
i think the first mission is to get text up with acceptable quality. (which, by the way, most of the in-process stuff at d.p. is _not_)
and _then_ we can use the _public_ to drive it toward perfection.
I'm am very much with you on this one. However, it rather duplicates the efforts of the wikimedia foundation (at least the last time I looked into their efforts), so I am not sure of the 'need' for it. In what manner do you suggest that this would be differentiated from the current wiki book editing process at wikimedia that justifies the creation of the project for the purpose of PG? I ask because I am considering giving it a try if I can find a single reason to do so that would not make me feel as though I am reinventing the wheel. I have not been able to find an acceptable reason on my own. I love PG and promote it to all that will listen, but love alone is not enough: There must be logic as well. :) For that matter, it wouldn't be that hard to cull the data from google books, cut their 'plain text' into pages (and/or re-ocr and dif) and just slap the result into a wiki (which is about what it looks like wikimedia has done). I believe the google use license allows for this for non-commercial entities such as PG? Then, someone can verify the google 'plain text' images as inline images or as content that could not be processed by the OCR (type in missing text and annotate images). Tidy it up (scannos, etc.), add some assumed markup for lines, paragraphs, pages, etc. using scripts with no human interaction. Have a human assisted 'spell check' fix the majority of remaining errors and end of line hyphenation issues, etc. and add some basic markup for italics and whatnot (although the google OCR text appears to already contain this). Run the text through an 'X number of people say this page matches the scan' proofing process (roudless process). Have a human assisted markup process to wrap the content in properly formed (but easy) XML. Convert the document to plain text (and any other formats that are desired). On a 300 page work of uncomplicated fiction, depending on the scan quality and speed of the humans doing the interactive processes, I would estimate this would take about 2-6 hours _plus_ the proofing time to create a document ready for first release and wiki storage. Then, you let it sit in a wiki for eternity being edited into perfection (or until a very large X number of people again say a page is perfect, create a final edition when all pages in a book pass this test, post the output, archive the XML version and scans and consider the book done for all time - or at least until a PG reader reports an error that must be checked). For user supplied scans, same process with OCR done on the server (if desired, can diff against the scan submitter's OCR, if supplied). I haven't thought about the process in a while and the new press for speed cuts out some of the way I recall my original ideas. I'm probably completely forgetting a topic or two, but this would pretty much work (after a few logic tweeks) to accelerate the release process and should retain some quality factors in the first release. As you know, ultimately, in regards to quality, you will always have to rely on the quality judgements of individuals working on the page or project to determine the quality of the final output. Quality is a human concept and so there is no way to automate it. I wish there were. :) I am very interested in your text processing scripts though and always have been. I just haven't been processing any OCR text for a while.
If the volunteers at DP are blissfully unaware that their time is being wasted then they do not feel like victims of the process
but they _are_ victims, whether they _feel_like_ victims or not...
They are not victims in the sense that an abused child is an abused child whether the child realizes it or not. We are talking about proofreading and wasting manual hours doing tasks that could have been done by (or assisted by) automation. I may lose sleep over the former, but I'll never lose sleep over the latter. Not that I do not regret the wasting of volunteer time, I just don't have it at the top of my 'causes' list.
and, like i said, despite a constant influx of volunteers sent there by p.g., the steady d.p. base seems to be a fairly constant number, which means they are experiencing severe "churn" and "burnout", neither of which bodes well for the future. the well _will_ go dry. (especially since more and more people will come to see google as "the source" of the books that they try to find online, since google offers people several million more titles than project gutenberg.)
Loss of volunteers is not good, obviously. Just as with any venture, it is twice as hard to win someone back as it was to gain them in the first place. The last I checked, the wiki book project also appears to have very few active members, so it is also possible that book proofing has a cap on the number of people willing to do it with any regularity that has nothing to do with the format of presentation or the process. As I said before, I don't believe it is possible to catch up to google. If DP produces 2400 books per year, then even if the process I outlined above (or your own or anyone else's) increased the production by 5x or 10x or even 25x (for first releases), we would still be far behind the scanners. (Although I do believe your scan figures for daily output by IA and google include a lot of material that is not public domain.) I would very much love for someone to prove me wrong on this point and have a million texts in PG in no time flat.
If the inefficiency bothers you and me (and others), it's a moot point because it is not our time that is being wasted.
but the potential of our society _is_ being spent inefficiently, and the time of _good_people_ is being wasted, for no good reason.
so i think the point is _not_ moot. i believe i should _speak_up_.
Yes, you should do as you please. We all should. You do fight the good fight as they say. As much as I may agree with you most of the time, it just isn't in me to voice out about it. After all, you took my crown. ;)
And, we have no power to change what is at DP. At least that is the way I look at it.
i'm not looking for "power". i'm looking to enlighten people.
'Power' is often the easier goal to attain. ;) I merely meant that we are not in a position to have any say in what DP does with itself. Nor in what PG does with itself. Carel