There need to be more volunteers working in the latter stages of the

production. And under the current DP "high priesthood" system the only way to get more volunteers working in the latter stages of the production is to get new people working on the earlier stages of production, which then perpetuates the problem. You have to be willing to adjust or modify the "high priesthood" system.

On 3/3/2010 11:29 AM, Jim Adcock wrote:

You have to be willing to adjust or modify the "high priesthood" system.

I have participated, or attempted to participate, in a number of FOSS projects over my career as a programmer, and I have a few observations which you may find relevant. Every successful FOSS project I have ever observed has started with the vision of a single individual. In the years leading up to 1995, Eric A. Young single-handedly managed to implement the full suite of cryptosystems used in SSL, and in that year made it available on the internet for free. This effort became the foundation of OpenSSL. Until he was lured away by RSA, Mr. Young was the driving force behind OpenSSL. Today, the role of visionary is played by Ralf Engelschall and Ben Laurie. In 1991 Andrew Tridgell, another Australian needed to mount disk space from a Unix server to his DOS PC. Using a packet sniffer he was able to reverse engineer the System Message Block protocol used by IBM's NetBIOS system, and which was the basis for DOS and Windows networking. This work eventually became Samga, a Unix/Linux software suite that provides file and print services to Windows-based clients. Mr. Tridgell still participates, and is the driving force behind the Samba open source project. While many have criticized his alleged heavy-handedness, I believe that the success of the Linux kernel is primarily due to the fact that Linux Torvalds still has absolute authority over what changes go into that kernel. Michael Hart plays the same role at Project Gutenberg that these programming giants played in the development of their respective software projects. Project Gutenberg was the brainchild of Mr. Hart, and he continues to be the driving force and visionary behind the project. While he, with uncharacteristic modesty, primarily credits the volunteers for the nature of Project Gutenberg, I disagree. For better or for worse, Project Gutenberg is the product of Mr. Harts vision and tenacity. Distributed Proofreaders was founded in 2000 by Charles Franks to assist in the production of electronic texts specifically to be distributed by Project Gutenberg. According to my recollection, Mr. Franks' theory was that production of e-texts was hampered by the fact that few people were willing to take on the task of producing an entire e-text, particularly through the arduous text proofreading process. His vision was to take a text and break it up into discrete units (in this case, pages) so that many people could be involved in the proofreading process and lightening the burden. Thus, the one time DP catch-phrase, "Proofread a page a day, that's all we ask." The volunteers at Distributed Proofreaders have become very good at proofreading texts. I have also seen any number of FOSS projects which have attempted to begin through consensus and team building. I can't name any of these projects for you, because they have all either failed or were still-born. I think I have learned this lesson from my observations of these projects: to be successful you must have one single visionary who controls, more or less, the project. Having that visionary will not guarantee success, but not having it will surely doom it. At if a project loses its visionary, or marginalizes him or her to the point where he or she no longer controls the vision, the project will become increasingly ineffective and inefficient, and will descend into in-fighting and turf wars as others try to control the vision. Vision cannot be obtained by consensus. When someone criticizes Project Gutenberg for supposed failings, or the inability or unwillingness to keep up with the times, and Michael Hart responds with his now inevitable suggestion to "JUST GO FOR IT," what he is saying is "what you are suggesting does not match my vision. If you feel your vision is better than mine I encourage you to go elsewhere to pursue it. We can offer some infrastructure support (disk space) and you are welcome to invite Project Gutenberg volunteers to go help you actualize your vision, but I will not substitute your vision for mine." I am not prepared to say that Distributed Proofreaders has lost its vision. It is still proofreading a lot of pages every day. It is clearly /not/ an efficient process, but efficiency was not one of the project goals. We are all familiar with the old saw that while one woman can have a baby in 9 months that doesn't mean that 9 women can have a baby in one month. I don't believe that DP is saying "if one person can proofread a text in 10 days, then 10 people can proofread it in one day," but they are saying "100 people can proofread it in two days." Distributed Proofreaders goal was to increase the speed that texts would be proofread, to lighten the load from any one individual and to make the process more fault-tolerant (if one volunteer quit, the project would not need to be restarted). What has happened is that the needs of the consumer has changed. I'm fairly certain that the proofread texts now sitting in DP's Post-Processing queue would meet Michael Hart's standards (or lack thereof, as he is continually telling me he has no standards) and could be released to Project Gutenberg as is. Other consumers, however, have higher standards, and Distributed Proofreaders is now trying to satisfy those standards as well, and those new standards require post-processing of a work as a single unit by a single person. DP's vision and expertise is in the area of distributed proofreading, not in the area of efficient e-book creation. This is why texts languish in the Post-Processing queue. Your problem, Mr. Adcock, is that you believe you can change the vision underlying either of these organization through rational argument. Vision is an intuitive, almost religious, experience, and blind faith is immune to rationality. It is virtually impossible that you will be able to change the vision either of Mr. Hart or whomever is currently the visionary at Distributed Proofreaders. I suspect that this is why Roger Frank has created his own web site for "roundless proofing;" his vision differs from that of Distributed Proofreaders, and it was simply easier to go his own way than to try and change someone else's vision. I believe I agree with every criticism you have leveled at both Project Gutenberg and Distributed Proofreaders, which is to say, I believe I accept your vision. So let me mimic the words of Michael Hart: GO FOR IT! Put together your own project to complete high-quality public domain e-books. You could certainly harvest all of the files currently in the DP post-processing queue to start with. You might be able to grab the HTML files from PG if you can find scans to go with them. Take advantage of the hardware resources that Mr. Newby has offered. Post messages here and at DP inviting volunteers to help you out. No need to return the e-books to either of those organizations; if they want them they will know where to find them. I will help out as much as possible. But please stop trying to convince Distributed Proofreaders or Project Gutenberg to accept a new vision. They are old and are set in their ways. They represent the last internet generation, not the current one. Show us the way forward, and let sleeping dogs lie.

Jim/James: re:

... I submitted two books in good faith to DP which are now stuck there indefinitely after I contributed many many hours of my own time

Of your two projects, the first went from creation to completion of all rounds solely through the normal operation of the queues in two months, and is being post-processed/verified. If you have concerns as PM about specifics of the project or its status, you should contact the post-processor or the PP-verifier of the project via the several means available to you on the DP site. The second project has also gone from creation to completion of P1, P2, P3 and F1 in two months, also solely through the normal operation of the queues, and has been waiting in F2 for seven months. In the normal operation of the queues, this project would release about six weeks from now. That's pretty far from "indefinitely." I have taken the liberty of releasing this project into F2 where a group of F2 volunteers are focusing their efforts on it and will easily complete it before day's end, possibly before this post reaches the list. re:

... and I have no way to get those books back out.

Since you are the project manager, you could have assigned yourself as post- processor and requested that it skip F2. However, as the F2'ers are finding and correcting formatting and other errors, it's probably better that you didn't. Project managers have options within the DP process, including, but not limited to those mentioned above, either of which could have progressed your project. Any of the DP project facilitators, db-req, dp-help, or admins could have heard your concerns and and discussed options with you, which could have saved much of the frustration which you've expressed on this list, had you only asked. David (donovan) James Adcock wrote:

"My problem", if we have to talk about my problems of which there are many, is that I submitted two books in good faith to DP which are now stuck there indefinitely after I contributed many many hours of my own time and tears, and I have no way to get those books back out.

On 3/3/2010 7:21 PM, D Garcia wrote: [snip]

I have taken the liberty of releasing this project into F2 where a group of F2 volunteers are focusing their efforts on it and will easily complete it before day's end, possibly before this post reaches the list.

Man, you've got to love it! Mr. Adcock points out that the production process at Distributed Proofreaders is broken, and offers a sample demonstrating /how/ it is broken. In response, Mr. Garcia removes the sample from the standard process and deals with it as a special case. In other words, instead of trying to fix the broken process, Mr. Garcia has simply tried to neutralize the complaint! I've worked at a number of few different companies for which this was just Standard Operating Procedure ... most of whom are no longer with us.

I will in this case vouch for at least part of the representation given by David (donovan). What you experienced is in fact the primary method employed by the DP process managers for trying to ameliorate the consequences of their system. When a perceived deficiency is detected, it is defined as a "special case" and given "special treatment". So your first project probably qualifies as a "First Project", and therefore has access to a good deal of standard "special treatment" that you might not have been aware of (though it was your responsibility to be so, unfortunately.) Your second project may have also been qualified for another standard "special treatment"; I'm not very familiar with all the nuances, but put he certainly is - as he points out, he is one of those primarily responsible for it. (In tact, it's also true that he is one of the primary gatekeeepers for innovation and process improvement generally.) It's too bad you had the misfortune to be advised by someone not familiar with the proper navigation of the dp process. As is apparently also the case for whoever is responsible for the Shakespeare projects.

Lee Passey wrote:

... Rather the problem [Jim] identified is that somehow the current production processes allow /any and all/ projects to become backed up in that queue. Given that problem statement, I would have liked to have seen something more like one of the following responses:

[Lee's four hypothetical responses omitted]

Instead we saw a response more along the lines of:

[Paraphrase of my actual response omitted] I addressed Jim's issue here solely because gutvol-d is where he raised it.

It seems to me that this kind of response is designed, in fact, to ignore the issue at hand, which is that changes need to be made at D.P. to increase the throughput of e-texts.

I don't believe that there is anyone at DP at any level of participation who is unaware of the need for improvements in the process. However, the variously proposed "solutions" run the gamut from the obviously naive/simplistic, through horribly manual kludges, all the way up to byzantine complexities requiring considerable effort from the entire volunteer base. Your statement even reflects a common barrier to getting a grip on the issues: "to increase the throughput of e-texts" is actually a statement of goal. While increasing the throughput of the process should be and is a component of DP's long-term goals, the specific problems and their underlying causes need to be identified first in order to effectively address them.

Now it very well may be that this problem has already been recognized by The Powers That Be, and that a solution will be in place Real Soon Now. In that case, wouldn't it have been better to just say so?

I'm certain some of these problems have been identified and the underlying causes and potential solutions are being examined, but I'm equally certain that no consensus can be achieved within the DP community as to what the causes are, much less what solutions are feasible, achievable, or even desirable. Whatever solutions do eventually result, interim or otherwise, some fraction of volunteers at DP will disagree. By no means am I ignoring the broader issues at DP--but I feel those are usually better discussed in a more appropriate venue. Overall, experience has shown that any discussion of DP on gutvol-d generally and unfortunately serves little productive purpose. While positive and insightful comments do occur, (and are read and appreciated!), they are easily lost in the background of posts which far too often contain derision, belittlement, accusation, and misrepresentation. One almost wonders whatever happened to basic respect. But then I remember the synergistic relationship between media and popular culture (of which old books are an excellent reminder). :) David (donovan)

Robert A. Heinlein said it 1941. Not original with Robert J. Hanlon [many suspect error from Robert Heinlein to Robert Hanlon] Napoleon might have said something like it.

While Lee's comments are pretty great, there are a few comments/corrections: On Wed, 3 Mar 2010, Lee Passey wrote:

On 3/3/2010 11:29 AM, Jim Adcock wrote:

...

You have to be willing to adjust or modify the "high priesthood" system.

I have participated, or attempted to participate, in a number of FOSS projects over my career as a programmer, and I have a few observations which you may find relevant.

Every successful FOSS project I have ever observed has started with the vision of a single individual. In the years leading up to 1995, Eric A. Young single-handedly managed to implement the full suite of cryptosystems used in SSL, and in that year made it available on the internet for free. This effort became the foundation of OpenSSL. Until he was lured away by RSA, Mr. Young was the driving force behind OpenSSL. Today, the role of visionary is played by Ralf Engelschall and Ben Laurie.

In 1991 Andrew Tridgell, another Australian needed to mount disk space from a Unix server to his DOS PC. Using a packet sniffer he was able to reverse engineer the System Message Block protocol used by IBM's NetBIOS system, and which was the basis for DOS and Windows networking. This work eventually became Samga, a Unix/Linux software suite that provides file and print services to Windows-based clients. Mr. Tridgell still participates, and is the driving force behind the Samba open source project.

While many have criticized his alleged heavy-handedness, I believe that the success of the Linux kernel is primarily due to the fact that Linux Torvalds still has absolute authority over what changes go into that kernel.

Michael Hart plays the same role at Project Gutenberg that these programming giants played in the development of their respective software projects. Project Gutenberg was the brainchild of Mr. Hart, and he continues to be the driving force and visionary behind the project. While he, with uncharacteristic modesty, primarily credits the volunteers for the nature of Project Gutenberg, I disagree. For better or for worse, Project Gutenberg is the product of Mr. Harts vision and tenacity.

Distributed Proofreaders was founded in 2000 by Charles Franks to assist in the production of electronic texts specifically to be distributed by Project Gutenberg. According to my recollection, Mr. Franks' theory was that production of e-texts was hampered by the fact that few people were willing to take on the task of producing an entire e-text, particularly through the arduous text proofreading process. His vision was to take a text and break it up into discrete units (in this case, pages) so that many people could be involved in the proofreading process and lightening the burden. Thus, the one time DP catch-phrase, "Proofread a page a day, that's all we ask." The volunteers at Distributed Proofreaders have become very good at proofreading texts.

I have also seen any number of FOSS projects which have attempted to begin through consensus and team building. I can't name any of these projects for you, because they have all either failed or were still-born.

Sadly to say, this is all too true, both locally and nationally, not to mention internationally.

I think I have learned this lesson from my observations of these projects: to be successful you must have one single visionary who controls, more or less, the project. Having that visionary will not guarantee success, but not having it will surely doom it. At if a project loses its visionary, or marginalizes him or her to the point where he or she no longer controls the vision, the project will become increasingly ineffective and inefficient, and will descend into in-fighting and turf wars as others try to control the vision.

I would like think that Project Gutenberg, and Distributed Proofreaders will continue on without me until they can't find anything more to do on eBooks, and perhaps then even continue on to something else.

Vision cannot be obtained by consensus.

I suppose I have been lucky enough to have managed this once or twice.

When someone criticizes Project Gutenberg for supposed failings, or the inability or unwillingness to keep up with the times, and Michael Hart responds with his now inevitable suggestion to "JUST GO FOR IT," what he is saying is "what you are suggesting does not match my vision. If you feel your vision is better than mine I encourage you to go elsewhere to pursue

"I encourage you to go elsewhere to pursue it" is not quite correct, even though there is some amerlioration below. We are more than happy to house any free eBooks efforts right here at Project Gutenberg, with or without our gutenberg.org or pglaf.org domain being associated, it's pretty much up the the people in question, and if they don't want some asscociation with PG we will provide readingroo.ms, etc., etc., etc. We will provide ALL of the infrastructure possible, and ask volunteers to help, but, being volunteers, it is really up to them. To lead here at Project Gutenberg you have to lead by example. DO SOMETHING!!! [You'll probably have to do it a couple dozen times.] Then ask others to get on the bandwagon with you and do it some more. When this works it is like starting an avalanche with snowballs. /// I think if Mr. Bowerbird had been willing to follow such a plan and to post an example of a completed book he did once a month, or even once every two or three months, he/we would have dozens of them online by now and there would no longer be arguments of such hypothetical types, but much more concretized. I must state for the record that I have encouraged him to this, pretty much every single year he has been here. I would encourage anyone/everyone else to do the same. It's all you would have to do to wrest "control" of PG from me, and then I could go invent something else.

it. We can offer some infrastructure support (disk space) and you are welcome to invite Project Gutenberg volunteers to go help you actualize your vision, but I will not substitute your vision for mine."

Not quite right: What I will not do, as asked so many times, is to state for official record that YOU are the official boss of Project Gutenberg and that YOUR method IS THE ONLY OFFICIAL METHOD OF PROJECT GUTENBERG.

I am not prepared to say that Distributed Proofreaders has lost its vision. It is still proofreading a lot of pages every day. It is clearly /not/ an efficient process, but efficiency was not one of the project goals. We are all familiar with the old saw that while one woman can have a baby in 9 months that doesn't mean that 9 women can have a baby in one month. I don't

No, but a group of women can have an average of one baby per month. When you are dealing with larger numbers it's not exactly the same.

believe that DP is saying "if one person can proofread a text in 10 days, then 10 people can proofread it in one day," but they are saying "100 people can proofread it in two days." Distributed Proofreaders goal was to increase the speed that texts would be proofread, to lighten the load from any one individual and to make the process more fault-tolerant (if one volunteer quit, the project would not need to be restarted).

Actually, 10 people CAN do that kind of job in one day, and have!!! However, it is nice to have both someone at the wheel and a substitute.

What has happened is that the needs of the consumer has changed. I'm fairly certain that the proofread texts now sitting in DP's Post-Processing queue would meet Michael Hart's standards (or lack thereof, as he is continually telling me he has no standards)

Again not quite right: It's not that I have no standards, I just don't force them on people. Even when it comes down to hard and fast accuracy percentages, I will state the accuracy level I hope for at any given time. Right now it is 99.975% Earlier it was 99.95% [co-opted by the Library of Congress, hee hee!] Before that it was 99.9%, but that was when I started with a version 0.1 not a version 1.0, and worked up to 1.0.

and could be released to Project Gutenberg as is. Other consumers, however, have higher standards, and Distributed Proofreaders is now trying to satisfy those standards as well, and those new standards require post-processing of a work as a single unit by a single person.

We always had a single person as the last post-processor. First it was me, then Judy Boss, then me again, then Greg Newby, then me again, then Newby again, etc., etc., etc.

DP's vision and expertise is in the area of distributed proofreading, not in the area of efficient e-book creation. This is why texts languish in the Post-Processing queue.

Your problem, Mr. Adcock, is that you believe you can change the vision underlying either of these organization through rational argument.

Personally, I believe in rational argument, with stated premises followed by stated conclusions, stacked on top of each other to final conclusions. However, as many of you have undoubtedly note bened, when such arguments are put forth, the opposition ignores them in "fair and balanced" ways. [Just to make sure those who never heard of "fair and balanced" look it up]

Vision is an intuitive, almost religious, experience, and blind faith is immune to rationality. It is virtually impossible that you will be able to change the vision either of Mr. Hart or whomever is currently the visionary at Distributed Proofreaders.

While my faith in the whole of the eBook movmement and Open Source is pretty much unshakeable, it is a rational faith, not blind, based on the simple cost/benefit ratio. In then end just plain individuals can do all the eBooks and post them where seach engines can find them. It's nice to have large collections, but not necessary.

I suspect that this is why Roger Frank has created his own web site for "roundless proofing;" his vision differs from that of Distributed Proofreaders, and it was simply easier to go his own way than to try and change someone else's vision.

And so too could anyone else, with less effort, and more cooperation. However, doing it yourself has certain inalienable advantages!!!

I believe I agree with every criticism you have leveled at both Project Gutenberg and Distributed Proofreaders, which is to say, I believe I accept your vision. So let me mimic the words of Michael Hart:

GO FOR IT!

Put together your own project to complete high-quality public domain e-books. You could certainly harvest all of the files currently in the DP post-processing queue to start with. You might be able to grab the HTML files from PG if you can find scans to go with them. Take advantage of the hardware resources that Mr. Newby has offered. Post messages here and at DP inviting volunteers to help you out. No need to return the e-books to either of those organizations; if they want them they will know where to find them. I will help out as much as possible.

But please stop trying to convince Distributed Proofreaders or Project Gutenberg to accept a new vision. They are old and are set in their ways. They represent the last internet generation, not the current one. Show us the way forward, and let sleeping dogs lie.

I'm still interested in new visions, but just not those that tell me to do something YOU should be doing, even though I am willing to help. I am willing to help!!! Period. That's the bottom line. And you don't even have to give me or PG any credit. . . .