- gutvol-d - lists.pglaf.org

and the crickets chirp
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

don said: > One wonders about how wwers distinguish between this would be the time where the whitewashers come in and delineate their policies for all of you to understand. cue the fanfare for the entry of the whitewashers... but i'm betting they won't do that. rather than explain a policy that you won't like one bit, they'll just stay silent and continue to be the black-box that you have to try and figure out from their actions... (and/or "non-actions", as the case might be.) but hey, can you blame them? they're just trying to protect their territory. each of them has put a _ton_ of work into p.g. -- quite literally thousands and thousands of hours, and some of them _tens_of_thousands_of_hours_. is it any wonder they want to protect the "old" versions? if they don't make that the traditional convention for p.g., then they know that _their_ work will soon be replaced too. the producers from the early days had to work _hard_. they had to buy expensive scanners and o.c.r. programs, or type in books by hand. and they did darn good work, considering the difficulty inherent in the task back then. now you guys wanna waltz in, with your scan-sets that cost you no work or time or energy, because you just downloaded them from google or the internet archive, who also did the o.c.r. for you. it's all so easy for you. you need to look at these matters from their perspective. there's also the tacit admission, when they "replace" an "old" e-text, that it was flawed, and needed corrections. that's sometimes a painful place to be forced to go to... so the question of updating e-texts is always gonna be one that is fraught with misgivings and mixed motives. *** on the other hand, the other problem here -- which many of you are conflating -- is the low quality of e-book files. here the problem lies with marcello, not the whitewashers. now, as most of you know, i think marcello is an asshole, because he _is_ an asshole. (most of you know that, too.) and yes, the problem with the low quality of e-book files is partly due to the fact that marcello is indeed an asshole. but unlike the difficulties involved with updating e-texts, it would be extremely simple to fix the e-book file output. a series of small changes would correct most of the flaws, right out of the box, and it wouldn't take much time at all. (this is why people can do the modifications so quickly and easily by themselves, because there's not that much to do.) but let me just, for half a minute here, defend marcello too. i can well understand why he would be stubborn about this, and refuse to implement even that series of small changes. it's because the people from d.p. have proven that they are unwilling to do what _they_ have to do, in order to improve the quality of the e-book output. so they've forced marcello to engage in a stand-off. he's saying, "if they won't budge, neither will i, and the quality of the e-book output will suffer." ok, so that's not a very strong defense, i grant you, but still, at least it shows why he might be motivated in such a way... otherwise, you've got to assume that he is purposely doing a sabotage of the e-book files for no good reason, and that would be beyond the pale, even for an asshole like marcello. you've got to explain why he wouldn't make the simple and obvious changes everyone knows would improve the output. *** underlying everything? the people who have the power to change things for the better have no interest in doing that. which is why, when you ask them for change or explanation, you will hear nothing except the sound of crickets chirping... -bowerbird

1 0

Re: [gutvol-d] improving blah blah blah
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

jon said: > my hope is that all the PG powers-that-be > are part of PG for the right reasons, that > they recognise that a problem exists, and > that they are keen that it gets fixed. you can grant them all of that. but none of that means that they are going to agree with you that you have the right solution. and they have the keys to the castle. > From what you say, it cannot get fixed > without buy-in from the WWs and Marcello > to a specific course of action, so the problem is > really one of generating consensus around > a course of action. right. and that hasn't ever happened here. ever. and it is clear that your plan will not do it either. i mean, we could all agree to re-do some e-texts -- without committing to your ms/rtt plan at all -- but the new e-texts would still be ignored by users. so you're not paying attention to the _real_ problem. > with an MS, an RTT and Knuth's genius > I can do a solo project that shames Amazon > in a day or two. well, there's another problem. you wanna use tex. go over to d.p. and see the tex contingent there. it's tiny. the learning curve is obviously too hard. > What I can't do is create the MS and RTT by myself. you're wrong. a scan-set is easy to obtain. and getting the text correct is not difficult. so, you know, go ahead and do your demo. do it for one book, e.g., "pride and prejudice". get the text from that pointer i gave to my site. > There seems to be a consensus around > a 10 book pilot forming. Success would mean that > 1 in 20 downloads would be of improved quality. > Sounds good to me. you're counting chickens when you don't even have any eggs yet. > DP co-operation may not be as crucial as I once thought. it's not "crucial" in the slightest. it would be a hindrance. > I think it is worth at least _trying_ to bring them on board. you're biting off more than you can chew as it is. and now you want to stuff a big decaying head in your mouth as well? d.p. stopped growing a very long time ago... soon they will start falling from their plateau. you already have an albatross around your neck. don't try to tie an anchor to your ankle as well... > I fully admit to having precisely zero knowledge > or experience regarding book editions and > availability of scans. I naively thought that > "someone" would know which edition > a given extant text was derived from > and that it would be straightforward to > find a decent scan given that knowledge. > This is cleary very wrong. sometimes it is easy. sometimes it is not. i've been unable to find any scan-set for the text used for the p.g. version of "pride and prejudice". > That we don't necessarily know the edition > of the extant text raises the issue of > which edition we lock in to. Who makes that decision? that's a non-issue. get a scanset from the internet archive, which will give you the o.c.r. as well. you correct that o.c.r. by comparing it against a clean text, resolving the diffs. > I conclude that Don is absolutely correct. > The initial action of a 10 book pilot should > involve nothing more or less than determining, > in each case, which edition and hence which scan > we are going to work with. ok, start with _one_ book, not 10. seriously. one. because you still have _no_ word from a whitewasher or marcello that _any_ change will be made to the site that will allow a corrected e-text to float to the surface. until p.g. gives some indication that it _truly_wants_ corrected editions, you are simply wasting your time. if i were you, i'd make 'em prove their intentions by bringing the books jim re-did to higher prominence. > We need someone knowledgeable to > guide this selection process and if necessary > make a final executive decision in each case. again, you're looking for some type of agreement that has _never_ever_ happened on this list before. besides... there's no reason you can't do _multiple_versions_ of a book. again, take my "pride and prejudice". i took one e-text and diffed it against another one. that pointed out the discrepancies between the two, which were either (1) an o.c.r. error in one or both, or (2) edition differences. you decide which it is by comparing each of the e-texts against its scan-set. at the end of the process, both e-texts are correct, with the caveat that both could conceivably have the identical o.c.r. error located in the same place. (although my research shows that to be very rare, quite less likely than one would think it might be.) being able to present multiple editions, and generate pointed evidence of the changes made across them, is an empowering thing, one that might be considered to be of considerable value by some people out there. scan-sets are plentiful these days. there's no reason to think that we need to limit ourselves to one edition. (having said that, however, i see _no_ useful purpose served by an e-text that _cannot_ show its provenance in a solid demonstrable way by pointing to a scan-set. but that's a point that i have made many times before.) *** greg said: > This is exactly opposite the PG policy. > We specificaly do NOT adhere to any print edition. see what kind of lunacy you are up against, jon? a policy that once made sense was retained until it no longer produced any solid benefit, and then _retained_even_longer_ as it became a liability, and now _is_still_retained_ even when it is stupid. heck, these people can't even _spell_ "specifically". > (That is part of why you will find it really hard to > find a matching print source for many PG eBooks.) which makes it very hard to submit an error-report that the whitewashers can't reject if they want to... now we are in the sad situation where the world is awash in p.g. e-texts which have zero provenance. if this doesn't change, there will come a time, and it's not far down the line, where project gutenberg will come to be considered a _liability_ to e-books, an example of _how_not_to_do_it_. how depressing. -bowerbird

4 3

Summary: Improving PG library
by Jon Hurst 25 Sep '12

25 Sep '12

Back to RL work tomorrow, so I thought I'd summarise where this discussion stands, at least from my point of view: (1) It is recognised by all concerned that a problem exists with the accuracy and/or typography in PG's top downloaded books. There is a desire to do something about this from Greg and at least a number of members of this list. (2) The first vital step towards doing something is to link each title to a scan set. We do not know the editions of the extant text, so choosing a scan set means choosing an edition. For new entries to the library this is the prerogative of the creator. How to choose one for the updates has not yet been answered. I would prefer it done by someone or a group of someones nominated by Greg with reference to the PG community at large. As I do not have the required level of knowledge to make executive decisions about editions, I can be of no help here. (3) Once master scans are uploaded, it will be possible to make some progress. There will be no point producing final derivative versions that will be buried by the current version. What _can_ be done is to create a clean text by whatever means and diff it against the extant text, thereby producing a comprehensive errata list for the WWs. This would at least take care of the accuracy side of things. (4) According to Bowerbird, Step 3 is possible by simply diffing a fresh OCR of an MS against the extant version: most of the errors highlighted would be in the OCR, but it should also highlight the majority of the errors in the extant text. If this is correct, DP does not need to be involved. (5) The typography side of things cannot be addressed until a method is devised to determine whether a redo is superior to a current version of the same format, and should therefore replace it. Currently the method is to just publish both versions, meaning that from a user's perspective, the current version always appears superior to new versions, so there is no incentive to rework it. I am still keen to help with this work, and will be happy to attempt (3) for a pilot of 10 books over the Winter when I have some time off. (2) is, however, outside my control, and nothing can be done until it is complete. I am also happy to create LaTeX versions of these books if (5) is resolved. Cheers Jon

2 1

More useless data
by Lee Passey 25 Sep '12

25 Sep '12

Apparently the PG archives from 2007 and 2008 have been lost, so I'm resending this message from November of 2007 for the benefit of Messrs. Hurst and Kretz [begin quoted text] Quick ... what is the most commonly downloaded book from project Gutenberg in the last three years? I promised Jon Noring some data a few months back, and I thought I'd deliver it in this forum, because some other people might find it interesting. As most people here know, TPTB at project Gutenberg deny having any download statistics beyond the past 30 days. Fortunately, for years now the Internet Archive has been trolling the internet, making periodic snapshots of web sites, including Project Gutenberg. So I went to the Internet Archive and captured the Project Gutenberg statistics pages since September 2004. I collated all the data, and came up with a list of 408 files which have appeared in the "30 day - Top 100" since that date. I added and resorted them, and now have a list of the most popular downloads from the PG web site since Sept. 2004. And the most popular download during the past three years is: (drumroll please) The Notebooks of Leonardo Da Vinci - Complete by Leonardo da Vinci (etext 5000) The rest of the top ten are: 2 The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle (9551 & 1661) 3 The Art of War by Sun Tzu (132, 17405 & 20594) 4 Le Kamasutra by Vatsyayana (14609) 5 Pride and Prejudice by Jane Austen (1342 & 20686) 6 The War of the Worlds by H. G. Wells (36 & 8976) 7 Ulysses by James Joyce (4300) 8 Little Journeys to the Homes of the Great - Volume 01 of 14 by Elbert Hubbard (12933) 9 Manual of Surgery by Alexander Miles and Alexis Thomson (17921) 10 Hand Shadows to Be Thrown upon the Wall by Henry Bursill (12962) 11 The Adventures of Huckleberry Finn by Mark Twain (76 & 19640) 12 Alice's Adventures in Wonderland by Lewis Carroll (11, 19573 & 928) (I know this is 12, but I couldn't bear to leave out Alice and Huck.) (Caveat: I was unable to get precise 30 day intervals, so this list is an approximation. A /very good/ approximation, but an approximation nonetheless.) (Caveat bis: These data are derived from that reported on the PG web site. They are only as good as PG's reporting.) Of course, because the PG corpus is always growing, this kind of linear analysis may over-weight early downloads. So I changed the collation algorithm a bit. I started with a 6 month baseline, and then as I added each 30 day list I increased the weighting by 4%. That is, the data as of Feb. 2005 was counted at 100%, but the data from Feb-Mar was counted at 104%, the data from Mar-Apr was counted at 108%, the data from Apr-May was counted at 112%, etc. Thus, more recent downloads got counted more heavily that more distant downloads. So what is the adjusted top ten list? 1 The Notebooks of Leonardo Da Vinci - Complete by Leonardo da Vinci (5000) 2 The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle (9551 & 1661) 3 The Art of War by Sun Tzu (132, 17405 & 20594) 4 Le Kamasutra by Vatsyayana (14609) 5 Pride and Prejudice by Jane Austen (1342 & 20686) 6 Manual of Surgery by Alexander Miles and Alexis Thomson (17921) 7 How to Speak and Write Correctly by Joseph Devlin (6409) 8 Ulysses by James Joyce (4300) 9 Little Journeys to the Homes of the Great - Volume 01 of 14 by Elbert Hubbard (12933) 10 The War of the Worlds by H. G. Wells (36 & 8976) 11 The Adventures of Huckleberry Finn by Mark Twain (76 & 19640) As you can see, the Manual of Surgery is more popular recently, and Hand Shadows less so. Alice dropped to 14, so I didn't feel like I could include her. What is interesting is that the addition of new files to the PG corpus has not had much affect on the most popular file downloads. The data for all 400+ files can be found at http://www.passkeysoft.com/~lee/zero.txt and http://www.passkeysoft.com/~lee/four.txt. Bowerbird, if you want to know where to start in your conversion process to z.m.l., I would suggest the books on this list. I hand manipulated the first 50 entries in each file, to try to count multiple editions of the same book as a single entry, the remaining data is raw. Enjoy! -- Nothing of significance below this line. _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/listinfo.cgi/gutvol-d

1 0

presumed blah blah blah
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

don said: > Seems like the original contributor > could be presumed to have implicitly authorized, > indeed welcomed, corrected versions as replacements, > regardless of the magnitude of the resulting change. don appears to have made a few errors there. so let me "fix" them for him. > seems like the original contributor > could not be presumed to have authorized, let alone > welcomed, so-called "corrected" versions as "replacements," > regardless of the insignificance of any resulting changes. there. now it's been "corrected" and "replaced". and perhaps don now sees why he was wrong... -bowerbird p.s. "look at the scan", you object. "what scan?", i ask. there is no scan. i repeat, there is no scan. none at all.

1 0

Improving the PG library
by jon.a＠hursts.eclipse.co.uk 25 Sep '12

25 Sep '12

Hi All, I've had a project for PG library improvement niggling away at the back of my mind for a while now, and would be grateful if those on this list would shoot it down in flames so that I can abandon it and avoid doing a whole heap of work. I will add that Greg has badly let me down by being enthusiastic about it. The project is to target 40% of PG downloads for improvement, equating to the most popular 1000 titles. There are just two simple steps for each title: Step 1 is to either source or create a master scan (MS). The MS must be the same edition as the extant PG text, and it must be of high enough quality to support accurate OCR. It will be stored with the extant PG text and will become the unequivocal master document for that text, i.e. if any document that purports to be a version of that text disagrees with the master scan, then that document will be considered to be in error (typos in the original can be corrected directly in the MS if required). Step 2 is creation of a reference text transformation (RTT) from the master scan. The RTT is a line by line pixel to UTF-8 transformation without wrapping or additional markup. It is produced by diffing a new proofreading of the MS with the extant PG text so that only errors common to both will remain. The new proofreading will use DP's P1 and P2 rounds, possibly repeated, with a single project level directive to not clothe eol hyphens and dashes -- essentially a standard DP LOTE style project skipping P3, F1, F2 and PP. The RTT will be stored by PG alongside the MS. The point of all this? The immediate benefit is that the RTT can be used to produce a comprehensive list of errata for its extant text, thus allowing the WW to eliminate the vast majority of errors in one hit. The future benefit is that the MS is a perfect universal master format: it does not include any introduced errors and it codes every formatting nuance. All transformations from MS to either intermediate master formats (TEI, RST, ZML, LaTeX etc.) or final formats (HTML, PDF, epub, mobi etc.) initially follow much the same path, and the RTT represents a fairly late divergence point. Personally, I intend to use them to make a raft of Kindle sized PDFs via LaTeX. Hopefully someone else might take care of epub and mobi. Maybe Amazon might use them to sort their Kindle Store versions out. The point is you end up with the foundations to do interesting work without having to first do a whole lot of boring OCRing and proofreading, and you can easily track changes in the RTT to keep your own version up to date. Additional benefits: both the MS and RTT are usable ebook formats in and of themselves, and the combination will allow a pretty nifty errata system to be written, whereby a reader types in a suspect phrase, gets taken to the line in the MS where that phrase is found, and can deliver the errata by simply clicking on the line and clicking a "Please Check" button. Weak points that I can see: 1. I know little about copyright law and less about the editions used for current texts, so I'm not going to be able to find scans suitable for the MS without help. 2. Whilst 1000 books is only 2.5% of the PG library, it is still rather a lot of books. I reckon I might get through 1 a month by myself. To complete the project within 2 years, it would need 42 people working at the same rate. That's rather a lot of people. 3. The elephant in the room... most of the heavy lifting relies on DP, and that means convincing Louise to let us use DP's P1 and P2 rounds. Even though standard DP workflow is used and it will help balance the rounds, I have my doubts she will agree. So... get out your best negativity and fire away! Cheers Jon

11 27

pride and prejudice blah blah blah
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

internet archive has 3 scan-sets of "pride and prejudice". i will mount all 3 in a "continuous proofing" interface _if_ 5 people here agree to bring the text to perfection, which -- by my definition -- includes formatting as well. let's see who's willing to walk, and not just talk. -bowerbird p.s. i don't care if p.g. mounts the finished products. i will.

2 1

walking, not talking
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

don said: > I'm not familiar with the "Best Practices" mentioned. > It would be interesting to see if it includes it is perhaps sad that you're not included on the committee that's working on that, don, because you'd have a lot to add. but it's _definitely_ sad that they aren't even open enough about their process that you would know they're doing it... meanwhile, if anyone wants to catch up with z.m.l. and _jaguar_ -- the tool-set for turning z.m.l. into e-books -- you can check out the information webpage located here: > http://zenmagiclove.com/september.html there's a live demo up. if anyone here would like their own personal sandbox to play in, just send me a backchannel... it's too easy to pick at the first draft of jon's ideas. try coming and attacking my more-solid stuff instead. unless you're too cowardly to do anything but bully... -bowerbird p.s. the text for 1 of the 3 copies of "pride and prejudice" is now up, and viewable, for those of you who wanna walk: > http://zenmagiclove.com/prxpr/prxpr.zml

1 0

blah blah blah blah blah blah blah blah blah
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

haven't we talked about all this stuff enough? either _do_something_ or shut up. sincerely. -bowerbird

1 0

Re: [gutvol-d] procedures for contributions, forum for questions and ideas
by James Adcock 24 Sep '12

24 Sep '12

I repeated these experiments one more time in epub format. Again "yes" means "yes it is reasonably formatted" but "no" means "no it is not reasonably formatted." 40000 no 40001 no 40002 no 40003 no 40004 no 40005 no 40006 no 40007 no 40008 yes 40009 no

1 0