September 2012 - gutvol-d - lists.pglaf.org

Summary: Improving PG library
by Jon Hurst 25 Sep '12

25 Sep '12

Back to RL work tomorrow, so I thought I'd summarise where this discussion stands, at least from my point of view: (1) It is recognised by all concerned that a problem exists with the accuracy and/or typography in PG's top downloaded books. There is a desire to do something about this from Greg and at least a number of members of this list. (2) The first vital step towards doing something is to link each title to a scan set. We do not know the editions of the extant text, so choosing a scan set means choosing an edition. For new entries to the library this is the prerogative of the creator. How to choose one for the updates has not yet been answered. I would prefer it done by someone or a group of someones nominated by Greg with reference to the PG community at large. As I do not have the required level of knowledge to make executive decisions about editions, I can be of no help here. (3) Once master scans are uploaded, it will be possible to make some progress. There will be no point producing final derivative versions that will be buried by the current version. What _can_ be done is to create a clean text by whatever means and diff it against the extant text, thereby producing a comprehensive errata list for the WWs. This would at least take care of the accuracy side of things. (4) According to Bowerbird, Step 3 is possible by simply diffing a fresh OCR of an MS against the extant version: most of the errors highlighted would be in the OCR, but it should also highlight the majority of the errors in the extant text. If this is correct, DP does not need to be involved. (5) The typography side of things cannot be addressed until a method is devised to determine whether a redo is superior to a current version of the same format, and should therefore replace it. Currently the method is to just publish both versions, meaning that from a user's perspective, the current version always appears superior to new versions, so there is no incentive to rework it. I am still keen to help with this work, and will be happy to attempt (3) for a pilot of 10 books over the Winter when I have some time off. (2) is, however, outside my control, and nothing can be done until it is complete. I am also happy to create LaTeX versions of these books if (5) is resolved. Cheers Jon

2 1

More useless data
by Lee Passey 25 Sep '12

25 Sep '12

Apparently the PG archives from 2007 and 2008 have been lost, so I'm resending this message from November of 2007 for the benefit of Messrs. Hurst and Kretz [begin quoted text] Quick ... what is the most commonly downloaded book from project Gutenberg in the last three years? I promised Jon Noring some data a few months back, and I thought I'd deliver it in this forum, because some other people might find it interesting. As most people here know, TPTB at project Gutenberg deny having any download statistics beyond the past 30 days. Fortunately, for years now the Internet Archive has been trolling the internet, making periodic snapshots of web sites, including Project Gutenberg. So I went to the Internet Archive and captured the Project Gutenberg statistics pages since September 2004. I collated all the data, and came up with a list of 408 files which have appeared in the "30 day - Top 100" since that date. I added and resorted them, and now have a list of the most popular downloads from the PG web site since Sept. 2004. And the most popular download during the past three years is: (drumroll please) The Notebooks of Leonardo Da Vinci - Complete by Leonardo da Vinci (etext 5000) The rest of the top ten are: 2 The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle (9551 & 1661) 3 The Art of War by Sun Tzu (132, 17405 & 20594) 4 Le Kamasutra by Vatsyayana (14609) 5 Pride and Prejudice by Jane Austen (1342 & 20686) 6 The War of the Worlds by H. G. Wells (36 & 8976) 7 Ulysses by James Joyce (4300) 8 Little Journeys to the Homes of the Great - Volume 01 of 14 by Elbert Hubbard (12933) 9 Manual of Surgery by Alexander Miles and Alexis Thomson (17921) 10 Hand Shadows to Be Thrown upon the Wall by Henry Bursill (12962) 11 The Adventures of Huckleberry Finn by Mark Twain (76 & 19640) 12 Alice's Adventures in Wonderland by Lewis Carroll (11, 19573 & 928) (I know this is 12, but I couldn't bear to leave out Alice and Huck.) (Caveat: I was unable to get precise 30 day intervals, so this list is an approximation. A /very good/ approximation, but an approximation nonetheless.) (Caveat bis: These data are derived from that reported on the PG web site. They are only as good as PG's reporting.) Of course, because the PG corpus is always growing, this kind of linear analysis may over-weight early downloads. So I changed the collation algorithm a bit. I started with a 6 month baseline, and then as I added each 30 day list I increased the weighting by 4%. That is, the data as of Feb. 2005 was counted at 100%, but the data from Feb-Mar was counted at 104%, the data from Mar-Apr was counted at 108%, the data from Apr-May was counted at 112%, etc. Thus, more recent downloads got counted more heavily that more distant downloads. So what is the adjusted top ten list? 1 The Notebooks of Leonardo Da Vinci - Complete by Leonardo da Vinci (5000) 2 The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle (9551 & 1661) 3 The Art of War by Sun Tzu (132, 17405 & 20594) 4 Le Kamasutra by Vatsyayana (14609) 5 Pride and Prejudice by Jane Austen (1342 & 20686) 6 Manual of Surgery by Alexander Miles and Alexis Thomson (17921) 7 How to Speak and Write Correctly by Joseph Devlin (6409) 8 Ulysses by James Joyce (4300) 9 Little Journeys to the Homes of the Great - Volume 01 of 14 by Elbert Hubbard (12933) 10 The War of the Worlds by H. G. Wells (36 & 8976) 11 The Adventures of Huckleberry Finn by Mark Twain (76 & 19640) As you can see, the Manual of Surgery is more popular recently, and Hand Shadows less so. Alice dropped to 14, so I didn't feel like I could include her. What is interesting is that the addition of new files to the PG corpus has not had much affect on the most popular file downloads. The data for all 400+ files can be found at http://www.passkeysoft.com/~lee/zero.txt and http://www.passkeysoft.com/~lee/four.txt. Bowerbird, if you want to know where to start in your conversion process to z.m.l., I would suggest the books on this list. I hand manipulated the first 50 entries in each file, to try to count multiple editions of the same book as a single entry, the remaining data is raw. Enjoy! -- Nothing of significance below this line. _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/listinfo.cgi/gutvol-d

1 0

presumed blah blah blah
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

don said: > Seems like the original contributor > could be presumed to have implicitly authorized, > indeed welcomed, corrected versions as replacements, > regardless of the magnitude of the resulting change. don appears to have made a few errors there. so let me "fix" them for him. > seems like the original contributor > could not be presumed to have authorized, let alone > welcomed, so-called "corrected" versions as "replacements," > regardless of the insignificance of any resulting changes. there. now it's been "corrected" and "replaced". and perhaps don now sees why he was wrong... -bowerbird p.s. "look at the scan", you object. "what scan?", i ask. there is no scan. i repeat, there is no scan. none at all.

1 0

Improving the PG library
by jon.a＠hursts.eclipse.co.uk 25 Sep '12

25 Sep '12

Hi All, I've had a project for PG library improvement niggling away at the back of my mind for a while now, and would be grateful if those on this list would shoot it down in flames so that I can abandon it and avoid doing a whole heap of work. I will add that Greg has badly let me down by being enthusiastic about it. The project is to target 40% of PG downloads for improvement, equating to the most popular 1000 titles. There are just two simple steps for each title: Step 1 is to either source or create a master scan (MS). The MS must be the same edition as the extant PG text, and it must be of high enough quality to support accurate OCR. It will be stored with the extant PG text and will become the unequivocal master document for that text, i.e. if any document that purports to be a version of that text disagrees with the master scan, then that document will be considered to be in error (typos in the original can be corrected directly in the MS if required). Step 2 is creation of a reference text transformation (RTT) from the master scan. The RTT is a line by line pixel to UTF-8 transformation without wrapping or additional markup. It is produced by diffing a new proofreading of the MS with the extant PG text so that only errors common to both will remain. The new proofreading will use DP's P1 and P2 rounds, possibly repeated, with a single project level directive to not clothe eol hyphens and dashes -- essentially a standard DP LOTE style project skipping P3, F1, F2 and PP. The RTT will be stored by PG alongside the MS. The point of all this? The immediate benefit is that the RTT can be used to produce a comprehensive list of errata for its extant text, thus allowing the WW to eliminate the vast majority of errors in one hit. The future benefit is that the MS is a perfect universal master format: it does not include any introduced errors and it codes every formatting nuance. All transformations from MS to either intermediate master formats (TEI, RST, ZML, LaTeX etc.) or final formats (HTML, PDF, epub, mobi etc.) initially follow much the same path, and the RTT represents a fairly late divergence point. Personally, I intend to use them to make a raft of Kindle sized PDFs via LaTeX. Hopefully someone else might take care of epub and mobi. Maybe Amazon might use them to sort their Kindle Store versions out. The point is you end up with the foundations to do interesting work without having to first do a whole lot of boring OCRing and proofreading, and you can easily track changes in the RTT to keep your own version up to date. Additional benefits: both the MS and RTT are usable ebook formats in and of themselves, and the combination will allow a pretty nifty errata system to be written, whereby a reader types in a suspect phrase, gets taken to the line in the MS where that phrase is found, and can deliver the errata by simply clicking on the line and clicking a "Please Check" button. Weak points that I can see: 1. I know little about copyright law and less about the editions used for current texts, so I'm not going to be able to find scans suitable for the MS without help. 2. Whilst 1000 books is only 2.5% of the PG library, it is still rather a lot of books. I reckon I might get through 1 a month by myself. To complete the project within 2 years, it would need 42 people working at the same rate. That's rather a lot of people. 3. The elephant in the room... most of the heavy lifting relies on DP, and that means convincing Louise to let us use DP's P1 and P2 rounds. Even though standard DP workflow is used and it will help balance the rounds, I have my doubts she will agree. So... get out your best negativity and fire away! Cheers Jon

11 27

pride and prejudice blah blah blah
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

internet archive has 3 scan-sets of "pride and prejudice". i will mount all 3 in a "continuous proofing" interface _if_ 5 people here agree to bring the text to perfection, which -- by my definition -- includes formatting as well. let's see who's willing to walk, and not just talk. -bowerbird p.s. i don't care if p.g. mounts the finished products. i will.

2 1

walking, not talking
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

don said: > I'm not familiar with the "Best Practices" mentioned. > It would be interesting to see if it includes it is perhaps sad that you're not included on the committee that's working on that, don, because you'd have a lot to add. but it's _definitely_ sad that they aren't even open enough about their process that you would know they're doing it... meanwhile, if anyone wants to catch up with z.m.l. and _jaguar_ -- the tool-set for turning z.m.l. into e-books -- you can check out the information webpage located here: > http://zenmagiclove.com/september.html there's a live demo up. if anyone here would like their own personal sandbox to play in, just send me a backchannel... it's too easy to pick at the first draft of jon's ideas. try coming and attacking my more-solid stuff instead. unless you're too cowardly to do anything but bully... -bowerbird p.s. the text for 1 of the 3 copies of "pride and prejudice" is now up, and viewable, for those of you who wanna walk: > http://zenmagiclove.com/prxpr/prxpr.zml

1 0

blah blah blah blah blah blah blah blah blah
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

haven't we talked about all this stuff enough? either _do_something_ or shut up. sincerely. -bowerbird

1 0

Re: [gutvol-d] procedures for contributions, forum for questions and ideas
by James Adcock 24 Sep '12

24 Sep '12

I repeated these experiments one more time in epub format. Again "yes" means "yes it is reasonably formatted" but "no" means "no it is not reasonably formatted." 40000 no 40001 no 40002 no 40003 no 40004 no 40005 no 40006 no 40007 no 40008 yes 40009 no

1 0

improving blah blah blah
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

roger said: > Bowerbird has warned against proceeding > and I respect his historical perspective. > Can we avoid the mistakes of the past attempts? i "warned" against proceeding only because it will end up being something jon regrets... so, jon, here's what i can tell you... your goal is fine, but unless p.g. powers-that-be change what they have _always_ done in the past, the goal is not obtainable, and your work wasted. ask jim what happened when he redid a bad e-text... end-users basically ignored the new one for the old. unless _all_ the p.g. powers-that-be _guarantee_ that won't happen to this project, it will be sunk... (do _not_ let the blanket "yes" that greg always gives in response to any proposal _fool_ you... greg himself couldn't put his _own_ proposal into play back at the start of this year, once it came up against a too-deadly combination of (1) reality, (2) whitewashers, and (3) marcello.) so unless you're happy with the mere fact of "having done it", even though the downloads continue to accrue to the old flawed originals, you're better off simply walking away today... but if you wanna do it just for fun, go ahead! if you want more-specific advice, here it is: 1. maybe it's just me that's confused, but you don't seem to be very clear on whether you want to increase the quality of the text, or of the derivative formats. i suppose that you could care about both, but neither side is all that difficult to obtain, by means that are relatively straightforward, so i'm unsure why a big campaign needs to be mounted... but, you know, have at it, and all that rot. :+) 2. doing 10 books is more than enough to tell you whether you want to do any more... execute in 10-book chunks, for your sanity. that way you'll be able to stop without having to feel that you "failed" in your overall mission. it will also mean you don't have to recruit first; and later, if everything works well, your results will do the recruiting for you. (see also point 4.) and perhaps most important of all, you will see just exactly how much p.g. _really_ supports you. 3. skip d.p. _entirely._ it's hard enough to battle through one bureaucracy; two is nigh impossible. you will _never_ get d.p. to buy into your changes, because it'll be afraid the behavior will generalize. besides, you don't _need_ d.p. it's not the proofing that's the difficult part of this task, it's really not... all you need is a system to do "continuous proofing", and roger has already built such a thing. (so have i.) 4. as for volunteers, have p.g. route some your way. (that's how d.p. gets its volunteers.) that'll be a good test of the _true_ p.g. commitment to your project... you could also recruit from the unhappy volunteers who are in the process of abandoning d.p. nowadays, specifically p1/p2 proofers with years of experience. provide these skilled people with new appreciation! 5. although the suggestion to go with first editions _sounds_ good, i'd be careful. when i've examined the changes made from one edition to the next one, it certainly appears that a lot of errors are corrected. so i'm inclined to think later editions will be "better". for instance, in "pride and prejudice", see this change: > Mrs. Long and her daughters must stand their chance; > Mrs. Long and her nieces must stand their chance; i'm guessing whichever one came _later_ was _correct_. first editions appeal to collectors, not text-obsessives. *** in summary... absolutely, many p.g. e-texts need to be improved! sadly, however, the p.g. bureaucracy often resists it. -bowerbird

4 4

Improving blah blah blah
by Robert Gibbins 23 Sep '12

23 Sep '12

BB wrote: >5. although the suggestion to go with first editions >_sounds_ good, i'd be careful. when i've examined >the changes made from one edition to the next one, >it certainly appears that a lot of errors are corrected. >so i'm inclined to think later editions will be "better". >for instance, in "pride and prejudice", see this change: >> Mrs. Long and her daughters must stand their chance; >> Mrs. Long and her nieces must stand their chance; >i'm guessing whichever one came _later_ was _correct_. >first editions appeal to collectors, not text-obsessives. I think that BB's point of exercising a certain amount of care about which edition to use before launching into a time-consuming project, is a good one, actually I suggested, like Peter Hatch, that Jon use the 1923 R. W. Chapman edition. Since Jane Austen died in 1818 or thereabouts this could hardly be considered a first edition. My understanding is that what Chapmann did is strip away layers of Victorian compositors 'improvements' particularly to punctuation and spelling, a process somewhat analogous to restoring an Old Master, or going back to the original score of a Mozart symphony. It's a scholarly edition rather than a first edition. (And yes Chapman's edition does have 'nieces' not 'daughters') Bob Gibbins

1 0