- gutvol-d - lists.pglaf.org

Improving the PG library
by jon.a＠hursts.eclipse.co.uk 25 Sep '12

25 Sep '12

Hi All, I've had a project for PG library improvement niggling away at the back of my mind for a while now, and would be grateful if those on this list would shoot it down in flames so that I can abandon it and avoid doing a whole heap of work. I will add that Greg has badly let me down by being enthusiastic about it. The project is to target 40% of PG downloads for improvement, equating to the most popular 1000 titles. There are just two simple steps for each title: Step 1 is to either source or create a master scan (MS). The MS must be the same edition as the extant PG text, and it must be of high enough quality to support accurate OCR. It will be stored with the extant PG text and will become the unequivocal master document for that text, i.e. if any document that purports to be a version of that text disagrees with the master scan, then that document will be considered to be in error (typos in the original can be corrected directly in the MS if required). Step 2 is creation of a reference text transformation (RTT) from the master scan. The RTT is a line by line pixel to UTF-8 transformation without wrapping or additional markup. It is produced by diffing a new proofreading of the MS with the extant PG text so that only errors common to both will remain. The new proofreading will use DP's P1 and P2 rounds, possibly repeated, with a single project level directive to not clothe eol hyphens and dashes -- essentially a standard DP LOTE style project skipping P3, F1, F2 and PP. The RTT will be stored by PG alongside the MS. The point of all this? The immediate benefit is that the RTT can be used to produce a comprehensive list of errata for its extant text, thus allowing the WW to eliminate the vast majority of errors in one hit. The future benefit is that the MS is a perfect universal master format: it does not include any introduced errors and it codes every formatting nuance. All transformations from MS to either intermediate master formats (TEI, RST, ZML, LaTeX etc.) or final formats (HTML, PDF, epub, mobi etc.) initially follow much the same path, and the RTT represents a fairly late divergence point. Personally, I intend to use them to make a raft of Kindle sized PDFs via LaTeX. Hopefully someone else might take care of epub and mobi. Maybe Amazon might use them to sort their Kindle Store versions out. The point is you end up with the foundations to do interesting work without having to first do a whole lot of boring OCRing and proofreading, and you can easily track changes in the RTT to keep your own version up to date. Additional benefits: both the MS and RTT are usable ebook formats in and of themselves, and the combination will allow a pretty nifty errata system to be written, whereby a reader types in a suspect phrase, gets taken to the line in the MS where that phrase is found, and can deliver the errata by simply clicking on the line and clicking a "Please Check" button. Weak points that I can see: 1. I know little about copyright law and less about the editions used for current texts, so I'm not going to be able to find scans suitable for the MS without help. 2. Whilst 1000 books is only 2.5% of the PG library, it is still rather a lot of books. I reckon I might get through 1 a month by myself. To complete the project within 2 years, it would need 42 people working at the same rate. That's rather a lot of people. 3. The elephant in the room... most of the heavy lifting relies on DP, and that means convincing Louise to let us use DP's P1 and P2 rounds. Even though standard DP workflow is used and it will help balance the rounds, I have my doubts she will agree. So... get out your best negativity and fire away! Cheers Jon

11 27

pride and prejudice blah blah blah
by Bowerbird＠aol.com 25 Sep '12

25 Sep '12

internet archive has 3 scan-sets of "pride and prejudice". i will mount all 3 in a "continuous proofing" interface _if_ 5 people here agree to bring the text to perfection, which -- by my definition -- includes formatting as well. let's see who's willing to walk, and not just talk. -bowerbird p.s. i don't care if p.g. mounts the finished products. i will.

2 1

walking, not talking
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

don said: > I'm not familiar with the "Best Practices" mentioned. > It would be interesting to see if it includes it is perhaps sad that you're not included on the committee that's working on that, don, because you'd have a lot to add. but it's _definitely_ sad that they aren't even open enough about their process that you would know they're doing it... meanwhile, if anyone wants to catch up with z.m.l. and _jaguar_ -- the tool-set for turning z.m.l. into e-books -- you can check out the information webpage located here: > http://zenmagiclove.com/september.html there's a live demo up. if anyone here would like their own personal sandbox to play in, just send me a backchannel... it's too easy to pick at the first draft of jon's ideas. try coming and attacking my more-solid stuff instead. unless you're too cowardly to do anything but bully... -bowerbird p.s. the text for 1 of the 3 copies of "pride and prejudice" is now up, and viewable, for those of you who wanna walk: > http://zenmagiclove.com/prxpr/prxpr.zml

1 0

blah blah blah blah blah blah blah blah blah
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

haven't we talked about all this stuff enough? either _do_something_ or shut up. sincerely. -bowerbird

1 0

Re: [gutvol-d] procedures for contributions, forum for questions and ideas
by James Adcock 24 Sep '12

24 Sep '12

I repeated these experiments one more time in epub format. Again "yes" means "yes it is reasonably formatted" but "no" means "no it is not reasonably formatted." 40000 no 40001 no 40002 no 40003 no 40004 no 40005 no 40006 no 40007 no 40008 yes 40009 no

1 0

improving blah blah blah
by Bowerbird＠aol.com 24 Sep '12

24 Sep '12

roger said: > Bowerbird has warned against proceeding > and I respect his historical perspective. > Can we avoid the mistakes of the past attempts? i "warned" against proceeding only because it will end up being something jon regrets... so, jon, here's what i can tell you... your goal is fine, but unless p.g. powers-that-be change what they have _always_ done in the past, the goal is not obtainable, and your work wasted. ask jim what happened when he redid a bad e-text... end-users basically ignored the new one for the old. unless _all_ the p.g. powers-that-be _guarantee_ that won't happen to this project, it will be sunk... (do _not_ let the blanket "yes" that greg always gives in response to any proposal _fool_ you... greg himself couldn't put his _own_ proposal into play back at the start of this year, once it came up against a too-deadly combination of (1) reality, (2) whitewashers, and (3) marcello.) so unless you're happy with the mere fact of "having done it", even though the downloads continue to accrue to the old flawed originals, you're better off simply walking away today... but if you wanna do it just for fun, go ahead! if you want more-specific advice, here it is: 1. maybe it's just me that's confused, but you don't seem to be very clear on whether you want to increase the quality of the text, or of the derivative formats. i suppose that you could care about both, but neither side is all that difficult to obtain, by means that are relatively straightforward, so i'm unsure why a big campaign needs to be mounted... but, you know, have at it, and all that rot. :+) 2. doing 10 books is more than enough to tell you whether you want to do any more... execute in 10-book chunks, for your sanity. that way you'll be able to stop without having to feel that you "failed" in your overall mission. it will also mean you don't have to recruit first; and later, if everything works well, your results will do the recruiting for you. (see also point 4.) and perhaps most important of all, you will see just exactly how much p.g. _really_ supports you. 3. skip d.p. _entirely._ it's hard enough to battle through one bureaucracy; two is nigh impossible. you will _never_ get d.p. to buy into your changes, because it'll be afraid the behavior will generalize. besides, you don't _need_ d.p. it's not the proofing that's the difficult part of this task, it's really not... all you need is a system to do "continuous proofing", and roger has already built such a thing. (so have i.) 4. as for volunteers, have p.g. route some your way. (that's how d.p. gets its volunteers.) that'll be a good test of the _true_ p.g. commitment to your project... you could also recruit from the unhappy volunteers who are in the process of abandoning d.p. nowadays, specifically p1/p2 proofers with years of experience. provide these skilled people with new appreciation! 5. although the suggestion to go with first editions _sounds_ good, i'd be careful. when i've examined the changes made from one edition to the next one, it certainly appears that a lot of errors are corrected. so i'm inclined to think later editions will be "better". for instance, in "pride and prejudice", see this change: > Mrs. Long and her daughters must stand their chance; > Mrs. Long and her nieces must stand their chance; i'm guessing whichever one came _later_ was _correct_. first editions appeal to collectors, not text-obsessives. *** in summary... absolutely, many p.g. e-texts need to be improved! sadly, however, the p.g. bureaucracy often resists it. -bowerbird

4 4

Improving blah blah blah
by Robert Gibbins 23 Sep '12

23 Sep '12

BB wrote: >5. although the suggestion to go with first editions >_sounds_ good, i'd be careful. when i've examined >the changes made from one edition to the next one, >it certainly appears that a lot of errors are corrected. >so i'm inclined to think later editions will be "better". >for instance, in "pride and prejudice", see this change: >> Mrs. Long and her daughters must stand their chance; >> Mrs. Long and her nieces must stand their chance; >i'm guessing whichever one came _later_ was _correct_. >first editions appeal to collectors, not text-obsessives. I think that BB's point of exercising a certain amount of care about which edition to use before launching into a time-consuming project, is a good one, actually I suggested, like Peter Hatch, that Jon use the 1923 R. W. Chapman edition. Since Jane Austen died in 1818 or thereabouts this could hardly be considered a first edition. My understanding is that what Chapmann did is strip away layers of Victorian compositors 'improvements' particularly to punctuation and spelling, a process somewhat analogous to restoring an Old Master, or going back to the original score of a Mozart symphony. It's a scholarly edition rather than a first edition. (And yes Chapman's edition does have 'nieces' not 'daughters') Bob Gibbins

1 0

improving the blah blah blah
by Bowerbird＠aol.com 23 Sep '12

23 Sep '12

jon said: > would be grateful if those on this list > would shoot it down in flames so that I can > abandon it and avoid doing a whole heap of work. the list apparently prefers that you will go ahead and do a whole heap of work. which it will then shoot down in flames. go back and read the archives of this list in january and february of (yes) this year to get a feel for this topic's uselessness: > http://lists.pglaf.org/mailman/private/gutvol-d/2012-January.txt.gz > http://lists.pglaf.org/mailman/private/gutvol-d/2012-February.txt.gz meanwhile, for pride and prejudice, try: > http://zenmagiclove.com/prapr you will find various work i did on that book -- in the swirl of that exchange back then -- which might help guide you, if you continue... but my best advice is this: "it's a trap! turn back!" -bowerbird

2 1

Improving the PG library
by Robert Gibbins 23 Sep '12

23 Sep '12

Re: 'Pride and Prejudice', you could try one of the Oxford texts, starting with the R. W. Chapman edition of 1923 which tried to go back to Jane Austen's original, in so far as that could be determined. I don't have it, but I believe the Oxford English Novels edition, is/was the latest version of this. The new Oxford World's Classics paperback edition, which I bought recently, also says it contains the same text. Bob Gibbins

2 1

Readable epubs
by Robert Gibbins 20 Sep '12

20 Sep '12

This thread was kicked off by Stevan White saying that he had taken some PG-generated epubs apart and re-formatted them, and that he though they were improved by doing this. Having been through this process myself about 18 months ago, and got nowhere with persuading PG to do anything about its epubs (my original proposal was simply that PG should add hand-corrected epubs to its current repository, which may now seem naïve, but was well-intentioned), I contemplated the feasibility of starting a web site myself, persuading others to join me, etc., etc. (and also having an offer to host my books on his site from BB), but eventually concluded that since my basic interest is in books, not running a web site, the simplest thing to do was to add them to the MobileRead library at (http://www.mobileread.com/) Now my personal belief (though I can't prove it) is that: 1. What Jim Adcock calls the dirt simple books constitute the vast majority of downloads from PG. 2. Much, if not most, of the great literature of the English language that interests me is already in the PG repository (although sadly often in a somewhat unsatisfactory state) and furthermore is contained within the above-mentioned dirt simple subset. 2. Most of these books could be satisfactorily encoded by limiting HTML markup to the styles contained in a relatively small css stylesheet, and by obeying a small number of rather uncontroversial HTML coding standards (things like what heading levels to use for volumes, chapters, sections within chapters, etc.). (They are, after all, dirt simple.) 3. If these books were thus encoded, then it would be a relatively simple matter to generate epubs from the HTML (and maybe even simpler to do it the other way round). Probably also you could generate plaintext from the HTML, so that you could (for these books) have a single master version in HTML. Over the last eighteen months as I've reformatted various PG-generated epubs in order to read them, I have slowly refined a set of styles which seem to work across at least a certain number of the dirt simple books. I don't kid myself that what I am currently doing is the best or most flexible possible answer. I didn't know HTML particularly well when I started, and I have already had one major re-think earlier this year when I went through at least two design iterations for the style sheet before eventually recognising that I was re-inventing the 'Decorator' design pattern from the book 'Design Patterns' by Gamma, Helm, Johnson & Vlissides. I've also had some very useful help from several contributors to this forum. (Anyone who is interested can see a fairly up-to-date version of the stylesheet I am using contained in http://www.mobileread.com/forums/showthread.php?p=1371707#post1371707 which is an Anthony Trollope novel "The Belton Estate".) I'd like to know if Stevan's books can teach me anything, so I would like to request that Stevan puts his books somewhere (maybe MobileRead, maybe he has a better place) and appends their URLs to this forum so that those of us who are interested in this problem have the opportunity to learn something. Bob Gibbins

1 0