January 2012 - gutvol-d - lists.pglaf.org

Re: [gutvol-d] Recording the quality of a book's OCR
by Lee Passey 29 Feb '12

29 Feb '12

On 1/3/2012 12:52 PM, Lee Passey wrote: > On Tue, January 3, 2012 11:46 am, Edward Betts wrote: > >> I agree we need a way to correct our books. I find it difficult to >> read the ePub or Kindle versions of scanned books because of the >> OCR errors. I'm sure the OCR errors are irritating when using >> text-to-speech. > > As a first step to move this process along I have created a servlet > which parses the Abbyy output file and converts it to HTML. [snip] > My next step will be to allow the document to be gzipped before > downloading. After that, I will add an option to break the HTML file > into multiple files, each of which matches a single page image, and > return the collection as a zip archive. This step is now complete. Additionally, there is now an option to omit (or include, depending on your perspective) word coordinates. > To use this service, in a browser navigate to > "http://www.ebookcoop.net/ebookcoop/FromIA?[iaid]", where [iaid] is > the Internet Archive IDentifer for a specific work. By default, Abbyy output is returned as a single HTML file without word coordinates. To add coordinates add "&coords" to the query string; e.g. "http://www.ebookcoop.net/ebookcoop/FromIA?cu31924097556546&coords". To return the file as a gzipped HTML file add "&gzip" to the query string. To return the file as a zip archive of HTML files where each file represents a single page (and should have the same naming convention as the image files) add "&zip" to the query string. Note that "&zip" and "&gzip" are incompatible, with "&zip" taking precedence; if you use both options, "&gzip" will be ignored. If you were building an online editing tool you would probably want to use a query string like this: "http://www.ebookcoop.net/ebookcoop/FromIA?tarzanofapes00burruoft&coords&zip" Again, let me remind you that this service is running on an old Pentium /// in my basement at the end of a DSL line, so expect it to be quite slow (it could require a matter of minutes to construct the file). As always, feedback is encouraged. Cheers, Lee

2 3

Goals and scope (Re: Version control systems)
by Greg Newby 26 Feb '12

26 Feb '12

(New subject+thread) On Mon, Jan 30, 2012 at 08:54:18PM -0800, James Adcock wrote: > ... > Again, the direction I hear you guys heading in is that you-all want to > reinvent DP because you-all think you can improve-upon DP. Not saying you > can, not saying you can't. I'm just say that is not *what I heard* Greg > talking about. Thanks. This is just what I was thinking of pointing out (catching up on today's emails...). Below are details on what I was envisioning. The need I'm trying to address is reformatting or editing eBooks, not proofreading them. For starts, let's consider books that get into the PG collection in the usual way (i.e., culminating with a WWer posting them). I think we can do what's below while leaving the existing workflow out of scope. (It's not holy or untouchable or anything, just out of scope for what follows.) What I'd like is (as someone else nicely put it) a continual improvement opportunity, provided to essentially anyone, for eBooks in teh PG collection. This boils down to a handful of critical activities. It's mainly the third one (III) that involves crowdsourcing and new tools. I. making changes to the master file(s) [let's imagine that we retain the practice of every PG eBook having a small number of master files, in a small number of master formats]. The short list of master formats includes RST, HTML, TeX/TEI, and plain text (perhaps with light markup). Maybe this list will grow in the future; maybe it will shrink. The main feature here is that typos or fixes or additional master formats can be contributed. Challenges have been noted (revision wars; concurrent editing; bogus fixes; spam/inappropriate additions; inconsistent files...) II. from those master files, various other file formats can be [and are, currently] derived automatically. These include EPUB, Kindle variants, variations on HTML or text (especially if they were not previously provided), RTF, and a few others. Again, maybe this list will grow, maybe it will shrink. I do hope to offer conversion on-demand, which will let people select conversion options, and maybe even different conversion programs, for their purposes. The main features here mostly exist, but not as flexibly as I'd like to see. For example, applying a variant CSS. Or making a PDF with a specific font and paper size. Many challenges are technical, such as increased sophistication in dealing with text and HTML as master formats. Others need to be addressed by policy or social means, such as the ongoing tendency to use HTML for layout that is difficult to automatically convert. These, and others, have also been discussed deeply. III. from those master files, various other file formats that are created/contributed by individuals. I get offered these (via help@) practically every day. Usually EPUB, but also RTF/DOC, PDF. Often with typos applied. These are what I called "lovingly prepared," though of course some are better than others. These can be better than automatically-generated versions in various ways. They might have advantages over master files (for example, improved HTML). The main feature is that these would, in many cases, provide an improved reading experience (at least for some people, on some devices). If we accept that anyone could contribute such a new file (or set of files) for an existing PG eBook, then the main challenges I see are (a) how to help readers select among them, and (b) dealing with the fact that, over time, master formats will be fixed, but not these hand-crafted derivatives. I believe the solutions are related, and fairly easy. For (a), we need a community recommender system. Stars, batting average, +1, new/novell, etc. And, "dislike," "report a problem," "report abuse," etc. For (b), "time is on our side" (yes, it is). For a derivative format, we simply need to note when a master format was updated, but hand-derived ones were not. Plus, perhaps, a metric of how different the master format is from when the hand-derived one was created. Combined with (a), a recommendation would be attenuated based on such a metric. So, for example, a hand-derived file that gets a 90% quality rating from readers would slowly lose quality points as the master format is increasingly different. You get the idea... details TBD. Soooooooooooooooooo.... my main starting point was to ask about existing software for group editing. Version control systems seem a reasonable fit for this. Plus, sophisticated systems like TRAC also take care of managing users and their passwords. We do need to think of all the other stuff, but the basic idea was crowdsourcing for eBook formats/conversions/presentations. >From that standpoint, I've left many other (important!) things out of scope: proofreading (distributed or otherwise), tools for auto-conversion (we have some; others will plug-in but will be developed separately), and even arguments about master format (we have some, and better exposure of conversion tools and options will create a positive feedback loop to guide contributors to do a better job. And/or, open capabilities for OTHERS to fix the masters). Cataloging, metadata and supplemental info (such as author bios). Note especially that I don't envision developing tools to help potential contributors do the conversion. Not our bailiwick: there are entire ecosystems of tools already, and all we need to do is support the community of interested donors of resulting files (including pointing them at recommended tools). The PG toolchain for automated conversion should remain available (i.e., http://epubmaker.pglaf.org) I hope this helps clarify my original suggestion a little better. There has been some great discussion on this and related topics. -- Greg Dr. Gregory B. Newby Chief Executive and Director Project Gutenberg Literary Archive Foundation www.gutenberg.org A 501(c)(3) not-for-profit organization with EIN 64-6221541 gbnewby(a)pglaf.org

14 128

Producing epub ready HTML
by hmonroe1＠verizon.net 25 Feb '12

25 Feb '12

19 130

how to improve the p.g. book files
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

don said: > OK, I'm willing to listen to how I can improve any > "book files" (whatever they are) PG provides today. are you kidding? have you ever even _looked_ at any of the output -- .mobi or .epub -- that is being generated by p.g.? don't you feel any need to inform yourself before you speak? > What great secret do you share with them > that the rest of us aren't privy to? um, it's no "great secret" what the obvious flaws are, don... because -- d'uh -- the obvious flaws are terribly _obvious_. but, since you don't seem to be able to _see_ the obvious, i'll give you a few examples to sharpen your observation... first, the e-books start with legalese. you know that, right? and maybe you're accustomed to it, and you've "accepted" it, but if you want an improved product, move it _all_ to the end. what usually comes next is the title-page gunk of the p-book. that's typically too much information, too fast, and undesired. you should open with the equivalent of the cover of a p-book; that is, the title (and subtitle, if any) and the author (and the co-creators, if there were any), contact info, creation date of this latest version, and the e-book's canonical online address. that's all! all of this info should present itself compactly, on one screen, even if the end-user has their fontsize cranked up fairly high... p.g. e-texts _often_ get this "cover-page" screen badly wrong. *** next should come the table-of-contents. _without_question_. people should know -- with certainty -- where the t.o.c. is, and it should be in a convenient location, so let's make it standard. nor should this be one of those sprawling 10-screens things... indeed, _this_ table of contents should fit _on_a_single_screen_. even if, once again, the person sets their font-size fairly large... if you've got one of those sprawling table-of-contents monsters, make that the _next_ section, and call it "the extended contents". but the second screen should be a compact overview of the book, and a quick way to jump even to the sections at the very end of it. another thing which you can consider is to split up the monster. let's say the book has 5 sections, with 10 chapters in each one... the table-of-contents in the front could just list the _sections_. then, at the start of each section, you have another linked list, which gives the person all the chapter names for that section, plus links to the previous and next section-headers. this allows the reader to gracefully page through the sections of the book, getting a good idea of the chapters contained within each one... this is far preferable to a sprawling table-of-contents up front, where the screen-breaks will never coincide with each section... there are other ways you can make this work, too. for instance, p.g. has a book along the lines of "365 recipes"... yep, there's a recipe for every day of the year. pretty nifty, eh? so in that book, i turned each month into its own section, with the appropriate number of entries for the days of that month... depending on the book, there might be other ways that you can separate the table of contents into chunks that are meaningful. again, the idea is to make something that works as an _e-book_. that's the ideal, anyway. but, getting back to our reality... project gutenberg derivative formats often botch the t.o.c. badly. in many books, the items get joined together in one big jumble... in the worst cases, not al that atypical, the links don't even work. this sets a bad impression even before the person starts reading, not to mention the worse fact that it hurts the t.o.c. functionality. on the one hand, i know that the problem largely revolves around the fact that it's difficult to anticipate all of the t.o.c. snowflakes... so i'm tempted, for half a second, to excuse marcello's poor work. on the other hand, it shouldn't be that difficult to detect the t.o.c., and do whatever work is necessary to make it work in a useful way. (sorry, marcello, i tried to be nice, because i know you're a big fan.) *** a list of illustrations might come next, if the book has any... even if there wasn't one in the p-book, there should be one in the e-book. there's no law against improving the product. *** after that, you can put the obtuse crap from the title-and-verso. but... as far as most front-matter goes, it's not a bad idea to _move_it_ to the end of the book. an introduction should stay in the front, obviously, and a dedication doesn't take much effort to skip over, but a lot of the other stuff that's often presented as front-matter could -- quite easily -- be read at the end, and be just as useful. if you _do_ leave this stuff up front, though, you absolutely must give the person a link to skip it, and jump right into the book... do p.g. derivative formats do any of this? no, sadly, they don't. so these are ways that those formats could easily be improved. *** this is just to get you started, don. if you need more input, on matters involving the interior, i can give that to you as well. but you really should look for yourself. -bowerbird p.s. oh, and i know that a _lot_ of you other people are just as uninformed as don is, but -- unlike don -- you're too stupid to even ask any questions to get the knowledge you are lacking...

4 5

Producing epub ready HTML
by Robert Gibbins 01 Feb '12

01 Feb '12

Almost exactly a year ago I put an append on this forum about what was wrong with PG-generated epubs, offering my own hand-crafted ones to replace them. The response at the time could probably be summarised as "yes, we know about that, go away" Over the last year I've probably re-mastered (for my own consumption) about 50 of the PG-generated epubs, many of them Anthony Trollope's novels produced by Joseph Loewenstein and others. Mostly what I am doing to the actual works is, as Greg Newby described in a recent append, removing stuff to do with formatting/layout and instead tagging with structural information that relates it to layout specified in a css file. I also have standard templates for the PG front and back matter wrappers and the xml files that the epub v2 standard requires. The css file I use is not particularly complicated, it contains remarkably few styles, those that are there being largely derived from existing PG stylesheets. I make no claims for the excellence of my markup or stylesheet, I'm still learning, no doubt you chaps could do better, but I achieve results that are satisfactory to me, and without using any more complicated tools than emacs and winzip. My main point is that to do this sort of thing you don't need to be an ace compiler writer or rocket scientist. A year later I look again at this forum and you are still arguing about it, but nothing seems to have been achieved! Some of you want to produce a grandiloquent change management/source code control systems, that gives fine-grained cooperative shared access to pages in the work. (Personally I found that the only significant productivity improvement I got was to take the generated epub and stick all the individual xhtml files back together so I could do emacs global changes on them (regular expressions are wonderful!)and then split them back up into individual chapter files afterwards.) Greg Newby seems to want to kind of shunt everyone's epub contributions into a siding where they will wither and die from not being updated. Nothing is sacrosanct except no one else is allowed to touch David Price or David Widger's work. It seems to me that really all that is required is to recognise that a lot of the html in PG's archive is pretty antiquated by modern standards and needs to be updated. And basically this is a labour-intensive task which can't be completely automated because it wasn't originally done to consistent standards, because it happened over a decade or two during which html and ideas about how to use it evolved. So really what needs to be done is to call for volunteers to do the clean up. And since Greg seems to imply that he is currently deluged with offers of documents from people who think they have already done this, it seems to me that there is plenty of enthusiasm out there (including me!). And before that happens, some PG group (contributors to this forum, Wwers, whatever) needs to get together and write a 'How To' describing at the nuts and bolts level how to do the clean up. This should include a recommended set of styles (and the css that implements them) lots of advice about what not to do for people that don't already know, also including access to templates, css style sheets, etc. As my ex-employer used to say, 'there are no technical answers to management problems, but sometimes there are management answers to technical problems'. Bob Gibbins

5 7

i don't know -- third base!
by Bowerbird＠aol.com 01 Feb '12

01 Feb '12

greg said: > I'm entirely in favor of > automatic conversion > to & from different formats. > Of master formats. the problem is you don't _have_ a master format... except for maybe r.s.t., and its uptake is very slim. and you don't have any adequate converters either. your .rst-to-.html converter makes people unhappy. your .rst-to-.epub converter makes people unhappy. your .rst-to-.mobi converter makes people unhappy. your .html-to-.mobi converter makes people unhappy. your .html-to-.epub converter makes people unhappy. your .txt-to-.mobi converter makes people unhappy. your .txt-to-.epub converter makes people unhappy. your .txt-to-.html converter isn't as bad as the others, but it's still not anything near to what it _could_ be... > I was personally deeply involved in a whole > lot of our current setup for those things that's nothing to brag about, as clearly indicated above. > and am hugely optimistic about > improvements in the future. ok, there's something we can all agree would be nice... until you have a master-format, however, i'd think that your "optimism" might be a tad bit unfounded. but, hey, i'll be more than happy to admit it if you prove me wrong. the point is this: yes, your converters definitely need work. but the problem lies much deeper, i.e., in your input-files... until you have some consistency in your input-files, you will constantly be battling against nature's ability to make fools. and yes, greg, yes, yes, a thousand times yes, this is _exactly_ the pro-consistency argument which i've been making forever. and now you see why. and my constant prediction that you would have to face this very demon, sooner or later, has now also been vindicated... > Meanwhile, we have an immediate and useful way > to improve many reader's experiences, by > providing access to hand-crafted alternatives. except you're just setting yourself up for future complications. _inconsistency_ is the problem. you can't solve it with "more". > It should be obvious that I'm constantly seeking > the long-term view of Project Gutenberg. it's not obvious, because your "solutions" cause more problems. that's not a way to take "a long-term view". if you said instead that you were "seeking a short-term, temporary fix", then yeah, i might buy what you're selling. but even then, it is a bad idea. > That is why I'm thinking of solutions that > involve keeping all derived files or changes > we create/publish/distribute, forever. keeping crap around forever is not a long-term solution either. in fact, it's the exact opposite: a big albatross around your neck. > But that doesn't mean I'm against providing > some files that will only have a few days, months or years > of useful life before they are superseded. except you don't have a mechanism for deciding to supersede. and once those files are out in the wild, you can't "expire" them. the more you talk, the worse this idea sounds. especially since "the problem" which you're attempting to solve has _already_ been solved. we know, for the most part, how to mark up .html that will give us a good .mobi and a good .epub. (at least once we give up the ghost that .html can be a "master", such that the exact same .html gives good .mobi _and_ .epub.) the hard part, then, is getting d.p. postprocessors to _use_ that. they've each developed their snowflake workflow, and are loathe to give it up. especially since -- and this is the _hardest_ part -- they refuse to settle for a "dumbed-down" .html file that doesn't "look like" the paper-book. but... if you can give 'em "the look", they _will_ standardize their .html, for the greater good, because they actually _do_ want the work to become useful in more places. and, of course, if you would have listened to me _six_years_ago_, when i was telling you the same thing, project gutenberg could've been the role-model for google, and internet archive, and amazon. but you were off lost in your xml/tei haze of confusion instead... and even to this very day, you are _still_ overcomplicating things! -bowerbird

2 1

Re: [gutvol-d] Producing epub ready HTML
by Lee Passey 31 Jan '12

31 Jan '12

On Tue, January 31, 2012 2:52 pm, Greg Newby wrote: > On Tue, Jan 31, 2012 at 12:53:06PM -0700, Lee Passey wrote: >> On Tue, January 31, 2012 12:48 pm, Robert Gibbins wrote: >> >> [snip] >> >> > And before that happens, some PG group (contributors to this forum, Wwers, whatever) needs to get together and write a 'How To' describing at the >> nuts >> > and bolts level how to do the clean up. This should include a recommended set of styles (and the css that implements them) lots of advice about what not to do for people that don't already know, also including access to templates, css style sheets, etc. >> >> I, for one, would like to see the style sheets you have created. Mr. Newby, can you provide us with some sort of repository where we can share this kind of stuff? > > I already did, and you're using it: > http://lists.pglaf.org/mailman/private/gutvol-d/ Well, this isn't very persistent, but if it's the best you can do... Robert, if you can zip up your style sheets and attache them to a message, I'll try to post them on my web site so they are available for discussion. I'll post my own reformulation of the PG CSS there as well.

1 0

Re: [gutvol-d] it's monday, after noon, let's get the crowdsourcing started
by Bowerbird＠aol.com 31 Jan '12

31 Jan '12

al said: > My suggestion, start with etext03 (etexts 3601-4800). that's a consideration. let's see what greg says... and what the other teams in the tournament say. looks like so far it's: > 1. bowerbird. > 2. marcello. > 3. lee with dkretz second in line right behind him. > 4. jim "j-mad" adcock. > 5. john "limpid" redmond. > 6. keith. > 7. karl eichwalder. > 8. hunter "don't forget guiguts" monroe. > 9. carlo "garweyne" traverso, and his students. no offense intended if i left anyone out, just let us know. and if you're listed, but don't want to play, please say so. 3601-4800 would be 1200 books... if we want to cut that number down -- at least for the opening round -- i suggest 4321-4567 -- about 250... that's enough to weed non-contenders. it strikes me that, with e-text numbers this low, this'll be a clean-up job rather than an exercise in how to generate better derivative file-formats, which is what i think greg was aiming at, but i'm ready to go with the consensus of the tournament. al, you'll need to point us to _canonical_versions_ of these books -- preferably at internet archive, or someplace where you can furnish us with scans -- so we're all working toward the same end-product. -bowerbird p.s. oh, and all you other teams, please take all of your preparation plans elsewhere; we don't need to hear 'em. the proof will be in the pudding -- where it _always_ is... unless, that is, you want to amuse us with your fumbling.

3 2

PG governing body
by Karen Lofstrom 31 Jan '12

31 Jan '12

On Tue, Jan 31, 2012 at 11:32 AM, Greg Newby <gbnewby(a)pglaf.org> wrote: > It should be obvious that I'm constantly seeking the long-term view of Project Gutenberg. That is why I'm thinking of solutions that involve keeping all derived files or changes we create/publish/distribute, forever. But that doesn't mean I'm against providing some files that will only have a few days, months or years of useful life before they are superseded. I'm starting a new discussion here. I hope that I can present this topic in such a way as to avoid an immediate descent into acrimony and personal attacks. I have the impression that when Michael Hart was alive, he *owned* PG. It was his baby, he had kept it going for years, he owned the name, etc. There was a board of directors, but it was not elected by PG volunteers. Greg Newby was MH's deputy, and over the years, took on much of the workload. Greg has served selflessly, silently, devotedly. (Correct me if I'm wrong in my impressions, please.) However, I note with some disquiet that Greg seems to have stepped into MH's shoes, as the man who runs PG and gets to make all the decisions. See the quote above. I am not comfortable with this as anything more than an interim measure, until MH's estate is settled and the copyrighted name problem is sorted out. I hope that PG can make a transition from a monarchy to an elected board, just as DP did. I am not sure what form this board should take, and how elections might be run. Who counts as a volunteer with voting rights? That was easier with DP, which is more organized. It might also be a good idea to have ex-officio members from other digital preservation organizations, such as Internet Archive and Hathi Trust. Perhaps a certain number of seats could be set aside for such organizations, and the choice of organizations NOT be set into stone. Old groups may falter and new ones form. I do hope that Greg will see this not as a personal attack against HIM, but a move towards an organization, and a library, that is not dependent on any one person's efforts. I do not see this as deposing Greg, who will continue to be essential. I just would like to see him as the CEO, not the king. With some trepidation ... -- Karen Lofstrom

1 0

who's on first
by Bowerbird＠aol.com 31 Jan '12

31 Jan '12

greg said: > typos or fixes or additional master formats > can be contributed you want people to fix typos without reference to a scan? please tell me that you didn't just suggest that seriously... and how can a master format be made with any certainty without full reference to a canonical version of the book? > The main features here mostly exist, > but not as flexibly as I'd like to see. where do any of these "main features" of customization exist -- not just "mostly" -- but in any form whatsoever? > These, and others, have also been discussed deeply. don't be ridiculous, greg. these issues have "been discussed" repeatedly, but they have never been discussed "deeply" at all. they have never come to anything actionable, usually because people here drag these threads all over the back-40 acres, and then drag in dead field-mice from the neighboring homestead and plop them smack-dab into the middle of every discussion. surely most of this crap would earn a "c" grade in your courses, and even _that_ assessment is probably being far too generous. the discussions weren't even "shallow"... perhaps "superficial". (but even that implies they were on-topic. but no, usually not.) > These can be better than automatically-generated versions oh please. haven't you learned anything from the "snowflakes"? hand-crafted versions are simply impossible to keep up-to-date. impossible. totally and completely. and now you want to _invite_ users to submit as many snowflakes as we possibly can? insanity. are you really in charge of this project? if so, i fear for its future... -bowerbird

4 4