October 2012 - gutvol-d - lists.pglaf.org

Re: [gutvol-d] Basic simple test case.
by don kretz 19 Oct '12

19 Oct '12

We need to be clearer about which stage(s) of the process we intend when we describe a given state and style of markup. He might have been referring to the text in a prepublication state before a css for any device or format has been applied. From: James Adcock Sent: 10/19/2012 5:18 AM To: lee(a)passkeysoft.com; Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] Basic simple test case. >The associated CSS file specifies the font to use (cursive), which you are apparently unhappy with. Just download the file, and change your standard gutenberg.css file to use the font that /you/ are most comfortable with. Or don't use a CSS file at all, and get a less appealing version, which is still superior to that provide by Project Gutenberg. (You can do this online with IE and Firefox by selecting "no styles" in your browser options). This is stupid. The job of the typesetter is to make intelligent typographical decisions -- including understanding how the limitations of the media he is targeting is going to affect those choices. The idea that the end customer is going to edit your file simply doesn't work. If you want to get people to buy into your view of how the world ought to be coded -- at least make something that looks half attractive and is readable. _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

1 0

have a little respect for the dead
by Bowerbird＠aol.com 19 Oct '12

19 Oct '12

most of you people didn't care even a little bit what michael hart thought when he was _alive_. yet you're arguing which one of you represents the true intent of his aim with this project now that he is dead and gone to greener pastures? yeah, right, like i really believe you now. have some respect for the dead, by leaving him _out_ of your never-ending balls of confusion... michael was wrong on more than one occasion, but he _never_ let himself get mired in a tar-pit of confusion like the messes y'all create here... the man sprinkled sugar on his pizza for energy, for crying out loud, and much more likely to err on the side of certainty than confused catatonia. which, if you haven't noticed, is what ails this list. -bowerbird

2 1

1. Choose a book to digitize
by don kretz 19 Oct '12

19 Oct '12

First of a series of notes I'm assembling in the blog to build from for a discussion of project 10X. Currently the identification of books to digitize and convert to ebooks is entirely up to the public. And it's not apparent that people are beating down the doors with titles they demand be published. The two publicly accessible DP sites often plead for new material for the beginning rounds. We see that even the production we do get is frequently a new version of an existing ebook. Yet we know there are major, well-known titles that are not available. My own projects, the volumes of Encyclopedia Britannica edition (the most renowned and most recently public-domain-available,) is not online in any quality digital form. There is no digital copy of Newton's Principia Mathematica in the English language. No copy of Ptolemy's Almagest. One wonders how many historically significant books are being irretrievably lost in the destruction and violence in Syria and Egypt and Libya, literarily among the most historically active areas in the world. So it may not be the most critical step in the process, but it will need to change and grow to support 10X, if the other steps progress. Even at the current rate, it would help to have some easily accessible set of lists of culturally valuable works, in all languages and traditions for people to use to search for sources, and a place to just post the images and/or urls if they find them. We now have a number of productive harvest sources for both images and crappy text digitization, but I've not seen a list of potential PG candidates, or a checklist of ebooks already in the existing catalog to guide anyone wanting to cull. (In fact, is there a checklist of existing titles in PG that's exhaustive and accessible? The rdf files don't qualify for casual use. Nearly every page of Encyclopedia Britannica is thick with both bibliographies, and with articles about authors and texts that are largely undigitized. I imagine many of them are already lost. How many other ebooks in the catalog would be similarly rich in authors and titles. As DP-Europe brought to our attention, there are entire cultures and languages whose written records are fast disappearing. We often take the identification and selection of books to be something we can assume takes care of itself; bu there's good reason to think curation in this area is important and in fact requires more attention if PG is to grow. Some will argue that we shouldn't be concerned about new texts while the existing catalog is in questionable condition. But that assumes that one needs to choose between the two; and that we will continue to suffer for the lack of a decent process for incremental improvement - something we'll need to discuss in further installments. But the most compelling counter-argument is PG's original mission statement - more free books for more people - combined with the known fact that books are disappearing from our reach daily. Some will argue that we shouldn't be concerned about new texts while the existing catalog is in questionable state. But that assumes that one needs to choose between the two; and that we will continue to suffer for the lack of a decent process for incremental improvement - something we'll need to discuss in further installments. But the most compelling counter-argument is PG's original mission statement - more free books for more people - combined with the known fact that books are disappearing from our reach daily.

7 13

huck finn has ben kickin my ass
by Bowerbird＠aol.com 18 Oct '12

18 Oct '12

it's true. huck finn has been kicking my ass. totally. all of those slanguage spellings screwed up my workflow, mostly because i didn't have the discipline to be rigorous about adhering to the correct order while doing the tasks. (in the past, violations of that ordering had few ill effects; but they were serious here due to those spelling variants.) at any rate, i have finally wrestled from under the bottom, and i'll soon be kicking huck's ass instead of vice-versa... there's a new version up, but i'll have a better one soon; > http://zenmagiclove.com/huckf/huckf-006.zml -bowerbird p.s. also, it's jarring to repeatedly encounter the n-word. i've considered changing it to "nogger" or "logger" while i work with the book, just to make a little humor out of it. after all this, i wish i woulda done it right at the outset...

1 0

Re: [gutvol-d] 1. Choose a book to digitize
by don kretz 18 Oct '12

18 Oct '12

The evident PG theme is More Free Books for More People. To that end, the following are good: More people involved. New titles. More publicity. More ways to digitize a book. More lists of work to do (undigitized books). More distribution channels. Simpler and fewer rules for contributing. More forms of books to use as input and free output. More access to free text sources at any level of completion or elegance. More ways to enable more people to improve existing free book source texts more easily. More books is more important than better books (yes, quantity over quality.) Readers' rights over copyrights. Lower barriers and more media to access for reading. Lower barriers and more ways to improve existing book texts at any time, place, or condition. More smaller simpler units of work offered to more volunteers with fewer restrictions. Fewer "Yes buts" from the gatekeepers. Fewer gatekeepers. That will do for a start. As for technique and format, start with what empowers readers and contributors, not what empowers computers and programmers who are only there for readers and contributors. From: jeroen(a)bohol.ph Sent: 10/17/2012 1:07 AM To: gutvol-d(a)lists.pglaf.org Subject: Re: [gutvol-d] 1. Choose a book to digitize The mission of PG is accessibility. Getting books to people, so they can actually read them. This is not exactly the same as preservation -- but touches to it in some ways. Yes, you can preserve a book in a sense by scanning it to very high standards, and then keeping those scans a few servers for patrons to use, and then lock away the original book in an ideally conditioned environment, taking wear-and-tear out of the picture for all but the most essential physical manipulations. But I think true preservation is more than keeping the artifacts for eternity, this is about keeping the expressions available and accessible, and part of our culture for as long as they deserve it ("deserve" being decided by the members of the public, not some artificial gatekeepers). For this, works need to be accessible.... Curiously, not our (lack of) standards, not even our limited capacity, but the laws of Copyright, cynically advertised as "promoting science and arts" are the biggest barrier to making works inaccessible by needlessly preventing even the slightest steps required to keep works accessible until they have become completely culturally irrelevant. Of course changing that will take some time, and we do have a huge backlog to work on until that change has been materialized. Turning then to the question of a master format. It didn't took me long back in 1996 on discovering TEI to be converted. This is exactly what we needed to get going with a master format. Yes, it takes time to learn, and yes, we do not yet have a fool-proof toolchain for it, but it will survive in the long run because it is reasonably well defined, and still flexible enough to deal with the large number of details we come across in real books. Still, TEI is not a stand-still target, and since its start has gone through several incarnations, and the whole customization stuff included with it make it hard as well (but compare that with the Unicode standard, few people will ever require the use of Cherokee or Deseret letters, but still they are in there, just in case). And yes, most tool-chains will require conventions to work well, but that can be solved... I don't see the point in defining yet another master format to get to something simpler than TEI. TEI has done the groundwork and heavy lifting, and works well, and while I agree that you need to make things simple, you can't really make them simpler than needed. Converting a simple work into TEI (a run of the mill novel) starting from reasonable clean HTML takes an hour or two; starting from Word doubles that, and plain text requires a walk-through of all pages (or scans) to add missing layout information. When I work on a text for PG for which a on-line source is already in existence, I often normalize both the OCR results and the alternative source towards TEI, and then run a compare to find issues in both. (As I have done with the "Old Frisian" text in the Oera Linda book recently posted). What I would understand is to have a more WYSIWYG interface on Distributed Proofreaders, that is, have people use a Wikipedia like mark-up, and then render that page as HTML internally, which will help to find more issues in the content. Jeroen. Quoting Lee Passey <lee(a)novomail.net>: > On Fri, October 12, 2012 4:47 pm, don kretz wrote: > >> One wonders how many historically significant books are being >> irretrievably lost in the destruction and violence in Syria and Egypt >> and Libya, literarily [sic] among the most historically active areas >> in the world. > > I do not believe that it is now, or ever has been, the mission of > Project Gutenberg to preserve world literature. According to the web > site, the mission of Project Gutenberg (written in Michael Hart's > inimitable style) is: > > "To encourage the creation and distribution of eBooks." > > There is nothing in the explanatory text surrounding this mission > statement that suggests that preservation plays any part in PG's > mission, although to be fair there is much in Mr. Hart's statement > that is at odds with the current practices of Project Gutenberg. > > It seems to me that the mission of Project Gutenberg has nothing to > do with the /preservation/ of literary works and everything to do > with the /popularization/ and /accessibility/ of those works. And > while Mr. Hart never said, "we encourage our volunteers to furnish > us with as many rare texts as they can," he did say, "[W]e are happy > to bring eBooks to our readers in as many formats as our volunteers > wish to make.... [P]eople are still encouraged to send us eBooks in > any format and at any accuracy level and we will ask for volunteers > to convert them to other formats, and to incrementally correct > errors as times goes on." > > I have come to believe that when Mr. Hart started Project Gutenberg > on the donated mainframe time he understood the potential of storing > mass amounts of text on computers, but he did not understand the > transformative power of computing. He understood the power of the > hard drive, but not the power of the CPU. Thus, when he first > started placing text into storage instead of using a rich format > that could be transformed into the Format Of Any Day, he chose to > carefully, manually transform each text directly into the Format Of > His Day, which in that day was 80 character lines, ASCII-only text, > suitable for use on the VT52 terminal. > > Over time, the Format Of The Day has changed, but given the > difficulty of up-converting VT52 format to more modern formats, and > the fact that most modern operating systems can still display VT52 > text files, however badly, most PG texts have remained in their > original, sorry state. This state of affairs has persisted for so > long that most of the PG old-timers see it as being not only normal, > but desirable. > > Most of the complaints now leveled at PG are not that the archive is > too incomplete, but that the contents of the archive are so visually > unappealing as to be unusable. Thus, the true mission of Project > Gutenberg, "[t]o encourage the creation and distribution of eBooks," > is now no longer being satisfied. > > So, the first advice /I/ would give to someone wanting to volunteer > at Project Gutenberg is to start by learning how to create an > electronic book from an existing file (a tutorial to this effect > should be created). Then s/he should practice what s/he has learned > by taking an existing PG file that s/he is interested in, and which > sucks, and make it suck less. The text can then be returned to PG > for the kind of incremental update that Mr. Hard envisioned. > > This kind of approach provides a gentle introduction to the creation > of e-texts. You can take an existing text, see how someone else has > done it, see where the mistakes are, fix a few simple mistakes, > check it in to PG, see what kind of feedback you get, fix more > mistakes, take on a more challenging text, and so on until you're > comfortable with markup. Then, go get some OCR'ed text from IA or > Google, fix that up and check that in. When you finally get around > to doing OCR yourself, you have all the underlying knowledge to fix > up your own OCR. > > Stay in the shallow end until you learn how to tread water, and do > not use the high dive until you are an expert swimmer. > > > _______________________________________________ > gutvol-d mailing list > gutvol-d(a)lists.pglaf.org > http://lists.pglaf.org/mailman/listinfo/gutvol-d > _______________________________________________ gutvol-d mailing list gutvol-d(a)lists.pglaf.org http://lists.pglaf.org/mailman/listinfo/gutvol-d

2 1

Re: [gutvol-d] Basic simple test case.
by don kretz 18 Oct '12

18 Oct '12

Why? To discuss as an illustration of proper handling of the issues you list, and others; and to see what alternative markup schemes look like in action. Sent from my Phone ------------------------------ From: James Adcock Sent: 10/10/2012 4:53 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] Basic simple test case. Re 14668: Well, the first question would be: Why? Contrary to the idea that PG needs to scale up efforts 10X and “do everything” maybe the right answer is to scale DOWN things by 10X and fix the books that people actually want to read, but which are currently hopelessly gone moldy, rather than offer more kiddie readers? Secondly, one needs to get page scans, which are at least available from Google in a variety of editions, you’d have to pick one. In terms of the current “automagic” HTML conversion from txt, this txt shows the problem that PG isn’t even currently “correctly” specifying that similar <p> formatting be used on each device. Seems given the PG txt conventions, PG should be specifying “no indent, 1em of white space between paragraphs” for the <p> styling – so at least the basics match the txt styling. This is important because txt “formatters” implicitly are using the txt formatting rules as an element of the formatting – i.e. syntax vs. semantics **cannot** be uniquely determined automagically by examining a PG txt file, so the best one can hope to do is to emulate the PG txt layout. In terms of hand-recoding the html/epub/mobi there appears to be no great problems other than understanding and dealing with the issue of merged/rounded top/bottom margins or not, which can be dealt with in the standard manner of using top margins only. In terms of design issues, there appears to be minor issues of poetry – not hard since the poetry lines are short. (how to “correctly” autowrap lines of poetry remains problematic in html since html doesn’t support poetry) There are issues of quasi-table listings of words, where the traditional solution is simply to linearize the lists. IE these word lists were “packed” on paper to save paper, but on ebook devices vertical landscape is “free” [horizontal landscape however definitely **is not**] so the word lists can simply be “unpacked.” And there appears to be a minor issue of plain rules vs. decorative rules. * * But all this would still beg the first question: Why? Who is the customer? What parent would want this for their kid today? Seriously? Is some researcher interested in this for historical reasons? Well – frankly they would be better off examining the bitmap scans. Fundamentally, one can’t code anything reasonable unless you decide who the customer is, and how they are going to actually be using your efforts.

5 16

meanwhile, elsewhere
by Bowerbird＠aol.com 17 Oct '12

17 Oct '12

james said: > I'm looking for this program https://www.dropbox.com/sh/ovgn8muzrn5nsku/soU0qln53Z -bowerbird

1 0

meanwhile, elsewhere
by Bowerbird＠aol.com 17 Oct '12

17 Oct '12

meanwhile, elsewhere... *** there's someone over at d.p. -- "tallforasmurf" -- who's programmed a post-processing application. it's called "ppqt". i wasn't able to get it to load up my page-scans, so i couldn't really give it a proper workout, but a cursory glance said it looks quite impressive... roger frank is doing a book with it, so maybe he will be kind enough to post his review of it here. it's built in python, meaning that it _might_ be relatively easy to make it a web-app as well, and the web-or-offline-whatever-you-feel-like-now mode _is_ the right way to fashion this toolset... so that's good news... *** and because it hardly ever gets mentioned here, let alone as often as it _should_ be, especially if we are talking about "p.g. times 10", check out: > http://en.wikisource.org their m.o. is progressive enhancement of _lots_ of books, which means they rarely get to certify _any_ of them as "finally finished", but that _is_ the wiki way; given that model, wikisource rocks. *** haveaniceweekend... :+) -bowerbird

3 2

"i had no idea"
by Bowerbird＠aol.com 17 Oct '12

17 Oct '12

james said: > I had no idea the process > would be as difficult > as it turned out to be. did you do any research about it first? where did your expectations come from? and since i know some of your false-starts, james, i think it's absolutely clear that project gutenberg needs to do a much better job of explaining this... sending people to d.p. is an _awful_ strategy, since the workflow there works so badly for independents. and, as i have shown, i developed a good workflow over 5 years ago, and documented it exhaustively... who else here -- besides james, in his book on it -- has done _anything_ to also describe and document? remember, putting "describe and document workflow" on a checklist _does_not_count_ as satisfying the task. unless, of course, your only task is to make a checklist. -bowerbird

2 1

(no subject)
by don kretz 17 Oct '12

17 Oct '12

For background to address Item 2 of the PG process, I went through a little exercise to put myself in the place of a potential volunteer - someone who would like to contribute a book. What would the experience be like? For a volunteer organization, this kind of person would presumably be worth attracting and trying to recruit. This post describes what I found. Imagine a person, Polly, who is new to Project Gutenberg, and she is only slightly acquainted with Distributed Proofreaders – tried it once, but found it daunting and left (as do most other people who try it). She has a book she wants to contribute. If she starts at PG, it’s likely she will begin with www.gutenberg.org. The first link on the page that looks like it might help is > > No fee or registration is required, but if you find Project Gutenberg > useful, we kindly ask you to donate a small amount<http://eb.readingroo.ms/wiki/Gutenberg:Project_Gutenberg_Needs_Your_Donation>so we can buy and digitize more books. Other ways to help include digitizing > more books <http://www.pgdp.net>, recording audio books<http://librivox.org>, > or reporting errors<http://eb.readingroo.ms/wiki/Gutenberg:Contact_Information#Electronic_Mail> > . > (She hopes that the order in which the list itemizes the ways she could help is the order in which they prioritize here help.) The option for “digitizing more books” looks promising. She clicks it, and it leads straight to Distributed Proofreading’s main page. Polly would like to try a more direct approach to PG before tackling DP again. Besides her personal experience, she knows about the overhead and delays involved at DP and she’d like more control of her text, and she’d like to do a lot of the work herself. So the next possibility presented on the page is the Self-Publishing site. But no, that’s probably not what she wants; it’s not a new book, and she isn’t the author. But there’s no further guidance on the main page. So she explores some of the non-specific links. Next she finds the “How-To” index, with links to further information. One (and only one) appears promising. It says “Gutenberg:Public Domain eBook Submission How-To”. She tries that next. This page<http://www.gutenberg.org/wiki/Gutenberg:Public_Domain_eBook_Submission_How-…>has a lot of information, but nothing like a checklist, and seems to assume she already knows how the pieces fit together. Primarily, though, it repeatedly refers her back to Distributed Proofreaders. And there are a lot of words and software application names that she doesn’t understand at all. This is beginning to look really complicated and, well, difficult. Intended for people who already know what to do. We could go on and describe what happens when Polly goes next to the DP site, and has similar difficulty finding help (which is mostly in the Wiki, which itself has its challenges.) And substantively, it leads right back to PG for information and procedures that Polly couldn't locate in the PG structure itself. What is clear is that PG doesn't really offer a useful way for potential volunteers to learn how they can contribute. It mostly points to DP. Even to contribute an etext. In terms of order-of-magnitude scalable growth, though, DP, for all its other benefits, is more like the problem than the solution. So new and different procedures need to be considered, not to replace DP, but to give more options to people who don't fit the DP work pattern. Next I'll post some notes I have about some alternatives for handling metadata checklistsand copyright clearance procedures (not necessarily for the PG process, with which I am totally unfamiliar and can't contribute anything, but on the public side of helping ease through to the next step where they have a genuine publishable project.)

3 3