- gutvol-d - lists.pglaf.org

Enlightened Self Interest
by Ron Aitchison 02 Mar '05

02 Mar '05

Understand the issue of editing. My proposal would be to supply an editable file in OpenOffice or MS doc format (BTW if you are not using the Open Source OpenOffice suite I recommend you check it out - the features are great, at least as feature rich as MS word, plus - one button PDF creation, output as doc, text or native XML format and a great price = $0! http://www.openoffice.org ). I propose to take nothing away you will have edit control over the file. This also opens up another question over what base document formats you have standardized for editability and portability e.g. OASIS etc.. Maybe that is a topic another list. Finally I note you have PDF formats available for some other books. Andrew Sly wrote: >One possible problem is that PDF files are not easily editable. > >All of our older texts are being gradually worked through, >corrected, supplied with a new PG header (which puts all the >legal "small print" at the end of the file instead of the >beginning) and REPosted into the currant directory structure. >When this process is done it will make some of the back-end >organization much easier to deal with. > >However, if during this process, we come across a non-editable file >(PDF, Lit, whatever), we cannot update it, and it's generally moved >into an "old" directory, where it is still availible if someone >goes looking for it, but otherwise is not shown in the catalog. > >Andrew > >> Having discovered Jane Austen regrettably late in life I have >> down-loaded a couple of novels and since I find the raw text format >> unpleasant to read I have reformatted for my own use. >> It seems to me since I have the ability to produce PDFs and OpenOffice >> formats and even - heaven forfend - MS doc format should they be wanted, >> it would be churlish not to make such an offer. >> If you can point me at a standard for PDF, page width, font size etc, >> etc., and let me know what formats you do want I would be happy to >> undertake the small additional work for the two novels I have currently >> downloaded. >> I cannot supply DocBook at this time but hope to have that available >> shortly. >> Regards >> >> > > -- Ron Aitchison

10 13

Re: quick question re: lack of date and copyright
by Doug Adams 02 Mar '05

02 Mar '05

>From: David Starner <prosfilaes(a)gmail.com> >If you can match it up with an edition in a library catalog >(Library of Congress or the British Library) are good for >this, it'll probably be clearable. Bonus points if it says it >was printed in the US, since then you only have to >establish printing pre-1989, not pre-1923 (in most cases.) Thanks David! I've found my version in the Library of Congress. The listing says it was published in: Chicago, Belford, Clarke [187-?] So even the LOC doesn't have a date for the book. Now a technical question. How do I submit this to get clearance without the date. Do I need to do it by email to someone. (I've previously used the internet form.)

2 1

quick question re: lack of date and copyright clearance
by Doug Adams 02 Mar '05

02 Mar '05

How do i get a work cleared for copyright if it doesn't have a date on the cover page? I am working on a book that I know for a fact is from the nineteenth century. The publisher, however, neglected to include the date.

3 2

Re: Enlightened Self Interest
by Juhana Sadeharju 02 Mar '05

02 Mar '05

Hello. The master format should be the digitized images of the original book pages. No font, nor footnote, nor math, nor any problems in readability, nor in representing the original text. I find the digitized images more pleasant than any ascii, html, word or TeX text. I don't know the reason but perhaps the art of typesetting and printing was better then than it is now! Any other format can be generated from the digitized images. If some conversion between html and TeX (say) does not go well, one can always check against the original typesetting from the images. So, keep archiving the digitized images!! 200 dpi with 32 grey levels starts looking ok but 300 dpi with 256 levels should be enough even for math texts. Forget 1-bit digitizations completely!!! Best regards, Juhana -- http://music.columbia.edu/mailman/listinfo/linux-graphics-dev for developers of open source graphics software

3 3

Repost: Public Knowledge Orphaned Works Project
by Brandon Galbraith 02 Mar '05

02 Mar '05

Sorry for the repost, just noticed the mailing list scrubbed my forward; Are you an artist, author, musician, or filmmaker? Maybe you're a scholar or librarian? If so, have you ever wanted to use a copyrighted work but been unable to locate the owner to clear the rights? It's a problem that happens all too often, and not only does it affect your work, but it also "orphans" the original owner's work. It's an unfortunate side effect of current copyright law that diminishes everyone's ability to create, innovate, and educate. Fortunately, we have good news: The U.S. Copyright Office wants to make it easier to locate copyright holders, and it's asking for the public's help. Before the Copyright Office can *address* the problem, it needs to gather evidence that there *is* a problem. This is where you come in: tell your story to the Copyright Office. Public Knowledge along with a number of other like-minded organizations have created Ophanworks.org: an easy way for you to submit your story to the Copyright Office. Now is your chance to tell the Office what personal difficulties you've had when trying to clear rights. To get started, go to: http://www.orphanworks.org Never tried to clear rights? Maybe you know someone who has. Forward them this message or visit: http://www.orphanworks.org to send them an email. You can always learn more about the problem of "orphan works" and the U.S. Copyright Office's notice, by visiting Public Knowledge's website: http:/www.publicknowledge.org/issues/ow ====================== Public Knowledge collaborated with the EFF to set up orphanworks.org as a resource for everyone to facilitate public participation in copyright policy. If you'd like to support this and future efforts, please make a contribution: http://publicknowledge.org/donate ====================== Thanks for participating! Your friends at Public Knowledge February 28, 2005 ____________________________

1 0

[Fwd: [Public Knowledge] Trouble Locating Copyright Owners? Tell the Copyright Office Your Story]
by Brandon Galbraith 01 Mar '05

01 Mar '05

I thought this would be of interest to list members =) -brandon

1 0

Re: [gutvol-d] Interesting message on TeBC from NetWorker (about fixing errors, Frankenstein, etc.)
by Bowerbird＠aol.com 01 Mar '05

01 Mar '05

jon said: > But the bigger issue is not constrained to errors (differences) > with respect to the source text used, as you continue to focus on. i think it was you who made "errors" the issue, revolving around the concept of "trustworthiness". if, once that house of cards falls down, you want to turn the issue to one of "which source-text to use", well then i think that michael's "i'm open to all of 'em" stance covers _that_ quite nicely, thank you very much. if you don't like the version of my antonia that's in the library now, add your own! the same goes for all the versions of "frankenstein". casting aspersions on the edition that _is_ there isn't constructive. provide all the meta-data you want on the version that you furnish; heck, you can even put a pointer in to your project at librarycity.org; these days i see a lot of e-texts referencing an .rtf version in france. > the PG version of Frankenstein, > which now exposes PG to legal liability. i don't agree. but if the lawyers to whom "bantam classics" is paying good money decide to send a cease-and-desist, let 'em. going by results obtained by the "gone with the wind" lawyers, the project gutenberg people will probably fold very quickly; without any money, you can't play poker against deep pockets. but hey, i would like to hear the laughter that would resound when bantam's lawyers argued that the way they can _prove_ that this e-text copied their book is because of the _errors_ (map-makers can pull that trick. but book-publishers? ha!) who knows, jon, maybe the project gutenberg lawyers will call _you_ to the stand, to throw your arms in the air and rant about how those terrible mistakes are ruining the fragile public domain, and therefore bantam doesn't _deserve_ the protection of the law. wouldn't that be ironic? :+) > The lack of proper processes, procedures and guidelines well, i don't agree with that either, jon. you might not agree with the procedures, but that doesn't mean there is a "lack" of them. maybe you don't agree with their choice of source-text for frankenstein. but it _was_ good enough for bantam. > is leading to serious questions about the integrity > and trustworthiness of the whole PG library not in my mind. and not in the minds of most people, i don't think. not any more so than with any paper-book i might find in a store. like the "frankenstein" version that was being _sold_ by bantam. > 1) redoing most of the non-DP works using DP, let's find out how many d.p. people want me to go over _their_ work with a fine-tooth comb. go ahead, speak up, i'd _love_ the challenge. > Well, at least you seem to indicate from > your interest in very low error rate OCR > that every etext PG includes in its archive > should be a textually faithful reproduction > of some known source. not necessarily. if someone wants to play editor and combine editions, i don't have any problem with that. in some sense, that's what the public domain is about. i don't see it in black/white terms as something frozen. if you _are_ going to represent something as faithful, i think it should _be_ faithful. but even then, that is _to_the_best_of_your_ability_. as long as you do that, and give your end-users a means of "checking your work", including a solid mechanism for improving it to perfection, then i think you've done your job. so yes, i agree with you, that scans should absolutely be furnished to the end-users, for works that purport to replicate that edition, certainly... however, i understand why they haven't been, up to this point, and so do you -- disk-space just hasn't been affordable enough, even now, if it were not for the largess of ibiblio and brewster, we couldn't even be entertaining the thought of posting the scans. > I doubt this error rate (let's say for even half of the public domain > printings out there) is accomplishable without sentient-level AI. i'm trying to get back off this listserve. i don't like contributing to the discourse in a place where my voice has been muffled before. so let me set up a place where you and i can fight... i mean, discuss... but this doubt of yours is rather easy to dispel, and quickly. you did a pretty good job of scanning that copy of "my antonia". and it looks like you processed (e.g., straightened) the scans well. so now we need to put them through o.c.r., using abbyy finereader; please have that done as follows: save results out to an .rtf file, one for each page; retaining line-breaks and paragraph indentation. do this for 20-50 pages, and zip the output up and e-mail it to me. i will reply to you with feedback on if the o.c.r. was done correctly. then i'll run it through programs that will soon be made available, at no cost, and we'll see what kind of an error-rate we end up with. or, if you prefer, follow this same procedure with some other book. then, if you still want to discuss this matter, we'll do it elsewhere. > But if proofreading is to be done anyway by the public, > as is *now done* by DP, what difference is there between > an OCR error of one every 10 pages, and one every page? when i talk about "the public", i mean _end-users_ who are reading the book for the purpose of reading the book, and _not_ specifically to be "proofreading" it per se. for that type of reader, one error on every page is too many, but one error on every tenth page is not. especially since -- if we give them an easy means of checking for errors and reporting them, and then reward readers for finding them -- errors won't persist for very long, and the e-text will instead progress very quickly on its merry way to a state of perfection. in a practical sense, this means that before you turn an e-text loose for download in an all-in-one file, you make it available _page-by-page_ on the web. anyone who might want to read it has to do so in that form. right alongside the text for each page is the image, so the person can easily check any possible errors. you let 'em know you are asking for their help to find mistakes. if they find one, they fill out a form right on the page, and their input is recorded -- wiki-style -- immediately. later readers can either confirm the error, or question it, or make comments. first person to find each error gets a credit in the final e-text. you also give people a viewer-program that allows them to download the appropriate page-image if they suspect an error -- displaying it right there in the viewer-app next to the text -- and which simplifies the process of reporting it if they find one. (by, for instance, filling out an e-mail they can send with a click.) > The key is that for the aspect of building *trust* in > the final product, it is a very good idea to involve > the volunteer proofreaders to go over the texts, > even if *you don't have to*. what i just described does a good job of doing that. this is the system of "continuous proofreading" i outlined on this listserve a very long time ago. you recently mistakenly credited it to james linden. my offer to develop this system was largely snubbed. for _that_, the project gutenberg "people in charge" rightly deserve to be criticized. for the tiny stuff that you have been complaining about, they do not... > Having (and proving to anyone who asks) at least > two independent people who proofed every page, > adds to its trustworthiness. not nearly as well as putting text and image side-by-side, and allowing any number of "volunteer proofreaders" to examine 'em. you might be surprised by the number of errors that "slip by" the proofreaders through two rounds of eyeballing over at d.p. (indeed, many even slip by the "third round" of post-processing and whitewashing, and sit there big and ugly in the final e-text.) even if a dozen people look at a page, an error might _still_ be there. but with eternal transparency, there is always hope it will be fixed. anyway, jon, i hope you take up the friendly challenge i issued here. and if any d.p. people want to call me on the challenge i made to them, you just let me know. in the meantime, i'll let you get in the last word on this thread, jon, because i _really_ need to be going. use it wisely... ;+) -bowerbird

1 0

Re: [gutvol-d] Interesting message on TeBC from NetWorker (about fixing errors, Frankenstein, etc.)
by Bowerbird＠aol.com 28 Feb '05

28 Feb '05

jon said networker said: > Project Gutenberg e-texts satisfy none of these wishes well, i guess networker will have to start his own project, eh? give him my best wishes! :+) > Conclusions > Of course, the goal of this exercise was not to establish the > provenance of the Project Gutenberg e-text of _Frankenstein_, maybe not. but having done so, it is _refreshing_ to know that -- when that's factored in -- only "a handful" of errors surface. so once again, in spite of some very big noises, it ends up that this fails to stand as a good example of an error-ridden e-text. > nor to discover if there are any errors in the PG e-text, but > to determine if there was an automated method of reducing errors > in newly scanned e-books for which a Project Gutenberg e-text > already exists. I'm afraid the jury is still out on this question. as for this "conclusion", the jury may still be out in _his_ mind, but in mine, the answer is very clear, and i've said it before here: if you do the scanning properly, manipulate those scans correctly, use abbyy in the best way, and subject its results to the right tools, you will reduce the errors in your text to a relatively small number. (the number we've been kickin' around is 1 error for every 10 pages, and at that point, proofreading by the public becomes very viable.) if you then have the rare luxury of evaluating your output against an existing version of the book -- like a project gutenberg e-text -- with the right tool (which networker obviously does not yet have), the comparison between the two, alongside the page-images, should make the process of coming to an error-free version simply a breeze. since this is _exactly_ what will need to be done _increasingly_, as the page-images from the internet archive and (we hope) google -- plus the work done by individual people scanning everywhere -- emerge into cyberspace, that's where my tool-development efforts are now being focused. i suggest networker start reading my blog; it should start being updated on a daily basis starting next week... -bowerbird

2 1

ok, this is my last post for the time being, really
by Bowerbird＠aol.com 28 Feb '05

28 Feb '05

jon said: > Bowerbird, I'll be happy to put up > a ZML regularized text version of My Antonia. if you prepare your browser-based versions carefully, copying text from the browser-window will give z.m.l., so there's no real reason to create a separate version... (of course, if you don't do the correct preparation...) for me, "round-tripping" is one of the biggest priorities. and by that, i mean that when a z.m.l. file is presented, the end-user should be able to copy text out _as_ z.m.l., so that -- with just a few global search-and-replaces -- when reloaded into a z.m.l.-viewer, it will look the same. (even if its text-styling is stripped away, as can happen when it's saved as a plain .txt file, it should be restored. automatically.) i have attained this in the z.m.l. viewer-program already. (this was pretty simple, as i control all the operations.) i've also attained it in my .pdf version, where i'm able to work around the limitations of acrobat's copy operation -- if you've ever copied text from a .pdf, you realize how awful it mangles the formatting -- by controlling what my viewer-program writes to the .pdf in the first place. (to answer the first question of a knowledgeable person, i write a dummy-line as a separator between paragraphs, so a global replace restores the blank line between them.) when i get around to making my zml-to-html converter, i will try to make sure that the .html that's created will copy out of the browser-window correctly too. however, browsers do some funky crap in their copy operations, so it might not be possible to preserve _everything_, at least until the browser-programmers tighten up their act there. when i do that work, i'll share any tips people need to know in order to prepare an .html version to copy out good .zml... but in many cases, even now, a copy out of a browser-window can produce text that is .zml, or can be easily converted to it. for instance, jon, your website that gives your listserve rules creates a nice .zml file. consistent formatting yields good .zml. *** jon said: > Yes, if this is the case, it is mysterious > since IA will gladly host them > once the etext version is out the door. sometimes i wonder if the internet archive is quite as accommodating as you always seem to make them out to be. i don't know otherwise, but it seemed if they were, then d.p. would've put their scans up long ago. (unlike their site, page-scans wouldn't need quick response-time.) pourlean said: > An accessible archive of posted > projects & images is in the works. i look forward to the day it comes online! (if there is any particular stumbling block, do please let me know, as maybe i can help.) *** jon said: > It's on me. In private email send me your address and > I'll burn and mail you a disk of the 600 dpi and 120 dpi scans hey, thanks for the gift, jon, i appreciate it! but i can't use 30 megs worth of scans in my little project; it's just a demo. so i wrote a quick program to grab a few dozen from the site. and now that i've done that, i can grab 'em all, if i ever need; since i was just looking for a way to do it in one fell swoop, i shoulda just done that straightaway, instead of bugging you. but maybe someone else will now be able to make good use of the zipped package of the scans that you added to your site... -bowerbird p.s. now if you'll all excuse me, i really need to go back to work... :+)

1 0

Interesting message on TeBC from NetWorker (about fixing errors, Frankenstein, etc.)
by Jon Noring 28 Feb '05

28 Feb '05

[I'm forwarding the following message by "NetWorker" posted to The eBook Community. Any followup to the specific points NetWorker raises are maybe better posted over there, especially if you'd like NetWorker to see your comments ("ebook-community" at YahooGroups). NetWorker is very thorough... Jon] NetWorker wrote [a few days prior]: > Project Gutenberg e-texts may yet have a role to play in the > production of high-quality e-books. Rising to the bait, I have placed > a hold on my local library's one(!) copy of Frankenstein, which I > will scan when it arrives. I will try to create a highly structured > e-text (certainly not as fine as what Jon did with "My Antonia"). I > will then try to find a way to preprocess both the OCR'ed text and > the Project Gutenberg e-text in such a way that the two files can be > meaningfully "diffed." Hopefully, I can come up with a method that > will allow existing PG e-texts to be an automated "proofread" of next > generation Public Domain e-books. Boy, _that_ was an interesting experience! The project goals: As an e-book consumer, I want an e-book that contains _lots_ of metadata; the more the better. I want the metadata to be patterned, so that I can use automated tools to manage a collection; sorting by author, genre, publication date, publisher, contributors such as editors and illustrators, etc. I also want the actual text to be marked-up in such a way that 1) I can view the text with all the presentational richness traditionally associated with a paper book, if I choose to do so, 2) I can convert unambiguously from one markup language to another, and 3) so that I can do a structural analysis of the book using automated tools. I also want a mechanism to know if apparent errors in the text are due to transcription errors or the author's intent -- this can be accomplished by including source information in the metadata, or providing access to page scans; both would be preferable. Project Gutenberg e-texts satisfy none of these wishes; to create an e-book which _does_ satisfy them pretty much requires starting from scratch. Scanning technologies are quite advanced these days, but OCR is still not 100% accurate, and automated spell checking can only go so far. Clearly the most time-consuming -- and most error-prone -- part of producing a reasonably accurate e-book is proof-reading by a human being. My goal was to discover if Project Gutenberg e-texts, which are presumably fairly accurate as to the _words_, if nothing else, could be used as yet another automated preprocessing step to reduce typographical errors to a minimum before the actual proof-reading begins. The process: To test my theory, I decided to use the novel _Frankenstein_, by Mary Wollstonecraft Shelly. _Frankenstein_ is clearly in the public domain, is known to have at least two versions, and has been the subject of a fair amount of discussion on this list in the recent past. I obtained a copy of Frankenstein from the public library; it was published in the "Barnes & Noble Classics" series in 2000. I was fairly pleased with the edition, as it was printed in a rather old-seeming type-face which gave the appearance that it was in fact a photo reproduction of a much older text; it seemed likely that it had not gone through much in the way of re-editing to modern conventions. I scanned and OCR'ed the book using ABBYY FineReader. I then did a spell-check of the book from within FineReader so I could compared "misspelled" words to the actual scanned image. I then saved the text as an HTML file. In the past I have written a couple of programs to help in the creation of e-books. TidyeBook is based on the HTML Tidy code base. It fixes some of the inaccurate HTML produced by ABBYY, strips headers and footers but leaves page numbers intact, if invisible, and merges broken paragraphs when it can do so without question. html2txt, based on an earlier C++ version of HTML Tidy, takes an HTML document and reduces it to simple text similar to that used by Project Gutenberg. Next I ran "frankenstein.html" through TidyeBook to clean up the HTML. I then hand-edited the HTML to fix paragraph breaks not fixed in the automated process, or which should not have been broken. I also fixed those instances where hyphenated words spanned a page break (very easy to do given the output of TidyeBook). I then generated an Impoverished Text Format version of the HTML text using html2txt. My strategy was to use the Gnu "diff" program to detect differences between the simplified version of my work product, and the Project Gutenberg version. Because "diff" is line-oriented I needed to normalize the two texts so there was a greater likelihood that lines would be correctly matched. I did this by writing yet another program (this could probably have been done more efficiently by a Perl or AWK script, but I am not very familiar with scripting languages, but am a highly proficient C/C++ programmer; it was easiest for me to use the tools at my disposal). The new program would reduce each file to lines of no more than 60 characters (the shorter the line the easier for a human to find the difference detected). Additionally, the program would start a new line whenever it encountered what is conventionally accepted as sentence-ending punctuation (!.?) or two newline characters in a row, which would signal the beginning of a new paragraph. All whitespace was reduced to a space character, including multiple whitespace characters. I used the new program to normalize the text produced by html2txt and that of frank14.txt from Project Gutenberg. I then compared the two resultant files using gnu diff and Microsoft's WinDiff. The results: I was quite surprised to find literally thousands of differences between the two texts. Most of the differences were changes in punctuation and capitalization. Many em-dashes were converted to semicolons or omitted altogether, and many semicolons were converted to commas. Some words capitalized in my scan (eg. Paradise) were converted to lower case (paradise). Some phrases were "fixed" ("our uncle Thomas's book" became "our Uncle Thomas' book"; "an European" became "a European"). Some words were Americanized ("tranquillise" became "tranquillize") yet other words are not ("favourite" remained "favourite"). In an attempt to discover the source of these differences, I visited a number of not-so-local libraries, and checked out a number of different printings of _Frankenstein_. Two of the most interesting are Leonard Wolf's _The Annotated Frankenstein_, Clarkson N. Potter, 1977, which claims that "In order to ensure the authenticity of the text, we arranged with the Library of Congress in Washington, D.C., to microfilm a copy of the first edition. That text has been reproduced in this volume by the photo-offset process," and the Penguin Classics edition which includes an appendix identifying the differences between the 1818 and 1831 editions (while significant, they are neither as pervasive nor as substantive as has been earlier suggested). Neither of these editions contained the punctuation or spelling changes of the Project Gutenberg edition. One of the books I checked out, rather serendipitously as it turns out, was the Bantam Classic edition, which was first published in 1981. Of all the editions I consulted, only the Bantam edition contains virtually all of the changes I noted in the Project Gutenberg e-text. The PG edition is apparently based on Mary Shelly's revised 1831 edition (although it has lost both the "Author's Introduction to the Standard Novels Edition (1831)" by Mary Shell, and the "Preface" to the 1818 edition by Percy Shelly). I thus believe that the PG edition is based on the Bantam Classic edition of 1981. <sidebar> Interestingly, copyright law provides protection to changed versions of public domain texts if those changes are of a nature that they are more than mechanical and provide some modicum of creativity. Clearly, the punctuation changes are not merely mechanical, and in some cases actually change subtly the nuances of the text. Ironically, of all the textual bases that Project Gutenberg could have used for its e-text of _Frankenstein_, it choose the one which is apparently still protected by copyright! </sidebar> I modified my text normalization program to discard all punctuation except hyphens and underscores (and of course excluding the sentence-ending punctuation mentioned earlier). This reduced the noise to signal ratio enough that the differences started to become meaningful, although it still resulted in at least 500 differences. It allowed me to discover a handful of OCR errors that had been missed by the earlier automated methods, and I have so far also found a handful of errors in the PG text ("But must finish." should be "But I must finish.", "every sight ... seem still to" should be "every sight ... seems still to", "destroy radiant innocence" should be "destroy such radiant innocence", etc.) Conclusions: Of course, the goal of this exercise was not to establish the provenance of the Project Gutenberg e-text of _Frankenstein_, nor to discover if there are any errors in the PG e-text, but to determine if there was an automated method of reducing errors in newly scanned e-books for which a Project Gutenberg e-text already exists. I'm afraid the jury is still out on this question. If the texts are a different as the PG edition of _Frankenstein_ and virtually all other editions, the process of sorting through the chaff to find the grain of wheat may not be worthwhile; I believe that the OCR errors discovered so far were blatant enough that they would have been easily discovered in the first proof-reading, and I believe that human proof-reading will always be required no matter how good our automated tools become. Some time ago I produced an HTML e-book version of Mark Twain's _Pudd'n'head Wilson_. I believe I will put that e-book through that same process. I will then attempt a new scan of some other, perhaps more obscure, PD work that already has a PG version. Having at least three data points, I will report again later. p.s. -- If someone from Project Gutenberg wants my diff file to update the PG e-text, I will be happy to e-mail it to you; it is approximately 95k in size.

1 0