- gutvol-d - lists.pglaf.org

Re: ok, let's take a look at gardner's book, just for the exercise
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

i did some more work on gardner's book, "the advocate". the (presumably incorrect) lines from the o.c.r. are in red, while the lines from gardner's proofed copy are in blue... (at least i hope i didn't screw up the colors like last time.) here's a sample url, using page-by-page mode: > http://z-m-l.com/go/gardn/gardnp123.html here's a single web-page containing the whole book: > http://z-m-l.com/go/gardn/gardn-hybrid6.html and here's a web-page with _just_ the diffs on it: > http://z-m-l.com/go/gardn/gardn-159diffs.html we've got 159 diffs. that's about all the preprocessing that can be done on this file. 159 diffs on ~4000 lines is _great._ saves a ton of time on word-by-word proofing. in this last round of preprocessing, i did a spellcheck... i should explain that i do not use a regular spellchecker. i copy all the text into a program (which i coded) that spits out a list of all the words _not_ in its dictionary. if you're interested, here's the dictionary that it uses: > http://z-m-l.com/go/regulardictionary.txt then i go through the list and cull out the words that -- for whatever reason -- look to me to be correct... (most typically, they're names and compound words, neither of which class happens to be in my dictionary; doing this culling saves me a lot of time that would be spent checking and then by-passing these valid words; it's basically a quick way to create a custom dictionary.) this leaves me with a list of words that i want to check, which i then feed into my banana-cream app, so it will show me all the occurrences of those words in the text, with the relevant page-scan displayed at the same time, so that i can check the text against its scan. it's handy. by the way, i can also feed banana-cream a list of lines, which it will then search for and display automatically... this is what i'll do to resolve those diffs displayed here: > http://z-m-l.com/go/gardn/gardn-159diffs.html -bowerbird

1 0

so what is so important about pagination?
by christine 23 Feb '10

23 Feb '10

The format text file is unsatisfying anyway, but as long as we insist on having it, we should make the best of it. Even if the page numbers are only given every second pages because of long paragraphs, it is IMO still better than not giving any at all. And putting it at the start of a paragraph would not make it useless in case of indices. It would reduce the search of the user to a couple of pages instead of a lot of them. Christine > > > ---------- Forwarded message ---------- > From: Marcello Perathoner <marcello(a)perathoner.de> > To: Project Gutenberg Volunteer Discussion <gutvol-d(a)lists.pglaf.org> > Date: Tue, 23 Feb 2010 19:22:27 +0100 > Subject: [gutvol-d] Re: so what is so important about pagination? > christine wrote: > >> I am not talking about adding big chunk of blank lines, but of either >> including the page number like that [p.101] in the text, I do not like this >> solution since it disturb when doing a search; >> >> or adding the page number at the start of the 1. paragraph starting on a >> page, e.g.: >> >> change of page, >> blah blah end of paragraph. >> >> [p.101]Text of the new paragraph. >> >> With a TN explaining that the page numbers are not accuratly placed at the >> start of the page, but at the start of the 1. paragraph starting on that >> page. The aim is to make the use of the files as easy and comfortable to the >> reader as possible. >> > > Probably would not work with Ulysses or Kant or many other texts that have > multi-page paragraphs. > > > > So we are left with: > > 1. put the page number where it belongs, even if its in the middle of a > word, or > > 2. put the page number where it irritates less, even if its up to one whole > paragraph off. > > Of course 2. would make it useless for indices, the use case you advocated > in the first place. > > Both cases are unsatisfying, IMO. > > > The only place where page numbers belong is in invisible markup that can be > made visible on demand (CSS edit or so). > > > -- > Marcello Perathoner > webmaster(a)gutenberg.org > >

1 0

Re: !@!!@!!@!Re: Re: so what is so important about pagination?
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

michael- i said it before, and i'll say it again. but i'm tired of repeating it. i'm on _your_ side. i've got your back. lately, you're beginning to make as little sense as jon noring used to, arguing "against" me when you don't even know what you're saying... i'm really going to have to stop talking to you. it's a waste of time. *** > Then say it is an OPTION. . . . ... > /then say OPTION!!! i don't know how to say it any more clearly than i already have... the whole point is that some people _want_ pagination, and i said "why not give it to them?" what is that, if it's not an _option?_ and as for the people who don't want to be bothered with pagination, i've said "why not give them what they want as well?" as _their_ option. i've even devised a format, and written the code, that will allow _both_ parties to get what they want. precisely because i believe in _options_. does p.g. offer people who want pagination a way for them to get it? no, it doesn't. and yet you yabber on about "options", like you owned the concept or something. why don't you start giving people options? you even go on to say "love it" when lee passey offers up a convoluted and cockamamie scheme with complex and unnecessary .html coding. does project gutenberg have a page on its website that will _unwrap_ an e-text? no, it doesn't. but i have a page on my site that'll do that... > However, if references were made to PHRASES and NOT PAGES, > this would work EVER SO MUCH BETTER!!!!!!! lay off the exclamation points, old man. they're making you punch drunk. making references to "phrases" might work if we did it from here on out, but there'd still be millions and millions of references to pagenumbers in the corpus that represents our cultural heritage. i know you to be a strong supporter of that cultural heritage, so i know you can't mean what you appear to be implying here, which is that we lose all those references. *** as for the rest of your reply, it just degenerates into meaninglessness... i have made my point, for the people who care to understand it, so i will opt out of stupid aspects of this conversation for the rest of the thread. -bowerbird

2 1

so what is so important about pagination?
by christine 23 Feb '10

23 Feb '10

The PG template should then be taken into account, but it differs depending on the WWer ... It would also require the reader having a reader with line numbers, mine does not have them. As for the page numbers disturbing the flow of reading, that is only true for the first couple of pages, you soon get use to it and do not see them anymore. Christine > ---------- Forwarded message ---------- > From: Joshua Hutchinson <joshua(a)hutchinson.net> > To: gutvol-d(a)lists.pglaf.org > Date: Tue, 23 Feb 2010 14:46:11 +0000 (GMT) > Subject: [gutvol-d] Re: so what is so important about pagination? > I'd suggest a different direction. > > If you have a situation like that, don't bother with page numbers (since > they do disturb the "flow" of reading). Rather, in the index, use line > numbers. Most modern editors will happily tell you what line number you are > currently on and even jump to a specific line number. The line number in a > text file is fixed, so it will be a consistent number to jump to and yet > doesn't disturb the flow of reading in the text itself. > > Now, before you say it, I realize this would require someone to write up a > quick and dirty script/program to convert page numbers to line numbers in an > index ... but it's doable. > > Josh > > On Feb 23, 2010, *christine* <lostpaces.dp(a)googlemail.com> wrote: > > I have been following this conversation, and would like it to lead to a > slight change in the PG pagination rule. > > As post-processor at DP-Int, I often have the feeling of uploading > unfinished works because I have to remove the pagination from the text > files. I don't mind in novels, but for books with index, it is another > matter. > > Some time ago, I received a mail asking me why I format my page numbers as > invisible in the html files. One of my reason is that I can not see the > point of showing them, since most of everything is linked (TOC, LOI, Index, > etc...) > > When I started post-processing, I included the page numbers in the text > files, and often had to remove them to have the project posted. When I asked > about the reason, I received the answers "it disturbs", "there is no need > for it, the reader can do a search", etc. > > Now, it does not seem very logical to show them in a linked file (html) and > leave them out in the text file. Doing a search is not a very good solution > either, in most of the case, it is like searching for a Smith in a british > telephon book, more finds than you care for. > > As example: I am currently working on "The Life of Napoleon", one of the > index entries says "N. returns to Paris". Well, searching for "Napoleon" or > "Paris" in that project would brings too many finds, as for "returns" this > would not bring you to the proper page either, because, there, you find "is > driven back". > > So, would it be possible to just relax that rule and let the volunteer > decides *if* the pagination in the text file of a project is needed or not. > > Christine (aka Lostpaces) > > > > ------------------------------ > > _______________________________________________ > gutvol-d mailing list > gutvol-d(a)lists.pglaf.org > http://lists.pglaf.org/mailman/listinfo/gutvol-d > > >

3 2

Mirroring the firehost? Re: Re: Many solo projects out there in gutvol-d land?
by Greg Newby 23 Feb '10

23 Feb '10

On Sat, 20 Feb 2010, Michael S. Hart wrote: > > > Does anybody have a serious objection to putting the 8,000, > > or so, books that were listed earlier as being in limbo, in > > something like our "PrePrints" section, where we put eBooks > > that are admittedly not ready for prime time??? > > Yea, there are people arguing that it's a horrible thing to do. I'm 100% with > you on this. Available with a few errors is far more useful than unavailable. > And it's not that they aren't actually available now, they are. DP has always > had the concatenated text available for download. It's behind a sign on and not > indexed by any of the search engines, so if you don't know it's there already > you can't find it. What's the URL? I could set up a nightly mirror... Do they automatically disappear from this area, after they are finally published? -- Greg

5 7

Re: ok, let's take a look at gardner's book, just for the exercise
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

gardner said: > Interesting. The colours go a bit wonky at um, actually, i have the colors totally reversed, all the way through. and the "bad" line was on the bottom, when i usually put it on top... i confused myself by looking at some diffs where the o.c.r. was right and your version was wrong. they are few and far between, but i still managed to convince myself that up was down and port was starboard. sorry! > substituted the correct spelling "Sainte Hélène" where the that's a different problem. my text-editor assumes mac-roman, so i canceled out the few places where there was high-bit text. > But there are places where "my" text contains errors > that I do not believe were in my text. again, that's because i mixed up the colors. your text was in red. i got it straightened out, and will be uploading a new version soon, so wait until i do that before you spend any more time checking it. (and actually, you don't have to do anything; i will do a full report.) i'm sorry, i apologize for the confusion. i ain't perfect... :+) -bowerbird

1 0

Re: ok, let's take a look at gardner's book, just for the exercise
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

i did some more work on gardner's book, "the advocate". i continued to do preprocessing on the archive.org o.c.r., and posted the results on my website. here's a sample url: > http://z-m-l.com/go/gardn/gardnp123.html the number of lines that differ between the cleaned o.c.r. and gardner's file has dropped to under 300, which is great, since the book contains about 3,900 lines in it. so it's <8%. so again we show the stupidity of a word-by-word proofing. i'm gonna go through one final round of preprocessing, and then we'll set out to resolve the differences. already, however, i can tell that gardner did a very good job on it. not perfect! but certainly something to be quite proud of. the (presumably incorrect) lines from the o.c.r. are in red, while the lines from gardner's proofed copy are in blue... if you prefer to view all the edits on one web-page, see: > http://z-m-l.com/go/gardn/gardn-hybrid5.html i also included the preprocessing edits i made on page 126, so you can get a feel for how little time i spent doing them... -bowerbird

2 1

Re: Bowerbird's software projects
by Bowerbird＠aol.com 23 Feb '10

23 Feb '10

gardner said: > http://unixcomputer.net:81/new-photo/cd/Advocate/ i scraped them from canadiana.org myself... :+) but hey, do you still have your original o.c.r.? or a latest version that has the original linebreaks intact? *** > I don't believe that a single pass is feasible, ok, i should elaborate. multiple passes, to check different aspects, will be required. but multiple passes to check the _same_ aspect are inefficient. > I don't believe that a single pass is feasible, > in particular for mismatched quotes and spaced quotes. i believe you're wrong, and that i can show you. > in particular for mismatched quotes and spaced quotes. > You fix the open quote, or in my case close more often than not, > then that reveals another quote problem further down/up. i have already demonstrated that you can fix spacey quotes, and -- in the vast majority of cases -- fix 'em automatically. leading and trailing spacey quotes are easy to fix, of course. from there, it's a simple matter of segmenting the text into _paragraphs_, and counting quotemarks in each paragraph, making sure that the odd ones are open, and the even closed. then when you come upon a spacey quote, fix it to be open if it is an odd one, and fix it to be close if it is an even one. if you come up against a case where there is an odd number of quotes in a paragraph, and the next paragraph does not start with a quote, then you have a case you need to look at. similarly, if any of the quotemarks come up as the wrong type (an odd that's close, or an even that's open), you need to look. you can test this for yourself. you'll find that it's very robust. usually there's no need to spend much time on spacey quotes. > In any event I am not troubled by multiple passes. ok. > Well it *is* a Gutenberg text after all. right. that point wasn't directed at you, as you correctly realized. > Thanks. well, the fact that you haven't wasted your time is only _part_ of the equation. the fact that you won't get much credit down the line (because _your_ text will be discarded because you threw away info that people will want) is yet another (bigger) part of the equation. > Sure. The book *is* public domain after all. Do what you like. i think you missed the point. you can mount a version of your work that doesn't throw away the important information, and then no one will have to re-do it, in which case they will be happy to continue to give you the credit. -bowerbird

7 7

Markup and Pagination (was RE: Re: !@!!@!!@!Re: Re: so what is so important about pagination?)
by cmiske＠ashzfall.com 23 Feb '10

23 Feb '10

The only justification that _I_ can see for the PG 'books' to contain some form of pagination would be in a markup version with a hidden reference to the source page (scan image - if one exists). This would facilitate referring to the original book pages for simple corrections in the released versions. If it is _your_ intention to preserve the book, then the retention of pagination and other formatting would make perfectly good sense as well. I have followed this project (and this list) for around a decade. I have always been of the understanding that Project Gutenberg was about making available the contents of public domain books to as many people as possible. It has never seemed to me that a goal of the project was to create a re-printable facsimile of an existing edition, but rather to create the Project Gutenberg edition. I’ve always thought of us as the ancient scribes who spent their lives copying the contents of aging editions of books onto new materials so as to preserve the content for another few generations of time. The wonderful thing about the digital editions at Project Gutenberg is that they will last ‘forever.’ I have wondered for quite some time why the primary format for archiving (not necessarily submission or release) at PG does not include markup. It would be amazingly easy to generate the PG plain text version (for release) to exacting standards from a rather simply marked up original, but it is rather complex and potentially error prone to produce any format other than plain text from the plain text document. I have done the final plain-text formatting stage before and it is a slow and potentially human-error riddled process to manually format to PG standards (many of the standards I refer to are actually from DP rather than PG). I believe volunteers could be found to markup texts submitted (current and past) that lack such enhancements and to work towards getting the backend of PG more standardized so as to facilitate greater accessibility and distribution of the content. I realize that DP is already producing HTML versions, but that is not precisely what I am referring to and does not address the multitude of files without HTML versions and those that follow individual markup standards. And, I would never suggest that submission of a book be blocked or release be delayed by a contributor not having the ability or desire to markup their document. I am only suggesting the PG use a more re-usable formatting method for archiving and on-the-fly content generation than the current standard of somewhat-standard plain-text (and somewhat PG/DP standard HTML). There has been a lot of talk on here over the years of markup and preservation of information from the original book. Markup would enable the retention of as much (or as little) information beyond the content as the volunteers wished to provide without interfering in the creation of a Plain Vanilla text file. It also would enable the creation of one document with multiple editions nested within (or at least the difference between editions). The later being what I believe Michael just indicated was his ‘dream.’ Also, markup, when done with that purpose in mind, is human readable. <chapter title=”Chapter I” subtitle=”The quest for solitude”> <paragraph>Helen was <emphasis display=”italics” edition=”1845, Dover”>not</emphasis> in the mood for company and she resented the <errata edition=”1845,Dover” type=”replace;typographical error” details=” nkock”>knock< /errata> at the door….</paragraph> <paragraph><quote>Just go away!</quote> she yelled and then went back to reading her copy of <title type=”magazine” display=”italics;underline” edition=”1845, Dover;1865,NY Press”>The Mystery Guild Weekly</title>.</paragraph> …. <page number=”1” edition=”1845, Dover” /> <paragraph>She really had no idea why there were so many people coming <page number=”1” edition=”1865,NY Press” />to her house on such a day as <errata edition=”1845, Dover” type=”replace” details=”to-day”>today< /errata>. One would <errata edition=”1865,NY Press” type=”addition” details=”almost” /> think it was her birthday….</paragraph> …. </chapter> This is a rather poorly thought out and quickly done markup schema, but I believe it serves the point in that it is human readable. It would not be an enjoyable read, but it would be readable. And, long after all of us are gone, if this were the only copy of this document left on earth, someone could make sense of it enough to convert it to plain text (or another more readable format). They could also create a reproduction of the 1845 book or the 1865 book. Or, create an edition of one year or the other with footnotes or sidenotes/marginalia of the differences in the other edition(s). In the meantime, while we are still around, PG could publish the Plain Vanilla version (and any other versions they choose to make available). I proposed something similar to this many years ago over at DP, but there was little interest in it at the time. I’m not sure there is much interest in it now…. In any case, thank you to all the volunteers for all of the hard work over the years and all the books you have provided for my pleasure and for the pleasure of so many others! I’ve never felt as though you get thanks often enough from all of us users of the end results of your dedication. Thank you! Carel

1 0

Re: !@!!@!!@!Re: Re: so what is so important about pagination?
by Bowerbird＠aol.com 22 Feb '10

22 Feb '10

michael, i wish you would've taken the time to read what i actually wrote, instead of just giving your kneejerk reaction. because your response doesn't address the point that i made. i am loathe to get into this argument, because it won't mean a thing in the long run... the future will have its own issues, and it will need to deal with them, and i laid it all out clearly. but let me just address a few things, to provide some clarity. > he fails to talk about the wide variety of Jane Austen's > p-books, and that paginations run rampant among them, > not to mention margination, spelling, etc. there are different editions of many books, to be sure... and i count each edition as a separate book. your e-book will have to mimic _one_ of the editions in a faithful manner, or it will be discarded... notice that when i say "mimic", i do _not_ mean that it has to match it _exactly_. so, for instance, if you wanna close up spacey contractions, or correct spelling, or make other kinds of changes, they might (or might not) be totally acceptable to any one specific end-user in the future... but you _will_ have to make it easy for that specific end-user to _compare_ your e-book with a p-book, in order to spot changes. i've shown how this comparison is done, by mounting a web-page which has the text on one side of the screen, the scan on the other. but if your e-book is a formless blob, that's not gonna cut it... and, for the record, i'm most certainly _not_ recommending that we create some "scholarly" version of our books. i laugh at that. the _only_ thing we know about the scholars of the future is that we do _not_ know what they will want and it'd be foolish to guess. put yourself in the shoes of the future. you have a dozen different e-book files, all purporting to be copies of "sense and sensibility". you know that some of them have been doctored, and others have been bowdlerized, and you _hope_ that some of them are accurate. you can, with some work, find the differences between them, but you'd prefer not to have to go through that exercise if you could, because you'd have to then do more work to find the _right_ copy. so, how do you proceed? well, i can tell you that if _one_ of those copies made it _simple_ for you to verify its accuracy by assuming the form of the p-book, that will be your obvious first choice. think about it. you'll agree. so, michael, if you want to respond to this, answer that question. -bowerbird

2 1