- gutvol-d - lists.pglaf.org

Formats and gripes
by Greg M. Johnson 08 Feb '10

08 Feb '10

I observed a bunch of things in the PG collection that were somewhere between maddening deficiencies in quality and simply how I wouldn't have done things. I could have made inflammatory demands that the whole system be rewrought according to my word. Instead I came up with my own custom DVD. And I'll testify that MH in discussing the idea was annoyingly non-paternalistic in not telling me how it had to be. ;) So, when someone complains, it is great when they can just go fix something themselves. There's the story of the king who demanded the whole world be covered in soft leather to keep his foot from being struck on a stone. Maybe someone should come up with a custom DVD with all the good works in EPUB format. I would respect PG if it were to refuse to host any and all of these bleeding edge formats. But I would love to be given the authority to create a raw text format without the 80-char line breaks. I agree that having these breaks (in the only TXT format available ) makes them unusable. The first-time visitor to PG probably won't be impressed that tools may be available to fix it.

9 26

roundlessness -- 003
by Bowerbird＠aol.com 07 Feb '10

07 Feb '10

we're looking at rfrank's "roundless" experiment at fadedpage.com... *** ok, it's been a while since i went through this little drill, so i'm gonna give you a little refresher-course on where i stand on a workflow that distribution proofreaders could be using, particularly from the viewpoint of rfrank's roundless research. *** first off, i would substantially re-ramp the "preprocessing"... this is the first step, where you're doing the scanning and o.c.r., or fetching the results from one of the major scanning projects. we'll start at the very beginning... *** i've advocated -- strongly -- a filenaming system that indicates, for each file, the contents of the file, so the name is meaningful. specifically, the _page-number_ of the file should be included in a consistent way in the name of a file. thus, for instance: > http://z-m-l.com/go/myant/myantp123.jpg the image-file for page #123 in the book "my antonia" is named myantp123.jpg. the "myant" prefix is common to files for the book, and the "p123" part refers to page 123. this is so straightforward that you couldn't be faulted for assuming that it's taken for granted. but it's not. many of the content-providers over at d.p. name their files differently, so the filename is _not_ unambiguously tied to the page-numbered contents inside; and a price is paid for this unclarity. yes, rfrank is one of these ill-advised content-providers, and he has carried over this bad habit to this experiment on roundless proofing. this shortcoming is even worse in a roundless system, because there is a need to refer to specific pages in such a system, and the absence of a sensible naming-scheme therefore becomes a bigger problem... (in a round-based system, where all the pages are treated as a "batch", the problem isn't as bad, but it's still something that should be fixed.) *** next, there are a number of things that are done in "preprocessing", at distributed proofreaders and by rfrank in his roundless experiment. some of these things should _not_ be done, in my considered opinion. on the other hand, there are other things that _should_ be being done. these are some things that are done which should _not_ be done: 1. run-heads and page-numbers are eliminated. 2. end-line hyphenates are being joined. 3. end-line em-dashes are being "clothed". these are some things that _should_ be done, which are not: 1. obvious and easily-located problems should be fixed. 2. spacey-quotes should be fixed. 3. ellipses should be standardized on a 3-dot ellipse. we can engage in debate on all of these suggestions, but it will be instructive for us to see some of the o.c.r. errors i'm talking about. appended to this post is a list of bad words or lines that were pulled from a book currently being proofed at rfrank's roundless experiment. these are almost all errors that will certainly need to be fixed. aggressive preprocessing can find these errors without looking for them. that's very important, in a roundless system, because if you can find and fix all the errors _before_ a page is subjected to a word-by-word review, then that word-by-word review can become the first "no-diff" in a chain of "no-diff" reviews, and the shorter that chain is, the more efficient you are. in the other scenario, if the page is dirty, you might have to have 1 or 2 (or even 3) proofings before the page is clean enough to receive a no-diff, meaning your efficiency has plummeted. there's no reason to make a human _search_ for errors when those errors can be located quickly and efficiently by a computerized search routine... -bowerbird p.s. sorry i've been sluggish with this series. i thought i'd been away from this stuff long enough that it wouldn't bore me to do it all again, but so far it has been a drag... i can only repeat this so many times... i'll try to get the motivation back again, but no promises if i cannot... p.p.s. here's that list of "probable errors" pulled out of rfrank's book: of'more Fd say pver t'other Enew a'slipped pn curtiss-robih somethin'to ght asighin' buncft whick mother'Ship ground^igood hefteaw iny wonderf ^Devolutions punkins outen tpreviously bustl gun'ls thaft ag^ny sud-v denly ij J>ack jumpin'his blame'em Jiack apture twise Weil numba chorteled pvounded Wowl oaly ^{w}Oh! haulded uae vre knovt apeak gink givin* stretch.'* J'ack wheen clost you'l althought eyefull weuns I"<cr> oa pHot etchin' jest'magine iframe ha.nd you'fe valk a'been outen morc'n MCGrath tc Unc' wuss'n pizen fresfi hirn orr hinsel| But you never can tell just what may hap]7^{en sion of weird noises springing up from the g<9^{a}^ 74-* /' 161 17S "Working over a bird with red feathers,'^{1 fall for such a decent game 93 taxidecentry or 18? was the work of a few second3. Hardly had the THE COMEBACK 227 HUNT OF THE S-18 belonged to the Hun pHot, Oscar Gleeb. must be pesU,@Jn' you like all get-out, so I made cruel to keep me a'guessing any longCi. so anxious to get started oA their way Porter Press disappears from an airpnQp .Watch my smoke, that's all." .until finally it died away completely. This gave .keep your eyes fastened on him. Whatever . A fortune hangs in the balance when young Dan Tierney, press , "Gripes! that was worth somthin'to glimpse, ; for Perk valued a few words of praise ;whooped the delighted Perk as he squatted pn you to hold my own. That's j^t how it should say,^{r} she bust out o' that little fog cloud right such as would tell ^ business being put through rendezvous and^ it's our game to chase after them, light, whether ^{r}'and or gulf, the chances were holding ground^igood--a heavily laden sailing path, and going through the most wonderf ^Devolutions. and be ready to pounce down on their inten<J^{e}d prey after the fashion of a hungry eagle striki ^{n} a fat duck that had been selected out of the flc^{c}k One thing he did do was to cut his intend^ wide circle short and again head toward ^{ne }scene of action, a move that certainly afford^ the eager Perk more or less satisfaction, he bel^{n} thrilled with the expectation of breaking into th^{e }game without much more loss of time. But you never can tell just what may hap]7^{en }when rival forces are striving against one ^^{n}' other. The best laid plans often go wrong a^{n}^ there was always a chance of the unexpected happening. Hardly had the airship whipped around ag^^{n }so as to head into the north than Perk beca<"^{e }aware of the fact that there was a sudden acc^{es}' sion of weird noises springing up from the g<9^{a}^ toward which they were now aiming. Jack, t^ must have caught the increased volume, for ^{ne }sheered off as if to hold back a bit so as to gr^^{s}P the meaning of the new racket. Men were no longer simply talking or laug^h" ing as they so cheerfully labored in transferring some of the contraband from the sloop to ^{ne }deck of the speedboat--their voices were rai^^{e}^ to shouts in which surprise, even the element ^ the frenzied sufferers in their ag^ny had been ^{w}Oh! That can be put through without muck said that name exactly three tin^s, like it meant operator as Oswald Kearns pick oui^an ordinary would rather have Jack praise him than ^ny one ^{r}fully five feet long and as thick through the body him, no matter where he goes-^-sorter dude, I'd will y^{u}> boy--two--three fellers jest swarmed "Working over a bird with red feathers,'^{1?? M^ans our gent has a raft o' ships comin' an' through fire or some similar means of destruc^ our man ditto. Mebbe now I'll soon^{x}get a chance "Gosh, amighty, we're flyin'\Mgh, buddy!"

1 0

roundlessness -- 002
by Bowerbird＠aol.com 07 Feb '10

07 Feb '10

we're looking at rfrank's "roundless" experiment at fadedpage.com... as i said yesterday, this test is a very very very very very good thing, because distributed proofreaders has been bogged down in a morass of "rounds" for many years now. their standard workflow now calls for _three_ rounds of proofing, followed by _two_ rounds of formatting... throw in a "preprocessing" round, and their "postprocessing", which is following by "postprocessing verification", and you've got 8 rounds. i don't know about you, but to me, that seems like a lot... but that's not the worst of it. the worst is the resultant backlogs... the problem arises because d.p. has thousands of proofers doing p1 (the first round of proofing), but d.p. only has hundreds that do p2 (the second round), and mere _dozens_ doing p3 ("final" proofing)... needless to say, the large number of proofers doing p1 can proof more than the smaller number doing p2, or the tiny number in p3. the backlog created is (understandably) frustrating and demoralizing for the proofers trying to keep up in p2, and is killing the p3 proofers. there is also the gnawing feeling that not all pages _need_ 3 rounds. indeed, _most_ pages in _most_ books are simple enough that they can be finished in one round, two at the most. so the _inefficiency_ of the 3-round proofing is rather striking as well. the thought is that each page should be proofed only as many times as that page needs; this has been labeled as a "roundless" system. aside from the backlogs of partially-done material, the other sign of a problem with the dp.p. workflow is that production has flattened... even though d.p. enjoys a constant stream of incoming volunteers, thanks to all of the good-will that project gutenberg's free e-books have generated over the years, d.p. output has leveled out at under 250 books per month, which works out to less than 3,000 per year. against the backdrop of the _millions_ of books google has scanned, this is a mere drop in the bucket. a small drop in a very large bucket. rfrank doesn't go into all of this on his site. perhaps he didn't need to, since the d.p. people he's recruited are well-acquainted with the issues. but rfrank is also unclear on many of the details of his little experiment, which is a more worrying matter. specifically, i don't see a lot of experimental rigor here. it seems to me that roger is unfamiliar with the mechanics of the scientific method and its applicability to human social experiments. i see no evidence of any stated hypotheses, nor any way such hypotheses can be disconfirmed... the reason people developed the scientific method was because we found that when we just fooled around "to see how things turn out", we often ended up fooling ourselves about what we had seen, and what it meant. we learned that we had to actually specify our hypotheses, and devise tests (experiments) specifically designed to disconfirm our hypotheses. otherwise, our brains are only too willing to accommodate what we find as being "supportive" of our initial impressions. ("experimenter bias" is the term by which this insidious phenomenon is most well-known.) if i'm correct, this problem will surface in rfrank's future results, and surface repeatedly, so there's no need for me to labor the point now. but i wanted to frame this particular issue, here and now, in advance. that's enough for today. see you tomorrow... -bowerbird

7 10

Re: Psychology of interacting with (Google's) ebooks.
by Bowerbird＠aol.com 06 Feb '10

06 Feb '10

greg said: > So what on earth are these big-pipe servers doing? collecting information, so as to assemble it, and reassemble it, thereby producing new information that might change the world. or make money. lots of money. lots and lots and lots of money... -bowerbird

1 0

Re: psychology of interactimg with ebooks
by Greg M. Johnson 06 Feb '10

06 Feb '10

<<discussion of poetry auto-generated files messed up>> IMNSHO, another fruit of making the TXT-80 format the default standard. The programming will ne a headache if one can't just delete *all*the CR's.

2 1

Re: psychology of interactimg with ebooks
by Bowerbird＠aol.com 06 Feb '10

06 Feb '10

greg said: > The programming will ne a headache > if one can't just delete *all*the CR's. nah. programming an unwrap routine is quite easy... > http://z-m-l.com/go/unwrap.pl -bowerbird

1 0

Re: autogenerated HTML
by Bowerbird＠aol.com 05 Feb '10

05 Feb '10

i said: > without even looking at those files, i can guess what's wrong... > > many of the books that are exclusively poetry are set flush to > the left margin, lacking any of the leading spaces that serve > as a signal to the conversion program not to wrap the lines... > > so of course the converter is gonna wrap the lines. > > this is an error, a major error, in the processing of these books. > > (and it's so easy to change every linebreak to a linebreak+space.) well, gee, i should have looked at those files earlier, because when i did get around to looking at them, i got a good laugh. see, the lines _were_ indented, so shouldn't have been wrapped. and wouldn't have been wrapped if most of the converters around would've been used. but i have since learned that there is an "experimental" converter, programmed by marcello, and that's what was used for this book. the irony of this gave me a big hearty laugh. you see, when i laid out the philosophy of z.m.l. here on this list, some of you will remember how much crap marcello threw at me. the guy was relentless. even though he almost never made _any_ posts to the list otherwise, he would respond negatively to anything i said. but never constructively. he'd just throw out pure bullcrap... for a long time i responded, just to clearly specify that it was crap; but after a while i decided to let his crap speak for its crappy self, and i stopped responding. after that, he stopped responding to me. (i guess that thing they say about not feeding the trolls is right on.) but if you wanna see what an ass he was, you can check the archives. you can also go look at a website he set up with a lot of quotes from messages i sent here. most of them are taken out of context, sure, but even then i stand behind them. you'll see that they were correct. (that's right, he set up a _fan_ page, on his web-site, to ridicule me; i don't know where the guy is coming from, but i think he should get a life.) anyway, marcello insisted that my approach was bunk, and that one could not successfully generate a full-on .html file from a .txt version. yet now he is writing code to do just that. (he's not doing it _successfully_ yet, so in regard to _himself_ alone, i guess he was right. but i have been successful for a long time now.) what's especially ironic -- and funny to me -- is that marcello is now suffering through the same complications that i experienced, namely maddeningly inconsistent .txt files, which he must program around, just like i did. further, this will lead him to the same conclusion that i came to, which is that this would be _much_ simpler if only the rules that p.g. has already established for .txt files were simply _followed_... even more irony, and thus even more humor? marcello is programming in _python_, a language where indentation is _meaningful_. this is ironic, and funny, because one of the things about z.m.l. which marcello once tried to lambaste is that whitespace is meaningful. every time he does an indent, i hope he chokes on it. meanwhile, i just have one thing to say to all you z.m.l. naysayers: i told you so. you were wrong. i was right. i hope you choke on it. (wait, is that one thing, or 4 things? oh well, guess it doesn't matter.) -bowerbird

1 0

autogenerated HTML
by traverso＠posso.dm.unipi.it 04 Feb '10

04 Feb '10

I have just become aware that PG now autogenerates HTML for texts that don't have it. Unfortunately however sometimes the autogenerated file is garbage (e.g. poetry rewrapped, see 31079). Would it be possible to have the autogeneration program to find what is the problem, or at least to preview the autogenerated file and possibly fix either the program or the files? Carlo

2 1

Re: autogenerated HTML
by Bowerbird＠aol.com 04 Feb '10

04 Feb '10

carlo said: > I have just become aware that PG now > autogenerates HTML for texts that don't have it. since a number of people are fond of rewriting history here, let's note for the record that i suggested this some time ago. indeed, my recommendation was that the .txt version should be used to autogenerate the .html version for _all_ the books, that hand-crafted .html be abandoned because it is too hard to maintain and to upgrade. i also suggested that conformance to this strategy would enable p.g. to improve the .txt versions... and i predicted that sooner or later, you'd all come around to this workflow. and how you have. so i will say "i told you so." > Unfortunately however sometimes the autogenerated file > is garbage (e.g. poetry rewrapped, see 31079). without even looking at those files, i can guess what's wrong... many of the books that are exclusively poetry are set flush to the left margin, lacking any of the leading spaces that serve as a signal to the conversion program not to wrap the lines... so of course the converter is gonna wrap the lines. this is an error, a major error, in the processing of these books. (and it's so easy to change every linebreak to a linebreak+space.) > Would it be possible to have the autogeneration program to find > what is the problem, or at least to preview the autogenerated file > and possibly fix either the program or the files? i've never tried to verify it with a closer analysis, but my impression is that some of the whitewashers use a slightly different converter... and then of course there are a number of different ones over at d.p., including the one in thundercat's app, and another by david garcia... without dedication to making the .txt program correct at the outset, however, it doesn't matter how good the converter might be... -bowerbird

1 0

Psychology of interacting with (Google's) ebooks.
by Greg M. Johnson 03 Feb '10

03 Feb '10

I downloaded two epub's from Google Books and one or both of the book reading apps on my Android phone didn't even see one of them. I think that some of these collections are designed with the idea that the repository should be on the web, and you the reader should go search the web interface to find a book you want, then download that one book, have perfect confidence it's going to be cool to read and functioning properly, then maybe you'll go on to the next one a few days later. I don't think humans work that way. First of all web interfaces, especially on a phone, are inherently slow, and sometimes unavailable either due to wifi/ 3G coverage or due to embarassment about using "work bandwidth". The Google Books interface isn't *bad*, but it's still like being fed at a gourmet banquet with a baby spoon. The user may have one bad experience with a downloaded text, no matter how small, and they want to curate their own collection first, maybe hoard-up more books than they or their family could read in a lifetime, cull out the icky or malfunctioning texts, and then have say 20 on their reader and 2000 on a DVD in a safe in their basement. At least that's how I respond to having one or two minor problems. ;) I don't think that Google Books at least gets this. I spent so much time at Google Books, browsing in apparently spider-like fashion, that I got this warning: "We're sorry... ... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now." I guess they're right. At any moment I was about to try to download a few hundred epub's. -- Greg M. Johnson http://pterandon.blogspot.com

10 14