February 2010 - gutvol-d - lists.pglaf.org

[SPAM] re: Many solo projects out there in gutvol-d land?
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

michael said: > Let's just forget the whole idea of error free texts. . . . well, that's going a bit overboard. better, let's try to achieve perfection, but let's not let that high goal get in the way of making work available even if it's not yet perfect... > I would prefer just to get these materials in circulation-- > then worry about approaching perfection along with Xeno. yes, except i don't see any part of project gutenberg that is doing very much at all in the way of "approaching perfection". once a text is posted, people seem to forget it completely... even when the whitewashers "correct" an e-text, they aren't doing nearly what they could be doing in order to improve it. it seems like there is a constant mad dash to do new books, but almost nothing is being done to fix up any older books. i asked at 10,000 books for a review in terms of quality control, and again at 15,000, and again at 20,000, and again at 25,000. i didn't bother to ask again at 30,000, because what's the use? but at some point, some hard questions will need to be asked... > Does anybody have a serious objection to putting the 8,000, > or so, books that were listed earlier as being in limbo, in > something like our "PrePrints" section, where we put eBooks > that are admittedly not ready for prime time??? well, i certainly don't... but many of the volunteers over at distributed proofreaders do. indeed, according to a poll (which has now received one of the highest number of votes on any poll that has been done there), there are split evenly -- right down the middle -- on this issue. i don't know what to make of that. but that's the way it is. it's also worth mentioning that those 8,000 books are _not_ "almost done". some of them really aren't even very close... some are full of typos, still. most contain pseudo-markup, which really should be converted to something more useful before the books are ever put in front of the general public. lots contain "proofer's notes", which would confuse people. it should also be noted that many of them are not in english, which might (or might not) have bearing on the question, but since i only speak english, i wouldn't have any idea what it is. considering all this, it would _not_ be a simple procedure to free up this matter. it could use up a lot of time and energy, and for very little benefit in return. (does any use "preprints"?) what _would_ be useful is for this material to be put on a wiki, in order to test notions of public postprocessing collaboration. instead of saying "here, take this unfinished work", we _should_ instead be saying "here, come help finish this unfinished work". at one point, i was tempted to build such a postprocessing system. but then i realized i didn't want to help d.p. get over their backlog; d.p. deserves to suffer the consequences of their terrible workflow, or they'll _never_ be motivated to fix it... so i decided to let it be... -bowerbird

2 1

Re: ok, let's take a look at gardner's book, just for the exercise
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

i've merged gardner's corrections into the archive.org o.c.r., and posted the results on my website. here's a sample url: > http://z-m-l.com/go/gardn/gardnp123.html gardner dehyphenated end-line hyphenates, so i rejoined (some of) them. i'll write another routine to do the rest... out of a file that contains about 4,000 lines, there are only 800 (at this point) which differ in the two versions. so even early in the merge, 80% of the o.c.r. lines were right, and that number will increase with more aggressive cleaning. the (presumably incorrect) lines from the o.c.r. are in red, while the lines from gardner's proofed copy are in blue... if you prefer to view all of the edits on one web-page, see: > http://z-m-l.com/go/gardn/gardn-hybrid.html -bowerbird

1 0

Re: so what is so important about pagination?
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

gardner said: > I'm going to take this as a jumping off point > for a more general question about > whether pagination of a published edition, > is worth saving. Obviously there is a range of opinion. > I'll give you mine. yes, there is a range of opinion. i can give arguments -- even good arguments -- on both sides. which obviously means that _some_ people have good arguments for retaining pagination. and you disenfranchise those people entirely when you throw out the pagination, no matter how good your intentions might be for doing so. i'd rather not disenfranchise those people... so i think it's necessary to include the pagination, and the original linebreaks, with end-line hyphenates. now, i think it's _imperative_ that we give people tools that enable them to discard that pagination, and unwrap those original linebreaks, and rejoin end-line hyphenates. to do otherwise would be to disenfranchise _those_ people, and i'd rather not disenfranchise them either. so, for me, the answer to the question is extremely simple. > What I believe, philosophically, I am shooting for > is to capture the core content, and reject the details > that have mainly to do with the medium of publication. i can understand that perspective. i can also understand the other perspective. and i see no reason that anyone has to be unhappy here. it's very important to understand that this does _not_ have to be an either/or question. we _can_ do _both_... > Maybe there are possible future uses of my text > that would want the things that I left out. > I tend to doubt that this could ever be very important. well, then, your imagination is starting to lag behind... :+) because we are now right in the middle of a situation here where "the things that you left out" _are_ "very important", namely a reproofing of your book, to test your accuracy... it's an order of magnitude more difficult to proof a book when the text has lost all of its linebreaks and pagination. are you of the opinion that the future will simply _accept_ that you did a perfect job in the digitization of your books? or do you think they will want to verify the quality of them? if you make it too difficult for them to undertake that job, they will just toss out your text and start anew. your loss... > As an individual contributor I do not feel that my time > is best spent capturing and encoding that, and so I don't. except the info was already captured. then you threw it away. > And I am happy that PG finds my efforts acceptable > despite this deficiency. except the future will throw out all the d.p. works because your deficiency is shared by the entire d.p. corpus, sadly... (even the d.p. people who save pagination toss linebreaks.) > I haven't done any sort of real research, but > a quick look shows me that > not many texts attempt to preserve line endings in any way. > Preserving line endings seems quite unpopular. the future needs to future-proof tens of millions of books... they can afford to throw out everything done up to this point, if they feel they need to, and they will, they most certainly will. (and advances in o.c.r. and o.c.r. correction will make it easy.) (well, as i pointed out, they might use some of the current texts to proof the new o.c.r. that they do, but then they'll toss them.) > Bowerbird wants to keep both actually, i don't need to take a personal stand on the issue, not as an end-user. and that's a good thing, because often i don't have a need for the original linebreaks or pagination. so i definitely want the option of discarding that information. what i am saying is that, as a "best practice" for digitization, the discarding of such information is clearly a terrible mistake. and if you're doing it simply because _you_ don't see a need for that information, then you're being selfish and shortsighted. plus you're giving an ultimatum to people who want that info: they're forced to toss your text as failing to meet their needs. and i am positive that you're going to lose that bet, gardner... -bowerbird p.s. by the way, i found 23 discrepancies in the paragraphing between your version of "the advocate" and archive.org's o.c.r. however, 21 were errors in the o.c.r., and only 2 in your book. the two errors in your book, both of them missed paragraphs: > http://z-m-l.com/go/gardn/gardnp087.html > "What, take the bird back to the bush where we > http://z-m-l.com/go/gardn/gardnp101.html > Yet who shall blame the sun and moon for that?

2 1

Re: Many solo projects out there in gutvol-d land?
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

postprocessing at distributed proofreaders is difficult only because the proofers are instructed to throw out meaningful data, which later then needs to be replaced, and the formatters insert obtrusive pseudo-markup, much of which later needs to be reworked or deleted. if the proofers used nonobtrusive zen markup instead, it wouldn't interfere with their proofing task, and there wouldn't need to be a separate formatting task, even if (in reality) people decided to specialize on that aspect. also, the conversion of proofed/formatted pages into a full-on electronic-book should be an automatic process. i've already demonstrated this many times, but would be happy to do it once again, on any book of your choice... this point is particularly important in a roundless system, where the object is to move a page to a "finished" status as quickly as possible. -bowerbird

2 1

rfrank reports in
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

ok, rfrank has made a report over in the d.p, forums on the latest set of results from his "roundless" experiment. so let's see what he says, and what i think in reaction... *** rfrank said: > So far in testing the roundless system as it stands, > I've left it to the proofer to say when they thought a page > was done. Turns out, that is reliable for only a very few proofers. > Those who wish to say "I told you so" can chime in now, rightfully. ok. > the post-processing clearable errors were > caused mostly by four proofers, and > each of those made several different kinds of mistakes. > These mistakes were found almost entirely > on pages that were one-and-done, that is, proofed once. > So what is to be done? inform those proofers they are making mistakes, and how, and that they not doing nearly as well as they think they are. and to put this into perspective, there are about a dozen committed proofreaders taking part in this experiment, with another 5 dozen people contributing fewer pages... so four "bad" proofers constitutes about 1/3 of the lot... in other words, even though "4 proofers" sounds _rare_, the actuality is the percentage of "bad" proofers is high. this fact should _not_ be surprising. when you fail to give people any feedback on their performance, many will think they're doing a fine job, even if they're doing a terrible job. (this is a big problem over at the d.p. mothership, but we probably shouldn't be getting into that can of worms now.) after all, if they didn't think they were doing fine, they would change what they were doing, so they _could_ be doing fine. so you absolutely need to give them good and fast feedback. > One solution is to have every page > looked at by at least two proofers. that seems straightforward, but it has some gotchas. > That seems straightforward but it has some gotchas. right. :+) > If every proofer knows that every page > is going to be looked at by someone else, > will they proof that page differently > than if they intended it to be one-and-done? it's likely. so you'd need to assume it, and work from there. > I think they might. Knowing the underlying mechanism > can undermine the process. well, you must assume people "know the underlying mechanism", because you want to be open and transparent about it with them. there's really no other option when you're working with volunteers. > Also, what if the second proofer is one of the four mentioned earlier? or what if they both were? > There is a good chance that many of the errors would slip through. right. > It's easy for me to change the site code to force two looks at every page, > and I'll probably do that, perhaps even with a project in progress. doesn't matter. even after two forced looks, some errors will remain. > A down side to the "every page looked at by at least two proofers" > approach is specific to fadedpage: that there are only a dozen or so > active proofers of the 60 or so registered users. The double-look > algorithm adds about 35% to the number of page looks on a project. doesn't matter. there's no need for any haste on the books coming out... > A better solution that just a double-look is > to actually instantiate Confidence in Proofer (CiP). i was afraid you were gonna say that. and it's absolutely the wrong approach. > For these four proofers, the system could schedule a second look at > their pages even if they check the "this is done" box when done proofing. > It would give them plenty of diffs to look at, and they would be > expected to look at those diffs that show some correction was made. well, it'd be better just to inform them and educate them in the first place, rather than impose an "expectation" on them that informs them (indirectly) and forces them to educate themselves (again, in a very indirect fashion)... > If diffs were not checked, then their access to new pages > would be reduced. The kind of proofer who checks diffs, > learns, and continues to contribute is exactly what is needed. well, yeah, maybe... but you're assuming a real luxury of an overabundance of volunteers, and a willingness to throw a good number of them away as "not exactly what is needed". it's better to figure out how to find a use for _all_ volunteers. > I believe for a roundless system to work, there has to be > a reliable mechanism for stopping a page as done. d'oh. there has been complete agreement that that is the issue from day 1. > I also believe that to have a reliable way to make that determination, > some form of Confidence in Proofer needs to be in place. some people have held that belief, yes. i think it's unobtainable, and wrongheaded, and basically a dead end. even if you get a rudimentary version, it won't turn out to be useful... > Therefore, CiP, which is important, and page tweets, which are > useful and fun, are currently my main coding efforts at fadedpage. yeah, well, you'll be coming back sometime down the line and saying "those who wish to say 'i told you so' can chime in now, rightfully"... the thread has more, on confidence-in-proofer, but i'm not gonna waste any more of my time dealing with that flawed concept... -bowerbird

13 41

Re: Bowerbird's software projects
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

i've uploaded the archive.org o.c.r. for "the advocate" -- the book that gardner suggested that i look at -- into the skeleton that was previously being used, at: > http://z-m-l.com/go/gardn/gardnp123.html the last page of the book shows the changes i made: > http://z-m-l.com/go/gardn/gardnp126.html the .zml that made these .html pages, as usual, is at: > http://z-m-l.com/go/gardn/gardn.zml if you compare the .zml file to the original o.c.r., you can see that it is very similar. it doesn't take much work to massage typical o.c.r. output to .zml. as no proofing has been done yet, the text is raw... (although this book came from the internet archive, it is a copy of a google book, which means the o.c.r. is very shoddy, since google puts out low-res scans. when archive.org does o.c.r. on its own scan-sets, the o.c.r. is fairly good, since they're using abbyy.) ordinarily at this point, in order to clean the o.c.r., i'd restore the linebreaks to gardner's p.g. e-text by using the linebreaks from the o.c.r. as a guide... however, gardner sent me a copy of the text as it was _before_ he rewrapped the original linebreaks, so i won't need to go through that boring exercise. i decided to post this o.c.r. text anyway, just so you could see what it would look like as it is "in process". at this point, the structure of the thing is pretty solid, in the sense that all of the paragraphing is correct, and the chapter-heads are in place, and all of that, it's just the scanning errors make the thing awful... but if you look past those scanning errors, you can see why this version is superior to the p.g. e-text: it is obviously self-validating against the p-book from which the text claims to have been generated, since it is easily compared to scans of that p-book. (or even against an actual hard-copy, if you prefer.) even if the p.g. text _is_ accurate, it can't be _verified_ as accurate, not quickly and easily, like this text can... -bowerbird

1 0

ok, let's take a look at gardner's book, just for the exercise
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

i said: > no, in such situations, i'll download the o.c.r. from archive.org, > which _does_ still contain pagebreak info, and unwrapped text, > and end-line hyphenates. and then i will use your proofed text > to make the corrections to the archive.org o.c.r. and then i will > throw your text away, and keep the corrected, unwrapped and > page-marked text with the original end-line hyphenates in it... it occurs to me that it would be quite instructive to demo this exercise. i'll be using gardner's book to show how i'd go through this process. to prep, i downloaded the scans for his book from canadia.com, and mounted them on my website, along with a skeleton copy of the text. here's a sample url: > http://z-m-l.com/go/gardn/gardnp123.html as you can see, the prefix for this book is "gardn", so if you put a copy of the banana-cream program in a folder of its own, and name it "gardn.exe", it will download the .zml file and the scans from the website, and you'll be able to see how to start to use it. -bowerbird p.s. mac users should name the app "gardn.app", of course... (or, since your .app extensions are likely hidden, just "gardn".)

2 1

Re: Bowerbird's software projects
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

gardner said: > That's kind of my experience, I guess. > Several fixes will suggest themselves, > in the context of a given specific text. > The next one might need different fixes. right. a rule i always follow is that when i find an error, i always search the rest of the text for other occurrences. > But that doesn't mean a long list of fixups > might be tried when there's no cost > to just adding tests/fixes to the list. well, except for what i mentioned about false alarms. it's a rare test that doesn't turn up any false alarms, but when one turns up too many of them, it becomes a liability instead of an asset. the question is always, "how many false alarms is too many?", and the next question is always, "how can i weed out false alarms?" > Better stick to Windoze, if it's a GUI. actually, they are both generated from the same code, so they should act identically. whether they really do... > Text in one file -- check. > I favour marking page boundaries with "===00123" > these days, but a global search/replace can fix that. my app is looking for separator lines that look like this: {{myantp123.png}} || the_runhead || (note that there is a space in the first column.) that .png filename there is the name of the page-scan, and the program assumes you name your files wisely... so, for instance, if you want to jump to a certain page, you simply type the page number and press enter, and the program automatically jumps to that page. nifty... > Yes. Looking at that. I am not 100% sure > I want to mess with Twister exactly, but > the list of regular expressions looks interesting. > I'm picturing building a perl script that > applies all of these fixes, then creates a patch set > based on the the differences it has introduced. > I could then edit the patch set as a file, > nuking changes that are wrong, and > finally apply the patches for the changes I like. i would be very surprised if you can make that workflow more efficient than simply editing text in the interface... the beauty of my app, and twister too, is that you can view the page-scan to help you make the edit decision. i'm well aware that you don't _need_ to view the scan in order to resolve the vast majority of questions, but the inefficiency in handling that thin minority is huge if the bureaucracy of viewing the scan is too convoluted. > Jeebies and gutcheck reference specific line numbers. try twister. seriously. the ability to jump right to the page where the question occurs, and view the scan in context, is a major boost to efficiency. i bet you will be surprised... > I find it takes a good couple of passes before > I am satisfied I have all the genuine hits covered. > Invariably the WW finds things I've missed anyhow. that's a sign of an inefficient workflow. you want to accomplish things in one pass, and you want to make sure you got all of it. > Got those. Like I say -- I will turn it into > a perl script and see where that takes me. a perl script is operating blind. get a seeing-eye dog. > Lots of choices there. > http://www.canadiana.org/ECO/ItemRecord/48293?id=16c79d4f15394e51 > http://www.archive.org/details/advocateanovel00heavgoog > http://books.google.com/books?id=ot4OAAAAYAAJ&oe=UTF-8 none of those options are all that useful to me, however... what i need is to have the individual scans available online, each of them individually addressed with their own address. for instance, that "myantp123.png" file i referenced above, the one that reflects the scan of page 123 in "my antonia"? you can find that right here, in sequence with all the rest: > http://z-m-l.com/go/myant/myantp123.png this is the way the library of the future will be organized... if you want your work in it, mount your files appropriately. yes, i can download the .zip file of the scans from archive.org, or pull 'em from the google .pdf, and then mount them myself, but that's too much work for me to do, when you could have mounted them correctly in the first place. > There are no page numbers in the Gutenberg text though. then you threw away some very crucial information, didn't you? probably rewrapped the text too, am i right? and dehyphenated? all these actions make any kind of reproofing an impossible task. which is not to say that your proofing work was a waste of time. no, in such situations, i'll download the o.c.r. from archive.org, which _does_ still contain pagebreak info, and unwrapped text, and end-line hyphenates. and then i will use your proofed text to make the corrections to the archive.org o.c.r. and then i will throw your text away, and keep the corrected, unwrapped and page-marked text with the original end-line hyphenates in it... and when i throw away your text, i throw away your credit-line. had you kept all that valuable information which i need to have, instead of tossing it out, i probably would keep your credit-line. you know, just so you know... -bowerbird

2 1

oops
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

crap. i meant to send that only to gardner and joey, not to the list. i don't care who gets the app, but i _would_ like to know who. so if you download it, please send me a backchannel saying so. (and if you download it more than once, let me know that too.) if there are unaccounted downloads, i'll have to delete the file, because i don't want too many copies out in the wild right now, beings that this is not meant as a finished copy in any respect... and since the cat is out of the bag, here's the mac version: > http://z-m-l.com/go/bananacream/bc2008f.app.zip -bowerbird

1 0

banana cream
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

hey guys- here's where you can get a windows version of banana cream: > http://z-m-l.com/go/bananacream/banana-cream2008.exe you'll also want to download this sample text: > http://z-m-l.com/go/bananacream/tjbus.zml put these two files in a folder of their own, and then rename the program to "tjbus.exe"... (by default, it loads the .zml file with the same name.) the program will then download the scans for that book ("the jungle", by upton sinclair) automatically from my site. (you can control the download method under one of the menus.) if you have any questions, let me know. i'd prefer to have the chance to address any complaints before you go public, but i have no desire for you to muzzle your truth however you see it. as stated earlier, i'd prefer you not distribute the app. thanks. -bowerbird p.s. the app follows my naming conventions, which require a 5-letter prefix at the start of each filename (e.g., "tjbus"), followed by a single letter declaring the type of page (either "c" for a cover, or "f" for forward matter, or "p" for a page), followed by the page-number (padded out to three places). if the page was unnumbered, you use the last page number, and append an "a", "b", "c", "d", etc., respectively, as needed. it's also the case that the .zml file needs a certain structure. i'll write something up for that, and send it along to you later.

1 0