- gutvol-d - lists.pglaf.org

Re: Bowerbird's software projects
by Bowerbird＠aol.com 20 Feb '10

20 Feb '10

gardner said: > That's kind of my experience, I guess. > Several fixes will suggest themselves, > in the context of a given specific text. > The next one might need different fixes. right. a rule i always follow is that when i find an error, i always search the rest of the text for other occurrences. > But that doesn't mean a long list of fixups > might be tried when there's no cost > to just adding tests/fixes to the list. well, except for what i mentioned about false alarms. it's a rare test that doesn't turn up any false alarms, but when one turns up too many of them, it becomes a liability instead of an asset. the question is always, "how many false alarms is too many?", and the next question is always, "how can i weed out false alarms?" > Better stick to Windoze, if it's a GUI. actually, they are both generated from the same code, so they should act identically. whether they really do... > Text in one file -- check. > I favour marking page boundaries with "===00123" > these days, but a global search/replace can fix that. my app is looking for separator lines that look like this: {{myantp123.png}} || the_runhead || (note that there is a space in the first column.) that .png filename there is the name of the page-scan, and the program assumes you name your files wisely... so, for instance, if you want to jump to a certain page, you simply type the page number and press enter, and the program automatically jumps to that page. nifty... > Yes. Looking at that. I am not 100% sure > I want to mess with Twister exactly, but > the list of regular expressions looks interesting. > I'm picturing building a perl script that > applies all of these fixes, then creates a patch set > based on the the differences it has introduced. > I could then edit the patch set as a file, > nuking changes that are wrong, and > finally apply the patches for the changes I like. i would be very surprised if you can make that workflow more efficient than simply editing text in the interface... the beauty of my app, and twister too, is that you can view the page-scan to help you make the edit decision. i'm well aware that you don't _need_ to view the scan in order to resolve the vast majority of questions, but the inefficiency in handling that thin minority is huge if the bureaucracy of viewing the scan is too convoluted. > Jeebies and gutcheck reference specific line numbers. try twister. seriously. the ability to jump right to the page where the question occurs, and view the scan in context, is a major boost to efficiency. i bet you will be surprised... > I find it takes a good couple of passes before > I am satisfied I have all the genuine hits covered. > Invariably the WW finds things I've missed anyhow. that's a sign of an inefficient workflow. you want to accomplish things in one pass, and you want to make sure you got all of it. > Got those. Like I say -- I will turn it into > a perl script and see where that takes me. a perl script is operating blind. get a seeing-eye dog. > Lots of choices there. > http://www.canadiana.org/ECO/ItemRecord/48293?id=16c79d4f15394e51 > http://www.archive.org/details/advocateanovel00heavgoog > http://books.google.com/books?id=ot4OAAAAYAAJ&oe=UTF-8 none of those options are all that useful to me, however... what i need is to have the individual scans available online, each of them individually addressed with their own address. for instance, that "myantp123.png" file i referenced above, the one that reflects the scan of page 123 in "my antonia"? you can find that right here, in sequence with all the rest: > http://z-m-l.com/go/myant/myantp123.png this is the way the library of the future will be organized... if you want your work in it, mount your files appropriately. yes, i can download the .zip file of the scans from archive.org, or pull 'em from the google .pdf, and then mount them myself, but that's too much work for me to do, when you could have mounted them correctly in the first place. > There are no page numbers in the Gutenberg text though. then you threw away some very crucial information, didn't you? probably rewrapped the text too, am i right? and dehyphenated? all these actions make any kind of reproofing an impossible task. which is not to say that your proofing work was a waste of time. no, in such situations, i'll download the o.c.r. from archive.org, which _does_ still contain pagebreak info, and unwrapped text, and end-line hyphenates. and then i will use your proofed text to make the corrections to the archive.org o.c.r. and then i will throw your text away, and keep the corrected, unwrapped and page-marked text with the original end-line hyphenates in it... and when i throw away your text, i throw away your credit-line. had you kept all that valuable information which i need to have, instead of tossing it out, i probably would keep your credit-line. you know, just so you know... -bowerbird

2 1

oops
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

crap. i meant to send that only to gardner and joey, not to the list. i don't care who gets the app, but i _would_ like to know who. so if you download it, please send me a backchannel saying so. (and if you download it more than once, let me know that too.) if there are unaccounted downloads, i'll have to delete the file, because i don't want too many copies out in the wild right now, beings that this is not meant as a finished copy in any respect... and since the cat is out of the bag, here's the mac version: > http://z-m-l.com/go/bananacream/bc2008f.app.zip -bowerbird

1 0

banana cream
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

hey guys- here's where you can get a windows version of banana cream: > http://z-m-l.com/go/bananacream/banana-cream2008.exe you'll also want to download this sample text: > http://z-m-l.com/go/bananacream/tjbus.zml put these two files in a folder of their own, and then rename the program to "tjbus.exe"... (by default, it loads the .zml file with the same name.) the program will then download the scans for that book ("the jungle", by upton sinclair) automatically from my site. (you can control the download method under one of the menus.) if you have any questions, let me know. i'd prefer to have the chance to address any complaints before you go public, but i have no desire for you to muzzle your truth however you see it. as stated earlier, i'd prefer you not distribute the app. thanks. -bowerbird p.s. the app follows my naming conventions, which require a 5-letter prefix at the start of each filename (e.g., "tjbus"), followed by a single letter declaring the type of page (either "c" for a cover, or "f" for forward matter, or "p" for a page), followed by the page-number (padded out to three places). if the page was unnumbered, you use the last page number, and append an "a", "b", "c", "d", etc., respectively, as needed. it's also the case that the .zml file needs a certain structure. i'll write something up for that, and send it along to you later.

1 0

banana cream for mac and windows is available
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

banana cream for mac and windows is available. the linux version seems to have a plug-in conflict, which i do not feel like debugging right now, so linux people can use the windows version, or wait. (note: i also have a version for the classic mac o.s.) if you want to be a tester for this nice little app, let me know and i'll tell you how you can get it... this is the version from the fall of 2008, which is the last time i worked on it in earnest, and i can't remember what kind of state i left it in, so there might well be some rough unfinished edges in it; but for the most part, it should be pretty smooth. like i said, though, it ain't pretty. ain't pretty at all. ;+) -bowerbird p.s. if people want to use it, i'd be happy to work on it again, to include incorporating all of those reg-ex tests that gardner was requesting. but... as only gardner and joey (yes joey, i did hear you) have asked for a copy so far, that seems unlikely...

1 0

[SPAM] re: Re: Bowerbird's software projects
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

gardner said: > Perhaps not, but over time you have described > checks that your tools can do and > fixes that you can automatically make > that sound a little to me like a super-duper gutcheck. yes, except that those checks and fixes are most often programmed only into one-off versions of programs... it's usually the case that making those checks and fixes useful in the general case, against any random book, is a more difficult matter. this isn't an apology of any sort; it's just that my intentions (for the most part) are to show that a particular check can be accomplished, and is useful. so far i haven't concentrated on building them into my app, because nobody's really expressed much interest in the app. the app has a general spellcheck ability, and that captures a very high percentage of the errors that occur within a text. > Also the workflow I picture is a little like gutcheck -- > I am thinking of text-in text-out command line tools, > not something that needs to look at image scans or > makes me talk to it in a fancy U/I. i'm not sure what you mean by the workflow you "picture". i was asking about your _actual_ workflow, the one whereby you currently digitize books. are you saying that you now do your digitizations without ever looking at image scans? because i have a hard time imagining how you can do that. you should also know i am a mac person, for good reason. for me, the interface is prime. if you're looking for tools that work on a command-line, in a text-in-text-out way, i'm the wrong tree for you to be barking up, that's for sure. i certainly wouldn't call my interface "fancy". to the contrary, it's extremely utilitarian, and not very pretty, not pretty at all. but it _is_ an interface, with buttons and menus and all that nice stuff that makes the program a lot easier to work with... > a Linux build would, I think. Windows would be fine too. i'll send you both. > This is still fairly accurate: ok, that was very useful... my tool assumes that the page-scans are in the same folder as the app, which is easy enough to satisfy. the tool also assumes that your text is all in one file, and that the page-boundary is of a certain type. i'd assume that your vi skills will enable you to satisfy this assumption in a fairly simple manner. other than that, i'd say you'll be good to go. > The last couple of books I've done instead by scanning, > bulk OCR and then proof from the scans and raw OCR text > which I can do on the road with my laptop or > anywhere I can mount a USB key for a couple of hours. that's how you'll want to operate with my software, yes. > After OCR I have a few basic things that I do > via regular expressions in vi: you can continue to do those things in vi if you like. global changes in vi are much quicker than going through one-by-one changes in the interface. > The thing is that I do not have > a specific set of checks and fixes that I consistently do. that's something you'll want to remedy. i did a series here a couple years back where i collected a list of checks that was necessary for the book i tested, and somebody turned that list into a set of reg-ex tests. you can find that set on the download page for don's app: > http://code.google.com/p/dp50/downloads/list indeed, since you are already using reg-ex, you'll probably find that you prefer don's tool over mine, since his program lets you actually _build_in_ your own list of reg-ex checks... > I rely a lot on jeebies and gutcheck. so, when you get a report from them on the possible errors, you enter vi and use search to locate each one of the errors? > I would like something perhaps with > a wider range of things that it can find > so I don't have to know all the things to look for. well, yes. and you can find some very extensive lists of reg-ex checks, right on d.p. the problem is that many of those checks have a low signal-to-noise ratio, in that they create far too many false-alarms. this is a problem even with gutcheck and heebe-jeebe, if i'm not mistaken. so you really have to fine-tune your list of checks to the particular corpus on which you are working, to be useful. this is why don's app is so useful, because you can build in the list of checks you want to do, and modify it at will, and even enter in a specific reg-ex to see if it returns any hits. > Over the years you have mentioned > several automated checks and fixes > that sounded sensible enough to me. sounds like you really want to use that reg-ex list that was based on the month-long series that i did. > I'm not keen enough to go back through the archives, > find them and implement them -- but I am > nevertheless interested in trying a tool like this out > on a project to see if it adds value for what I do. having heard all this, i'd guess don's app is the one for you. > http://code.google.com/p/dp50/downloads/list i'll send mine to you too, but his is based on reg-ex checks... > http://www.gutenberg.org/dirs/3/1/2/1/31212/31212-8.txt > and just tell me what you find. I have no doubt there is lots if the scans are online too, or can be, i'll certainly take a look at it... without looking at them, i can't know if something is an error or not. -bowerbird

4 3

Re: Bowerbird's software projects
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

gardner said: > Where was that Sourceforge project again? there is no sourceforge project. my source code has never been open-source. my enemies will need to write their own code. > I know you've talked about tools that > do more/better checking than Gutcheck actually, i don't think i've ever compared my stuff with any other software directly, because any tool is better than no tool. gutcheck has some charms. my tools do different things, and do things differently, but whether they do "more" or "better" is not an issue. > and have automated fixing and such. some have, yes. it's also important to remember that i do lots of experimentation, with quick-and-dirty code that serves to test the usefulness of a particular feature, but which might never be implemented further, perhaps because it doesn't prove to be worthy, or because the generalized code would take more time than i can give, or simply because that task just hasn't been done yet... > I would like to try them out. > Where can get get my hands on this stuff? i'll be happy to send you a copy, gardner, since you are an independent producer -- that's my target sweetspot. until i release the program generally, which might be very soon but also might not, you'll have to agree not to distribute the app any further, since i want to know who has it so that i can engage them in dialog about it, but that's the only restriction at this point. you'll also need to tell me what version you want -- mac or p.c. or linux. your signature-block screams out linux, which is fine, but you'd be one of my first linux users, so if you want the more-well-tested windows version, say so. finally, please give a short description -- frontchannel -- of your _current_ workflow. how do you do your books? do you use an editor, or some other tool? use gutcheck? what kind of preprocessing do you do on the raw o.c.r.? if you need to view a scan to check the text on some page, how do you do that? how do you find errors, with reg-ex?, or via a word-by-word proof of every page? anything else? if anyone else wants to get a copy of my program, say so, either frontchannel or back. the same conditions apply... also, if you're interested, you should check out don's app: > http://code.google.com/p/dp50/downloads/list his tool is similar in many ways, and you might like it too. -bowerbird

3 2

roundlessness -- 005
by Bowerbird＠aol.com 19 Feb '10

19 Feb '10

we're looking at rfrank's "roundless" experiment at fadedpage.com... *** i've been looking at the data for one of the books rfrank used... the book is titled "eagles of the sky", and it runs about 240 pages. in total, there were about 500 lines that were changed by proofers. here's a list of roughly 300 of those lines: > http://z-m-l.com/go/frabf/frabf300diffs.html the list shows the original line, from o.c.r., and the edited line. there's also a link to each page-scan, if you want to view that... i could've included all 500 changed lines in this list, except that the 200 lines i've excluded contained errors that _should_have_ (most definitely) been fixed in preprocessing, and it burns me up to be a witness to such a tremendous waste of proofer resources. it's just a crime against the generous contribution of the proofers to put such shoddy text in front of them and expect them to fix it. roger frank may think i'm talking shit about him again, but i tell you, i'll talk shit about _any_ producer who gives shoddy text to proofers, and, on top of that, i will feel _justified_ and _moral_ about doing it. i mean, it's not exactly a _mystery_ how to do good preprocessing. i have spent a lot of time writing posts here documenting _exactly_ how to do good preprocessing, and i did a heckuva lot of research to learn and test those preprocessing methods to prove their utility. so when someone just _ignores_ what i've done, and continues to burn out proofers by wasting their time and energy with material that should've had the obvious-and-easy-to-detect-automatically errors fixed before it went in front of 'em, i have a right to be mad. someone needs to talk some sense into roger's head, and do it fast. not to say that roger is the only one, or the worst one. not by far. i don't even bother looking at what the p.g. content producers are giving to their proofers these days, to protect my blood pressure. but since the odds that anything has changed over there are none and none, someone should talk some sense into _their_ heads too. it's a _shame_ to be wasting proofer time, and you producers who fail to do good preprocessing should be _ashamed_ of yourselves. *** at any rate, if that list of 300 changes is too dense for your brain, you can also thumb through the pages and see the changes made: > http://z-m-l.com/go/frabf/frabfp123.html all changed lines are marked in red (o.c.r.) and blue (edits), so if any particular page (like page 123, linked to above) is all-in-black, it means that none of the lines on that page had any changes made. you will find this "stepping through the edits" to be very friendly... -bowerbird

1 0

Re: "The Inheritance" by Susan Edmondstone Ferrier
by Bowerbird＠aol.com 18 Feb '10

18 Feb '10

al said: > I'm not aware of any general PG practice to send newcomers to DP. i'm not surprised to learn that you're not paying any attention, al. > Your motivation? yeah, you know, the reason why i would spend time and energy on this. > What do I care? why did you suggest i do this book? > Be altruistic, and do a book. i'm altruistic just by being here, sharing my analyses with people, al. i'm altruistic when i _do_ the research that _leads_ to those analyses. i'm altruistic when i design and program tools that do what's required, because that proves that such tools can indeed be designed and coded, and it lets me be precise and experienced when i assess their value... i'm altruistic when i work up my suggestions for an improved workflow. i'm altruistic in a number of ways, al. to "do a book" seems unnecessary, unless you're intentionally suggesting that i should lower my sights a lot. i clean text for the fun of it, al, just like my girlfriend does her sudoku. so i don't need a _lot_ of motivation... but i certainly do need _some_... you've given me no good reason to work on the book you've suggested, al, so i'm not sure why you even bothered to mention my name at all... > If you want something challenging in the exact same way that _i_ will decide how i will be altruistic (or even _if_ i will be altruistic), i will also be the one who decides what is "challenging" to me. but, like, thanks for your suggestion, and have a nice day, ok? -bowerbird

2 1

obsoleting etext #1250?
by Peter Saint-Andre 18 Feb '10

18 Feb '10

[resending to gutvol-d after mail to subs(a)pglaf.org bounced] Project Gutenberg etext #1250 (Ayn Rand's novel Anthem) contains a fair number of errors, which I realize is not unusual for early PG texts. Last year I completed a thorough check of all published editions of Anthem in an effort to produce a definitive text. Although I have not yet had an opportunity to check the galley proofs of the 1946 edition housed in the Library of Congress (which I have reason to believe will answer several nagging questions about the author's intentions), as far as I know the text I have produced is the most accurate version on the Internet. Would Project Gutenberg be interested in accepting the text I have produced? You can find an XHTML version here: http://www.monadnock.net/rand/anthem.html For your convenience I have moved my textual notes to a separate page: http://www.monadnock.net/rand/anthem-notes.html Thanks for Project Gutenberg! Peter -- Peter Saint-Andre https://stpeter.im/

1 0

"The Inheritance" by Susan Edmondstone Ferrier
by Al Haines (shaw) 17 Feb '10

17 Feb '10

If anyone's looking for a project, look no further than the above. Internet Archive has assorted editions, none of which are projects in DP. All editions appear to be clearable under Rule1 (pre-1923). Hmmm... maybe bowerbird is up for them? Al

1 0