January 2012 - gutvol-d - lists.pglaf.org

Re: [gutvol-d] book of james -- 008
by Bowerbird＠aol.com 09 Jan '12

09 Jan '12

james said: > I will set this project aside nope, no time for that... :+) here's the next step, a list of some of the words that didn't pass the spellcheck. (might not be all of them, as i'm not certain i've handled all the encoding issues, but these are definitely some words that will not pass.) > http://zenmagiclove.com/bhaga/flag-by-pages.html this list of flagged words is ordered by _the_page_... first of all, as an overview, i don't know about you, but when _i_ see a list like that, i begin to get a feeling that i am the master of the o.c.r., rather than a victim of it... i'm no longer on a "hunt" for the errors, i have a _map_, so i know exactly where to find them and apply the fix... or, for some cases, to check them once (and for all) to make sure the name is spelled correctly, so i can ignore other cases (if it is) or do a global correction (if it's not). that's where this particular list gives us another edge, namely that it has automatically culled a lot of names. it won't always get them all, especially in a book with american names, some of which are "good" words, like "will smith", "may brown", "mike cane", or "brook springs". but on a book like this one, it does a _very_ good job... and there's an added benefit on this book, because many of those names have diacritical marks on them, so being able to pull them out like this means that we can easily generate some global changes to fix them... so that will be one of your next jobs, james, to prepare such a list of global changes to the names listed there. the matter of encodings also brings up another issue... at the top of this page of flags, there's a little "v?" link... you will see this on a lot of my pages, at top or bottom. if you click it, it will send the page to the "validator" site -- in a separate window -- to see if the page "validates". it's not that i care that much about "validation" (i don't), it's just that having a link like this one makes it painless. anyway, if you click the validator link on this page, james, you'll see that it generates an error and a couple warnings. the error is for a naked ampersand i'd "failed" to escape... (i did the others, but i left this one in, for an illustration.) the first warning is because it's html5, so you can ignore it. but the second warning is a troublesome one, which says: > Text run is not in Unicode Normalization Form C. i don't know what that means, and i don't particularly care to have to track it down, so if you can figure it out, james, that would be dandy. the problem also occurs in your file, the one in your dropbox, so i think it's likely on your end. but there might also be parts of my workflow that are plain oblivious to utf8 issues, so i can't rule out that possibility. but it probably has something to do with the way that you generate some utf8 characters, e.g., by "composing" them. i'm so ignorant of these issues that i don't even know when these characters _look_ right or wrong, so i can't help much. but if you can get your text to go through the wc3 validator without generating a warning, that would be a good thing, and i will make sure that my apps don't screw anything up. -bowerbird

2 1

Re: [gutvol-d] book of james -- 007
by Bowerbird＠aol.com 09 Jan '12

09 Jan '12

james said: > I'm not sure what you mean by try it now... > http://zenmagiclove.com/bhaga/bhaga-2edit.py there are some new buttons, at the top of the scan, which allow you to make the text smaller or bigger, and also to make the scan wider or more narrow... you can also, as before, resize the scan more crudely by adjusting the size of your window of your browser, and resize the text by adjusting the "text zoom" menu. that should allow you, james (and everyone else too), to customize the display so that it will work for you... if not, let me know... -bowerbird p.s. changes are retained during the life of the session, but will need to be redone at the start of a new session, because i don't wanna be storing any persistent cookies.

2 1

Re: [gutvol-d] book of james -- 007
by Bowerbird＠aol.com 09 Jan '12

09 Jan '12

james said: > My two monitors at home have > a maximum resolution of > 1440x900 and 1360x768 respectively. that's good to know. i can plan for that. > I would guess that very few of us have > monitors of high enough resolution that > we could view a page image without scrolling > and have that page image be readable. i'm lucky to have a very big screen, and i know that makes me out-of-touch with many people; that's why i try to make sure to ask these things. perhaps you could send me a screenshot where you have arranged things to be optimal for you? another thing to keep in mind is that you won't actually be _reading_ the scans, not per se, but only referring to them as your reference-point. i know it's tough to let that go, mentally, but do it! when the time comes for a word-by-word proofing, the arrangements can be different. but that's later. > your software messes up some UTF-8 characters. > (If you can't read what I typed, it was a capital'S' > with an accent acute above it). um, that was an intentional change that i made, because something is getting screwed up in utf8. i haven't done enough sleuthing to know whether it's on your end or mine, but it ain't validating... (well, it only gives a warning, not an error, but...) but we'll get to that later... > Third, your "smaller" button does not work. > It seems to advance the page instead. oops! i'll fix that. in the meantime, the number in the small box between "smaller" and "fetch" is the text-size-percentage, change it and then press "fetch" and find the optimum value for you. > Fourth, your "good fit" example shows > what many Hindus would consider to be > the most important page in the whole book. i didn't know that. > You must be a very old soul. i've been told that, a good many times. -bowerbird

2 1

book of james -- 007
by Bowerbird＠aol.com 08 Jan '12

08 Jan '12

ok, james, let's begin the process... :+) i will attempt to reintroduce the correct linebreaks tomorrow, but you can do some stuff meanwhile... *** first, here's a draft version of the editor: > http://zenmagiclove.com/bhaga/bhaga-2edit.py you can't do any editing in it yet, but that's the point. the first thing you'll do with a page is to _look_at_it_, to see if the formatted text looks like the page-scan... if it _doesn't_, then the first type of editing you will do is change the text so that it _will_ look "like it should". (once you can edit, that is, not now; just look for now.) once the formatted text _looks_like_ the page-scan, you will do the proofing. this is the reverse of what distributed proofreaders does, because (quite frankly) they're doing it wrong... in my most humble opinion. *** so we first need to ensure the interface "works right" on whatever machine(s) you'll be using to do editing. so take a look at it, using such machine(s)... the editor shows the text on the left, scan on the right. and, for now, it's showing both my text and your text, with mine at the top because mine is formatted better, as the formatting is our specific focus at the moment. eventually, my text and yours will be merged into one. you'll adjust the scan by resizing the browser-window. start with a window the full-width of the screen, then narrow it until the entire page-scan comes into view... you want to have a window which is as _big_ as possible, so everything is easier to read, but also you wanna have everything in the window, all the text and the whole scan, and you'll want all of the text lines to remain unwrapped. here's an example of a good fit: > zenmagiclove.com/bhaga/good-fit-interface.jpg if the lines wrap inappropriately, or are just too narrow, you must make the text smaller or bigger, respectively. you can do this by clicking the "smaller/bigger" buttons you will find at the bottom of the text. that resizes text by an increment of 2%, so it might take you a few clicks. you can also resize the text by zooming it (under "view", in most browsers, although it'll differ on some of 'em)... use whichever seems to work best for you, and gives you a view includes the entire page of text and the full scan. tell me if it works for you. and if not, save the source, and see if you can fiddle with the .css and make it fit... *** then, once you've got a good fit, give it a good test -- run through a few hundred pages... you proceed forward one page by clicking the scan... there are some pages -- like the tables and trees -- which i didn't even attempt to format, but the rest of them _should_ look fairly accurate. tell me about any which do not appear to be similar to the page-scan... because the paragraphs are indented, and headings are centered and bigger, and the font is proportional, it's easy to tell if the text is being formatted correctly. the paragraphs on both sides should even align nicely. this will make it easier to check any proofing questions, which will be the next step, once the linebreaks are ok... *** you can also start to work on the text... here's a list of the "good words" in the text, the ones which are found in the dictionary, and thus the ones which we'll use as this book's dictionary from now on. > http://zenmagiclove.com/bhaga/good-unique.html what you'll want to do is run these through a spellcheck. they _should_ all pass -- since they passed mine! -- but you might be surprised... but then you will understand... the other thing you _can_ do with the list is to go over it. just because a word passes spellcheck does _not_ mean that it's the _correct_ word. it could be a stealth scanno. so that's what you'd be looking for, words that might be "in the dictionary", but which _don't_belong_ in this book. the example often given nowadays is the word "modem". they didn't have modems when this book was published, so that would almost certainly be a scanno for "modern". again, this check is only if you _want_ to do it. there are a lot of words in this text -- the book has 450 pages -- and it'd be a big job to look closely at every single one... plus the odds of catching a glitch are really very small... (but i found a few "possible" ones -- "para" and "tanks"; go ahead and search your text to see if they're scannos.) so, all in all, this task has a low return of benefit to cost. nonetheless, i often enjoy looking over the "good words", because it gives me a feel for the vocabulary of the book, (it's a bit similar to reviewing the index of a paper-book.) *** next up, we'll look at the _flagged_ words -- ones that were _not_ found in the dictionary -- for more action... -bowerbird

2 1

book of james -- 006
by Bowerbird＠aol.com 06 Jan '12

06 Jan '12

james- i'm on the run, so no time to respond to your post, but here's the results of my efforts to reinstall pagebreaks: > http://zenmagiclove.com/bhaga/bhaga-2view.html not perfect, but not bad either. it will at least get you back to the state where you can confidently look at a scan and the text for that page... the next step would be to reinsert the correct linebreaks. from that point in, you'd be back to the normal situation. of course, "normal" for this book _still_ lacks the italics, and the em-dashes, and the utf8 characters, and _some_ of that _might_ be available in the archive.org abbyy.gz, i haven't looked. but if you went to grab _that_ version, you'd be starting from scratch. so either way is a pain... most of the _layout_ in my version is close to correct now. more later... -bowerbird p.s. don't start a new project without talking to me first!

2 1

book of james -- 005
by Bowerbird＠aol.com 06 Jan '12

06 Jan '12

so, james, i'm finally getting around to trying to figure out what to do about you and your book, and whether i can help you. so i did a compare between a file of the book from december 22nd and just the other day... and i see some interesting stuff... :+) i expected to find you'd done a few more pages, with everything before them basically the same, and everything after them still left untouched... but what i actually found is that you have been doing some book-wide global-change editing, including a mass fix of spacey doublequotes, and some global-change name replacements. looks like you also did a hunt for em-dashes... well done, my friend. i'm sure it was all your idea, and a good idea it was, sir! i applaud your accomplishments... anyway... i think what i would need to do is to _rewrap_ the file back to its original p-book linebreaks, and reintroduce indicators for the pagebreaks. that's what it would take to put it in my system. i'm willing to _try_ to do those things, to see if it's relatively _easy_ to do them, and -- if so -- (but there are no guarantees!), then i would be able to build a system for you. will you use it? if not, i won't bother. but if so, i'll give it a try. -bowerbird

3 5

selling your own dogfood
by Bowerbird＠aol.com 06 Jan '12

06 Jan '12

once again, someone over at distributed proofreaders has raised the issue of people selling hard-copy of p.g. e-texts at amazon.com... so a brand-new newcomer, "exilefromgroggs", posted this: > I would suggest an alternative strategy. > Set PG up as an Amazon seller (yes, seller!). > Charge $1/£1 per book for books that are > bought that way, and point out that 100% > of the money that PG receives will be ploughed back > into the project to make further books available. > In the seller's blurb for the book, point out that > the books can, in fact, be downloaded for free > from PG's website, and include a link. I would imagine > that being up-front and open, and being clearly linked > to a "good cause" would mean that PG would rapidly > become a high-profile seller in its own right through > Amazon, and would take the wind out of the sails of > people who are trying to make a fast buck for no effort. because, you understand, that's always the exact way that those d.p. volunteers picture these amazon resellers -- as "people who are trying to make a fast buck for no effort." i fully understand that most of these resellers put as little effort as possible into churning out the "product" they sell. but the truth of the matter is that it takes time and money to turn an e-text into a printed book, and even more work to handle distribution, and the profits aren't _that_ great... but nobody bothered to tell that to the brand new newcomer. because doing so would force the existing volunteers there to confront the fact that their workflow doesn't have a way to create a decent print product. the ascii text is unstyled. the .html product is formatted for a web-browser, not print. the e-book products are formless blobs that barely work in the machines for which they're intended, let alone for print. that's why the vast majority of the resellers start with the text-file, instead of the .html file. so all of the work that post-processors put into making the .html file "look good" is just wasted energy. every one of these html-books is a "snowflake" -- unique unto itself, per the post-processor -- and it's far too much work to try to figure out each one, so you could port it to print. or, for that matter, anything else, including e-books, which is why those turn out rather badly. with a library this big, there has to be some standardization, or else you'll be unable to keep the files updated over time... of course, it's _easy_ to say that now, with all the experience we have from having the books _not_ being _convertable_ to the formats we want. but some of us said it all _years_ ago, that letting each post-processor go off in their own direction was a sure-fire way to make sure their work'd be short-lived. -bowerbird p.s. there is also _another_ newcomer over there who wants to program an html5 web-app that interacts with the d.p. site, and nobody is bothering to tell him not to waste his time and energy, because even if he codes it up, the powers-that-be will ignore it. p.p.s. ...waving back to lucy... :+)

22 99

Re: [gutvol-d] the betty lee fun parade
by Bowerbird＠aol.com 05 Jan '12

05 Jan '12

speaking of "betty lee", my latest version is on my site: > http://z-m-l.com/go/betle/betle.zml that's as far as my "preprocessing" will go on that file... any further edits will be the product of "beta-readers"... the last round of tests i did on this file were regarding _single-quotemarks_. i coded a routine to check them, and was a bit surprised to see exactly what turned up... there were a lot of splotches on the scans of this book. (as i said before, this scan-set isn't up to roger's par.) and, as we know, o.c.r. is duty-bound to report splotches. what that meant, in this file, is that awholelotta splotches got misrecognized as single-quotemarks. dozens of 'em. and bad single-quotemarks don't show up in spellcheck... but they _did_ show up in this test, big and bright as day. so i believe i'm almost perfect on the single-quotemarks. of course, i would be very interested if you find an error, because it _might_ expose a shortcoming in my routine... (on the other 'hand, it just might be two' splotches which happen to _appear_exactly_like_ something "legitimate", like the open-close pair in the top line of this paragraph.) then again, unless you happen across one "by accident", you're probably not gonna find any errors in my results _unless_ you write a routine of your own to check them. so, you know, _be_my_guest_... ;+) -bowerbird

1 0

Re: [gutvol-d] the betty lee fun parade
by Bowerbird＠aol.com 05 Jan '12

05 Jan '12

john said: > betlepythonB.html so i took a look at this revision, john. only it still has the image-references, which are broken, and which interrupt the paragraphs. and nobody will find this acceptable. so i'm a bit confused. this is little improvement over before. so here's some stuff for you, although it's "not soup yet" on my timetable, so i really shouldn't be releasing them yet. here's a script that "depaginates" the file: > http://zenmagiclove.com/depage.py that is, it deletes page-boundary lines and the _correct_ blank lines surrounding 'em, according to the rules i gave you yesterday; this ensures the paragraphs rejoin properly. still in this file are the line-end hyphenates, soft and hard, and the tilda-pairs i discussed. so here's a script which deals with _them_: > http://zenmagiclove.com/unwrap.py again, i have a schedule when these two were to be released, and this is premature, so the results might not be perfect at this time, but they will be more than good enough for you to do whatever you need to do for your demo. -bowerbird

1 0

more fun with betty lee
by Bowerbird＠aol.com 05 Jan '12

05 Jan '12

john, thanks for joining the betty lee fun parade! :+) can we retire the "mount high horse" header, though? it was an honorific for 2011, intended to end an era... i've looked at your work, and have a few notes for you. first of all, those hyphens are not "spurious", they're the hyphens which actually exist in the printed-book. but yes, you want to get rid of them, and here's how... (first i'll tell you that i will make web-apps available to perform these routines for people automatically, so it's not like you really _need_ this documentation, but i happily tell you this information if you want it.) here's how you might wanna rework a .zml text-file... 1. a hyphen at the very end of the line is a "soft" one, meaning it can -- should -- be eliminated on unwrap. you can do this programmatically. in python, it'd be: > thebook=re.sub("-\n","",thebook) but note that you _must_ attend to em-dashes _first_: > thebook=re.sub("--\n","—",thebook) 2. if there's a hyphen-tilde combo at the end of a line, this indicates a _hard_ hyphen that should be retained. the tilde is eliminated. (think of it as "sacrificing itself", with a heroic gesture, so that the hyphen can be saved.) > thebook=re.sub("-~\n","-",thebook) 3. you'll also find cases of a tilde-tilde pair at line-end, or -- to be more accurate -- at the end of a _paragraph_. this tilde-tilde pair indicates a doublequote mark which was _dropped_ because the speaker's dialog _continues_ in the next paragraph. (this will enable you to perform a check on the balancing of doublequotes, but otherwise, the only thing to do with the tilde-tilde pair is delete it.) 4. you'll need to do the other lines to finish an unwrap. all lines should be unwrapped _except_ blank lines and any lines which start with a space as their first character. i do this as a multi-step procedure in my wordprocessor, but i'm sure some reg-ex person can "show us the way". any linebreak with whitespace on either side is retained; all other linebreaks are deleted. shouldn't be that hard. (but if no one spills, i will give you my python routines.) 5. "{{" lines give the filename of the _scan_ for that page. "[[" lines are the pagenumber of the page, printed or not. the "[[" lines can be preceded by up to 3 blank lines, and _all_ of those blank lines must be deleted on an unwrap, as should the "[[" lines themselves (although you _might_ wanna save some kind of reference to the pagenumber). "{{" lines should be deleted as well, of course, and they are always followed by at least _one_ blank line, which must also _always_ be deleted. sometimes there'll be _more_ than one blank line following a "{{" line, _but_ additional lines after the first one _must_ be retained. you'll understand these rules _implicitly_ once you've been told the _reasons_ for them, but that's for later. 6. looks like you figured out everything else you need. i especially like how you handled the letter from rose... the p-book didn't really set it off much, so neither did i, but i generally think that such material should be set off. as criticism, i'd say that your table of contents is skimpy. > The conversions are with a python script, > as you will have guessed. If you want it, > you can have it---Now! you guys should take up john on his open-source offer. > The HTML looks generally acceptable to me, > but the images are missing. the scans are in the same directory as the .zml file. > http://z-m-l.com/go/betle i leave my book subdirectories (under "go") wide open, and name my files intelligently, so it should all be clear. > BB, could you be persuaded to add the HTML to your > treasury, so that we can see the result in context? do you want me to add _my_ .html file to my site? if so, then that will be coming very soon, john, yes. or do you want me to add _your_ .html to my site? i'm willing to do that, if you'd consider it an honor, or something, but i'll assure you that it's really not. your .html version was serviceable, but i believe you could probably do a better job on a conversion now, by using the information which i've given you above. a book this simple isn't really much of a test, however. try working on the test-suite which you can find here: > http://zenmagiclove.com/test-suite-2012.txt -bowerbird

4 4