- gutvol-d - lists.pglaf.org

Re: [gutvol-d] updated PP editor
by Bowerbird＠aol.com 24 Dec '11

24 Dec '11

roger said: > The DHYP button dehyphenates the page > so spellchecking has whole words to check. minor question, for roger... why not have the spellchecker join the hyphenates, solely to do its checking, and leave the text intact? *** major questions, for everyone... can someone give me a 2012 justification for the reductionistic attitude toward the proofing task? why are we still doling out pages one-at-a-time? in 2002, i understand why d.p. was conceptualized in such a fashion... o.c.r. wasn't nearly as reliable, so there were corrections necessary on every page, and each one might demand a chunk of your time. but today, you can clean an entire book in an hour -- in one pass, or in six 10-minute sessions, say -- and send it to the smoothreaders for a final check... so why don't you guys want people to do it that way? smoothreaders are lots easier to recruit from scratch. i've been beating this drum for a long time now, and maybe my own noise has deafened me to hear you, so i'm seriously asking you for an answer and i will promise to do my best to hear what you have to say, because i honestly believe there _is_ no justification. so if you believe you have one, can i hear it please? *** for my side, i scraped roger's text and images, and have already cleaned the book and placed it online, as if for smoothreaders. i didn't even bother to do a final spellcheck, or an exhaustive _italics_ search, so those are two tasks you could do if you want to... i'm quite confident this meets my standard criterion, which is a rate of 1 error or less in every 10 pages... by the way, if you wanted to clean roger's text as well, i've appended a list of questionable things to examine. some of 'em are undoubtedly right, but i'd look at 'em anyway, if i were to seriously take the job of this book. (which is to say i didn't actually examine all of this list.) and there were other errors i'd already fixed before i generated this list, so it won't be an exhaustive list... anyway, here you go... > http://z-m-l.com/go/betle/betlep123.html > http://z-m-l.com/go/betle/betle.zml -bowerbird p.s. list of easily-found possible-errors in "betty lee"... > %ut > *Y > Adeste > Ah's > Ain > And'me > Avalking > afteh > agis > ar.e > arivals > artiele > atternoon > audtorium > Bam > Barnett > Baruett > Bradshaw > Buxton > bing > blackeyed > bootleggeh > boydom > butleh > C > Carlin > Caroline > Chanucey > Chauneey > Chetr > Chistmas > Christinas > Cie > Crede > cheks > cher > childf > couse > cousip > crede > D > Dat > Dey > Dodie > Dorrance > Dorranee > Dorry > de > dem > dere's > dictu > dinneh > doin > doublequick > drew'her > drummajor > E > Einma > Eosy > est > evidentally > Fascisti > Ferris > Fidelis > Finn > fatted > flyin > foh > G > Galilee > Geewhilikers > Glaus > Gr > Grood > Gwynne > git > goin > goodhumoredly > goodnight > guestspolitely > gwine > H'm > Hallowe'en > Hm > Howland > Huxley's > Huxleys > he'worked > hehse'f > high'spirits > hise'f > horribile > hurrried > hurryf > hurryin > ing > io > ioas > ivere > Josie > jus > Kose > LeEoy > Lena > LeRoy > Lovel > Lyon"Y > leftiit > longue > loquor > M > Mary > Maytime > Mm > Monday > mawnin > mayn't > me.v > mihi > miserabile > mon > muddly > mustn't > mysteriouser > Nup > neded > nevah > nosense > nuffin > O > O.K. > Onee > Ophelia > Orme > o > ol > on's > operam > orchestra"'next > oughtn't > Papa's > pail."I'm > palazso > palazzo > peccato > playin > prosecomp > proudj > puritanic > pusson > quaveringly > Rardon > r > re > Savilia > Savilla's > Savillas > Sehna > Selma > Sevilla's > Sevillas > sauve > schooltime > searchingiy > selfconscious > semed > serius > sewin > shound > siicceeded > slangily > slep > somthing > spect > spuzzy > suah > Titania > Tommy > t > t'ook > taMng > tempora > than,the > that.would > the'best > thore > tion > tiredlooking > tnat > togeteher > tol > trubble > ubi > under,his > unecessary > vero > victrola > vocabularly > Well,/remember > Willie > wanta > was'glad > welldressed > weren > wlio > woodchopper's > wouldn > wouldn'd > xnay > Y > Yes'm > year's > yo'se'f > yonr > > -Laura? > as-of > Bim-bang > car-fare? > co-operate > direc-i > distinguished-looking > E-r-r-boom > easy-going > entered,-her > faculty-versusstudent > G-od > G-ood > G-wynne > How-de-do > Howdo-you-do > have-something > how-do-you-do > hymn-book > ice-cream > inter-plays > it-go > junior-boys > keen-looking > Lu-chee-a > leave-takings > Mid-years > mid-years > non-sorority > One-two-three-go > pos-i-tive-ly > read-Cicero > room-and > school-girl > semi-order > senior-girls > Tooral-looral-loo-oo-oo-oo > Toot-toot > Trade-Last > tear-stained > termed-social > think-a > thought-Father > tooral-looral > tooral-looral-loo-oo-oo-oo > tr-ragic > try-out > twelve-thirty > What's-his-name > Ye-ah

3 3

book of james -- 003
by Bowerbird＠aol.com 23 Dec '11

23 Dec '11

james said: > Carlo, > I defined a compose key on my Linux box last night and > it works great. I can finish the book twice as fast now. carlo is handing out the holiday gifts, isn't he? :+) thanks again, carlo, for the djvu-text-extraction info... *** james, i just wanna make sure we're on the same page. (see how i did that?) ;+) i'm cleaning the text now, and i will hand it over to you sometime next week. that's my understanding, anyway. is it your understanding as well? i want to be positively _certain_ that we do _not_ have the situation where we're working on separate tracks, because merging 2 sets of corrections is a nightmare. good version control is an absolute requirement. -bowerbird

3 3

updated PP editor
by Roger Frank 23 Dec '11

23 Dec '11

I've made some changes to the Page Proofing Editor and I'll freeze the code and use PPE to proofread two books I've scanned with Abbyy. The major changes are all to the analysis window contents. I've added visible spell checking, with suspect words highlighted in light blue. Other colors on the analysis window indicate other suspect conditions. I've added quite a few from the excellent set in gutcheck and some others from pptxt. There are more I will add over time, but I'm anxious to use it for a few books. I've also added a way for the user to add good words from any page. I see this most often for proper names. By the way, the text is scanned with end-of-line hyphenation intact. The DHYP button dehyphenates the page so spellchecking has whole words to check. Feedback always welcomed. I'll report back again after I've used it on an entire text. --Roger

1 0

Seeking advice on formatting of magazine-type content
by Roger Lee 23 Dec '11

23 Dec '11

I've previously completed a reasonable number of proof-readings of book content and now I've begun preparing the content of a large annual of weekly magazines - I have already received my copyright clearance. My annual was published in 1905-6 and is in fair condition, but not suitable for flat-bed scanning; the pages are somewhat yellowed. I'm using a LG LSM-100 Scanner Mouse and I've completed initial scanning of the first issue of this magazine. I'm creating separate PNG files for each complete page and of each distinct illustration; these need further work. Based on my previous proof-reading, I'm working on correcting the initial OCR text in plain text files. I'm using the folder structure as follows... Annual pngs (of annual only) text (of annual only) Issue 1 pngs (of Issue 1 only) text (of Issue 1 only) (etc) Each weekly magazine comprises fictional and non-fictional material. Initially, I plan to treat each issue of about 16 pages as a project and the annual as a separate project. I'm also thinking that where articles are episodic, I feel it is worth considering side projects in which the episodes for a given article are assembled as a unit. The annual has a good table of contents using page numbers; in the ebook format, the page numbers might usefully be converted into issue number and article-in-issue number (like 1395.1). In the first issue, I've seen only one page reference to an issue pre-dating this annual. In the Project Gutenberg catalog, I see there are quite a number of other magazines, so I can use them as a guide though I realize there are experienced final-formatters out there. I would welcome pointers to relevant information to help me proceed and, if possible, contact with someone who has worked on this kind of material. How do I go about creating the project structure for others to help with proof-reading, etc, in due course? Where can I find advice on cleaning up illustrations? Thanks for your patience, Roger

5 9

Re: [gutvol-d] book of james -- 001
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

for want of better text, i've now loaded in the original o.c.r. for the "book of james". > http://z-m-l.com/go/bhaga/bhagap123.html this isn't the edit interface, it's just a skeleton to display the o.c.r./scan combo for each page. resizing your browser-window resizes the scan. so james, if you want me to pursue this at all, send me your corrected files. otherwise, i will just be done with it. i'd love to see an xhtml tutorial on this book... *** roger said: > http://etext.ws/ppe.php yes! roger's back! :+) > different users with different browsers. iphone, no. ipad, also no, unfortunately. doesn't appear to size itself very well... > http://z-m-l.com/misc/cinema-screen-shot.png lotta unused space on my 23-inch screen. anyway, all you people who bellyache about "open source" need to step up to the plate... go over and help roger code something cool. show that you really _deserve_ open source... *** don said: > Is this closer? i don't think so. i'd say you're getting colder. but i _will_ read that story about the turnip... *** keith said: > elves aha! that's great. i didn't think the gerbils would have enough manual dexterity... but _elves_ are very handy with their... hands... > Now, if I could only get then to program. oh, they don't have to be able to code at all. just tell them to run these commands: > pandoc -i criterion.html -o markdown.md > pandoc -i criterion.html -o rst.rst > pandoc -i markdown.md -o markdown.html > pandoc -i rst.rst -o rst.html if criterion.html and markdown.html and rst.html are not "sufficiently similar", let me know why not. *** carlo said: > djvutxt --page=123 djvufile.djvu 123.txt ok, i get it now. it's on _my_ machine. i thought you issued it on archive.org. i can't thank you enough for that, carlo; this ends a years-long headache for me. -bowerbird

4 5

Re: [gutvol-d] book of james -- 002
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

james- there is now cleaner text on those web-pages: > http://z-m-l.com/go/bhaga/bhagap123.html this text is from the .djvu file, so the pagebreaks are much more accurate than they were before... i was hoping the em-dashes would reappear too, but no luck there. perhaps someone will offer to re-do the o.c.r. i will also play with the abbyy.gz file at archive.org, and see what i can coax out of that, but i'm having too much fun tinkering with the .djvu text for now. like i said, hang loose until after christmas... :+) -bowerbird

1 0

book of james -- 002
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

james said: > I started doing the page-at-a-time thing and gave up. > Your pages are already better than mine because > I used Tesseract and archive.org uses ABBY Fine Reader. except the o.c.r. from archive.org, in this case, is screwed up. it's missing its em-dashes. i've dealt with this problem before, and it's less work to re-do the o.c.r. than to fix the em-dashes. will someone with a good version of abbyy please re-do this o.c.r.? > 2). This book really requires a way to enter UTF-8 characters. if someone does the o.c.r. for you, they can specify utf8 output... > If I could just stick a circumflex above a's, u's, and i's > (both lower and upper case) that would be 99% of what I need. if you can pull out a list of the words that require circumflexes, we can create a script that does a global change in one swoop. > (after de-hyphenating do not dehyphenate! the program will do that for you. > re-wrapping do not rewrap! if you need to rewrap, the program can do it. rewrapping is evil. it just makes it harder for the next guy... -bowerbird

3 2

Re: [gutvol-d] book of james -- 002
by Bowerbird＠aol.com 21 Dec '11

21 Dec '11

james- just hang loose for now. we'll get it all straightened out. :+) -bowerbird p.s. roger, that screenshot from earlier happened because i had "zoom text only" selected. now it resizes much better.

1 0

book of james -- 001
by Bowerbird＠aol.com 21 Dec '11

21 Dec '11

james said: > I am interested in trying out your tool. cool. let's mount it as a demo project. > I'm currently working on a book > that has a special meaning to me > but which is hell to format. ok. > http://www.archive.org/details/studyofbhagavata00benaiala i'll check it out. > I've more or less committed myself to > doing family tree tables using ASCII text. i'd suggest doing them as text _and_ images, and putting both in the book, as a fall-back... text can be copied; images preserve the look. > There are a LOT of these in the book. > I'd need some way to indicate that these > should be surrounded by "pre" tags in the HTML. that's easy enough to do. > Second, I need to have some illustrations. > How would your tool handle those? with simplicity, grace, and elegance, of course. ;+) > My understanding of your ideas is that > I can make a file with some simple markup > and generate many different formats from that. you understand correctly. > I used the HTML generator in guiguts for > my PG submissions so far. I'm not entirely happy > with it, especially the way it uses "span" tags > to indicate indented text. I end up going through > and replacing these with "blockquotes". > The style sheet guiguts gives me is good for HTML > but needs to be tweaked to produce a good Kindle book. > And of course once I've generated my HTML > I now have two files to correct. i hear everything you're saying. > Also, if the book has multiple levels of headings, > how would you deal with that? you simply indicate the level of the heading by the number of blank lines preceding it... for instance: > h1 heading = 14 blank lines preceding > h2 heading = 12 blank lines preceding > h3 heading = 10 blank lines preceding > h4 heading = 8 blank lines preceding > h5 heading = 6 blank lines preceding > h6 heading = 4 blank lines preceding or, if you prefer: > h1 heading = 9 blank lines preceding > h2 heading = 8 blank lines preceding > h3 heading = 7 blank lines preceding > h4 heading = 6 blank lines preceding > h5 heading = 5 blank lines preceding > h6 heading = 4 blank lines preceding note it's the _relative_ number that counts. (the absolute number merely determines how much vertical space the header uses when the book is output in .pdf format... some designers like to put it way down.) the program creates a nice _outline-style_ indented list of the headings for you, and keeps track of any change that happens, so once you've determined the outline is right, you can pretty much stop thinking about it. (if you've already used the h1-h6 terms, you can just do a global replace on them. if you like, you can _leave_in_ the h1-h6 terms, in addition to the blank lines, and wait to take them out until you "go final".) *** so i'll mount the scans. send me the text, and i'll incorporate it into the public demo. however... ...here comes the bad news... it might well be the case that what you want is impossible. because the kindle can't do it. is _incapable_ of doing it. not just kindle, but _any_ screen that's "too small" (e.g., a phone). for instance, most of the family trees that you gave as examples in your december 18th post _won't_ work -- dependably -- in _any_ system. > http://lists.pglaf.org/mailman/private/gutvol-d/2011-December/008590.html not as text, anyway. you can "fool yourself" into thinking they work, by setting the text-size down to a small level... but if you want a test, do the opposite instead: set the text-size to its biggest level, and then see how many characters fit on a kindle screen. you will then be tempted to say something like "well, as long as the text is a reasonable size..." but again, that's just fooling yourself, because if you ask a 50-year-old person, you might get a totally different answer m a 20-year-old as to what text-size constitutes "a reasonable size"... and your problem is worse than the "poetry" one, because your lines, if split, _become_gibberish_. so you are going to have disaster. perhaps not all the time, on all of your trees. but sometimes. and you need to face that fact before you start, or you're only gonna be disappointed later... so, if you wanna work with me, send me the text. -bowerbird

4 5

exploratory code
by Roger Frank 21 Dec '11

21 Dec '11

Though I've run many books through DP's proofing and formatting workflow, I always thought it would be easier and more fun if the computer gave feedback about what it thought of the page as it was being proofed. To test this out, I have written a multi-pane editor that may be useful for etext production. I loaded a book into the demo version of the program at http://etext.ws/ppe.php and I'm interested to know if it works for different users with different browsers. It uses several HTML5 pieces that I think are safe. Open source, MIT license, as is all my software tools for etext production. The left pane is a text edit area for a page. Center pane shows the results of analysis for that page. Top pane is a clickable image of the page. Various status information is lower right. Still to do: spelling check, common scanno flags and many more regex checks. Right now it includes spaced or crunched punctuation, poetry detection, end-of-line hyphens (which may be resolved with the DHYP button) and a few others. I also want to use a magnifier jquery plugin on the image to dynamically enlarge the area under the mouse instead of having to open another window. I will solo post-process a few books using it. If I can make it work well enough, I'll add the page queue code (i.e. "next unproofed page"), instead of how it is now, where you select the page to proof or it gives you a random page. As you proof, the status area displays how many times the page has been proofed, as this is all tracked along with all earlier revisions in the database. There is no login required and I'd appreciate if you would take a look at it. Feedback welcomed to the email shown in the application. I'll summarize (if there is any interest) back to this thread. --Roger

2 2