December 2011 - gutvol-d - lists.pglaf.org

updated PP editor
by Roger Frank 23 Dec '11

23 Dec '11

I've made some changes to the Page Proofing Editor and I'll freeze the code and use PPE to proofread two books I've scanned with Abbyy. The major changes are all to the analysis window contents. I've added visible spell checking, with suspect words highlighted in light blue. Other colors on the analysis window indicate other suspect conditions. I've added quite a few from the excellent set in gutcheck and some others from pptxt. There are more I will add over time, but I'm anxious to use it for a few books. I've also added a way for the user to add good words from any page. I see this most often for proper names. By the way, the text is scanned with end-of-line hyphenation intact. The DHYP button dehyphenates the page so spellchecking has whole words to check. Feedback always welcomed. I'll report back again after I've used it on an entire text. --Roger

1 0

Seeking advice on formatting of magazine-type content
by Roger Lee 23 Dec '11

23 Dec '11

I've previously completed a reasonable number of proof-readings of book content and now I've begun preparing the content of a large annual of weekly magazines - I have already received my copyright clearance. My annual was published in 1905-6 and is in fair condition, but not suitable for flat-bed scanning; the pages are somewhat yellowed. I'm using a LG LSM-100 Scanner Mouse and I've completed initial scanning of the first issue of this magazine. I'm creating separate PNG files for each complete page and of each distinct illustration; these need further work. Based on my previous proof-reading, I'm working on correcting the initial OCR text in plain text files. I'm using the folder structure as follows... Annual pngs (of annual only) text (of annual only) Issue 1 pngs (of Issue 1 only) text (of Issue 1 only) (etc) Each weekly magazine comprises fictional and non-fictional material. Initially, I plan to treat each issue of about 16 pages as a project and the annual as a separate project. I'm also thinking that where articles are episodic, I feel it is worth considering side projects in which the episodes for a given article are assembled as a unit. The annual has a good table of contents using page numbers; in the ebook format, the page numbers might usefully be converted into issue number and article-in-issue number (like 1395.1). In the first issue, I've seen only one page reference to an issue pre-dating this annual. In the Project Gutenberg catalog, I see there are quite a number of other magazines, so I can use them as a guide though I realize there are experienced final-formatters out there. I would welcome pointers to relevant information to help me proceed and, if possible, contact with someone who has worked on this kind of material. How do I go about creating the project structure for others to help with proof-reading, etc, in due course? Where can I find advice on cleaning up illustrations? Thanks for your patience, Roger

5 9

Re: [gutvol-d] book of james -- 001
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

for want of better text, i've now loaded in the original o.c.r. for the "book of james". > http://z-m-l.com/go/bhaga/bhagap123.html this isn't the edit interface, it's just a skeleton to display the o.c.r./scan combo for each page. resizing your browser-window resizes the scan. so james, if you want me to pursue this at all, send me your corrected files. otherwise, i will just be done with it. i'd love to see an xhtml tutorial on this book... *** roger said: > http://etext.ws/ppe.php yes! roger's back! :+) > different users with different browsers. iphone, no. ipad, also no, unfortunately. doesn't appear to size itself very well... > http://z-m-l.com/misc/cinema-screen-shot.png lotta unused space on my 23-inch screen. anyway, all you people who bellyache about "open source" need to step up to the plate... go over and help roger code something cool. show that you really _deserve_ open source... *** don said: > Is this closer? i don't think so. i'd say you're getting colder. but i _will_ read that story about the turnip... *** keith said: > elves aha! that's great. i didn't think the gerbils would have enough manual dexterity... but _elves_ are very handy with their... hands... > Now, if I could only get then to program. oh, they don't have to be able to code at all. just tell them to run these commands: > pandoc -i criterion.html -o markdown.md > pandoc -i criterion.html -o rst.rst > pandoc -i markdown.md -o markdown.html > pandoc -i rst.rst -o rst.html if criterion.html and markdown.html and rst.html are not "sufficiently similar", let me know why not. *** carlo said: > djvutxt --page=123 djvufile.djvu 123.txt ok, i get it now. it's on _my_ machine. i thought you issued it on archive.org. i can't thank you enough for that, carlo; this ends a years-long headache for me. -bowerbird

4 5

Re: [gutvol-d] book of james -- 002
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

james- there is now cleaner text on those web-pages: > http://z-m-l.com/go/bhaga/bhagap123.html this text is from the .djvu file, so the pagebreaks are much more accurate than they were before... i was hoping the em-dashes would reappear too, but no luck there. perhaps someone will offer to re-do the o.c.r. i will also play with the abbyy.gz file at archive.org, and see what i can coax out of that, but i'm having too much fun tinkering with the .djvu text for now. like i said, hang loose until after christmas... :+) -bowerbird

1 0

book of james -- 002
by Bowerbird＠aol.com 22 Dec '11

22 Dec '11

james said: > I started doing the page-at-a-time thing and gave up. > Your pages are already better than mine because > I used Tesseract and archive.org uses ABBY Fine Reader. except the o.c.r. from archive.org, in this case, is screwed up. it's missing its em-dashes. i've dealt with this problem before, and it's less work to re-do the o.c.r. than to fix the em-dashes. will someone with a good version of abbyy please re-do this o.c.r.? > 2). This book really requires a way to enter UTF-8 characters. if someone does the o.c.r. for you, they can specify utf8 output... > If I could just stick a circumflex above a's, u's, and i's > (both lower and upper case) that would be 99% of what I need. if you can pull out a list of the words that require circumflexes, we can create a script that does a global change in one swoop. > (after de-hyphenating do not dehyphenate! the program will do that for you. > re-wrapping do not rewrap! if you need to rewrap, the program can do it. rewrapping is evil. it just makes it harder for the next guy... -bowerbird

3 2

Re: [gutvol-d] book of james -- 002
by Bowerbird＠aol.com 21 Dec '11

21 Dec '11

james- just hang loose for now. we'll get it all straightened out. :+) -bowerbird p.s. roger, that screenshot from earlier happened because i had "zoom text only" selected. now it resizes much better.

1 0

book of james -- 001
by Bowerbird＠aol.com 21 Dec '11

21 Dec '11

james said: > I am interested in trying out your tool. cool. let's mount it as a demo project. > I'm currently working on a book > that has a special meaning to me > but which is hell to format. ok. > http://www.archive.org/details/studyofbhagavata00benaiala i'll check it out. > I've more or less committed myself to > doing family tree tables using ASCII text. i'd suggest doing them as text _and_ images, and putting both in the book, as a fall-back... text can be copied; images preserve the look. > There are a LOT of these in the book. > I'd need some way to indicate that these > should be surrounded by "pre" tags in the HTML. that's easy enough to do. > Second, I need to have some illustrations. > How would your tool handle those? with simplicity, grace, and elegance, of course. ;+) > My understanding of your ideas is that > I can make a file with some simple markup > and generate many different formats from that. you understand correctly. > I used the HTML generator in guiguts for > my PG submissions so far. I'm not entirely happy > with it, especially the way it uses "span" tags > to indicate indented text. I end up going through > and replacing these with "blockquotes". > The style sheet guiguts gives me is good for HTML > but needs to be tweaked to produce a good Kindle book. > And of course once I've generated my HTML > I now have two files to correct. i hear everything you're saying. > Also, if the book has multiple levels of headings, > how would you deal with that? you simply indicate the level of the heading by the number of blank lines preceding it... for instance: > h1 heading = 14 blank lines preceding > h2 heading = 12 blank lines preceding > h3 heading = 10 blank lines preceding > h4 heading = 8 blank lines preceding > h5 heading = 6 blank lines preceding > h6 heading = 4 blank lines preceding or, if you prefer: > h1 heading = 9 blank lines preceding > h2 heading = 8 blank lines preceding > h3 heading = 7 blank lines preceding > h4 heading = 6 blank lines preceding > h5 heading = 5 blank lines preceding > h6 heading = 4 blank lines preceding note it's the _relative_ number that counts. (the absolute number merely determines how much vertical space the header uses when the book is output in .pdf format... some designers like to put it way down.) the program creates a nice _outline-style_ indented list of the headings for you, and keeps track of any change that happens, so once you've determined the outline is right, you can pretty much stop thinking about it. (if you've already used the h1-h6 terms, you can just do a global replace on them. if you like, you can _leave_in_ the h1-h6 terms, in addition to the blank lines, and wait to take them out until you "go final".) *** so i'll mount the scans. send me the text, and i'll incorporate it into the public demo. however... ...here comes the bad news... it might well be the case that what you want is impossible. because the kindle can't do it. is _incapable_ of doing it. not just kindle, but _any_ screen that's "too small" (e.g., a phone). for instance, most of the family trees that you gave as examples in your december 18th post _won't_ work -- dependably -- in _any_ system. > http://lists.pglaf.org/mailman/private/gutvol-d/2011-December/008590.html not as text, anyway. you can "fool yourself" into thinking they work, by setting the text-size down to a small level... but if you want a test, do the opposite instead: set the text-size to its biggest level, and then see how many characters fit on a kindle screen. you will then be tempted to say something like "well, as long as the text is a reasonable size..." but again, that's just fooling yourself, because if you ask a 50-year-old person, you might get a totally different answer m a 20-year-old as to what text-size constitutes "a reasonable size"... and your problem is worse than the "poetry" one, because your lines, if split, _become_gibberish_. so you are going to have disaster. perhaps not all the time, on all of your trees. but sometimes. and you need to face that fact before you start, or you're only gonna be disappointed later... so, if you wanna work with me, send me the text. -bowerbird

4 5

exploratory code
by Roger Frank 21 Dec '11

21 Dec '11

Though I've run many books through DP's proofing and formatting workflow, I always thought it would be easier and more fun if the computer gave feedback about what it thought of the page as it was being proofed. To test this out, I have written a multi-pane editor that may be useful for etext production. I loaded a book into the demo version of the program at http://etext.ws/ppe.php and I'm interested to know if it works for different users with different browsers. It uses several HTML5 pieces that I think are safe. Open source, MIT license, as is all my software tools for etext production. The left pane is a text edit area for a page. Center pane shows the results of analysis for that page. Top pane is a clickable image of the page. Various status information is lower right. Still to do: spelling check, common scanno flags and many more regex checks. Right now it includes spaced or crunched punctuation, poetry detection, end-of-line hyphens (which may be resolved with the DHYP button) and a few others. I also want to use a magnifier jquery plugin on the image to dynamically enlarge the area under the mouse instead of having to open another window. I will solo post-process a few books using it. If I can make it work well enough, I'll add the page queue code (i.e. "next unproofed page"), instead of how it is now, where you select the page to proof or it gives you a random page. As you proof, the status area displays how many times the page has been proofed, as this is all tracked along with all earlier revisions in the database. There is no login required and I'd appreciate if you would take a look at it. Feedback welcomed to the email shown in the application. I'll summarize (if there is any interest) back to this thread. --Roger

2 2

gerbils expose chilling factoid
by Bowerbird＠aol.com 21 Dec '11

21 Dec '11

keith said: > just submit pandoc, and will > we will see what Lee has to say. oh, so now you want to try and send _me_ on that wild-goose chase! no way, mister. :+) but really, keith, what did you mean by that puzzling "no human intervention" comment? do you have some gerbils who'll run the test? :+) *** that $100 "challenge" can't be taken seriously. for anyone who wants to know why, seriously, i am all too happy to explain the subterfuge... lee _thought_ that he could demonstrate that light-markup languages are not fully specified. thus, by requiring different ones to produce what is essentially the same underlying code, he thought he could expose the holes in each. what he didn't know is that the schemes _are_ fully specified, and thus can indeed create code which accomplishes even a fairly complex book, in a fairly standard manner, such that that code would indeed qualify as "essentially" equivalent. in fact, the degree of full-specification obtained is complete enough that these markup systems can even be translated in a one-to-one fashion. and not only can they "be translated" that way, the conversion can be done _programmatically_, and pandoc is the existence-proof for _that_... so lee was significantly underinformed, and his "challenge" merely exposed that chilling factoid. but lee is a clever and devious person, who knows how to cover his ass, so he found a way to do that, in the off-chance that he unknowingly was wrong. his challenge specified that _he_ would prepare the various light-markup files, meaning he could _sabotage_ the full-specification of the schemes. lee made it _appear_ as if he was betting that you couldn't make a basket shooting from mid-court. (or what he believed would be a mid-court shot, even though, with pandoc, it was a mere lay-up.) but, with a little shift at the end, he changed it, such that he was betting that _he_ would miss a shot from mid-court. would you take _that_ bet? would you take a bet that he could miss a lay-up? ok, fine, lee wanted to make sure that he wouldn't have to give away his $100. who could blame him? but to think this proves anything is just ridiculous. nonetheless, a simple contradictory point remains. namely, none of the benefits of any light-markup system _ever_required_ that such a system could be converted flawlessly into _other_ light-markup. the object of light-markup is to convert to .x/html, .epub, .mobi, .pdf, or some other _output_format_, and _not_ to convert to some other _input_format._ i mean, it's _nice_ that pandoc can convert between markdown and restructured-text and asciidoc, and many other formats, but it's certainly not necessary. markdown does what it's _meant_ to do, regardless. and so does restructured-text. and so does asciidoc. lee is trying to shift your attention to a trivial thing. he wants you to focus on a meaningless "conversion" between light-markup formats, instead of the _more_ important matter that light-markup converts to .html. because lee wants you to do your authoring in .html, not light-markup. but light-markup is much simpler, so i'm not really sure why he wants to waste your time. and let's be absolutely clear about something vital... this choice of a "master format" does _not_ entail any need to give anything up. you don't have to "choose" between using light-markup and .html. because light-markup can be converted to .html... with the simple click of a button -- boom! -- html! not to mention .mobi. and .pdf. _automatically_... -bowerbird

2 1

Re: [gutvol-d] book of james -- 001
by Bowerbird＠aol.com 21 Dec '11

21 Dec '11

carlo said: > You seem to be unaware that > you can get the text for a single page > from the Internet Archive djvu files, through > the djvutxt command and the -page option. holy crap... i did not know that... i figured that there _had_to_ be a way to do it, since the .djvu knows the text on a page, but i didn't know how. how/where exactly does one issue this command? -bowerbird p.s. james, if you could still zip up your files, that would let me splice in your corrections...

4 4