December 2011 - gutvol-d - lists.pglaf.org

a review of some digitization tools -- 023
by Bowerbird＠aol.com 19 Dec '11

19 Dec '11

ok, so let's regroup once again for this new week... this thread's purpose is to review digitization tools. so we need to know what's involved in "digitization". our analysis gave us this: > 1. obtain the text (via scanners, and o.c.r. software) > 2. clean the text (the first set of tasks for our tools) > 2a. do a spellcheck > 2b. fix spacey punctuation > 2c. restore styling, e.g., italics > 3. turn the text into e-books (the second set of tasks) > 3a. tagging the structural aspects of the text > 3b. converting the text into .html (intermediate and final) > 3c. converting intermediate .html into e-book output-files task #1 is largely taken care of for us, thanks to the public output of the major scanning projects. (although sometime this week i'll discuss the issue of how to get the best results from abbyy's output.) task #2 is something we discussed earlier, finding that the jobs can be done fine in a wordprocessor. that leaves us with task #3. on some day this week, i will discuss the specifics of applying zen markup, and that post will cover the main thrust of task #3a. and then, of course, the two "conversion" tasks -- #3b and #3c -- are being performed by the script which we're in the process of discussing right now. *** so we need an application that does #3b and #3c. i do programming. so my first instinct is to code. so if i have a job which needs to be accomplished, once i've coded something -- or tried to do so -- i can compare it with the tools that exist out there, and do an assessment of how everything stacks up. what we'd like, for this app, is one that: 1. creates good html. (gotta do the job, and well.) 2. is cost-free. (because we don't have a budget.) 3. is user-friendly. (because we rely on volunteers.) 4. runs online and off. (so we have multiple options.) 5. is open-source. (because why not ask for it all?) those are listed in decreasing order of importance. but everything through #4 is pretty much required. we need an online component, because we will want our system to be capable of being used in a setting like project gutenberg or distributed proofreaders, a cyberlibrary meant to digitize and display e-books. an online presence is required by such a cyberlibrary. at the same time, it would be best if the tools used by this cyberlibrary could also be run by people offline... if we can meet all of our desires with our own code, it's gonna be hard for any other tool to get traction. so let's see how we do... *** up to this point, the sample content for this thread was "books and culture", by hamilton wright mabie, a book that is distinguished by the fact that it was the very first public-domain book released by the big library scanning project being done by google. it was a simple book, so it was good starter content. but now we want to work on more complicated stuff. what we need, as a "proving ground" for our script, is a good test-suite, with the structures we require. fortunately -- if you ascribe luck to elbow grease -- we have such a test-suite, one that i created in 2005. i've updated it a bit for this thread, and put it here: > http://zenmagiclove.com//test-suite-2012.txt so now let's give our script a new name as well: > http://zenmagiclove.com/cgi-bin/t2.py that's just a first pass, so everything might not look exactly right just quite yet while we get started, but we'll work our way up to handle everything we need. so look at that input test-suite, satisfying yourself that it doesn't have any sneaky tricks up its sleeves, and then take a look at the .html the script creates -- feel free to view the source too, and comment -- and then you'll be ready as we depart on this ride... *** z.m.l. input file: > http://zenmagiclove.com//test-suite-2012.txt python script to create .html output: > http://zenmagiclove.com/cgi-bin/t2.py -bowerbird

1 0

a review of some digitization tools -- 022
by Bowerbird＠aol.com 19 Dec '11

19 Dec '11

keith said: > The convert code has to be substatially equivalent HTML. it will be. > Beware! That is a very strong factor. try it before you ramble off into the hypothetical brambles. > And that prize would go to the developers of pandoc. i have no quarrel with that. but... john is a professor of philosophy at california berkeley. so i sincerely doubt he needs a hundred bucks from lee. somebody should take his money, though, if he'd gonna be dumb enough to leave it laying in the street like that. -bowerbird

2 1

Re: [gutvol-d] let's get some tutorials from the xhtml crowd
by Bowerbird＠aol.com 18 Dec '11

18 Dec '11

paul flo said: > As a PG newbie, I've found the "do what you want" > attitude to be one of the biggest barriers to entry. > I prefer instruction, guidance and examples of best practice. if i would have known you are a "newbie", i wouldn't have even bothered to ask you about your workflow, because you really haven't even developed one yet... still, you did seem to be very firmly convinced that your way was the correct way, and my way was not. > (Another barrier was my own dumb choice of > a 700 page two volume work to start on. Ho hum.) it sounds like the content of the pages is probably even more daunting than the sheer number of 'em. but, you know, that's how you learn... :+) > It's been a 15-month labour of love, > but I'm nearly there now! ouch. if i had known _that_, i definitely wouldn't have asked about your workflow. except as a bad example. i certainly hope that time included several months-long gaps where you did no work except to regain motivation. xhtml crowd: get to work on those tutorials, right now! because your followers seem to be wasting a lot of time! -bowerbird

1 0

a review of some digitization tools -- 021
by Bowerbird＠aol.com 17 Dec '11

17 Dec '11

this week, we looked at the e-book file-formats which were auto-generated for our sample book. recall that this is _not_ just some "random" book. it's a book that has had a very high profile in the history of book-digitization for the entire planet. it was also an extremely simple book to mark up, as it's almost entirely paragraphs or headers, but d.p. managed to whiff their chance outright, and that hack-job has sat at p.g., unfixed, for 7 years. *** it's quite easy to make vague accusations against project gutenberg, alleging low-quality e-texts... and people do. sometimes with sheer nonsense. (some who _were_ on this list, some who still are.) it's also extremely easy to _ignore_ vague claims. especially when some of 'em are sheer nonsense. so that's how the whitewashers treat vague claims, _all_ the vague claims -- as nothing but nonsense. it's much harder to clearly document your criticism. first, it takes work; but also your audience must then pay attention to be able to understand the problems. i did the work. so... were you paying any attention? i hope whitewashers too pay attention to my demo. and realize what they have to do to prevent shoddy work like this from being posted in the p.g. library, which includes telling distributed proofreaders that the postprocessors will need to step up their game. i hope that happens. but i doubt that it will. it's just too easy, once you start ignoring criticism, to continue in the comfortable path of head in sand. but if someone wants to send me a positive sign, it'd sure be easy to correct the flaws in pg#16736. (like it woulda been easy to do it right originally.) *** i hear someone out there asking "couldn't marcello improve his script, so it would make this file work?" well, yeah, he could. and he probably will, now that i've laid this embarrassing example in front of you... but his converter ain't the only tool out there that'll _expect_ headers to be tagged with a [h] header tag. you do not have to accept the strictest definition of a "paragraph" that someone (like lee) can dream up, but you certainly can't tag a header as a "paragraph" and still expect that anybody will take you seriously. and, in general, it just won't work to try to overcome gross stupidity at the tag level with "clever scripting". as the saying goes, just when you think you obtained a fool-proof system, the world devises a bigger fool. so even if marcello plugs this one hole in the boat, it won't matter if postprocessors punch a new one. and if the postprocessors can do anything they want, poking a hole in the boat is exactly what they will do. *** and, if you want the bigger picture, well here it is... lots of books now sit in the d.p. postprocessing queue. lots and lots and lots and lots of them. what i am showing here in this current thread is that post-processing doesn't need to be all that difficult... i'm basically boiling it down to a single python script. the d.p. books as they come out of f2 are not as clean as the text i am working with here. they have "notes" in them, and pseudo-markup, and other artifacts, but they aren't really _that_ far away from what i'm using, if you only take the step of seeing that that's the case. on the other hand, if you continue to see that job as "difficult and time-consuming", it continues to be so. at any rate, that's the bigger picture... *** i find it interesting that nobody at all has weighed in on the quality (or lack of it) inherent in the .html code that's generated by the python script in the latest demo. this crew regularly lambastes html that is generated by ms-word, and calibre, and sigil, and everything else, but not a peep about a converter they can actually influence... it's as if they don't even _want_ a high-quality converter. *** z.m.l. input file: > http://zenmagiclove.com/grapes009.txt python script to create .html output: > http://zenmagiclove.com/cgi-bin/tday2011.py and next week we go back to the python script and work on installing routines for more-complicated structures... remember how people used to insist that light-markup was incapable of handling anything but simple books? i wish i had a nickel for every time someone wrote that. and when i asked them to show me a hundred books that were "too complicated" for me to do, guess what? nobody showed me 100. or even a dozen. or even one! now, surely it must be the case that there's books that are so complicated that my z.m.l. cannot handle them! heck, i _know_ it's true. i can point to a dozen myself! but i can't point to many more. and i don't think that you can point to many more either. but just try... ok? -bowerbird

1 0

preshashay2002
by Sankar Viswanathan 17 Dec '11

17 Dec '11

good day http://www.mothersdaygiftsideas.net/wp-admin/images/sca5iv.php?upygforum=62 Sat, 17 Dec 2011 1:15:10 _____________________ "He says--I dont suppose it makes much difference what he says; he simply doesnt know what hes talk--You think you know, but do you? Youve read a lot of gush that-- Terry began, but Gus interrupted him, almost a new thing for the quiet chap.Listen, Terry: get right on this." (c) Kellyann vsi9a3a

1 0

Re: [gutvol-d] The problems with paragraph formatting at PG
by Bowerbird＠aol.com 16 Dec '11

16 Dec '11

paul flo said: > And yet it is so difficult > to have these ASCII files work for readers. light-markup converts well to any format you want. i'm in the process of demonstrating that, right now. *** > to show that more people would rather work in XHTML > than ZML, I'd only have to find two XHTML people. xhtml? as far as i can read what the wc3 has declared, xhtml is a dead-end. even .epub is leaving it behind. light-markup, on the other hand, has been on a sharp upward slope for years now... even with trendy geeks... heck, _most_especially_ with the trendiest of the geeks. markdown, in particular, is _exploding_ at the moment. but, you know, use what _you_ like the best, eh? :+) *** you know that one of the _output_ formats for markdown is xhtml, right? but the point is, nobody wants to _write_ in an xhtml environment. not any _real_writers_, anyway. real writers want the-words-and-nothing-but-the-words. even the main .epub programmers use asciidoc to _write_. but use whatever you like! what _do_ you use, by the way? -bowerbird

4 4

never bet every bit of your credibility
by Bowerbird＠aol.com 16 Dec '11

16 Dec '11

i said: > last year i did this demo in perl. > wanna see those scripts? > (no, of course you can't have them. > but you can _run_ em.) well, i did do some demos in perl last year... so i poked around, to dig them out for paul, because paul had said that he scripts in perl. but those weren't the first things i did in perl. so i decided to be curious, and see how far back that i started doing perl. i'm not sure even now, but i was definitely actively using perl by 2007, because i still have content dated february 2007. and it's not even in an obscure location, either... it's right off the main page of my z-m-l.com site. > http://z-m-l.com which means that, in 2 months, it will have been up on my site, available publicly, _for_5_years_... and lordy lordy, it's basically the very same demo which i am currently scripting in python. get out! no wonder i am feeling nostalgic -- i'm in a rut... gonna have to take my game up a couple notches. so paul, if you're listening to people who say that i never produce anything, you should know that _they_are_telling_you_lies,_speaking_untruths..._ and just so you know, paul, these were the demos that made my antagonists realize they had _lost_ the little war they'd started against me, and thus they had to sit by and watch in utter helplessness as i showed the winning hand in our poker match and started sweeping the big pot over to my side, a pot which contained every bit of their credibility, which they'd unwisely thrown on the pile in a wager that proved to be _incredibly_ stupid on their part. and thus a very loud silence fell over the crowd... the moral is "never bet every bit of your credibility". another good lesson is that it is always unwise to bet the other guy on what he has in _his_ pocket. he's probably got a much better idea than you do about what _is_ in his pocket, and what is _not_... even if he says he has a perpetual motion machine or a time machine in there, be careful with that bet. it's _safer_ to take a bet on what's in _your_ pocket. because you can check, so you should know, right? regardless, you still shouldn't bet _everything_ on it. because when the time arrives for you to _prove_it_, you just might discover your pocket has been picked. -bowerbird

1 0

abbyy finereader on sale -- last day!
by Bowerbird＠aol.com 15 Dec '11

15 Dec '11

the new version of abbyy finereader -- v11 -- is on sale... but today is the last day of the sale -- at least supposedly. the full version, new, is $119 on sale, reduced from $170. the update is $70 on sale, and it will be $100 afterwards... finereader express, for the mac, is $70 today, $100 later. i don't know if v11 is an improvement on earlier versions or not, so i can't give any advice on whether to _upgrade_. but this is a good price to buy a version if you have none... most _especially_ if you believe the lies that jim and lee have been telling you that finereader does good markup. it does _presentational_ markup, which is well and good -- and honestly, that's all that a regular person needs -- but not what you really want if you have to edit the .html. finereader _does_ give excellent o.c.r., though, the _best_! so again, if you don't have an o.c.r. program, grab it now! > http://abbyyusa.com -bowerbird

3 2

... and then there are conversion tools
by D Garcia 15 Dec '11

15 Dec '11

I'm surprised that no one has yet mentioned pandoc as a useful piece of kit. http://johnmacfarlane.net/pandoc/

2 1

a review of some digitization tools -- 020
by Bowerbird＠aol.com 15 Dec '11

15 Dec '11

so, we have compared the .html version from my converter with the one from p.g., and learned they look virtually identical, although the p.g. version has "tag abuse" in its coding, while my version does not... (you are _welcome_ to give full criticism on the .html being generated by my converter... give me feedback now before it goes public.) then yesterday, we compared the _.mobi_ i created with the p.g. auto-generated one, and found the p.g. version had many flaws, most of them stemming from the tag abuse. so today we're gonna compare the .epub i made with the p.g. auto-generated one. an .epub file is mostly a "wrapper" around the .html, so i'd say we can now guess that the .epub is going to have some problems, especially since we have been experienced. here's where to download the p.g. .epub: > http://www.gutenberg.org/ebooks/16736 and here's where you can find mine: > http://zenmagiclove.com/misc/ukagl-epub.epub > http://zenmagiclove.com/misc/ukagl-epub.zip *** the results are exactly as we would fear. so i'm not even going to step through the drill. here are some screenshots; you'll figure it out. > http://zenmagiclove.com/misc/bac-epub001.jpg > http://zenmagiclove.com/misc/bac-epub002.jpg > http://zenmagiclove.com/misc/bac-epub003.jpg > http://zenmagiclove.com/misc/bac-epub004.jpg > http://zenmagiclove.com/misc/bac-epub005.jpg > http://zenmagiclove.com/misc/bac-epub006.jpg > http://zenmagiclove.com/misc/bac-epub007.jpg > http://zenmagiclove.com/misc/bac-epub008.jpg > http://zenmagiclove.com/misc/bac-epub009.jpg > http://zenmagiclove.com/misc/bac-epub010.jpg > http://zenmagiclove.com/misc/bac-epub011.jpg > http://zenmagiclove.com/misc/bac-epub012.jpg garbage in, garbage out. sorry, no way around that. and again, one more time, because it must be said, this was a _simple_ book! headers and paragraphs! oy vey... *** to repeat, i welcome feedback on my .html coding! z.m.l. input file: > http://zenmagiclove.com/grapes009.txt python script to create .html output: > http://zenmagiclove.com/cgi-bin/tday2011.py once this is released publicly, i'll likely be deluged with requests to fiddle with stuff, so i will probably deny all of them, with little or no consideration, so this is your _only_ chance to get it how you want it. *** tomorrow, i sum up the comparisons from this week. next week we get back to building that python script. -bowerbird

1 0