- gutvol-d - lists.pglaf.org

Re: [gutvol-d] Producing epub ready HTML
by Bowerbird＠aol.com 17 Jan '12

17 Jan '12

james said: > First, you moved all the front matter like title page, > TOC, preface, etc. to the back of the book. that's of no consequence. this is a temporary work-product. (i did it that way so the vast majority of pagenumbers worked, because i was too lazy to build in the offsetting mechanism.) > Second, you have put <pre> tags around some content > that should be just blockquoted text. which paragraphs, specifically, as a general class? and yeah, i realize that question is self-contradicting, but... the decision on formatting was made by a crude algorithm, and i can probably improve it significantly, if i merely know which paragraphs it formatted incorrectly. so please tell me. i would look myself, but i've grown tired of this book, sorry. all of this formatting can be changed by you, at a later time, but i'm trying to do as much as i can for you, automatically, since correcting the text will cause you enough pain as it is. > The family trees should be pre-formatted, but much of > what you made pre-formatted text doesn't really need to be. right. that's what i was asking you to check. > italicized words also have underscores before and after. no consequence. > Fourth, I noticed that this book is incredibly long > and I was crazy to work on it. 880k. it's not _that_ long. it does have a lot of names that are not that familiar to westerners, but they can be handled in a bulk manner, so that doesn't have to be so hard either. diacritics can be hard, but that's just one global change too, once you have a correct version of how each word should be. (you'll laugh at yourself for having done 'em one-at-a-time.) once you boil it down this way, it's not any harder than doing two 225-page books. in fact, it's a little bit simpler than that. > I prefer having the contents of the book be > in the same order as the original you don't really think i would move frontmatter to the back in the final product, do you? no one would do that, would they? > and I think PG prefers TOCs with no page numbers. you must learn not to confuse what p.g. wants with what should be done. but yes, those pagenumbers can be eliminated... but no, you shouldn't do that... and finally, no, p.g. doesn't really care these days... > I'm not really sure what I'm supposed to do with this HTML. you aren't supposed to do _anything_ with this .html... except what i said -- look at the colorized paragraphs, and tell me which of them, if any, is improperly coded... > Is this the output of a light markup file you will give me? it's output from the current version of the file, found at: > http://zenmagiclove.com/bhaga/bhaga.zml but the book is still in my hands, and i'm not done yet... i know that the idea you (and most people) now have is that .html is something that you create "at the end", when you're "all done" with the book. but i use it as _a_tool_ that visualizes the work as it is in-progress. try to rewire your brain to think that way. > Is there some sort of web interface coming for > working on this? I could use some clarification. yes. -bowerbird

2 1

kf8 and mobi_unpack
by James Adcock 15 Jan '12

15 Jan '12

Mobileread has a good discussion of kf8 issues, and a python tool mobi_unpack.py (suitable for unpacking and understanding public domain kf8 files) at: http://www.mobileread.com/forums/showthread.php?t=61986 <http://www.mobileread.com/forums/showthread.php?t=61986&page=17> &page=17 It runs correctly and without any modifications or installation hassles on my windows CPython installation. I haven't been able to get it to run under IronPython, but then again I don't have much experience with IronPython (or with CPython, for that matter) If anyone else is interested in playing with KF8 (and presumably has a Kindle Fire and/or Kindle Previewer software) or in reading pleasingly formatted PG books on their Kindle Fire, I'm having good luck running PG html files directly through the new kf8 kindlegen, [as opposed to compiling a PG epub for example, or reading the PG version of the .mobi file] - especially if I take a minute or two to fix obvious html coding errors in the PG html files -- such as specifying side margins in the body.

1 0

book of james -- 012
by Bowerbird＠aol.com 15 Jan '12

15 Jan '12

james said: > I don't pass myself off as an expert. well, be that as it may, now i know you _do_ have expertise, in many things, which was being obscured from my view by the glare of a few "inadequacies", for want of a better word. i'll deal with the shortcomings in a later post, but for now, what's important is moving the lessons to a different point on the learning curve, one that will leverage your strengths. what was unclear before, which i now know from your book -- and will detail outright, so the others here know it too -- is that you have a fairly solid grasp of the d.p. software, and scan tailor (which is a big plus), and o.p.d.s cataloging stuff. heck, you are even comfortable setting up your own server... in addition, you've used stuff like calibre, sigil, and booki, which means you know how to make an .epub, and a .mobi, so what you seek is a _superior_ solution for those formats. (which is what i have. thus, it makes it a lot easier, for me, if we both know that that's exactly what you're looking for.) another admirable quality is you are willing to do research. you don't sit and wait for someone to hand you something; you go out and perform an active search to find solutions... but your most vital strength is that you are a programmer. i knew that, in the back of my mind, since you said you had done stuff for the o.l.p.c. machines, but if i would've known that you code in python, we coulda cut to the chase earlier... because that means that you're probably a new subscriber to this list, and were not here last september (and earlier). when i have given numerous e-book app-coding tutorials. i mention september in particular because it was then -- for the first time -- i was coding in python _and_ actually sharing my source... i suggest you go back and read that series, running under the title of "these grapes are sweet". the posts started in september, and some ran into october: > http://lists.pglaf.org/mailman/private/gutvol-d/2011-September/date.html > http://lists.pglaf.org/mailman/private/gutvol-d/2011-October/date.html again, that's the first time i coded _anything_ in python, so my scripts will be silly and crude to you, but you will start to see how easy it can be to write code to help you. and i invite you to riff on my work. let's get it going on. i've been long waiting for a programmer to come along who would spur me into a positive collaborative spiral... and you, james, just might be that guy. -bowerbird

1 0

book of james -- 011
by Bowerbird＠aol.com 14 Jan '12

14 Jan '12

ok, finally! after way too many hours of work, james, your text is now back with its original pagebreaks and linebreaks, but retaining the edits that you had made to it. whew. all that, just to undo what never should've happened... i learned a lot of little things along the way, so i don't regret having made the decision to go down that path. but the biggest lesson i learned is that it's just one big waste of time and energy to have to undo such a thing -- which is a lesson i had supposedly learned before -- so we can be sure i've gone down that path the last time. anyway, james... i'll turn the text back to you very soon. but this morning, i wanted to give myself a little reward, after all of that hard work, so i took a look at your book. > http://www.archive.org/details/EBookEnlightenment wow... that is quite a book you wrote there, james, 247 pages, chock full of graphics, on the topic of... ...making e-books! and... as far as i can tell, you covered the subject well, based on the quick cursory look i gave it this morning. so now i'm puzzled. if you're "the person who wrote the book on the topic", then what am i doing, explaining to you how to do it? can you "enlighten" me, james? (bet you got that a lot.) and what can you tell me about your co-authors? and the booki system which you used? clear me up, james. i don't like being puzzled... -bowerbird

2 1

Re: [gutvol-d] Amazon releases new KF8 and MOBI tools
by Jim Adcock 14 Jan '12

14 Jan '12

>Not sure what "we all know." Are you saying Kindle Fire is only internally HTML 3.2? Are you saying KF8 is only internally HTML 3.2? Also, reading the Amazon documentation on the KF8 release they seem to be saying that they are planning new releases soon of the software in the "current generation" Amazon eInk devices to make them KF8 compatible. How compatible remains to be seen.

1 0

Re: [gutvol-d] book of james -- 010
by Bowerbird＠aol.com 14 Jan '12

14 Jan '12

james said: > I concede your point about ABBYY Fine Reader. well, first of all, there's no need to "concede", because i'm not trying to "win" an argument... i'm trying to help you figure out what you are doing wrong, to make you feel like every book is the last one that you're ever gonna digitize... even as you lay out plans to do a dozen more... :+) so i will have much more to say, later, but i want this file off my plate, today, if i can find a way to make some time, so i will have to delay the temptation... james, you're taking all this really great. -bowerbird

3 2

Amazon "Send to Kindle"
by James Adcock 13 Jan '12

13 Jan '12

Amazon has created a new app for your desktop or laptop which makes it easier to send an (unencrypted) book file from there to your Kindle. Free if you use wifi. You do not have to save what you send in Amazon's cloud. Useful during book dev, or if you download a book from PG via your computer, and then want an easy way to transfer it to your Kindle. You don't have to do it this way of course. You can use a USB, or a local wifi server, or use the Kindle web browser to download directly from the web, or .. Just right click on a file and send. This solution is not useful for people who do not use Amazon/Kindle. http://www.the-digital-reader.com/2012/01/12/amazon-releases-sendtokindle/#m ore-28952

1 0

Re: [gutvol-d] book of james -- 010
by Bowerbird＠aol.com 12 Jan '12

12 Jan '12

james said: > I did some work for One Laptop Per Child. yes, i saw that. > http://www.archive.org/details/EBookEnlightenment haven't seen that yet, but i'll look at it when i get a chance. > The goal of this book is to get children and teachers > in poor countries to digitize their own books. right on. right arm. > Again, cheap and time-consuming > is better than expensive and efficient. but we can agree that cheap and efficient is _best_, yes? > Kid's labor is worthless i'm sure you didn't mean that the way it sounds... ;+) > and doing it the hard way > might even be educational. if you emphasize the "might"", i might agree. but it's beside the point, right? the point is that we do things that are educational, and whether they are easy or hard is immaterial. the important thing is that _we're_learning_. there's a time and place for _difficulty_ too -- when we want to instill capabilities for discipline and tenacity and persistence and self-confidence and asceticism and resolve and dedication and a sturdy work ethic -- so i have full appreciation for its qualities, but a love for difficulty _per_se_ is twisted. > (I'm assuming that the kid is > doing this because he *wants* to do it. > I don't advocate child labor). and thus you touch on what might well be _the_ most important issue -- motivation. i have found -- most especially for kids -- _a_sense_of_mastery_ is one of the crucial -- even _vital_ -- sources for motivation... if you feel like you're good at something, you are much more likely to want to do it. likewise, when you feel you're not good at something, you're unlikely to want to do it. so, do you feel you're a master of digitizing? or more like you've been beaten down by it? and another thing lurks right behind mastery -- so close it may be its twin, or its shadow -- namely _a_sense_of_fun_... it almost sounds redundant to say that people are much more likely to want to do things they think are "fun". the linkage is so tight it's almost tautological. similarly, if you're good at doing something, you're probably gonna come to think it's fun. and if you find something fun, you will do it, and the experience will lead to more mastery. so the intertwine of these three factors is quite compelling and creates a fine braid. and "difficulty" is not present in the picture. *** so, are there difficult parts to book-digitization? yes, without any question. but _on_the_whole_, if book-digitizing is hard, you're doing it wrong. and people who tell you otherwise... are wrong. -bowerbird

1 0

Re: [gutvol-d] book of james -- 010
by Bowerbird＠aol.com 12 Jan '12

12 Jan '12

james said: > I'm not exactly sure what > "start from scratch and do the job correctly" > would mean in my case. well, if it's any consolation, i didn't mean to imply that anything done "incorrectly" was _your_ fault... mostly this is a botched job by the internet archive. they did a lousy job of scanning the book, and then their stupid workflow went and sabotaged the o.c.r. so i consider you to be an innocent victim of them... the one thing you did do wrong was the rewrapping. but again, your guide in that was a misguided policy from project gutenberg, which _might__ have made some kind of sense back when it was established, but has outlived all of its usefulness by now, to the point it's a bad practice which can only be preached against. so, you know, once again, you are the victim here... having said all of that, though, my advice to you has not changed since i first gave it to you a while back, which is that you would be better off starting over... you could spend a little time cleaning up the scans, and then ensure that the o.c.r. gets done correctly, with the options specified intelligently, and output saved properly, and _still_ finish the thing _faster_ than continuing to work with the file as it now exists. (and that's _after_ we've both put in a bunch of work!) i'm just about done doing my part, so i no longer have a dog in this fight, so my advice is for _your_ benefit... nobody else here will care. they aren't being affected. but that's my advice. for you. and i feel some kind of urgency and obligation to tell everyone down the line to avoid a situation like this... because it simply doesn't need to be _this_ difficult... you're willing to take on the burden, and that's fine, but even then, it's my opinion that your time would be better spent doing four books the easy way than one the hard way. > I do not own and don't intend to buy ABBYY Fine Reader. i guess mr. haines didn't convince you with his argument that -- at a value of $50 an hour for your time -- it would _pay_ for itself very quickly... :+) truth is, even at $1/hour, it will pay for itself soon enough, if you think you're gonna keep doing more digitizations... you can also pick up a "used" copy at a really low price... and it's not just the money; it's your _sanity_ you're saving... but that's beside the point, too. there are enough people here that _do_ own finereader that you should be able to find one of them to do you the favor of doing the o.c.r. for you. you could do them a favor back... heck, if you woulda asked me, in the first place, i would have done it with my mac copy of finereader (which is way behind, yet would still probably give you better results than you have). but you wanted to pursue the file you had, so i went that way. if i had it to do over, however, i would reverse that decision... > I did attempt to make my own page-at-a-time OCR > with Tesseract yeah, tesseract is an even bigger waste of time and energy. > Obviously room for improvement, but > much better than what I started with. there might be different classes of trash, but it's all garbage. don't start comparing one pile of garbage to another one, or you're just gonna convince yourself to end up with garbage... > Putting in em-dashes and accents is painful, > but realistically what alternative do I have? re-doing the o.c.r., so those things are _retained_, instead of being thrown away. that is exactly what i am saying, james... you _did_ have an alternative. but you choose not to use it... because you didn't want to throw away the time you'd spent. so, instead, you will end up throwing away even more time, working on a decidedly-inferior file. i told you that goin' in, and now that i've spent a bunch of time with the file myself, i can confirm that my original advice was extremely sound... and i shoulda stuck with it, until i eventually convinced you. i took up the challenge because i knew i could learn from it, and i did, so it wasn't a waste of _my_ time. but i can tell you that you'll be wasting your time, doing unnecessary hard work. and let me reiterate: you _still_have_ that better alternative! you can still find someone to do the o.c.r. for you, and you can clean up the scans first, so they give you better results. and then, because you will not do the stupid stuff that was done by internet archive, you won't have to waste your time undoing the damage that they did. you still have a choice... and it's not just you, james. there are a whole lot of people who're beating their heads against the wall to digitize books, because they think they have "no alternative" to head-beating. > My understanding of what you're currently trying to do > is get my partially corrected text to match line-by-line > and page-by-page with the page images. i'm not "trying" to do it. i'm actually doing it. and i'm also doing _some_ cleaning of the text, and a lot of formatting. but you're going to have to do a whole lot _more_ cleaning... much of the work will be painful, and some of it unnecessary. oh, and just for the record, i have spent a good many hours getting those pagebreaks and linebreaks back like they were. it might not have taken you more than a minute to rewrap it, but it's taking me hours and hours and hours to undo that... it's not a job that i _like_ doing... in fact, i totally detest it... and the reason i hate it so much is that it is _unnecessary_... you should have never rewrapped that text before proofing. but let me tell you why i did that job. i did it for you, james. i did it because i knew that every hour i spent "fixing" that would save you 3-to-7 hours of proofing time, and _sanity_. so i _spent_ my own time, james, so you could save _yours_. (i wouldn't have done it if it would've been a 1-to-1 payoff; but with a 3-to-1 payoff, maybe even 7-1, i decided to do it.) so i think i've "earned" the right to give you some good advice. (as if i really had to earn something like that in the first place.) the important thing is that you learn that you should _never_ do a book this way ever again. it's simply not worth the work. > Once you have that I'm supposed to go through > page by page and change the text to look > exactly like the page images. i will have done most of that by the time i hand it over to you. once the original linebreaks are restored, it's a piece of cake. that's why i was able to use the text from the internet archive to show well-formatted model-pages right from the outset... > I add underscores for italics, right, and that's something you shouldn't have had to do, since abbyy does a good job of catching most of the italics. but then the internet archive went and threw them away, along with the em-dashes, which you must also restore... in the future, i'll make available an app that will go and grab that information -- it's sitting there, in one of the data-files at archive.org -- so people don't have to suffer because of a bad decision made outside of their control, and i could've helped _you_ to avoid that unnecessary work as well, but... one thing you will not have to do is to worry about diacritics. at least for the names you've done thus far, i can capture 'em, and make a global search-and-replace to put in the diacritics. but that's another thing you shouldn't have had to worry about, because i'm pretty sure abbyy can recognize those characters... > but I do not re-wrap the text, remove page numbers, > join paragraphs split between pages, or anything else > I've been doing so far. that's right. don't do any of that. the machine can do that. don't waste any of your time doing what the machine can do. > I just correct spelling and accents. no accents either. but it's more than just "spelling". indeed, the "spelling" part is the easy part, because the spellcheck routine flags most of the bad words. but where the bulk of the work exists for this text is in the correction of the punctuation. the scans were bad, so that means the o.c.r. of punctuation is bad... but the problem is exacerbated because the book was typeset very poorly, with lots of punctuation glitches... and the exacerbation continues, because many of the punctuation glitches _can't_ be found with algorithms. and, just as another example of a task that took hours, the doublequote checking was a bear, because the o.c.r. was bad, but -- even worse -- because the typographer made lots and lots and lots of doublequote mistakes... this includes a failure to mark continuing dialog with an open-quote at the start of subsequent paragraphs. so i installed most of them, to the best of my abilities, but it's very difficult sometimes to make that judgment when you're not actually absorbing the book's content. and this means i also introduced lots of doublequotes that were _not_ in the p-book itself, and you might be uncomfortable with that, and want to go delete them... (which is fine, by the way; i have no attachment to 'em; if you decide that's what you want to do, i can probably write a routine that could locate them rapidly for you.) but most of your work will revolve around punctuation. and whether you'll want to standardize inconsistencies in the book's typography, some of which are... weird... and this is picayune stuff. it's... _irritating_. not fun. now, this punctuation work would have been necessary even if you would have had reasonably good o.c.r. text. but after having done all the unnecessary work required, i'm not sure that i would have the stomach to be able to face the stresses of handling those types of judgments... so i suggest that you gird yourself... > After I've done all that I go back and put in > the ZML stuff like extra blank lines for page headings. again, most of that will have already been done. > Then the magic happens and we get several formats > of the book from one source. I'm not entirely clear > on that part. you don't need to be clear. you just need to click a button. -bowerbird

2 1

book of james -- 010
by Bowerbird＠aol.com 11 Jan '12

11 Jan '12

james- work is coming along fine on the file, but i'm sorry that i haven't gotten it turned back to you sooner than this... i got caught up in some other things, plus the clean-up is taking longer than i'd hoped it would, mostly because the typography of the book really wasn't quite up to snuff. in the meantime, you can always work on the family-trees. *** to make this post more widely applicable, i can probably reiterate the point i usually describe at the very beginning: take care to select a well-done book which will scan nicely. or, if you're using pre-existing o.c.r., make sure it's good. otherwise, you might well have to spend a lot more time. for instance, on this book which james is digitizing now, he faces the triple whammy -- poor p-book typography, bad scans, and internet archive's careless text handling (where the book is missing em-dashes, utf8, and italics). so even after all the time he and i have spent cleaning it, my guess is that he would still be finished with it sooner if he were to start from scratch and do the job correctly... -bowerbird

2 1