December 2011 - gutvol-d - lists.pglaf.org

a review of some digitization tools -- 011
by Bowerbird＠aol.com 13 Dec '11

13 Dec '11

ok, here we go, in our series on digitization tools... we're concentrating now on the _tagging_ part of the digitization process. we have cleaned the text and marked the italicized words in previous steps. so we need some sample books to work with now... you might recall that i used "books and culture" in our "sweet grapes" series back in september, when discussing the clean-up phase, so we can now use that e-text to demo the next phase in our process. the file, after the bulk of the clean-up, was here: > http://zenmarkuplanguage.com/grapes006.txt we then did a transform of that file, to get this: > http://zenmarkuplanguage.com/grapes007.txt and now, after yet another transform, we have this: > http://zenmarkuplanguage.com/grapes008.txt all the versions are more-or-less reminiscent of: 1. the output you could expect from abbyy o.c.r. 2. the text you find for each book at archive.org 3. a c.t.f. (concatenated text file) from pgdp.net 4. the .txt format utilized by project gutenberg to that extent, then, we're on very familiar ground. still, i encourage you to examine "grapes008.txt" closely, so you know i have nothing up my sleeve. here are some of the notable things you might see, which adhere to the rules of zen markup language: 01. headers are preceded by at least 4 blank lines. 02. headers are followed by exactly 2 blank lines. 03. structural elements are bounded by blank lines. 04. paragraphs are not indented (in the input file). 05. the linebreaks are consistent with the p-book. 06. linebreaks don't necessarily need to be retained. 07. end-of-line hyphens are considered to be "soft", 08. but a tilde indicates a preceding hyphen is "hard". 09. italics are indicated by surrounding underscores. 10. lines beginning with " > " indicate a blockquote. 11. the first section is considered as "the title page". 12. the table-of-contents must be the second section. and again, none of this "tagging" would be hard to do in a typical text-editor, or your favorite wordprocessor. this, again, is one of the bare-bones books that has practically no structural elements aside from headers, front-matter, and plain old paragraphs in the body... the only other structures here are a few blockquotes. so take a good look at that file... -bowerbird

5 26

Re: [gutvol-d] seriously, feel free to ask any questions
by Bowerbird＠aol.com 13 Dec '11

13 Dec '11

james said: > I'm pretty happy with the way the web page came out. ok, i'm unclear... are you happy with the .html version mounted at p.g.? are you happy with the p.g. autogenerated .mobi file? are you happy with the p.g. autogenerated .epub file? if you're happy with all three, then the case is closed. if you're unhappy with any, then we should pursue it. :+) -bowerbird

3 4

Re: [gutvol-d] a review of some digitization tools -- 017
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

alex said: > I suppose there's no way to get the source for that .py? the best way in the world to get it is to write it yourself, alex. i'll be happy to answer every question you have along the way. here's a push to get you started: 1. read in the .zml file. 2. separate text into chunks, splitting on double-linebreaks. 3. examine each chunk to determine its structure, and tag it. 4. ... 5. wrap each chunk with its appropriate markup, and output. (you'll find there is a step necessary between step 3 and step 5, which is why i labeled it thus, but don't worry about it at first.) this is not a hard program to write. i rewrite it often, for _fun_. and for my own education. each time i try to write it differently. this time i am using python, with .html5 as the output format... doing it time after time teaches me about the hidden "gotchas", and allows me to locate the most streamlined and graceful path. but it's also taught me the value of actually _doing_ the coding, rather than just picking up code from somebody else to run it, which is why i am disinclined to simply hand it to someone else. i worked very hard, for a good many years, to make this simple. so i need to see a similar willingness to work, that i can reward. -bowerbird p.s. i started the current python script on thanksgiving day -- hence its name -- when i was stranded without my computer (and thus had to start from scratch), while waiting for turkey. so i can say, with great certainty, it doesn't take a lot of time.

1 0

a review of some digitization tools -- 017
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

ok, are you ready to go back to work? :+) our time using "books and culture" is coming to an end. it's a very simple book, and we've squeezed all the instructional juice out of it... so for today, we'll just delete the colorization which we were using to separate the "chunks", and remove the "tags" that we were displaying. so our output will start looking like it "should". we will also ensure that we achieve "validation", because we wouldn't want marcello to scold us. i've also put in a "width" component, because lines that run the width of a browser-window give me the creeps. i've got a 23-inch screen, and no line has any business being that long... finally, as "an exercise", we'll borrow some .css that was used in the .html version of this book when it was mounted up on project gutenberg, as pg#16736, to see how closely we match that. that p.g. html uses block-style indents _and_ paragraph indents, which some folks consider to be weird, but i'm ok with it (in general) and my object now is to clone their look-and-feel, so that's what i'm gonna do, just so you know. ok, so here we go... z.m.l. input file: > http://zenmagiclove.com/grapes009.txt python script to create .html output: > http://zenmagiclove.com/tday2011.py i guarantee this week is gonna be interesting, because we'll be doing some _comparisons_... also, now that we've built our "engine" on the "simple" stuff, we'll be able to add the rest of what we'll need to handle complicated books, and that'll make this series more interesting. -bowerbird p.s. resending this hours after the first time, as the system appears to have swallowed it...

2 1

seriously, feel free to ask any questions
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

james said: > Bowerbird, Well, you've given me some things to think about. good deal. glad to hear it, james... :+) -bowerbird

1 0

Re: [gutvol-d] seriously, feel free to ask any questions
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

james said: > On the Kindle I formatted it like this: > > _Krishna:_ > > Some damsel I saw, supremely fair-- > A moon unstained, that slowly rose, > Or a golden vine. yes! that's exactly what i woulda suggested you do -- move the name up, to use the horizontal space better. but i would've done that for the _web_ version as well. because more and more, the web is being accessed by people with smaller screens -- ipads and iphones, etc. and -- especially when it comes to _reading_ -- they increasingly use a portrait orientation, not landscape. so the old assumption of wide-screens is crumbling. thus, even for the version that's "for the web", it'll be increasingly wise to use that horizontal space wisely. take it from me, i know the sense of accomplishment and feeling of mastery that a person attains when we are able to "make an e-book look like the print-book", because i have _experienced_ those feelings myself... so i'm not downgrading them, not in the slightest... but these days, there's almost no real payoff in that. if anyone wants to "see what the print-book looks like", the scans are online, just as available as your e-books... for instance, here's the page with the text you quoted: > http://www.archive.org/stream/cu31924011697525#page/n21/mode/2up (humously, archive.org has the recto/verso mixed up.) just try to provide such a focused link -- targeting a single poem -- in the versions which you uploaded... there is no payoff in having you _replicate_ something that's online already, in a more-or-less "original" form. indeed, as you experienced, if there is any payoff at all, it is a _negative_ one, regarding the generated e-books. if you make your master-format "look like" the p-book, your generated e-book outputs _might_ look like crap... (or they _might_ look and work just dandy, which would be the type of situation we will be aiming for, but if we must change something, change "look like the p-book".) > I will know to avoid "span" in my web pages > in the future, and to put in the special style > that prevents kindlegen from indenting > the first line of every paragraph, etc. > I am gradually learning the tricks. that's good. tricks are good. :+) the thing is, the e-book environment is still shifting. the ground isn't steady. so a trick you learned today might not work tomorrow. it might even back-fire... which makes you look bad. that's just the way it is... > However, the reformatting is not something > I could have done without having two different > HTML versions, and the Kindle really needed it. well, james, if you are willing to change your "web" version in the way that i suggested up above, then you might well be able to do everything very simply. that's what i will be willing to show you, if you like... > So archive.org has a nice Kindle version, > and the Kindle Store, but PG (which has by far > the most downloads) has a disappointing one and > the Kindle Store will have two disappointing ones > competing with my good one: one by Amazon and > another by some reprobate who seems to make > a quick and dirty version of everything released > by PG before Amazon makes their free version. the way i would suggest doing it is to use light-markup. and then _generate_ the .html version -- automatically!, at the same time _generating_ the .epub and the .mobi, and a nice .pdf version as well. less work, better results. more fun, and with faster throughput, to do more books. moreover, when the rip-off publishers rip off your .html, they'll rip off your .mobi and .epub versions as well, and at least the book (and maybe your name) will look good. > So it would be nice to have the option of preparing a > hand formatted version for EPUB and MOBI so when Kindle > owners grab them the results are as good as possible. yeah, except that approach then requires _everyone_ to "hand format" an .epub and a .mobi, and they won't do it, or they won't do it well, so that's the wrong path to pursue. far better to make it _easier_ on everyone, in the first place, and get dependable high-quality versions across the board. -bowerbird

3 2

Re: [gutvol-d] a review of some digitization tools -- 011
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

keith said: > Jim's world is not bizarre to me. > It is more a philosophy thing. i understand what you're saying here, keith, i do. but i'm not sure that you understand just exactly how retarded the discussion is that you're having. admittedly, i'm missing something, since i am only reading _your_part_ of the conversation, and not the parts "contributed" by jim and lee. but step back for a moment and get the picture... you're talking about the _table_of_contents_, man! what is there to talk about? i mean it. _seriously_. and you're saying stuff like "the t.o.c. should link to the chapters in the e-book"... well, um, yeah. d'oh. was there ever any doubt about that? _any_at_all_? i'm guessing that even _jim_ agreed with that part, albeit without ever acknowledging that you wouldn't have had to "clear up that point" if jim hadn't _first_ said something so convoluted that you really _did_ feel a need to restate the obvious to the stupid idiot. but, jeez, keith, if you're gonna do that, at least have the decency to _call_ him "a stupid idiot", if only so the rest of us know that you're still aware we are not so dumb that you need to remind _us_ of the obvious. otherwise, we might think you are as stupid as he is... -bowerbird

3 2

a review of some digitization tools -- 011
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

keith said: > O.K. let's take this slowly so that > we are all in the same context. keith, welcome to the bizarre world of an extended discussion with jim adcock. as a recent escapee from bizarre-world, i'm not sure it's a place you should go... you are now sounding as twisted as jim. everyone knows what a "table of contents" is. there's really no need to discuss it. _really_. even with the confusion of e-book formats, the correct course of action is quite clear... (even if p.g. didn't follow it well in the past... last i checked, marcello finally figured it out.) anyone else, anyone else at all, think there is any need to discuss anything about the t.o.c.? keith, if no one else weighs in, take the hint... -bowerbird p.s. speaking of which, it'd appear lee and jim are having a great time discussing my threads. i'm sure _that's_ a model of clarity to emulate!

3 2

reflections on the nature of argumentation
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

keith said: > it truly shows that he has > no true argument that is decently valid one more thing here before i leave this all behind... no one here has spent more time over the years entertaining jim and his stupidities... like i said, jim gets stuff "correct" at times, but there are also _many_ other times where he is off the wall... which is probably why the vast majority of you simply ignore him, and have done so for all of the years he has been on this listserve. any cursory examination of the archives will clearly reveal that i am the person who has responded to most of jim's messages. indeed, if my memory serves me correctly, i would say that the only other people who've engaged in any extended interaction with jim on _anything_ approaching "a regular basis" have been keith and michael (who was totally exasperated by jim's posts)... indeed, i'd be willing to bet that _all_of_you_combined_ have responded to jim fewer times total than i have all by myself... i chose to converse with jim since i was attempting to stimulate _any_ kind of dialog here, with _hope_ it would turn interesting. but, as everyone saw, a dialog with jim was far from interesting. it was usually a confusing mess, maddening in its inconsistency. a good dialog is like an extended rally in tennis, where each of the players executes clean return of the ball to the other's court. i was constantly baffled when having dialog with jim, because all of a sudden he'd make points for _my_ side of the debate, while giving the full impression he was arguing _against_ me! it was as if i was playing tennis against someone and suddenly i turned around and found him on _my_ side of the net, hitting the ball back to the empty space on the other side of the court! it woulda been totally bewildering if it wasn't so darn hilarious! (even i am forced to concede that jim _amused_ me quite often, and the arguing-both-sides tack was one of the funniest of all.) at other times, it was as if we were both playing on court #1, but jim was hitting his serves into court #2, or even court #3. and then acting like he had hit an ace! really, jim? seriously? what are you supposed to do when someone "plays" like that? so i was constantly asking myself whether the guy was _really_ that stupid, or if he was just _acting_ that stupid to irk me off. because there coulda been a certain method to his madness. as i've said, there are lots of people who have kill-filed me... so the only things that they "know" that i've supposedly said are the misinterpretations which other people feed to them. so i needed to consider a possibility his crap was deliberate. if he could give the _impression_ that he was defeating me to his audience, then they would actually believe he _was_. there's an old t.v. show called "three's company", where the landlord of an apartment thinks a male occupant, "jack", is _gay_, which is why it's acceptable he's living with two girls. but the guy isn't really gay, they just told the landlord that. still, the landlord _misinterprets_everything_ that happens within his mistaken belief that jack is gay, so he constantly "accumulates" more and more "evidence" that jack _is_ gay, thereby, of course, strengthening future misinterpretation. this is one of the main sources of the comedy in the show. as the audience is wise to the fact that jack is _so_ straight. we know he's mistaken, but we see it "makes sense" to him. and it's just as silly to see the same thing happening here. there is this one group which sees me as "a troll", and they are constantly accumulating more "proof" that that's true, because their "evidence" continues to show that i'm a "troll", thus their misimpression has become truly self-sustaining, even though people who read all posts know that it ain't so. so it could be at least plausible jim takes advantage of that. but after a while, it became clear that it really doesn't matter if jim is "really" that stupid, or if he's just "acting" that stupid. either way, the stuff that comes out of his mouth is _stupid_. so i just decided to stop rewarding that with my own dialog. and now that i've stopped amusing him, jim knows that he has nothing left here anymore, so he's whining about that. he says i'm "not man enough" to respond to his sniveling... hey jim, i _tried_ to dialog with you, for _years_. i gave up. i was the only one who recognized jim on a regular basis, and now i've stopped, and he realizes he has nothing left. tough luck, jim. you used up your last lifeline. so you die. and i can already tell y'all that my decision is a _good_ one. which i always knew it would be, since every other time that i've put someone in my kill-filter, i've been happy that i did, and wondered why it took me so long to come to my senses. i'd guess jim will continue to whine for a while. but i'm done. -bowerbird

5 12

Re: [gutvol-d] TOC in EPUB
by Bowerbird＠aol.com 12 Dec '11

12 Dec '11

keith said: > Please excuse if this seems OT, but we are > in general talking about creating ebooks. more on the table-of-contents? _more?_ really? it's not "off-topic", but it's a big waste of time... > If I understand you correctly, the NCX is being > deprecated/susperceded for the HTML 5 <nav>-tag. oh no. now you're off lost in the swamp of .ncx? no wonder you're bogged down and wasting time. > Now, I can see the elegance of the approach, but > isn't this going to cause a big mess when authoring > an ebook and cause older EPUB not to function properly. keith, i wrote a reply to you a month or two ago, when we were talking about .epub. i held it back, because i thought it was probably _too_ blistering, but in retrospect, i darn well should have posted it. because you still do not seem to realize that i.d.p.f. uses .epub for the _purpose_ of "causing a big mess". the corporate dinosaurs that compose the i.d.p.f. are doing whatever they can to retard a movement toward electronic-books, because they realize it will kill them. they _want_ e-books to fail. or at least to slow down. so they're constantly mucking stuff around, _hoping_ that they can confuse people enough, and make the e-book experience so painful, that we will "give up" the revolution, and let them continue to rob us blind. don't fall in their trap. > How should I deveolp a new ebook. > 1) still use a NCX > 2) just use the HTML5 <nav>-tag > 3) use both methods you forgot choice 4 -- throw out the corrupt bastards, and use an easy-to-author e-book format that works. > This may seem as a trivial question, yet > is important expect an ebook to work with > different ereaders or develop a tool. the corporate dinosaurs don't want e-books to work. they certainly do not want anyone to "develop a tool". that's why they keep changing the rules of the game. the only way to win is to overturn their little chess-set. even as it is, they've managed to put blinders on us. electronic-books used to _enthrall_ people, because the opportunities that are created seemed _endless_. remember when we used to let our imaginations roam, and we thought of wonderful things e-books could do? now we actually _have_ much of the infrastructure that we need to make many of those dreams come true -- such as handheld devices which are very powerful, and widespread and persistent wireless access to the internet -- but are we acting to instantiate those dreams in reality? well, no, we're not, because we're bogged down in useless bullshit like figuring out how to make a table of contents... why did someone constrain our vision? (the answer, like always, has to do with money.) and why did we allow that? if we do not act, we are acted upon. it's time for us to act. > Otherwise there will ebooks to redo and redo! until you give up. or get smart. which one will it be? -bowerbird

2 1