December 2011 - gutvol-d - lists.pglaf.org

Re: [gutvol-d] seriously, feel free to ask any questions
by Bowerbird＠aol.com 09 Dec '11

09 Dec '11

keith said: > For $100??? You can be serious! > Make that $1000 and I might think about it! really? you would think about going on a wild goose chase for $1000? well, ok, so would i. but how much would it cost to have you actually _go_ on that wild goose chase, and not just "think about it"? $10,000? $100,000? or what? the good news is that, if you want to code a text-to-html converter, you can do that _extremely_easily_ if you design a wise format, even if you are an unskilled coder like i am. anyway, if you did want to convert various light-markup formats and .html, you should check out pandoc, open-source from john macfarlane. > http://johnmacfarlane.net/pandoc/ here's the lede from that page: > If you need to convert files from > one markup format into another, > pandoc is your swiss-army knife. > Need to generate a man page from > a markdown file? No problem. > LaTeX to Docbook? Sure. > HTML to MediaWiki? Yes, that too. > Pandoc can read markdown and > (subsets of) reStructuredText, textile, > HTML, and LaTeX, and it can write > plain text, markdown, reStructuredText, > HTML, LaTeX, ConTeXt, PDF, RTF, > DocBook XML, OpenDocument XML, > ODT, GNU Texinfo, MediaWiki markup, > textile, groff man pages, Emacs org-mode, > EPUB ebooks, and S5 and Slidy HTML slide shows. > PDF output (via LaTeX) is also supported with > the included markdown2pdf wrapper script. whew! just _reading_ that makes me break out in a cold sweat and worry about fever dreams... john macfarlane is a professor of philosophy at u.c. berkeley, so he can keep all of this straight. mere mortals should just use pandoc instead... -bowerbird p.s. be forewarned that there are a half-dozen versions of markdown, all of them just _slightly_ different...

2 1

Re: [gutvol-d] seriously, feel free to ask any questions
by Bowerbird＠aol.com 08 Dec '11

08 Dec '11

because, you know, if this kind of text-to-html thing really _was_ "impossible" -- like you were once told -- then it seems we're now doing something "impossible". and you should be asking questions, like "how did you do that?", and "what kind of magic are you using there?" but i think we all know that this isn't really "impossible". and it never was. ok, i suppose before .html was invented, people would have looked at you strange if you would have told them that they could "convert" plain-text to "html", but still... *** indeed, you know that #013 post i sent out this morning? it had this in it: > http://en.wikipedia.org/wiki/triskaidekaphobia now, i didn't specify that u.r.l. as a link. i didn't create any "markup" around it, or anything, i just copied it in. but take a look at that message on the pglaf website: > http://lists.pglaf.org/mailman/private/gutvol-d/2011-December/008447.html what you will see is that the u.r.l. to the wikipedia page was turned into a link. you can click on it, and it'll work. that is, [a href=] anchor markup was inserted around it, so that your web-browser would treat it as an active link. so _that_ is an example of a text-to-html conversion! indeed, the entire plain-text contents of that e-mail to this listserve were converted from text into .html, so that it could be mounted on the web. imagine that! and not only that, but the entire plain-text message of _every_ e-mail to this listserve are _routinely_ converted into .html, so the post can be mounted in the archive... do you want to have to code your e-mail in .html format? heck no! but do you want clickable links? heck yes! do you want archives available on the web? heck yes! and -- ever since august 2004 -- the archives _have_ been available on the web... (due to some sloppiness, the archives between 2006 and 2009 were _lost_, but e-mail during that time had been converted to .html.) the package which does this conversion is "pipermail" (the mailman edition), and it's open-source software. pipermail includes other "conversion" routines too... > for instance, because of the right angle-brackets > that precede each line here, these lines will be > rendered in italics, because the conversion routine > presumes this should be treated as a blockquote, > because that's a common convention in e-mail apps. so it's not as if this is some exotic new capability which has only recently become manifest in our consciousness. on this very listserve, where people were insisting that conversion from plain-text into .html was "impossible", _the_archive_itself_ proved those people were _stupid_, and it used their own ignorant e-mails as its "pudding". so yeah, i think that all of this is fairly _obvious_, if you just think about it for half-a-second or so, but _really_, if you have _any_ questions, do feel free to ask them, ok? -bowerbird p.s. if you're interested in a rst-2-epub converter, try: > https://github.com/mattharrison/rst2epub2

4 4

a review of some digitization tools -- 014
by Bowerbird＠aol.com 08 Dec '11

08 Dec '11

ok, to review, let's consider the list of structures that we found in the set of 4 books digitized by jim adcock. they were: > a. headers -- [h1]-[h6] > b. paragraphs and breaks -- [p] and [br] > c. styling -- [i] and [b], plus cover-page prettyness > d. blocks -- [pre] and [blockquote] > e. horizontal rules -- [hr] > f. images -- [img] > g. links -- [a name] and [a href], t.o.c., and index so, we've covered headers, paragraphs and breaks, styling, blockquotes, horizontal rules, and links... which means that we just have to take care of images, and we've handled our first set of "simple" structures. so i inserted some images, just for fun... for good measure, i threw in a poem as well, and a little index too, so i could do the coding for those... the index is a "mere" search routine, but it's useful because it reveals important keywords in the book, with an explicit list of the number of hits returned... (i'll do more work on indexing, but this is a first pass.) the index functionality required that each paragraph be addressable, so now each "chunk" has an .html "id". you'll find all of this new stuff at the end of the book. z.m.l. input file: > http://zenmagiclove.com/grapes009.txt python script to create .html output: > http://zenmagiclove.com/tday2011.py -bowerbird

1 0

seriously, feel free to ask any questions
by Bowerbird＠aol.com 07 Dec '11

07 Dec '11

i said: > http://zenmarkuplanguage.com/grapes008.txt ... > http://zenmagiclove.com/tday2011.py ... > it shouldn't be too hard for you to figure out what this > script has done, and how you would replicate the code. > but if you have questions, please feel free to ask them... seriously, feel free to ask _any_ questions you might have. because, some 6-8 years back, people on this listserve were _insisting_ that what i'm doing here is "impossible". they said it was ludicrous, evidently so much so that they did not even feel any need to refute the arguments, but instead went straight to lambasting me for proposing it... hey, they went so far that they thought a mere _mention_ of z.m.l. was enough to put a smirk on the reader's face... "prove it", they'd say to me, thinking it was a "challenge", as if the taunt somehow framed their technical prowess... all it did was show how little they knew, and how reckless they were, just throwing away every bit of their credibility. so i let them bet all of it, knowing that i'd win the hand... sure enough, over time, they grew more and more quiet, as i systematically doled out piece after piece of evidence. until eventually, a few years ago, they became fully silent, as i _kept_ showing y'all one demonstration after another, right up to today's python script. impossible? yeah, right. so here's your "proof", you stupid idiots; it was right there, in the pudding, just like i was always telling you, all along. and the silence i hear in return is _deafening_... -bowerbird

2 1

a review of some digitization tools -- 013
by Bowerbird＠aol.com 07 Dec '11

07 Dec '11

http://en.wikipedia.org/wiki/triskaidekaphobia -bowerbird

1 0

a review of some digitization tools -- 012
by Bowerbird＠aol.com 06 Dec '11

06 Dec '11

now you have examined our current sample book: > http://zenmarkuplanguage.com/grapes008.txt i've used the ".txt" extension on that file, but really it has been marked-up to satisfy the rules of z.m.l. -- also known as "zen markup language", my baby. one common aspect of text-files is that paragraphs are separated by blank lines. z.m.l. takes it farther, mandating _all_ structural elements be so bounded. this makes it easy for you, as a user, to "grok" how text will be handled in z.m.l. automatic processing. every "chunk" which is surrounded by blank lines is treated as a distinct unit that's virtually independent. here's a script that converts our sample to html5: > http://zenmagiclove.com/tday2011.py to make it easy to see the distinct "chunks" of the text, the script alternates background-color for each chunk. it also numbers the chunks, and shows a visual display of the .html coding which it has inserted into the text, to give you an idea about the processing that it's doing. it shouldn't be too hard for you to figure out what this script has done, and how you would replicate the code. but if you have questions, please feel free to ask them... if you examine this html-book, you'll see that it contains everything we'd expect, plus a few things we might not... for instance, in addition to the linked table-of-contents, chapter-headers also link back to the table-of-contents. there is also, above each chapter-header, a pair of links which navigate to the previous chapter and the next one. i consider these navigational capabilities to be essential, but i think they are not present on any p.g. html versions. the book also has curly-quotes, typographic em-dashes, indented paragraphs, among other customized settings. tomorrow we will begin ramping up the difficulty-level, with the strategy of introducing one structural element each day, until we are handling everything we will need. this gradual build will show how _easy_ this task can be, if we just approach it the right way with a good mindset. to that end, if you'd like me to use any particular book you believe will present some "challenge" to my system, go ahead and post a message pointing me to that book. this approach will serve to dispense the silly nonsense that z.m.l. can only be used to handle "simple" books... -bowerbird

1 0

Re: [gutvol-d] a review of some digitization tools -- 011
by Bowerbird＠aol.com 05 Dec '11

05 Dec '11

i've discussed how many of the digitization tasks can be accomplished in a regular wordprocessor... so, if you're on a mac, and you're interested in this, you might want to check out some recent threads on the d.p. forums, where a volunteer named "ernie" is posting some automator and applescripts he uses... ernie's user-name is "eschaal". here's a thread where some discussion is happening: > http://www.pgdp.net/phpBB2/viewtopic.php?t=2586&start=495 roger frank and the world-famous lucy24 have joined in... here's the other thread: > http://www.pgdp.net/phpBB2/viewtopic.php?t=49617 ernie has also set up a wikipage: > http://www.pgdp.net/wiki/User:Eschaal/PPTools/Mac' -bowerbird

1 0

a review of some digitization tools -- 010
by Bowerbird＠aol.com 02 Dec '11

02 Dec '11

ok, let's regroup... you will recall our three overarching requirements for the process of digitizing an existing paper-book: 1. obtain the text (via scanners, and o.c.r. software) 2. clean the text (the first set of tasks for our tools) 3. turn the text into e-books (the second set of tasks) the first requirement is largely taken care of, thanks to the big scanning efforts of google and internet archive. it keeps getting harder to find a book they haven't done. under the second of those requirements, we covered: 2a. do a spellcheck 2b. fix spacey punctuation 2c. restore styling, e.g., italics all of these tasks can be managed with a text-editor, whether offline or online. nothing here is "difficult"... so we're working on the tasks for the third requirement: 3a. tagging the structural aspects of the text 3b. converting the text into .html (intermediate and final) 3c. converting intermediate .html into e-book output-files tasks 3b and 3c will be the product of automatic processes, so our biggest task is 3a, the tagging of structural aspects... we have reviewed 5 simple books, to see what they involved. a. headers -- [h1]-[h6] b. paragraphs and breaks -- [p] and [br] c. styling -- [i] and [b], plus cover-page prettyness d. blocks -- [pre] and [blockquote] e. horizontal rules -- [hr] f. images -- [img] g. links -- [a name] and [a href], mostly t.o.c., but also index this is a bare-bones list, to be sure, but it's also the case that a surprisingly large number of books only require bare-bones. i believe internet archive currently claims ~3 million volumes... i'd say that 1 million of those books only require bare-bones, while another million of 'em would need only a little bit more. and even the final million are not _that_ difficult, to be honest, not if you're willing to bend a little on what you want from 'em. and if you won't bend, nobody will be able to make you happy... so, starting next week, i'll show you how you can tag structure. we will review how various tools can help us, offline and online, and i'll show you how easily i can build a tool that does the job. we'll start with the simple structures we already found, above, and then work in any additional structures we learn we need... to that end, if there are some books you want to point me to -- with complicated structures -- do please feel free to do so. it might take a little time to work up to the complicated books, but there's no reason we can't put 'em on the table right away. have a nice weekend! :+) -bowerbird

1 0

Re: [gutvol-d] a review of some digitization tools -- 007
by Bowerbird＠aol.com 02 Dec '11

02 Dec '11

keith said: > Then you disclaim that your idea do have merit! ;-)) jim is like a hyperactive clock that's running backward... he's correct several times a day, completely by accident. but nobody, including jim, could tell you when that'll be. > Talking, about discussing things hypothetically. exactly! see what i mean! pretty pointless, isn't it? :+) > I will forgive you here, as I believe you do not > have a formal education in Computer Science. and i'll forgive you here, for thinking that matters... > I am home and whether you believe me or not I think code! > 99% of the time my code works out of the box. > An important paradigm in CS is think first, code later. ok, well then, when you think the code you are thinking can be written, and you are ready to write it, and then ready to have it evaluated as to whether it works or not, you let us know, ok?, and we'll be ready to evaluate it... because i'll let you evaluate my code, to see if it works... in the meantime, there's not much point talking about it. because, in the end, it will either work right, or it won't... the proof is in the pudding, keith, right in the pudding... > So, extensive light-mark-up is NOT heavy mark-up. then? oh, yet bet your sweet ass that light can become heavy! and if you want the proof of _that_, go look at ascii-doc. even restructured-text, in marcello's hands, is too obese. it takes real discipline to keep it light. but i can do that. > Then using XML is light-mark-up then? no, those angle-brackets all by themselves make it heavy. anything that makes you press the caps-key is oppressive. > Or those features you have proposed are not a minimal ! minimal work. maximum power. that's the sweet-spot, keith. > Can be is hypothetical! can be can be real as well, if you decide you will make it real. > C'mon, you know better than play games with me! after all these years, i certainly _should_ know better, right? perhaps those cyberarchaeologists will think i am dense too, just for continuing to persist in conversation with all of you... and thus, keith, you have given me something to think about, so i will reward you with my $2 prize for positivity. ironic, eh? -bowerbird

3 2

a review of some digitization tools -- 008
by Bowerbird＠aol.com 02 Dec '11

02 Dec '11

one more review of a book digitized by jim adcock... this one is pg#32540, "my first summer in the sierra", by john muir, thus of interest to a great many people. as in the previous books, we have the standard tags, for paragraphs and breaks, headers and t.o.c. links, formatting of front-matter (e.g., sizing, centering), horizontal rules, and italics within the body-text... we also have those div's with a class of "blockquot", plus other cruft introduced by some tool, i'd guess. other than that, there are a few new additions here... one is that jim marked the p-book page-numbers... he did this in at least one of the previous books, but i forgot to mention it then. in this book, however, he also included the _index_, and did a nice job of linking the index-entry page-numbers to the relevant page... so that's a pleasant advance in this e-book's usability. (one brief historic note: i had to fight like the dickens against people here who _insisted_ that pagenumbers were "unnecessary in an e-book", and must be deleted. i was right, and they were wrong, and i'm so glad that more and more postprocessors are agreeing with me, as time goes on, including ones like jim adcock here.) as you might imagine, the making of those index links entails much markup, so it's a task we wanna automate. jim has indicated that he uses regular expressions a lot, so i'd guess that's how he tackled this job for this book. but he might've used some other tool to do it instead, an app like calibre, for instance, although i don't know which of the available tools can handle this exact task. whatever the case, we'll want our tool to be able to do it. you'll see that jim uses some _table_ markup in here too, for the table of contents and the list of illustrations too... again, there will be some people who call this "tag abuse", and who'll argue that table markup only be used for tables. and they also insist that a table of contents is _not_ a table! isn't that just too rich? i couldn't make this stuff up, folks. but i regularly tell those people to zip it, and i do that here. as long as this markup does what jim wants it to do (and i'll assume it does) across those machines he wants to support, i am fine with it, even if it fails some definition of "purity"... jim also used those .css spans to create custom left-indents, both for poems and index-entries. i still don't prefer it, but it works, and that's the most important consideration of all. oh hey, i see that those div's marked as "poem" and "stanza" are not really poetry, but rather just a transcriber's note, so yeah, i guess that would have to be considered "tag abuse"... they could just be labeled, accurately, as "transcriber-notes". once more, a pretty simple book, all things considered, so i'm starting to wonder why jim has argued in the past that this digitization stuff is really hard for a person to perform. perhaps he is tripping over his own feet... -bowerbird p.s. here are examples of all the types of tags jim used... > [p] > > [br] > > [p][span class="pagenum"][a name=][/a][/span][/p] > > [p class="center"]Sierra Edition[/p] > > [p class="figcenter"] > > [i] > > [div class="blockquot"] > > [h1]MY FIRST SUMMER IN[br][br]THE SIERRA[/h1] > [h2]THE WRITINGS OF JOHN MUIR[/h2] > [h3][b]FROM SKETCHES MADE BY THE AUTHOR IN 1869[/b][/h3] > [h4]THE SIERRA CLUB OF CALIFORNIA[/h4] > > [hr] > [hr style="width: 45%;"] > [hr style="width: 65%;"] > > [a href="#Page_171"]171[/a] > > [span style="margin-left: 1em;"]a nerve-trying experience in, -;[/span] > [span style="margin-left: 1em;"]sunrise in, ;[/span] > [span style="margin-left: 1em;"]thunder storm, , ;[/span] > [span style="margin-left: 1em;"]grandeur, , , .[/span] > > [span style="margin-left: 2em;"][i]From a photograph by Herbert W. Gleason[/i][/span][br][/div] > [span style="margin-left: 2em;"][i]From a photograph by Charles S. Olcott[/i][/span][br][/div] > > [span style="margin-left: 3em;"], , , , , , ;[/span] > > [div class="centered"] > [table summary="CONTENTS" border="0" cellpadding="2" cellspacing="2" width="100%"] > [tbody][tr] > [td align="left"][span class="smcap"]The Yosemite Falls, Yosemite National Park[/span][/td] > [td align="right"][a href="#Frontispiece"][i]Frontispiece[/i][/a][/td][/tr] > [/tbody][/table][/div] > > [div class="centered"] > [table summary="CONTENTS" border="0" cellpadding="2" cellspacing="2" width="100%"] > [tbody][tr] > [td align="left"][span class="smcap"]The Three Brothers, Yosemite National Park[/span][/td] > [td align="right"][a href="#photo7"]208[/a][/td][/tr] > [/tbody][/table][/div] > > [p class="figcenter"] > [a name="map" id="map"][/a] > [br] > [img src="images/maps.jpg" alt="Map" title="" height="601" width="600"][br] > [a href="http://www.gutenberg.org/dirs/3/2/5/4/32540/32540-h/images/mapl.png"][small]Larger view[/small][/a][/p] > > [div class="poem"][div class="stanza"] > [span class="i0"]The following changes were made:[br][/span] > [span class="i2"]p. [a href="#Page_156"]156[/a] Hoffman and Cathderal Park to Cathedral[br][/span] > [span class="i2"]p. [a href="#Page_166"]166[/a] only curly currus wisps scarce to cirrus[br][/span] > [span class="i2"]p. [a href="#Page_171"]171[/a] most of their acomplishments to accomplishments[br][/span] > [span class="i2"]p. [a href="#Page_267"]267[/a] Adenostema fasciculata to Adenostoma[br][/span] > [span class="i2"]throughout text: chamœbatia to chamÊbatia[br][/span] > [/div][/div] > > [div class="poem"][div class="stanza"] > [span class="i0"]foothills and foot-hills[br][/span] > [span class="i0"]goldenrod and golden-rod[br][/span] > [span class="i0"]rootstocks and root-stocks[br][/span] > [span class="i0"]thunderstorm and thunder-storm[br][/span] > [span class="i0"]gristmill and grist-mill[br][/span] > [span class="i0"]lifelong and life-long[br][/span] > [span class="i0"]overleaning and over-leaning[br][/span] > [span class="i0"]snowstorm and snow-storm[br][/span] > [/div][/div] > > [h2]CONTENTS[/h2] > > [div class="centered"] > [table summary="CONTENTS" border="0" cellpadding="2" cellspacing="2" width="70%"] > > [tbody][tr] > [td align="right"]I.[/td] > [td align="left"][span class="smcap"]Through the Foothills with a Flock Of Sheep[/span][/td] > [td align="right"][a href="#Page_3"]3[/a][/td][/tr] > > [tr] > [td align="right"]II.[/td] > [td align="left"][span class="smcap"]In Camp on the North Fork of the Merced[/span][/td] > [td align="right"][a href="#Page_32"]32[/a][/td][/tr] > > [tr] > [td align="right"]III.[/td] > [td align="left"][span class="smcap"]A Bread Famine[/span][/td] > [td align="right"][a href="#Page_75"]75[/a][/td][/tr] > > ... > > [tr] > [td align="right"]XI.[/td] > [td align="left"][span class="smcap"]Back To the Lowlands[/span][/td] > [td align="right"][a href="#Page_254"]254[/a][/td][/tr] > > [tr] > [td align="right"][/td] > [td align="left"][span class="smcap"]Index[/span][/td] > [td align="right"][a href="#Page_267"]265[/a][/td][/tr] > > [/tbody][/table][/div]

2 1