- gutvol-d - lists.pglaf.org

Re: [gutvol-d] seriously, feel free to ask any questions
by Bowerbird＠aol.com 09 Dec '11

09 Dec '11

keith said: > Lee want substantially identical HTML. > That is a completely different animal. i don't know that much about lee's proposal, and i don't intend to do anything to find out. from the little bit of material that you quoted, i got the impression he will provide four files that _he_ will create (i.e., probably garbled), from which he expects you'll generate output that will match a criterion which _he_ makes (i.e., probably one that's chock-full of quirks), and if you "succeed" -- in _his_ estimation -- he will give you the "prize" of a hundred bucks. he wants to give you 4 separate piles of garbage, get one brick of purified gold back from you, and then pay you peanuts. ergo, a wild-goose chase. and all to distract you from the important point: that any of those light-markup systems will work for the purpose of being the p.g. "master format", if you chose one and did the refining grunt-work. and indeed, one of them has already been chosen! but the d.p. workflow and their volunteer mind-set had -- alas -- long ago become as hard as concrete. so even that decision will mean zilch in the long run. oh well. that's the way the cookie crumbles... -bowerbird

1 0

a review of some digitization tools -- 015
by Bowerbird＠aol.com 09 Dec '11

09 Dec '11

ok, let's add some simple lists today... :+) do feel free to point me to more complicated lists, either moderately so (which i'll get to rather soon), or even severely complicated (for some later time). z.m.l. input file: > http://zenmagiclove.com/grapes009.txt python script to create .html output: > http://zenmagiclove.com/tday2011.py or jump directly to the new stuff, if you prefer: > http://zenmagiclove.com/cgi-bin/tday2011.py#chunk220 and tomorrow, tables! -bowerbird

1 0

Re: [gutvol-d] seriously, feel free to ask any questions
by Bowerbird＠aol.com 09 Dec '11

09 Dec '11

keith said: > For $100??? You can be serious! > Make that $1000 and I might think about it! really? you would think about going on a wild goose chase for $1000? well, ok, so would i. but how much would it cost to have you actually _go_ on that wild goose chase, and not just "think about it"? $10,000? $100,000? or what? the good news is that, if you want to code a text-to-html converter, you can do that _extremely_easily_ if you design a wise format, even if you are an unskilled coder like i am. anyway, if you did want to convert various light-markup formats and .html, you should check out pandoc, open-source from john macfarlane. > http://johnmacfarlane.net/pandoc/ here's the lede from that page: > If you need to convert files from > one markup format into another, > pandoc is your swiss-army knife. > Need to generate a man page from > a markdown file? No problem. > LaTeX to Docbook? Sure. > HTML to MediaWiki? Yes, that too. > Pandoc can read markdown and > (subsets of) reStructuredText, textile, > HTML, and LaTeX, and it can write > plain text, markdown, reStructuredText, > HTML, LaTeX, ConTeXt, PDF, RTF, > DocBook XML, OpenDocument XML, > ODT, GNU Texinfo, MediaWiki markup, > textile, groff man pages, Emacs org-mode, > EPUB ebooks, and S5 and Slidy HTML slide shows. > PDF output (via LaTeX) is also supported with > the included markdown2pdf wrapper script. whew! just _reading_ that makes me break out in a cold sweat and worry about fever dreams... john macfarlane is a professor of philosophy at u.c. berkeley, so he can keep all of this straight. mere mortals should just use pandoc instead... -bowerbird p.s. be forewarned that there are a half-dozen versions of markdown, all of them just _slightly_ different...

2 1

Re: [gutvol-d] seriously, feel free to ask any questions
by Bowerbird＠aol.com 08 Dec '11

08 Dec '11

because, you know, if this kind of text-to-html thing really _was_ "impossible" -- like you were once told -- then it seems we're now doing something "impossible". and you should be asking questions, like "how did you do that?", and "what kind of magic are you using there?" but i think we all know that this isn't really "impossible". and it never was. ok, i suppose before .html was invented, people would have looked at you strange if you would have told them that they could "convert" plain-text to "html", but still... *** indeed, you know that #013 post i sent out this morning? it had this in it: > http://en.wikipedia.org/wiki/triskaidekaphobia now, i didn't specify that u.r.l. as a link. i didn't create any "markup" around it, or anything, i just copied it in. but take a look at that message on the pglaf website: > http://lists.pglaf.org/mailman/private/gutvol-d/2011-December/008447.html what you will see is that the u.r.l. to the wikipedia page was turned into a link. you can click on it, and it'll work. that is, [a href=] anchor markup was inserted around it, so that your web-browser would treat it as an active link. so _that_ is an example of a text-to-html conversion! indeed, the entire plain-text contents of that e-mail to this listserve were converted from text into .html, so that it could be mounted on the web. imagine that! and not only that, but the entire plain-text message of _every_ e-mail to this listserve are _routinely_ converted into .html, so the post can be mounted in the archive... do you want to have to code your e-mail in .html format? heck no! but do you want clickable links? heck yes! do you want archives available on the web? heck yes! and -- ever since august 2004 -- the archives _have_ been available on the web... (due to some sloppiness, the archives between 2006 and 2009 were _lost_, but e-mail during that time had been converted to .html.) the package which does this conversion is "pipermail" (the mailman edition), and it's open-source software. pipermail includes other "conversion" routines too... > for instance, because of the right angle-brackets > that precede each line here, these lines will be > rendered in italics, because the conversion routine > presumes this should be treated as a blockquote, > because that's a common convention in e-mail apps. so it's not as if this is some exotic new capability which has only recently become manifest in our consciousness. on this very listserve, where people were insisting that conversion from plain-text into .html was "impossible", _the_archive_itself_ proved those people were _stupid_, and it used their own ignorant e-mails as its "pudding". so yeah, i think that all of this is fairly _obvious_, if you just think about it for half-a-second or so, but _really_, if you have _any_ questions, do feel free to ask them, ok? -bowerbird p.s. if you're interested in a rst-2-epub converter, try: > https://github.com/mattharrison/rst2epub2

4 4

a review of some digitization tools -- 014
by Bowerbird＠aol.com 08 Dec '11

08 Dec '11

ok, to review, let's consider the list of structures that we found in the set of 4 books digitized by jim adcock. they were: > a. headers -- [h1]-[h6] > b. paragraphs and breaks -- [p] and [br] > c. styling -- [i] and [b], plus cover-page prettyness > d. blocks -- [pre] and [blockquote] > e. horizontal rules -- [hr] > f. images -- [img] > g. links -- [a name] and [a href], t.o.c., and index so, we've covered headers, paragraphs and breaks, styling, blockquotes, horizontal rules, and links... which means that we just have to take care of images, and we've handled our first set of "simple" structures. so i inserted some images, just for fun... for good measure, i threw in a poem as well, and a little index too, so i could do the coding for those... the index is a "mere" search routine, but it's useful because it reveals important keywords in the book, with an explicit list of the number of hits returned... (i'll do more work on indexing, but this is a first pass.) the index functionality required that each paragraph be addressable, so now each "chunk" has an .html "id". you'll find all of this new stuff at the end of the book. z.m.l. input file: > http://zenmagiclove.com/grapes009.txt python script to create .html output: > http://zenmagiclove.com/tday2011.py -bowerbird

1 0

seriously, feel free to ask any questions
by Bowerbird＠aol.com 07 Dec '11

07 Dec '11

i said: > http://zenmarkuplanguage.com/grapes008.txt ... > http://zenmagiclove.com/tday2011.py ... > it shouldn't be too hard for you to figure out what this > script has done, and how you would replicate the code. > but if you have questions, please feel free to ask them... seriously, feel free to ask _any_ questions you might have. because, some 6-8 years back, people on this listserve were _insisting_ that what i'm doing here is "impossible". they said it was ludicrous, evidently so much so that they did not even feel any need to refute the arguments, but instead went straight to lambasting me for proposing it... hey, they went so far that they thought a mere _mention_ of z.m.l. was enough to put a smirk on the reader's face... "prove it", they'd say to me, thinking it was a "challenge", as if the taunt somehow framed their technical prowess... all it did was show how little they knew, and how reckless they were, just throwing away every bit of their credibility. so i let them bet all of it, knowing that i'd win the hand... sure enough, over time, they grew more and more quiet, as i systematically doled out piece after piece of evidence. until eventually, a few years ago, they became fully silent, as i _kept_ showing y'all one demonstration after another, right up to today's python script. impossible? yeah, right. so here's your "proof", you stupid idiots; it was right there, in the pudding, just like i was always telling you, all along. and the silence i hear in return is _deafening_... -bowerbird

2 1

a review of some digitization tools -- 013
by Bowerbird＠aol.com 07 Dec '11

07 Dec '11

http://en.wikipedia.org/wiki/triskaidekaphobia -bowerbird

1 0

a review of some digitization tools -- 012
by Bowerbird＠aol.com 06 Dec '11

06 Dec '11

now you have examined our current sample book: > http://zenmarkuplanguage.com/grapes008.txt i've used the ".txt" extension on that file, but really it has been marked-up to satisfy the rules of z.m.l. -- also known as "zen markup language", my baby. one common aspect of text-files is that paragraphs are separated by blank lines. z.m.l. takes it farther, mandating _all_ structural elements be so bounded. this makes it easy for you, as a user, to "grok" how text will be handled in z.m.l. automatic processing. every "chunk" which is surrounded by blank lines is treated as a distinct unit that's virtually independent. here's a script that converts our sample to html5: > http://zenmagiclove.com/tday2011.py to make it easy to see the distinct "chunks" of the text, the script alternates background-color for each chunk. it also numbers the chunks, and shows a visual display of the .html coding which it has inserted into the text, to give you an idea about the processing that it's doing. it shouldn't be too hard for you to figure out what this script has done, and how you would replicate the code. but if you have questions, please feel free to ask them... if you examine this html-book, you'll see that it contains everything we'd expect, plus a few things we might not... for instance, in addition to the linked table-of-contents, chapter-headers also link back to the table-of-contents. there is also, above each chapter-header, a pair of links which navigate to the previous chapter and the next one. i consider these navigational capabilities to be essential, but i think they are not present on any p.g. html versions. the book also has curly-quotes, typographic em-dashes, indented paragraphs, among other customized settings. tomorrow we will begin ramping up the difficulty-level, with the strategy of introducing one structural element each day, until we are handling everything we will need. this gradual build will show how _easy_ this task can be, if we just approach it the right way with a good mindset. to that end, if you'd like me to use any particular book you believe will present some "challenge" to my system, go ahead and post a message pointing me to that book. this approach will serve to dispense the silly nonsense that z.m.l. can only be used to handle "simple" books... -bowerbird

1 0

Re: [gutvol-d] a review of some digitization tools -- 011
by Bowerbird＠aol.com 05 Dec '11

05 Dec '11

i've discussed how many of the digitization tasks can be accomplished in a regular wordprocessor... so, if you're on a mac, and you're interested in this, you might want to check out some recent threads on the d.p. forums, where a volunteer named "ernie" is posting some automator and applescripts he uses... ernie's user-name is "eschaal". here's a thread where some discussion is happening: > http://www.pgdp.net/phpBB2/viewtopic.php?t=2586&start=495 roger frank and the world-famous lucy24 have joined in... here's the other thread: > http://www.pgdp.net/phpBB2/viewtopic.php?t=49617 ernie has also set up a wikipage: > http://www.pgdp.net/wiki/User:Eschaal/PPTools/Mac' -bowerbird

1 0

a review of some digitization tools -- 010
by Bowerbird＠aol.com 02 Dec '11

02 Dec '11

ok, let's regroup... you will recall our three overarching requirements for the process of digitizing an existing paper-book: 1. obtain the text (via scanners, and o.c.r. software) 2. clean the text (the first set of tasks for our tools) 3. turn the text into e-books (the second set of tasks) the first requirement is largely taken care of, thanks to the big scanning efforts of google and internet archive. it keeps getting harder to find a book they haven't done. under the second of those requirements, we covered: 2a. do a spellcheck 2b. fix spacey punctuation 2c. restore styling, e.g., italics all of these tasks can be managed with a text-editor, whether offline or online. nothing here is "difficult"... so we're working on the tasks for the third requirement: 3a. tagging the structural aspects of the text 3b. converting the text into .html (intermediate and final) 3c. converting intermediate .html into e-book output-files tasks 3b and 3c will be the product of automatic processes, so our biggest task is 3a, the tagging of structural aspects... we have reviewed 5 simple books, to see what they involved. a. headers -- [h1]-[h6] b. paragraphs and breaks -- [p] and [br] c. styling -- [i] and [b], plus cover-page prettyness d. blocks -- [pre] and [blockquote] e. horizontal rules -- [hr] f. images -- [img] g. links -- [a name] and [a href], mostly t.o.c., but also index this is a bare-bones list, to be sure, but it's also the case that a surprisingly large number of books only require bare-bones. i believe internet archive currently claims ~3 million volumes... i'd say that 1 million of those books only require bare-bones, while another million of 'em would need only a little bit more. and even the final million are not _that_ difficult, to be honest, not if you're willing to bend a little on what you want from 'em. and if you won't bend, nobody will be able to make you happy... so, starting next week, i'll show you how you can tag structure. we will review how various tools can help us, offline and online, and i'll show you how easily i can build a tool that does the job. we'll start with the simple structures we already found, above, and then work in any additional structures we learn we need... to that end, if there are some books you want to point me to -- with complicated structures -- do please feel free to do so. it might take a little time to work up to the complicated books, but there's no reason we can't put 'em on the table right away. have a nice weekend! :+) -bowerbird

1 0