October 2012 - gutvol-d - lists.pglaf.org

several small-to-medium-size units of work
by Bowerbird＠aol.com 12 Oct '12

12 Oct '12

don said: > The path away from the quagmire > needs to reintroduce simplicity, and > especially single simple jobs that > iteratively improve a piece of text, > with immediate gratification. > There probably need to be several simultaneous > small-to-medium-size units of work that > offer closure and positive reinforcement you said all that before. > for the benefit of people who > don't have toolchains they know intimately > and have been refining for years; > and other people who can be > persuaded to adopt and learn them. ok, dude, fine. but i'm guessing that, once you have your little system that everyone buys, your start-up page will look like this: =============================== > > "book-dig" -- for book digitization... > > thanks for helping to digitize books! > > "book-dig" is a system to _finalize_ > a paper-book that's being digitized. > > a person who's done the bulk of the > preliminary prep and o.c.r. clean-up > can then use "book-dig" to let other > volunteers help polish off the book... > > (book-dig also lets you do finalization > by yourself if you prefer to do that, or > don't want to have to "recruit" people.) > > this "finalization" process helps to turn > corrected o.c.r. into a refined e-book... > > in some circles, this has been known to > be called "post-processing", as if it were > just an "afterthought", but professionals > know these tasks _are_ "the processing". > > but that doesn't mean it must be difficult. > > by breaking it into a series of small tasks, > it actually becomes simple (and fun) to do! > > *** > > step through these tasks to polish a book. > > (you probably should do the tasks in order, > since the later ones depend on the earlier, > at least until you have enough experience > to know if/when it's ok to go out of order.) > > * task #01. check paragraphing. > * task #02. check doublequotes. > > * task #03. check any flags left. > * task #04. check formatting. > * task #05. check illustrations. > * task #06. check the styling. > * task #07. check hyphenates. > * task #08. check singlequotes. > * task #09. check look and feel. > * task #10. check dictionaries. > > * task #11. search function. > * task #12. fix flagged items. > * task #13. there is no 13th task. > > * task #14. final certification. > * task #15. prelim read-through. > * task #16. final read-through. > * task #17. mount the book. > * task #18. check the mount. > * task #19. open up champagne. > * task #20. start the next book. > =============================== so, you know, if you need to "borrow" this design, go ahead. i figure you'll get around to it in 8 years. -bowerbird

1 0

several small-to-medium-size units of work
by Bowerbird＠aol.com 12 Oct '12

12 Oct '12

don said: > There probably need to be several simultaneous > small-to-medium-size units of work > that offer closure and positive reinforcement. at the beginning of this year, when we had this same thread, i coded an entire workflow of single-purpose components... mercifully, the thread dribbled away, and i can't even recall if i mentioned it, because i was glad all this misery stopped. and now that it's been started again, i haven't mentioned it, because i'm still hoping this current thread will die as well... (y'all are not just sadists, it seems, you are also masochists.) but perhaps i will show you what i've done the _next_ time someone is foolhardy enough to bring up this b.s. again... which i'm thinking should be right around june of next year. -bowerbird

2 1

huckleberry pudding.002
by Bowerbird＠aol.com 11 Oct '12

11 Oct '12

got a new version of huck finn up... :+) > http://zenmagiclove.com/huckf/wisdom.py > http://zenmagiclove.com/huckf/huckf-004.zml it's getting really clean now. time to splice in the italics. i used a brain-deadish approach with the aux dictionary, partly because i wondered how difficult it would make it, which was _not_ a smart decision to make with this book, and all the funny spellings twain used for the "dialects"... i left a few really bad pages, so you could see the horror. > page 67-73 > page 111-114 > page 151 > page 182 > page 202 lots of words flagged in red on those pages... (at any rate, with the words almost fully correct, and reflecting a very close relationship to the actual scans, what with retaining page-breaks, line-breaks, and even end-line-hyphenates, this particular version of the file is quite close to the type of file jon hurst was suggesting. of course, it's silly not to mark the italics. i'm just sayin'.) *** here's the version of huck finn that i'm using: > http://archive.org/details/adventureshuckle00twaiiala that's the version that has been downloaded most often -- some 14,508 times, according to their listed figure -- and it's also the one which they themselves used for their first demo of their online web-based scan-flip program, for which brewster had been lusting since the first time he saw a similar program running in the british museum. and -- as fate would have it -- it is also a version which has lost its em-dashes from its o.c.r. file, i'm sad to say. (meaning i had to scrape the text from the abbyy.gz file.) > https://ia600506.us.archive.org/5/items/adventureshuckle00twaiiala/adventur… that's right, its most prominent version of "huck finn" is fatally flawed, and nobody cared, or even noticed much. (yet another demonstration that normal people don't even need full clean digital text of a book. which is not to say that society wouldn't benefit by having a complete library of clean digital books; but "society" isn't too willing to pay for such an asset, so i guess we don't really _deserve_ it.) but in spite of my own general preference for digital text, i will admit that the archive.org book-flipper is awesome. > http://archive.org/stream/adventureshuckle00twaiiala#page/122/mode/2up indeed, on an ipad with good wifi, it's a great experience. you can have a 2-up page-spread view, which is nice, _or_ switch over to single-page display if you want bigger text. and, with this huck finn, you get all the great illustrations, plus the lovely design with its rustic olden-dayes feel to it. *** anyway, i wanted a little more than book-flipper offered (such as the ability to save the scan-files to my own site), so i did a primitive-but-with-special-tricks version of it: > http://zenmagiclove.com/scanarchiveorg.py buttons at top-left switch between "huck" and "mcguff"... the "huck" scans are pulled from my site, but the "mcguff" ones are called from archive.org, so you might notice that the page-numbers are incorrect, because archive.org was not smart enough to put page-numbers in its scan names. (the stupidity that i have to deal with is so mind-boggling.) this special-tricks app will be very useful to me long-run... *** more later. because you don't get enough torture... :+) -bowerbird

1 0

blah blah blah feasibility blah
by Bowerbird＠aol.com 11 Oct '12

11 Oct '12

jeroen said: > I will continue then... great! i'm glad nothing i said served to dissuade you. your workflow is so idiosyncratic that i don't think it will ever be a specific aid to other d.p. post-processors, but your general attitude of wanting to improve your e-books -- even in the face of d.p. and p.g. stupidity -- _will_ be a good role-model for others. at least i hope that it will. > Agreed, but I have a feeling this is a passing phase well, the lack of power will certainly change in the future. but that doesn't mean all of the viewer-inconsistencies will immediately go away. my guess is that they _won't_. even with web-browsers, the incompatibilities still exist. yes, the "basics" have all achieved a degree of uniformity, but on the edges, not so much, as illustrated best by c.s.s., where you still have to specify lots of stuff _by_browser_... for instance, we still need code like this: > background: url('linear-gradient.png') 0 0 repeat-x; > background: -moz-linear-gradient(top, #a1e048, #6a942f); > background: -ms-linear-gradient(top, #a1e048, #6a942f); > background: -o-linear-gradient(top, #a1e048, #6a942f); > background: -webkit-linear-gradient(top, #a1e048, #6a942f); those browser-prefixes are an indicator of the sickness... and .html5 as a "living standard" (i.e., never on solid ground) will only exacerbate the problem. what worked yesterday... well, it might work today, or it might not, we'll have to check. and that's in an arena where the vendors have a motivation to match up with each other, to conform to the specification. but the e-book world has evolved differently, such that the commercial interests there believe their business imperative compels 'em to _be_distinct_, so as to build a walled garden. apple didn't bother with the tedious "embrace" step on .epub; just skipped straight to the "extend" step with "ibooks author". and apple didn't even try to disguise this with technical talk; it was so bold it included a "you must sell output through us" clause right in the legal terms for the authoring-tool itself... so, if apple is willing to be that brazen with their customers, do you really think it's going to work for greater cross- > But everybody in the "scene" of pirated books is using it. > (Maybe those two facts are related) no, _those_ facts aren't related. the point is whether you must maintain the markup inside. calibre rewrites that markup, and the rewrite is _extreme._ trying to rework markup after calibre rewrote it is painful. > I know it wrecks my books... _that_ fact might be related. due to that extreme rewrite. > I will try to do some testing on iPads then apple has about 30% of the total market, 77% of the .epub. kindle has about 60% of the total market, almost no .epub. everyone else splits the rest; 10% of the total, 22% of .epub. so, for your .epub version anyway, ibooks is all that matters. > Chaining conversions is a pain, > that is what I complained about several times. it's a bit more work to have to make a .mobi _and_ an .epub. that's why most people feed their .epub to kindle-previewer. but it will hurt your .epub to force it into .mobi double-duty. so i prefer to make the .mobi separately, and _specifically_... (the .opf and .ncx are a bit easier to make than for an .epub; plus, yay, you don't have to split the .html into 300k chunks.) > For that I will need the spec of both. well, for .mobi, you pretty much know what will work now, and sadly, that's probably all that'll _ever_ work for .mobi. kf8 -- which is now not much more capable than .mobi -- should start showing regular boosts in its capabilities, and that's why you're gonna wanna split it off as a separate flow. > I thought it was pretty closed, > but maybe there are some pointers. the #eprdctn hashtag on twitter is utilized by many people who're employed to make e-books, or freelancing at it, so you can keep up with developments by looking in on that... > ePub3 is the xml-variant of html5, but ePub3 isn't there yet. yeah, i know that's "the line". but considering that .html5 is the route that we switched to, once enough people finally realized .xml was the wrong way, that "official line" is best viewed as a contradiction-in-terms. > I think that should be answer to some of the complaints, > with a proper explanation of the state of technology. but that was my earlier point. it is _not_ just "technology". there's a lot of business shenanigans going on that mean we're not gonna get what we want out of e-books until we loss all these corporate leeches and establish our own way. -bowerbird

1 0

huckleberry pudding
by Bowerbird＠aol.com 11 Oct '12

11 Oct '12

ok, so i'm re-doing huck finn. david's version has no provenance. jim's version has none of the images. neither one is as error-free as i can be. i thought i'd have it all finished by today -- a week after "pride and prejudice" -- but proofing is going longer than allotted, thanks to all that delicious twain slanguage, so i'm resetting my deadline to this weekend. if you want to see the work in progress, though, you can see the text as it gets updated by visiting: > http://zenmagiclove.com/huckf/wisdom.py "wisdom" is what came after "water/wine/whiskey". -bowerbird

2 1

blah blah blah blah test case blah blah
by Bowerbird＠aol.com 11 Oct '12

11 Oct '12

don- i see there's some heat in my spam-folder, but i have neither the time nor the desire to look at any of that... but again, i will need correct text if you want me to do the mcguffey. too much work to restore the diacritics. so if anyone has that correct text, don, do let me know. even with utf8 text, though, the .html that'd be created by epubmaker would still be a disaster, totally worthless. in the files section for the book at p.g., though, there's _gold_ (believe it or not) in the form of a word .doc file. the text lacks the page-breaks and line-breaks i require, and the diacritics are not there, but otherwise it is clean. so it is probably the most advanced place to start from... moreover, the .doc has a ton of nicely-clipped graphics, which would also provide a great head-start for people... i'm just sayin'... *** again, this is _not_ a good test-case. and it's probably more work than it's worth. plus did you notice how easily you could just point me to the scan-set on archive.org? for most people, who would want to read the book, or maybe just look at it, the scan-set is more than enough to meet their needs. this is what i was talking about a few days back, when i said that most people are very happy with a scan-set, with no need for clean digital text for the whole book, which is the only benefit that project gutenberg offers. (and, at least with this 7-bit text lacking the diacritics, p.g. can't even offer _that_ benefit, which is a bit sad.) if nobody uses the digital text, why bother cleaning it? *** considering all that, i'd think that it's probably best to just put this book aside. but if you still want to do it... -bowerbird

1 0

blah blah blah feasibility blah
by Bowerbird＠aol.com 10 Oct '12

10 Oct '12

don said: > So what are you suggesting? ... > What is a constructive suggestion? as per usual, for this _particular_ topic, we have the otherwise uncommon phenomenon where jim adcock is one of the only people making sense in the dialog. so let me translate for you. jim recommends that p.g. should _require_and_enforce_ submitted .html to be convertible to quality .mobi/.epub. as _part_ of this, jim recommends that the converter-tool should be improved, since its current performance sucks. jim suggests that an improved converter coupled with just a _little_ bit of bending on the part of the producer's whims will mean that _everyone_ will end up with a better outcome. he would probably also suggest that the converter-tool be made more widely available, so that producers could use it to check the viability of their files _while_ working on them. (the current p.g. web-app tool is extremely clumsy to use.) it is clearly _possible_ to implement all these suggestions, because the systems i've built have routinely included them. so i would certainly agree with all of jim's suggestions. indeed, i can't see how there could be much opposition. oh, right, now i remember. you don't like the messenger. *** as for the simple/complex dimension... i'll ignore for a moment that the label has its problems... i believe it is quite easy to make a concrete suggestion: use the simple-basic workflow for the 99%+ of books in the library which can be finished with that approach. portion out the remaining less-than-1% of the books to people who are in love with their complex methodology, and see who comes back with the best-working product. for each e-book, give the winning contender a gold-star sticker to put on their forehead, for the bragging rights. *** > 1. Choose a book to digitize. > 2. Collect and record metadata, and confirm Public Domain status. > 2. Obtain images of all the pages. > 3. Produce text with OCR. > 4. Refine the text to match the OCR. > 5. Locate and mark significant artifacts in the text. > 6. Submit package of text(s) and images to PG. > 7. PG review and post to catalog. > 9. Select layout and format for desired device or application. > 10. Select or create layout and format for text and artifacts. > 11. Issue ebook per spec (possibly including production.) > 12. Identify errors or other revisions to text. > 13. Locate and describe revisions. > 14. Submit revision to PG. > 15. Back to step 7 to validate and apply revision. well, first off, you might wanna check your numbering. more importantly, can you please explain how this list sheds any light or understanding on contended issues? it's reminiscent of your earlier call for "a feasibility study". we're about a decade past the time for such suggestions... there are simple, _obvious_ problems right in front of us -- one example is a whitewasher who posted an "update" less than 6 weeks ago with headers tagged as paragraphs, and not with some "header" class, but bracket-p-bracket -- yet you want to fuzz this all up with a kindergarten outline? is p.g. paying you to be a "consultant", or what? i don't get it. -bowerbird

9 36

blah blah blah blah test case blah blah
by Bowerbird＠aol.com 09 Oct '12

09 Oct '12

don said: > I appears these are the images: great. now i'll just need correct text that contains the line-breaks and pagebreaks from this scanset. -bowerbird

1 0

blah blah blah blow your socks off
by Bowerbird＠aol.com 09 Oct '12

09 Oct '12

don said: > but they are inherently unsemantic again, you have confused "semantic" with "explicitly tagged". human beings follow relatively simple rules to endow meaning. so it's silly to think we can't program computers to do the same. it's totally unnecessary to tag a section as "table of contents", when we can easily code a routine that can recognize it readily. and if you include the word "chapter" in each chapter-header, which you should, so you will get an automatic deep-link to it, it's quite easy to write the code that leverages that knowledge. again, stop treating computers as if they are some simple idiots that need to have every single bit of minutia explained for them. > so getting people to restrict themselves to > marking up a master text with only semantic tags > will be difficult. don't waste the time and energy of human beings by having them engage in the stupid task of applying tags. > 2. Similarly, I find it helpful to have an easy visual distinction > between structural markup and presentation markup. i find it helpful to avoid the need for "mark-up" entirely. > 3. Also, since it's only divs and spans, the markup > tends to become verbose and obscure pretty quickly, > with a lot of noise to the signal. again, you don't need it. > 4. And consequently marked-up XHTML is > less transparent when you want to see only the text. not when you have the markup applied automatically, after-the-fact, solely for feeding it to a web-browser that's programmed as a dumb-and-blind rule-follower. but if you think those brutes are gonna avoid extinction, you certainly don't have a very good handle on the future. google is gonna blow your socks off... -bowerbird

1 0

Suggestion: Convenient Downloadable Archives Format
by R Vijay 09 Oct '12

09 Oct '12

Hi all: Presently the Gutenberg DVD is available only once every few years. If the following formats of archives are made available, then it is expected to benefit the users to have a more uptodate collection and also increase the popularity of the Gutenberg site/service: 1. Collection of Books for each month as a torrent 2. Collection of Books for every 3 months as a torrent 3. Collection of Books for every 6 months as a torrent 4. Collection of Books for every 12 months as a torrent Thank You. Vijay

1 0