- gutvol-d - lists.pglaf.org

Re: [gutvol-d] Thoughts on Page Images at PG
by Bowerbird＠aol.com 15 Feb '12

15 Feb '12

carlo said: > The individual files are more valuable > if one wants to check a possible error, > since one does not need to download > the full zip file, just to look at one page. and, more to the point, mounting them individually means that every page has its unique u.r.l. address pointing to it... you _could_ unzip one page, on-the-fly, and give each page an address that way; indeed, that's what internet archive does. but i think it's better to mount each one, and _zip_ 'em on-the-fly when necessary, as desired for downloading as a package. > Your images are 20 MB; assuming that > this is representative of the collection, > all the 40000 books could fit in one disk of 1 TB. there aren't 40,000 books. more like 30,000. but now that michael's gone, who's counting? > all the 40000 books could fit in one disk of 1 TB. and again, more to the point, my demo showing p.g. could grab the d.p. images via the backbone (and not force volunteers to do 2 slow re-routes) was a solution from a complete outsider like me... but p.g. and d.p. have an ongoing relationship, not? i mean, greg's on the p.g. board of directors, isn't he? so what they should _really_ do is buy a hard-disk and duplicate the entire contents of the d.p. site -- images, text, forums, all of it -- and then send that hard-disk to p.g., where the scans could be mounted the next day. plus d.p. would have another back-up. it's surprising that they haven't already done it, and mind-boggling that it hasn't even occurred to them, at least as far as you could tell from all appearances. > But a few years ago, when they started to > keep page images it costed much more. several policies seem not to have been revisited in many years now. p.g. is increasingly out-of-touch. and, as this very issue indicates so well, when it does try to update stuff, not much thought goes into things. -bowerbird p.s. so carlo, got your rewrap of p&p yet?

1 0

rewrapping p.g. to an existing scan-set
by Bowerbird＠aol.com 15 Feb '12

15 Feb '12

carlo said: > Much simpler, one can use wdiff (or dwdiff) > and preserve whitespace from one > and non-whitespace from the other > through regexp. I am interested in > comparing your tool with my wdiff approach. i'd love to see the results from both of you. and compare it to what my program can do, because, at least thus far in its development, my tool still needs considerable baby-sitting for the level of quality one needs to produce. i suggest everyone use their approach on the p.g. e-text for "pride and prejudice"... > http://www.gutenberg.org/cache/epub/1342/pg1342.txt let's mold it to this version at archive.org: > http://www.archive.org/details/prideprejudiceno00austuoft (or, if some other version was the basis for #1342, or you prefer it for any other reason, say so, and we can use that.) -bowerbird

2 1

happy valentine's day!
by Bowerbird＠aol.com 14 Feb '12

14 Feb '12

happy valentine's day to all you lovebirds out there! chocolate and flowers and a little red satin pillow with "love" embroidered across it! and remember that _communication_ is the foundation for a good relationship! -bowerbird

2 1

2020 vision
by Bowerbird＠aol.com 14 Feb '12

14 Feb '12

don said: > If PG is to have a hope of > citizen-provided proofing, > crowd or otherwise, there are > two non-negotiable design requirements. > > The page images must be available as the basis > for resolving all questions about content > (understanding they aren't always unambiguous, > but no images is hopeless.) > > The text must have the page numbers embedded > as data so the error submission process includes > the ability to easily confirm the text with the image. well, don, yes, your general point is certainly spot-on. if i wanted to quibble, i could mention that it would certainly be possible to build capabilities that could handle much of the task via a variety of other means, using other ways, maybe being even more effective. (such as auto-comparison of multiple digitizations.) nonetheless, i would still have to admit that, at many points in time, along the way, it is simply _necessary_ to be able to refer to the original, to quell some issue. it's an unavoidable need, for a digitizing system. *** and yes, i understand quite well that you are merely probing to see if the powers-that-be will "concede" that they will indeed have to install such capability. since if they don't agree to _that_, we just go home, knowing that the job before us would be impossible. *** still, don, perhaps you're getting ahead of yourself. because even if they _do_ concede, would it matter? is there anyone to build such a correction system? (and by "build", i mean "program". it's obvious we have no shortage of "architects" here who're willing to draft up their favorite fancy blueprints. but without anyone to actually build the thing...) so, would anyone code a p.g. correction capability? as someone who has volunteered -- many times -- to program a system for project gutenberg that did exactly that, i do believe i can answer that question. today, 2012, i would never build that system for p.g. and not just because p.g. lost all its credit with me... because i'd advise anyone else _not_ to do it either... the reason is simple: not enough bang for your buck. first of all, the requirements are hard for p.g. to meet. even just these two don listed are hard, because p.g. never paid any attention -- at all -- to pagenumbers. and even though they give a little lip-service to scans, it's not part of their d.n.a. either. so the infrastructure is wholly and completely unprepared for such demands. indeed, d.p. has the scans for most of the books they've digitized over the years -- online as we speak, i believe. (if a provider asks other sites not to repost their scans, d.p. will respect that request, which is a bad response, to my mind, because nobody should put that restriction on scans from a public-domain book, and we shouldn't honor such a restriction if anyone does try to impose it; but that's the d.p. policy.) at any rate, the fact remains that d.p. _has_ the scans. but p.g. doesn't mount them. and yes, for all the p.g. books, you can probably go find a scan-set somewhere, either google or internet archive, or somewhere else. but who is it that'll do that legwork? and how do we decide which version, if there are many? and then there are the practical issues, like file-naming. and the p.g. linebreaks which do not match the scan-set. and scan-sets not atypically will manifest some glitches, which -- at d.p. -- requires a "hospital". who mans that? none of these are trivial issues. they can be surmounted, at least individually, but it's work. en masse? more work. and if you think _programmers_ are in short supply here, take a look around and see how many gophers you count. but even if we ignore all of that, for now, we can't ignore that p.g. _could_ be mounting scans now. but it doesn't. which tells me the p.g. infrastructure can't handle scans, not with any volume, which means this thing won't scale. so that's a difficulty facing you. but it's not the hardest one. no sir. not by a long shot. the hardest part would be the obstacle of the politics. the navigation of marcello, all by itself, would impose such a huge cost on you that it wouldn't be worthwhile. no way to go anywhere without marcello blocking you. then add greg, and the whitewashers, plus all the flack you'd have to take from d.p. for invading "their" space, not to mention all the "cooks" in the gutvol-d "kitchen". all with the best of intentions, mind you. (well, maybe except for marcello.) but a huge obstacle nonetheless. heck, it'd be a _nightmare_ -- and that's putting it mildly. *** but even if, with some miracle, you were prepared to deal with such immense costs, the benefits are feeble. the most you're gonna get by fixing the p.g. library is 40,000 books. and that is a total fantasy number. at least 10,000 of the "books" are repeat fragments, audio files, and various things which are not "books". so the _real_ number is closer to 30,000, if that many. but even if it _was_ 40,000, or 80,000, that number is _dwarfed_ by the collections now available elsewhere. moreover, a very high percentage of the p.g. e-books are _already_ very clean, especially in a relative sense, which lessens further the bang you'd get for your buck. nobody cares much if you remove the last 100 errors from a book. and if they knew you spent _100 hours_ fixing those last 100 errors, they'd say you were crazy. they'd rather you spent that time taking 10 new books from "unacceptably dirty o.c.r." to "ok, this will work". and where can you do that? well, over at internet archive, of course. so, if any programmers want to build such a system, my advice to them is to "build it for internet archive". their infrastructure _does_ support page-scans, and it _does_ keep track of page-numbers, and their stuff is exposed on the web, in a somewhat-documented way, so you can grab it from them without talking to them. (which is a good thing, because they're just as deaf as the p.g. powers-that-be are, so count your blessings.) so the costs of building their system will be much lower, and with millions of books the benefits will keep comin'. plus correcting their text would be a real contribution, rather than simply gilding the lily on a few p.g. books. dollar-for-dollar, pound-for-pound, bang for the buck is gonna be _far_greater_ at internet archive than at p.g. much lower costs. much greater benefits. no-brainer. > People will self-verify if they are given the tools. > If they can't, the whitewashers are screwed again. the whitewashers screwed themselves by not installing such a verification-system in the first place, and that's something i have been saying since december of 2003, when p.g. had 10,000 e-texts, as it was apparent then that we needed a way to go back and repair the books. a significant mark of the uselessness of this endeavor is the fact that -- more than 8 years after that date -- we are _still_ having a "dialog" about "how" we "could" perhaps "build" a "system" that "might" (or might not) "accomplish" the "goal" of "correcting" those "e-books". it is testament to michael hart's vision and achievement in building from the grass-roots an actual cyberlibrary, that so many good people have shown that we're willing to waste valuable years of our lives in service to that goal. but maybe, just maybe, it's time to stop wasting our time. michael is gone, folks. there is no vision here any more... i, for one, am not gonna discuss this until the year 2020 before i see the p.g. powers-that-be are deaf _and_ blind. -bowerbird

8 15

let me get this straight
by Bowerbird＠aol.com 13 Feb '12

13 Feb '12

ok, let me get this straight... p.g. wants volunteers from d.p. to download a scan-set, and then zip it, and then upload it to p.g. for mounting. why doesn't p.g. just get it directly from d.p.? here are the first 8 pages of "when it was dark": > http://zenmagiclove.com/pgstupidity/012.png > http://zenmagiclove.com/pgstupidity/013.png > http://zenmagiclove.com/pgstupidity/014.png > http://zenmagiclove.com/pgstupidity/015.png > http://zenmagiclove.com/pgstupidity/016.png > http://zenmagiclove.com/pgstupidity/017.png > http://zenmagiclove.com/pgstupidity/018.png > http://zenmagiclove.com/pgstupidity/019.png that book just went into postprocessing at d.p. i pulled those 8 pages over to my i.s.p. in mere seconds. i could pull all the pages, if you needed a bigger demo... but i assume you're smart enough to get it. (although you weren't smart enough to have the idea originally.) i used to think that p.g. stupidity was at its _peak_ when you wasted the time and energy of volunteers. but you've topped yourselves. because now you want to waste their _bandwidth_ as well! congratulations... i can't wait to see what you come up with next. oh right, i remember now. you want to "crowdsource" _corrections_ to your files, without giving anyone the scans that they would need to verify their corrections. who knows how much of our time, energy, bandwidth, and _frustration_ we'll burn up chasing that phantom? i feel like yossarian, on first learning about catch-22. -bowerbird

1 0

Page scan naming convention
by Al Haines 13 Feb '12

13 Feb '12

This is the convention used by Joshua Hutchinson, when he provided page scans from DP: Basic format: The prefix for the cover pages is: "c". The prefix for the roman pages is: "f". The prefix for the arabic pages is: "p". *** For blank pages there should be no file and the page number should be skipped. Optionally an image saying: "This page is blank in the original." may be inserted. *** Example of file naming: front cover c0001.png back cover c0002.png spine c0003.png i title page f0001.png ii title verso f0002.png iii dedication f0003.png iv is blank v contents f0005.png page 1 p0001.png page 2 p0002.png image on page 2 p0002-image1.png image on page 2 p0002-image2.png page 3 p0003.png page 4 is blank page 5 p0005.png ... ... page 9999 p9999.png

1 0

hunter must gather, not scatter
by Bowerbird＠aol.com 12 Feb '12

12 Feb '12

elsewhere, hunter said: > if you follow gutvol-d you already know that > many consider the HTML guiguts produces > and that many PPers choose to produce otherwise > is a disaster when viewed on most readers and as epubs > which is the most popular format for > users downloading pg books. > So the whole convert to HTML feature is one huge bug. hunter, just one word: careful! first, i can see quite well that you _qualified_ that -- perhaps quite meticulously, to your mind -- with the "when viewed as" phrase which followed. but your audience, over there in the d.p. forums, turned off their rational brains just as soon as you said the .html they lovingly produce "is a disaster". they never even got to the condition of specification. i know this is true, because i used to use that trick against 'em all the time. but _my_intention_ was to make them stop listening, while still covering myself. you, on the other hand, _want_ them to listen to you. so you really need to be careful about that. *** but i think, aside from that, you got the fact wrong. to my mind, nobody here is complaining about the .html output from guiguts. i know that _i_ am not. i don't even know what kind of .html guiguts makes. it might be bad, or might be good, i just don't know. what i _do_ know is that it's the output of a script, thus can probably be depended on to be consistent. so even if it is bad, it's bad in a _dependable_ way... meaning the odds are good that it can be corrected by using a procedure which is, likewise, automated. (whether or not marcello's app does that correction is an open question, but a correction is _possible_.) the real "problem children" are the "snowflakes", the .html versions which contain "unique" .html, inconsistent with the majority of the other books. just as one big example, there are lots of ways to indicate a header, using .html, or .css, or _both_... (ask a dozen people on this list, and you will get at least a half-dozen different ways of doing it.) and even though most of them can't be considered "wrong", the simple fact is that, if we allow 'em all, it becomes virtually impossible to write a converter that can discern all of them, i.e., spot the headers... and, obviously, if a converter can't recognize even something as crucially important as the _headers_, it's not gonna be able to make a quality conversion. one reason i used this as the example is because all these different ways of indicating the headers essentially produce an equivalent _functionality_. at most, they give a different "look and feel", but it'd be easy for all of us to agree on the "purpose". none of your postprocessors are so heavily invested in their own particular snowflake style of _headers_ that they couldn't switch to some consensus method. another reason i used headers as the example is that it is possible to _find_ books -- in the library now -- where the purposes and functionalities of the headers are being compromised, because the headers are _not_ being correctly recognized. this is the kind of problem that you need to bring to the attention of your people. it's a real problem, that exists right now, in the library. you can say "look at this book you worked so hard on". and then point out that "the headers are not working; they're not doing what they're supposed to be doing". perhaps they didn't appear in the table-of-contents, because they weren't getting recognized as headers. or maybe they showed up as dinky wimpy side-notes, rather than being big and bold, like headers should. whatever the case, show people how their "snowflake" methodology sabotaged the functionality of their book. let them know that their _solution_ to this problem is the use of a workflow that produces consistent output. now, sometimes you are going to find that the glitch is _not_ in the input that you feed p.g., but rather in marcello's converter... your people are going to say "well, look, i did the headers the way i'm supposed to, and they _still_ didn't show up correctly in the .epub." and hunter, your answer is that _your_ converter _will_ create the correct output, when it's given correct input. so y'all won't need marcello's damn converter no more. -bowerbird

1 0

pride. and prejudice. incremental.
by Bowerbird＠aol.com 12 Feb '12

12 Feb '12

ok, greg, new versions of "pride and prejudice", incremental improvements for the tournament. > http://zenmagiclove.com/prapr/prapr.mobi > http://zenmagiclove.com/prapr/prapr.html > http://zenmagiclove.com/prapr/prapr.zml > http://zenmagiclove.com/prapr/prapr.ncx > http://zenmagiclove.com/prapr/prapr.opf > http://zenmagiclove.com/prapr/prapr.jpg *** i learned some interesting things doing this... i began by stripping some cruft from my .css. as i did, more and more, i started to wonder how much of it i could strip out completely... it turned out that i could strip out quite a lot, and still get out essentially the _same_ .mobi. until eventually i dispensed with the stylesheet. what's in the .html file above is now just this: > p {} > blockquote {background-color:yellow;} > i {color:blue;} and that's only to prove that kindlegen doesn't disregard the stylesheet, like some people say. because you will see those colors _do_ work -- when you display the .mobi on a color machine. but it's also true that kindlegen _does_ ignore a lot of the stuff you might put in a stylesheet. most especially, it seems, margins and padding. it's the exact equivalent of the hard lesson that designers had to learn as they went from print -- where they could control almost everything -- to the web, where you "control" almost nothing. they had to force themselves to abandon their desire for full control, and embrace flexibility. and that's why i wanted to run an experiment where i relinquished all that i possibly could, where i said, "ok, then show it how _you_ like." and know what? it didn't do a bad job at all. i was fine with it, aside from a couple details... one such example: it didn't center the headers. so i moved those exceptions into in-line styles, and boom, a simple-yet-quite-acceptable mobi. a list of the .html tags -- the complete set that is used in this book -- is appended to this post. but the bottom line is simple: let go. go zen. there are thousands and thousands of books in the p.g. library that can be treated this way. -bowerbird [mbp:pagebreak] == pagebreaks [div id="chunk0""] == sections [h2 style="text-align:center;"] (h1, h2, h4) == headers [p] == paragraphs [p style="text-align:left;text-indent:0;"] == table-of-contents [p style="text-align:center;text-indent:0;"] == scenebreaks [i] == italics [br] == breaks [hr] == horizontal rules [blockquote style="text-indent:1.25em;"] == blockquotes [a id="chapter_1"] == internal-idnames [a href="#chapter_1"] == internal-links [a href="http://zenmagiclove.com/prapr/"] == external-links [p style="text-align:right;"] == chapter-links

10 33

Re: [gutvol-d] hunter could do it with guiguts
by hmonroe.pglaf＠huntermonroe.com 11 Feb '12

11 Feb '12

I agree there is in fact a golden opportunity to change the face of PPing by using guiguts to encourage better practices by changing default behavior and flagging bad practices. Most PPers will download the next release when announced given the bugginess of earlier versions. All we need is some volunteers to help with development (just send me your sourceforge ID). For instance if someone wants to add HTML markup to identify a cover image I can point you to the right spot in the code. If I understand, it is up to the WWers to decide whether to accept vanity epubs. Perhaps they would be more willing if someone gave them a tool to check epubs, i.e. explode them and check for a cover, valid HTML, etc. I would be happy to include ZML scripts and test cases with guiguts if they are shared with me. Perhaps a system for crowdsourcing corrections could also be used for distributed PPing. I am working towards encapsulating guiguts functionality into libraries that could be used in a web based approach. Hunter

2 1

Search
by don kretz 10 Feb '12

10 Feb '12

If your site offers eBooks, and it says "Search All" (rather insistently, 3 times,) why wouldn't people assume it would search the texts? Where is the screen that searches texts? Sent from my Windows Phone

2 1