- gutvol-d - lists.pglaf.org

Re: [gutvol-d] roger, your code _could_ scale just fine, if only you'd...
by Bowerbird＠aol.com 20 Oct '11

20 Oct '11

keith said: > This wisdom is absolutely a must! sigh. keith, you're a perfect exemplar of the technocrats that i was just talking about... and making fun of... they saw lots and lots and lots of cases where someone used the file-system as a database and then ended up having problems with scaling when it eventually grew. so those technocrats then came to believe that they'd seized upon something "inevitable", so they started to enforce the iron-clad rule that "you must _never_ use your file-system as a database, because it won't scale". and they're proud of themselves, and puff their chests, and slap themselves on the back for this bankable info. furthermore, they repeat the mantra to each other often, confident that they've come across some ultimate truth... the problem is, there are some exceptions. and this is one of them. as long as you have a script working on only one book, consisting of just a few hundred page-scan images and a few hundred corresponding text-files, and _further_, you have _no_more_ than a dozen people working on it (if that, and usually not simultaneously), then you are _not_ going to have any "scaling" issues with that script, especially if 80% of the text-files will never be changed. (and none of the graphics files will ever change, save for occasional swapping-out of some badly-imaged pages, which isn't even something that the script will manage.) and even if you have a team of a dozen people working simultaneously on a book, they'll finish in half-an-hour. so you are _not_ going to have any problems with "load". and the important thing is that we _know_, for _certain_, that the per-book task will _never_ exceed beyond that... so we will never be taken "by surprise" by scaling issues. we might gain a ton of users, and eventually be working on thousands of books at the same time, but _each_book_ will still be a very small bite, easily chewed and swallowed. more users would mean more hardware, and bandwidth, and more copies of the script running in different folders, but each script is still going to be doing a very easy task... that is why this particular usage is _an_exception_ to the rule that you shouldn't use the file-system as a database. so your "advice" is wrong, keith. it's just flat-out wrong. and i was watching closely as roger developed his system. so i saw him make the error as he was in the process of it, and i saw the ramifications of the error... all of a sudden, things were squirreled away in a database, thus opaque... in this type of workflow, you want content out in the open. and equally important here is the point that roger needs to know that he _can_ mount a full-on site, if he _wants_. he's now suffering from the misimpression that he can't. maybe he doesn't _want_ to do that, that i'd understand, but i don't want him to think he can't, when really he can. > What most do not understand that in order to have > a "proper" database for any non-trivial task, you > must add layers of abstraction in order to administer > all transactions and function needed. Without them > a system will eventually bog down and will not scale well. this is the other sickness that technocrats have. first they introduce complexity. then, when the complexity starts to make it difficult to handle a very simple situation, they add _more_ complexity, as if that's gonna solve it all... and _that_ is what causes things to "eventually bog down"... -bowerbird

2 1

epubeditor.sourceforge.net
by Bowerbird＠aol.com 19 Oct '11

19 Oct '11

lee said: > parsimonious i'll comment on that later, but for now let's focus on this: > http://epubeditor.sourceforge.net lee, please generate o.c.f. and .ncx files for these books, both of which have received some attention here recently: > the art of the book, by charles holme, 1914. > http://www.archive.org/details/artofbook00holm > the practice of typography: modern methods of > book composition, by theodore low de vinne, 1904. this is a timed test, so don't dawdle... -bowerbird

2 1

Re: [gutvol-d] ePubEditor
by Lee Passey 19 Oct '11

19 Oct '11

On Wed, October 19, 2011 11:12 am, James Adcock wrote: >> On Oct 7, 2011 I registered the project "ePubEditor" with SourceForge.net, >> and uploaded all my code there (http://epubeditor.sourceforge.net/) > > I tried this for a couple minutes but it always crashes when I try to open > an epub file. Perhaps include a sample file that works in your downloads? Thank you for your feedback. When ePubEditor opens an ePub file, the first thing it does is explode the entire contents of the .zip file into a temporary directory. The location of this temporary directory is read from the file "ePubEditor.ini". By default, this location is set to "C:/My Library/temp". There was a bug (not surprising) in the downloadable package that caused an error message to be displayed if the temporary directory does not exist. You can work around this problem by creating the directory "C:/My Library/temp", or you can edit the ePubEditor.ini file to point to some other temporary directory that does exist. At some point I intend to create a fancy user interface to edit the preferences but for now, I guess notepad is the tool. I have fixed the code to create the temporary directory first before trying to extract the files. I will probably upload it to SourceForge before the end of the week. I will try to add the work-around to the readme at SourceForge tonight.

1 0

chasing away the p1 proofing crew
by Bowerbird＠aol.com 19 Oct '11

19 Oct '11

jim said: > I don't think DP has changed anything else > so what I think they have done is simply > reduced the rate of books accepted into the system > in the first place, increasing the (implied) queue > of books digitized but not begun proofreading. > What this does, imho, is simply chase away > volunteers who want to proof the lower rounds. > But that is probably better than simply > wasting those volunteer efforts by having > too much stuck on queue in later rounds. i'm constantly amazed by the people who think they can "analyze" what's happening at distributed proofreaders from a distance, by looking at a chart or some statistics. it's a social system, and thus a lot more complex than that. but if you wanna reduce it all down to something simplistic, _this_ is what has been happening at d.p. the past few years: they have systematically driven away the p1 proofing crew... the mantra at d.p. used to be "do whatever job you choose". alas, no more. you can no longer opt to stay in p1 forever. much new material is _officially_ made "unavailable" to you, either permanently or "temporarily" -- for a length of time that usually means it's finished before it becomes available. (material that doesn't get finished is typically undesirable.) perhaps just as bad (or maybe even much worse), if you read between the lines in the forums, you can tell that volunteers who choose to stay in p1 are being harassed for that choice by some other volunteers. (at times, it even becomes overt; the harassers operate with the indignation of the righteous.) so frontchannel and back, officially and via "peer pressure", volunteers are being chased away from doing p1 proofing... this is a travesty, pure and simple. probably even a tragedy. many of the old-school p1 proofers, who had been doing it for _years_, with literally thousands of pages of experience -- even _tens_of_thousands_ -- decided to leave the site... most of them were proofers who could be counted on to catch every single error on a page and leave it _perfect_... it is not inaccurate, in the slightest, to say that some of these p1 proofers were the very best proofers on the whole d.p. site. i've done lots of d.p. research, and everything i have looked at -- everything! -- indicates that p1 proofers _in_general_ are _just_as_good_ as p3 proofers, and _better_ than p2 proofers. now, we know that p1 includes some _beginner_proofers_ and others who are "unclear about the guidelines" (to put it kindly), so -- to wash out those negatives -- it's patently obvious that _some_ of those p1 proofers are doing an _outstanding_ job... and it is those excellent proofers who are being driven away. simply because they don't want to do p2 (which _bores_ them), or p3 (even worse!), or formatting in any round (it's _stupid_), or post-processing (because all they really wanna do is proof). d.p. should _treasure_ these experienced, skilled p1 proofers. instead, they are doing shit to make their lives uncomfortable. and all in the name of "balance" and to "empty some queues". because p1 proofers churn out the work, and the (admittedly undermanned) p2 and p3 proofers cannot keep up with them, and the post-processors can't keep up with any of the others. but the reason the p1 proofers are being actively harassed is because they churn it out. thus "something needs to be done about the excessive output from p1". it's just plain ridiculous. you know what d.p. should do? send every page through p1 a half-dozen times. or until it goes unchanged by 3 successive p1 proofers, which means it's as clean as it's gonna get, no matter who else looks at it. indeed, roger's work shows that, for some of the p1 proofers, the best of them, you don't even need a "confirming" proof... if they say a page is _good_, odds are that the page is _good_. but if you want to _really_ be sure, have 3 of 'em say it's good. that's what d.p. _should_ be doing -- repeated p1. d.p., stop dissing your p1 proofers! put 'em to work, dammit! but hey, guess how many times i have made this suggestion? _a_lot..._ guess how many times d.p. has listened? _never..._ d.p. sent itself down this wrong path when juliet decided to set up a proofing hierarchy. any "workflow" which demands that _dozens_ of people (with the most difficult jobs) match the output of _hundreds_ of people (with less difficult jobs) who must in turn match the output of _thousands_ of people (with ostensibly the least difficult jobs of all) is bound to fail. it's _guaranteed_ to fail. there's nothing else that it _can_ do. indeed, it's sheer stupidity, on its face. pure sheer stupidity. so now p1 is being brought down to the level of p2 and p3... and everything looks "balanced", and people think "it worked". in reality, they have only succeeded in _deluding_ themselves. but you know what's comical? d.p.'s most congested segment -- post-processing -- really could be completely unnecessary! perhaps you didn't notice the "post-processing" part of when we just went through those lessons on how to digitize a book? that's because _my_ workflow -- like any _decent_ workflow -- has reduced that "compilation" part to just a few button-clicks. but the workflow at distributed proofreaders is outright stupid, so over there, post-processing is a messy complicated endeavor. and that has now levied a human cost, on the p1 proofing crew. the saddest part is that the p1 proofers who are chased away will never come back to help project gutenberg again. never. so the world at large loses extremely valuable contributions. that's the saddest part. *** i haven't done many posts on d.p. lately, mostly because i've completely given up on them. they have created the seeds of their own destruction, so their demise is just a matter of time. the saddest part is they burned out valuable human resources. -bowerbird

1 0

Re: [gutvol-d] these grapes are sweet -- lesson #21c
by Bowerbird＠aol.com 19 Oct '11

19 Oct '11

a little more, while i'm thinking about it... there is some other code i will not share, for various reasons. i'm not going to show how to generate the .ncx and .opf files. that's because it cost me days and days of time and frustration to find out just exactly what was required in these stupid files. the .epub people didn't care how much complexity they piled onto the person who wanted to create an .epub file... indeed, it rather seems they attempted to make it as hard as possible. perhaps because it is the corporate publishers who controlled this format, and they didn't want self-publishers to compete? it's not the first time we've caught them dragging their feet on the e-book revolution, and i suspect it won't be the last, either. when you've done similar legwork to crawl through the crappy "standard" they crammed down our throats, and you're ready to curse the people who created .epub, we can compare our notes. next, i'm not gonna show you how to generate a .pdf either... i bought a commercial plug-in to help me, so if you want to struggle through the slough of open-source stuff, have at it. i just don't wanna work so hard to get something "at no cost". finally, all this stuff that i'm showing you is stuff i developed years and years ago. i've done a _lot_ more, especially since, and i'm certainly not going to hand over the code for all that. or even any code that paves the way to simplify reproducing it. but if you are willing to pay the price -- it runs six figures -- you can buy it all and release it as "open source" to the world. of course, if you just want to _use_ the software, it's cost-free. -bowerbird

2 1

since jim is unwilling to give up his disingenuous pose
by Bowerbird＠aol.com 16 Oct '11

16 Oct '11

since jim is unwilling to give up his disingenuous pose, let me spell things out for you, so everyone understands. jim has looked at my pdf's, and what he found there was plain old unstyled text. now, since this particular book consists of very little styling to begin with -- it's almost totally and completely plain-old-paragraphs, with just the chapter-headers and front-matter and two cases of blockquotes -- that's not much of a "flaw", but if you're jim, you're willing to grasp at any possible straw you can. so let me be clear: the chapter-headers are unstyled, as is the front-matter, and those blockquotes, and the occasional cases of italics. so, for instance, the headers are _not_ big, bold, and centered, as in a final product. italics are indicated only by _surrounding_ _underlines._ blockquotes have the ">" at the beginning of each line... the front-matter isn't centered, like it was in the p-book, or bold, like a title-page should be. so... get the picture? now, could i have done all that styling and formatting? well, yes, obviously, i could have... i've done it before. and, if you care to notice, i put the chapter-headers in the .pdf table-of-contents sidebar, so i recognized 'em. (and because the sidebar is a useful capability, i did it.) but any styling and formatting was _beside_the_point_ i was making with the current demos, so i didn't do it... my point was that you can chop the output many ways, including by page, paragraph, chapter, or _not_at_all_. you can put formatting in, or leave it out, _your_option_. because sometimes ya don't need formatting in the way. most especially when -- as was one of the use-cases -- you have an editor go over every paragraph in the book. my workflow is broad enough to embrace the author who might be _in_the_process_ of actually _writing_ their book. maybe jim doesn't care about that; fine, he doesn't have to. but there's no reason to let his blinders cut off your vision... -bowerbird

1 0

jim seems to want me to start ignoring him again
by Bowerbird＠aol.com 16 Oct '11

16 Oct '11

jim said: > if those are your examples, then I also > encourage readers of this thread to go take > a good look at them and judge for themselves. why don't you make your points directly, jim, so they can be addressed directly, instead of going the slimy route by trading in innuendo? *** jim said: > [Not exactly sure how BB > comes up with his measures!] again, then why don't you just ask directly, jim, instead of trying to imply there is subterfuge? i wrote a program that delivers those numbers. (what?, did you think i count them one by one?) if you want, i can have it generate some output that would give everyone irrefutable proof, jim. do you want that proof, jim?, or are you happy and satisfied just that you got a smear-job in? i guess i'm going to have to kill-file you again. not only do you actively fail to contribute value to the conversation, you actually make it worse. -bowerbird

2 1

modern methods of book composition
by Bowerbird＠aol.com 16 Oct '11

16 Oct '11

andre said: > The Internet Archive has 3 copies from 3 different sources. i see 4. but who's counting? and of the ~15000 lines in this book, approximately 9,000 have identical o.c.r. across all 4 digitizations, with another 3,000 that have identical o.c.r. across 3 of the 4 digitizations, meaning that the "problem lines" can be boiled down to a fairly small subset... i'm busy with another project right now (thanks, alpha-testers!), so if someone else wants to do that research, go ahead please... -bowerbird

2 1

Re: [gutvol-d] some pdf's for you to look at, and consider
by Bowerbird＠aol.com 16 Oct '11

16 Oct '11

jim said: > I'm sorry, but *what* were the links in this latest BB email > supposed to be examples of ??? just providing more and more concrete examples, jim, of the power and flexibility of the z.m.l. format as a "master". so when the tei/xslt folks get their shit together (...right...) they'll know what kinds of output they will need to emulate. (yes, jim, i know that -- unlike 2003, when i arrived here -- the vast majority of techies don't see x.m.l. as "the future", but rather view it as a wrong turn they shouldn't have taken, and have long since abandoned that wrongheaded approach. but some of the crusty people here didn't receive the memo. so it's great entertainment to make fun of their ineptitude... just like how it was when my nephews gave me a jump-rope as a birthday present; what could i do except laugh along?) *** it was also a demonstration of the equivalence of outputs in html form and pdf form, since there is the longstanding belief that those formats are at opposite ends of the pole. when really you can make them as similar as you want... *** and, if you must know, it was laying down the groundwork to provide cover-fire for my upcoming assault upon epub, because it's time to put that travesty out of its misery too... -bowerbird

2 1

Re: [gutvol-d] call for alpha testers
by Bowerbird＠aol.com 14 Oct '11

14 Oct '11

benjamin said: > I will volunteer, then. :) great! but let's do all these things backchannel, folks. no sense cluttering the list with it. thank you... -bowerbird

1 0