October 2011 - gutvol-d - lists.pglaf.org

Fwd: Re: epubeditor.sourceforge.net
by Lee Passey 22 Oct '11

22 Oct '11

Comment from the current author: [11:47:01 AM] KH: It's a big file because it includes lots of the original geometry information and wraps each word in its own span to support corrections. He didn't say this, but you're right in that it's also to help support the flippy setup. As far as the script, the actual php itself is still internal only, as it's part of the main codebase (which isn't open). Perhaps I can see about making it public. You can use it on any item, however, by visiting "http://www-kenh.archive.org/download/<IDENTIFIER>/<IDENTIFIER>_abbyy.html" Alex -- Alex Buie Network Coordinator / Server Engineer KWD Services, Inc Media and Hosting Solutions +1(703)445-3391 +1(480)253-9640 +1(703)919-8090 abuie(a)kwdservices.com On Fri, Oct 21, 2011 at 11:33 PM, Lee Passey <lee(a)novomail.net> wrote: > On 10/21/2011 9:58 AM, Alex Buie wrote: > >> Lee, >> >> There's a very alpha script we're testing you can try out. (No >> guarantee, may burn the house down, etc) >> >> >> http://www-kenh.archive.org/download/artofbook00holm/artofbook00holm_abbyy.… > > What we have here is not the script, but the output of the script as applied > to "The Art of the Book." As you might expect, it's a mess. Like all > FileReader output it is in SGML HTML, which XML parsers cannot use. It > appears that in standard usage at the Internet Archive, the primary (perhaps > only) reason for doing OCR is to support searching inside the FlipBooks (or > the PDF files which appear to be the same thing in a different format). > > What I see in this file is that /every/ word in the file has a surrounding > <span> of class "abbyyword" and a title attribute containing the coordinates > indicating where that word appeared in the image. The end of every line that > was detected is indicated by <br class="abbyybreak">, and every line begins > with an <a>nchor of class "abbyyline" that not only has an identifier it > also has data indicating where the line starts in the image. > > The file is full of this sort of cruft; if we want data that is not > inextricably tied to a picture, we don't need any of this stuff. > > BUT... > > If this is the best I can get, I'll take it. I think I can write some XSL > scripts, maybe together with some real programming, that can reduce the > garbage to a manageable level. I would much rather have too much data than > too little, because its easier to ignore irrelevant data than it is to guess > at the existence of unknown data. > > Whoever wrote this should be continued to improve it; don't let her/him > become discouraged. And if IA doesn't have the resources to pursue it, > please make it available in the condition it's in now. > > If you could provide a pointer to the actual script, that would be really > great. >

1 0

epubeditor.sourceforge.net
by Bowerbird＠aol.com 22 Oct '11

22 Oct '11

lee said: > When you say "o.c.f." am I to > assume you mean the ePub file? oh, my apologies, i screwed up. i meant the .opf file, required in an .epub, much like the .ncx file. sorry... > Where do you want me to upload them? anywhere is fine. you could even copy it into a post here. > Be warned, the .epub file is almost 6 megabytes in size. oh, heavens no, i wouldn't ask you to do anything like that. i mean the simple file that lists the manifest of other files... (but now i'm curious as to what this 6-meg file would be?) again, though, my apology for the error. -bowerbird

3 3

Re: [gutvol-d] epubeditor.sourceforge.net
by Bowerbird＠aol.com 22 Oct '11

22 Oct '11

lee said: > I didn't want to clutter up the mailing list with > files that probably only BowerBird is interested in, > so I sent them to him back channel. never got 'em. and yes, i looked in my spam folder, because that's where your posts go, automatically... (you know, that kill-filter thing.) but that's ok, i don't really need the files. seriously. i'd much rather have you provide people with a primer on them, as well as some easy-to-understand templates, since -- if you will remember -- you had _criticized_ my unwillingness to provide source-code that would create such files. if i could've shown that source-code without giving too much information to the people here who have made me their "enemy", i certainly would have. so that's why i did not. but if you would be willing to do some tutoring of them, so i knew that they were actually in possession of the same knowledge that i already have, then i _might_ be willing to part with my source-code... or, ya know, you can always give 'em _your_ source-code. which they could turn into perl or python. (yeah, right.) > You see, I cheated. well, what you did wasn't really "cheating", but... > ePubEditor is not a tool that is used to take > simple text files and convert them to HTML. ...that was pretty much what i wanted you to admit. because without .html, one cannot make an .epub. so your "epubeditor" cannot _create_ a new .epub -- at least not using a text-file as its input-file -- it can only _edit_ an existing .epub or .html, right? so it's not as useful or powerful as we might have reasonably expected, given the name of the app... and certainly not very useful to the d.p. people... i'm not saying it's useless. it still has a purpose... but turning plain-text into .html is "the hard part". after that, it's just generating some metadata files. > There are other programs out there > which do that at least as well as I could well, maybe you under-rate your own skills, lee, since many of those "other programs out there" get thumbs-down from most e-book-designers, since the .html they create is considered crappy. _you_ wouldn't turn out crappy .html, would you? > (not the least of which is FineReader itself). well, first of all, not everyone owns finereader. more importantly, it is one of those programs that creates .html that e-book-designers hate. just so you know... > ePubEditor's "CreateNCX" function relies on > the existence of an Table of Contents made > up of one or more lists, optionally nested. so if a person makes an .html table of contents, your program can create an .ncx version from it. that's better than nothing, to be sure, but it would be even better if it could create _both_. (and yes, i see where you later noted that it can create both, provided the person has marked up their .html file appropriately, but all that is just resetting the place where you put responsibility on the person, rather than having the tool do it.) > This file may be created by hand, and in the > case of small works it is trivially easy to do so. um, yeah, but isn't it the "trivially easy" tasks that we want our computers to be performing _for_ us? just sayin'... > I have no idea why IA didn't OCR this page as well they did... but their quality-assessment routines said the results were too shabby to use, so they substituted the page-scan instead... not ideal, but understandable. they would sub in every page this way, if they could, but the e-book would be too bloated to do anyone any good. > Referring to the image I was able to create > an HTML TOC in about 15 minutes i really didn't mean for you to go to all of that work. i'm sorry now that i asked... it won't happen again... oh, and you kids out there, don't try this at home! if you want to _create_ and/or _edit_ an .epub file, your best bet is with "sigil" -- a free/free program that behaves quite like a regular word-processor, and then saves its files in .epub format. sigil does a bang-up job of creating your .opf and .ncx files _for_ you, based on the text that you have entered (or retrieved from an o.c.r. file), all _automatically_. (although it also lets you go in and edit manually.) there are other .epub creators out there nowadays, including "legend maker", but that one costs $40, so why not just go with the free/free "sigil" instead? and remember, if you want .pdf and .kindle as well, you'll be able to get that from my "jaguar" editor... it's _not_ open-source free, but _is_ cost-free free. > In the case of the "Art of the Book", there were > not headers marked. In fact, every block of text > was marked as though it were a paragraph. welcome to the second-biggest book-scanning operation on the entire planet, lee. and yes, we're in deep doo-doo. > (If you're going to call everything a paragraph, > why add any markup at all? because you can't make an .epub without having markup. but nobody said that the markup needed to be any good! that was one of the things you "forgot" to put in the spec. > It makes no sense. on the contrary, if you need to be able to "offer .epubs", it makes perfect sense. it's the only thing you _can_ do. > a division of the text that has no content? > I'm beginning to think the people at Internet Archive > simply don't understand how to use HTML. surely you must be mistaken, lee. and _badly_ mistaken. the internet archive is one of the _leading_authorities_ on "books in browsers". indeed, they run a yearly conference with that very title. and -- as a matter of fact, since i have just mentioned their yearly conference -- i should tell you that this year's version is actually being held _very_ soon! here, let me google that for you... ok, yes, it's next week: > http://bib.archive.org/ and lee, one of the "themes" of the event is "beautiful books". which introduces me to a post i was going to make next week, but will probably make today, now that i stole my own thunder. anyway, lee, good luck with the app. when you have the mods necessary so it can run in toto on a mac, please let me know... (i got it to execute ok, and could do some things, but not all.) -bowerbird

2 3

amazon announces yet another milestone
by Bowerbird＠aol.com 21 Oct '11

21 Oct '11

ok, this is a milestone that i've been waiting for... amazon has announced the successor to its .mobi format. "is it .epub?" of course not, you idiot. that's the dinosaur format. kf8 -- "kindle format 8" -- replaces .mobi version 7, and is based on html5, with css3. (anybody got a calculator?) it also contains some features related directly to e-books, like drop-caps, callouts, and sidebars... another inclusion which i believe will prove to be extremely valuable is s.v.g. as i said, i have been anticipating this development, and am now more excited than ever about the upcoming release of "jaguar", my text-editor with a unique e-book orientation... -bowerbird

2 1

Re: [gutvol-d] chasing away the p1 proofing crew
by Bowerbird＠aol.com 20 Oct '11

20 Oct '11

andrew said: > I've seen a more concerted effort to > encourage users to work in higher rounds this is the way the harassment of the skilled long-term p1 proofers tends to be "officially" described. not that andrew is "an official", or anything. unofficially, seasoned p1 proofers are accused of "poaching the easy stuff", which is, of course, plainly ridiculous. (most of them prefer p1 precisely because it gives 'em more to do.) everyone who has been paying attention to what happens at d.p. knows about this. -bowerbird

2 1

Re: [gutvol-d] roger, your code _could_ scale just fine, if only you'd...
by Bowerbird＠aol.com 20 Oct '11

20 Oct '11

sorry, keith, but i don't wanna have this dialog with you. you're talking about this in a general "theoretical" sense, which allows you to be extremely vague and nonspecific. on my side, i have actual code that i have already written, which runs and does the job well and uses few resources. so, you know, nothing here is "broken", so i don't need to "fix" it, even if i thought you had "the cure", which i don't. i was communicating something important to roger, and i trust that he understands it, or will ask me if he doesn't. -bowerbird

1 0

Re: [gutvol-d] roger, your code _could_ scale just fine, if only you'd...
by Bowerbird＠aol.com 20 Oct '11

20 Oct '11

keith said: > This wisdom is absolutely a must! sigh. keith, you're a perfect exemplar of the technocrats that i was just talking about... and making fun of... they saw lots and lots and lots of cases where someone used the file-system as a database and then ended up having problems with scaling when it eventually grew. so those technocrats then came to believe that they'd seized upon something "inevitable", so they started to enforce the iron-clad rule that "you must _never_ use your file-system as a database, because it won't scale". and they're proud of themselves, and puff their chests, and slap themselves on the back for this bankable info. furthermore, they repeat the mantra to each other often, confident that they've come across some ultimate truth... the problem is, there are some exceptions. and this is one of them. as long as you have a script working on only one book, consisting of just a few hundred page-scan images and a few hundred corresponding text-files, and _further_, you have _no_more_ than a dozen people working on it (if that, and usually not simultaneously), then you are _not_ going to have any "scaling" issues with that script, especially if 80% of the text-files will never be changed. (and none of the graphics files will ever change, save for occasional swapping-out of some badly-imaged pages, which isn't even something that the script will manage.) and even if you have a team of a dozen people working simultaneously on a book, they'll finish in half-an-hour. so you are _not_ going to have any problems with "load". and the important thing is that we _know_, for _certain_, that the per-book task will _never_ exceed beyond that... so we will never be taken "by surprise" by scaling issues. we might gain a ton of users, and eventually be working on thousands of books at the same time, but _each_book_ will still be a very small bite, easily chewed and swallowed. more users would mean more hardware, and bandwidth, and more copies of the script running in different folders, but each script is still going to be doing a very easy task... that is why this particular usage is _an_exception_ to the rule that you shouldn't use the file-system as a database. so your "advice" is wrong, keith. it's just flat-out wrong. and i was watching closely as roger developed his system. so i saw him make the error as he was in the process of it, and i saw the ramifications of the error... all of a sudden, things were squirreled away in a database, thus opaque... in this type of workflow, you want content out in the open. and equally important here is the point that roger needs to know that he _can_ mount a full-on site, if he _wants_. he's now suffering from the misimpression that he can't. maybe he doesn't _want_ to do that, that i'd understand, but i don't want him to think he can't, when really he can. > What most do not understand that in order to have > a "proper" database for any non-trivial task, you > must add layers of abstraction in order to administer > all transactions and function needed. Without them > a system will eventually bog down and will not scale well. this is the other sickness that technocrats have. first they introduce complexity. then, when the complexity starts to make it difficult to handle a very simple situation, they add _more_ complexity, as if that's gonna solve it all... and _that_ is what causes things to "eventually bog down"... -bowerbird

2 1

epubeditor.sourceforge.net
by Bowerbird＠aol.com 19 Oct '11

19 Oct '11

lee said: > parsimonious i'll comment on that later, but for now let's focus on this: > http://epubeditor.sourceforge.net lee, please generate o.c.f. and .ncx files for these books, both of which have received some attention here recently: > the art of the book, by charles holme, 1914. > http://www.archive.org/details/artofbook00holm > the practice of typography: modern methods of > book composition, by theodore low de vinne, 1904. this is a timed test, so don't dawdle... -bowerbird

2 1

Re: [gutvol-d] ePubEditor
by Lee Passey 19 Oct '11

19 Oct '11

On Wed, October 19, 2011 11:12 am, James Adcock wrote: >> On Oct 7, 2011 I registered the project "ePubEditor" with SourceForge.net, >> and uploaded all my code there (http://epubeditor.sourceforge.net/) > > I tried this for a couple minutes but it always crashes when I try to open > an epub file. Perhaps include a sample file that works in your downloads? Thank you for your feedback. When ePubEditor opens an ePub file, the first thing it does is explode the entire contents of the .zip file into a temporary directory. The location of this temporary directory is read from the file "ePubEditor.ini". By default, this location is set to "C:/My Library/temp". There was a bug (not surprising) in the downloadable package that caused an error message to be displayed if the temporary directory does not exist. You can work around this problem by creating the directory "C:/My Library/temp", or you can edit the ePubEditor.ini file to point to some other temporary directory that does exist. At some point I intend to create a fancy user interface to edit the preferences but for now, I guess notepad is the tool. I have fixed the code to create the temporary directory first before trying to extract the files. I will probably upload it to SourceForge before the end of the week. I will try to add the work-around to the readme at SourceForge tonight.

1 0

chasing away the p1 proofing crew
by Bowerbird＠aol.com 19 Oct '11

19 Oct '11

jim said: > I don't think DP has changed anything else > so what I think they have done is simply > reduced the rate of books accepted into the system > in the first place, increasing the (implied) queue > of books digitized but not begun proofreading. > What this does, imho, is simply chase away > volunteers who want to proof the lower rounds. > But that is probably better than simply > wasting those volunteer efforts by having > too much stuck on queue in later rounds. i'm constantly amazed by the people who think they can "analyze" what's happening at distributed proofreaders from a distance, by looking at a chart or some statistics. it's a social system, and thus a lot more complex than that. but if you wanna reduce it all down to something simplistic, _this_ is what has been happening at d.p. the past few years: they have systematically driven away the p1 proofing crew... the mantra at d.p. used to be "do whatever job you choose". alas, no more. you can no longer opt to stay in p1 forever. much new material is _officially_ made "unavailable" to you, either permanently or "temporarily" -- for a length of time that usually means it's finished before it becomes available. (material that doesn't get finished is typically undesirable.) perhaps just as bad (or maybe even much worse), if you read between the lines in the forums, you can tell that volunteers who choose to stay in p1 are being harassed for that choice by some other volunteers. (at times, it even becomes overt; the harassers operate with the indignation of the righteous.) so frontchannel and back, officially and via "peer pressure", volunteers are being chased away from doing p1 proofing... this is a travesty, pure and simple. probably even a tragedy. many of the old-school p1 proofers, who had been doing it for _years_, with literally thousands of pages of experience -- even _tens_of_thousands_ -- decided to leave the site... most of them were proofers who could be counted on to catch every single error on a page and leave it _perfect_... it is not inaccurate, in the slightest, to say that some of these p1 proofers were the very best proofers on the whole d.p. site. i've done lots of d.p. research, and everything i have looked at -- everything! -- indicates that p1 proofers _in_general_ are _just_as_good_ as p3 proofers, and _better_ than p2 proofers. now, we know that p1 includes some _beginner_proofers_ and others who are "unclear about the guidelines" (to put it kindly), so -- to wash out those negatives -- it's patently obvious that _some_ of those p1 proofers are doing an _outstanding_ job... and it is those excellent proofers who are being driven away. simply because they don't want to do p2 (which _bores_ them), or p3 (even worse!), or formatting in any round (it's _stupid_), or post-processing (because all they really wanna do is proof). d.p. should _treasure_ these experienced, skilled p1 proofers. instead, they are doing shit to make their lives uncomfortable. and all in the name of "balance" and to "empty some queues". because p1 proofers churn out the work, and the (admittedly undermanned) p2 and p3 proofers cannot keep up with them, and the post-processors can't keep up with any of the others. but the reason the p1 proofers are being actively harassed is because they churn it out. thus "something needs to be done about the excessive output from p1". it's just plain ridiculous. you know what d.p. should do? send every page through p1 a half-dozen times. or until it goes unchanged by 3 successive p1 proofers, which means it's as clean as it's gonna get, no matter who else looks at it. indeed, roger's work shows that, for some of the p1 proofers, the best of them, you don't even need a "confirming" proof... if they say a page is _good_, odds are that the page is _good_. but if you want to _really_ be sure, have 3 of 'em say it's good. that's what d.p. _should_ be doing -- repeated p1. d.p., stop dissing your p1 proofers! put 'em to work, dammit! but hey, guess how many times i have made this suggestion? _a_lot..._ guess how many times d.p. has listened? _never..._ d.p. sent itself down this wrong path when juliet decided to set up a proofing hierarchy. any "workflow" which demands that _dozens_ of people (with the most difficult jobs) match the output of _hundreds_ of people (with less difficult jobs) who must in turn match the output of _thousands_ of people (with ostensibly the least difficult jobs of all) is bound to fail. it's _guaranteed_ to fail. there's nothing else that it _can_ do. indeed, it's sheer stupidity, on its face. pure sheer stupidity. so now p1 is being brought down to the level of p2 and p3... and everything looks "balanced", and people think "it worked". in reality, they have only succeeded in _deluding_ themselves. but you know what's comical? d.p.'s most congested segment -- post-processing -- really could be completely unnecessary! perhaps you didn't notice the "post-processing" part of when we just went through those lessons on how to digitize a book? that's because _my_ workflow -- like any _decent_ workflow -- has reduced that "compilation" part to just a few button-clicks. but the workflow at distributed proofreaders is outright stupid, so over there, post-processing is a messy complicated endeavor. and that has now levied a human cost, on the p1 proofing crew. the saddest part is that the p1 proofers who are chased away will never come back to help project gutenberg again. never. so the world at large loses extremely valuable contributions. that's the saddest part. *** i haven't done many posts on d.p. lately, mostly because i've completely given up on them. they have created the seeds of their own destruction, so their demise is just a matter of time. the saddest part is they burned out valuable human resources. -bowerbird

1 0