October 2011 - gutvol-d - lists.pglaf.org

Re: [gutvol-d] Fwd: Re: epubeditor.sourceforge.net
by Bowerbird＠aol.com 27 Oct '11

27 Oct '11

jimmy said: > if you're only interested in the text content, it's quite useless. that's right. > It is useful for OCR research to have that data that's right too. i'm not saying they shouldn't make that data available. but there's a huge middle-ground they're ignoring. they provide dirty o.c.r. as their "text", and this bloated x.m.l. and when they are confronted about their dreadful o.c.r. text, they respond with "well, you could trawl our x.m.l. instead...". as if throwing _all_ of the data at us is an acceptable answer to our reasonable request for _better_ text in the first place. (and again, i love that alex actually provided a script to us, or at least the location of a script we can run against every book. like i said, i wish there were more people like him over there. he is an excellent example of the way that they _should_ be.) > so I'm glad they provide it i'm glad they provide it too, for the one-tenth-of-one-percent of the population who _might_ be expected to make use of it... but i'm not glad they're ignoring the needs of the much-larger percentage of people who would make use of cleaned-up text. why are they just now, after having collected 3 million books, bothering to develop a script to give people "useable" output? > not as useful as corrected text, granted let me be clear that i'm not asking for "corrected text"... i am asking for text without any globally-replaceable flaws. i am asking for text that has some reasonable resemblance to the structure that finereader can _easily_ give its output. look at the o.c.r. that is offered for "the art of the book" -- with faults that are typical of _all_ of the archive.org books. there's no clear indication of pagebreaks, for crying out loud. yes, the pagebreaks have 3 blank lines, and that's consistent. but there are many other places in the book with 3 blank lines, places which are _not_ pagebreaks. so you can't count on that. really, if they can't even get _that_ right, what are we left with? (coincidentally, it's _possible_ for them to get it right, because the .djvu file itself, which produces the text-file, gets it right. just have abby put a formfeed at the end of each page, stupid.) or consider that, where there were illustrations, this text is littered with nonsense characters. why weren't they filtered? again, it's not as if finereader has no knowledge of pictures... many of the illustrations are saved out specifically, so abbyy had to know the locations, and could've ignored those areas. but the people who devised the workflow simply did not care. they made garbage. and now we have to live in their garbage. or try this on for size. here are two scans from the scan-set: > http://z-m-l.com/misc/artofbook00holm_0108.jpg > http://z-m-l.com/misc/artofbook00holm_0109.jpg ok, i understand, mistakes happen, yadda yadda, and i note that the pages did get rescanned, except without the hand... my question is this: why were these 2 scans still included in the scan-set which stands as the representation of this book? why weren't they simply discarded? and since they didn't get included in any of the output that was generated by archive.org, that means that archive.org has in its workflow a mechanism that says "ignore these"... which seems to my mind like an unnecessary complication, a grafted-on non-solution to a problem that _should_have_ been solved by addressing its cause, not one of its effects... and now every other developer who wants to "add value" to that scan-set is going to have to build in an "ignore these". it's true. the more you know about the archive.org workflow, the more you realize that it's just one jerry-rigged _disaster_. it's all dressed up, in the emperor's new x.m.l. clothes, but the guys who designed it obviously have little competence. or i dunno, maybe their main priority was on "job security" -- nudge, nudge, wink, wink, know what i mean? -- but if that was the case, boy, are they in for a surprise, because archive.org laid off more people than they hired in 2011, if my guess is correct, and as the economy gets _worse_ -- which it certainly will -- they will "let go" even more... and we will be left with a complex mess that no one can even _understand_, let alone _fix_ so it works _correctly_. we might as well not even bother spending the money to create a cyberlibrary, if all we're going to do is waste it... > but I think the clearest example of > the value of such data is reCAPTCHA, > which (in part of its operation) > compares the output of two OCR systems, > and extracts images from > the coordinates where they disagree. you might not know that google bought recaptcha, shortly after which internet archive stopped using it. but yeah, that's one example of using coordinate data. the _clearest_ example, for archive.org, is using the data to highlight a hit found if you search in their flip-books. which, by the way, reminds me about this screenshot: > http://z-m-l.com/misc/prime-example.jpg oops! there's a prime example of a search fail! go ahead. try it yourself: > http://www.archive.org/stream/booksculture00mabiuoft#page/236 the stupid. it's everywhere. (but jimmy's point was good.) -bowerbird

3 2

Re: [gutvol-d] sometimes i despair
by Greg M. Johnson 27 Oct '11

27 Oct '11

From: a(a)aboq.org > I asked you many months ago why you -- as the only person here, > to my knowledge -- insist on contributing to this list in ugly, > bloated HTML emails, instead of in plain-text, which is the gold > standard of mailing-lists, especially (!) tech lists. Constantly > posting in HTML makes you look like a perpetual clueless > newbie. Also, I made you aware many months ago that your > cr*ppy AOL email software -- again, as the *only* contributor's > here -- incessantly breaks the proper threading of messages > from this list. With every contribution you send to this list using > AOL, you break a discussion thread, creating havon on (for > example) my iPad and iPhone, etc. If you don't give > a damn about messing up other people's inboxes on a daily basis, > you can't be surprised at the lack of civility you're experiencing, > or the lack of interest in your innumerable projects. Completely agreed on every point. I cannot read pg emails on an ipod touch. I also subscribe to a Linux User's Group which appears to be using exactly same email software, and I can read them. -- Greg M. Johnson http://pterandon.blogspot.com

1 0

barbara freethy owes michael hart a large pizza
by Bowerbird＠aol.com 26 Oct '11

26 Oct '11

barbara freethy. nope, i've never heard of her. have you? she is an oft-published and award-winning author, with novels from pocket, n.a.l., and harper-collins, 30 of them all told, constituting a hefty backlist... but no, sorry, the name doesn't ring a bell with me. however, according to the sacramento bee, which quotes a vice-president from barnes&noble digital, ms. freethy has sold a lot of e-books of her backlist. > http://www.sacbee.com/2011/10/25/4005206/author-barbara-freethy-sells-over.… over a million e-books. of her backlist. just in 2011. (which, according to my calendar, is only 10/12 over.) her e-books are priced at $3-$6, which means freethy is taking home $2-$4 on each one. times one million. thank you, michael, for setting writers like freethy free... -bowerbird

3 3

Re: [gutvol-d] Fwd: Re: epubeditor.sourceforge.net
by Bowerbird＠aol.com 26 Oct '11

26 Oct '11

i'm not directing _any_ of this at you, alex... none... these are things i've said before, and will say again, here and elsewhere in cyberspace, many many times. you seem swell, alex, all reasonable and everything. i only wish there were more like you at archive.org... especially the people who _decide_ on the workflow. and yes, one-off corrections don't impress me any. until the entire library is fixed, no one can develop apps that will add value to the library qua library... just in case anyone is unclear about what i've been talking about, i've appended the x.m.l. data saved for a page in "the art of the book" that has nothing more than a heading on it that says "great britain". > http://www.archive.org/stream/artofbook00holm#page/n12/mode/1up i think we can agree that that's a whole lot of mud that we need to scrape off a page that has 2 words. -bowerbird p.s. i changed the angle-brackets to square-brackets, so as not to confuse them with any .html processing... ---------------------------------------------------- [page width="2935" height="4285" resolution="300" originalCoords="true"] [block blockType="Text" l="594" t="802" r="2082" b="966"] [region] [rect l="594" t="802" r="2082" b="966"] [/rect] [/region] [text] [par lineSpacing="128"] [line baseline="958" l="608" t="832" r="2064" b="960"] [formatting lang="EnglishUnitedStates" ff="Times New Roman" fs="44." bold="true" spacing="-16"] [charParams l="608" t="836" r="736" b="960" wordStart="true" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="40" wordPenalty="0" meanStrokeWidth="205"] G [/charParams] [charParams l="748" t="832" r="852" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="100" wordPenalty="0" meanStrokeWidth="205"] R [/charParams] [charParams l="864" t="832" r="964" b="960" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="100" wordPenalty="0" meanStrokeWidth="205"] E [/charParams] [charParams l="972" t="836" r="1096" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="69" serifProbability="98" wordPenalty="0" meanStrokeWidth="205"] A [/charParams] [charParams l="1100" t="832" r="1232" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="255" wordPenalty="0" meanStrokeWidth="205"] T [/charParams] [charParams l="1232" t="832" r="1304" b="958" wordStart="false" wordFromDictionary="false" wordNormal="false" wordNumeric="false" wordIdentifier="false" charConfidence="255" serifProbability="255" wordPenalty="0" meanStrokeWidth="0"] [/charParams] [charParams l="1304" t="836" r="1404" b="956" wordStart="true" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="98" wordPenalty="0" meanStrokeWidth="236"] B [/charParams] [charParams l="1416" t="832" r="1524" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="100" wordPenalty="0" meanStrokeWidth="236"] R [/charParams] [charParams l="1536" t="836" r="1584" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="100" wordPenalty="0" meanStrokeWidth="236"] I [/charParams] [charParams l="1596" t="832" r="1728" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="255" wordPenalty="0" meanStrokeWidth="236"] T [/charParams] [charParams l="1728" t="836" r="1852" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="100" serifProbability="98" wordPenalty="0" meanStrokeWidth="236"] A [/charParams] [charParams l="1864" t="836" r="1916" b="956" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="72" serifProbability="100" wordPenalty="0" meanStrokeWidth="236"] I [/charParams] [charParams l="1932" t="836" r="2064" b="960" wordStart="false" wordFromDictionary="true" wordNormal="true" wordNumeric="false" wordIdentifier="false" charConfidence="82" serifProbability="100" wordPenalty="0" meanStrokeWidth="236"] N [/charParams] [/formatting] [/line] [/par] [/text] [/block] [/page]

2 1

Re: [gutvol-d] Fwd: Re: epubeditor.sourceforge.net
by Bowerbird＠aol.com 26 Oct '11

26 Oct '11

somebody at archive.org said: > KH: It's a big file because it includes > lots of the original geometry information and > wraps each word in its own span to support corrections. just another example of their ludicrous workflow over there... the important content from the o.c.r. process -- the text -- gets muddled inside a huge amount of other data which they consider to be "necessary" solely because their major focus is on the scans. and, just in case you didn't realize it, _this_ is why the _search_ capability at archive.org is so darn slow, often to the point that the machine gives up, meaning you get no results at all, because the search routine has to plow through so much irrelevant data. -bowerbird p.s. in the rare case when one needs to know the coordinates of a word's placement on the scan, one can easily determine it. the routine for finding a line's baseline is simple, and very fast, and -- turned on its side -- works fine for locating each word. i once posted a sample graphic, but i can't run it down it now. but for heaven's sake, if you feel that you _must_ save the data, then at least have the good sense to store it in a separate file...

3 2

Re: [gutvol-d] roger, your code _could_ scale just fine, if only you'd...
by Bowerbird＠aol.com 25 Oct '11

25 Oct '11

keith said: > Are you sure you know what a revision control system is i'm absolutely positive i do. > or how they work! do you wanna have a discussion about the relative merits of c.v.s. versus git versus mercurial? or what? (i roll my eyes.) > I am inclined to think not. think what you like. i still don't wanna discuss it with you. but if you want to inform yourself a little better, you could trot over to d.p., and see how they handle diffs over there. for tiny text files, typically with the same number of lines even after their most extensive edits, diffs are _a_breeze_. to involve "a revision control system" -- as the term has come to be known today -- would be _massive_overkill_. -bowerbird

1 0

roger, your code _could_ scale just fine, if only you'd...
by Bowerbird＠aol.com 25 Oct '11

25 Oct '11

roger, i hope you're not listening to this interchange between keith and lee. as i demonstrated when i did my example site using your "sitka" book, it's easy to save successive iterations of the text for each page, and then simply run diffs on them to spot changes... saved iterations also make it easy to revert a change. so there's just no need for a "revision control system", or such nonsense, even if we could find "a simple one" (which i am inclined to think doesn't even really exist). continually resist inclinations to unnecessary complexity. -bowerbird

2 1

roger, your code _could_ scale just fine, if only you'd...
by Bowerbird＠aol.com 25 Oct '11

25 Oct '11

roger said: > Even if I agreed with all the technical points, > there is what feels like a showstopper to me > that's equally if not more important. > Running a cooperative workflow of volunteers > involves motivating people. A leader of > such an effort can't just tell them what to do, > or sometimes even how to do it. They need to > gain their good will, evoke their interests, > stimulate their creativity, and communicate > with them in an open manner. this is, of course, very astute. and it reminds me of something that juliet said in her recent post, which i will return to shortly... but for now, i want to concentrate on this, instead: > I coded it myself and ran it on a non-commercial, > personal server. Neither the quality of the code nor > the bandwidth to the hardware were scalable, and it > wasn't my intention to run a full production site anyway. i do believe you are mistaken about "the quality of the code", roger, in terms of whether it could _scale_ to full production. or rather, you're correct, but you fail to see the real problem. you followed "the conventional wisdom" which tells coders that they should _not_ use the file-system as their database, but should instead use a "proper" database, typically mysql... but for your purposes, that was a mistake, roger. trying to funnel all of your users, and all their transactions, through a single "gate" -- a.k.a., your mysql database -- is _what_ complicated your code and caused the scaling issues. what you should have done instead is to write a program that can be dropped into a folder _with_ the files for a single book, which then _administers_ the transactions involving those files. since most of the files are small, changes are not a big deal... besides, the fact is that most of the files will never be changed. even ones that _are_ changed will only undergo a few iterations. so there is nothing that's going to overwhelm the file-system... thus with -- maybe -- _ten_ people on a book simultaneously (at the _most!_, and more typically probably just one to three), and few files being changed, there _are_ no "scaling" issues... load is distributed across separate scripts in different folders. that was your main mistake, roger, especially vis-a-vis scaling. (your other big mistake, which increased your complexity load unnecessarily, was your failure to use intelligent file-naming.) your system can easily be changed to accommodate a site that simultaneously digitized hundreds of books, even a thousand... (of course, it'd be smarter to set the number to a few hundred, and not let a new book come in until a current one is finished, so as to avoid the d.p. situation where thousands sit undone.) but you carried away the wrong lesson if you think your code "won't scale", roger... -bowerbird

3 3

Re: [gutvol-d] Fwd: Re: epubeditor.sourceforge.net
by Bowerbird＠aol.com 24 Oct '11

24 Oct '11

alex said: > There's nothing stopping someone > writing an abbyy.gz to plain (non-encumbered) html converter, > aside from effort, you know... this is the type of response we often get from archive.org. because it insulates them from criticism of their workflow. and allows them to float mindlessly in their little bubble... look, people, it's _stupid_ to embed the important data in a mass of unimportant data. it's the _wrong_ way to do it! _stupid_. and _wrong_. do you understand those words? yes, of course, we can clean the crucial data out of the mess. but why should we have to do that? why are _you_ taking action that _requires_ us to do that? i mean, really, when you boast with pride that you've made "three million books available to the public", do you really want us to have to decode that boast to its more accurate "we've put three million books inside of a big pile of mud, and there's nothing stopping you from pulling them out..." is that _really_ what you want to say? because that is _really_ what you are _actually_ saying now. besides, even though it might be rather simple to _write_ such a "converter" -- although it shouldn't output "html", as you said, but rather the text directly -- it's another thing entirely to then execute that converter on _3_million_texts_. _any_ kind of bottleneck, no matter how small, grows large when you have to do it a million times -- let alone 3 million. and if you are talking about a _big_ bottleneck, like parsing x.m.l. files that are 58 megabytes (like the current example) -- again, across a corpus that includes millions of books -- then we're talking about very _significant_ processor-time, not to mention bandwidth issues if we have to retrieve them, and storage concerns (even to save only the smaller output). not to mention that then we've essentially forked your library. so _you_ doing it _right_ in the _first_place_ is very important. > (not trying to snark at BB) nor am i trying to snark back. -bowerbird

2 1

some questions for peter brantley and archive.org
by Bowerbird＠aol.com 24 Oct '11

24 Oct '11

i see internet archive is doing another "books in browsers" conference, with one of the themes being "beautiful books". so i have two questions. first, isn't it a bit presumptuous to speak of "beautiful books" when the actual .epub and .mobi e-books you serve to people are filled with the distressingly ugly crap of uncorrected o.c.r.? second, the point about "books in browsers" in something that is a bit misleading. i don't believe anyone would question that the e-books in our cyberlibrary must be viewable in a browser. but is that the _only_ way that they should be viewable? or even the most important way they could be viewed? let me rephrase the question, to illustrate the point. assume i said "twitter in browsers", or "facebook in browsers". again, i don't think anyone would argue that twitter or facebook should _not_ be available in a browser. that would be ridiculous. but it would be equally ridiculous to put forward the argument that twitter or facebook _only_ be available in browsers, or even that browsers should be the _predominant_ platform for them. we've learned that app access is typically _much_ more friendly. the world of software applications has lately come to appreciate in a very deep manner the importance of decoupling the content of a website from the browser-interface used to see that content. nowadays, the a.p.i.'s which you offer are _extremely_ important in terms of leveraging the relevance of your content by allowing outside developers to add value to it in ways you hadn't foreseen. particularly with e-books, the chrome and functionality that is offered by the web-browser are _not_ particularly appropriate to the interface, the content, or any of the reading experience. to the extent that you focus on the browser, you miss the point. -bowerbird

1 0