- gutvol-d - lists.pglaf.org

Re: Obituary for Michael Hart
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

greg said: > In short, I think anyone who wants to > may edit the main Wikipedia page > (it's not locked or otherwise protected). the page is not locked, or otherwise protected. but the idea that "anyone who wants to" can edit a wikipedia page has been a myth since... oh, i dunno, about 2005 or so. you can _change_ a page, at least temporarily, but if one of the wikipedia insiders doesn't care for what you've written, they'll revert it. and there's not too much you can do about it. unless there are some wikipedia insiders here?... otherwise, let's look and see what they _have_, because that's likely to be what they will keep... first, they were very prompt entering his death. the timezones are a bit confusing, but it looks like they had it entered before greg's post here. (that's not surprising; there's competition over who will be the first person to enter any news.) and, as you'd expect, the attention meant that a flurry of edits have been made in the interim, most of them are rather inconsequential, but at 23:04 on the 8th, kaldari did an edit noting michael as "best known as the inventor of the e-book and the founder of project gutenberg", which i believe that michael would consider as the most important phrase describing himself. (and face it, "the most important phrase" is all that history and the great masses recall.) however, at 12:10 on the 9th, theo10011 did an edit (which s/he labeled a "minor rewrite") that changed "the inventor of the e-book" to "an inventor of e-book". this strikes me as a cop-out (and in addition the grammar's bad.) the washed-down version might be a bone that's being thrown just to avoid contention, because a lot of wiki editors shy away from statements that might be "controversial"... certainly there were people who wanted to bestow that "inventor" title to someone else; a common suggestion was vannevar bush, but i got the firm impression that some of michael's detractors would have been happy to have _anyone_ be awarded that little title, as long as it wasn't michael. petty bastards. now _perhaps_ kaldari will come back and revert the change that theo10011 made, so michael will again be called "the" inventor... or maybe it will just stick as "an inventor"... in the long run, it might not matter much. it's certainly true that michael is laughing about such trivia now that he has passed. he never got enough credit when he was alive and kicking, so why would he expect to get any more credit now that he's dead? but if the page _does_ get changed, it must be a change from one of the insider editors, who are all quite familiar with outsiders who want to barge in and "spruce up a page" as a way to deal with their grief from a death. -bowerbird

1 0

these grapes are sweet -- lesson #05
by Bowerbird＠aol.com 09 Sep '11

09 Sep '11

we set aside today's prepared lesson, in order to wish godspeed to our dear friend michael hart... we will attempt to continue tomorrow, because that's what michael would have wanted us to do. -bowerbird

1 0

oh man that hits hard
by Bowerbird＠aol.com 07 Sep '11

07 Sep '11

oh man, that news really hits me very hard. michael lived a very good life, and he got to see his dreams come true, so i am sure that he's happy upstairs. but it's gonna take me some time to get used to being without his spunkiness down here on the mortal plane. -bowerbird

2 1

these grapes are sweet -- lesson #04
by Bowerbird＠aol.com 07 Sep '11

07 Sep '11

the thread explains how to quickly and easily digitize a book... i did the book "books and culture", by mabie, which you can find by searching archive.org for "booksculture00mabiuoft". *** first off, a little note for the future lessons in this thread... in order to do your digitization, you're going to need to use a text-editor or word-processor that can do reg-ex searches. on the mac, i use a program called "tex-edit" (not "text-edit", but "tex-edit", where "tex" stands for texas) by tom bender... if you've been confused by regular expressions in the past, and are still gun-shy, don't worry about it. i promise that we won't become obsessed in the overly-complex details... but you need to be able to search for specific simple patterns, such as "lowercase followed by space followed by uppercase". i'll specify that combo as "lowercase-space-uppercase", and you'll need to know how to translate it into a reg-ex search. for text-edit, just to give you an example, that would be this: > [a-z] [A-Z] and you would specify that the search is to be case-sensitive. (tex-edit has a button for that, but your system might differ.) you also need to learn how to search for "whitespace", which might be a regular space, but might also be a linebreak or tab. (it's very important that you be able to search for a linebreak; some word-processors, even advanced ones, will not do that.) finally, another handy thing about a reg-ex search is that you can specify a search for multiple items, which is a time-saver. so you could do one search that would find a forward-slash and a backward-slash and a left-bracket and a right-bracket and a cent-sign and a tilda and a downward-sloping accent and an @-sign and a #-sign and a $-sign and a %-sign and an ^-sign and a & and an asterisk and "=" and an underbar. all in one search. which is faster than doing 15 searches... my pledge, in this series, is to do all of the digitization with _only_ my word-processor. if i do need a specialized tool, i will program it, and make it available online for you to see, and use, if you would like to replicate the work i am doing. (which would be a great way to learn how to do it yourself!) *** in the last lesson, we discussed "auto-cleaning" -- changes that are made to the o.c.r. file globally, without monitoring. auto-cleaning can fix a huge number of errors -- at times literally _thousands_ of them -- so it's incredibly efficient, and its great usefulness should never be underappreciated. for instance, in our mabie book, auto-cleaning made fixes to 543 lines, out of the 5700 in the body-text, right at 10%. those are fixes that cost you zero time or energy. right on! but that's just the first type of cleaning you will want to do... *** so now let's discuss the second type of cleaning you can do. after having done auto-cleaning, i turn to "high-probability" searches that quickly locate o.c.r. errors which are common. over the years, i've posted messages to educate this list of an amazing number of routines that help find o.c.r. errors. once again, here i am, doing the same thing another time... right about now, you're probably starting to get pissed off with my pedagogical tone... you might be saying "boy, that bowerbird certainly thinks he smart, doesn't he?" or maybe you'd like to take me down a few notches, knock me off my high horse. that's why the people on this list _rarely_learn._ they're too busy getting all riled up because of my "tone"... if you woulda been paying attention, instead, you would've learned quite a lot over the years, and you'd be thanking me. lucky for you, though, someone once created a collection of my tips, translating them into regular-expression searches, and posting them to the web somewhere... i'm not going to tell you where -- mostly because i never wrote that down -- but maybe someone did, and is willing to share it with you. for now, as usual, i'll tell you the routines that would make the "minimal set" for finding the unique errors in this book. *** there is a great assortment of cleaning routines that i do which are ones where i manually approve whether a change will be made or not. a very good example of this is the search for "11" (eleven). the replacement-term will be "ll" (lowercase letter "l", twice). here are the lines this search locates in the mabie book: > Chapter 11. > 112 > 8 113 > 114 > 117 > 119 > "They *11 take me there some day," > 211 as i expect whenever i run this search, there are a number of "false-alarms" that turn up, where the number 11 is correct... nonetheless, this is still a very productive search, because it helps to locate some problems. first of all, that "chapter 11" isn't actually chapter 11, it's chapter 2, in roman numerals, so that's an o.c.r. error that needs to be two uppercase "i" letters. next, the "8 113" line should really just be "113", because it's the pagenumber for page 113. but the "8" _was_ on the page, yes indeed, as the signature-number, so this wasn't "an error". but since we don't need the signature-number, we'll delete it. (books are usually printed as a collection of 16-page sections, which are called "signatures"... if you look closely at the spine of a book, along the top edge, you'll see these signatures and how they were sewn or glued together. back in the olden days, printers numbered the signatures, so as to keep 'em straight.) next, in the "take me there" line, there _is_ a misrecognition of "11" (eleven) for "ll" (two lowercase "l"), which is _exactly_ what we were seeking with this search. but there is _also_ an error with the apostrophe plus a floating-space before it, so this line needs hand-editing, not just a search-and-replace... finally, there are some "dogs that did not bark" in this search. where is the pagenumber for page 11? it shoulda turned up. likewise with the pagenumbers for 111, 115, 116, and 118... the fact that they are missing means we should go look there. so this is an instance of a search yielding a rather high payoff, both in the lines that it found (3 of the 8 of them had errors) and also (once you grasp it) in the lines that it did _not_ find, since it indirectly pointed to 5 other errors in the book's text. *** another neat example is the _mid-line_ floating-doublequote. take a look at one of the sets of lines from our last lesson -- where we fixed floating-doublequotes at the start of the line. we see that most also have mid-line floating-doublequotes: > "thrift of time," which brings ripe- > "Chronicles " and North's trans- > "Still studying Dante ? " said the > "Divine Comedy " or in Goethe's > "Faust " for the first time discovers > "Master and Man," is one of those > "Odyssey " are of more importance > "In any museum," says Mr. La > "Anna Karenina " leaves no reader if we do a search for this space-doublequote-space combo, there will be two very-common replacement-terms, namely _either_ space-doublequote _or_ doublequote-space. your first instinct might be just to find the next occurrence, and then edit it by hand. but that's not the right approach... if at all possible, you want to resist the urge to hand-edit. it's much smarter, faster, and easier to search-and-replace. so, in this case, you do the search-and-replace _two_times_. first, you specify doublequote-space as the replacement-term, and you approve all cases where that is the correct replacement (this is the right fix for the lines given above), and skip the rest. then the second time, you specify space-doublequote instead, and approve all cases where _that_ is the correct replacement. (since you already fixed the other cases, it should be all cases, but there might be some exceptions, so you still need to look.) some people here will undoubtedly recall that i have a program that can correctly fix such floating-doublequotes automatically. but, as i said above, i'm trying to work only in a word-processor. since there were only 59 cases of these floating-doublequotes, the task was manageable enough that i could handle it manually. *** a lot of times, people ask me to provide a "full set of searches"; they want to have a list that can find every error in every book... you _can_ get something that comes close. but you pay a price... almost every search-routine will have some "false-alarms"... (remember that we had a couple when we searched for "11"?) if a routine finds enough errors, the false-alarms are "worth it". but still, it takes time and energy to process those false-alarms. so you only want to run the routines that will locate errors... i know what you're thinking: "i'm running the routines so i can _find_ the errors, because i don't know what errors are there, so how do i know which routines will be the ones to find errors?" it _does_ seem to be a bit of a knot, but the escape is simple: if you listen to the text, you will hear it telling you the errors. let me explain... when you're correcting one error, you'll notice other errors... when you are doing your non-automatic cleaning, you need to have a certain mindset. on the one hand, you must be focused. when you start a specific change-routine, you have to finish it; don't let yourself be distracted by other errors that will pop up. on the other hand, don't put blinders on yourself. you must be open and attentive to the text. when you're correcting an error, be mindful of other text around the error, and any errors in it... whether you correct the other errors -- or leave 'em for later -- is immaterial. what is important is that (1) you retain your focus on the process of continuing what you had been doing _and_that_ (2) you make a note of the nature of the other error you spotted, so that you can later do a specific search for that _type_ of error. because if an error exists, it's likely that other such errors exist. and if you "listen" to the text, it'll tell you about all of its errors... another thing that people have asked is about any "order" they should perform the search-routines in. it doesn't matter much. because each search-routine will lead to other search-routines, if you're listening to the text. so eventually you'll find all errors. in this regard, however, i will say that a simple _spellcheck_ is quite likely to find the greatest number of errors most quickly. one thing about spellcheck is that it isn't much of a challenge... i like to clean up text because it's something akin to a _puzzle_, and i enjoy the game. a brain-dead spellcheck ruins all my fun... but if you just want to speed up the job, do a spellcheck. be aware, though, that spellcheck will _not_ find everything, so just because you spellchecked doesn't mean you're "done". there are lots of digitization errors that can survive spellcheck, and i'll discuss one of them right after this. before concluding with spellcheck, though, i should tell you that the secret of spellcheck is to add all the "correct" words to your dictionary, so subsequent occurrences are not flagged. this means proper names, of course, but it also means things like archaic spellings (if you're retaining them in your output), and slang, and dialect contractions, and all of that other crap. this means that it's best if your spellcheck program can use a _customized_dictionary_. (because you don't wanna be adding archaic spellings to the dictionary you use for other documents.) but once you've added all the "unique" words to your dictionary, spellcheck is a breeze, because -- unless you screw stuff up -- spellcheck will run on the whole document without a single stop. which means you can do it often, to ensure you didn't screw up. but remember -- a clean spellcheck doesn't mean a clean text. *** there was an interesting situation in this book. if you want to follow along, you can pull up the o.c.r. file: > http://ia700300.us.archive.org/1/items/booksculture00mabiuoft/booksculture0… (that u.r.l. will probably be split, so you'll need to rejoin it.) i'll draw your attention to page 9, where we find these lines: > thoughts, acts, dispositions, and pas- > sions of humanity. There is no > getting to the bottom of Shake- > speare, for Instance, or to the end of > his possibilities of enriching and In- > teresting us, because he deals habit- > ually with that primary substance > of human life which remains sub- > stantially unchanged through all the > mutations of racial, national, and > personal condition, and which Is al- > ways, and for all men, the object of > supreme interest. Time, which is > the relentless enemy of all that Is > partial and provisional. Is the friend > of Shakespeare, because It continu- > ally brings to the student of his more specifically, look at these lines: > his possibilities of enriching and In- ... > the relentless enemy of all that Is > partial and provisional. Is the friend > of Shakespeare, because It continu- you can see that we have a problem with the o.c.r. program turning small "i" words into capitalized ones. this is a glitch i've seen before, and i've even discussed it on this list before. it's easy to understand how the dot on the lowercase-i could mislead the o.c.r. that it is the solid line of an uppercase-i... (and, of course, just to follow up on the point made above, spellcheck doesn't flag this as an error -- its spelled write!) so we'll need to fix these glitches. specifically, we need to change lowercase-space-capital-i to lowercase-space-lowercase-i, replacing the mistaken capital. in most books, this replacement won't introduce many errors, if any at all, so i have auto-changed this before (albeit rarely). so, since it was a common mistake in the o.c.r. for this book, with a count of 69 such instances, should we auto-change it? i've appended the list of instances. take a look at it, and see whether this change would be good to do as an auto-change, or whether we'd need to approve each as a monitored change. go ahead and make your assessment now, i'll wait... so, did you think it would work as an auto-change? starting at the top, we see auto-correction would apparently work just fine, including for all those in the upper two-thirds. so you might've got tired of looking, and said "auto-change". however, toward the bottom, we see we may have a problem. and yes, a glance at one of the scans tells us we do indeed... specifically, this author has given "honorific capitalization" to the terms "idealism", "idealist", "idealists", and "ideal"... (elsewhere in the book, he contrasts idealism with realism, so "realism", "realist", and so on, also receive capitalization. and, for the record, so does "nature"; mabie's a tree-hugger.) so, in this book, this replacement as an auto-change would have meant more than a typical number of introduced errors. and not just auto-change. sometimes even when you are "monitoring" a search-and-replace, you go on auto-pilot, and just start pressing "replace" without actually _looking_ to consciously confirm that the change is really necessary. so be aware of that, and guard against it. keep your focus, and remember to _also_ be aware of the surrounding text... *** so this capital-i thing was an interesting twist in this book. but it gets even better! take a close look at the middle line: > the relentless enemy of all that Is > partial and provisional. Is the friend > of Shakespeare, because It continu- this error would _not_ have been isolated by our find routine, because the capitalized-word "is" was preceded by a _period_. it just so happens that -- in the reality of the print-book -- that "is" word was preceded by a _comma_ -- not a period -- but the o.c.r. _also_ misrecognized that comma as a period... so my error-finding routine wouldn't have isolated this line, because it's looking for cases of lowercase-space-uppercase. we just happened to see the error in this line because we're looking at the identical error in the lines which bordered it. such are the intricacies of correcting an o.c.r. file... :+) *** in our next lesson, we'll start getting down to the nitty-gritty of finding and fixing the errors in the text, so stay tuned... -bowerbird 69 instances of lowercase-space-capital-i > speare, for Instance, or to the end of > his possibilities of enriching and In- > personal condition, and which Is al- > the relentless enemy of all that Is > of Shakespeare, because It continu- > is to Invite an intimacy with It which > is to Invite an intimacy with It which > The man who Is bent on getting > is by no means Irresponsible ; It may > is by no means Irresponsible ; It may > of giving Itself up to idle reverie ; > "Spectator" and turned them Into > The man who is thrown Into constant > association with Inferior work either > of the genius of a race or an ageo It > to enter Into the thought of the un- > Idea and feeling so deftly woven to- > gether, and follow each back to Its > man of culture shares In no small > Demeter this Is precisely what Mr. > " TT Is undeniable," says Matthew > teachers Impart the breath of life by > giving us inspiration and Impulse. > paintings and of a gallery of Italian > he receives it into himself, and In the > and he is sufficient unto himself In > 'X'HERE IS a general agreement > tion, " Home for Incurable Children." > other words, culture had wrought Its > tinct from himself. There Is no easy > because It Is a matter of growth. > because It Is a matter of growth. > and possessing them in the Imagina- > education ; but genius Is not only in- > the Immense majority of men any ef- > must not be Inferred that a man Is > must not be Inferred that a man Is > laws, habits, or Institutions. It is the > itself out In material and social rela- > a main or central movement In > prominence, Imbedded themselves > These events are described In narra- > tive form, with episodes, Incidents, > be found In the pages of Homer, > Idealism, that, like other spiritual > The true Idealist has his feet firmly > as much as Idealism from weak practi- > The essence of Idealism is the ap- > he defined the Ideal as the comple- > rational Idealism sees all these things, > that the Idealist asks is that life shall > This is true Idealism ; but it is > acterises true Idealism, and which > makes all the greater writers Idealists > of the representative Idealists of his > bits of true Idealism which has been > Idealists in the breadth of their vision > Idealism which not only discovers the > reality of the Ideal as Plato saw it is > A rational Idealism is, therefore, > of the Ideal in personal and social life > stroying a false Idealism, convention- > true Idealism which the world has yet > Arden and on Prosperous Island there > Idealism which grows out of the vision > such an Idealism, and no spiritual > great Idealists is one of the greatest > ter *'As You Like It" must give > of "As You Like It" is not only

1 0

these grapes are sweet -- lesson #03
by Bowerbird＠aol.com 06 Sep '11

06 Sep '11

the thrust of this thread, once again, is how to digitize a book. i went through all the steps, and will regurgitate them for you. i used the book "books and culture", by mabie, which you can locate by searching archive.org for "booksculture00mabiuoft". as i said in my last post, the o.c.r. was clean, which is typical, with around 97% of the body-text lines recognized correctly, at least _after_ i'd run my automatic cleaning-routines on it... people who've been here for a long time will recognize that this is a finding that i have obtained over and over and over. most of the time, you'll get errors in roughly 3% of the lines. that depends, of course, on the quality of the printed book, how much care you take in the scanning process, and so on. but another big variable is how much _cleaning_ you can do. *** the first type of cleaning is something i call "auto-cleaning". when i say "auto-cleaning", i'm talking about global changes that can be made to the text _without_any_ manual approval. ("manual approval" is where you accept/reject each change.) an example of such auto-cleaning would be a global change of all instances of a space-comma (" ,") to just a comma (","). in old books, o.c.r. often puts a space in front of punctuation. the punctuation characters were actually _printed_ with some extra space in front of 'em, at least according to the standard of modern-day typesetting, so it isn't really an o.c.r. "error"... but it's something we'll want to replace in our digitized text, so it's a global change we can make without manual approval. if you can do a lot of auto-cleaning, you can drop the errors down to the 1% range,,. if you can't, it might float up to 5%... *** one of the risks that you take with auto-cleaning is that you aren't actually looking at the changes that you're making, so _might_ change something that's right to something wrong. is it possible that auto-changes might _introduce_ an error? sure it's _possible_. but with some changes, it's so unlikely we don't need to worry about it. i certainly don't feel i must "manually" approve every change of space-comma to comma. if such an auto-change does introduce an error, i'll just trust that a reader from the public will find the error and report it. now, obviously, if the auto-cleaning fixes 998 errors but also _introduces_ 2 new errors, that's probably "worth it" anyway, as it's easier to clean the 2 new errors than the 998 old ones. but it helps to know what kind of errors you're introducing, if/when you are, so i always like to review the results of any new routine that i might be considering for auto-cleaning... *** another example of an auto-clean routine is to correct the "floating-doublequote" when it happens at the start of a line. o.c.r. often has trouble with doublequotes in older books, putting an extra space in front of them or behind them... (in real life, with modern-day conventions, a doublequote has a space _either_ in front of it _or_ behind, or _neither_ if parentheses or brackets are involved, but _never_ both.) a floating-doublequote in the middle of a line is ambiguous, since the extra space might've been introduced in front of it, or might've been introduced behind it. but at the beginning of a line, the extra space was obviously introduced behind it, so can be removed with a no-look global search-and-replace. some line-starting floating-doublequotes in the mabie o.c.r.: > " thrift of time," which brings ripe- > " Chronicles " and North's trans- > " Still studying Dante ? " said the > " Divine Comedy " or in Goethe's > " Faust " for the first time discovers > " Master and Man," is one of those > " Odyssey " are of more importance > " In any museum," says Mr. La > " Anna Karenina " leaves no reader likewise, floating-doublequotes at the _end_ of a line are also candidates for a no-look auto-cleaning search-and-replace... again, the 3 examples from the current book: > guidance ? " > they make. The student of" Faust " > No man can read "In Memoriam " as that first example shows us, question-mark characters are also often "floating", and a good candidate for auto-cleaning. (likewise with exclamation-points, semicolons, and colons...) *** auto-cleaning also brings up some "philosophical" questions about the digitization process... for instance, i close up any spacey-ellipses (". . .") so they are tight ("..."), _plus_ i globally change all 4-dot ellipses to 3-dotters (wanna fight about it?); i also merge each ellipse with the word that it follows (usually) or that it precedes (when the ellipse is at the start of the line). i always change "to-day", "to-night", and "to-morrow" so that they lose their unnecessary dash. these are "in-house styles", so i don't expect you to agree, and don't care if you disagree, so it really won't be productive to discuss this kind of issue... you don't have to do all my auto-changes if you don't want to. *** so that takes care of auto-cleaning... there are other types of cleaning, and i'll talk about them in our next lesson... -bowerbird

2 1

these grapes are sweet -- lesson #02
by Bowerbird＠aol.com 04 Sep '11

04 Sep '11

hello. i hope you're having an enjoyable holiday weekend. :+) as i said, the thrust of the thread is how to digitize a book. if anyone had suggested a book, i woulda used their choice; but since they didn't, i decided to do an old standby again... "books and culture", by hamilton mabie, is a pleasant book. it's 280 pages, with about 220k of text, so it's average size. i used a version scanned by the internet archive last november. find it by searching archive.org for "booksculture00mabiuoft". as many of you will remember, we've looked at this book a lot. in fact, it was one of the examples which i just gave benjamin. this means that i can test my efforts in this current digitization by comparing my results to a very-well-proofed criterion-file... i'll share all my in-process versions with you later in the week, but for now i can post some preliminary results. as i expected, the o.c.r. results were very good. ignoring front-matter pages, the text in the body of the book contained ~5700 lines of text. i did a few standard cleanup routines on the o.c.r. file, such as closing up spacey punctuation and converting utf8 em-dashes. after that cleaning, of the ~5700 lines, over 97% were correct... specifically, about 5,555 were correct, and about 123 were not. that, in a nutshell, is the problem with the workflow at pgdp.net. d.p. has volunteers examine every line in a file, multiple times!, rather than laser-focusing on the 3% which are "problem lines". as i have shown before, and will show once again, it's rather easy to find those "problem lines". heck, most show up in spellcheck! after the cleanup, it's a breeze to turn the file into a nice e-book. -bowerbird

1 0

Re: awfully quiet -- 2555 chances at redemption
by Bowerbird＠aol.com 03 Sep '11

03 Sep '11

i said: > how the finally situation will resolve itself... mixed got i words there up some. *** rickywong said: > I know the guys at bookglutton. They aren't alone. i'm not sure what "alone" means there... > I think the term "social reading" is > quite broad and for different purpose. ok. > I'm looking more at enhanced content. then i'm not sure you picked the best examples. because i'd say it's gonna be pretty difficult to "enhance" things like "alice in wonderland" or "art of war" or the bible. or maybe not, i dunno. at any rate, i'd love to see you take your stab at it. i hope other people here give you feedback too... -bowerbird

2 1

these grapes are sweet -- lesson #01
by Bowerbird＠aol.com 03 Sep '11

03 Sep '11

ok, well, i'll be popping in and out over this 3-day weekend, and indeed for the next couple weeks, but let me now show how i can create and maintain a productive thread on this list _all_by_myself_, if need be. of course, you're invited to join, if you'd like, and can force yourself to be _constructive_, but -- lacking that -- i'm capable of carrying the thread myself... the thrust of the thread will be how to digitize a book. i've covered this many times already on this list, of course, so there will be pointers to previous posts, here and elsewhere, seasoned with updates gained from newly-won experience... so let's review the basics first. o.c.r. is pretty good nowadays, even at google and archive.org. with some elbow-grease, the right tools, and the proper focus, a person can digitize a typical book in about an hour these days. that's not a new development, either. i documented it years ago. in september of 2006, to be exact, so we're talking 5 years back. here are the posts: > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-26,1 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-27,6 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-28,5 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-09-29,5 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-10-02,6 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-10-03,2 > http://onlinebooks.library.upenn.edu/webbin/bparchive?year=2006& post=2006-12-22,1 i did the digitization with some tools that i programmed myself, which were unavailable to the general public, but you can achieve similar results today with "standard" apps and tools i'll give to you. and i'll show you exactly how to achieve those results... so if you want to play along, go and grab a book at archive.org. (if you like, tell me which book it is, and i will provide guidance.) -bowerbird p.s. when i was googling for those posts, i came across this: > http://www.mobileread.com/forums/archive/index.php/t-15544.html unbelievably, in january of 2008, i had to argue _against_ people to tell them that the iphone would be a device people would use to read. this on a site called "mobileread.com". i'd guess they're smarter now.

1 0

Re: precisely why i called the bluff
by Bowerbird＠aol.com 01 Sep '11

01 Sep '11

jim adcock said: > This is silly word gaming. > Anything programmatic "fixes" which > can be applied without human input can be > considered effectively part of the OCR algorithm. you know, jim, there were reasons i stopped even bothering to have discussions with you, namely that your arguments lacked real weight, and that you'd constantly flop from side to side. first you say that google's o.c.r. is improving -- ("towards 'reasonable' 100% human-free o.c.r.") -- then you say that o.c.r. will never be perfect... both those statements are true, yet lightweight, because their truth-value is quite self-evident, plus their valence points in opposite directions. and besides, they are beside the point, because -- as i noted just yesterday -- i have _proven_ here a great many times that it is quite easy to improve on the results of o.c.r. programmatically, such that the text is perfect even if the o.c.r. is not. i'd say that's precisely what google has been doing. which is why their _text_ is now close to error-free, even though their ever-improving _o.c.r._ is still not -- and never will be -- perfect. this is most assuredly _not_ "silly word gaming", and if you think that it _is,_ then there's no use in even discussing it any further... if you want to define "o.c.r." as something that happens _after_ the "optical character recognition" process, fine; but don't be so disingenuous when you have done that to say that _i_ am the one who is doing "silly word gaming". and the simple fact that i need to _remind_ you exactly what "o.c.r." stands for shows how stupid your dialog is. and here's the bottom-line: it doesn't matter one whit whether google's o.c.r. improves from this point on, because the post-o.c.r. fixes it already knows how to do will be sufficient to take their text close to perfection... > After having looked at these examples > I stand by what I said earlier about > the difficulty moving from demoware to product > which "real world customers" actually accept. where, precisely, is "the difficulty"? i see none at all. > Insulting the customers is indicative of the problem, > not the solution. i assume by "customers" you mean "volunteers", but i haven't the faintest clue what you mean by "insulting". and i sincerely doubt that any "explanation" you give will make any more sense than you have made thus far. so, jim, how about this? if someone else understands what you mean, they can explain it to me. and if nobody else understands what the heck you're talking about, but they remain curious, they can ask you to explain, to them, what you mean, so that then they can -- in turn -- explain it to me... and if nobody else knows what you're talking about, and nobody _cares_ to know, either, they just shut up, and you stop talking to me, and i stop talking to you. and if benjamin has any questions or comments for me, he and i can talk to each other, without your interruption. how does that sound, jim? -bowerbird

1 0

Re: precisely why i called the bluff
by Bowerbird＠aol.com 01 Sep '11

01 Sep '11

jim said: > As one who has actually worked professionally > on a number of different recognition system > I respectfully disagree with your prediction of > the future. No OCR system will ever be "error free." i didn't say the o.c.r. would be error-free, jim... i said their _text_ would eventually be error-free. as i have proven here, exhaustively, there are _plenty_ of post-o.c.r. fixes that are programmatically applied, first and foremost among these many measures being comparison of separate digitizations of the same book. (and google sometimes has half-a-dozen digitizations.) if i had their corpus, i could create hundreds of fixes... plus i expect google has invented a good number more of these effective techniques than i could ever dream of. > "Demos" often remain "demos" forever, because > turning "demos" into "real world products > accepted by real world users" is so danged hard. well, yes, sometimes that charge has merit... and sometimes it's nothing but a cheap shot. which is it in this case? well, jim, you might have wanted to check the subject-header here. check out the rest of this reply, to benjamin... *** benjamin said: > And would your lordship stoop to provide cut the cutesy crap, ok? :+) i've done far more than enough grunt work here to prove that i don't consider myself as anything more than a grunt. no matter how loudly i yell... the only people i am superior to here are the stupid, who refuse to learn, who stick their head in the sand. (unfortunately, there are quite a few of 'em here, but the good news is most have put me in their kill-files, so they don't know i told the truth about them again.) > And would your lordship stoop to provide > the location (as in URI) of these demos to > a relative newcomer with the best of intentions? here's the standard set of things that i point to: > http://z-m-l.com/go/myant/myantp123.html > http://z-m-l.com/go/mabie/mabiep123.html > http://z-m-l.com/go/sgfhb/sgfhbp123.html these are all page 123 from respective books, nestled in the set of all pages from each book, shown with text on one side, scan on the other. here are the master files that generated the sets: > http://z-m-l.com/go/myant/myant.zml > http://z-m-l.com/go/mabie/mabie.zml > http://z-m-l.com/go/sgfhb/sgfhb.zml each of the pages in the set contains a form at the bottom where errors can be reported... go ahead and make a sample report on a page; you'll find the report is appended to that page, and also collected on a separate page with all the reports that have been made for that book. as i said, though, error-correction is quite easy. the hard part -- or so people here seem to think -- is turning the text into a respectable e-book, but i've got that base covered quite thoroughly as well, as you see if you go to the main page at z-m-l.com and follow the link for examples. text-to-html, and the next step (.mobi and .epub) follows from there... > I always thought that that project was for > independent producers producing ebooks > on their own for PG, as opposed to the > general public proposing fixes to PG volunteers. well, that whole sentence shows you're very confused, but that's to be expected in "a relative newcomer", eh? i have suggested all kinds of methodologies, ranging from independent producers to collaborative methods (e.g., d.p.) to encouraging and using feedback from the general public. but slicing up the world that way isn't particularly productive. the tools i've created can be used by independent producers, or in collaborative workflows, or after-the-fact by the public. they can exist offline or online, and the behavior is the same. the tools don't know (or care) how the humans split the tasks. if you want more demos, i've got a ton. i'm also willing to program new ones, quite specific to exactly what you want, providing you agree to host them online for people to use... same offer goes for you, jim. note that i'm calling your bluff. -bowerbird

3 2