September 2011 - gutvol-d - lists.pglaf.org

Re: [gutvol-d] a new bevy of kindles
by Bowerbird＠aol.com 29 Sep '11

29 Sep '11

jim said: > The "fun" part of all this is that to be effective PG has to > be willing to meet PG users "where they want to read" > which ever-increasingly means the need for useful and > effective implementations of ePub and Mobi books, > an area where even a casual inspection of PG offerings > will display that PG continues to frequently fail. you're right, jim. p.g. does continue to frequently fail there. however, other entities -- most significantly archive.org -- fail in even worse ways, indeed often in _spectacular_ ways. which introduces us to my next post... -bowerbird

1 0

some questions
by Bowerbird＠aol.com 29 Sep '11

29 Sep '11

alex said: > I'm a collections management intern at the Archive. oh boy... do i ever have some questions for you... *** oh yeah, and because i haven't said this recently... i ignore lee passey's posts, because if i read them, i would feel an obligation to do corrective surgery, and i have learned that that is a pointless exercise. be careful about believing his misrepresentations... -bowerbird

2 1

these grapes are sweet -- lesson #19
by Bowerbird＠aol.com 29 Sep '11

29 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** ok, let's finish up with just a couple more cleaning routines... again with a search for capitalized words in mid-sentence: > http://zenmarkuplanguage.com/grapes117.py > http://zenmarkuplanguage.com/grapes117.txt this list is a little long, but it found a handful of instances my earlier routine had missed, obviously due to some bug. *** i thought i'd also do a routine on mid-line hyphenates: > http://zenmarkuplanguage.com/grapes118.py > http://zenmarkuplanguage.com/grapes118.txt that didn't turn up anything, but i'm now off to do a check on the end-of-line hyphenates, for the sake of completion. -bowerbird

1 0

Re: [gutvol-d] a new bevy of kindles
by Bowerbird＠aol.com 28 Sep '11

28 Sep '11

jim said: > Not sure exactly who is doing what > re Kindle support on archive.org, > but something new is happening there. "something" might be happening, and it might be "something" that "someone" wasn't bothering to do before now, but it most assuredly isn't anything _new_... > I downloaded a "Kindle" book > from archive.org "at random" what book? and what was the exact method by which this "random" selection was made? > and what I find is much more readable > than what I have found there in the past can you be less specific? i mean, seriously... > -- clearly some new OCR technology, archive.org uses abbyy finereader. always has. > readable, but surprisingly the OCR > doesn't bother to "heal" all the linebreaks > found within words. thank you for your "summary", but without any evidence for your points, it doesn't mean much. there are certain things that archive.org and/or amazon.com could be doing to make those books useful and beautiful and powerful and accurate, as i have been documenting in my current series... whether archive.org and/or amazon.com _is_ doing those things, or not, i don't really know -- or care... -bowerbird

1 0

dinosaurs benefit by acting like mammals (but for how long?)
by Bowerbird＠aol.com 27 Sep '11

27 Sep '11

throwing in the towel on the many benefits formerly obtained by "economies of scale", harpercollins embraces print-on-demand... > http://www.harpercollins.com/footer/release.aspx?id=959&b=&year=2011 -bowerbird

1 0

taking on the textbook gougers
by Bowerbird＠aol.com 26 Sep '11

26 Sep '11

"we're sick and tired of being gouged, and we're not gonna take it anymore". > https://torrentfreak.com/pirate-service-makes-textbook-rentals-last-forever… > http://librarypirate.me/ -bowerbird

2 1

these grapes are sweet -- lesson #17
by Bowerbird＠aol.com 26 Sep '11

26 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** i forgot to tell you i fixed that bug in the search routine. and i'm running it with the latest edited version, which -- because we're out of sequence a bit -- is grapes006.txt... > http://zenmarkuplanguage.com/grapes111.py > http://zenmarkuplanguage.com/grapes111.txt everything that's returned in the "notfound" list looks ok, which means we've finally got clean text. what we need to do now is to include the "notfound" words in the "specials" dictionary, so we can create the dictionary that will constitute _the_custom_dictionary_ for this book... *** after that, the next step is a complete visual run-through; that is, we'll go through every page of our text and check it against its pagescan. there are a couple of objects here: 1) to check that all our _formatting_ is correct throughout, such as paragraphs, blockquotes, headers, and so on, and 2) to find and insert any _styling_ that appears in the book, such as italics, bold, and so on, and 3) a final clean-up q/c. for this task, it's best to use the page-viewer: > http://zenmarkuplanguage.com/grapes203.py -bowerbird

1 0

these grapes are sweet -- lesson #16
by Bowerbird＠aol.com 25 Sep '11

25 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** another demo-only lesson, again with our page-viewer: > http://zenmarkuplanguage.com/grapes203.py there are several big advancements in this version... first, there's now a "toc" button, which shows you a list of the pages (with links) that head a section. (in z.m.l., this is a line preceded by 4 blank lines.) so you have more navigational power than before. next, when you do a search now, it not only lists the _pages_ that contain your search-term (with links), it now lists the actual _line_ that contains the term, which gives a heckuva lot more context on the hit... finally, in a development that increases your power tremendously, you can now specify a _text-file_ as your search term (it must have the ".txt" extension), and the program performs the search of the pages using every _line_ contained within that text-file... (with the restriction that each line is just one word.) say there's a file "simple.txt" containing these lines: > love > books > wonder > mystery > dry-as-dust if you were to give "simple.txt" as the search term, then the program will search for all of those terms. as you can see, such a file exists on my website: > http://zenmarkuplanguage.com/simple.txt so try entering "simple.txt" as your search-term. bingo! a list (with links) of 134 lines with at least one of the five search-terms, from pages 7 to 277. nifty. useful. powerful. that's all for this lesson... -bowerbird

1 0

these grapes are sweet -- lesson #15
by Bowerbird＠aol.com 24 Sep '11

24 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** ok, time to start in with our "final checks" on this book... you might recall that we noticed -- right off the bat -- that the o.c.r. was having a problem misrecognizing the letter "i", especially in conjunction with a lowercase "l" (where the pair were mistaken as a capital-h), and at the start of many words, where the lowercase "i" was often misrecognized as a capital... we took care of all the "li"-misrecognized-as-capital-h errors when we reviewed the spellcheck, as those words got flagged. but the improperly-cased i-words passed spellcheck just fine, so we need another check, to catch them and similar errors... this check is a relatively easy one to do -- you just look for capitalized words which are not at the start of a sentence... the problem is that this check gives too many false-alarms, because it finds _names_, not just improperly-cased words. on the other hand, you can turn that _flaw_ into an _asset_. you might remember that we _want_ to find names, because we want to add 'em to our customized spellcheck dictionary. it ends up that this routine is a _great_ way to locate names. what you do is you search for mid-sentence capitalized words. when you find one, you check it against the regular dictionary. if it's not there, you put the word on the "possible names" list. if the word is in the dictionary, you flag it as a "possible error". sometimes the error is due to an inappropriate upper-casing. other times it can also be due to some missing punctuation, or a period that was misrecognized as a comma, and so on... but most of the time in this book, it was inappropriate casing, so the refinement to our routine is an extremely welcome one. this refinement doesn't remove all of the false-alarms (or all of the "misses", for that matter, if you know signal-detection). there are a good number of personal names that _do_ appear in a dictionary, such as "smith" and "brown" and "tanner", etc. (there's a humorous example in our book here, which dotes on the many contributions from the classic epic poet "homer", a name that passes spellcheck _nicely_, praise be to babe ruth, albeit probably not in a way the greek storyteller anticipated.) likewise, there are sometimes mid-sentence initial-caps that are _not_ o.c.r. errors. (you might recall that the current book gave honorific capitalization to "idealism", "realism", and such.) but all in all, this is a very worthy routine, which finds errors in a good number, without too many misses or false-alarms... so that'll be the first part of our program today. *** the second part will check the balancing of our doublequotes. you might remember that i fixed the doublequotes manually, in the initial clean-up, since there weren't very many of them. but it's always good to have the computer go back and check your work, make sure you did it correctly. ("to err is human.") besides, there might be some that the o.c.r. just plain missed. or there might be some that the _printer_ accidentally forgot. the best check, for doublequotes, is to split on paragraphs, and check that each one starts with an open-doublequote (if it has any doublequotes), and then alternates between close-doublequotes and open-doublequotes, ending with -- preferably -- a doublequote. it is grammatically correct to not have the close-doublequote when a quoted passage continues with the next paragraph. but, for the most part, throughout the file, we're gonna expect alternation between an open-doublequote and a close-doublequote, so we can write a routine that actively tests for that specific pattern... to be exact, we will set up a print "toggle", and _output_ any text between an open-doublequote and a close-doublequote. and we'll wave a flag if we encounter two open-doublequotes in a row, or (especially) two consecutive close-doublequotes. *** the third part of this miscellaneous clean-up program today will check any ambiguous paragraphing at the top of a page. the o.c.r. at archive.org has no information about a paragraph that starts at the top of a page, but we need to know that info, so we'll check anything that looks like it involves such a thing. there are three tip-offs. one is sentence-ending punctuation at the end of a page. another is a capital letter at the top of a page. last of all, a third is a short line at the end of a page. any of those tip-offs will result in a check by us of the scan... as you can tell from those tip-offs, this check involves both text at the bottom of one page _and_ at the top of the next. for the goal of getting proper paragraphing, it's enough to view the text at the top of "the next page" -- if it's indented, you have a new paragraph, and if not, you don't -- but you also want to check the text at the bottom of "the one page", because you sometimes find the o.c.r. made an error there. (indeed, it happened once, maybe twice, in the current book.) this check will tell you the pages that you'll need to review, and you will check the pagescans to make your judgment, and then edit the text-file appropriately, if there is a need. *** the final check performed by today's program will be on the runheads. i include these in the z.m.l. version of an e-book, so i want them to be correct. we expect they'll be repetitive, so we ignore 'em if they are, print them if/when they change. *** here's the program: > http://zenmarkuplanguage.com/grapes112.py > http://zenmarkuplanguage.com/grapes112.txt the output is kinda sloppy, so i won't repeat it here. just run the program yourself if you want to see it... let's review our findings on the 4 things we checked... suffice it to say that there were a lot of places where the o.c.r. misrecognized a lowercase-i as uppercase, and a few other places where we had an inappropriate mid-sentence capped word, for one reason or another. what the program also revealed is that i'd made some errors when i manually "fixed" spacey-doublequotes... that's the glitch with "manual" fixes -- human error... just a couple of errors, yes. but errors nonetheless... third, it listed out _16_ possible page-top paragraphs, about half of which will require a reference to the scan. those on 4-5, 7-8, 45-46, 111-112, 112-113 do not, because we can tell the trigger was fired unnecessarily. a few of the rest obviously constitute a new paragraph, specifically those on 168-169, 171-172, and 251-252. and we're tipped to an o.c.r. error at the bottom of 208. finally, we see that we have a couple runheads that have misrecognitions in them, so we'll have to fix those too... *** based on this output, i did corrections to the text-file: > http://zenmarkuplanguage.com/grapes005.txt *** so here's the same program, run on the new "005" file: > http://zenmarkuplanguage.com/grapes113.py > http://zenmarkuplanguage.com/grapes113.txt as you can see from the output, i didn't catch all of the bugs when i did my edits. again with the human error. but this teaches us something that's extremely important. after you do edits -- automatic or manual, but especially the manual kind -- you need to re-do the check, to ensure that (1) you made all of the changes you intended to make, and that (2) you didn't make any other unintended changes. *** anyway, so i did the corrections _again_ to the text-file: > http://zenmarkuplanguage.com/grapes006.txt and here's the check: > http://zenmarkuplanguage.com/grapes114.py > http://zenmarkuplanguage.com/grapes114.txt ok, it finally looks clean. whew. thus ends today's lesson. -bowerbird

1 0

these grapes are sweet -- lesson #14
by Bowerbird＠aol.com 23 Sep '11

23 Sep '11

this thread explains how to digitize a book quickly and easily. search archive.org for "booksculture00mabiuoft" for the book. *** of course, if there is no lesson #13, then lesson #14 essentially is the unlucky one... so let us use this lesson to discuss a bug... i gave you the list of words that were "notfound" in our book. i know, i know, i'm giving you a lot of dense lists of stuff, so i really can't blame you if your eyes just gloss over, but _still_, i was testing to see if you actually look at this stuff or not... evidently "not", or you woulda noticed these were "notfound": > conceit > conceived > concentration > conception > conceptions > concern > concerned > concerning > concert > conclusion > conclusions well, quite obviously, those words should be in our dictionary. and -- again, obviously -- they _are_. but for _some_ reason, our program didn't find them, so we've gotta track that down... it ends up, it was caused by a rejoined end-of-line hyphenate. (not just an end-of-line hyphenate, but an end-of-page one.) the hyphenate "con-crete" occurs across pages 144-145, and "crete" on page 145 was misrecognized with an initial capital. this caused our code, as written, to misbehave just a little bit, because that mid-word capital-letter confused our sort logic, resulting in a glitch on all the "con..." words after "concrete"... i will provide a fix in the next version of the spellchecker code. *** meanwhile, i've fixing the newly-exposed errors in our text: > http://zenmarkuplanguage.com/grapes004.txt after a few more checks and corrections, it'll be good to go... *** for now though, the code is good enough that we can install it in our page-by-page viewer-program. you can see that here: > http://zenmarkuplanguage.com/grapes202.py i'm "color-coding" the words now. first of all, the "common" words are rendered in light-gray. the list of those "common words" -- 555 of 'em -- is here: > http://z-m-l.com/common555.txt it's amazing how much of the book is these 555 words... next, the british words are blue, "regular" words are gray, "special" words are green, hyphenates are light-blue, and "notfound" words are rendered in red, for best visibility... thus, all the problem words can be noticed at first glance. i've also installed a _search_ capacity... just enter the term to find, and then click the button labeled "sf" ("search for"). you'll be transported to the next page containing the term. clicking "sf" again will take you to the next occurrence, etc. you also get a list of links to other pages with that term... our "engine" for page-by-page stuff is coming right along... -bowerbird

1 0