and here's today's post... again, i'm talking about the scan-set that exists at:
those scans were subjected to o.c.r. by finereader v7, with the resultant text-file located here:
one of the things you should notice about this text-file is that the original linebreaks were retained by the o.c.r. linebreak retention is very important when it comes to comparing the text-file against the page-scans because it makes that comparison many times easier to accomplish. one of the hassles with those original linebreaks, however, is with the words that were hyphenated at the end of a line. you can't just rejoin all words that have end-line hyphens, because some of 'em might be words that are hyphenated even when they're in the middle of a line. you _could_ use a dictionary, to determine whether the word is a compound, a better way is to use the actual text of this particular book, in case the author had some idiosyncrasies. so, in order to identify such words, we need a computer routine that finds all of the words that were hyphenated in the middle of a line. it's not hard to write such a routine, and the results from it -- for the "my antonia" text-file -- are appended to this post... once you have these "truly-hyphenated" words, you can then write a routine to correctly rejoin end-of-line hyphenated words. of course, you would only do this _after_ you are done proofing. (it is _here_ that you would do a compare against a dictionary, to make sure that you were doing the rejoins "correctly" and to "flag" any exceptions, so they could be checked more closely, to ensure that they were indeed idiosyncratic to the book/author.) you probably should also retain a copy with the original linebreaks, just in case it will come in handy for someone else down the line. another option would be to code the original linebreaks so that a viewer-app could keep them if the end-reader wants them, or delete them (i.e., convert 'em to spaces) if that's what's desired. after all, isn't the best general attitude to _retain_information_ that can't be reproduced, and not throw it away, if at all possible? there has been a growing realization that page-break info should be saved. why isn't the same realization extended to linebreaks? some people will want to reproduce the original p-book as closely as possible, and linebreaks would be an important aspect of that. distributed proofreaders keeps linebreaks up until the last minute -- because they understand that linebreaks help proofing immensely -- but then they throw them away, seemingly oblivious to the fact that they also could be extremely useful to someone else later on... don't throw this valuable information away! someone might want it! *** as your bonus for today, i've uploaded the .rtf file that finereader generates automatically out of the o.c.r. process. as i remarked wednesday, _styling_ information -- which is included in the .rtf -- can be very valuable in helping ascertain the book's structure... the .rtf can be found at:
for those who prefer a .zip, it's at:
as illustration of today's lesson, however, i have uploaded an .rtf that does _not_ retain the original linebreaks. if you compare it to the .txt version -- to which it is identical -- you'll see just how difficult the comparison can be when that linebreak information is discarded, and all unnecessarily so, because it could easily be kept. *** so, was anyone able to "figure out" the heuristics that i was using in wednesday's routines to ferret out the headers in "my antonia"? speak up, folks, don't be shy, or you won't get any source-code... *** see you on monday, with part 4... -bowerbird p.s. here is the list of "truly-hyphenated" words in "my antonia": (this is the list before some unwanted one-of-a-kind exceptions have been culled fro it, such as the first two listed. a double-dash is an end-of-line hyphen that was part of an otherwise hyphenated word.) An'-ton-ee-ah An-tonia Ar-r-r-mond Good-bye Good-evening Good-morning High-School Indian-like Jake-y May-basket Preserving-time Snow-White Sunday-School Te-e-ach To-day Wash-day a-goin' a-going a-quiver absent-minded alligator-skin ashy-gray awkward-looking ball-gloves bar-room bar-tender barbed-wire bath-water bay-window be-frogged bed-ticking bee-bush black--and-white-check bluff-top boarding-house boarding-houses boot-tops bottle-tomatoes bow-legged box-couch box-elder brand-new bread-board breast-pocket bright-flowered bringing-up broad-backed broad-minded bronze-green buffalo-peas buggy-riding bull-snake bull-snakes bunk-bed butter-maker cabinet-maker cabinet-work cake-cut--ters candle-moulds car-window car-windows cared-for carpenter-shop case-hardened cattle-pond cave-house cheek-bones cheerful-looking chicken-bone chicken-house cigar-stand clap-trap clasp-knife close-clipped close-cropped close-growing cloud-hung coat-tail coffee-cake coffee-pot cologne-scented cone-flower cone-flowers cook-stove copper-red corn-knife court-plaster cow-pumpkin crab-apples cross-roads cuff-buttons curly-headed cutting-tables day-coaches deep-seated deep-seeing deep-set developing-room dining-room dog-town dog-towns door-bell door-crack down-feathers down-hearted draft-horse dragon-slayer draw-bank draw-bottom draw-head draw-side dressing-room dress--ing-room drug-store dry-goods dumb-bell dun-shaded dusty-smelling earth-owls easy-blowing easy-chair elder-hunting ever-falling every-day faint-hearted fair-skinned fancy-work far-away far-seeing farm-boy farm-work fashion-plates faun-like feather-bed feather-beds feed-bag fellow-countryman fellow-countrymen fifty-five finger-bowls finger-exercises finger-tips fire-box first-cabin flat-chested flat-topped forget-me-nots full-grown fur-worker gaming-tables garment-makers' ghost-moon gold-green gold-headed gold-washed golden-rod good-bye good-byes good-looking good-nature good-naturedly good-night goods-box grain-cars grain-sack grammar-school grape-arbor green-topped griddle-cakes ground-cherries ground-cherry grown-up guinea-hens half-buried half-ear half-fares half-grown half-mile half-past half-section half-story half-swooning half-window half-windows hand-painted hand-sheller hard-and-fast hard-worked hay-cave hay-fights head-first heart-broke heavy-odored hide--and-seek high-collared high-heeled hitch-bar hitching-bar hollow-chested home-coming hoo-hoo horse-collar horse-pond horse-sense house-cleaning husking-gloves ice-cold ice-cream ink-smeared iron-gray ironing-boards jack-rabbits kawn-tree kawntree-man ketch-on lady-teacher lard-pail large-minded level-headed liberal-minded lifting-up light-hearted lightning-flashes lining-silk living-room lodging-house lone--some-like long-winded low-branching lunch-basket merry-go-around merry-go-round merry-making mid-day mid-ocean milk--ing-pails mixing-bowl money-lender moth-like mountain-ash mournful-looking music-master natural-born natural-like nev-er never-ending newly-created non-existent nose-bleed nut-cake oil-can oil-cloth oilcloth-covered oil--cloth-covered old- old-fashioned old-maidishness one-night open-grazing orange-colored out-of-door over-considerate over-delicate owl-feather pagoda-like pale-blue pale-gold paper-hanger paper-rack passers-by pea-vines peck-measure picture-books pig-yards pillow-cases plough-handles plum-patch pocket-mirror pocket-money porch-posts post-office prairie-dog prayer-book prayer-meeting pump-handle quick-changing quick-footed rabbit-skin race-course raw-boned reading-table ready-made reap--ing-hook red-lined ripple-marks rocking-chair roll-call round-collared round-eyed round-shouldered run-over rye-field saddle-bags sand-blast school-teachers scroll-work scrub-oaks sea-coast section-lines self-asser--tiveness self-possessed self-possession self-sacrifice shaving-mug sheep-fold sheet-iron shirt-sleeves shooting-coat side-road silver-rimmed sing-song sitting-room sky-line slack-wire sleigh-bells sleigh-ride slim-waisted small-town snake-cane snake-skins snow-banks snow-blocked snow-covered snow-field snow-men snow-on-the-mountain snow-white soft-hearted stage-driver steel-gray stop-watch stopping-place store-boxes storm-win--dows straw-colored strong-minded sturdy-looking sugar-cakes sun-down sun-up sun-warmed sunflower-bordered supper-table sway-backed te-e-ach tea-kettle ten-dollar thick-set thoughtful-look--ing thrashing-machines title-page to-day to-morrow to-night train-crew tree-tops trom--bone-player trotting-horse trot--ting-buggy trout-fishing trumpet-blowing turning-lathe twenty-dollar twenty-four twenty-six un-wishful under-side up-bringing violin-teacher violin-teachf waffle-irons wagon-box wagon-seat wagon-tracks wagon-trail wall-paper war-times warm-blooded warm-hearted wash-basin wash-boiler watch-case watch-charm water-pipes water-spaniel water-tap water-tin weak-minded wedding-ring week-day well-behaved well-dressed well-fed well-grown well-made well-planted well-prepared well-set-up well-to-do west-bound wheel-rim wheel-ruts whinny-laugh white-handed white--and-gold wild-eyed wild-looking win-ter windmill-frame window-blinds window-shade wine-colored wine-stains wo-man wolf-cry women-folk work-basket work-horse work-horses work-room work-table work-team work-teams working-man working-slippers yard-measure yoke-mate