- gutvol-d - lists.pglaf.org

re: [gutvol-d] header detection revisited
by Bowerbird＠aol.com 23 Jun '05

23 Jun '05

lee said: > The question is a bit ambiguous. only if you haven't been following the drama for the last year-and-a-half. welcome to this listserve, lee. have you dropped the handle for good now? it's been some time since we chatted, especially frontchannel... > What are you trying to detect headers _from_? > AFAICT, Gutenberg e-texts don't have big and don't have bold, > so neither can be the hallmark of a header in Gutentexts. that's right. so for that i need to call on some of the other items in my 30-item checklist. the very best way to detect headers in a p.g. e-text is to test for blank-lines above the line in question. three blank lines will grab almost all of the headers, as well as a dose of false-alarms. the job then is to toss the false-alarms, and to do the best job possible of discerning the missed headers. and actually, in perhaps 25%-30% of project gutenberg's e-texts, pulling lines that start with "chapter" will net most headers. :+) > Presumably, therefore, you are trying to detect headers in some > marked-up text that uses some sort of presentational markup. "markup" doesn't usually enter into the equation. it can, of course, but if something has been marked up, a good way to find the headers is to examine the markup. nonetheless, i _can_ use my system on the _presentation_ of text that has been marked-up; many of my examples will be just that. as such, it can be used in cases where the mark-up is not available, for one reason (print) or another (.pdf), but its presentation is. but of more direct concern to this listserve, however, is its application toward the task that many people here do, which for the most part is to digitize text from scans of paper-books. a routine that recognizes headers in o.c.r. output -- because they are relatively big and/or set in bold -- saves the digitizer from that chore. i haven't discussed the importance revolving around header-recognition, so that might not seem like a big deal. but it is indeed rather important. (any e-book programmer, like yourself, lee, knows why it's important.) and, getting back again to the existing e-texts -- some 16,000+ now -- a routine for determining the headers in them would be quite valuable... if you're looking for a general overview, i focus on 3 distinct arenas: 1. strict z.m.l., where header-structure is defined by certain rules. 2. "fuzzy" mode, where texts are somewhat consistent, but not always. 3. "wild" texts, where all bets are off and you do the best that you can. project gutenberg's e-texts generally fall in the second category. as the examples i give will show, it would be relatively easy for me to make software that inputs text from the second category and then modifies it and outputs a file conforming to the strict first category. but nobody from project gutenberg took me up on my offer to do that... i've done enough work on arena #3 to know that it will be possible, although you can't expect perfect output from the tool on a wild text. i largely abandoned arena #2 when project gutenberg people passed, although there will be wide-ranging applicability of this arena on texts with some kind of regularity in them, such as listserve digests. but my main focus now is on spreading the gospel of arena #1 -- z.m.l. in z.m.l., headers are indicated simply by having blank lines above them. (and the more blank lines, the higher the priority-level of the heading, so it's a cinch to handle even the most complex of heading-structures.) this simplicity means that it's easy to write fast code to find headers in a z.m.l. file, and it's simple for users to understand how to make 'em. there is still a big explosion of self-publishing that will be happening, and i want to spare all those new writers the pains of doing mark-up. i'd much rather have them concentrating on their _content_ instead! once i've got all the tools in place to do what i want with arena #1, i'll return to arena #3. being able to take text "from the wild" and ascertain its underlying structure, and then output it in strict z.m.l., so it can be handled with my tools, will be an awesome achievement. again, this is an arena where markup is impractical, perhaps impossible. consider all the content that is being generated _every_single_day_ on yahoogroups. nobody's going to mark-up all that content, so we need to have a way of pulling it into our e-books and have it be nicely formatted. > Given your assumption that headers are > 1. conspicuous, > 2. hard to miss, and > 3. easy to find > (all variations on a theme) thanks for noticing the theme... ;+) but it's not really an _assumption_. (nice try to spin it that way, though.) it's actually an _observation_ on the very _nature_ of _being_ a _header_, one of those things that seems totally obvious once realized and verbalized. and of course, once you have realized that headers are _hard-to-miss_, it becomes very silly to maintain that it is "impossible" to detect them. of course you can detect them -- because they stick out like sore thumbs! > it seems to me that the best way to detect a header is to > determine the general characteristics of the majority of > all paragraphs in a document (size, indentation, amount of > punctuation, location of punctuation, capitalization, etc.) and > identify as headers any "paragraphs" which fall way outside the mean. now you're thinking. looks like you're on your way to replicating my 30-item checklist. > I presume you have a reliable way to identify paragraphs > (not always possible when using text derived from PDF files). well, yes. and the fact that text copied out of a .pdf loses its blank lines -- which then makes paragraph-detection exceedingly more difficult, -- does indeed make the detection of headers more difficult as well. which means you have to solve the paragraph-detection problem first, as best as you can, anyway, with text that you've copied out of a .pdf. restoring the paragraphs is a much bigger task than detecting headers. if you can't perform that hard task for end-users, why do the easy one? but the solution isn't as hard as you might think, although it's not 100%. when i'm done discussing headers, if you want to discuss this, we can... and besides, dealing with text copied out of a .pdf is not a high priority. the best way to deal with _that_ kind of text is to go to the producer and say, "can i instead have the file that you used to produce the .pdf?" but even without having solving this .pdf paragraph-detection problem, -- i.e., with all blank-lines removed -- my checklist does pretty well... > Consider the shortest verse of the Bible: "Jesus wept." > Biblical verses are merely numbered paragraphs. > Can your algorithm determined that it is a paragraph and not a header? um yeah. "headers" in the bible are "paragraphs" that are not numbered. and -- as you yourself just pointed out -- the actual verses are. voila. > This is the problem of the false positive: > it is as important to identify not-headers as it is to identify headers. yes it is. and much of the 30-item checklist is attuned to that issue. once you've accepted that this is part of the job, it's not all that hard. > You would be much more likely to > increase your list of special cases > if you would share the thirty-odd > special cases you have already identified. i haven't identified "thirty-odd special cases". i've abstracted 30 rules that act in combination to answer the question at hand -- is this a header? and it wasn't that hard. you can probably come up with 10-15 right off the top of your head, without even thinking too much. and if you subjected those to empirical testing on lots of e-texts, as i have over the course of the last 2-3 years, you would probably discover the rest of my 30 items. and then you too would be saying, "it's not impossible, folks, and in fact, it's not even all that difficult." there's no magic here. just hard work... -bowerbird

1 0

Volunteers in New Jersey area?
by N Wolcott 23 Jun '05

23 Jun '05

Are there any PG volunteers near Rutgers University in New Jersey. Need some one wo scan a few pages of microfilm to disc or email, there is no charge for this apparently. N Wolcott nwolcott2(a)post.harvard.edu

1 0

Re: [gutvol-d] header detection revisited
by Lee Passey 22 Jun '05

22 Jun '05

>it's been one month since my post about "detecting headers", >in response to jon noring's "challenge" in that specific regard. > >in case you've forgotten, here's a quick recap: > > big and bold. that's what headers look like. > conspicuous. real hard to miss. easy to find. > >as i said last month, i have developed a 30-item checklist. >that's how many ways a header can make itself conspicuous. >but the main way -- by far -- is simply to be big and/or bold. > >so it's time now for part 2. > >but first, any questions? don't be shy, step right up, because >headers are the first step toward detecting all types of things. >(which is why we need to discuss them in some more detail.) > >-bowerbird > > The question is a bit ambiguous. What are you trying to detect headers _from_? AFAICT, Gutenberg e-texts don't have big and don't have bold, so neither can be the hallmark of a header in Gutentexts. Presumably, therefore, you are trying to detect headers in some marked-up text that uses some sort of presentational markup. Given your assumption that headers are 1. conspicuous, 2. hard to miss, and 3. easy to find (all variations on a theme), it seems to me that the best way to detect a header is to determine the general characteristics of the majority of all paragraphs in a document (size, indentation, amount of punctuation, location of punctuation, capitalization, etc.) and identify as headers any "paragraphs" which fall way outside the mean. I presume you have a reliable way to identify paragraphs (not always possible when using text derived from PDF files). Consider the shortest verse of the Bible: "Jesus wept." Biblical verses are merely numbered paragraphs. Can your algorithm determined that it is a paragraph and not a header? This is the problem of the false positive: it is as important to identify not-headers as it is to identify headers. You would be much more likely to increase your list of special cases if you would share the thirty-odd special cases you have already identified.

1 0

re: [gutvol-d] header detection revisited
by Bowerbird＠aol.com 21 Jun '05

21 Jun '05

as i step you through example after example after example -- all of 'em pre-existing, including many from the library itself, and handled with solid honest-to-goodness working source-code -- you'll come to realize fully how _easy_ it is to detect headers (and even header-level!), and the people who insist on telling you "it's impossible" will seem curiously illogical and out-of-touch... oh yeah, feel free to recommend any e-text from the whole library as a "test-case" that you would like me to consider in detail! :+) (to be fair, test-cases should include scans to resolve any doubts.) thank you! enjoy your first full day of summer! 90 degrees in l.a.! -bowerbird

1 0

header detection revisited
by Bowerbird＠aol.com 21 Jun '05

21 Jun '05

it's been one month since my post about "detecting headers", in response to jon noring's "challenge" in that specific regard. in case you've forgotten, here's a quick recap: big and bold. that's what headers look like. conspicuous. real hard to miss. easy to find. as i said last month, i have developed a 30-item checklist. that's how many ways a header can make itself conspicuous. but the main way -- by far -- is simply to be big and/or bold. so it's time now for part 2. but first, any questions? don't be shy, step right up, because headers are the first step toward detecting all types of things. (which is why we need to discuss them in some more detail.) -bowerbird

2 1

Baha'i Faith texts - Terms of Use acceptable for us?
by Joshua Hutchinson 21 Jun '05

21 Jun '05

The Baha'i Faith makes available quite a bit of material in eBook form available on its website. Further, the Terms of Use (http://reference.bahai.org/en/terms.html) seem to make it perfectly acceptable to further distribute this work as long as the copyright and attribution is intact and it is for non-commercial use. Does anyone see a problem with "raiding" their material for inclusion in PG? I realize the texts will need to be reformatted to our standards and formats, but I can do that. Plus, as a side-benefit, they have many of the texts in Persian and Arabic as well as English, so we would be getting multiple languages represented in one swoop. Josh PS FYI, I am a Baha'i, but I don't speak in any official capacity. I'm going by the Terms of Use linked above.

1 0

Re: [gutvol-d] Derivative works, or, what is copyrightable?
by shimmin＠uiuc.edu 20 Jun '05

20 Jun '05

As another poster pointed out, in the U.S., Bridgeman v. Corel says that some mechanical reproductions are not "original works" for the purpose of copyrightability; indeed, the point of creating these works is to be unoriginal. Whether PGLAF wants to stand on Bridgeman in this particular case is their own decision; as always, the only sure test as to whether something is clearable is to try and clear it. That said, if it's not the illustrations you're interested in, but merely need to consult another edition to repair lacunae in the text you're dealing with, then just consult whatever editions you have easily at hand, and repair the text accordingly.

3 3

Re: Print-on-demand and dead-tree copies of Gutenberg texts.
by grendelkhan 17 Jun '05

17 Jun '05

Thanks to everyone for their comments so far! I'm learning quite a bit as I go. As of October 2003, the US Commerce department reported that about three-fifths of households had a computer; a little over half had internet access. https://www.esa.doc.gov/Reports/NationOnlineBroadband04.htm So it's not as bad as I was led to believe. Still, Perhaps I should have stated my goals a little more clearly. I have no particular interest in making money or making a business out of this. I'd simply like to make the books available---through whatever means that may be---in dead-tree form. I suppose it's a terrible idea fo tie the actual Project to a commercial entity by developing a working relationship with them---I don't think an "Official Project Gutenberg Edition" is a good idea. lulu.com, as mentioned, has no setup fees, but their pricing is a mite stiff---$4.53 plus $0.02/page. Certainly better than buying stuff from most university presses, but not exactly bargain-basement. Lightning Source charges (based on some quick googling at http://com1.runboard.com/bthescribesmessageboard.fwritingarchives.t45%7Coff… ), $0.90 plus $0.013 per page, but I don't know what kind of binding that requires, or what sort of setup fees they charge. Perhaps they'd waive them if DP put out some sort of print-ready version in addition to human-readable text. I'm thinking TeX->PDF here, as it's pretty much the stablest human-readable-yet-fully-marked-up format available. Thoughts? I suppose I should take a relatively short etext, mark it up and see how it looks. I concur that simply throwing plain text, or even decent HTML, at paper is a horrible idea. So, what I ask is---is there a way to prepare the etexts as, in addition to HTML, whatever format is print-ready for these machines? Since typesetting a ready copy is a simple matter of feeding it to a Xerox DocuTech or whatever the $100,000 piece of hardware the print shop uses is, how can we do the necessary preprocessing ourselves? What exactly does the "setup fee" include? Thanks to everyone again for being so helpful with this. --grendelkhan

4 10

re: [gutvol-d] roundtripping formatted text through a .pdf
by Bowerbird＠aol.com 17 Jun '05

17 Jun '05

jon gorman said: > I think bowerbird just claimed the pdf that he exported > was easy to import back as pdf (via copying out the text), > not necessarily that he converted an existing pdf file. > Of course, I'm probably wrong about that. > Without capitols I sometimes get lost ;) i understand... :+) and what you've said is pretty-much correct. yes, i'm _only_ talking about .pdfs that _my_ viewer-app creates. (if some other program created the .pdf, then blame that program.) so, you put plain-text into my viewer, and it formats it nicely. you can print that nice formatting to a .pdf (which looks nice). and then you can copy the text out of the .pdf. when you do that, much of the nice formatting has been stripped away, of course, and we're back to plain-text again. (if i remember correctly, .pdf _does_ retain italicizing, but it _doesn't_ retain bolding. i don't have the faintest idea why, it's kinda weird like that. and it definitely stores the color of the text, which is cute. but it definitely strips the _size_ of the text, which is bad. all of this is in _my_ version of acrobat reader, which is v4. we talk about acrobat/.pdf like it's one straightforward thing, but it's a crazy mish-mash of different-and-changing versions, so all of our discussion needs to be couched in careful clauses.) but the loss of formatting doesn't matter, because after you have made a few global changes (which, among other things, restore the blank lines between paragraphs that get stripped), you can put the text back into my viewer-program, and it will redo the nice formatting, just like it did it in the first place... with zen markup, this is all pretty easy to accomplish... :+) > If I had time, I'd write one by hand for ya > that had none of the encoding mess. i'd love to see that! -bowerbird

2 1

roundtripping formatted text through a .pdf
by Bowerbird＠aol.com 17 Jun '05

17 Jun '05

recently i've worked on "roundtripping" styled z.m.l. text through a .pdf. my viewer-program can write z.m.l. text to a .pdf such that copying the text out of the .pdf gives a user the same text that went in. make a few global changes -- which restores the whitespace acrobat usually strips from text -- and you can load the text back into my z.m.l. viewer-program and generate the same .pdf once again... the proof is in the pudding, and the structure is in the presentation. that is _not_ something you can do with text that's copied out of a .pdf created with other programs i know. generally, the .pdf format is known as the "roach motel" of file-formats -- content goes in and can't get out... :+) -bowerbird

7 15