anna said:
> can I count on these markers
> appearing in each and every text?
> Or are there other delimiters and/or tags?
you can count on them appearing, except when they don't.
but if you search for lines with three (or more) asterisks,
which have the word "start" and "gutenberg" (uppercase),
you're likely to have very good results across the corpus.
however, you will then want to screen the first paragraph
after that, for words like "distributed" and/or "produced"
(and a handful of others that i don't remember right now),
so as to get rid of the production-credit for the book.
there are also some huge (literally) gotchas that you will
_not_ want to download, including the human genome files
and an innocuous e-text (whose number i have repressed)
that includes the entire library up to that point. (i dunno
whose idea that was, but i wish somebody else would have
been smart enough to shoot it down as a very stupid idea.)
-bowerbird
