We're encouraging the use of our bulk download file instead of spidering our website (often hit hard by aggressive AI bots): BTW if anyone wants training data from Project Gutenberg to do WHATEVER, there's a one click file you can download containing ALL the cleaned utf-8 .txt files at Project Gutenberg, updated weekly. No scraping necessary! https://gutenberg.org/cache/epub/feeds/txt-files.tar.zip (9.9GB) Eric Hellman http://ebookfoundation.org <http://ebookfoundation.org/> https://bsky.app/profile/gluejar.com