I don't mind their desire to slurp up all the things... my bigger/biggest issue is probably the reproduction/distribution by making what they slurp generally available (when they bypass said robots.txt as an example).

For expired works, or works that are no longer published, I'm pretty morally flexible here and lean towards making it available.