Yes, this has been the traditional reason for robots.txt -- protects the bot as ...

		paulddraper 6 months ago \| parent \| context \| favorite \| on: Robots.txt is a suicide note (2011) Yes, this has been the traditional reason for robots.txt -- protects the bot as much as it does the site.