Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yes, this has been the traditional reason for robots.txt -- protects the bot as much as it does the site.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: