Hacker News new | past | comments | ask | show | jobs | submit login

Please post contact information for bots wanting to be whitelisted. (Either here or in the robots.txt file itself).

For example, our ChangeDetection.com site has a couple hundred users monitoring pages on facebook. (User-Agent: ChangeDetection). We always honor robots.txt so if we are not whitelisted all these monitors will be disabled.

Edit: looks like you already have posted contact info in robots.txt




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: