Please post contact information for bots wanting to be whitelisted. (Either here or in the robots.txt file itself).
For example, our ChangeDetection.com site has a couple hundred users monitoring pages on facebook. (User-Agent: ChangeDetection). We always honor robots.txt so if we are not whitelisted all these monitors will be disabled.
Edit: looks like you already have posted contact info in robots.txt
For example, our ChangeDetection.com site has a couple hundred users monitoring pages on facebook. (User-Agent: ChangeDetection). We always honor robots.txt so if we are not whitelisted all these monitors will be disabled.
Edit: looks like you already have posted contact info in robots.txt