Does Ahrefs not respect robots.txt? Their website says they do, https://ahrefs.c... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		devicenull on Oct 18, 2013 \| parent \| context \| favorite \| on: Who exactly is crawling my site? Does Ahrefs not respect robots.txt? Their website says they do, https://ahrefs.com/robot/index.php

bediger4000 on Oct 18, 2013 [–]

I don't know if I tried that. Originally, I wanted to let them know they were forbidden, but now I just want to jerk them around.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact