Those cornercases can exist on sites that don't have a robots.txt and still have... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

mburns on Nov 5, 2013 | parent | context | favorite | on: Wayback machine gets a facelift, new features

Those cornercases can exist on sites that don't have a robots.txt and still have to be crawled correctly.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact