Actually, if you want to interpret robots.txt that way, it raises the problem of... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

_wiv7 on Nov 5, 2013 | parent | context | favorite | on: Wayback machine gets a facelift, new features

Actually, if you want to interpret robots.txt that way, it raises the problem of "how long can I consider a robots.txt valid for?"

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact