Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

They do respect robots.txt (supposedly), but they also introduced a new user agent that nobody would yet have in their robots.txt as part of this feature[1], and looking at my server logs it's already crawled a bunch of sites.

[1] https://platform.openai.com/docs/bots/overview-of-openai-cra...



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: