Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
claudiulodro
on Nov 1, 2024
|
parent
|
context
|
favorite
| on:
ChatGPT Search
They do respect robots.txt (supposedly), but they also introduced a new user agent that nobody would yet have in their robots.txt as part of this feature[1], and looking at my server logs it's already crawled a bunch of sites.
[1]
https://platform.openai.com/docs/bots/overview-of-openai-cra...
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
[1] https://platform.openai.com/docs/bots/overview-of-openai-cra...