Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
wilg
5 months ago
|
parent
|
context
|
favorite
| on:
OpenAI Announces SearchGPT
If I were making a search engine or AI crawler, I would simply pose as Googlebot
jsheard
5 months ago
[–]
Google actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expect the big CDNs to flag your IP address as malicious if you fail that check.
https://developers.google.com/search/docs/crawling-indexing/...
wilg
5 months ago
|
parent
[–]
You could maybe still only follow robots.txt rules for Googlebot.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: