Hacker News new | past | comments | ask | show | jobs | submit login

If I were making a search engine or AI crawler, I would simply pose as Googlebot



Google actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expect the big CDNs to flag your IP address as malicious if you fail that check.

https://developers.google.com/search/docs/crawling-indexing/...


You could maybe still only follow robots.txt rules for Googlebot.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: