I was regularly searching, but I was rarely indexing any of the sites. I struggl... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

l72 on March 26, 2021 | parent | context | favorite | on: Only Google is really allowed to crawl the web

I was regularly searching, but I was rarely indexing any of the sites. I struggled to even get an initial index of many of the sites, due to being blocked or being reported.

jedimastert on March 26, 2021 [–]

If you were getting blocked on an initial load, you were either hit with rate limiting or an unrecognized user agent

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact