Hacker News new | past | comments | ask | show | jobs | submit login

I would like to target the Brave search crawler/bot, but they’re hiding themselves like every other spambot: https://twitter.com/tinusg/status/1453862793933897729?s=21



This says that the index is created from users' own web browsing, not from a bot.


Wait brave indexes what you browse? How has there not been some bug yet? Can't imagine that going well...


It all seems quite scammy. They claim it's opt-in, but as a Brave user I can't find the option to opt-in or opt-out.

They look the searches you do at Google and other search engines. The search terms and the results you click on gets sent their way, including metadata from the page.

The content itself is downloaded by their bot (more like a fetcher). That bot has the user agent of a regular browser, so you won't see it.

You also can't specifically block their fetcher. It only adheres to disallow *.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: