Absurd. They have been doing search for so long and couldn't build their own search index? I understand it's a huge investment, and clearly for some data sources you need to create partnerships and depend on others (e.g., youtube videos). But you would think that a search engine company would invest more in this area.
Do you really understand the scale of this (ongoing) investment? I'm not sure I'd classify this as "Absurd" - the value of DDG is not yet-another (tm) search index but the specific values they add on top. When you have limited resources I'd say they're making the right choice.
Brave rolled their own completely independent search [1] on what I assume is a relatively limited budget. It seems that regularly grabbing the data would be pretty easy. The harder part would be searching/ordering it in an efficient and meaningful way while avoiding SEO, but that seems more like a fun problem than a difficult one (if not both).
There are two things the general HN sentiment genuinely believes is impossible to build unless you have billions of dollars:
- Search Engines
- Browser Engines
I don't quite understand how we got there because neither of those things are impossible. Both are achievable with a small team and a couple years of runway. As proved by Brave and Ladybird.
As proved by Mojeek; a full web scale international search engine built from the ground up with £3 million in funding, and no dependencies. Admittedly longer than a couple of years; 5 years as a hobby project, 10 years building, and more recently GTM [0].
I declare a bias as CEO, but the determination of our founder Marc Smith deserves more recognition, so here is the story up to 2021 [1]
As I understand it, Brave shipped a white-labelled Chromium, and piggybacked their search results on Bing+Google until they had enough cached results to start using their own index.
These are both orders of magnitude easier than building a browser or a search engine from scratch (although they may require similar capital investment)
If you have a web browser that isn’t Firefox, it’s probably Chromium these days. Also irrelevant to the search product.
Regarding the search product, you’re suggesting they used to use Bing/Google but now don’t. … Does that not mean they run their own indexers? So you’re just diminishing their value today based on what they used to do?
Not as irrelevant as it might seem. They do run their own indexers now, but my understanding is that they didn't actually build the index that way. They used the Brave browser traffic to guide which pages to index, which was primarily driven by the Google/Bing search results the browser was vending at that time. it's more of an embrace-extend strategy than a build-from-scratch
The point of my previous response stands... What is the difference in how one reaches a destination, if they in fact did make it to that destination? Granted, this does read a bit like an ethical dilemma where people with different values may come to different conclusions. But if you use Brave search today, you are getting results sourced from Brave's index, and not Microsoft/Google's, so in my mind, how they got to this point is now irrelevant. It's just smarter to do it how they did instead of starting from square 1.
In my honest opinion, Brave is a bit better than a crappy browser. They have their little crypto widgets, which you can disable very quickly if you’d like. Which depending on how hot blooded you are on the subject of crypto can be a turn off. But “crappy” seems a bit on the harsh end of the spectrum, no?
Building an organic dataset with decent results is the expensive and hard part. Weather, wiki etc are the kind of things you can develop on a $20/m dedicated server.
