I've used DDG for the past ~5 years, and it is typically worse without using a hashbang like !so for technical queries. I guess that is what the web has evolved to-- knowing which mega-site you want to search against rather than discover new sites?

Also, I am by now 100% sure that Google has just stopped indexing the long tail. Like if I search for function names of public source code that I downloaded from GitHub, Google won't find it. But of course, it's still on GitHub.

Similarly, Google will sometimes not find a single result for some Windows API function names, despite them being publicly documented on docs.microsoft.com.

Then searching for "odicht" out of curiousity, they auto-correct it to "olight". So I start with almond-based sugar sweets, follow their auto-correct twice and now I'm staring at headlamps. And even Google has no idea what "odicht" might have been, so I really wonder how Amazon decided to auto-correct from an existing product into a non-word.

Searching for "odense marzipan" including the quotes then works, but it yields the cringe-worthy message:

Your search ""odense marzipan"" was automatically translated into "„odense marzipan“".

(where the only difference between the first and the second thing is that they converted the ascii quotes to up and down sentence quotes)

A while back I spent a lot of time looking around for a basic SATA bluray drive on Amazon before finally giving up. All I could find was burners for several hundred dollars when I really just needed a drive to quickly rip a single disc I'd bought. I spent probably an hour scrolling through results and trying all sorts of variations on search strings.

I eventually gave up and punched it in Google to get... kicked back to Amazon to a simple, cheap SATA bluray drive that had been there all along.

This is exactly it. Amazon isn't a shopping site, they're a corporation using third party sellers to offload the risk and cost of providing a wide array of goods. They let customers experiment with product offerings, find products that sell using their web site, then cherry pick the most lucrative ones to produce and stock to compete with their own "customers".

They're not a service for anyone but themselves.

What I don't get about this companies (because this seems a problem shared between google, YT and Amazon) is when they optimize for clicks or whatever their KPI, what are they thinking is the outcome in the mid term?

I mean, IDK around you, but I'm the prosumer regular folks ask for recommendations. They may be safe for the time being but of course I'm going to contribute for their competitors getting klout.

I don't have an alternative for YT, but people watches me using DDG and whe they ask for recommendations for buying stuff I don't even bother with Amazon.

I'm not the type who pushes his decisions onto others, but I already got asked why I don't use google and amazon.

I just tried using Amazon.com to order something to Hong Kong for Christmas - sweet Jesus I don't know why anyone would ever visit that site more than once. I ended up giving up and just ordering the things with the co.uk version and sending them myself.

They extra postage cost is nothing compared to the insanity of the .com site.

only to have it disappear somewhere to page 2 or 3 (if even!)

There's got to be a reason no-one (to my knowledge) has done this: They probably forbid affiliates from doing their own indexing and ranking. Does anyone know for sure?

They do. I came across a reddit thread where a guy had built a simple php-based search indexer for Amazon and managed to pay for his college through it. After this incident they apparently put changed their developer terms of service!

Just try searching for a high-end appliance and watch all the Chinese knockoffs that will get ranked higher than the actual thing you're looking to buy. Sure, they cost less, therefore you're more likely to buy something therefore such changes tend to win web labs and go into production. But they've completely destroyed the intent of the search doing so.

Now shopping on Amazon feels like shopping on eBay or Aliexpress with fast shipping. Everything name-brand could be counterfeit, and everything no-name is a complete gamble.

There is an actual product (https://www.amazon.com/gp/product/B084GKQB9T) with all of my keywords in the title. I found it via a google search because I gave up scrolling through Amazon.

I actively avoid shopping Amazon where possible these days, just because it's such a trashy experience.

I very much get the feeling that amazon and eBay "Don't like" Denmark – the former redirects amazon.dk to amazon.de saying "we deliver!" and the latter doesn't exist (but it does wholly-own a local alternative). Going to Denmark always makes me aware of how good the local Danish things are, and how inflexible and annoying the "global" options usually are by comparison. Riding roughshod over the language is a good example of this.

(And that leaves aside the issue of the product page of a really nice wastebasket I bought years ago being hijacked by a meat slicer.)

The product is there. It works adding the company name.

Rather weird if true, but I can't really disagree with your observation. It seems like large parts of the web have disappeared in the last five to ten years.

Google do most likely index the sites, but their current algorithm just don't use them, because it as much a promotion algorithm at it is search.

Search for "github drv8830pico" - it's literally not in any results.

Ask HN: Has Google search become quantitatively worse? https://news.ycombinator.com › item 4 hours ago — But of course, it's still on GitHub. ... Search for "github drv8830pico" - it's literally not in any results.

DuckDuckGo works fine, though.

Maybe there's no links to it from other places, or somewhere on github thats crawled / indexed frequently enough.

Considering this comment thread is seemingly quickly indexed, just posting this link might be enough!

GitHub does fit the "huge website with lots of duplicate content" description very well.

Then again, Twitter seems to be doing fine. But I have a suspicion that they have some sort of agreement (and GitHub doesn't), because I have to regularly "report as spam" (note how that implies Google decides what's spam and what isn't) Twitter emails and they still pop up in the inbox after a while.

It might be anti-spam, but how is it this shit?

Man, ads really are the devil.

Google News too has become flaky. Often does not find stuff you know is there, or finds it one day, but not another. Hrmph.

There was a time I thought I'd trained it to stop telling me about what celebrity said or wore what, but those are creeping back into my feed.

I absolutely hate the Google has changed the semantics of search so that it doesn't really pay attention to your keywords unless you quote them. In Google's mind you're not serious about a keyword until you quote it! Garbage!

What does the business model look like? Ads (it would be in front of a very valuable audience of technical folks)? Or paid subscriptions (perhaps the community votes which resources get crawled / indexed)

Most of my searches are for really old pages or really long tail stuff and Google just simply doesn't bubble them up, if it has them at all. I keep finding web sites lately from links on other sites and find myself asking "Why the fuck did Google not find this?" .. then I go back to Google and try to find it with keywords from the site, and nothing...

This filter takes care of that box completely:

  www.google.com##.exp-outline
  www.google.com##[style="display: block; opacity: 1;"]
  www.google.com##[data-hveid]>div:style(height: auto !important)

This happens _all_ the time on the Twitter app search bar.

I'm sure I'm not the first one to think of it, so there must be some reason why it isn't done - does anyone know why?

The browser should do it. Websites often do this intentionally in order to trick users into clicking ads. The only way to stop these abuses is for the browser itself to stop enabling them.

You're likely not the first, but it's an excellent suggestion.

Gaming is exactly one of the problems and all those sites that lie on spectrum between news to gaming and utilize some aspect of dynamic presentation.

theverge.com presents a “show comments” button that shifts out of the way to reveal the worthless taboola links, I honestly don’t know how it isn’t considered click fraud

Hit the nail on the head there my friend. Super annoying.

I wonder why their search is so slow though. Any ideas?

But it isn't MUCH better. DDG is just slightly better than Google, that has become infuriating.

And somewhere a team of designers and PMs got their bonus for increasing the engagement OKR. Clearly users love the animation and added delay because look at the metrics skyrocket!

> Results for exact term [...]. If no results are found, we'll try to show related results.

But very regularly it fails to find results I know exist in not so unpopular places.

[0] https://help.duckduckgo.com/duckduckgo-help-pages/results/sy...

This really got to me about 6 months ago, so I changed all my default searches on all my browsers and mobile to DDG, and haven't looked back. I tried DDG ~5 years ago, and there was no way it could have replaced google for me then, but when I did it 6 months ago, it didn't seem any worse, maybe a little better.

You'd think from the responses herein that using quotes was a panacea.

It isn't. I see what you see. I think it's ignoring the quotes.

DDG is heading this wrong direction too. Today I've searched for some ecommerce platform called Comerzzia and it showed me some Comerzia or Comercia or whatever shops near me. It shows maps if it thinks they're related and apparently I can't disable that feature.

The fundamental, unavoidable problem is that the cost of providing high-quality results on the long-tail of possible searches tends to grow faster than the revenues that can be earned from those increasingly rare, obscure, long-tail searches. Any search service seeking to maximize profit, like Google or DDG, ultimately always evolves to perform less and less well on the long tail of possible searches.

The search service we all wish we could have -- a service seeking to maximize the quality of individual searches, no matter how obscure -- may not be feasible as a profit-maximizing business.

I think even two years ago, Google searches had far more depth and yet Google was quite profitable (then the searches were still biased but now stuff is simply gone). Sure, if someone looked at the marginal profitability of every single search result, it would look like what we're seeing. But there was a time when good indexing of stuff that didn't turn a profit by itself was done as a service to attract people to Google and/or to improve the Internet generally. That time has passed, clearly but it was a decision.

[a] I'm paraphrasing Ernest Hemingway: https://www.goodreads.com/quotes/102579-how-did-you-go-bankr...

What are the ways to direct more air into the likes of you.com? You.com was an “Show HN” topic 3 weeks ago[1]. The you.com improvement in 3 weeks is noticeable.

This post has been simplified for sentiment parsers...

[1] https://news.ycombinator.com/item?id=29165601

i have a directory of duckduckgo !bang operators, for easy access https://mosermichael.github.io/duckduckbang/html/main.html

This helps to find the specialised search engine, if you need one. (it scans duckduckgo for any new !bang operators, in a nightly built, here is the script: https://github.com/mosermichael/duckduckbang )

I suspect, that most of these specialised search engines are powered by elasticsearch. It may be, that elastic is starting to cut into google search, from the low end.

Try searching for any product outside your wheelhouse and it quickly devolves into an undergrad research endeavor.

I can't trust the first or second page results because of SEO. Then every page after quickly veers off topic or just features sites that aren't as good at SEO.

Google is simply maximizing profits by giving users results that would cause either more clicks on ads or show more ads. It's mission is to make money this quarter/year. If you believe any of their Silicon Valley-style new age talking points you probably don't have critical thinking skills.

If their products are getting worse for you perhaps you are not part of a profitable segment for them.

I remember once searching for how common same-sex relationships among teenagers are in Japan, as some say it is very common, and all I received were political opinion pieces that did not in any way come with the numbers I sought on Google, so I then tried DuckDuckGo and to my amusement what I received with the same query was mostly pornography.

Neither particularly useful, but the contrast in how both prioritize was interesting to me.

Maybe it's just me but those always seem vastly gratuitous. Like shouldn't the engine figure that out automatically? It's like half its job.

Does doing the hashbangs actually throw you directly onto the site? I'd assumed it just restricted results to that site.

It’s almost as if the government and big companies have spent a lot of effort understanding human biology, cognitive function and applying what was learned.

While selling the masses on a contrived story keeps them believing there’s a universe of infinite life available to humanity if you just follow these steps…

Things like human colonization of space, and political memes about wealth are taking advantage of the same biological quirks as religion. It’s just now we can quantify the effects rather than wave it off as mysticism.

But the human story is already set on a path of building a bridge to nowhere.

You can sort of fight it by including a term showing your topic in your search, I think.

Based on the search box suggestions I get, it seems many people work around this by appending reddit to their searches. If I search for "warmest winter coat" it's a bunch of untrustworthy content marketing until you try something like "warmest winter coat reddit"

Unfortunately I prefer to avoid reddit (which also has a fair amount of astroturfing), but I haven't found a good alternative. I severely miss Google's old "discussions" (or was it forums?) filter.

For some reason though (probably because they used AMP) they basically allow them to do anything they want. Multiple popups, hijacking click events for login modals, and hiding the content with no impact to search results.

So now, in all the glory of the Internet, the person who genuinely wrote the best blog post on the "warmest winter coat" is completely unfindable on normal Google search or you force yourself via a reddit query to a completely hostile user experience unless you login.

It would be cool to just be able to do something like this "warmest winter coat --hobbyist-only".

I like to compare with search.marginalia.nu results from time to time, but the restrictions it puts on the content it traverses do not make for a good daily driver.

Just also poking fun of how Google both killed the "real" blog industry and forced all the good content on reddit (which ironically does every anti-SEO thing ever and where people are trying to go).

warmest winter coat --hobbyist-only

Top 10: 1. Canada Goose Parka 2. Patagonia Down Jacket 3. Marmot Precip Jacket 4. Columbia Winter Jacket 5. North Face Thermoball Jacket 6. The North Face Nuptse Jacket 7. Rab Neutrino Endurance Jacket 8. Mountain Hardwear Ghost Whisperer Jacket 9. Black Diamond Fineline Hoodie 10. Outdoor Research Cathode Hoody

Google should be opinionated. It should have a huge bias towards quality. It should not be hard for a small army of employees to be blackholing ANY crap product roundup site. Real product tests, where multiple items were actually purchased and compared, should always float to the top.

Just as much as needing to pay attention to what spam to suppress, they should be asking "what do we we want at the top" and whitelisting really great sources that always cut in front. Why should healthline ever appear before examine.com?

Instead, they have thrown their hands up, said the algorithm is in charge, and to interfere with it would be improper. Bollocks.

when i search for something specific, i usually include a random niche tangental hyper specific keyword about the thing i want in quotes (until it gets turned into the SEO-buzzword of the day)

"impedance" for analog electronics stuff, "ring-spun" for clothes stuff, etc

Sometimes I feel like that internet isn't for me anymore, and that's a little distressing.

We need a PBS/NPR of search engines.

PBS/NPR like solutions would be subject to co-opting forces as with other search engines. Look at wikipedia, touted as a “social good” but undermined by collective activism edits in the name of various ideologies claiming pure intentions of “social good”.

But it is still not so dire. I went back to bookmarks, reading lists and keeping note of writers I check out. It's not bad at all as long as I keep in my interest bubble. Google or not, I still would prefer today's internet world to the decades before the internet.

And of course, new search engine, something distributed and in the GNU domain.

Besides, all that can be done with img HTML tag (browsers do have 'disable images' thing already).

Ironically we’re watching this play out now with products and techs that market themselves as “decentralized.” Maybe after this phase the tech community will consider this isn’t something we can tech our way out of.

The reason Google isn't explaining it is because they can't explain it. They don't know why the ML is returning the results it's returning. And whatever it's doing in that black box, it's making gobs of money—Google's sole purpose since Ruth Porat started squeezing every dime out of every orifice–so just keep on keeping on.

https://en.m.wikipedia.org/wiki/Eternal_September

banning advertising would help for awhile, but other profit streams would be optimised and expand to fill void, such as data collection/mining

- ad revenue would drammatically decrease

- buying ads would actually be affordable by local sellers

- the market would self regulate against monopolies

If surveillance capitalism is super-profitable, "making cell phones" capitalism must be even better since Apple is bigger than those companies.

I can only presume that so many people have given up on the web as indexed by Google and are just searching for "<whatever> reddit" now as the only way to get any kind of content written by real people on a subject instead of SEOd filler "content".

Presumably it won't be long before Reddit itself is flooded with spam content to take advantage of this - I'm sure it's already happening to high value keywords.

This happens on Amazon reviews all the time as well.

What I end up doing is trying to find a post that isn't all-in on any specific solution... but lists pros and cons of multiple options, because it seems less likely that a content advertiser will post anything negative (or positive about a competitor).

In a few years Reddit will be just like the generic google searches for everything but the most heroicly moderated or obscurely small subreddits.

first result is a list from a blog by some "Emergency prep guy" it basically lists 27 coats with information.

Second result is RT online with black Friday recommendations

Third is oprah daily with recommendations and shop links

So, reddit is also, as you said, full of false information + astroturfing as well. Besides not everyone is interested in diving into reddit rabbithole to find information on warm coats.

What do we want google to do? It tries to blend whatever is available, I don't think google got worse on this particularly, but it is probably a hopeless pursuit considering the status of the web. As for forums, adding "forum" at the end sees to work, but I agree it would be nice to have the option in the toolbox.

Most of the time, this is just a list of coats someone googled and copy-pasted info from the marketing pages. This page is an affiliate-marketing site masquerading as a review site.

Not sure if specifically that page is, but that's what the majority of "product review" results in Google are nowadays.

Emergency prep guy seems like at least owns or knows about the products he lists, however he also discloses his site is owned by a company which is a participant in the Amazon Services LLC Associates Program.

So maybe Google should stop feeding this affiliate spam loop and try to create algorithms that try to actually understand which information is based on personal experiences and not created by financial interest. This is a very hard task to do and also prone to manipulation as well. In the end maybe Google needs some real competition for this to happen.

As you say, this is harder to do well than it sounds. There are definitely a lot of hobbyist types who review things and make a side (or even major) income from affiliate sales. With human judgment, it's quite easy to differentiate the honest hobbyist from the spam sites. I fear that any automated attempt to punish the spam sites would end up hurting the hobbyists more.

The study was poorly done and there were tons of comments pushing the same message: "vegetarians/vegans are annoying hipsters who will lecture you for eating meat and they'll be so deservedly upset by this."

Found it and most of those comments are deleted now (https://redd.it/qskxol). Is the meat industry losing a sizable chunk of profits to more people swearing off meat for moral reasons, or ditching meat as a financial decision?

Edit: Threw that link into a website that restores deleted comments (https://www.reveddit.com/v/science/comments/qskxol/meat_cons...).

Mods deleted all references to fact that the study was funded by a beef company. Blatant corruption?

SEO blogs are full uncanny valley for me.

It's also funny how Google basically nuked groups and made it unsearchable, while once in a blue moon you get a search result to alt.coats.winter or something

If I'm asking google or DDG for advice on a product, it's either going to be a reddit or Wirecutter for me. 99% of results on "best *" results in _*literally hundreds*_ of domains like "best*for2021.com" "buybest*.com" "top10*reviews.com" that are all generated by bots containing only the worst knockoff / counterfeit / Chinesium products and tons and tons of Amazon affiliate links.

E.g. I was trying to remember the name of a top-of-the-line soldering station brand (Metcal) I used back in college, so I kept trying permutations of "best professional soldering rework station" on google [0] and DDG [1]but it only comes up with low-end Chinese stations, a few mentions of Weller and Hakko, but no impartial reviewers, no forums or blogs, no discussions...nothing leading to Metcal.

Then I searched "best professional soldering rework station site:reddit.com" [2], I clocked the first 3 links, scrolled, and found Metcal on the second hit. [3]

I was surprised to see Wirecutter did a review [4], and arguably the Hakko FX-888D is the best soldering station ever made (and the X-Tronic is a fine budget runner-up) for *_MOST_* people, but it's still not a Metcal (the thermal capacity and regulation of their iron tips is just unparalleled even with nice Wellers and Hakkos - you can really feel the difference when working with THICC power ground planes and RF connectors).

[0] https://www.google.com/search?hl=en&q=best%20soldering%20rew...

[1] https://duckduckgo.com/?t=ffab&q=best+soldering+rework+stati...

[2] https://duckduckgo.com/?t=ffab&q=best+professional+soldering...

[3] https://www.reddit.com/r/electronics/comments/2c4hnl/best_so...

[4] https://www.nytimes.com/wirecutter/reviews/best-soldering-ir...

On the other hand, a different search for "R-values of winter coats" produces a few real gems, like https://outdoorcrunch.com/jackets/

> "R-values of winter coats"

This is a valid alternative... but I don't want to be an expert on winter coats to be able to Google basic information. I'd have to weed through a fair amount of marketing content to even find that the phrase "r-value" exists. In the past this wasn't necessary.

The internet used to be primarily a place for people to connect and share information... and now it feels like primarily a place to be advertised to. There's also the fact that many ads have evolved beyond simple billboards to psychological manipulative clickbait.

It's completely anecdotal and tangential to this topic, but I have a sneaking suspicion that the way marketing manipulates people has created unhealthy amounts of skepticism that further fuels the affinity for conspiracy theories... which tend to be so toxic that they're almost inoculated from marketing.

You can search "warmest winter coat site:reddit.com" and filter by past year, only to get a result from 7 years ago.

I really don't know how to search the web anymore.

Eg.

'good cheap mountain bike' -> 10 results out of the first 10 are commercial spam and listicles.

'thread good cheap mountain bike' -> 5 human discussions on entry-level MBs, 1 link to a MB forum home page (not a specific thread), 2 commercial spam, 1 paywalled magazine article testing MBs, and 1 online shop product page for a MB that happened to mention a "73mm Threaded BB shell" multiple times.

I don't use them enough, but the replacements are interesting: they seem to mostly search for indicators of a forum (e.g the existence of a next page button).

This always gets brought up but the problem is far deeper:

When I search for:

> "weirdly specific ab345"

and the results contains thousands of pages without "weirdly specific ab34" then the problem isn't spam sites.

It is Google not respecting my queries.

- captcha (because vpn)

- spam results (based on location, my location was never very good for technical content)

- paywalls

- no pictures cause photos is now paid (i've signed up for unlimited forever)

And on bing basically I get shopping coupons, games and, well, and microsoft's "anything's valid, except customer sat" approach.

And the entire point of the discussion is that astro-turfing has papered over everything useful on the internet in order to sell low-quality crap.

The fact it doesn't scale is exactly why I use this method. Nobody is paying people to sit in a discord chat and recommend fishing poles. I know the information I am getting is from an enthusiast, if not a professional.

What is needed is a way to archive actual useful information for lookup later.

My deepest apologies for saying this, but for any type of query that has a monetization angle, I now add "site:www.reddit.com" to the query to find actual discussion about it.

Normal Reddit disclaimers apply as much of what you find is garbage but at least if you search "best exercise bike" confined to reddit you'll get real opinion not hellbent on monetizing you.

https://www.tomsguide.com/best-picks/best-exercise-bikes https://www.cyclingweekly.com/news/best-exercise-bikes-40742... https://www.menshealth.com/fitness/g23064646/best-exercise-b...

Not sure if they are amazing results but it's decent for such a generic query.

For stuff like this you will only get spec comparison articles or comparison articles from big sites who get gear sent to them or make so much from affiliate sales that they can afford to rent/buy gear. It's simply not likely for anyone to put so much time and effort into it for a passion project.

However, where you can still find independent reviews is if you look up a specific item. Granted, they may still put an affiliate link in, because why not try to make some money when you've put in the work of making a good review, but at least they'll have bought the gear themselves and are using it because it's what they like best, and can speak to the pros and cons of the equipment in detail.

So the big lists are still useful - check out a couple "top 10" lists, read the specs, and get an idea of what's suitable. Then look up the promising candidates individually, and find more "real" content about them. I find this usually gives results which are better than just reading Reddit discussions, at least for higher value items.

With something like exercise bikes, the calculus changes. You're going to have much fewer ratings, and people tend to only buy one bike every 10+ years, and they'll want to buy the best one they can afford. Now there's a real incentive for a company to manipulate the ratings, and it's also much easier to do so.

I totally agree though that the current inability to answer the question isn't new. And I don't think that will change - I don't really see a viable alternative to the current situation. But if someone comes up with one then I'll be very excited to see it, as the current approach of using lists from big sites to go find higher-quality reviews from smaller sites is not the most efficient. But maybe that's not the worst thing - I like that there's still value to small independent reviewers.

IMO this is true.

It used to be however that the search results contained the phrase you looked for. This was later I think revised to also include pages that had links with the text in question pointing to them.

This kind of made sense for some queries (opinion based?), but for other queries (looking for technical content) it is actively harmful as it drowns out every useful site.

I feel like this exact question was asked in a meeting then peloton appeared. It turns out a bike also needs to have an ipad strapped to the front with a fitness instructor reading your name off a list of connected users and offering personalized praise. That'll be $1500 up front then $40 a month please.

I slightly worry when I have some product I rely on in my daily life that just does its job well. How long until the corporate race to the bottom hits this industry too?

It’s infuriating.

But now that I've typed that, I can see exactly why they haven't responded: a search company telling the world that its search results are f**ed. That would do wonders for the stock

Two things:

1. https://duckduckgo.com/traffic DDG has seen exponential adoption(yes actually mathematical exponential it seems) until last year. Now its traffic just increases at steady rate that most growth hackers would envy.

2. this happens despite their search results being just as bad as Googles: they (pr probably Bing) also sends me tons of completely irrelevant results because they ignore what I write in favour of what they think I write or want me to search for.

DuckDuckGo.com is my go to not because it is better but because:

- it is equally good now

- moving from DDG to Google is faster (just add !g)

- they don't have a history of showing me irrelevant and insulting ads.

With Duck Duck Go, there’s much less stuff to scroll past to get the results.

Test query: Samsung Galaxy

Google has 3.5 cell phone screens of header to scroll through. DDG has two screens, then the results, then the remaining screen or so of stuff google jams in the header.

This is much better.

For instance, the list of suggested searches is after the search results. Why does Google think I want to refine my search or switch to a different Google product before I read the results my first attempt at searching yielded?!? Do search logs show that most searches are from people that are typing things they don’t care about into the incorrect website?

Ignoring that, let’s quantify the results by position on the page. DDG has the top result in position 4 (samsung) or 6 (wikipedia).

Google has the first two hits at positions 16 and 27. (And stutters a lot of similar looking Samsung pages for some reason.)

I’m counting horizontal rows of content, skipping headers and columns of additional content.

(Edit: typos)

Good point. Also their main focus seems to be privacy, not quality.

The privacy problems of Google just recently (i.e. last 5 years) started to bother me, their complete inability to fix their broken search operators has bothered me for a decade.

My guess is if DDG provided the same quality as the old Google instead of just this nostalgia play it could possibly rise even faster.

I've been on both sides of paid reviews on Reddit.

I still do the same thing because in some subs you'll get actual conversation in the comments, but it's definitely being manipulated.

Instead of fixing the spam they are instead encouraging companies to spend more and more time on SEO and coming up with their own shenanigans like better ranking for using AMP (defunct now).

People who generally make great content (think a researcher or a great software maker) can't compete with billion dollar companies like Canva, Shutterstock and Pinterest who spend millions of dollars on SEO and have dedicated SEO employees who spend all day sending outreach emails and doing experiments. Henceforth the good content never even sees the light of the day; drowned by all this "SEO" optimized content.

FWIW i still believe it's the job of the search engine to find great relevant content and show it to the user instead of the other way round. Though I know it's much easier said than done.

(1) https://news.ycombinator.com/item?id=25538586

I do not think this is true, some businesses got special stuff in the white pages.

Even with the yellow pages, listings were alphabetical, and grouped by business type, and businesses could pay for bigger listings. I’d forgotten how information dense listings used to be (probably due to lack of filtering, but it still makes me want yelp to do better)

https://www.loc.gov/resource/usteledirec.usteledirec04924x/?...

- Many of the problems are self-inflicted (dropping search terms, pages of other stuff before first link).

- They’re getting worse faster than their competition.

- Google apparently helped at least one external SEO team game image search into relevance-oblivion.

(Also, “narrative” usually implies “fiction” or “concerted disinformation campaign”, and is either used as a weasel word by liars referring to their own writing/reporting, or it’s used as a pejorative. I don’t think you meant to imply either.)

What did I miss?

The fundamental problem is that Google and the SEO spammer’s interests are aligned. Google is both the search provider and ad network. I think this makes Google tremendously vulnerable to competitors who don’t have that conflict of interest, and presents a massive opportunity to those with enough courage and cash.

Google makes money regardless of the quality or 'originality' of the content your search comes up with so they currently have no motive to change things.

This eliminates the conflict of interest where spammy results kick off ad auctions that the search engine profits from.

At this point, revenue would be proportional to: market-share * expected-number-of-searches-before-success

So, the dominant player (with market share near 100%) would be incentivized to make the first search or so produce crappy results.

2) This incentive can be eliminated by throttling the rate at which ads are delivered to a given user (such as by repeating the same ads for free on search refinements, or simply skipping ads on search refinements).

DDG is still in step 1, where the upside on market share is much greater than the upside from spamming low quality results to increase incremental ad display rates.

All of these threads devolve into anecdotes and reminisces about the "good old days" and complaining about Pinterest. None of which is in the least quantitative.

I'd be interested to see some actual data or research on the subject, if it exists.

Or maybe it's not Google that's gotten worse but the web itself? Again, quantitative results, please, not anecdata.

Google has. They use this data expertly to improve search. Common sense and technological advancement tells us that, quantitatively, Google search has become better year over year, for all their relevant metrics/cost functions.

And likely, exactly because it has become better for all its users in aggregate, it has to become a bit worse for a certain group of power users. There, we can only rely on anecdotes and personal experience, but these tell us it actually has gotten worse.

Similarly, the web can become both worse and better. The really useful articles today are better researched, multi-modal, solid web of links, internet-first. Spam has also evolved. And "top 10 ways to do X"-McContent outranks better articles, because that is what the majority of Google users wants to see and clicks on. They truly have a better experience, while others' experiences suffer. It depends on what you measure.

lmfao. so you're telling me "quantitatively" that google search results have gotten better, without citing any data at all, but with an appeal to common sense and "technological advancement"?

what if i told you that search is an adversarial problem, and that it's possible for google's tech to be getting better slower than the aggregate tech power used to game google search is getting better? is this not a patently obvious possibility? it's not some kind of gotcha impossibility for google's tech to get much worse over time, even if they weren't hamstringing themselves by lots and lots of user-hostile changes which benefit google's interests rather than their users.

Yes. If that sounds so unacceptable or strange to you, I suggest you try it. Works really well when reasoning about unavailable data, or researching a field with slow peer-review process.

> what if i told you that search is an adversarial problem

Then I get an adversarial reaction and a downvote from you.

> it's possible for google's tech to be getting better slower than the aggregate tech power used to game google search is getting better? is this not a patently obvious possibility?

Yes, that's plausible. Should be measurable quantitively too. Can you cite some data on this? :)

We could compare to the available data on the quality of (HTML) e-mail spam filtering over the years, which all have kept up. Like pg said: Spam is solved, when skilled spammers start creating content which does not look, feel, or talk like spam. So webcontent-spammers still are on the first 2 pages of Google, but with content not classifiable as spam/content farm.

> which benefit google's interests rather than their users.

One of Google's interest is their user. But perhaps not the type of user you are. Studies have shown the value that the tech of Google is delivering its users per year. This value was in the thousandths, and this value has risen. Meanwhile, Google makes about tens of dollars per user per year, less for technical users which don't click ads or block these.

> it's not some kind of gotcha impossibility for google's tech to get much worse over time

It really is, no way to mince it. Google search is funded by Google ad tech. Google ad tech has improved ML by a ton. To say Google tech is getting worse, is to totally overlook deep learning revolution, word2vec, transformers, BERT, etc. etc. etc. To state that, is to reveal the truth that you are ignorant of major technological advances in the past decade, only looking at the issue from the viewpoint of a single atypical Google-search user. What would you even do with quantitive SEQ data?

Do you really think it is possible that Google runs an implementation test of a new ranking model, and deploys it, while all measurements, human labeling, and user tests show it is doing worse? Of course not! Only if you think you are smarter, could you think that Google search changed and has gotten worse.

If Google does ML like the rest of the industry, all model changes move up their designed levers, or these changes are not committed. So if Google was unable to improve search, then Google would have looked exactly like 2008 Google. The fact that it does not, shows either you or Google is wrong. If I had to make a bet...

It's a bit bloody rich for you to come out with that attitude when your entire point revolves around your opinion of 'common sense'

> Do you really think it is possible that Google runs an implementation test of a new ranking model, and deploys it, while all measurements, human labeling, and user tests show it is doing worse? Of course not! Only if you think you are smarter, could you think that Google search changed and has gotten worse.

You seem to be confusing the concepts of 'Google have made their tech more profitable' and 'Google have made their search capability better'.

> You seem to be confusing the concepts of 'Google have made their tech more profitable' and 'Google have made their search capability better'.

No, you are confused. Try to Google the article I was talking about. It talks of perceived value to the user. What value would you lose without access to Google maps, search, youtube, gmail, etc.?

Google made their tech more valuable. That sounds like an improvement to me.

> Google have made their search capability better

If you want an explanation for this obvious statement (and not be demanded to ask in return how capitalism works), I suggest you first try to code a simple search engine. I think a 100-line Python script with some imports would do. Only this would make talking about capabilities possible.

I think they've accepted that SEO has killed past ranking algorithms and are rebuilding rank from the ground up using ML with the entire internet as guinea pig. All of the crap results we're weeding through now is grist for the ML mill.

I am literally typing exact phrases for content I know is there and not getting the results I should.

I think they've cut the cord with past algorithms, not an incremental update, a major one.

My thinking so doesn't make it so, however, so just my two cents.

I've actually noticed Gmail spam detection doesn't work as well in the last year or two. I get an obvious spam message maybe every other day. Ads with all images and all.

The exact clones of StackOverflow and Wikipedia are all over the first two pages of Google.

Is it the any surprise if it ranks highly on alternative sites, spam or not?

I also think you're underestimating average users. Anecdotally I've heard my parents complain repeatedly about the incoherent, auto-generated, affiliate link spam that plagues product searches.

They have multiple levers, of which user search quality is a big set. There is always trade-offs and a balance that must be found, which aligns with company vision and strategy. Having these levers allows business-decision makers to direct focus top-down (on a certain set of users, on producing great ad numbers, etc.).

It is clearly hard and important to design these levels and find the right balance, given a rapidly changing company and user-base. So a lot of expertise and power is invested to measure the right things, and to find the right balance (an incorrect/risky balance should also be adjustable with other levers).

So for me: either Google is trying really hard, but essentially failing. Or they have the best of the world, with all the right context, designing these levers. While hard and sometimes wrong, I do not expect to contribute anything which may improve their lever settings. If someone does know, Google would like to hire them.

So while true, that accurately measuring things with proxies, is really hard, and sometimes done wrong at companies. I do not think Google gets this wrong, or at least, gets this to be the best of breed. If their metrics still cause long-term search engine quality loss, would show them to not know what they are doing. I think they do know very well, better than me at least.

I would agree too that the balance of levers right now is in line with Google's strong market position. Search engine quality could take a small hit, if justified with extra adsense income. But when search engine quality noticeably start going down, then all other metrics will suffer. You should have teams with sole focus on improving quality. Other teams will have to realize that favoring their lever over the search-engine-quality lever must lead to worse outcomes for Google in general.

About product searches, I myself was not able to do this satisfactory 15 years back. It improved. But need to stop viewing things as a single lever, a single metric. To say search has become "worse" in general, is to exactly fall into the trap of not accurately measuring and losing too much nuance/details for competing objectives.

If Google really really wanted they could find out why they more often than not include results that doesn't contain my keywords even after I have put doublequotes around them and hunted down and applied their verbatim setting!

After that they could think really hard about how relevant the text:

> and something someone said xyz.

>

>abc is next up and something something

is for someone searching for "xyz.abc"

Or maybe see if they can dig out an old cheat sheet with all the operators they used to support and invite some old Googlers to secretly come in and teach about it but that can wait until they got the basics working again.

Without access to both Googles, the best you can do is compare across different search engines: special-cased ones like search.marginalia.nu can net you a quantitative feel for what exists out there that's less likely to be content marketing, but I am not sure if you can figure out where those pages rank in Google search results for the same terms programmatically?

You can also prepare for the future: record some data today, and compare in 10 years time.

It is pretty quantitative yes, but it happened at different times for different people.

But it used to be that when you searched for something you got pages containing the thing you searched for, and if you couldn't find it at first then you could do a search on the page and find out some enterprising scammer had included your keywords in white text on white background.

Today Google and Bing has teamed up with the scammers so they don't need to use such hacks anymore. Google and Bing will include the results even if they don't contain said keywords at all.

Uncharitable? Yes. Do I hope Google and Bing engineers read this and fix it or do I hope some Russian enterprise launch a better engine?

I actually hope Google find back to its roots! I don't hate you guys but you really don't make it easy for us in between using all the oxygen in the room and annoying me all the time with useless time wasting results.

- There are way more Google users, including grandmas.

- Conversations have moved from discussion boards to walled gardens and chats.

- Google relies more on neural network embeddings, so does a better job when you type full sentences and semantic similarity.

- Google relies on authority signals and incoming links to a website, so non-commercial, hobbyist, or controversial content ranks way lower.

- Websites rely on Google for income, so they start producing what Google and its readers want to see.

- Spammers rely on Google for income, so those surviving after decades, have created massively successful linking rings and spam production pipelines looking at keyword search statistics.

- You were really good at Google searching years ago, having a harder time updating and letting go of what worked for you. Easier to blame Google for this.

As for tips: Anything academic, search on specific websites or Google Scholar. Anything technical/coding, search on StackOverflow. Anything cultural/commercial you want a peer answer, instead of a salesman answer, search on Reddit. Try to join like-minded communities where you can ask expert questions, and research new things in your field. Exact keyword match still works by enclosing keyword in double quotes:

    "sal dulu antasma"

This is a completely miserable experience, and walls off useful information into classes of people who "are in the know" about where the most relevant information exists.

And if you're that grandma searching for a birthday present for your grandson? Good luck. She's likely to be devoured by ads, if not an outright scam.

Agreed on the miserable experience. Do you have any ideas on how to attack this? Perhaps Google started out with the right experience, but ads eventually toppled it. Perhaps Google never hit on the right experience. What gives?

This is much more like what Ye Olde Webbe was like. Sites competed to build communities that were repositories of information. Things like Reddit tried to build a generic silo so that they could silo information there, which I think is a bad thing long-term.

The biggest problem, as I see it, is sites just give up on doing their own search. Not surprising, as search is a hard problem, but it plays merry hell with the democratization of the Internet to foist the problem off onto Big Corporation Inc. to do the heavy lifting.

A related problem is that many sites simply don't have what could be called a "webmaster" anymore. Everything is contracted out, or part of a subscription service, or otherwise disconnected from the owner of the site having full control. If you're a small business that sells locally produced products, you're never going to appear in Google or Amazon searches, even if you have an Amazon store. You can't afford a full-time webmaster just for your site, and all of the various platforms, like Wordpress/Shopify/etc, deal in such volume that these small businesses will be largely ignored.

The ISV model for products like AutoCAD is possibly a good route. A team of well-versed engineers and designers can build things, but you need a direct customer representative to get at the juicy meat of what the end-user needs. Apply this sort of model to search, and you can aggregate over larger swathes of customers.

Do you find this better? In my experience it’s nicer to just put stackoverflow, reddit, or (often, in my case) seriouseats in my google query. Reddit search in particular is pretty miserable.

That said, your tips are good. Thank you.

Edit: I'd love to see a some-of-the-web search engine like this. Start just with university sites, prepress archives, quality forums, public dev Slacks, etc.

Even though it's a bit broken, it has some lucid moments. Just compare:

https://www.google.com/search?q=mechanical+keyboards

https://search.marginalia.nu/search?query=mechanical+keyboar...

The take-away I want to drive home is that it's absolutely possible to build something the scale of Google c.a. 2003 and run it on consumer hardware. I think, due to general difficulties in making these things profitable, the ideal approach is to make the operation so absurdly cheap it can be run non-profit instead. I'll gladly pay out of my own pocket to have a good search alternative.