Unfortunately all these changes mean that it's now tougher to find old websites with specific information that you're looking for on Google nowadays, in general. It feels like Google used to be much more useful for that ten years ago than it is now.
Google's purpose seems to have changed from helping people find the most useful information, to funelling people to information that Google wants them to find on a particular topic, whether its the most useful or not.
Are we really still pretending like Goog/FB/etc. aren't effectively mass information control mechanisms first and foremost?
We have a blog on agrifood tech associated with our site and we’ve published over 1,000 articles often interviewing some of the most successful entrepreneurs, VCs, and senior execs at food and ag companies. We’re by far the most authoritative thought leader in the industry and if you go to any industry you’d hear our name a few dozen times but we get massively outranked by Forbes articles, TechCrunch, CB Insights (which is a poor authority in our space). It’s quite sad just how bad Google can still be.
If your industry doesn't have a large web community, then it's difficult for Google to know that you're authoritative. From Google's outsider perspective, all the links are going to Forbes.
You are thinking in terms of the old world, the PageRank algorithm.
Everything you say is true in that old world.
In today's world, Google is optimizing for ad revenue.
I could see how optimizing for PageRank could go out the window in such a world.
Even if PageRank still has influence, it's not clear to me the site doesn't have a large web community with a lot of links. Maybe it does, and the ad revenue considerations are overriding that.
That sounds to me like Google is optimizing retrieval for deeper searches within a narrower set of "authoritative" websites. The ideal solution would involve putting in the logistical and theoretical engineering work to classify and validate a huge number of websites good for fewer, specialized answers.
But for whatever reason, the reality seems to be a relative few websites are selected to provide "good enough" answers on shallow queries for a huge number of topics or objects, Wolfram Alpha style.
Maybe this is due to developer laziness, gradual dilution of cutting edge engineering, explicit desire for simpler solutions or real practical limitations, even at Google's scale. In any case, the result appears to be that sites like yours are poorly surfaced because they're not very generalizable. Whereas CB Insights claims it knows about your query already, there's an internal API setup for exposing CB Insights already, and CB Insights passes a basic sanity check on the answers.
This is really unforunate and I sympathize with you. I think there is a lot of opportunity for a search engine that can optimize for very specialized queries across many domains, including arbitrarily historical resources. But I think such a search engine would have very different goals and incentives than Google from the outset.
People still seem to find us depending on the exact search terms you’re using but it’s Google’s user experience that really suffers. If someone randomly publishes “the top 25 Agtech companies” on Forbes it’ll jump to the top of the list when searching for Agtech and stay there for half a year. Google is embarrassing itself with these kind of shallow and superficial results.
I think you’re onto something. Google’s “fake news” algorithm change probably didn’t want to be seen as politically or viewpoint driven, so an easier approach was to just de-prioritise large “authoritative” websites even for deep searches where more relevant results exist.
Actually, that’s almost exactly how Google described the algorithm update...
But not domain expertise. Google has a rich get richer ranking that favors broad generalist publications. If you’re seen as an authority in anything then it seems like you can also rank high for something tertiary.
> Google's purpose seems to have changed from helping people find the most useful information, to funelling people to information that Google wants them to find on a particular topic
Normally I only use Google, but I've also experienced this. In contrast with a few years ago, more recently Google gave me such bullshit that I was like, "Just give me the pages that have these keywords, this is so obvious" and I went to duckduckgo.com. There, with literally the exact same query, every one of the top links was relevant and had my search terms. On Google none of the first page of links had what I searched for. This certainly supports "they show me what they want me to see", rather than what I asked for.
On the flip side, I still find Google useful and I must say that I often am happy to read what they decide to give me: even if it's not what I searched for.
I've done searches on Google with 3-4 words and the results given back had excluded all but one word. The "not on page" results are usually pretty infuriating since that's what I am looking for
One of Google's cofounders once said that if there is more than one search result then that is a bug. That sort of thinking inevitably leads to this sort of thing. There are always shades of grey.
I'm looking for a link, will update when I find it.
yikes, i really hope they are not so shortsighted these days.
a lot of the time people are searching for something, they do not know exactly what they are looking for enough to articulate the perfect string of search terms and need a plate of more general results to get to what they want... this seems painfully obvious.
Page and Brin are saying that they want to deliver that specific final result even when someone starts off not knowing what they’re looking for by using context and their user data to understand what the search is really about. Not that they want to deliver one general result when someone makes a general query.
I’m not necessarily agreeing; just explaining their thinking.
I definitely think that's an incredibly legitimate concern and wouldn't want to dismiss it because it's such an important issue. It is also important to remember, though, that a lot of these changes are made in ways that (seemingly, at least) attempt to prevent sites from gaming the search algorithms. I have seen the quality of results I'd expect from a Google search go down, and I attribute that to quite a bit of factors, including Google's self interest. I also try and consider just the massive rise in general content available through the web, the further and further growth of it as a tool for manipulative information or propaganda, as well as its growth as a marketplace. I only mention these things because I think a lot of people, including myself, will tend to ignore a lot of the external factors that forced as well as allowed Google's search and results algorithms to be in the position they are in.
As far as search engines go, DuckDuckGo is pretty neat, have been using it for 3 years now and can count with my fingers and toes how many times I've used Google since then. And half of those times it was because I typed the search term directly into another person's browser's address bar which defaulted to Google.
I haven't used ddg as it used to be fairly terrible, but google may have slipped enough and ddg may have improved enough now, I'm going to give it another shot.
i've also used DDG for the last few years. it has indeed improved a lot. i don't really ever need to turn to google to get what i'm looking for these days.
Here's your monthly reminder that "relevance" is fundamentally subjective, and therefore all search and discovery services, both on the internet and offline, give you their opinion of what you'll find useful.
How useful you actually find a given service to be is entirely a function of how closely you agree with that site operator's opinion.
This may be a good thread to let people know about a hidden setting I found out about recently called Google Verbatim. I thought certain sites I used to search for had fallen off the Internet, but the change was how Google ranked its search results.
In any search result page, click Tools > All results > Verbatim. That is also how you restrict e.g. the temporal range of results (Tools > Any Time > ...)
We really need good, alternative search. I miss Lycos, Hotbot, Yahoo and the days where if you didn't find someone in one place, you checked another. Searching for things took real work and you could find a lot of different rabbit holes.
Today, rarely do we go past the 2nd page of results. About 1/3 of the time, I get frustrated with Duck Duck Go and add !g.
