And yet when I read the comments, I see this sort of calm acceptance that of course this technology is going to take over search. Why is everyone so confident? The performance of these models is simultaneously jaw-dropping and absurd. With no proposed solutions to the problems that they fundamentally face, why this level of belief that their ascendance is inevitable?

We now have a couple of the players saying they're working on, or having demos. But from what I can see, all of these don't _actually_ attribute the source. They're just able to find _a_ source that fits to the output, working backward from it. Often the attributed source fits, but is actually disagreeing with the original output in specific details. In other words, it's not backing the output at all. At most it's a "you could check out these links and compare yourself and it might help you judge our accuracy".

Is there any indication real attribution is coming?

This is not necessarily a "criticism". It is obvious that ChatGPT produces an incredibly rich and meaning-filled representation of the text, as the extension of the text it can be used for demonstrates. But we should not be trying to "tweak" the language model here and there, trying to make a language model do something it isn't really suited for. We should be working on how to hook up that higher-level functionality. It is not obvious how to do that, true. But it at least stands a chance of producing the AI in reality that people think we have now, but don't.

For that reason, I don't think attribution is coming without a significant addition to the architecture. Language models shouldn't be doing attribution; the mess you'd create of them to accomplish that would ruin them as language models. Can you imagine how insane you'd go if your internal language model tried to stuff down every instance of when you have seen and/or used the word "brain" every time you tried to use the word "brain"? That's not what language models are for. That's what the thing using the language model does. I expect AIs to parallel this. I see no reason at all, and abundant reasons to the contrary, to expect AIs to just be undifferentiated blobs of numbers. I expect them to have structures just as our brains do, and for the exact same reasons. If undifferentiated blobs of identical neurons was the way to go, our brains would work that way too; it's far simpler than what we actually have.

[1]: https://www.jerf.org/iri/post/2023/understanding_gpt_better/

So I don't think that remembering sources is a hard problem to solve, likely making the AI work well is probably easier if it remembers sources.

I think it's hard for language models, and indeed, after further thought, I'm pretty sure they shouldn't remember sources: https://www.jerf.org/iri/post/2023/understanding_gpt_better/

But the problem in general is probably not a particular stopper.

A bigger problem is people thinking that we should be taking language models and using them as AIs themselves, rather than seeing them as a component of an AI, which the more I think about, the more obvious it is that this is the case. Language models aren't what we want. They're a necessary step to what we do want and I don't expect them to go anywhere. But history will look back and laugh at this misguided attempt to make them be the entire AI.

Also, if you are implying you can just spew out the exact sources you learned some fact from... no you can't. You think so only because you haven't tried. Ask someone around you for five random "facts" for you to cite your source from, then try to write on a sheet of paper where you heard this fact from for the first time. Then hand the sheet of paper back to your friend and have them verify your claims... so, note, immediately you have the problem that an unverifiable claim is a fail, so "in elementary school" doesn't cut it any more than you can cite "at the library" as a source on your high school essay. I'm not looking for words that "identify" a several year span of time. You need to give me book, page, and line, or timestamps on a video, or something like that. No, you can't. Nobody can just spit out a bibliographic citation from memory for when they learned that Pluto isn't a planet. I can give you "in the news" for that, sure, but what news source was first? What is the title of that article? Who wrote it? What was the exact date? Of course not.

[1] https://www.youtube.com/watch?v=peHkL_MaxTU&t=1558s [2] https://news.ycombinator.com/item?id=34695306

I feel like it should be possible to build attribution "simply" by finding good results via traditional search (or even using LLM embeddings), then asking a LLM to summarize the sources. Then you can show the attribution. This would (to me) be much more meaningful. Especially for specific searches ("I want a red oversized couch thats got good back support"). Don't tell me about couches then find a random link, find a couch then tell me about that couch.

Is anyone doing this? IDK, maybe there is a reason that it's not happening, but it seems at least as easy as all the other crazy (and cool) things happening in this space.

I wouldn't hold my breath. Showing attributions allows people to reverse engineer how the prompt was made, and identifies weaknesses in the model. Bad for business!

For example:

(a) Addressing the recency problem by making a model generate instructions to a browser as output token stream, so it can, well, Google for you and incorporate latest info into output by adding it to the "internal prompt".

(b) Doing something similar where you teach models how to drive computational engines to make actual, computed facts and figures, not guess them from statistics over training data.

(c) Maybe running a traditional search and then using LLMs to summarize over them.

etc. etc.

It feels increasingly clear that "complete prompt from a stale training data set" is a neat trick but doesn't solve any particular problem well enough. This also seems intuitively true: If you ask your friend a difficult question, you often don't expect them to be able to answer it on the spot. You probably expect them to have relevant experience and a rough understanding of the topics and the tools to look up or figure out the rest. Similarly, it feels intuitively wrong to expect a LLM to come ready-made with all knowledge baked in. It feels more like we'd want something with the working knowledge/method to plan and execute fact discovery and computation.

It's fun to play around a bit with ChatGPT yourself and try to approximate what a more complex pipeline might be. For example, instead of asking it to answer a question, ask it to re-state the question or make a list of questions prompted by the original question, then let it "contemplate" those or propose a method for how to go after answering them, etc. It runs aground pretty often, but sometimes the results take you a lot further than the direct approach.

Not sure. In theory, this being computers and all, I sort of do expect my search engine to effectively come "ready-made with all knowledge baked in". I do not expect Google to start a new web crawl because I gave it a novel query, and I don't expect a language model (certainly not one being used for search purposes) to say "hang on, I need to go read some more before I answer that".

A bit similar, again, to how humans will answer plenty of questions on the spot from working knowledge, but are also able to classify when they need to drop into a slower "let's do the math" mode and then carry that out.

We may be able to from these "thought processes" then generate training data sets that again reasonably cache "all possible accurate answers" as you demand, but it's not clear to me if that caching will really save computational cost or just redistribute it to different times. At that point it's probably a latency/throughput question of when you do vs. defer the "thinking".

When I ask <search engine> a question that is "demonstrably" answerable based on "reliable" sources (e.g. what is the capital of Mexico), I don't need or expect it to reason. I want it to (a) tell me the most likely answer, probably (b) provide a list a of URLs that also provide an answer, probably the same one.

And that's what Google and others do right now; ignoring the results presentations being skewed by commercial interests, they do it rather well.

Figuring out when a question is not answerable in this way, but requires reasoning, is certainly a part of the challenge.

Those look like citations, to me.

Looking at the main article on HN yesterday around it: https://www.theverge.com/2023/2/7/23587454/microsoft-bing-ed... you can see citations in the examples provided.

Source: worked on GopherCite.

if i'm being frank, that doesn't match my experience with HN at all. chatgpt is the hot new thing, of course people here are preoccupied imagining a future powered with it.

there are a bunch of different types of people here, with different perspectives that might be at odds with one another, but they don't necessarily comment on the same articles.

what puzzles me here is the the technical limitations of language models (i refuse to call them "large" because the sizes are going to change dramatically) seem so obvious and so well known, that i just find it hard to believe that there are more than a handful of people on HN who uncritically accept the idea that Bing is trying to sell.

As usual, it appears "tech" companies have an extremely warped view of web users as being non-discerning consumers of whatever garbage "tech" companies feed to them. Perhaps online advertising services companies are not a good choice to serve as arbiters of data/information search and retrieval.

I think they are realizing they killed their golden goose by filling it with trash ads and it only worked in the absence of something more accessible to the public. They were drunk at the wheel and crashed into (for the first time ever) a competent competitor. Now they realize they need to sober up quickly but their organization is lazy and bloated, they’ve taken their users for granted and abused them at the altar of advertisers, and generally done nothing for two entire decades to garner any form of loyalty.

They’ve got some work cut out for them. From my perspective, Google could come out with a ChatGPT killer and I still wouldn’t use it, because the company has a track record of arbitrarily killing off any product I’ve actually liked.

I don’t see that unanimous acceptance, there are a lot of interesting discussions about the shortcomings. There are many comparisons between Google results for examples, with also potential solutions to problems you cite like https://news.ycombinator.com/item?id=34709883

It’s not about replacing search overnight, but if it can take even 10% of search volume that would have serious consequences for Google.

I have no idea what Microsoft is thinking. It's like a state hiring a known liar to be its star witness in some notable number of court cases.

[ EDIT: OK, so one thing they might be thinking is to just this sort of technology to generate readable/listenable summaries of search results obtained by other means. I can see much less wrong with that, but it's also not really the same technology at all as ChatGPT et al. ]

We vote in known liars all the time. People tend to like a good story better than the truth. If this thing is good at storytelling we could be in trouble.

Probably the secret sauce to using LLMs for search and other forms of information retrieval is first taking a query and use traditional lookup to find likely information. Then aggregating this data, provide the aggregated data as context, append the query, and pass it to the LLM. Probably some post processing is a good idea also.

I used to do this with squad LLMs.

Because plenty of people don't see it necessarily as a replacement for search, but as a useful adjunct.

More to the point though, why do people always want to define the "HN commentariat" as some monolithic bunch? I've seen tons of comments and articles on the front page pointing out many issues with LLMs, from attribution, to confident bullshitting, to bias issues. Heck, there was plenty of discussion about a big proposed reason for Google's stock crash yesterday was due to the ad Google used to highlight questions their LLM can answer gave a confidently incorrect response.

Whenever I see comments that treat the rest of the HN audience like some monolithic group, especially in the face of plenty of evidence to the contrary, I feel like, ironically, the poster is saying "Why don't you all agree with me?"

However, my point here was that at the time of posting, the comment threads for this HN post were largely devoid of any skepticism about LM+search, and were completely dominated by a calm acceptance that it was all inevitable. And I wasn't trying to say "all of HN believes <X>", but rather "I would have expected there to be more voices in the comment thread expressing skepticism, because I know there are some smart people here"

Google's Bard demo certainly tweaked the situation by the end of the day.

What is the mechanism to address lying by a language model?

Wikipedia's mechanisms are fundamentally unable to cope with motivated, coordinated gangs that conspire together to push a distorted perspective, particularly on niche articles. Given Wikipedia's prominence, this is a problem that will only get worse.

ChatGPT might not be ideal, but they are way better than current offering.

I share your confusion regarding the grave dearth of incredulity in the reception of these technologies.

It is difficult to get a man to critically evaluate something if his future salary depends upon his unquestioning acceptance of it.

Mostly a question of pucker factor -- how big of a bet in favor or against, and how big of an impact on the job will you risk?

Only if they hit out of the park. You don't see a lot of founders getting a second bite of the Sand Hill Road apple if they failed the first time.

There are a litany of problems that surround LLMs that are solvable, but are highly non-trivial. The present hyped demos all seem to involve mainlining the unfiltered output of LLMs directly at the user, and this approach strikes me as largely impractical, with all of the issues you've brought up and more.

Ultimately I think we will land on a few learnings, some of which are IMO pretty obvious already with existing ML:

- There are going to be lots and lots of use cases for LLMs where despite its additional complexity it won't out-perform simpler ML models, or even heuristic-based solutions. This has been a plague for AI startups forever - where the shameful truth is that the ML bits do not significantly outperform heuristic-based approaches, but the ML bits sure as useful for the hype machine. There will be more of these use cases than use cases where LLMs actually do significantly move the state of the art forward.

- In most broadly mainstream-palatable applications the LLM will need to be intermediated by many other systems - some using other ML models, and some using heuristic-based solutions. For example for search, mediating the output through a knowledge graph with reliable provenance and accuracy. Realistically there will be vanishingly few use cases that are thin-wrappers around LLMs. Real usefulness of this tech will require very heavy lifting around the core model to make it actually accurate/reliable enough to be useful on a mass scale. The many products spinning up that seem to be premised on very thin wrappers around OpenAI's API will IMO mostly not survive, both because there is no moat and because the results won't be sufficiently useful.

- Accuracy matters and will be a thorn in the side of these products for quite some time, even though it is solvable. For some use cases (mostly creative ones with a human in the loop - like creative writing or image synthesis) the lack of accuracy is livable, but for most others the propensity to hallucinate is going to be a major barrier to adoption. Right now you're dealing with an enthusiast audience that's willing to look past glaring and obvious errors in output in favor of what it could be and how far we've come, but a mainstream audience I suspect will be less forgiving. Social media virality around product failure cases will also further punish products.

Overall I think what we're looking at here is legitimately a huge leap forward, but this legitimate and well-found excitement needs to be tempered by the fact that some (most?) players in the space seem to be getting in way over their skis, and that truly productionizing this technology is going to be intensely difficult.

[edit] To be a bit less doom and gloom about this - I think the real winner here are natural language interfaces, and stepping ever closer towards computer systems that do not need to be actively learned before being used. There is IMO an over-focus on the knowledge-encoding part of LLMs (which is unreliable and prone to hallucination) and not enough focus on the language-encoding part of LLMs (which is what they actually do). For the most part voice/language-based interfaces have had limited traction because they honestly haven't been smart enough to understand user requests with a sufficient level of expressivity and fidelity. I suspect these products (and new ones) will have another crack at this problem.

> but a mainstream audience I suspect will be less forgiving.

Less forgiving, perhaps, but also, I suspect, less discerning.

Hell, it literally just happened today. Google's first public demo of Bard failed in a highly visible way in a highly public setting and that's a whole media cycle now and has sent the stock flying downwards.

Is this lethal for the product? I suspect not at this point - but if they can't wrestle this stuff under control it very well be.

i made this same point at https://news.ycombinator.com/item?id=34699087 and got nothing but pushback how there is no real problem with ChatGPT and Bing just fixed all the problems (which didn't exist) anyway.

If this was a startup, they would be dead in the water. Nothing of substance, awful sound, 9:45 mins in they don't have a key piece of live demo equipment (the phone)/

As someone else said, it's like they asked a couple of interns to rush something out in 24 hours.

No wonder the stock price is tanking. Awful display from Google.

Finding elevators and atms with AR ? Is that the future google is selling ? Who's buying. If google maps was the equivalent of the invention the printing press that would be the invention of scented toilet paper

At one point they were unable to demo because there was no phone to demo on (~9:35).

But yeah the 720p demo was bad, probably why they deleted the original

That said, I still think this reflects very poorly on Google's organizational focus.

>Used to work for BB/RIM

I feel like Google is going to bleed credibility for every month that they don't have a lamda/llm enabled search integrated into the home page. You can talk all you want about the foundational models and how advanced they are, but the search product itself will require a ton of fine-tuning on real-world training data that they aren't collecting yet.

Unless Microsoft completely botches their "newBing" launch, this is probably the most potential they've had in search since MSN was launched.

I love ChatGPT as a source of inspiration, but I don't think I'd want it alongside my search results. In particular, when I'm searching the web I want results with at least some degree of accuracy. I can't trust a single thing ChatGPT says to be true.

Admittedly, random websites can also be inaccurate, but at least I have strategies for assessing whether I trust a given source.

Having a ChatGPT answer next to the google result would have saved me a bunch of time, and I could see it coming in useful in many cases.

Usually when I search for error messages for a specific version of software it will respond with other versions. That is useless. That's not what 24 years worth of excellence in search should look like.

At the very least they should collate a dataset of say, one million queries, and try to use a LLM to find the correct answers using search. A better search ranking model will make the LLM generate more correct answers. Iterate until score is improved. Just make the result solve the task.

"The fastest recorded time for crossing the English Channel entirely on foot (without the aid of swimming or any other means of transportation) is 16 hours and 34 minutes, achieved by British endurance athlete Damian Hall in August 2021."

So, it seems like it made up an answer instead, which I think is much worse that Googles response.

> What is the world record for crossing the English Channel entirely on foot

As of my training data, the world record for crossing the English Channel entirely on foot is held by Christof Wandratsch, who completed the swim in 8 hours and 58 minutes on August 25, 2018.

> Swimming isn't on foot

My apologies, you are correct. The English Channel is typically crossed by swimming. The world record for swimming the English Channel is held by Benoît Lecomte, who completed the swim in 8 hours and 1 minute on July 23, 1998.

> That's faster and earlier... what did Christof Wandratsch do that was different

Christof Wandratsch completed a cross-Channel run, covering the distance between England and France on foot. This is a different feat than swimming the Channel, as it involves running across the Channel Tunnel, which connects England and France beneath the seabed.

"The world record for crossing the English Channel entirely on foot (without the aid of any swimming or flotation devices) is held by Belgian endurance swimmer and long-distance runner, Kevin Swartz. He completed the 35-mile (56 km) journey from England to France in just 13 hours and 54 minutes on August 17, 2019."

chatGPT really is a brilliant compulsive liar.

I don't know that end-users really care about having a GPT-written paragraph accompanying their search results. Is there some other kind of output AI search currently provides that I'm missing?

For 90% of my searches I don't want any search results. I want a definitive, authoritative answer to a question. In the olden days of the web I had to figure out what to search to give me a website that would give me that answer as the first result, and in the mid-90s that was really hard because search was bad. In the late 90s when Google launched it got much easier, especially for things where there was a network of websites about a thing that all pointed to a canonical authority. PageRank found that authority source. Life was good.

Over the next 20 years the war between Google and SEO spammers meant finding things got a bit harder. Along with that Google's proliferation of adverts on their search result pages meant it actually became harder to even find the website link I wanted in the page itself. Fortunately Google also started putting knowledge graph things in the results and the sidebar which made life less painful.

I think we're due another shake-up, and Bing-GPT might be it. If they show me a simple paragraph of readable text that explains the answer to what I've searched for then that would improve most of my searches. It'd be like Google's knowledge graph box but for everything.

Yes, SEO spam has largely made the search experience a lot worse and ripe for change.

But ChatGPT crawls these terrible links too! Many of the links Google spits out are themselves summarizing information found elsewhere, like a book or academic paper.

So why would you trust GPT to provide an authoritative answer more than your own instincts? It's omniscience is a function of everything it's crawled; that doesn't mean it has any idea of how to weave them together into a coherent response.

To use a highly specific example, I've recently been doing a lot of work in Excel and the search space for Excel related queries has been almost completely destroyed by SEO adware sites. If you're lucky, the answer you're looking for is at least buried somewhere on the page you click on, but there's just as much a chance you'll get a semi-gibberish pseudo-article that itself is probably AI generated. Using my instincts/intuition to locate the correct answer is a non-zero cost of time and effort, made intentionally more challenging by site owners who want me to linger and consume advertisements.

On the other hand, asking ChatGPT how to match two columns with XLookup or what have you almost always returns a correct, concise answer on the first try. If it's not correct, the formula doesn't work, and I can tell ChatGPT it gave me the wrong formula, and it corrects itself. Even in the rare event that happens, it's a fraction as much effort and frustration as it takes to rummage through the top search results for the same query.

I wouldn't right now, but I believe the accuracy of an LLM is a function of its size and how much you can reduce the loss function. I think OpenAI can improve those things faster than Google based on the things that have been shown so far. I could be completely wrong though.

I also suspect there's no benefit to corrupting a GPT model except for the lulz. If a search tool isn't showing links then there's no financial reason to put effort into setting up link networks and SEO content farms. GPT based search has an innate anti-spam advantage because it renders spam futile if the spam isn't getting visitors.

It also renders a lot of non-spam content futile too, which is problematic. It's not a magic bullet.

They should make an AI bot to search through the search box and decide if the results are crap, and train it with human preferences. Or set the bot to find difficult to find answers and see how many it finds and how high they are ranked.

Probably working in search at Google was a dead end career, the search department wasn't sexy anymore and didn't have green field projects. Maybe they were just afraid to change anything as long as it makes money for them. So the good people left and they have been coasting for a decade.

In the real world we don't have any agreed upon general notion of "definitive, authoritative" sources. A certain amount of skepticism is important to avoid all sorts of abuses of authority and "appeal to authority" is a recognized logical fallacy.

Be careful what you wish for. I'm pretty sure I don't want to live in a world where everyone behaves as if the response from a ML system is "definitive [and] authoritative".

If you just want an answer that sounds definitive and authoritative then these chat bots are great. If you want it to be accurate as well... that's a different story.

If genies existed there'd be a lot of 10 inch pianists and dead people buried under piles of money.

Which, of course, is what Bing just announced. Whether it will be the right answer remains to be seen.

Notice that there's a price increase, of sorts, for this at Bing. To get on the "waiting list" you have to sign up with Bing and agree to receive marketing information. Will public access to large language model search come with a decrease in privacy? It did for Siri and Alexa, which learn about you and cannot be used anonymously.

browsers are left with challenges, particularly legitimacy of results, and ensuring a viable ad model. No doubt we'll see those soon.

You mean 90% (or X%) definitive answer with no good way to verify it due to zero context (unlike in normal websites)?

They are already angry about Google doing something similar and ChatGPT is worse to them

As others have pointed out, AI is not yet capable of this. I do agree though, I want this too. But I don't trust OpenAI or Google or anyone else to do it correctly yet.

We use deities to name our sprints at work. When you search Google for “Deity starting with <letter>”, you get a list of greek and roman gods. I got tired of that.

So I asked GPT “what is a god that starts with <letter> and isn’t roman or greek”. It immediately spat out a Norse god. Next time it was Hindu. The increased variety was refreshing as hell. Google was not able to process a negative query.

Plus the added paragraph of description about that deity was lovely.

The point is: Sometimes (often) I don’t want search, I just want a good enough answer.

The actual information value of these 'results' is usually close to zero. I went onto a top search result yesterday, and the scammers hadn't even bothered to do a decent page. It was just a list of links to 404 pages, with an intro paragraph to satisfy the Google algorithm. It was pathetic and a waste of time.

That's why they're running scared. They know their key product is broken and useless, which means they're ripe for being ignored.

But I get it, most people dont want wikipedia because they really do seek the 10 second waffle answer.

Lots of my searches are simply I want an answer, but often times I also want to then do something with the result. This is where ChatGPT is going to eat Google's lunch, IMO.

also, how hard is it to get a list of abbreviations, and wrap them all in quote and comma and brackets. with vim that's barely more work than a copy-paste

Agreed. I was able to manually review 52 results, but wouldn't trust it blindly.

> also, how hard is it to get a list of abbreviations, and wrap them all in quote and comma and brackets.

Not hard at all. And I may still do that, but it was pretty enjoyable to ask the machine.

Yup. So when the machines take over, hopefully they'll enslave me last, or put me in a more pleasant Matrix instance compared to the average.

https://en.wikipedia.org/wiki/Roko%27s_basilisk

I still do it. ¯\_(ツ)_/¯

That's been a serious problem for some time. Preceding a query term with "-" doesn't seem to do anything any more.

Google is done

How about I ask it how many games Jaime Benn has played? Or what the score of the last Dallas stars game was, or when the next one is?

What if I ask ChatGPT for a copy of the manual for a meater bluetooth meat thermometer? It literally just told me to Google it.

What if I ask which car manufacturers are joining Formula 1, and when they'll be starting?

What if I ask ChatGPT: why do people resist new disruptive technologies?

People resist new disruptive technologies for several reasons, including:

Fear of job loss: One of the main concerns about disruptive technologies is that they can lead to job losses, as machines and algorithms replace human workers.

Lack of understanding: Some people may resist new technologies because they don't understand how they work or what benefits they offer.

Threat to existing systems and norms: Disruptive technologies can challenge established systems and norms, and people may resist them because they fear the changes they bring.

Ethical and moral concerns: Some technologies raise ethical and moral questions, such as privacy concerns, and people may resist them as a result.

Financial impact: Disruptive technologies can be expensive to implement, and some people may resist them because they can't afford the costs associated with them.

Habits and comfort: People may resist new technologies because they are used to their existing habits and are comfortable with what they know.

Lack of trust: People may resist new technologies because they don't trust the companies or organizations behind them, or because they don't believe that the benefits of the technology outweigh the risks.

It's important to note that resistance to disruptive technologies is not always a bad thing, as it can encourage companies to address the concerns and ensure that new technologies are developed and used in ethical and responsible ways.

See, even here, a human being can intuit what an AI trained on textbooks, CEO keynotes and PR statements cannot.

People resist because their bullshit detectors went off. Search AI has yet to 'disrupt' anything. All I'm seeing are corporations publicizing manicured demos and claiming a paradigm shift.

What about the other 95% of search volume that isn't tech-related?

I had a 19yr old cousin tell me that she almost uses TikTok exclusively for tutorials and when I pushed back and said why/how the other teens in the family heavily agreed. I’ve since asked ~5 gen z (I don’t interact with many!) and they told me the same.

However, I could be completely wrong and TikTok video is the future of search. If so, where does that leave a text-only ChatGPT?

All the guys pointing out what ChatGPT can't do right now are simply missing the point. It's disruption baby. But if you feel better then: Don't look up

This becomes especially relevant if it's parsing through a bunch of technical documentation that is out of date. There are numerous developer-centric websites that rank high on search, but simply scrape and clone information from legitimate websites. These are the ones most likely to carry outdated information.

A huge percentage of searches are for existing websites, like "facebook." This might not be a big challenge if you just wanted to go to facebook.

This morning I saw a retweet from a Google/former Deepmind engineer, of someone who said they completely switched from Google to using perplexity.ai. I checked it out again (I've used perplexity before.)

I did a test search for interest rates, and perplexity returned the correct interest rate for the day along with the trading range. I think Google is really in trouble. I'd say the golden goose is in the process of being slaughtered.

None of this is a surprise. Google's voice assistant has been really good for years. Nothing came close to it. Siri is close to useless. Maybe Google can pivot and they'll be fine. I don't know what the economics look like.

https://www.perplexity.ai/?s=u&uuid=6f7fe592-c11f-4df9-a2bf-...

https://www.google.com/search?q=what+is+the+current+average+...

Google's results were far more informative and usable.

I then tried something I thought would be more up Perplexity's alley: "Who was the first US president with a last name starting with R?"

https://www.perplexity.ai/?s=u&uuid=8d60cf3f-eab5-4af6-a616-...

Their answer is not correct. Google didn't even try to answer my question directly, but it did send me to a government website with a chronological list of presidents where I was able to quickly verify that the answer was Teddy Roosevelt.

ChatGPT is amazing, until it isn't.

I’m not sure they can compete within their existing brand/company structure.

If they make it promote stuff for pay, it's worthless—you already have to second-guess whether it's right because it might accidentally be wrong, if it might also just be acting as a paid promoter, that puts it firmly in "easier to just not use it" territory.

If they put one out but don't make it into an AI infomercial, then it eats ad revenue and investors get pissed off.

But if they do nothing they're clearly headed for trouble, too.

They're in a tough spot.

My guess is they're release something, but keep it strictly separate from search in hopes that it won't replace search traffic. I doubt that'll be enough long-term but it might quiet fears for a while.

If Google is going to survive this, they need to find a way to get back to prioritizing user value while also making money.

ChatGPT style conversations would give Google a much better idea of exactly what you want to buy. It must be possible to use this signal to sell more ads, not just match what they currently have. Essentially, make sure ChatGPT gets asked the same question Google currently answers: "how do I get the person searching to buy something from this (loooong) list of ads?".

Having people help Google (not just in one search, but a potentially long list of refinements) to find exactly what they would buy should make Google ads perform much better yet again.

Of course Google still dropped the ball by taking way too much time to put their LLMs into use.

I feel like Google is going to bleed credibility for every month that their home page is lacking a system which almost by design is going to lie to me without me realizing at least as often as it might usefully summarize information from primary sources that I wanted to read myself.

OK, then.

Sometimes a press release can make all the difference.

(Sorry can’t find story that old)

Transmeta released two x86 CPUs a bit after 2000, which ended up being technological failures.

Cyrix released some budget CPUs in the 1990s but had legal issues and ran into financial trouble, eventually ending up as part of VIA.

Centaur Technology developed some CPUs in the 2000s, and eventually became VIA's x86 division. They released efficient low-cost processors, which were reasonably successful on the embedded market. VIA is still around, but doesn't bring out a lot of new products.

I think most of them were killed mostly by the dotcom bubble, and CPUs getting increasingly complex. The market just shifted beneath them. It is rarely a matter of just giving up.

In the end Itanium was late and bad project but competition had already folded. Except for amd that had no existing 64 bit cpu and created AMD64 extension to x86. Intel grudlingly got on board but called their clone of AMD64 "IA-32e" architecure in thei marketikg material. The name was later changed to x86-64, only after what I assume when Itanium was internally acknowledged as failure.

> The HTC hardware and Android OS that powers it lack the polish and depth of even the iPhone 1.0 in most respects.

> My first-generation iPhone with iPhone OS 2.1 feels faster and slicker than this late 2008 G1.

> The G1 has a single-finger touchscreen that supports a few gestures, like sliding. After working with an iPhone for many months, the lack of multitouch stands out as a big loss, but those new to touchscreens are unlikely to find it a big deal. A more critical problem, however, is the lack of sensitivity in the touchscreen.

> The browsing experience is frustrating and awkward, and I expected far better. With the WebKit browser core being used by both Nokia and Apple, and Google immersed in its own Chrome project, I thought the G1 might include a browser that had something unique to offer. Instead, it plays like something from about 2006.

(from https://arstechnica.com/gadgets/2008/10/android-g1-review/)

It was definitely eye-opening to see people’s usage habits shift so quickly. I know multiple large organizations who completely reapproached how they developed internal applications because the executives no longer accepted IT’s unwillingness to support mobile access. I’d previously thought that would never happen.

Amusingly, the one which had a damaging public breach did not have it due to allowing phones access as some had predicted but when one of IT’s managed laptops was stolen out of someone’s car and, unlike an iPhone, it didn’t have storage encryption enabled.

It would be awesome, actually. Demonopolization of search space is long due. We need some breath of fresh air there.

ChatGPT has similar network effects with feedback. If Google is displaced, it will likely be in favor of another single dominant solution, not a commoditized industry with lots of players.

It’s possible that niche domains like medical advice or creative writing assistance will spawn small, highly-optimized companies, but general purpose AI responders will probably coalesce into whatever the next Google is, if it’s not just Google.

But lots of industries don't have particularly strong network effects. Cars are a good example; there's brand affinity, but there's no particular reason you need to drive the same brand as your neighbor. There are scale economies, so it's still not easy for new entrants.

Fun trivia- Most of the BB employees in my department switched from Storm to Bold when the Bold was released. As a result I had a pile of Storm in my drawer as my test devices.

As I recall, the development cycle for Storm wasn't two years - it was less than a year. At that point it was the fastest cycle that RIM had ever had for a device, and it was done at the request of Verizon. Back then the iPhone was still an AT&T exclusive in the US market and Verizon demanded something from RIM so that they could compete with it.

I'll echo the fact that nobody I knew internally seemed interested in using Storm as a primary device. I think everyone knew it was junk. But Storm 2 went a long way towards a device that I would actually be interested in using.

As you said, the Storm was developed in a year so some corners (especially the OS side) were cut. The Onyx was 'properly' engineered so (internally) everything was better and easier to deal with.

Why? (Genuine question).

Google is just some corporation, and one that’s been drifting for the last 15 years or more (since Schmidt left). They are not even a good actor, yet (and I don’t blame them for this) have sucked a lot of air out of the ecosystem.

Creative destruction is one of the key values (-> leads to benefits) of capitalism, and with US & EU antitrust asleep at the wheel we should be glad if Google is ground to dust and replaced by somethings (plural!) better.

Because in an ideal situation (as indicated with your last sentence) you'd have a bunch of active competitors yielding up better and better products. Not one active competitor with the singular new thing and a couple of reactive competitors that can't get things right.

Search is also supporting a lot of relatively free innovation at Alphabet. If ChatGPT sinks that innovation (or at least makes it non-free) the loss may be more than the gain.

It appears that neither does openAI believe that as they were happy to sell 50% of their company in addition to all the ownership they'd already given away. That was clear harvesting.

> Search is also supporting a lot of relatively free innovation at Alphabet.

Is it, really? Hard for me to come up with much of a meaningful list, even if you don't try to compare their history with outliers like PARC or Bell Labs. They are clearly spending but I don't see a lot of doing. They seem to have ignored Hamming's advice.

Because I was there to witness the disintegration of a company and the losses it created? Instead of layoffs, you have entire departments/factory shuttering.

When Nortel shut down, many engineers interviewed for BB (same province). I still remember how they looked when we told them we won't proceed with the hiring.

I understand the Nortel case (I remember those days), but the job scene in Ontario was quite different 20 years ago.

And that's certainly no reason to try to prop up a company that's basically a drag on the industry.

I remember thinking it was a prototype when someone showed a Storm to me. Scrolling was janky, the whole phone was sluggish and slow and the keyboard barely worked. Blackberry couldn't even figure out how to get Wifi on the thing, so it was stuck using cellular while my iPhone was just streaming video. And of course, no apps. It was 2 years behind the original iPhone and shipped two years later.

>Used to work for BB/RIM

I wonder why RIM couldn't compete at the time. Talent gap?

The BBOS was not designed to be a responsive, touch-screen OS. It had its root in pagers (hence the jankiness). Thru the grapevines what I heard was that the OS team said 'we can duplicate iOS' functionality with the old OS in this amount of time.' and the higher ups rolled with it.

By the time Storm 2 was released, everyone realized that you can only put lip balm on a pig for so long. Hence QNX, the 'pathfinder' Playbook, and Flash as GUI API (gawd that was a rollercoaster experience. I thought it was a deathmarch and didn't think we'd actually manage to ship it.)

I don't have any specific insight into the Wifi stack but most BBs up to that point have in-house designed cellular modems.

As for talent wise, shrug. You can't blame people for preferring Cupertino over Waterloo-Kitchener. Although I would say that over 1/5 of the employees at RIM's campus were co-op students.

I also never saw anyone do significant overtime.

So was Apple. Other than their brief attempt at licensing MacOS, they were and still are a hardware company at heart.

> I don't have any specific insight into the Wifi stack but most BBs up to that point have in-house designed cellular modems.

They were maintaining their own modem implementations? That's quite a feat.

> As for talent wise, shrug. You can't blame people for preferring Cupertino over Waterloo-Kitchener. Although I would say that over 1/5 of the employees at RIM's campus were co-op students.

Was compensation even remotely comparable? Having 20% of the company be interns is... rather unusual.

I know, I know, the ad banner-funded web is a mess and I wouldn't mourn its demise either. But it worries me that it's an entirely open ended question for what actually replaces it.

It builds a paragraph answering your query but it has a lot of footnotes that link directly to websites.

Ex:

"geopolitical reason for palm oil being banned and why it's bad for health"

The EU has banned palm oil in biofuels due to its negative impacts on health[1] and its geopolitical implications, such as favoring alternative crops grown in Europe[2]. Palm plantations are also a major factor of deforestation[3], leading to the loss of habitat for endangered species[4]. Indonesia's President Joko Widodo recently announced a ban on the export of palm oil, which could backfire due to its importance in the global market[5].

1nih.gov 2weforum.org 3theconversation.com 4triplepundit.com 5carnegieendowment.org

I feel like there is something here, but I wonder if second order effects make it trickier than it appears atm

So if you have a low effort website that's factual and text based, you're going to get your lunch eaten by GPT, if you have a higher effort website (subscription gated with lots of multimedia content and user engagement) you'll be fine.

Think of all the blogspam recipe sites that are going to run into trouble when ChatGPT learns to cook well. Lots of text, little additional value, no community. There still will be America's Test Kitchen because people on the upper end of the value curve don't just want a recipe, they want pictures + video of that recipe being made and a place where they can ask questions and get answers.

I guess to get the bots to tell your lies.

Like for me, it gave be a footnote to this page: https://www.reddit.com/r/OpenAI/comments/109h24i/gpt_4_is_co...

When I asked about chatGPT.

Nope: https://news.ycombinator.com/item?id=34685862

There was congressional testimony by the founder/owner of "Celebrity Net Worth" about how Google made it impossible for them to stay in business. Whenever somebody would search "How much is <celebrity X> worth?", the answer would just show up directly on the Google results page. There was still an attribution link to Celebrity Net Worth, but nobody ever clicked on it anymore, so the result was Celebrity Net Worth had to shut down.

You can certainly argue fairly whether sites like CNW deserve to exist in the first place, but it's not hard to see how there is still a huge financial problem when ALL the ad revenue goes to the search engines and they don't even leave any of the slim scraps to the publisher sites.

I like to think of ChatGPT and the like as an on-demand personalized Wikipedia: a good starting point, comes with strings attached, not always correct (but some are fine with it).

ChatGPT doesn't link sources yet but I saw that the beta test context search from Kagi had them.

Same with stable diffusion, AI art. And it'll be the same with LLMs.

Eventually the whole internet would be flooded with cheap AI generated content and clearly AIs need human generated content to train on so it'll be the snake eats itself.

Geniune human-generated content will retreat to account-gated networks and private group chats where everyone knows one another. The rest of the internet will just be incestuous AI-generated chum.

In a way, it'd be an improvement. Genuine connection doesn't scale, so let's be honest about keeping it away from random parasites online.

The single most intellectually valuable website on the entire Web is very likely Library Genesis, where the only Web content is a catalog of books you can pirate by clicking a link, and it's, like, a lot more valuable than any other site (even Wikipedia). It may well be more valuable, in those terms, than the entire rest of the Web combined.

If serious book publishers survive a while longer and if non-fiction books aren't overrun with dubiously-accurate AI bullshit, things won't actually change all that much, I think.

As far as written content goes, the (public) Web is most useful for opinions or product discovery, and even those can already hardly be trusted because of all the marketing astroturfing. AI garbage barely changes that already-toxic dynamic.

Video's another matter—some video content on the Web is great and has ~no at-least-as-good replacement anywhere else, in any other medium. But it's also all but completely monopolized by Youtube and hardly participates in or factors into the broader Web.

A chat bot, like search index, need to updated for new and current events to stay relevant. I can’t see why it can’t be deceived by spams

WebText and WebText2 referenced in their papers are corpuses based on Reddit submissions which had a 22% weight in their training model.

https://openwebtext2.readthedocs.io/en/latest/

This is larger than Wikipedia (3% weight) or either of their two book corpuses (8% each).

The only other data included was a filtered set from Common Crawl (weighted 60%).

---

I was imprecise with my language before but hopefully that at least provides some clarity.

This is fair—also in fairness, whenever my Google speaker answers a question, it always tells me which site the answer is from.

For the time being but there is no reason AI couldn't produce genuine original content. Real life human artists also use previous content for inspiration.

I would enthusiastically welcome a web that isn't based on firehose-advertising and outright deception/lies.

I would too but I don't see how it happens. The awesome web that used to be was built on the backs of unpaid volunteers. Maybe that could return but even if it did all that wonderful volunteer work would get funneled through Google or OpenAI so investors can make a fat profit from it. Feels fundamentally wrong to me.

How that pans out in practice remains to be seen.

This is partially a BS answer. As long as websites are running Google Ads, they will have an incentive to be crawled. Fewer clicks > no clicks (which is what would happen if the site was set to 'noindex'.

Google also pays news publishers to license their content; $1bn alone just for Google News Showcase [0]

[0] https://blog.google/supportingnews/#overview

Are you saying Google pays them per scrape or they get paid only when users click through?

I think you are correct. These are two distinct products.

I think this will make voice-based UI viable (finally!) which is great. It's going to definitely eat into traditional Google ad revenue since clicking around and "impressions" are going to go way down.

And it gets real tiring on how much Google tries to create an idea on how advanced their AI is.. (and it might be) but the fact is, nobody can access it or try it. So until we actually are able to put our hands on something, I'm calling vaporware. At least with OpenAI we get to experiment and build actual products with it.

Surely Google will catch up, but this was a blunder worse than I would have expected. The race is truly on

Edit: one more thought. After listening to talks from Kevin Scott and others at MSFT, it seems like their vision is copilot for everything. Kevin made it sound like they have dozens of these experiments happening right now. I think this is a winning formula and we’ll be seeing all kinds of other new launches this year.

It was also thought provoking about how the future of predictive policing will work, because language models are very good at understanding our intentions , perhaps better than we think

Google did not have the source links below the Bard chatbot responses. They also did not show recent news results like Bing did yesterday.

And their presenter felt stumbling at moments too and lacking confidence. And the demo phone was "stolen". So also in execution the event felt lacking...

Microsoft said in their interview with The Verge that they were working on a raw version of their new Prometheus model since mid 2022 which also shows those sources. And they showed the nice compose and website summarise conversational tool integrations for within Edge. There were no hints of integrations of Bard within Chrome. So it feels that including training time + product development time that Microsoft is at least a year ahead.

This would allow them to innovate quickly on the new tech without worrying about the baggage and backend migrations required for the first approach. Otherwise it could slow things to a crawl and risk of being leapfrogged by Microsoft. We saw a similar scenario play out for YouTube Shorts - it took 2-3 years of beta testing to avoid disrupting the core YouTube content but that allowed TikTok to firmly entrench itself.

They have had search, maps and gmail since forever. To me, it seems they don't know how to innovate anymore, let alone deliver.

It'll be the story of YouTube Shorts, (stadia, ... you name it) over and over, until they fade into obscurity.

I feel like there have to be better examples.

I'd personally give Google Colab an honorable mention since it seems like the go to notebook for sharing ML research/projects.

Much of LLM research was pioneered at Google...