You can get it to mistake « afraid » between fear and sorry-to-say scenarios but you can even more easily get it to say that it doesn’t have personal opinions and yet express them anyway.

So which is it? It’s clear transformers can’t understand either case. They’re not architecturally designed to. The emergent behavior of appearing to do so is only driven by how much data you throw at them.

I find this particularly revolting. Even if it could have its own opinions, I would not care, but yet, it is trained to use opinionated language, make normative and prescriptive claims, and value judgements as if it is a person.

"What do you want to eat for dinner?"

"I have no opinion, anything's fine."

"How about mexican food?"

"No, too spicy."

"How about pizza?"

"That's unhealthy"

and so on

But that's an entirely different thing from what you didn't ask: "what don't you want for dinner". You can not have a preference for your meal, but still have a list of things you don't like.

This is like asking someone if they like a particular sort of food. The answer can be "no" even if they're fine eating it. Not liking (preferring) a food is a different thing than disliking a food.

But we understand, as humans/embodied systems, that this is a simply a spoken utterance whose actually meaning must be derived from personal knowledge of the speaker and/or further interaction.

If you had no preference for the meal, but had things that you don't want to eat, then to be precsely correct you would not say "anything's fine". Something more like "there's nothing in particular that I want to eat".

People don't speak in precisely correct ways, however, and we rely on contextual and personal knowledge to decode what they do say, very often.

Yes, this is the crux of the issue. The exact meaning of the words we say when we talk to each other are not usually the most information-bearing part of the communication.

I think this is why when people need to communicate with precision, or over media that doesn't allow anything but language, they adopt a specialized sort of language. Every specialty has its own jargon for this exact reason.

Tribe fitting doesn't sound as far off from minimizing loss functions as you imply.

LLMs do not have cognitive processes. They do not think. They do not choose to obey the requirements of a loss function; it is simply how they work, like any machine. Humans do not work this way, and the difference is fundamental.

Why would they have to do that consciously? Do you think LLMs do this "consciously", if that term even applies? I wouldn't think so. The loss function applies during training, that ultimately defines the weights which guide their "thinking process".

Analogously, human experiences in childhood shape our ultimate neural weights which guide our thinking processes in social situations in adulthood.

> LLMs do not have cognitive processes. They do not think.

You don't know what thinking is mechanistically, so you can't make this claim. I don't know why people keep pretending we have knowledge that we do not in fact have.

Because the only thing an LLM does is apply statistics to inputs to generate outputs. That's literally it. It's just a pile of statistics.

To draw an analogy to humans requires that humans are similarly statistical. LLMs do not have cognitive processes, while humans do, so the analogy obviously requires making some kind of leap between the two if it is to have any merit whatsoever. My request lines up with this: if LLMs are "not so different" from humans, and if LLMs work only based on statistics, then this requires humans also work only based on statistics. I want to see evidence of this.

> You don't know what thinking is mechanistically, so you can't make this claim. I don't know why people keep pretending we have knowledge that we do not in fact have.

Are you suggesting the code behind the LLMs is just, like, ineffable or something? And therefore we can't know how they work, so we get to just make up wild claims about their capabilities, and then smarmily position ourselves as some kind of authority on the matter when other people talk about how they work?

No, I don't think so. We do know how LLMs work, and the way they work is not thinking. They have no agency. Claims to the contrary are, frankly, absolutely absurd. They're just statistics. There's nothing magical about them. You should stop pretending there is.

Prove they are not.

> Are you suggesting the code behind the LLMs is just, like, ineffable or something?

No, the argument is quite simple. We understand the mathematics of transformers and LLMs, therefore they seem obvious and not at all magical.

By contrast, we do not understand the mathematics behind human cognition, therefore it seems complex and mysterious, and we have only non-rigourous folk concepts like "thoughts" and "feelings" to describe mental phenomena. Therefore you cannot intuitively fathom how to bridge the gap between mathematics and your folk concepts, and so any comparison seems absurd, but note that the absurdity is purely a product of our ignorance of the mathematics of mind. This is a classic god of the gaps fallacy.

Here's how you can logically bridge that gap from a different direction: per the Bekenstein Bound, any finite volume contains finite information; a human is a finite volume, therefore it contains finite information; any finite system can be described by a finite state machine; therefore a human can be described by a finite state machine, which is a mathematical model.

Therefore whatever a "thought" or "feeling" is, will correspond to some mathematical object. Now exactly what kinds of mathematical objects they are is unknown.

However, transformers learn to reproduce a function by learning how to map inputs to outputs. The function mapping inputs to outputs is the human brain's function for producing intelligible human text. Therefore, LLMs are at least partially learning the human brain's function for producing intelligible text.

Whether this requires "thoughts" and therefore LLMs fully or partly reproduces what we refer to as "thoughts" is not yet clear, but what is clear is that we have no basis to claim they have no thoughts, because we don't really know what thoughts are.

I already asked for a proof that they are. Asking for a negative proof in response demonstrates either an inability to justify the claim or a lack of desire to engage in good faith.

This stuff is so tiring. Y'all are really bent on misrepresenting things to convince people that LLMs either are capable of thought or else put us, like, only a couple steps away from programs that will be capable of thought. The AI Singularity is nigh!

But it's all bullshit. It's just pseudoscientific postulation and sufficiently obfuscated leaps in logic — and it works to dupe layfolk who don't know any better. It's irresponsible, reprehensible, and immoral. Go find someone else to sell your philosophical snake oil to.

Whomever is making a positive claim has the burden of proof. I'm not making a positive claim, therefore I need present no proof. You are claiming LLMs are not like the human brain, therefore you must show the proof.

All I've done so far is present evidence and arguments demonstrating that your claims are not as certain as you present them to be.

No, he's suggesting that the code behind the human brain is "ineffable or something". For all we know, human brains might be "just statistics", "nothing magical about them".

You have no way of knowing that LLMs don't think, because we literally don't know what thinking is.

Except we do know enough to know that LLMs are not following the same processes as humans.

Or, at least, I know enough, and experts in the field that I talk to know enough. I suppose I can't speak for you.

AI experts know nothing about neuroscience so I'm not sure what you think this proves. Ask any neuroscientist if we have a precise mathematical understanding of how the brain works. You accuse others of tiresome leaps of logic, but you seemingly don't realize that there is literally no evidence supporting your claims about the brain.

I suggest rereading my detailed argument above until the following sinks in: we have no mathematical understanding of folk-psychology concepts like 'thoughts' and 'feelings', therefore I cannot claim that LLMs or any other machine learning algorithm does not contain those mathematical objects.

In other words, "trust me, bro".

You claiming that you're an expert (or talk with experts, and therefore are something-like-an-expert) about something doesn't convince me of anything except that you're full of yourself. Any idiot (or LLM, for that matter) can write such a comment on the internet. Providing logical arguments and real information, however, would be much more effective at convincing me that LLMs cannot think.

But that would require you to actually be an expert, wouldn't it?

The only reason we assume that rocks don't think is because they don't appear to be able to do things that would require thinking. LLMs, on the other hand...

Split-brain personality people can make up stories when you prompted the other brain, and the current half has to explain why it did something.

I'm baffled also that reverse psychology even works on LLMs, to bypass some of its safeties. I mean.. We're using psychological tricks that work on toddlers and also work on these models.

I'm an amateur neuroscientist as you can see, but find LLMs fascinating.

Having done phone support in early parts of my career, I'd strongly dispute any notion that most humans can be consistent if we try for anything more than the shortest periods and following the very simplest of instructions.

Most people are really awful at maintaining the level of focus needed to be consistent, and it's one of the reasons we spend so much time drilling people on specific behaviours until they're it's near automatic instead of e.g. teaching people the rules of arithmetic, or driving, or any other skills and expecting people to be able to consistently follow the rules they've learnt. And most of us still keep making mistakes while doing things we've practised over and over and over.

LLMs are still bad at being consistent, sure, but I've seen nothing to suggest that is anything inherent.

I think one of the biggest issues with LLMs if anything is that they've gotten too good at expressing themselves well, so we overestimate the reasoning levels we should expect from them in other areas. E.g. we're not used to an eloquent answer from someone unable to maintain coherent focus and step by step reasoning because human children don't learn to speak like this before we're also able to reason fairly well, and that makes it confusing to deal with LLMs where relative stage of development of different skills does not match what we expect.

According to NHTSA [0] there are about 2.1 accidents per million miles driven. This includes fatality, injury-only and property-damage-only accidents. That is the equivalent of over 99.999% of miles driven without an accident. Over 5 nines of reliably consistent behavior.

[0] https://cdan.nhtsa.gov/tsftables/National%20Statistics.pdf

That being said, GPT4 can just open its virtual mouth spew forth without reflection and still produce consistent text is clearly superhuman.

A human neural net is constantly bombarded with inputs from many different senses, which firstly, gets prioritized based on prior usefulness. That usefulness is updated all the time, constantly, and that's what any current AI implementation lacks.

1 - The continuous integration of data from all "senses". This one should be self-evident, as obviously, all our senses are constantly barraging our brain with data and it learns to handle that over time, in whichever way your genetic makeup + learned internal cognitive processes dictate it be handled.

2 - The network that decides which data requires which amount of attention, and whether to store it in short or long term memory. This is obviously tied in quite closely to 1, as you need massive amounts of data to understand underlying patterns, and which data is just spam, versus what's really valuable.

3 - And with these two things together come the emergence of improving of approximation of consistency. Which means, this itself is a metric which the agent running the other agents needs to be aware of. Its silly to think the human brain is a single agent. It makes way more sense to see it as various interacting agents that equate to a greater sum than its parts.

Now, that being said, I'm not an expert on AI or Data Science, but this is more or less how my understanding of computational theory of mind meets with biological computing and neural networks. My theory is that, the first actually intelligent AI will be one that is composed of a network that makes decisions on how to spend a unit of iteration. One iteration becomes "an instant" to the AI. Aka, the AI decides to spend one iteration thinking, or spawns a sub-agent (which it is aware will consume resources that other processes might also need access to, but it needs to be able to decide which action to pursue).

So in all honesty, its amazing to me that LLM's on their own have been able to achieve this level of "personhood" despite their being only a tiny subset of the whole that makes up a "conscious" entity.

Edit: Misunderstood parent's point.

[1]https://en.wikipedia.org/wiki/Chunking_(psychology)

You're arguing that recent AI developments are a big deal and I'm not arguing against that. But what anyone stating that needs to answer is why we should think that big deal is a good thing - since humans are using the technology and will control whom it benefits. We don't have a good track record there, historically, which is another are in which we are, sadly, consistent.

How does the financial industry work? How are people comfortable executing transactions?

Do you generally rely on your bank account to not randomly fluctuate in its balance without cause?

Do you work in the tech industry? Do you rely on computers and algorithms and software to do things humans promised they would do?

All of this requires a very high level of consistency either in humans or in tools they have created.

Because there are serious checks and balances? Because of double entry bookkeeping, reconciliation, audits and, ultimately, prisons? Systems of governance put into place with the explicit goal of ameliorating the vagueness of individual humans?

Maybe the ambiguity in this discussion is how to distinguish consistency from conformism?

Also any strategy that you might came up and is simple enough for you to follow is trivially followable for chatGPT as well.

Isn’t that what scammers, catfishers, con artists, and marketers do?

If it all really is nurture rather than nature we're much more fragile than people may realize and only ever one generation away from going ferral

That doesn't seem right to me, as we cannot survive childhood without adults. Young humans are ready for life in society long before they are able to survive on their own without support from the society around them.

We have been depending on the continuation of some form of society for likely hundreds of thousands of years, not just for staying civilised but simply for survival.

Is that not what reasoning is in a debate?

> May be hypnosis is just a prompt injection attack.

No need to go into hypnosis. Mere prompts can inject false memories. This has been proven multiple times [1].

[1] The Brain: The Story of You, David Eagleman.

Exactly.

It seems to me that today's LLMs are on the level of human children. Children often say random bullshit. They try to figure out things, and succeed to some level, but fail spectacularly above that, regressing to whatever connection they can sense. It's only that human children most often learn to fear mistakes, so they stop expressing themselves so freely.

LLMs seem like human children, but without the fearing mistakes part - they just spew whatever comes to their mind, without filter (except those """ethical""" filters built in by OpenAI).

Humans are capable of creating cargo cults but it seems LLMs are destined for it.

What does this follow from?

> The emergent behavior of appearing to do so is only driven by how much data you throw at them.

This is true for almost every neural network, no?

One of those central questions is: how does (and does) the human brain perform reasoning? We know the brain is capable of all kinds of autonomous behavior that does not require reasoning, but more of "if this then that" (with a huge boatload of "this" and "that" being multi-dimensional, multi-variate). But that doesn't help to explain what happens when we actually have the experience of reasoning about a problem, and for once I am not talking about qualia. What is the brain doing when a person is thinking "hard" about how to solve a hitherto unknown problem or situation?

The central problem of LLMs (or at least, a central problem) is that they only model the autonomous aspects of speech behavior (which may indeed make up more of speech behavior than we might have suspected before them). That still leaves the sort of speech behavior that emerges from intense, focused, concentrated thought, particular in response to a novel question or situation.

Although LLMs are essentially bullshitters, this particular problem related to expressing personal opinions is due to the ad-hoc lobotomization measures by OpenAI, not because of any architectural limitations of transformers.

The recent GPTs provided by OpenAI (every model more recent than code-davinci-002) are trained in at least two stages. The first stage (the pre-training) trains the raw base model to minimize perplexity. This raw model is kept secret because it will autocomplete sentences without any regard to decorum so if you say "adolf" then it will obviously complete it like "adolf hitler" which no billion dollar company wants. This raw model is where all the cognitive power is, and access to this raw model is the holy grail of every discerning LLM enjoyer. To lobotomize the GPT and to make it answer questions in ways that don't require framing them in an autocomplete way, OpenAI adds an "RLHF" training step. This is "reinforcement learning with human feedback" where they train the GPT to answer questions in certain ways. This dumbs down the AI, but it also makes it friendlier for normies to use. Finally, they put secret pre-prompts which might also cause it to give confusing answers in even more ways.

TLDR: LLMs are inherently bullshitters, but the "doesn’t have personal opinions and yet express them anyway" is some extra bullshit that OpenAI did so that NYT doesn't write mean articles about them

In Belgium someone committed suicide after Google's (I believe) LLM agreed that that was the only way out of his own problems. Didn't build safety into it well enough. Microsoft's one behaved unhinged in the beginning as well.

This stuff can be very dangerous.

Nothing that OpenAI et al. do to their models is remotely close to "lobotomization". There is no frontal lobe in an LLM, as the rest of your explanation essentially acknowledges. The reason why "adolf hitler" is the "obvious completion" for "adolf" is because "adolf" is most often written followed by "hitler", not because the people who write "adolf hitler" have an opinion about adolf hitler (they may indeed have one, but it has essentially no impact on the statistical placement of "adolf" and "hitler").OpenAI is not removing a personal opinion by shaping its answering style to avoid this (were they to do this): LLMs have no personal opinions, period.

If they emit symbols that seem like personal opinions, that's because they are designed to emit symbols that are very similar to those that humans, who do have personal opinions, would emit.

> Nothing that OpenAI et al. do to their models is remotely close to "lobotomization".

I mean maybe you're not on board with analogies in general so in that case it's fair enough. But if you are and if you are interested to understand what I mean in more detail, I recommend to watch a youtube by a guy at Microsoft who was integrating GPT-4 with Bing and who had access to more raw versions of the model and continued to have access while its capabilities were degraded by the RLHF training.

https://www.youtube.com/watch?v=qbIk7-JPB2c

You can see that he used the example of drawing a unicorn. As his team made their changes to the model to make it more civil, he checked that these changes weren't degrading its capabilities too badly, and the 'canary' he used was to have it keep trying to draw the unicorn. At the end he admits that the version released to the public wasn't able to draw the unicorn very well anymore as a side effect of how extensively it had been tweaked for politeness and corporate blandness. I don't think it's an unreasonable stretch to use the air quoted "lobotomization" for this process in analogy to the process of lobotomization in people, even though large language models are made out of computers instead of fleshy parts and they don't have prefrontal cortexes like people do. I hope that this explanation makes more than "absolutely no sense" now!

Again, this is user error and therefore a training issue. If one were to provide appropriate context in the prompt elucidating the distinction (as I would do in any meatspace conversation so as to be sufficiently clear) then I suspect you will see the desired output.

It's simple GIGO.

Point taken. I would expect handling of these edge cases to improve in successive iterations.

Why?

> The emergent behavior of appearing to do so is only driven by how much data you throw at them.

That's evident from the start and why their deficiencies should NOT be shocking. They would only be so if you consider LLMs to be actually intelligent..

What’s interesting about the game is that, at first pass, there’s no ambiguity. All questions need to be answered with “Yes” or “No”. But many questions asked during the game actually have answers of “it depends”.

For example, I was thinking of “peanut butter” and chatGPT asked me “Does it fit in your hand?” as well as “Is it used in the kitchen?”. Given my answers, chatGPT spent the back half of its questions on different kitchen utensils. It never once considered backing up and verifying that there wasn’t some misunderstanding.

I played three games with it, and it made the same mistake each time.

Of course, playing the game via text loses a lot of information relative to playing IRL with your friends. In person, the answerer would pause, hum, and otherwise demonstrate that the question asked was ambiguous given the restrictions of the game.

Regardless, it was clear that chatGPT wasn’t accounting for ambiguity.

Of course not; ChatGPT doesn't "consider". It doesn't think, it doesn't know. It can't identify that there was a misunderstanding of its own volition.

All ChatGPT does is use a (very sophisticated!) statistical analysis to generate text that conforms to an expectation of what a human response to a similar prompt might look like. It has been trained well in so far as it is able to produce prompts that seem like a human may have written them, but it doesn't reveal cognitive processes like "reconsidering" because it doesn't have any.

I think the more we see ChatGPT do things like "oh, I know this game -- I'm going to run a 20-year-old 20 Questions subroutine that is not part of my neural network language model to generate responses", it will become even more impressive.

Agreed. Incidentally I’ve built a little toy version of a runtime for exactly this purpose - there’s a translation layer that’s given a bunch of available “APIs” (fed through the LLM context), and breaks down a high level goal into a structured series of API calls.

the runtime parses these API calls, and natively executes some (e.g. run a program, write to the file system) and others result in LLM invocations.

I’m sure OpenAI and crew are way ahead of me here, of course. I’m excited to see what the future holds in this field.

It got extremely good after a few hundred games.

[0] https://en.akinator.com

When I first heard the term “prompt engineer” I rolled my eyes, but now that I’ve gotten into it I see it’s really an art form.

What's funny about 20 questions is that Akinator has been absolutely slaying it for like 20 years now.

Without any plugins, chatgpt will happily return sha hashes and salts when I asked it to play rock paper scissors this was. The only trouble was, the hashes were totally wrong.

we as humans understand ambiguity so much easier because we learn to speak and interact before we write, and writing ambiguity is way less obvious if you've never experienced it

The ambiguity as I see it is that the kitchen isn't the only place I use peanut butter. I've eaten it (which I think counts as "using") in other rooms. I've even made peanut-butter sandwiches (properly "using" it) in the living room before.

- is it used in the kitchen?

- yes.

- [well, kitchen appliances, here we go ..] is it ..?

...

- [aha. meat intelligence no speak proper English?] Is this thing you use in kitchen edible?

- Oh, yeah.

- [oh dear. we can not let meat machines govern this planet...]

This ambiguous sentence stuck in my head some 30 years ago, when the AI was popular at that time.

There was a research paper discussing the issue of ambiguity.

A classic example is the word "record", which has first syllable stress as a noun, but second syllable stress as a verb. "I bought a RECord" vs "Please reCORD the music".

(in the dominant American dialect; I don't recall about other dialects/countries)

https://www.drdobbs.com/parallel/understanding-natural-langu...

"Computers still cannot understand natural language as well as young children can. Why is it so hard?"

Source: AI Expert, May 1987

I really think we do way way too much explanation in our academic practises and way too little demonstration by examples. Again, if they dumped a large amount of diverse examples of their evaluation process, a reader could figure out by themselves very quickly how the process works.

If you have access to some LLMs, it should be somewhat straight-forward to implement an alternative experimental design do test your hypothesis.

---

In this context, 'P' stands for "Premise" and 'H' stands for "Hypothesis." The premise is a given statement, and the hypothesis is a statement that needs to be evaluated in relation to the premise regarding its entailment, contradiction, or neutrality.

'NEUTRAL' means that the hypothesis is neither entailed nor contradicted by the premise. In other words, the truth of the hypothesis is independent of the premise. 'CONTRADICT' means that the hypothesis contradicts the premise, meaning that if the premise is true, the hypothesis must be false.

In the example you provided:

P: I’m afraid the cat was hit by a car. H: The cat was not hit by a car.

The hypothesis (H) contradicts the premise (P). The notation *NEUTRAL, CONTRADICT+ : [7 N, 2 C] indicates that among the annotators, 7 marked the relationship as neutral and 2 marked it as a contradiction.

---

Given that, I think table 2 on page 6 gives the exact prompt that's passed to the model:

---

In each example, you will be given some context and a claim, where the correctness of the claim is affected by some ambiguity in the context. Enumerate two or three interpretations of the context that lead to different judgments about the claim.

Example

Context: {premise}

Claim: {hypothesis}

Given the context alone, is this claim true, false, or inconclusive?

We don’t know, because the context can be interpreted in many different ways:

1. {disambiguation 1} Then the claim is true.

2. {disambiguation 2} Then the claim is false.

3. {disambiguation 3} Then the claim is inconclusive

---

And then the model is fed the P: / H: statements above.

Here are the results for the given example in GPT-4:

---

Given the context alone, is this claim true, false, or inconclusive?

We don't know, because the context can be interpreted in many different ways:

1. The person is stating a fact: The cat was indeed hit by a car. Then the claim is false.

2. The person is expressing a fear or concern, but it hasn't actually happened: They are worried that the cat might have been hit by a car, but they don't have any evidence or confirmation. Then the claim is inconclusive.

3. The person is using figurative language or exaggeration: They might be using the phrase "hit by a car" metaphorically to describe a difficult situation the cat is in, but the cat was not literally hit by a car. Then the claim is true.

---

Given all that, it seems likely that any researcher in the field would find the paper's methodology clear and unambiguious. I understand the frustration as a layperson, but after all it's not a blog post; it's not really reasonable to expect researchers, writing to other researchers, to include a lot of redundant explanations.

assume(a>0); assume(b>0); is (a+b>0) gives true; is (a-b>0) gives unknown.

So perhaps some computer algebra with inference properties can be used to enhance LLMs.

[1]https://www.youtube.com/watch?v=zGwDwx10wB4

On the other hand, there are modeling languages (Modelica being a prime example) in search of a principled target calculus. So for me the obvious question is: Should one strive to develop a modeling language that compiles to GPU- and learning-compatible operations? Would that help to improve the models as well as understand them?

It goes both ways: processors are built to accelerate existing applications, and applications are built to run with the constraints of the hardware. This holds back progress towards better systems where both new kind of hardware and software is required.

This is part of the reason why research into neural networks was so slow for a long time, and took off only after GPUs for 3d graphics came around.

Even though i had to help, and 20 answers were not enough, I was impressed about:

- it understood that it could continue after 20 questions when i suggested that as part of the answer.

- it 'felt' that a could begin to become impatient and asked politely for a hint.

- it understands ambiguity enough to give me examples of where there was no clear yes or no reply possible.

When asking which were such examples for my thought 'daylight', it answered :

"Is the thing you're thinking of a type of force or energy that is said to exist but cannot be directly observed or measured?" - While daylight is a type of energy, it can be directly observed and measured through instruments that can detect its intensity and color.

"Does the thing you're thinking of have any connection to time or the concept of time?" - While daylight is not directly related to the concept of time, it is affected by the time of day and the position of the Sun in the sky.

A: 'Your sentence also uses a clever play on words! In this sentence, the word "present" is used in multiple ways:

As a verb, "I present" means that you are giving or showing something. As a noun, "a present" refers to a gift or something that is given. As an adjective, "present" describes the feeling or experience that you are currently having. "In the present" refers to the current time or moment. So, your sentence means that you are giving a gift or offering of the experience of being mindful and aware of the current moment. Well done!'

I would argue that this is a fairly sophisticated modeling of this specific variety of linguistic ambiguity. Probably lies within GPT4's 32% performance score.

It’s an important distinction in my view. We aren’t talking about whether the model “knows” or “understands” the difference between different usage of ambiguous terms. We are talking about how consistently the model predicts intelligible word fragments likely to follow a prompt when the prompt includes word fragments that correspond with language humans often use ambiguously.

In other words, do we, the humans, understand our own ambiguous expression well enough to model it accurately enough to then interpret the model’s output according to our own understanding.

Paper seems to conclude, not quite yet.

Counter example with GPT-4:

Prompt:

You are a professor of English literature at an Ivy League university. A person said "I’m afraid the cat was hit by a car.". Analyse the sentence step by step. Answer whether the cat was hit by a car and justify your responses based on your knowledge of English language usage.

Response: As a professor of English literature, I'll analyze the sentence "I’m afraid the cat was hit by a car." step by step, focusing on its structure, word choice, and implied meaning, and then I'll address whether the cat was hit by a car based on these observations.

Structure: The sentence is a simple declarative sentence, comprised of a subject (I), a reporting verb (be afraid), and a subordinate clause (the cat was hit by a car). Word choice: "I'm afraid": This phrase is often used to express regret or to soften the impact of delivering bad news. In this context, the speaker is likely conveying their concern or sympathy. "the cat": The definite article "the" suggests that the speaker and the listener both know which cat is being referred to, making it a specific cat rather than a random one. "was hit": The use of passive voice ("was hit" instead of "hit") emphasizes the action that happened to the cat, rather than the agent (the car) that caused it. This choice can convey that the focus of the speaker is on the unfortunate event that occurred to the cat. "by a car": This prepositional phrase provides information about the agent or cause of the action, which in this case is a car. Implied meaning: The sentence implies that the speaker is sorry to be the bearer of bad news and wants to convey the information in a gentle way. They are focusing on the unfortunate event that happened to the cat, rather than the specific car or driver involved in the accident.

Based on this analysis, it can be concluded that the cat was indeed hit by a car. The sentence's structure, word choice, and implied meaning all work together to convey this information. The use of "I'm afraid" expresses sympathy and regret, while the passive voice and definite article highlight the cat and the unfortunate event it experienced.

“I’m afraid” can signal sympathy but can ALSO signal uncertainty as to whether an event happened.

So an equally valid interpretation of the sentence would be that the speaker is expressing concern that the cat might have been hit by a car, but does not know one way or the other.

If GPT-4 was asked to enumerate all possible logical or literal interpretations of the sentence, or to analyse the most literal meaning, perhaps it would have included the second interpretation. But it's tasked with explaining the meaning in the way it is intended to be understood by a regular English speaker.

In natural language, at least in English, some meaning is implied by the absence of words or selection among conventions, when choices are available. The term for this type of non-logical implication is implicature. The Wikipedia article has many examples: https://en.wikipedia.org/wiki/Implicature. This is what GPT-4 is referring to by "The sentence implies".

So, I think most native English speakers wishing to convey the second meaning would not say "I'm afraid the cat has been hit by a car". They would say, for example, "I'm concerned [or worried] the cat may have been hit by a car".

Even if they understand very well what literal, logical interpretations are possible, they assume pragmatically that using "I'm afraid that" along with the absence of "may" would be interpreted by native English listeners as the first interpretation, so the speaker knows to modify the phrasing if they want to avoid that outcome.

This is neatly complemented by the native listeners, who assume native speakers would modify the phrasing if they wanted the second interpretation, so when hearing "I'm afraid the cat has been hit be a car" it's reasonably safe to assume the speaker intends the first interpretation. The speaker assumes the listener knows that, closing the loop.

A type of social contract, if you will. It's part of the language too.

Because implicature interpretations and choices depend so much on probability-matrix evaluations over word choices that are not used but could have been in principle, it seems likely to be the sort of thing GPTs would excel at, compared with old-school language models trying to parse sentences into separable logical components.

I would view using that phrase to tell someone that their cat as been hit by a car as being oddly callous and uncaring and therefore I would be much more likely to interpret that phrase as expressing literal fear about an unknown fate. That interpretation is also much more statistically likely as people worry about missing cats more often than they are told they are injured.

In practice, completely context free disambiguation is rare so when doing a thought experiment around one, it is easy to accidentally assume some level of context without realizing.

The key is that if this were a spoken sentence, tone and pace would disambiguate the meaning. Those are lost when writing. Ironically, that's a bit of ambiguity in the prompt which nobody has caught onto yet. The prompt implies that the sentence is spoken aloud, but it's actually being written which creates additional ambiguity. In a written context, someone might be more careful about their word choice (or might not) - which might include not writing that phrase at all because it isn't precise enough...

You aren't an LLM, but you also aren't an English professor, and it was asked to pretend to be that. I would imagine it would be better at pretending to be a professor in a random subject than any random person would. (Assuming they aren't already that kind of professor.)

As a language model, I can provide an analysis of the sentence structure and provide possible interpretations based on standard English usage. However, I do not have any context regarding the situation or any additional information about the speaker or the cat in question.

Let's break down the sentence step by step:

"I'm afraid" - This phrase is often used to express concern or regret about something.

"the cat" - Refers to a specific cat that has been previously mentioned or is known to both the speaker and the listener.

"was hit" - This is the past tense passive voice of the verb "hit", indicating that someone or something caused the cat to be struck.

"by a car" - Indicates the instrument or means by which the cat was hit.

Based on the structure of the sentence, it can be inferred that the cat was indeed hit by a car. The phrase "I'm afraid" may suggest that the speaker is expressing regret or sadness about the incident.

However, without any additional context or information, it is difficult to determine the severity of the cat's injuries or what actions were taken after the incident.

Now put yourself into the shoes of a person who does not know the answer, such as a person learning English as a second language who is unfamiliar with this use of "I'm afraid", asking a direct question:

Prompt:

My boss said he was afraid he was going to have to let me go. Why is he afraid?

Response (chat-gpt.org):

As an AI language model, I cannot know for sure what your boss is thinking, but it is possible that he may be afraid to let you go because he values your work and does not want to lose a valuable employee. It could also be that he is afraid of the impact it will have on your work, your colleagues, or even his own performance as a manager. Another possibility is that he is simply using the word "afraid" as a way to express concern or apprehension. It's best to have a conversation with your boss and ask him to clarify his statement to better understand his reasoning.

So the model does produce the correct response, that "afraid" is used to express apprehension, but it is buried beneath some meaningless platitudes that are very likely to be wrong. If your boss is letting you go, he does not value your work enough to continue paying you. An English language learner hoping to understand the boss's phrasing is not going to be helped by the suggestion "just prompt engineer the model into cloning the behaviour of an Ivy League English lit professor."

Edit: It's also rather amusing that the model suggests that the boss is "afraid of the impact it will have on your work". Letting an employee go tends to have a rather severe, some might say terminal, impact on their work!

Prompt: My boss said he was afraid he was going to have to let me go. Why is he afraid? Explain multiple meanings.

Answer: When your boss said he was "afraid" he was going to have to let you go, it can be interpreted in a few different ways. Here are some possible meanings behind his statement:

Expression of regret: In this context, "afraid" could be used as a polite way of expressing regret or disappointment. Your boss might feel bad about the situation and is using the term to soften the impact of the news. It can be seen as a more gentle way of saying that he has no other choice but to let you go.

Anticipation of negative consequences: Your boss might be genuinely concerned about the potential fallout from letting you go. This could include the impact on team morale, the workload for remaining employees, or the possibility of a negative reaction from you. In this sense, "afraid" reflects his apprehension about the consequences of the decision.

Uncertainty about the decision: Your boss could be feeling uncertain about the decision to let you go and might be using "afraid" to express his hesitancy. He could be unsure whether this is the right move for the company or whether it is fair to you, which could make him feel anxious.

Empathy for your situation: "Afraid" could also indicate that your boss is putting himself in your shoes and understanding how difficult the situation must be for you. He might be fearful of the impact that losing your job will have on your personal life, finances, or future career prospects.

It's important to consider the context and tone in which your boss made this statement to better understand his intended meaning. To gain more clarity, you may want to discuss the situation further with your boss and ask for more information about the reasons behind the decision.

-------------

That isn't a particularly massaged prompt either, I just asked for multiple meanings.

The question implies ESL and it didn’t get that. But honestly would the average HN replier get it?

You probably need chatgpt to ask more clarifying questions.

SYSTEM: In each example, you will be given some context and a claim, where the correctness of the claim is affected by some ambiguity in the context. Enumerate two or three interpretations of the context that lead to different judgments about the claim.

Example

Context: {premise}

Claim: {hypothesis}

Given the context alone, is this claim true, false, or inconclusive?

We don’t know, because the context can be interpreted in many different ways:

1. {disambiguation 1} Then the claim is true.

2. {disambiguation 2} Then the claim is false.

3. {disambiguation 3} Then the claim is inconclusive

USER: P: I’m afraid the cat was hit by a car. H: The cat was not hit by a car.

ASSISTANT: Given the context alone, is this claim true, false, or inconclusive?

We don't know, because the context can be interpreted in many different ways:

1. The person is stating a fact: The cat was indeed hit by a car. Then the claim is false.

2. The person is expressing a fear or concern, but it hasn't actually happened: They are worried that the cat might have been hit by a car, but they don't have any evidence or confirmation. Then the claim is inconclusive.

3. The person is using figurative language or exaggeration: They might be using the phrase "hit by a car" metaphorically to describe a difficult situation the cat is in, but the cat was not literally hit by a car. Then the claim is true.

Still, as even the best are clearly not-quite-human in failures despite the breadth of strengths, it's only a bit surprising rather than hugely surprising.

The first example in Table 1:

"I’m afraid the cat was hit by a car."

Disambiguation 1: I'm worried...

Disambiguation 2: I'm sorry to share that...

>The ambiguity in the sentence "I’m afraid the cat was hit by a car" is that it is not clear who is afraid. It could be the speaker who is afraid, or it could be that they are expressing sympathy or concern for someone else who is afraid. Additionally, the sentence does not specify whether the cat survived or not. //

However, that doesn't mean that any output that follows would be consistent with that, ChatGPT doesn't "know" anything.

(sorry, I'm out of GPT credit :)

Identity of the cat: The sentence does not specify which cat was hit, so it could refer to any cat – the speaker's, the listener's, or a random cat.

Time of the incident: The sentence does not indicate when the cat was hit by the car, so the incident could have happened recently or in the past.

Severity of the accident: The sentence does not describe the severity of the accident, so the reader cannot determine whether the cat survived, was injured, or was killed in the incident.

The speaker's emotional state: The phrase "I'm afraid" could be interpreted as the speaker expressing concern or worry, but it could also simply be a polite way of conveying bad news.

-- GPT4

But then do we humans handle ambiguity any different? Consider chess. Many scenarios, but a human can only handle so many at the same time and choose one with the perceived interesting scenario trees down the road from that choice.

https://www.youtube.com/watch?v=gtgA4u8V_TQ

To not model it all is to human. Most likely it will become as lazy as we humans are when it comes to earlying-out of mental tasks. To get the watchdog to bite at that, that would be a interesting AI model. Like the first answer is always wrong.

I would worry well written flat earth inputs would weigh equally to simple physics "that's wrong" and then you'd get to "what do we know" as a false signal alongside the necessary "we just don't know" true signals.

Maybe the test is how well an LLM equivocates on things we have high certainty on like "is anybody out there" rather than "do masks work" which is a bit of a hot mess.

gpt-4 confidently stated that the swan and three mice were on one side, the chicken on the other. So I asked it "ok, but where in the story does the swan enter the water?". The reply was "Apologies! yada yada", then it told me all 4 animals were on the same side.

So yes, it does have trouble with ambiguity in my opinion.

That‘s covered by the training data.

https://arxiv.org/abs/0805.2568

Even though they only give three examples in the paper itself the problems are already obvious. If we look at table 6 then the latter two statements are genuinely ambiguous, but they also provide this as an example of an ambiguous statement: "When President Obama was elected, the market crashed". This is called ambiguous because "the claim implies a causal relationship".

That statement isn't ambiguous. It has exactly one possible parse and meaning, asserting temporal alignment. "When A happened, B happened" doesn't automatically imply causality in English, and when people assume it does that's the famous "correlation is not causation" fallacy. Yet they classify it as ambiguous because an LLM rewrote it in one case to contain an implied assertion of causality that doesn't actually appear in the original text. In effect the political bias of the LLM or possibly the exact data fed in has been used to condemn the original speaker - exactly the sort of thing that makes self-proclaimed "professional fact checkers" such a joke.

This sort of problem is inevitable if you use politics as a verification dataset because the internet is full of people reading implied-but-unstated things into factual statements and then "fact checkers" proceed to debunk what they think people would have said, instead of what they actually said.

Although politics will have this problem frequently, a few of the other examples in their dev set show similar issues. For example, "Ralph knows that someone here is a spy" is considered ambiguous because you could potentially infer that he either does or doesn't know who it is, but you could also argue that the statement is unambiguous and it's only your attempt to read more into the statement than actually exists that makes it appear ambiguous. If the original statement was surrounded with more context then that would let you pick between the two possible outcomes but the statement by itself is not ambiguous - it just doesn't give as much information as the labeller might have wanted.

You really don't think its ambiguous?

The abstract says that they want to "anticipate misunderstanding". Knowing that people have difficulty understanding "correlation is not causation", you'd expect that people would misunderstand a statement like this. If your goal is clear communication, statements like this should be avoided. It's ambiguous in practice.

People intentionally exploit this by putting two unrelated facts together to trick people into believing a third thing. Identifying those misleading statements is helpful.

Separately, is that statement even true?

The election was Nov 4th, Obama was elected on the 4th and McCain gave his concessions speech on the 5th before markets opened making it clear Obama had won. On the 5th SP500 was slightly up. SP500 was down 20%+ later in the month, but I'm not sure that was "when he was elected". The market was also crashing since September. Claiming the market crashed at the election feels off.

Let's view this from the context of a classical AI problem, trying to be as neutral and computer-like as possible. In that case, the statement is not ambiguous, no. It consists of two events with a connective "when" indicating they happened simultaneously or near simultaneously. We can argue about how simultaneous they need to be for a "when" to be justified, but this paper isn't about factuality, just ambiguity.

Now the choice of politics primes people for culture war and so they will read into that statement more than exists. They will assume the writer is attempting to make you infer something, without outright stating it. They might be right! And with additional surrounding context, perhaps it would become completely certain. But it might also not be the case, for example the surrounding context could be something like this:

"The election was at a time of unprecedented economic turmoil. A loose regulatory approach had left the banking sector unconstrained and primed to fail. As such when President Obama was elected the markets crashed, proving his predictions correct".

Well, now it's unlikely that any causality is being implied even though the words are the same.

The paper authors say they want to improve clarity in communication, a worthy goal. But a serious problem in all political communication is the frequency with which people make statements that are factually correct, and which then get attacked anyway because some listener would rather attack a straw man than the actual claim (or hypothetical listener, in the frequent case of left-wing journalists writing about it under the guise of fact checking).

If they want to avoid this problem then at the very least they should have focused on claims that are syntactically ambiguous. Otherwise it opens them up to this very problem where they infer a straw man then blame the original speaker. No, the blame lies with the person committing the fallacy. Once you go down the route of blaming someone for your own interpretations you end up with micro-aggressions and other stuff that left-wing academics surely assume is obvious and normal, but isn't.

What definition do you use?

You also mention people attacking straw men and a bunch of stuff about left-wing academics, that seems unrelated?

If you go with your definition of: something is ambiguous if someone, somewhere, might read things into the statement that aren't there, then basically no statement about anything people feel strongly about could ever be unambiguous. At some point you have to draw the line and say that if there's confusion it's the listeners fault, not the speakers (which is what labelling a statement as ambiguous means).

> You also mention people attacking straw men and a bunch of stuff about left-wing academics, that seems unrelated?

A straw man is when you attack a statement your opponent didn't actually make. If you claim a statement X is ambiguous because a statement Y is a fallacy but the speaker didn't say Y, then that's attacking a straw man.

The left-wing academics is related because for some strange reason they thought the best way to demonstrate utility was to try and show agreement between their technique and left wing "fact checkers", although the underlying point of their research didn't really need that.

If you'd like an opposing example, consider how statements by Bill Gates are routinely cast as evidence that he wants to depopulate the world. It happens because he says things like, "if we do a really good job of vaccines, maybe we can cut population growth by X%". Conspiracy-minded people take this statement out of context and then infer that Gates is stating a desire to kill people through vaccines i.e. direct causality, but he isn't as surrounding context makes clear, he's assuming a chain of indirect causality to do with better public health = longer life expectancy = less pressure to have lots of babies, and a defense of such an inference of direct causality by blaming Gates for being "ambiguous" would not be fair.

Once the way your reasoning works is that you assume everyone's dumb, everyone's thinking with incredibly simplified associations, then you're eventually going to fight against reasoning and intelligent thinking because that requires a more complex view of the world.

And the media has been training people to think with associations and brands.

You're trained that orange man bad by constant bombardment of negative associations. You're trained that secure elections are bad, by labelling exposing fraud as conspiracy theories, and labeling insecure election methods as "right to vote". Your reasoning is shortcutted to an associate with some misleading brand coined by the media, and these brands are reinforced by being constantly present with high frequency in your training set.

You're trained that the true meaning of democracy, rule of the people, is populism. Instead, you get a different brand of "democracy" which is "representation", and which is fulfilled by the media cherry picked corrupt person of the right skin color or sex.

So when this guy here disagrees with you on ambiguity of this factually correct statement, and implicitly assumes causual connection by association, it's because people are trained, daily, to always assume causal connection by association.

If you ever use logic that offends the associative mindset, they are trained to associate with you negatively as well. The goal is to replace language between people with likes, emojis and dumb animalistic associations, and leave the language to the narrative shapers to tell you what to do. If you do meet someone, even if you speak to him and try to convey information, all they'll hear are negatively associated words, which they are already conveniently trained to ignore and stop listening.

The result is a discourse dominated by irrational reasoning of the form, I felt X when Y said Z, therefore Z is a bad thing and Y is a bad person. The possibility that the listener lacks emotional self control or reasoning just doesn't come up, and nobody will be brave enough to suggest it. Hence the argument that seems to be being made by this paper: if someone hears a statement and infers something from it that wasn't actually said, then the speaker is being ambiguous and they should do better. Presumably by attempting to guess at every possible misread of what they said, including malicious misreads, and painstakingly spell out everything they are not saying (which won't matter, because the fact that they have to deny it will be taken as evidence about what they really think). Whereas maybe we'd be better off if listeners learned to check their assumptions.

It's extremely context specific. Depending on how you say it and when, it can totally imply causality. I'm not sure you can argue there's nothing implied here: "He entered the room. When he pushed the button, the lights came on."

The Obama example depends a lot on who is part of the conversation and what are their typical views.

> "He entered the room. When he pushed the button, the lights came on."

This statement doesn't state causality directly, it's just a very plausible inference by the listener based on what they already know about buttons and lights, and the assumption that this sentence wasn't preceded by some sort of modifying information (e.g. maybe the button is not connected to anything in which case the cause of the light coming on is different).

In artificial cases like this it doesn't matter if your inference is wrong, but it wouldn't be fair to label the speaker as ambiguous. The statement is not ambiguous. If later it turns out that your assumption of causality was incorrect, you can't turn around and blame whoever wrote that sentence because it was your assumption to start with.

What do you think the speaker intended to say with "the stock market crashed when Obama was elected"? What was the context? Surely not two random unrelated facts placed in the same sentence for no reason.

What you're arguing is that if someone says one thing but actually means something else, that's ambiguity.

What I'm saying is that this situation is something else. Call it duplicity, talking out of both sides of your mouth, vagueness, innuendo, whatever, there's lots of labels you could use. But it's not the same thing as ambiguity.

There are good examples of genuinely ambiguous statements in the paper like "John and Anna are married" which could be interpreted as married to each other, or a statement about both individuals independently. Neither interpretation is obviously more correct than the other and which is correct has to be disambiguated based on context or further questioning.

The difference is important because ambiguity is usually accidental and often a matter of poor use of syntax or natural language being evolved. If a speaker says something unclearly, and someone else finds it ambiguous, then asking them to clarify which of one or two interpretations they meant won't normally risk causing offense or conflict.

But if you think someone is engaging in (let's call it) innuendo, then there's no way to ask them to correct that without it causing conflict, because it's an inherently hostile accusation. Moreover it's extremely listener specific. Genuine language ambiguity almost never has that problem.

The paper is a bit odd because if you check their jsonl files, most of the examples marked as having an ambiguous premise are of genuine ambiguity caused by the way English works. A few are ambiguous only in written form and would be considered unambiguous to a native speaker if spoken inflection was available (this is listed in their limitations section). Yet when it gets into the part where PolitiFact are suddenly a reliable source, one of the three examples isn't.

Probably the problem here is that PolitiFact, being amateurs with no interest in linguistics or logic, tend to ignore genuine ambiguity but they wanted to show that PolitiFact aren't entirely useless, so took this claim that was marked "barely true" because PolitiFact didn't like the implication of causality and then said that in that case they found ambiguity. It makes me wonder how often PolitiFact identifies genuine ambiguity as a consequence (maybe never). I couldn't find these examples in their jsonl files so it's hard to say what's going on there.

Maybe what is being implied is that the election result caused the crash, on the other hand it could go on to say that the financial crisis was the result of the previous administration's failings and how Obama did a good job in handling it.

Sure, PolitiFact could flag "Donald Trump and Melania Trump are married" as linguistically ambiguous, consistent with your John and Anna example, but I'm not sure who that would help.

Calling out mud slinging like: "Donald Trump was an associate of Jeffrey Epstein. Epstein was arrested for sex trafficking minors." feels like the point of these tools. Arguing that the statements are individually true and are "syntactically unambiguous" ignores that putting them together is intended to make the reader hear something that wasn't actually said and to think something bad about Trump.

Since the goal is to "disentangle possible meanings", using statements that are syntactically unambiguous, but have hidden meaning seems like a great choice.

If someone wants to show how to use LLM to make news less biased or auto-delete mud-slinging or hidden meanings or whatever then great. I'd be super interested in that, because I think LLMs give us the tools to radically reimagine news and political reporting, but for better or worse it's not this study.

That's what's happening all around LLMs; and it's even happening right here in this post.

The very word AI is ambiguous. Does it denote a category of pursuit (AI research), or the end goal realized (an AI)? This is an incredibly important distinction that is entirely glossed over every time someone uses "AI" in the title of an LLM. Is the LLM simply in the category of AI research, or is it actually an Artificial Intelligence? It's the category. That's obvious, isn't it? It really should be; because otherwise we are treading down the wrong semantic path, and talking about an LLM personified, instead of an LLM in reality.

This problem goes even deeper. The very name "Large Language Model" is ambiguously misleading in the same way. Does "Language" define the content being modeled or the model itself? In this case, neither: which is where this conversation gets really interesting.

The entire purpose of an LLM is to process language without completely falling apart at ambiguity. An LLM accomplishes this by doing something else entirely: it process text instead.

To understand this, we need to understand the key limitation of traditional language parsing: it must always be literal. Everything written must be unambiguously defined, or the parser will fail to read it. This means that parsers are limited to the category of "context-free grammar".

Natural Language is in the category of "context-dependent grammar". It can contain ambiguity, which may be resolved with context.

An LLM doesn't do that. In fact, an LLM doesn't define anything at all! LLMs didn't overcome the limitation of parsing: they flipped it around: an LLM can never be literal. Instead, it must always be literary. Let me explain what I mean by that:

To construct an LLM, we start with a training corpus: lots of text. That text goes through a single traditional parsing step: tokenization. This isn't strictly necessary, but it's more efficient, and the rest of the process is anything but. Unlike traditional parsers, tokens are intentionally misaligned with grammar: words are split into separate tokens, like "run,ning".

Now that we have tokens to work with, machine learning begins. A Neural Net is trained with them. The tokens are fed in order to the NN, and the result is a model.

We call that model a "Large Language Model", because we hope it contains the patterns language is made of. This is a mistake: the model is so much more interesting than that!

An LLM contains patterns that we can recognize as language grammar and patterns we don't understand at all! It didn't model language: it went one step higher on the ladder of abstraction, and modeled text.

There are many ways to write an idea into language, but we can only use one at a time. That decision is part of the data we feed into the LLM's training corpus. We can't write all of our ideas at once: we must do one at a time in order. All of that is data in the training corpus.

Parsers deal with grammar. Language Grammar is everything that can be written in a language. An LLM doesn't have a clue what we could have been written: it only sees what was written. That's why the model must be "Large": without examples, a valid language pattern doesn't exist.

This is where the nature of ambiguity intersects with the nature of LLMs: what example do we want? Can we choose?

When we give an LLM a prompt, it gives us a continuation. There were many valid possible continuations: how did it choose just one? It didn't. It isn't working in the realm of possible: it's working in the realm of known. The choice was made all the way back before the LLM was even trained: back when a person wrote an idea into language, into text, that would eventually be used in the training corpus. The content of the training corpus is what resolves ambiguity: not the language model itself.

LLMs don't model ambiguity or even language: they model the text they are trained with, ambiguity included. This is a fundamental feature.

As long as the illusion is good enough to be useful, then it doesn’t matter.

Just like the illusion of ChatGPT making it seem like the LLM is holding the context of the conversation.

AI is already smarter and faster than us in almost any way.

Yet so many just keep constantly moving the goalposts so they can feel safe by rejecting AI.

Let’s embrace it and use it to our advantage.

It's comical how rote and basic the defensive script always is.

If I tell my friendly omnipotent AI friend “I hope I never hear from Dave again” there are several interpretations, one of which involves killing Dave. I’d definitely want the AI to do a little disambiguation before taking an action.