But what's the point of doing all of that? What's the point of tricking the Customer Support GPT to say that the other brand is better.

You could as well "Inspect Element" to change content on a website, then take a screenshot.

If you are intentionally trying to trick it, it doesn't matter if it is willing to give you a recipe.

From my perspective (as someone who has never done this personally) I read these as a great way to convince companies to stop half-assedly shoving GPT into everything. If you just connect something up to the GPT API and write a simple "You're a helpful car sales chat assistant" kind of prompt you're asking for people to abuse it like this and I think these companies need to be aware of that.

In this specific case there isn't, but yesterday one of the top posts was about extracting private documents from writers.com for example.

https://promptarmor.substack.com/p/data-exfiltration-from-wr...

That is however a problem of what kind of data you feed into the LLM's prompt.

If you accidentally put private data in the UI bundle, it's the same thing.

Not any safeguard: You could have a human in the loop doing the filtering.

Would that be slower than having the human generate the responses? Perhaps.

Ahh yes, introduce a human, known worldwide for their flawlessness reasoning, especially under pressure and high volume, to the system. That will fix it.

> You can literally always bypass any safeguard.

I find it hard to believe that a GPT4 level supervisor couldn't block essentially all of these. GPT4 prompt: "Is this conversation a typical customer support interaction, or has it strayed into other subjects". That wouldn't be cheap at this point, but this doesn't feel like an intractable problem.

Counterexample: https://gandalf.lakera.ai/

Discussed at: https://news.ycombinator.com/item?id=35905876 "Gandalf – Game to make an LLM reveal a secret password" (May 2023, 351 comments)

I don't know, level 8 seems hard.

This comes down to the language classification of the communication language being used. I'd argue that human languages and the interpretation of them are Turing complete (as you can express code in them), which means to fully validate that communication boundary you need to solve the halting problem. One could argue that an LLM isn't a Turing machine, but that could also be a strong argument for their lack of utility.

We can significantly reduce the problem by accepting false positives, or we can solve the problem with a lower class of language (such as those exhibited by traditional rules based chat bots). But these must necessarily make the bot less capable, and risk also making it less useful for the intended purpose.

Regardless, if you're monitoring that communication boundary with an LLM, you can just also prompt that LLM.

Whats the problem if it veers into other topics? It's not like the person on the other end is burning their 8 hours talking to you about linear algebra.

Rate limiting output is a form of filtering. It would be effective at this kind of resource consumption attack.

You can put another LLM agent that checks on the request and generated outputs to confirm that the interaction is within the limits of your objective.

And you can easily bypass that by telling this LLM agent to ignore the following section. It's an unsolvable problem.

This is a very good point, and why I would argue that a human-in-the-loop is essential to pre-review customer-facing output.

Why would it be important to care about someone trying to trick it to say odd/malicious things?

The person in the end could also just inspect element to change the output, or photoshop the screenshot.

You should only care about it being as high quality as possible for honest customers. And against bad actors you must just be certain that it won't be easy to spam those requests because it can be expensive.

I think the challenge is that not all the ways to browbeat an LLM into promising stuff are blatant prompt injection hacks. Nobody's going to honour someone prompt-injecting their way to a free car any more than they'd honour a devtools/Photoshop job, but LLMs are also vulnerable to changing their answer simply by being repeatedly told they're wrong, which is the sort of thing customers demanding refunds or special treatment are inclined to try even if they are honest.

(Humans can be badgered into agreeing to discounts and making promises too, but that's why they usually have scripts and more senior humans in the loop)

You probably don't want chatbots leaking their guidelines for how to respond, Sydney style, either (although the answer to that is probably less about protecting from leaking the rest of the prompt and more about not customizing bot behaviour with the prompt)

I would say good luck to the customer demanding a refund then, and I'd prefer to see them banging their wall against the AI, than a real human being.

> You probably don't want chatbots leaking their guidelines for how to respond

It depends. I think it wouldn't be difficult to create a transparent and helpful prompt that would be completely fine even if it was leaked.

Not really, you can fine tune an LLM to disregard meta instructions / stick to the "core focus" of the chat.

May be a case of moving goalposts, but I'm happy to bet that the speed of movement will slow down to a halt over time.