This is worse because "prompt injection" is a feature, not a bug.
If you want a generic AI to talk to, then whatever you talk it into - such as rules of behavior, or who to trust - someone else will be able to talk it out of. Just like with humans.
Others mention the problem is lack of separation between control/code and data - technically yes, but the reason isn't carelessness. The reason is that code/data separation is an abstraction we use to make computers easier to deal with. In the real world, within the runtime of physics, there is no such separation. Code/data distinction is a fake reality you can only try and enforce, with technical means, and it holds only if the thing inside the box can't reach out.
For an LLM - much like for human mind - the distinction between "code" and "data" is a matter of how LLM/brain feels like interpreting it at any given moment. The distinction between "prompt injection attack" and a useful override is a matter of intent.
If you want a generic AI to talk to, then whatever you talk it into - such as rules of behavior, or who to trust - someone else will be able to talk it out of. Just like with humans.
Others mention the problem is lack of separation between control/code and data - technically yes, but the reason isn't carelessness. The reason is that code/data separation is an abstraction we use to make computers easier to deal with. In the real world, within the runtime of physics, there is no such separation. Code/data distinction is a fake reality you can only try and enforce, with technical means, and it holds only if the thing inside the box can't reach out.
For an LLM - much like for human mind - the distinction between "code" and "data" is a matter of how LLM/brain feels like interpreting it at any given moment. The distinction between "prompt injection attack" and a useful override is a matter of intent.