If I recall correctly, there is a proof or conjecture suggesting that it’s impossible to build an “LLM firewall” capable of protecting against all possible prompts—though I may be misremembering, just search for resources like this [1].

[1] https://arxiv.org/abs/2406.03198