Yes, this is an area of active research. For example https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm...