> deciding if they should escalate things to engineers feels like a hard thing for an LLM

Especially since most attempts will have a "under no circumstances should you voluntarily involve a human" in the prompt.