In the "immutable architecture" style, the only time you see a shell on a production box should be when it's on fire (and even then, only if you need to do some "perimortem" debugging because the problem keeps reoccurring even when you just blow the server's state away with a reboot, and nothing in the "postmortem" logs tell you enough to diagnose or even know what more logging you should add.)
I'd hope an immutable architecture wouldn't break inside the box, since that would indicate it mutated :) And if it did, you'd just blow it away and replace it.
At that level, it's the interactions with different components that can break down, and the tools you use are more about testing configured interactions. Also, bad data. That bit necessarily mutates.
I'm a big fan of developing in a similar environment to production, so that the debugging tools and interaction are familiar long before an emergency. At our company, we develop on Linux and deploy on Linux.
