Even a large model has to behave fairly predictably to be useful; it's not totally random, is it? The same thing applies to humans.

Interpretability can mean several things. Are you familiar with things like this? https://distill.pub/2018/building-blocks/