Because human interaction (whether human to human or human to machine) rely heavily on trust. Humans implicitly trust other humans more because we've evolved to have empathy. It makes it easier to predict what will happen with another human because we can model their behavior by putting ourselves in their shoes. That's also why we are so uneasy around those who we cannot empathize with (e.g., socipaths).

While validation is a proxy for this trust, it's not the same because the modeling may be incomplete or we don't have the same evolutionary heuristics to understand aberrant behavior. So we tend to want to interpret the model explicitly.