The usage of "you" is very ambiguous. The usage of "cold", or "warm", or "right" or "north" is very much as ambiguous as can be.

Ambiguity –or specificity– has nothing to do with why we talk the way we do. Sometimes, ambiguity is the whole point of a sentence.