One needs to know your wife to know that "hand me a tractor" is wrong. Perhaps she makes model farms as a hobby, perhaps you work in plant rental and that means "pass me the keys to one of the tractors", the possibilities are endless. You need vision, memory, profiling, etc., to even begin to properly contextualise day-to-day conversation.
Hm, I think I disagree with you and GP. The answer is Bayesian inference to both of you isn't it? The prior is going to give very low weight to your domestic partner asking you to "hand them a tractor", despite the fact that it's not impossible that it's the correct words.
