It likely is a genetic mechanism rather than "discrimination." Fact is that in an adverse encounter, the big (tall) fellow is more dangerous. So its wise for smaller persons, and especially women and children to minimize encounters with taller/bigger persons, and strangers also.

So they're doing the right thing.