There's a strong correlation between genes and memes (you tend to pass your ideas on to your offspring). In absence of anything else to go on, appearance is indicative of ideas held, making rapid intellectual filtering of "us vs. them" easy using physical characteristics. Yeah, it's wrong in so many ways, yet the correlation is enough for humans to develop it as a survival mechanism early on - important and persuasive when odds are "they" are going to kill you for limited food & resources in preservation & procreation of genes & memes.