Every action we take affects lives of everyone around the globe, and it significantly affects lives of thousands, when You take into account that all society's visible pressure on your life is built upon small decisions of all people that form this society.
When i buy apple, i affect lives of all that stand ready to provide the apple to me. I affect lives of all that stand ready to provide me peaches, grapes, bananas. I affect the prices of all above and so i affect lives of all interested in buying any of the products.
One of the driving forces of market economy - competition - is because of third parties choosing either me or you, and thus affecting my life positively, and yours, conversely, negatively.
What's the problem with prejudice? We are all prejudiced in million ways. That is the essential of being efficient in achieving our goals. We make decisions based on optimal amount of knowledge and information, not more or less. When You are in a hurry in the morning, in winter, You take warm clothes, rather than checking the temperature prospects for all locations you are going to be this day. Why so much prejudice against light clothes, it just might be a warm day! Do You know exactly why is it going to be cold? No, but you reasonably expect it. That is what a prejudice can be - a reasonable, with amount of knowledge and information you posses, expectation.
So when not to be prejudiced? Well, one certainly ought not to be prejudiced when it's unreasonable. If one could really benefit, long-term, from getting information, he ought to do just that. It's reasonable to know some about things that affect your actions.
One then should not be prejudiced when it comes to other human beings, because it does not cost much to get information, and altough it does not much good to yourself, if you don't care, but it can be very important for people around you if you are not prejudiced against them, but get to know them before passing judgements.
But can we force people to first or second one? Not on my watch - unnecessary force is worse than unreasonable, yet peaceful, actions. The only situation when one MUST not be prejudiced is - when you use force. When you are a policeman, a judge, a tax collector. These people should only treat people different based upon difference in their actions, not any kind of traits that are only correlated.
Now back to the point. When I do nto give You a job, i do not affect your life forcible. Assuming you accept private property rights in their basic form, my business is mine, and as long as i keep it away from hypothetical Mr.X's life, it's only him that affect forcibly my life, when he gets a job I did not want to give him. It's actually a hidden ongoing robbery.
I have to admit it, you're pretty good. Saying racism is an "assessment", comparing it to chosing clothes and then actually accusing them of "robbery"... masterful!