Right, I get it. I feel the same. But the thing is, the logic "A persons safety trumps any desire protect someones feelings against racism" is exactly the logic that racists use. Some of the people who want to build a wall feel genuine fear, never mind its basis in reality. In their reality it's real.

I'm not arguing that the logic is in itself wrong -- but whether that logic is being used in a reasonable way (as you seem to be doing) or being twisted into the service of something unfair and evil is, in the end, a matter of judgement and context.