Suppose you're of the view that a mass influx of people into your country that are willing to work for less than the locals are will do harm to the economy. If one country's economy is weaker than yours and is a source of immigration from people seeking greater opportunity (and who'll do anything to get it, including undercutting locals), is it racist to want to stop immigration from that country?
It's definitely protectionist and perhaps xenophobic but I don't think it's racist. (There is a clear distinction between xenophobia and racism - xenophobia is purely about an unjustified dislike for those from other countries while racism is an unjustified dislike for those who look different to you)
You do know that historically (and like recently as well) white people -- specifically British and British-descended white people -- have been all sorts of racist towards other people we now consider white (but didn't always), including but not limited to, people of Italian, Irish, Jewish, and, yes, Polish heritage?
I find that quite hard to believe, as I don't see any visual distinction between Brits, Ashkenazi Jews, Irish (unless ginger), northern Italians and Poles. Southern nations could be distinguished (e.g. southern Italians, Spaniards, Greeks), in that case it could have been racism (e.g. in the US they still consider "Hispanics" a different race, although I don't).