You're right, I don't think EU freedom of movement is based on ideology. But it was definitely used as an ideological argument by both sides during the Brexit debate.