We have no scientific evidence of the contrary. There are some studies pointing to mass migration being (slightly) detrimental to the wages of low-income natives, but that's pretty much it. You will find a lot of opinions though.
So-called "slight" decreases to wages add up to quite a lot of money over whole careers. I don't see any reason to deliberately institute a policy that helps advantage capital over labor, when it is already so very advantaged.
Hmmm I can invest my capital in 30 countries easily with a carefully selected index fund. If I could move across borders freely, it would be hard (if not impossible) to do the same thing. So I'd say it isn't the same footing that applies.
If you could literally sign an order right now to completely remove any control on who could come and work in your state (setting aside criminal records, diseases etc), would you do that?
I think you would probably not, because you know it would be a war zone within weeks, with the poorest and most desperate people from all around the world swarming to your small part of the world, bringing with them huge social issues.
Does that happen now? There are plenty of poor, desperate people all over America, do they "swarm" the more affluent areas? Do we have poor people from rural Pennsylvania coming to Los Angeles and undercutting hard-working Californians?
I would. With background checks for criminal records.
People who have the wherewithal to move across the globe are not "the poorest and most desperate people from all around the world", they are the ones with the means to pay thousands of dollars to mafias because we have no better system to bring them in.
Letting women vote had downsides for certain people too. That didn't make it less right.
As for stability, you need to consider the benefits of having the best people from around the world in the jobs where their comparative advantage is greatest.