If you supply labor at X skill level, an influx of immigrants at X skill level (but not of Y or Z skill level) will reduce the buying power of your wages.
Taking it to real life - people tend to oppose immigration when they are at X skill level, and support it when they are at Y or Z skill level. It's raw self-interest often dressed up as altruism or principle.
You can see this understanding here whenever H1B is discussed - suddenly supply and demand becomes an obvious economic concept.
And often they're in they're twenties, meaning society didn't have to pay for the expensive first two decades of growing up. Basically a live is 8 decades, 4 decades of which are unproductive, and 4 decades are productive. If you get an immigrant, you get a 6 decades person that's unproductive for only 2 decades -- and some other society paid for those 2 other unproductive decades.
Thank you for backing up my point with some numbers.
In particular see the second link, the age pyramid, showing how there are very few people in the foreign-born population who are younger than 20.
And indeed, when you remove most of the population who are younger than twenty, the medium age of the group overall will increase.