Also don’t forget emigration. I don’t know the numbers for the US but for example in the EU some countries had more emigrants than immigrants during some years (e.g. Spain in 2010).

(edit: clarification)