Giving them back to whom, exactly? Texas voted to join the US, Puerto Rico has repeatedly had independence referenda, and Hawaii is... complicated, but you certainly couldn’t argue that the unelected Hawaiian monarchy was a more legitimate government than the elected US government (also Hawaiian independence polls at around 6%).