Yeah, I understand this argument but I think it's bad philosophy. You need to think 4-dimensionally: everyone was a "theoretical" person if you go back far enough and making rules that harm people in the future but not in the present leads to wild temporal inconsistencies.

Perhaps it's clearer to think about a person-slot of a particular type in a particular time. If I go back to say 1930, you were probably just a person-slot, not an actual person.