Sure, and instead, people use apps like Signal or WhatsApp, which are tied to phone numbers, on which the attacker can now register to your phone number thanks to his receiving your SMS...
If you tell Signal not to allow anybody else to re-register from your phone number without your PIN it will enforce this until at least seven days passes without you using Signal.
If you've uninstalled Signal or just never use your phone then yeah, after a week or so this proposed attack "works" and the safety numbers for any ongoing conversations with anybody reset (the attacker doesn't know the long term identity key for your phone so they'll get a new one, thus generating a different safety number), which will be notified to the other participants although since you presumably never use Signal there may not be any such conversations.
