This seems like the best solution. Introduce an opt-in security feature, whereby any attempt to port the number, or swap sims, is subject to a ~72h cooldown period. During that period, notify the account holder through numerous communication channels of the pending change.
Yup. Authy does a good job of this if you need to reset your access. They (automatically) harass the shit out of you for 24h before doing it. I think I got 10+ calls/texts plus emails.
Email, SMS, automated phone call.