People will get even more frustrated when their flimsy physical authenticators break down. Passwordless authentication is perfectly okay for a service like a physical bank, employer etc. that can always check your identity and re-enroll you via some other means if need be. It's a disaster in the making for something like Google, Facebook, MS etc. etc. that offer no such thing. Make no mistake, the same people who pick weak passwords today will neglect proper care about any kind of disaster recovery scenario.

The best you could do is use a "software-based" or "virtual" authenticator that explicitly refuses to protect against a user cloning and subsequently restoring its identity, at least as a last resort. Not good enough for bank payments, ofc. but plenty usable for everything else.