Indeed, the problem with lots of fallbacks is that they can invalidate user's requests for higher security. Security can sometimes end up being only as strong as the weakest link.

Make the fallback too lax and you might as well not bother with 2FA/Passkeys at all.