No, the inherent problem of SMS is that it can be stolen/redirected. Given that, and given that companies are too eager to use it as 1FA, you shouldn't use it.
If I'm giving advice to companies, I say "don't use SMS 2FA as 1FA" (well, I actually say "don't use SMS 2FA at all, it's too tempting for a support person to use it as 1FA"), but this thread is about the user, and as a user, you shouldn't use SMS 2FA.
I wonder if companies that have your phone number and do such careless things as "phone-number based 1fa", will not also simply do that if you don't even have 2fa enabled... As long as they have your phone number, they'll abuse it.
So you are not against phone based 2fa or 1fa, your are against giving companies your phone number. But them, if they are soooo careless to try phone based 1fa when they can get away with it, they are also probably open to some social engineering.
In the words of RMS: "We should all try to make those companies fail."