Yes and no; some countries have much more stringent rules about what you have to do to avoid being resident for tax purposes. You might want to route your income through Panama, but very, very few of the ultra-wealthy want to be forced to actually live there.

Countries that currently have lax rules about tax domiciles do so in large part because it has no real impact due to other loopholes, but if you start to remove those, then tightening up residency rules will have a much larger payoff. Nothing stops the UK from saying that spending more than, say, 20 weeks a year in the UK makes you liable for UK income tax on your entire worldwide income (which is, very very roughly, the rule the US has).

That being said, it's always going to be easier to hide income, which is the practical advantage of a strong, universal VAT. It's much harder to hide consumption than income. (Or even better, land value taxes!)