It depends on the alternative you are suggesting. No taxes? A bureaucracy so vast that can handle all imaginable edge cases? Individual discretion of the tax assessor? I think all these could fix this particular problem but might lead to worse outcomes overall.
Ugh I must have been half asleep when writing the original comment (toddler woke up at 4am). I mistook your comment for a knee-jerk libertarian reaction, but what you propose does make a lot of sense. I stand corrected.