You can Google around and find lots of (these things of course will always be subjective) comparisons where people claim they can hear a difference.
I think my point is that for people who have a hi end setup and are used to listening to music -- don't come under the classification of "most people".
I agree with you for that for the average listener with $200 headphones or a club DJ, MP3-320k is fine.