In my experience, it's already a problem. When I'm shopping online with a brand whose fit I don't know, I order everything in medium and large and then return the size that fits less well. I'd imagine most online clothing stores have sky-high return rates.

I don't think many brands would want to do this, but it would help if they had a chart showing how their fit compares to other brands. E.g. "much looser than Dolce & Gabbana, looser than Calvin Klein, tighter than Gap".