Of course it matters, you're being defrauded as a consumer.

Aside from that, it could have health ramifications. Most people eat organic to avoid herbicide exposure, and most people use olive oil because it's considered healthy. Obviously there are countless examples. You should get what you pay for.