> I hear this quite often but where does it come from?

A part of it might be stereotypes and depictions in pop-culture. The drama Billions shows hedge funds getting their consistent edge from insider info. Not sure how true that is but for people outside the industry that doesn't come off as a good look.