I've always wondered this myself. The claim of "we're open source so you can trust us" has always seemed a bit off to me for the exact reasons you give - sure, you publish this codebase, but who says that's what you're actually on your servers?

And from the answers so far, it's disheartening to hear that there doesn't seem to be any great way to guarantee fair play. But I suppose it generalizes - we don't have any way to guarantee that the people we interact with on a daily basis have our best interests at heart, and we do a lot of trusting just because it makes life easier.