Good luck with that, we don't have any app stores that let third parties verify checksums of open source projects. The trusted computing base is controlled by Google and Apple basically, and they don't allow certificates from third parties who do audits.
You can't completely remove the trust factor no matter what you do. Can you completely 100% trust your computing device? Maybe. Can you 100% trust your network? Maybe.
But things like encryption help build trust. They ensure that there are safeguards in place in case something gets compromised. They reduce number of bad actors from unlimited to just one (the vendor).
In my opinion, all these are very good reasons to always opt for E2EE even if it isn't perfect.
