It's not that clear cut of a tradeoff. If Apple were to allow limitless sideloading of apps and that became the standard way that people installed apps on iOS devices, it would seriously limit their ability to keep the platform secure against malicious apps.
No no. I'm talking about keys to my phone alone. If I choose to trust my own certificate then that only impacts what software I can run on my phone. An app cannot "sideload itself onto all your contacts". This extends to the OS itself. Let me change the certificate governing the OS software. Only once you allow that do I truly own my phone.