Ah, I see your point now: my cell phone carrier can still trick the bank into unlocking my safe deposit box.
But can't we do something similar for every possible system? By controlling my computer's OS, Apple could in principle have a copy of my GPG keys and passwords right now. Ok, I'll use Linux... but do I need to worry about Intel recording the same data somehow? The point is, I'm constantly being forced to store my credentials in someone else's hands.
Well, not really. You could build your own HSM, that would store your keys and sign things for you. Obviously, only with your physical permission (a button press or even a PIN entry).
An AVR (like Arduino) would suffice without much need to trust hardware vendor. It has no communication channels except for the one you define (and you can and should define quite a restricted one) and not much die space (and too low cost) to have a backdoor to begin with. (And really paranoid ones could always go with TTL.)
The only serious problems are cryptographer-vetted RSA implementation that would fit in an AVR and writing a PKCS#11 driver for such HSM.
Yeah, this is totally on tinfoil-hat-grade paranoid side of things, but a CPU-level backdoors are not far from that.
But can't we do something similar for every possible system? By controlling my computer's OS, Apple could in principle have a copy of my GPG keys and passwords right now. Ok, I'll use Linux... but do I need to worry about Intel recording the same data somehow? The point is, I'm constantly being forced to store my credentials in someone else's hands.