OpenSSH by default tries authenticating with all your identities. You should probably turn that off too.

Or keep sensitive identities in a different location, so they aren't automatically picked up like that. That forces you to specify an identity file when you need one of those, but that small inconvenience is probably not a concern if the identity is that sensitive.