Not as far as I'm aware, at least not without some really clever attack. The design is such that all you're doing is asking the remote agent to use it's private key to sign things, it never actually exposes the private key. Your agent doesn't even always have access to your private key- it could be on a TPM or Yubikey where the key never leaves the device. you are just passing around a connection to your agent that is able to authenticate with a private key.
I would not be surprised if there is some way to attack this into getting a private key, but it would either be a direct attack on the agent code like sending it malformed messages to somehow get remote code execution to then read the key. Or some more complicated attack on the cryptography where you repeatedly force it to auth and can somehow use the results to reduce the key space needed to brute force the key - along the lines of a known plaintext attack.
But I'm also just a hobbiest here who has looked a little into the security model but I am by no means a cryptography expert, so take all of this with a grain of salt.
I would not be surprised if there is some way to attack this into getting a private key, but it would either be a direct attack on the agent code like sending it malformed messages to somehow get remote code execution to then read the key. Or some more complicated attack on the cryptography where you repeatedly force it to auth and can somehow use the results to reduce the key space needed to brute force the key - along the lines of a known plaintext attack.
But I'm also just a hobbiest here who has looked a little into the security model but I am by no means a cryptography expert, so take all of this with a grain of salt.