The server doesn't care what private key you use. If it's signed by the CA, the server trusts it.

Unless I'm missing something, anyone with control over your CA can easily sign in as you.

This is only true if you don't trust the machine you are connecting to.

Private key (only you know) -> signed by CA -> public key you share.

You put the public key on the server , the CA can't change that file... And they can't make that particular cert work without a private key that you hold.

Most CA can do is revoke and break the chain on you.

Uhm. No... he's right.

In the browser world.. A a rouge CA, I could generate a certificate pair for https://google.com and your web browser would trust it.

The same is true with this setup... the solution, for this is to be your own CA and add the CA to all of your clients, rather than adding all of your clients to the server. So this reverses the problem, which for many should be easier.

(I honestly can't see anyone using a public CA for this.. it would be nuts)

You might be surprised to see the nut things people would do for convenience... but in this case you are probably right as the average computer user probably don't use SSH.

The whole point of using a CA is to not put every public key on every server.

If the public key is going to be there, then the CA is not doing anything.