How many people have reviewed Paramiko? In particular, how about that ecdsa patch[1] to Paramiko that you'll need to be accessing modern Ubuntu or Fedora (and before long, RHEL/CentOS). What about the python-ecdsa[2] (that paramiko's provisional support for modern Fedora and Ubuntu's default configs is based on)? This entry from its README seems pretty frightening:
This library does not protect against timing attacks.
Do not allow attackers to measure how long it takes you
to generate a keypair or sign a message. This library
depends upon a strong source of random numbers. Do not
use it on a system where os.urandom() is weak.
I'm not saying Paramiko (or its patch sets) are insecure, just pointing out that the same arguments can be made against the libraries and code that Ansible is based on.
Which is one of the reasons I like them as much as I do. I can watch the iterations, they work hard on their product are committed to open source, and release regularly and often.
The other big win for me is I can read their code, I understand python & have a number of items I'll be able to contribute to upstream that will help others use the product.
I'm not tptacek, but note that IP addresses are not exactly cryptographically secured. If you only restrict access by IP address, you tend to lose your whole intranet as soon as one host falls.