Right, with Google 2-step verification you don't have to worry about number porting attacks. It's just vulnerable in the sense that a phishing site you've entered your username and password into can still trigger the prompt.

>How does the security key/browser pair communicate without involving the site? Does it involve more of Google's interference then? While I know you're not saying "yes" to the site, isn't the key doing roughly the same thing?

When you use a hardware security key with a browser, your browser tells the security key the page's domain, a user id, and a random challenge token if I remember right. The security key signs a message containing all of these things and gives that back to the browser. If you're on a phishing site, the page will have a different domain than the true site, the message signed by the security key will have the phishing site's domain instead of the true site's domain, and the signed response generated by the security key won't be valid for the attacker to use on the true site.