Yeah my problem is more similar to this. I get impatient pretty often, and I worry that I seem impatient, and that it makes the person I'm pairing with anxious about me rushing them, and then whether they're actually anxious about that or not, I'm still anxious about trying not to be impatient or at least not seem like I am.

This is a totally personal problem, and I've had pretty long successful periods of my career that I've pair programmed most hours of most days, and I do think it's a technique with a lot of advantages, but it really isn't my favorite thing to do because of this kind of dynamic.