That's a good outlook. I can't tell you how much it irks me when someone is annoyed that someone else doesn't speak English well. I always say "at least their English is better than your <other language>".
Invariably, that outlook is expressed by people who happen to only speak one language.
I don't mind when I meet someone in normal daily activities that speaks English poorly. My thoughts on the subject mirror your own.
What _IS_ frustrating is dealing with a customer service representative in an overseas call center that speaks English poorly. How many man hours have been lost in forced communication with foreign contractors in an effort to save a few dollars an hour over an American counterpart? Also, those cost savings to a company like Comcast also damage our economy by not employing Americans in need of work. It's a double-whammy of frustration.
Invariably, that outlook is expressed by people who happen to only speak one language.