That's a fair point. I use either a ssh config file with all the correct options in it, or the -p option if I'm doing it without a config file, perhaps that's why I've never had a problem.