I gave up on a non-standard port because I kept having to figure out configuration for programs that implicitly run ssh (darcs, git, I think there were others..)
On Mac/Linux, setting a different default port for your host in ~/.ssh/config will allow git and other applications to automatically use the new port binding on the server.
…which allows me to just use "ssh server1.example.com" and "git clone foo@server1.example.com:blah/project" to access those resources without explicitly defining the port number.
---
Edit: The standard msysgit installer for Windows may look in the same place too — I haven't tried.
