True, right up until you need to use something that must check the SSH_AUTH_SOCK envar and then it's game over man (c.f., Chef Workstation).