My workflow is: Upload/backup often, Download only manually to continue working on the thing from another machine.
When you use the SSH port 23 instead of 22 you can use the regular authorized_keys mechanism for authentication.
