This is a pretty old school method for using a master connection. Now-a-days, if you don't like screen for some reason, you can still do much better by using the ssh ControlMaster feature. This allows you to have an SSH session open up a socket that allows subsequent connections to that machine to re-use the secure communication channel.

That said, fanout can still be a useful utility.

I should make a blog post detailing my setup, but for now, I'll share the critical pieces:

In my local .ssh/config:

==========

  Host vpn
    ForwardAgent=yes
    ProxyCommand none
    ControlMaster auto
    ControlPath=~/.ssh/%r@%h:%p
    HostName myvpn.domain.com

  Host dest??
    ProxyCommand ssh vpn 'nc -w 60 %h.internal.domain.com %p'

==========

Then, if I ssh to any dest machine like dest01 or dest05, the first one will create the connection to the vpn jumphost, and all the subsequent ones will reuse that connection.

The ControlMaster is a great feature. We use SSH as a mechanism for running remote monitoring plugins in Nagios, using the ControlMaster socket socket saves us a lot of crypto overhead.

An alternative to ProxyCommand is ssh -t, as in "ssh -t gateway ssh inside_machine". ControlMaster is definitely the way to go for the original problem w/ the OTP setup. "ssh -t" is usable on arbitrary terminals for places where gateway is password accessible (and handy when you have to pay per static public IP).

Thanks for the tip. I fought with ssh -t before and just couldn't get it working in my example above. I'm trying again now. Maybe what needs to happen is I need to give up on having my ssh_config as the central location defining my aliases and instead use bash aliases to give me the short name connect.

In the discussion of sudo -- having "PASSWD: ALL, NOPASSWD: /bin/dd" is effectively "NOPASSWD: ALL". The legitimate user will have to enter his password for other commands in a semblance of security, but an attacker who compromised his account can bypass that. Consider the following (with /sbin/route also in NOPASSWD):

  sudo /bin/dd if=/path/to/anything of=/sbin/route
  sudo /sbin/route # pwned!

So I can now run ANY program without a password. If you like, throw in a few more dd commands to backup/restore the abused command (route).

He also has /sbin/insmod in NOPASSWD, which is another "get out of jail free" card. If I can load a kernel module, I can do anything.

First thing I noticed too. The sudo maintainers should modify visudo to do some checking for known "safe" apps versus "unsafe" apps.

I often use a program called shmux to run commands on a number of remote machines via SSH in parallel. There are many different programs (including fanout) that do this, but I think shmux is the best.

It can be run in both interactive and non-interactive modes. The interactive mode is curses based and lets you easily control how many of your ssh tasks run in parallel, lets you pause and resume, and nicely separates out the stdout and stderr output. The non-interactive modes let you define what success means in terms out output and return code, and splits out output and return codes into separate files for further processing.

I don't recall if I ever tested out shmux when I was looking for terminal multiplexers. For interactive usage, the best one I ever found was Omnitty. Unfortunately, I was never able to get it to compile for Mac OS X so I normally use it by first sshing in to a remote Linux machine then using omnitty from there to fan out to the cluster I want to play with.

Neat, but couldn't he just use Screen on the gateway terminal to get around the issue of wanting to have more than one session at a time (assuming its installed)?

It's probably a questionable assumption that screen will be installed on the jump host. Those machines usually have pretty minimal systems.

Sometimes multiple xterms are easier than using screen with multiple buffers.

In that case you could connect to the same screen session in all of the xterms (see screen -x), but have them display different buffers.

How does that help? You'd either still have to connect to the remote server multiple times (if screen runs remotely), or you're going to have to connect multiple times from inside screen (if screen runs locally)...

Using the ssh host as a SOCKS proxy in combination with http://www.meadowy.org/~gotoh/ssh/connect.c is a lot easier than adding a separate tunnel for each host on the other side of the wall, as long as the intermediate host allows you to use it as a SOCKS proxy (DynamicForward option of ssh).

I tried to figure out why he couldn't log to more than one machine using the same connection to martha?

ssh martha -> ssh waldo -> do stuff -> exit ssh othermachine -> do stuff -> exit exit

Port-forwarding surely works but did he explain the origins of this problem somewhere?

He means simultaneously.

So why not do everything at 'martha' shell prompt instead of his local shell prompt?

You only need some place where you can do for loops and spawn subsequent ssh connections -- 'martha' sounds like a good candidate for that.

Another good source of ssh tricks and tips are Brian Hatch's articles - http://www.hackinglinuxexposed.com/articles/

Great techniques indeed. Bookmarking!