The lines between the nodes were actually meant to show that those nodes use the Rackspace internal network (ServiceNet) for communication - see the colored labels on the diagram.

Ohh, I see, blue = unencrypted. That makes sense, thanks. So there really isn't any gain over using SSH with private key authentication with this?

Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything).

Second, getting a dump of a table from your MySQL DB with SSH+keys:

  ssh prod-db 
  mysqldump what-i-need > file.sql
  exit
  scp prod-db:file.sql .

With VPN (assuming the connection is running):

  mysqldump -h prod-db whatever > file.sql

(I'm not saying it's neccesarily a good idea to du dumps of your production db, but that was the example that came to mind.)

There's also stuff like instrumenting a running Tomcat with JVisualVM on a server in your remote environment (which, no, you can't easily do over SSH-forwarded ports.. also, portforwarding is annoying).

Finally, VPN allows you to run internal services. A jabber-server was mentioned in the article, and perhaps a webserver with some reports/dashboard thing, cacti/nagios stuff etc. without worrying about setting up authentication and SSL for each. Each VPN user gets his own IP, so you can just do crude IP allow/deny if you need, or you can put the customer people on one subnet and allow them to access the reports server, and the devs/ops on another that can reach everything.

While I think you're right in that VPN setups make a lot of things smoother, SSH actually compares pretty well with the first two of your examples.

"Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything)."

You can work around much of the annoying stuff in multi-step setups by using SSH's ProxyCommand configuration variable. For example, you could have something like this in your local .ssh/config:

  Host prod-db
    ProxyCommand ssh bastion nc -w1 %h %p

After which 'ssh prod-db' and 'scp prod-db:...' work as if the bastion host wasn't between you and prod-db.

"Second, getting a dump of a table from your MySQL DB with SSH+keys:"

If you can access prod-db with SSH, you can dump the database with a single command:

  $ ssh prod-db mysqldump what-i-need > file.sql

  > ssh prod-db 
  > mysqldump what-i-need > file.sql
  > exit
  > scp prod-db:file.sql .

I remember running across a tutorial/blog post once about how to automatically setup back-channel on an ssh connection so that you could do:

  > ssh prod-db
  > mysqldump what-i-need > file.sql
  > scp file.sql mymachine:.

Edit: Here's a similar article (the one I was thinking about is older than this one) --

article: http://codysoyland.com/2010/jun/6/ssh-tip-automatic-reverse-...

hn discussion: http://news.ycombinator.com/item?id=1409263

  > ssh prod-db 
  > mysqldump what-i-need > file.sql
  > exit
  > scp prod-db:file.sql .

You can definitely do better than that with ssh:

  ssh prod-db "mysqldump what-i-need" > file.sql

If the dump is large and the network is the bottleneck, you can also do it like this:

  ssh prod-db "mysqldump what-i-need | p7zip" | p7zip -d > file.sql

Edit: Noste gave the right oneliner, sorry I didn't notice it at first. Consider mine only as an expansion of his second point.

That diagram was drawn like that just for exemplifying purposes.

Just to illustrate what network interfaces you would use for communicating between server processes.