I'm not totally wild about OpenVPN. If you asked me which I'd rather expose, SSH or OpenVPN, I'd have to flip a coin. In either case, it's easy to realize the "right" configuration, with a single "bastion" host (hate that word) that provides authorized clients with transparent access to servers.
One thing you can say for SSH is that it gets a lot more audit attention than OpenVPN does.
In either case, the problem I think most startups should move to solve early is multi-factor authentication. I have friends building things with Yubikeys, and others building things with SMS-based auth; either are better than straight passwords, which are (among other things) a policy nightmare.
OpenVPN can be configured to be stealthy (to start a connection, the first packet must be signed with a shared-secret, otherwise it doesn't respond), while SSH doesn't. And OpenVPN has the performance advantage (tunneling is done over UDP instead of SSH's TCP - TCP over TCP brings about a couple of performance problems, especially when there's congestion).
The whole stealth thing (and "port knocking" along with it) is wildly overrated. If your security depends on people not knowing you're running SSH or OpenVPN, you have bigger problems.
If you want to get all cutesy, go hack SSH to add an extra round or two to OpenSSL's AES. That'll learn 'em.
I think the parent comment is referring to OpenVPN's HMAC packet authentication of the TLS control channel, the --tls-auth option. From the man page:
The rationale for this feature is as follows. TLS requires a multi-packet exchange before it is able to authenticate a peer. During this time before authentication, OpenVPN is allocating resources (memory and CPU) to this potential peer. The potential peer is also exposing many parts of OpenVPN and the OpenSSL library to the packets it is sending. Most successful network attacks today seek to either exploit bugs in programs (such as buffer overflow attacks) or force a program to consume so many resources that it becomes unusable. Of course the first line of defense is always to produce clean, well-audited code. OpenVPN has been written with buffer overflow attack prevention as a top priority. But as history has shown, many of the most widely used network applications have, from time to time, fallen to buffer overflow attacks.
So as a second line of defense, OpenVPN offers this special layer of authentication on top of the TLS control channel so that every packet on the control channel is authenticated by an HMAC signature and a unique ID for replay protection. This signature will also help protect against DoS (Denial of Service) attacks. An important rule of thumb in reducing vulnerability to DoS attacks is to minimize the amount of resources a potential, but as yet unauthenticated, client is able to consume.
tls-auth does this by signing every TLS control channel packet with an HMAC signature, including packets which are sent before the TLS level has had a chance to authenticate the peer. The result is that packets without the correct signature can be dropped immediately upon reception, before they have a chance to consume additional system resources such as by initiating a TLS handshake. --tls-auth can be strengthened by adding the --replay-persist option which will keep OpenVPN’s replay protection state in a file so that it is not lost across restarts.
and I just realized who I'm responding to... :) I'd be interested in knowing if the above isn't a strong/decent rationale.
there's probably a bug in some code on your system that's exploitable. maybe openvpn or ssh or its configuration (c.f., debian crypto bug).
under this model, anything that increases the time til (inevitable, under this threat model) success, works.
running ssh or openvpn on a different port or whatever to "hide" it, will help against mass scans for common exploits. it increases the work to find the problem. of course it won't really help against a concerted, targeted attack. but i'm pretty sure unskilled, mass attacks are more common than targeted attacks.
Still, if OpenVPN drops packets lacking the HMAC packet authentication without processing them further, then either
1) the guy would also need the HMAC key, or
2) the zero-day is in the code that looks at the HMAC signature.
It's not that I only count on this for security, but it's a matter of reducing the attack surface. Likewise, I don't have passwordless guest accounts on all my servers, since that would make the attack surface even greater.
if (1) then i certainly agree that none of this really matters (i apologise if that wasn't clear from my previous comment).
on the other hand, if his objective is (2), and not everyone is using some of this obfuscation, then this separates you from the crowd for a little while (but my model is that as time goes on, you are inevitably exploited).
i realise after your company was exploited it might make you feel like (1) is always the case but for many (2) is the salient threat.
Generally speaking, the people running around with OpenVPN and SSH zero day are looking to break into your machine. The people looking to break into any machine are either targeting Windows clientsides, or weeks-to-months-old web vulnerabilities.
People with SSH zero-day are not, by and large, looking to burn those vulnerabilities by spraying them into every busybody's honey pot logs.
> The people looking to break into any machine are either targeting Windows clientsides, or weeks-to-months-old web vulnerabilities.
Actually the people looking to break into your machine are targeting windows clientsides and weeks-to-months-old web vulnerabilities.
There's a cost involved in developing 0day, droppers, remote access trojans, maintaining breach and exfil teams etc. If these guys can get into the developer laptops with an email, a wink and a PDF then why waste the 0day? If you're putting all your effort into a custom SSH daemon without expending equivalent effort on your connection sources (especially when connecting to the Internet) then you're doing it wrong.
Multi-factor is a big deal, especially if you ever want to deal with credit cards at all. We looked at WiKID (open source two factor solution) but building a LDAP backed VPN solution around it was going to be a fair bit of work, so we ended up using PhoneFactor, which is a pretty cheap commercial offering using your cell phone instead of physical token. Big fan.
If you would rather use ssh than OpenVPN but still want the convenience of "direct" connections to computers on the other side of the gateway host, and you want to avoid the tcp-over-tcp slowness problem, try my sshuttle project: http://github.com/apenwarr/sshuttle
Yeah I'd go the ssh route myself. Nothing against OpenVPN, but the (or maybe another) big problem in this scenario is that people are using passwords to log into servers in the first place.
OpenVPN supports certificate based auth but key-based SSH auth is considerably easier to do (and depending on your experience may be easier to manage).
You can also restrict the commands that can be run so you can use things like Git or tunnel MySQL loopback connections without handing over the box.
If they only thing you're planning to run over your VPN is SSH, then yeah, it's a coin toss whether you should run a VPN or just expose OpenSSH to the internet. But if you're planning on running anything else over your VPN, then I'd wildly disagree that the two are equivalent.
Someone else already mentioned the performance advantage, but just the user convenience advantage alone would be a deciding factor for me.
Well aware of that, but it can be a bit of a pain if you're tunneling more than a couple protocol or if your tunneling to more than a couple of hosts. Also, VPN let's you use the same configurations as you would if you were local to the machines, you don't have to use localhost, etc.
And of course, there's also well known performance problems tunneling over SSH because you incur a double hit from the TCP backoff algorithm. Not to mention that you can't (easily) tunnel udp over SSH....
Come to think of it, there's no real comparison. SSH is a great remote shell protocol, but a shitty VPN protocol (not surprising since it's not * intended* to be a VPN protocol).
We have been using OpenVPN to secure one of our Rackspace servers dedicated to development for the last couple of years. We also have ejabberd running on that box (just talking to the vpn interface). It works great for distributed development and the internal secured jabber IMs are great for transferring sensitive stuff like passwords between remote folks. We also have a persistent conference room for a shared "watercooler".
The main problem we have is key management. Does anyone know of a good admin app for generating, storing and revoking keys?
I've been using tinyca2, which has a really nice gnome interface for managing a certificate authority. I'm really happy with it and it takes a lot of the pain out of CA and certificate management.
Interesting. We ran ejabberd over SSL for years without needing a VPN. Most DVCS' support SSH in some form or another. You might find SSH key management a bit easier.
tinc is really good little secret I have found if you want a true mesh P2P secure overlay network. It's open source, has a good community in freenode #tinc, and really easy to install and use, just read the man page http://linux.die.net/man/5/tinc.conf
I strongly recommend not relying on the VPN written in C that nobody's heard of that uses a hand-rolled custom RSA authentication algorithm to authenticate connections for your perimeter security.
can you tell me what the actual difference between OpenVPN and tinc is? I know OpenVPN very well and really like it. Why would I prefer tinc over OpenVPN (or the other way around)?
In my opinion, they share a lot of the same features like running in userspace, easy configuration, ssl/certificate based authentication - just the same.
"Automatic full mesh routing. Regardless of how you set up the tinc daemons to connect to each other, VPN traffic is always (if possible) sent directly to the destination, without going through intermediate hop"
OpenVPN doesn't do that, and it sounds very useful when multiple machines are involved
we are doing that in our production environment. We use different boxes (1 on US and 2 on EU), they are connected with openvpn to serv our different needs.
Our personal development boxes are also connected to each others using OPENVPN and firewalled. All the internal code made for our app is using the VPN ips, and every application is listening on these IP only (not localhost nor 0.0.0.0)
The lines between the nodes were actually meant to show that those nodes use the Rackspace internal network (ServiceNet) for communication - see the colored labels on the diagram.
Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything).
Second, getting a dump of a table from your MySQL DB with SSH+keys:
(I'm not saying it's neccesarily a good idea to du dumps of your production db, but that was the example that came to mind.)
There's also stuff like instrumenting a running Tomcat with JVisualVM on a server in your remote environment (which, no, you can't easily do over SSH-forwarded ports.. also, portforwarding is annoying).
Finally, VPN allows you to run internal services. A jabber-server was mentioned in the article, and perhaps a webserver with some reports/dashboard thing, cacti/nagios stuff etc. without worrying about setting up authentication and SSL for each. Each VPN user gets his own IP, so you can just do crude IP allow/deny if you need, or you can put the customer people on one subnet and allow them to access the reports server, and the devs/ops on another that can reach everything.
While I think you're right in that VPN setups make a lot of things smoother,
SSH actually compares pretty well with the first two of your examples.
"Yeah, there is. First, doing it with SSH+keys requires either multi-step (which is annoying, esp. for copying files), or exposing each servers SSH to the net (which might be unfeasible if you don't have public IPs for everything)."
You can work around much of the annoying stuff in multi-step setups by using SSH's ProxyCommand configuration variable. For example, you could have something like this in your local .ssh/config:
One thing you can say for SSH is that it gets a lot more audit attention than OpenVPN does.
In either case, the problem I think most startups should move to solve early is multi-factor authentication. I have friends building things with Yubikeys, and others building things with SMS-based auth; either are better than straight passwords, which are (among other things) a policy nightmare.