What a terrible name. So it is non-private private networking, great distinction there DO. I wonder why people are confused..?
What about "Shared Intranet" instead. Why has the word "intranet" fallen out of style? And why use the word private to describe something that is inherently non-private by design?
I actually think the addition of this free inter-droplet channel is amazing (and a potential massive cost saver), but the name sucks.
Technically, I think they're correct with their definition[1]. However, they should be more explicit with the "Shared" part and have some better documentation rather than an announcement blog post[2], a community guide[3], and a moderator comment on that community guide [4]. Especially in today's privacy-conscious world.
Having said that, people who are concerned about privacy should always be validating their setups themselves. The maxim of "trust but verify" comes to mind.
Couldn't you technically call the internet a non-private private network? It's all networks. The private/public part depends on the context you're talking about.
An example is that a network at starbucks is private to the people at the starbucks, but public in a sense that anyone there can connect.
Perhaps they should indicate "shared private" more clearly in the droplet settings. Their usage of private network seems to be just referring to RFC 1918 addresses an not much else but many people will infer more than that.
When setting up multiple VPSs connected by "private networking" with a company like Linode, or Digital Ocean, or what-have-you, you need to assume that the inter-VPS links are not secure. It's a little piece of knowledge that comes with experience, hard for newbies to realize.
The first time you set up one of these clusters, you might follow one of Linode or DigitalOcean's handy guides, where they might suggest i.e. a reverse proxy server receiving (and decrypting e.g. HTTPS) inbound traffic, routing it out to multiple worker machines, and a single backend database system. Linode sells dedicated Load Balancers for the front end of exactly this sort of set-up.
These guides almost always fail to mention that the data is observable as cleartext in the internal network. They ought to be reedited with big bold warnings starting that these links ought to be secured. (Can Linode's Load Balancers even secure these links?) Besides other eavesdropping customers, there could potentially be little magic government agency plugs installed -- or the eavesdropping customers could be government security agencies themselves. (Sorry, tinfoil, I know...)
OpenVPN connections are a lightweight, efficient solution. They're also transparent once you change IPs from those of the virtual network interfaces to those of the secure virtual network interfaces. Such a configuration is still non-trivial, though, for someone configuring their VPS via a control panel rather than the command line.
Hm, I'm not an "expert". Believing that there's any level of security, at this point in time, in any cloud based service is a sign of a semi-retarded IT person IMHO.
That doesn't mean I don't support the cloud, don't use or don't like the technologies that we have today. I wish I could learn and use all modern services out there.
There are various levels of security needed, depending on a case-to-case scenario. I don't believe any company wilfully would create any kind of problems to any of it's users/clients. There are hosting providers (the PirateBay was using one) who are renowned for resisting subpoenas and what not.
BUT if you need absolute security - for whatever reason - in today's world you start by combining carefully the HARDWARE, you don't even buy ready-made products, then you place the server in a place where is physically safe from discrete eyes and hands. Then we can talk software...
People don't "need" to do that, and many don't. They are aided by the false assertions of jackass vendors like this that make false claims further reinforcing their mistaken beliefs.
The configuration overhead of OpenVPN on an internal network like this is lunacy. The correct answer is a secure L2 network... a private one.
Your VM vendor has access to your host ram. Encrypting isn't going to protect you from them. The threat model here is other customers, which are trivial to partition. They just didn't do it.
In your experience, is a single OpenVPN server sufficient on such a network, or is it possible to have a fallback (server)?
I previously tried using a distributed VPN setup without a single main server; that didn't work out so well however, mostly because the software was somewhat unreliable.
Most data centers actively want the default private networking to work across the datacenter with everyone. A better term would be non-routing. Rackspace calls theirs "service net". These non-routing networks encourage people to use cheap internal bandwidth (rather than bounce of an expensive leased line just to come back in) and possibly offer a special service just to that data center's customers. There are tons of companies offering sql and nosql databases as a service, and any number of other offerings.
Most cloud providers recommend you configure your firewall rules keeping in mind that other people could be on your internal network connection. This is really a security best-practice anyway.
This is basically a billing feature, so that you can use "free" internal networking capacity instead of metered external networking usage. They are just following what Linode and other VPS providers do.
I think part of the problem is that people confuse DO, who just rent out cheap boxes, with more "full service" cloud providers. DO is partly to blame because they keep marketing themselves as a cloud platform when really they're yet another company who rents out servers with a few extra features.
It's "private" in the sense that it's not going over the public network, rather than private to your account. This was pretty clear when the feature was announced. I don't know why it's a surprise now.
You still have to secure your server, even if it's only communicating over the "private" interface. I usually use this feature for database servers, and lock down the firewall so that only the web servers can communicate with the db servers, and only via the "private" interface.
This is the same way Linode's works as well. I'm curious, how many people here thought that Private networks were walled to only droplets/nodes on the same account?
Amazon's private network is private by default, I believe Google's is as well, and I believe Azure's is. Digital Ocean is acting much more like a traditional VPS company than a modern cloud computing company here.
Well what you are describing is VPC (which for new accounts for a while now is enabled by default), but VPC is software isolation which creates private networks, but EC2 Classic is technically shared private IP space between all customers. There is a big difference between the two.
Even with EC2 classic, every machine is locked down from every other. The typical way around this is to modify the default security group, which is limited to your account and your instances. It's a different approach than VPC, but everyone had effectively private networks with EC2 Classic, unless they really went out of their way to open up everything to everyone.
Granted we're probably now distinguishing between being able to see unencrypted traffic and being able to access ports on a machine. They're both bad, but I'd argue having your DB port exposed to the world is probably worse.
EC2 (and OpenStack) uses security groups to give you the best of both worlds: shared address space so you _can_ route to other customers; but the firewall means you have to explicitly allow the traffic.
Yup EC2 security groups are great for ease of management. Group to Group communication for something like load balancers to auto scaling back-end app servers is great as you don't have to worry at all about tracking IP addresses.
But the fact it it does exist, and I'd venture to guess most major customers who have been using EC2 for a long time now still use it currently and it is still available to many people and is not going away anytime soon.
The glib answer is that a cloud provider understands the need for a private network.
But, in that is a deeper truth: a VPS is a cheaper machine. It becomes "cloud" when you use many machines together. If you're using many machines, you really want intercommunication between those machines to be easy, fast and - yes - reasonably secure.
Most cloud providers gce, aws (vpc), azure, rackspace, etc. provide multi-tenant networking, where private network addresses are specific to an account. So there is legitimate reason for confusion.. that said i did read the release notes on DO's announce when i started using this feature and it was pretty clear that's not what they offered.
I don't understand the outrage. There's a clear distinction between "Private Network" and "Private To Just My Servers Network." I've been a DO user for years and I've never been confused about this. Anyone I've ever recommended DO to has never been confused about it either.
I just automated the firewall and SSH tunnels/TLS certificates setup to restrict access and encrypt communication between my droplets when necessary.
I see a lot of people get tripped up by this via my work on rubber [1]. It's probably selection bias, but most of them are switching to DO as a cheaper alternative to EC2 and in EC2 your private network is "private to just my servers network" unless you go well out of your way to change that. It also seems like a pretty sensible default for a company that has a lot to lose if malware is able to propagate through a datacenter by anyone willing to invest $5 for the month.
Seems like a very small sample size to be convinced it's clear.
When you call something private, that carries a lot of connotations. To me that distinction that you think is so clear is anything but. While I use Linode they also talk about a private network that I assume their switches are configured in such a way that the data doesn't leak outside of my VMs. That I could ping and connect other linodes needs to be made explicit. Why not call it the "Linode-wide Network" or "Shared DO-only Network"?
Ambiguity is not a good thing when the consequences are high.
I agree with you - it's like having one NIC to the internet, one to a big LAN switch with every other device. No surprise for me... especially when the touted benefit has never been privacy, but cheaper network costs. I suppose they could change the name to "Local networking" or "DigiNet™" or something.
A private network has always meant "a network not accessible through the public internet". If people are using something they don't understand their outrage should not be taken seriously. Learn iptables, it's not hard to use!
It continues to amaze me how many people blame the user for "doing it wrong" or being stupid or something when a company is found to be blatantly lying to reduce their costs.
It's like the whole industry has Stockholm Syndrome or something from decades of vendor abuse.
People, really: private means private. They lied. There's no two ways about it. We didn't misunderstand because we're stupid, we misunderstood because they were cutting corners and being intentionally misleading by using words that have definitions different than the thing that was actually happening.
This isn't new with Digital Ocean. Their contempt for their customers is palpable. A year or two ago I caught them leaking customer data and they made a huge handwavey blog post full of lies about how nothing was leaked.
This is a complete non-issue; anyone really concerned with privacy wouldn't rely on DO's security.
Anyone who was using DO's private network for security reasons and complains is just being hypocritical. They don't care about security: if they really cared about security they would spend the time to implement it themselves.
Security and privacy aren't black and white topics, they're shades of grey, with the exact shade depending on your specific personal or business requirements and appetite to risk.
It could be posited that anyone who is really concerned with these things will want to protect data in transit, but also won't want to use a hosting service relying upon multi-tenant hypervisors which may be susceptible to exploit.
One has to assume AMZN didn't invent AWS Dedicated Instances just for shits and grins. :)
Amazon didn't invent AWS Dedicated Instances for security, they invented them for money. Now they can sell services to people who, for example, fall under HIPAA. And often those people don't care about security either--they just care about complying with regulation so they can do business.
Security is pretty black and white. There are a limited number of people who have reason to look at your data. If people other than those people can see your data, then you aren't secure. Amazon is typically not part of that group.
You may as well written "anyone really concerned with privacy wouldn't use a VPS." You don't have to outsource security to screw this up. You're given a non-routable IP. It's called "private networking". And you've just been sold on moving from EC2, where you're effectively on your own VLAN. It's very easy to mess this up. And adding a firewall rule doesn't work because you probably added the subnet anyway. Even if you were diligent enough to detect that, you'd have to propagate rules per droplet every time a new droplet is created or another destroyed (wouldn't want a new account to get your old IP). Doing that in a fault-tolerant way can be a lot of fun. Oh, and your iptables rules look great, but Ubuntu won't restore them for you upon reboot. So even more chances to mess up.
This is deceptive and it shouldn't be acceptable. Maybe you shouldn't be a sysadmin without knowing all this, but I don't know how you learn it without getting burned by it every step upon the way. It's not like the defaults are sensible in most cases.
> And you've just been sold on moving from EC2, where you're effectively on your own VLAN.
Mrh, not quite. If you are a relatively new EC2 customer, then you are a VPC customer, which might effectively be your own VLAN.
EC2 has also been this way for ages, but it has also always had security groups. If you open something up on EC2 Classic to 10.0.0.0/0, any other customer within that region will be able to access your services.
Sometimes this is desirable for collaboration sake, but bridging VPCs is probably a better solution.
> I don't know how you learn it without getting burned by it every step upon the way. It's not like the defaults are sensible in most cases.
Welcome to #SysadminLife ;)
I do agree that everyone could be clearer about this, Rackspace, Linode, Digital Ocean, and Amazon. A substantial motivation is to keep your inter-VM traffic off the edge routers.
> EC2 has also been this way for ages, but it has also always had security groups. If you open something up on EC2 Classic to 10.0.0.0/0, any other customer within that region will be able to access your services.
Except that's not how anyone did it. You could define rules that used a security group as the source. And since every instance belonged to a default security group, locking down access to other machines you owned was quite trivial -- you only added rules to allow access from that default security group. If you wanted to add subnet rules, that was usually limited to special cases you would have to actively think about.
> Welcome to #SysadminLife ;)
> I do agree that everyone could be clearer about this, Rackspace, Linode, Digital Ocean, and Amazon. A substantial motivation is to keep your inter-VM traffic off the edge routers.
My source of issue was with the grandparent blaming people that get burned by it. As long as the only way to learn this stuff is to get burned by it, it's counterintuitive to blame people for getting burned by it. And in this case, there are measure DigitalOcean could take to make things easier on everyone. E.g., configure the droplets to save and restore iptables rules on shutdown/power-on and lock the instances down by default to only have port 22 publicly exposed.
I can affirmatively tell you that more than one of my employers, successful startups several years in, _did_ do this when I started.
Anyway, I don't blame people that get burned by this, very little in hosting is obvious.
That's a great example of why people need skilled ops help. Understanding how one computer connected to a cablemodem works and understanding what happens inside of a datacenter are very different things.
When you buy a VPS you should assume that it's not private.
There have been plenty of POC's of side channel attacks against sharded/virtual infrastructure including retrieval of encryption keys from a co-hosted VM.
On top of that you inherit the risk of potential vulnerabilities within the hypervisor itself which could be potentially exploited to execute code on the host or to compromise co-hosted machines.
There is no mode for any virtual infrastructure to give you neither truly private networking (since at best it's going to be on a virtual switch) or a private host for that matter.
Unless you control everything including the datacenter it self any true sense of privacy should not be considered, especially not in a virtualized multi-tenant environment.
When a PaaS/Cloud provider says private networking, they mean RFC1918 address space. It's not publicly routed IP space which is private. So saying they offer private networking is accurate against that long standard definition. The technology and effort involved in providing the equivalent of AWS' VPC is huge and most smaller providers don't have the scale, man power, or resources to do it. It's a non-trivial solution to deploy.
Not understanding the platforms you choose to use is not an excuse to write something like this article. It's also not an excuse to not understand how to manage the platform(s) you run your services on.
I agree with shaggy. I don't know how you would expect to have a VPN accessible only by your account's droplets for $5. A private network means an internal network. So it's just using the internal infrastructure of the data center to communicate with other droplets in the data center. It's more performant than using the external network. Usually companies' advertise a fully private account infrastructure as a "Private Cloud" or something of that nature.
"Private Network" is often analogous with "Internal Network". They aren't lying,it's private as in it's using private IP assignment within their own infrastructure. Thus, private.
I would have thought the fact that you can't set static internal IP's would have implied this. If you're looking for something that's stupid easy to setup and super flexible checkout tincd. A co-worker turned me on to it and it's fantastic so far, but requires a bit of re-thinking how you want to layout your VPN (if you want to REALLY use it the way it should be, otherwise you can just set it up like a normal VPN).
As a secure alternative, would something like ZeroTier [1] or cjdns [2] work to create a private cloud? Or is there a simpler solution that would give you seamlessly encrypted networking between nodes (even across data centers)?
It seems to me that would be quite easy for virtualization software to prevent individual VPSs from reading networks packets not meant for them, thus providing decent privacy. What's the technical hurdle that prevents this in practice?
It's tricky because the routing is completely dynamic. You also can't use the typical hardware tricks at scale - for example providers will have more customers than available vlan numbers.
So you have to have a dedicated solution running something comparable to openvswitch in software, or interface with your hardware really fast and really often. Usually the hardware is not really built to be reconfigured constantly every second (or more often).
Once you start adding more than address filtering, you get an explosion of rules. For example in openstack (neutron), it's pretty common to just put a hard limit on the number of entries - otherwise you end up with one vm spawning and suddenly every router has to add another entry for each of the ports to each of the possible 1000s of other vms. And then clean up and keep everything consistent when vms are killed again. It really does get hard beyond the bare basics.
Yes, that's called network virtualization and it's complex to operate at scale. It's built in to Amazon VPC, Google GCE, Azure, etc. but often not found in second-tier providers.
It's amazing to me how people can not do their research, not understand the service they are choosing to sign up for, and proceed to publicly malign a company because of their (the customer's) ignorance about how the product works.
One has a reasonable expectation that services called "private" will be. It requires no further research.
Why is it amazing to you that people take a company's statements at face value? Why blame the user when the other party is actively misleading them to cut corners and reduce costs?
This is news to no one. Every datacenter I've ever done business with offered 'internal transfer' for free and used basically identical verbiage to explain it.
Private: belonging to or for the use of one particular person or group of people only.
"all bedrooms have private facilities"
synonyms: personal, own, individual, special, exclusive, privately owned
"his private plane"
