Well I think that's more of a cost question -- AWS can get expensive pretty quickly. Three t2.micros (one coordinator, 2 nodes) are absolutely pitiful in terms of processing power but that's already ~$30/month when you could get a way beefier machine on Hetzner whether dedicated or cloud (also Scaleway[0]).
I'm a fan of Hetzner because I think their cheap dedicated machines are worth the operational costs for me, and the issues with upkeep I'll face are good for me because that knowledge has value. Also, I want to note that if you actually start subscribing to the immutable infrastructure movement that's going on right now, once you look past all the buzzwords it's a fantastic way to run servers stress free -- as long as your data is backed up/properly managed, just shoot servers left and right, spend a lot of time to get them into the right state ONCE, and never worry about it again. You can even use terraform to provision hetzner. Again, this kind of thinking and the related tooling is catchy/popular right now because it's useful at larger scales, but it can also free you of a lot of worry at lower scale. For example I have a post on using Container Linux to go from brand new machine to single node k8s cluster, with one file.
To be honest though, setting up a Hetzner dedicated machine is very very easy -- they've got great utilities. You could even go with Hetzner Cloud and things will be more managed.
I would say go with AWS if you want to experiment with AWS technology as well -- and want to use their value added services. If you run kubernetes on a dedicated machine on hetzner you're definitely not going to get the rest of that, of course.
BTW, kubeadm is better/less complex than kops -- It's almost impossible to fuck it up, but there are subtle things in kops due to the AWS integration that make things a little things ever so slightly more difficult.
I'd be really interested in how you're approaching persistence? I've also found self managing clusters provisioned with kuibeadm fairly hassle free until persistence is involved. Not so much setting it up (e.g. rook is fairly easy to get going with now), but the ongoing maintenance, dealing with transitioining workloads between nodes etc etc.
tl;dr - Rook is the way to go, with automatic backups set up -- using rook means your cluster resources are ceph-managed, you basically have a mini EBS -- Ceph does replication across machines for you in the background, and all you have to do is write out snapshots of the volume contents from time to time just in case you get spectacularly unlucky and X nodes fail all at once, in just the right order to make you lose data. Things get better/easier with CSI (Container Storage Interface) adoption and support, snapshotting is right in the standard and restore is as well -- barring catastrophic failures you can just lean super hard on Ceph (and probably one more cluster-external place for colder backups).
I'd love to share! In the past I've handled persistence in two ways:
- hostPath setting on pods[0]
- Rook[1] (operator-provisioned ceph[2] clusters, I free up one drive on my dedicated server and give it to rook to manage, usually /dev/sdb)
While Rook worked well for me for a long time, It didn't quite work for me in two situations:
- Setting up a new server without visiting Hetzner's rescue mode (which is where you would be able to disassemble RAID properly)
- Using rkt as my container runtime. The Rook controller/operator does a lot of things which require a bunch of privileges, which rkt doesn't give you by default and I was too lazy to work it out. I use and am happy with containerd[3] (and will be in the future) as my runtime however, so I just switched off rkt.
right now, I actually use hostPath volumes, which isn't the greatest (for example you can't really limit them properly) -- I had to switch from Rook due to my distaste for needing to go into Hetzner rescue mode to disassemble the pre-configured RAID (there's no way currently to ensure they don't raid the two drives you normally get after the automated operating system setup). Normally RAID1 on the two drives they give you is a great thing, but in this case I actually don't really care much for main server contents since I try to treat my servers as cattle (if the main HDD somehow goes down it should be OK), and I know that as long as ceph is running on the second drive I should have reliability as long as I have more machines which is the only way to really improve reliability, anyway.
Supposedly, you can actually just "shrink" the raid cluster to one drive, and then remove the second drive from the cluster -- then I could format the drive and give it to Rook. With Rook though (from the last time I set up the cluster and went through the raid disassembly shenanigans ), things are really awesome -- you can store PVC specs right next to the resources that need them -- this is much better/safer than just giving the deployment/daemonset/pod a hostpath.
These days, there's also local volumes[4], which are similar to hostPath, but offer a benefit in that your pod will know where to go because the node affinity is written right into the volume. Your pod won't ever try and run on a node where the PVC it's expecting isn't present. The downside is that local volumes have to be pre-provisioned, which is basically a non starter for me.
I haven't found a Kubernetes operator/add-on that can dynamically provision/move/replicate local volumes, and I actually wanted to write a simple PoC one of these weekends -- I think it can be done naively by maintaining a folder full of virtual disk images[4] and creating/mounting them locally when someone asks for a volume. If you pick your virtual disk filesystem wisely, you can get a lot of snapshotting, replication, and other things for free/near-free.
One thing Kubernetes has coming that excites me is the CSI (Container Storage Interface)[5] which is in beta now and standardizes all of this even more. Features like snapshotting are right in the rpc interface[6], which means once people standardize to it, you'll get a consistent means across compliant storage drivers.
What I could and should probably do is just use a hetzner storage box[7].
I'm a fan of Hetzner because I think their cheap dedicated machines are worth the operational costs for me, and the issues with upkeep I'll face are good for me because that knowledge has value. Also, I want to note that if you actually start subscribing to the immutable infrastructure movement that's going on right now, once you look past all the buzzwords it's a fantastic way to run servers stress free -- as long as your data is backed up/properly managed, just shoot servers left and right, spend a lot of time to get them into the right state ONCE, and never worry about it again. You can even use terraform to provision hetzner. Again, this kind of thinking and the related tooling is catchy/popular right now because it's useful at larger scales, but it can also free you of a lot of worry at lower scale. For example I have a post on using Container Linux to go from brand new machine to single node k8s cluster, with one file.
To be honest though, setting up a Hetzner dedicated machine is very very easy -- they've got great utilities. You could even go with Hetzner Cloud and things will be more managed.
I would say go with AWS if you want to experiment with AWS technology as well -- and want to use their value added services. If you run kubernetes on a dedicated machine on hetzner you're definitely not going to get the rest of that, of course.
BTW, kubeadm is better/less complex than kops -- It's almost impossible to fuck it up, but there are subtle things in kops due to the AWS integration that make things a little things ever so slightly more difficult.
[0]: https://www.scaleway.com/pricing/
[1]: https://vadosware.io/post/k8s-container-linux-ignition-with-...