You don't have to deal with operational aspects of consult, nomad and vault of that matter if you choose the managed kubernetes cluster of the cloud provider. If you are talking about container orchestration onpremise - the experience I had with kubernetes is terrible. There 1000 things that can go wrong. Onpremise, I would recommend using k3s which is super simple to setup or microk8s with comes with ubuntu 20.04. I am not sure what benefits you would get with nomad as compared to the lightweight kubernetes distros. One of the benefits of these lightweight solutions is that, you will basically find a helm chart for any serious application out there where as in nomad you might have to figure it out how to deploy. For instances deploying cassandra, you will find a helm chart, but nomad, you might find blog that does it or figure it out yourself. To my best of knowledge that how it is, may be I am wrong?
microk8s and k3s aren't fit or designed for production, they're mostly for testing/experimenting.
And even with managed Kubernetes, there's still a lot of complexity remaining ( GCP had to come out with a more managed service, GKE Autopilot, to address some of that), but you still have the evolution of APIs to keep track of every update you make, and there are still dozens of services that are updated each update, and each one can go wrong ( even if it rarely does).
> . One of the benefits of these lightweight solutions is that, you will basically find a helm chart for any serious application out there where as in nomad you might have to figure it out how to deploy. For instances deploying cassandra, you will find a helm chart, but nomad, you might find blog that does it or figure it out yourself. To my best of knowledge that how it is, may be I am wrong?
Indeed, and that's the main disadvantage for Nomad IMHO, the ecosystem is much smaller so there aren't that many ready-made equivalents to Helm charts and operators. Depending on how many of those you need, k8s can save you a lot of time.
I don't agree with you. What aspects of microk8s or k3s makes it experimental? It used to be the case, in case of k3s it is one of core offerings from rancher. Same goes for microk8s. Purpose of autopilot is something else, if you are talking about pure orchestration bare minimum kubernetes is actually not a bad option. API will keep evolving but basic objects like deployments and statefulsets that you need for paas like experience are quite stable.
> K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.
> [microk8s] Low-ops, minimal production Kubernetes,
for devs, cloud, clusters, workstations, Edge and IoT.
Microk8s started as an easy alternative to minikube for local dev.
k3s started as a simplified version of k8s for testing/experimenting on RPis, etc.
Today both seem to focus on IoT/"edge". Both do clustering and HA of the control plane though, so are in theory usable in production.
However, why would you use either of them in production? Yes, it's easier than vanilla k8s, but still has a lot of moving parts, and to top it off, it's a specific flavour by Rancher or Canonical of moving parts ( e.g. microk8s uses dqlite for storage instead of etcd). So you might stumble on specific to the platform edge cases, and you still have a big part of the k8s complexity to deal with ( in the case of microk8s it tries to abstract some of the complexity with wrappers, but when they fail, you're screwed).
> API will keep evolving but basic objects like deployments and statefulsets that you need for paas like experience are quite stable
Stable now, but Ingress was in beta for quite some time, and the when the beta API gets deprecated, you have to adapt.
