I’ve been working on using k3s for my home cluster for this exact reason. I run it in a vm on top of proxmox, using packer, terraform, and ansible to deploy. My thought process here is that if I ever want to introduce more nodes or switch to a public cloud I could do so somewhat easily (either with a managed k8s offer, or just by migrating my VMs). I’ve also toyed with the idea of running some services on public cloud and some more sensitive services on my own infra.
I have been doing k3s on a Digital Ocean droplet and I would say k3s has really given me an opportunity to learn some k8s basics without truly having to install and stand up every single component of a usable k8s cluster (ingress provider, etc) on my own.
It took a bit to figure out setting up an https cert provider but then it was pretty much off to the races
I use kind with podman running rootless, it only works on systems with cgroup2 enabled. But it's very cool. Conventional k8s with docker has a number of security gotchas that stem from it effectivly running the containers as root.
With rootless podman k8s, it is easy to provide all your devs with local k8s setups without handing them root/sudo access to run it. This is something that has only recently started working right as more container components and runtimes started to support cgroup2.
