Here's what I don't really get.. So, let's say you have three hosts and create your cluster.
But now, you still need a reverse proxy or load balancer in front right? I mean not inside the cluster but to route requests to nodes of the cluster that are not currently down.
So you could set up something like HAProxy on another host. But now you once again have a single point of failure. So do you replicate that part also and use DNS to make sure one of the reverse proxies is used?
Maybe I'm just misunderstanding how it works but multiple nodes in a cluster still need some sort of central entry point right? So what is the correct way to do this.
My solution for this setup is having ingress controllers on all three nodes, and then specifying all three IPs in all DNS records. That way the end user will "load balance" based on the DNS randomization.
Of course, if a node goes down, a third of the traffic will be lost, but with low TTLs and some planning, you can minimoze the impact of this.
It's an interesting approach.
I did it a bit differently. I set up three Proxmox nodes on three hetzner servers. Then I deployed virtual routers. I then set up HAProxy and k3s nodes as LXC containers.
What's nice about the whole setup is that a proxmox node can go down and it all still works. I will now set up keepalived as mentioned in the other reply so the HAProxies will also be fully HA. Proxmox also works well with zfs and backups.
I set up the proxmox nodes manually and did the rest with terraform + ansible. One `terraform destroy` cleans up everything nicely.
I wonder how the performance difference is between bare metal and k8s node in LXC.
You almost answered your own question. One common solution is to have 2 nodes with haproxy (or similar) sharing a virtual IP with keepalived that load balance de traffic to the control plane nodes and to the nodes where your ingress controller runs.
There are other options, like running the haproxy in the control plane nodes.
