> If what you're using it for is HTTP traffic, HAproxy seems to be the most mature tool out there for HTTP load balancing and failover.

HAproxy is a reverse proxy, it can do load balancing, but it's not useful if you need a HA cluster, as HAproxy becomes the single point of failure in your architecture. You still need something like wackamole, vippy or ucarp to activate the switch from a failed machine to a standby box.

I personally hate this model. Layer2 networking architectures like this need to finally die off, as they are extremely complex and difficult to troubleshoot.

For high-availability HAProxy I've found ECMP to be by far the most simple way to achieve very reliable redundancy, with a side benefit of more or less infinite horizontal scaling (depending on what routers you're talking to). I've served hundreds of gigabits with this model, and it works well. You can even scale it out on a global level by utilizing anycasting.

It works pretty simple. Run bgpd on your HAProxy boxes talking to your router. Each HAProxy box advertises your VIP, and your router will load balance this via ECMP. Should a HAProxy machine die, the BGP announcement gets withdrawn and traffic flows to the remaining proxy servers still advertising the VIP.

The only thing left to do is get some sort of service monitoring going that can automatically down bgpd should haproxy die/otherwise mess up on an individual machine. Add a couple checks in to ensure there is always a "path of last resort" should you have a bug in your app or monitoring code, and this proves to be very resilient, scalable, and is something nearly anyone can troubleshoot in a very short amount of time. It also works well in a cloud type on-demand/devops model - as it's extremely easy to simply spin up additional haproxy machines and have them automatically announce their configuration via bgpd.

I looked at ucarp however, as far as I understand, it only looks if node is alive/down, not if particular port or service is down.