A number of years ago, I set up a similar configuration with two OpenBSD boxes (which support PF and CARP out of the box) running Apache + mod_proxy_balancer for really granular load balancing and routing. It was a super-flexible and cheap way to route traffic to a mid-sized app server cluster, and generally worked really well.
We did have some network issues after the new topology went live that we unexpectedly tracked down to the LB pair, for one simple reason: CARP generates a lot of multicast traffic. Depending on how your hosts and network are configured, this can easily get routed out into a fairly large portion of your local network, and use a lot of bandwidth/router capacity with no benefit.
So, if you're setting up a CARP pair/cluster of your own, pay close attention to your multicast setup. Ideally, put the CARP multicast traffic on a dedicated subnet and watch your router and switch stats to make sure you aren't flooding the rest of your network with multicast spam.
Also if you're setting up in a virtual environment keep in mind that it requires promiscuous mode enabled.
This stopped the show for me once when setting up two OpenBSD load balancers in a shared virtual environment. I was told that to enable promiscuous mode on a single port group they would also have to enable it on the physical ESX host adapter for each ESX host since the pair was separated on different physical hosts.
If that is true then I would never enable it. However networking isn't my strong side so I can't verify this.
Well generally you're going to be putting multicast traffic on a default subnet of something like 224.0.0.0/4.
It's pretty easy to filter that out using iptables as well. I know for keepalived, you're generally only sending out notifications every second or two. Anything higher than that and yeah, you could end up generating some crazy traffic.
Here's a typical tcpdump of an advertisement for the curious:
172.20.1.xx > 224.0.0.18: AH(xxxxx): VRRPv2, Advertisement, vrid 51, prio 104, authtype ah, intvl 1s, length 20, addrs: 172.20.1.xx
We did have some network issues after the new topology went live that we unexpectedly tracked down to the LB pair, for one simple reason: CARP generates a lot of multicast traffic. Depending on how your hosts and network are configured, this can easily get routed out into a fairly large portion of your local network, and use a lot of bandwidth/router capacity with no benefit.
So, if you're setting up a CARP pair/cluster of your own, pay close attention to your multicast setup. Ideally, put the CARP multicast traffic on a dedicated subnet and watch your router and switch stats to make sure you aren't flooding the rest of your network with multicast spam.