> 2. WAN sucks. Where I work, I've seen mysql slaves get hours out of sync due to packet loss.
Have you tried the new BBR congestion control for TCP?
We've been seeing similar issues with PostgreSQL WAN replication and found that our TCP connections get throttled prematurely by the Cubic algorithm way before (0.5Mbps) the available bandwidth (~20Mbps) is reached and actual congestion kicks in.
From our research it seems that switching to BBR on our replication origins should completely solve this problem, but we haven't had a chance to put it into production yet (gotta upgrade Kernels first ...).
I'm just throwing this out there as an idea for you, but perhaps also because I hope to hear from those who have already had a chance to apply BBR to DB replication over pipes with elevated (0.1-1%) packet loss.
I'm still in the progress of setting up my product, but my day job does mysql replication. The issues come from when there is a good 5 minutes of 80% packet loss. It's caused some major issues.
Have you tried the new BBR congestion control for TCP?
We've been seeing similar issues with PostgreSQL WAN replication and found that our TCP connections get throttled prematurely by the Cubic algorithm way before (0.5Mbps) the available bandwidth (~20Mbps) is reached and actual congestion kicks in.
From our research it seems that switching to BBR on our replication origins should completely solve this problem, but we haven't had a chance to put it into production yet (gotta upgrade Kernels first ...).
I'm just throwing this out there as an idea for you, but perhaps also because I hope to hear from those who have already had a chance to apply BBR to DB replication over pipes with elevated (0.1-1%) packet loss.