Actually modern switches have 400ns latency. Is that too slow? Hardware PTP support in your switches can get you clocks synced to close to a nanosecond accuracy, surely that's good enough?