Doctor A decides to test a cure on 50 patients. 40 have gotten better. Doctor B independently decides to test the same cure on his patients. He will stop once he has reached 'significance'. Coincidentally, the results become significant at the 50th patient, and he also has a 40/50 success rate.
Doctor A says "I followed a fixed testing procedure, and the statistical analysis says that my data is not significant. We need more experiments."
Doctor B says "I followed an optional stopping procedure, and the statistical analysis says that my data is not significant: the cure is good."
A Bayesian would claim that if they both have the same data, then they should reach the same conclusion, regardless of their intent.
A "frequentist" would uphold that the doctors can legitimately disagree. I don't know much about frequentism, but it's the dominant perspective in statistics. Everything I've read about A/B testing is frequentist.
