I can't share the actual data, but the method of calculating the confidence was the standard formula from calculating the confidence for the Z-score as used in http://abtester.com/calculator/, Google Website optimiser, etc etc. The numbers did check out, it was just an unlikely fluke, which is why I don't trust the confidence interval as much as I used to...
Before running this test, did you estimate the sample size needed to detect the estimated difference in effects from A vs B? Since the expected difference is 0, the necessary sample size would be infinite. Therefore, we could at least estimate a very small effect, which would necessitate a very large sample. I would be very cautious, then, about trying to interpret the "confidence interval" before you had accumulated that large sample size. If you're going to peek before getting to that point, then you should almost certainly have implemented an alpha spending function (or similar) in order to maintain the desired overall error rate.
This should not reduce your belief in confidence intervals; this is a great, motivating opportunity that should prompt you to learn how to use them correctly.
No, we hadn't estimated the necessary sample size. To be honest, nobody expected the confidence metric to go so high for a test with no changes, which is why we were surprised. We've studied various A/B testing resources and we haven't seen this "peeking error" mentioned anywhere, so we'd appreciate any resources you may have (and would possibly summarise them in a subsequent post).
Sure thing. I just posted a bit more detail about this elsewhere in this thread. Also, googling "alpha spending function" brings back some useful results. (I can't paste into the HN text box with the latest Chrome dev, otherwise I'd paste some links here for you.)
