From 3 years ago but it certainly still holds true. As mentioned in the comments, many PPC'ers use http://www.splittester.com to help determine if there's enough data to determine a winner.
SEOBook recently opened another site focused primarily on PPC at PPCBlog.com and this was a recent discussion in the forums.
Which should be a pretty big wake up call to anyone not doing the math. 1 in 20 events happen all the time, yet naively reading the results makes it look like one is almost certainly better than the other.
Agreed, this is such a common problem. Another really common manifestation is when you A/B test while looking at multiple variables. If you observe the results of 20 variables, you should expect one of them to have 95% confidence!
Kludgey is right; and it won't always be accurate: numbers might happen to agree on a too-small sample; or diverge on a large-enough sample.
But I think it's a very simple, concrete and credible demonstration to users. And if they observe such an expt over time, over a few trials, they'll quickly develop an intuition for when the sample size is large enough. And this is more valuable than an abstract calculation. (Of course, ideal to have both.)
Unfortunately, the empirical approach doesn't take into account the effect of how near the probability is to 50% vs the extremes 0% or 100%; nor of the variance of the population. However, in practice, these are probably similar enough over all ads and populations for the effect to be negligible - especially if you give yourself a margin of safety.
Can't hurt to try this approach, as it takes approximately 10 seconds to set it up. Will give it a shot, and report back if the results are interesting...
SEOBook recently opened another site focused primarily on PPC at PPCBlog.com and this was a recent discussion in the forums.