See Peter Norvig's article "Warning Signs in Experimental Design and Interpretation"

http://norvig.com/experiment-design.html

for Warning Sign I9: Being Too Clever, which relates directly to the point of this thread.