Which is why scientific tests have a placebo option.
Very loosely, what hypothesis testing is telling you is whether your results are so aberrant that they can't just be explained by model variance (coins flipped at random). The hypothesis that can't really be tested is whether your model explain most of the variance in observed data.
If I take a random sample of people, ask them whether they're male or female and what's their yearly earnings, I'll find statistically significant differences in averages; however, that will explain only a very small portion of the total variance in wages, given that these also depend on education, skills, experience...
It could be that the main source of variance in the poster's experiment is the time of the day emails are sent. How would he know? When doing crude A/B testing one hopes that all the things in the world one _isn't_ controlling (is it raining? have local sports team won trophies?) are uncorrelated with the control variable. But here it might not, there might be something very different between the batches he doesn't know about.
He makes some great arguments. Especially for the solopreneur (like me), making maximum use of time and energy available to me during the day is critical. There is risk in spending time on A/B testing, with the results of much work put towards this coming back "Inconclusive".
I could see extreme cases where you need to test a major area of the business, testing an entire market segment, etc...but then you could argue just pick up the damn phone and call people in that market segment.
Very loosely, what hypothesis testing is telling you is whether your results are so aberrant that they can't just be explained by model variance (coins flipped at random). The hypothesis that can't really be tested is whether your model explain most of the variance in observed data.
If I take a random sample of people, ask them whether they're male or female and what's their yearly earnings, I'll find statistically significant differences in averages; however, that will explain only a very small portion of the total variance in wages, given that these also depend on education, skills, experience...
It could be that the main source of variance in the poster's experiment is the time of the day emails are sent. How would he know? When doing crude A/B testing one hopes that all the things in the world one _isn't_ controlling (is it raining? have local sports team won trophies?) are uncorrelated with the control variable. But here it might not, there might be something very different between the batches he doesn't know about.
And he's proud of that!