Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: A/B tests on low traffic site – is it possible?
2 points by faustas on Dec 31, 2020 | hide | past | favorite | 5 comments
Did you have any success with A/B testing on low traffic landing page?

Without for weeks or even months to see conclusive test results (95% statistical confidence).




Instead of A/B testing, why not try and figure out why the traffic is so low on the landing page? Seems like there’s SEO work to be done?

In my last job we struggled with this. It was more useful, frankly, to go with our gut and qualitative information from sales inquiries we got (questions like “how did you learn about us” or “what made you think to contact us”). We also had a PR firm interview clients to understand how they perceived our brand.

For better or worse we used these qualitative bits of info to try and build useful landing pages. But a lot of it was unfortunately subjective and involved debate over how our marketing actually worked.


It really depends what you mean by "low". It's certainly possible at the hundreds-per-day type volume. The key is to just look for bigger effects. If 200 people show up, 10/100 convert on A, and 25/100 convert on B then B wins (p < .01)

As your traffic scales, you can look for smaller and smaller effects. But the whole idea of testing variants can be applied pretty damn early.


If you flip a coin four times and get tails every time, do you conclude that flipping a coin isn't random?

Obvious answer: the more traffic, the better your results.


Yes, I know that. But there are many "suggestions" how to do AB with low traffic. Test significant changes, measure micro conversions instead of macro and etc. So I'm just wondering if all that is just "theoretical" things ir maybe there are some hacks that work in a real life...


Is this a bigger priority than increasing traffic?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: