I know of a direct-response guy who had horribly ugly sales pages that literally included misspellings. He tells this story about how all kinds of marketing consultants and designers would come in and say, "Your sales page is so ugly! You need a redesign." He would laugh and say, "Everyone says that. But I have data from the last 5 years, and this is the single best-performing page of all." He made $25m last year.
Sophisticated direct marketers are not to be trifled with.
However, if you're building a company that depends on other non-direct factors like brand, you can't really A/B test that. You just have to make a decision.
A few years back, I met a guy whose girlfriend worked on the optimization team at Amazon. Now THEY are sophisticated. They knew how A/B tweaks would affect users several months later. But even still, Amazon often makes strategic decisions that can't always be justified with data.
Bottom line: It's important to know the advantages and disadvantages of various types of marketing, including direct/brand/etc. But you will have to make some tough choices.
Personally, on my sites, I've seen lots of interesting results with testing designs...but I finally told my staff to stop micro-testing since we'd optimized the hell out of some of our stuff and were only getting incremental gains (e.g., going from 35% opt-in rate to 39% is prohibitively difficult). The biggest results I've gotten have been from strategic changes like changing the offer, adding a new product, pricing changes, back-end offers, deeper customer research, etc.
Well said Ramit. There was a great interview of Jeff Bezos by the now defunct Portfolio Magazine in which they asked him how important A/B testing is to Amazon's success. He said it was important but that still many decisions, such as launching the Kindle, need to "come from the heart".
In this age of analytics, I think a lot of small startups are overlooking the gut decisions they need to make and focusing too much on optimization (e.g. this HN question).
This might not be the interview referenced above (kinda brief). Wired acquired Portfolio's content? Or was it their progeny?
"Portfolio: Are you always extremely methodical about major decisions?
Bezos: With business decisions, yes. With personal decisions, I find that my methodical nature can confuse me, and so I think more about personal decisions, like what job you really want to take or whom you want to marry. Although I did have criteria for that...
Portfolio: What's a gut call you made?
Bezos: Amazon Prime. It's an all-you-can-eat buffet, $79, that gives you free two-day shipping on everything you buy for a year. When you do the math on that, it always tells you not to do it.
Portfolio: One of your big initiatives, a search engine called A9, fell flat. What happened?
Bezos: If you decide that you’re going to do only the things you know are going to work, you're going to leave a lot of opportunity on the table. Companies are rarely criticized for the things that they failed to try. But they are, many times, criticized for things they tried and failed at."
I know of a direct-response guy who had horribly ugly sales pages that literally included misspellings. He tells this story about how all kinds of marketing consultants and designers would come in and say, "Your sales page is so ugly! You need a redesign." He would laugh and say, "Everyone says that. But I have data from the last 5 years, and this is the single best-performing page of all." He made $25m last year.
Sophisticated direct marketers are not to be trifled with.
However, if you're building a company that depends on other non-direct factors like brand, you can't really A/B test that. You just have to make a decision.
A few years back, I met a guy whose girlfriend worked on the optimization team at Amazon. Now THEY are sophisticated. They knew how A/B tweaks would affect users several months later. But even still, Amazon often makes strategic decisions that can't always be justified with data.
Bottom line: It's important to know the advantages and disadvantages of various types of marketing, including direct/brand/etc. But you will have to make some tough choices.
Personally, on my sites, I've seen lots of interesting results with testing designs...but I finally told my staff to stop micro-testing since we'd optimized the hell out of some of our stuff and were only getting incremental gains (e.g., going from 35% opt-in rate to 39% is prohibitively difficult). The biggest results I've gotten have been from strategic changes like changing the offer, adding a new product, pricing changes, back-end offers, deeper customer research, etc.