I think that in almost all cases, the 'professional' design is going to perform better -- where I think you'll find the conflict in advice is whether or not it's worth it to spend the expense on an untested idea.
If the idea holds enough water, it doesn't NEED a good design (re: Craigslist, tarsnap, the original Digg, etc.) -- it just needs to work. If it works, and you can prove there's a business there, then by all means, spend the money to make it perform as well as you possibly can. Design in this sense is an 'optimization' technique, not part of the core product.
The flip side, is if you have a horrible idea, and you spend $10k on design, you've effectively thrown that money away.
The exceptions to this, of course, are in the demographic. If your target market is web designers, they're going to be hypercritical of any design, so it needs to excel from day 1. If your startup is an iPhone app, same thing. Otherwise, most people will be forgiving of a spartan design, so long as the damn thing works, and that'll give you leverage to design against.
Also, another big point in this is that as you build the product, you'll change how things work, a lot. In the application I'm building now, I had Photoshop mockups for every page, and how I thought they would look, but in the initial testing, I've changed almost every page somewhat dramatically. It costs me time, which I hate, but it's better than costing me time AND dollars I'd have to pay a professional designer to re-design because I changed the position of button x. And, while I hate having to re-implement, I'd much rather not move into production with button x in the wrong place because I couldn't afford to pay a designer to move it.
I know of a direct-response guy who had horribly ugly sales pages that literally included misspellings. He tells this story about how all kinds of marketing consultants and designers would come in and say, "Your sales page is so ugly! You need a redesign." He would laugh and say, "Everyone says that. But I have data from the last 5 years, and this is the single best-performing page of all." He made $25m last year.
Sophisticated direct marketers are not to be trifled with.
However, if you're building a company that depends on other non-direct factors like brand, you can't really A/B test that. You just have to make a decision.
A few years back, I met a guy whose girlfriend worked on the optimization team at Amazon. Now THEY are sophisticated. They knew how A/B tweaks would affect users several months later. But even still, Amazon often makes strategic decisions that can't always be justified with data.
Bottom line: It's important to know the advantages and disadvantages of various types of marketing, including direct/brand/etc. But you will have to make some tough choices.
Personally, on my sites, I've seen lots of interesting results with testing designs...but I finally told my staff to stop micro-testing since we'd optimized the hell out of some of our stuff and were only getting incremental gains (e.g., going from 35% opt-in rate to 39% is prohibitively difficult). The biggest results I've gotten have been from strategic changes like changing the offer, adding a new product, pricing changes, back-end offers, deeper customer research, etc.
Well said Ramit. There was a great interview of Jeff Bezos by the now defunct Portfolio Magazine in which they asked him how important A/B testing is to Amazon's success. He said it was important but that still many decisions, such as launching the Kindle, need to "come from the heart".
In this age of analytics, I think a lot of small startups are overlooking the gut decisions they need to make and focusing too much on optimization (e.g. this HN question).
This might not be the interview referenced above (kinda brief). Wired acquired Portfolio's content? Or was it their progeny?
"Portfolio: Are you always extremely methodical about major decisions?
Bezos: With business decisions, yes. With personal decisions, I find that my methodical nature can confuse me, and so I think more about personal decisions, like what job you really want to take or whom you want to marry. Although I did have criteria for that...
Portfolio: What's a gut call you made?
Bezos: Amazon Prime. It's an all-you-can-eat buffet, $79, that gives you free two-day shipping on everything you buy for a year. When you do the math on that, it always tells you not to do it.
Portfolio: One of your big initiatives, a search engine called A9, fell flat. What happened?
Bezos: If you decide that you’re going to do only the things you know are going to work, you're going to leave a lot of opportunity on the table. Companies are rarely criticized for the things that they failed to try. But they are, many times, criticized for things they tried and failed at."
I suspect you're referring more to visual design, correct? If you mean design in the broader sense then comparing "Professional" design with "Non-Professional" is like asking if you can test the difference between a product made by someone who knows how to build a product versus someone who doesn't. Anyone building a product is, in fact, designing. Whether they are good at it is another thing entirely. Design is combination of "how things work" and "how things look" (and I'd even argue the latter is still a child of the former).
Visual design as decoration is probably less effective, and I think you'll get a lot of that with "Unprofessional" designers. Visual design as a way to amplify/enhance existing well-designed interactions is definitely effective (i.e. strong button affordance, good use of color to convey meaning, etc). This is the type of output you'll get with a "Professional" designer.
Philosophy aside, I think you'd probably do better with qualitative testing. Do people tend to complete tasks easier if the action being tested "looks" easier? Does the brand identity match the audience you're targeting? Are your design patterns consistent across the site? Etc, etc. It's certainly a bit more nebulous but I think with the right questions you could certainly use A/B testing to answer some of these things.
I think it comes down to usability as opposed to design. I've seen some absolutely beautiful sites (professionally) that nobody could figure out how to navigate, and thus had a low conversion rate. Yet a site that look straight out of a 14-year old's bedroom hosted on Geocities could return more results since the call to action was very clear.
The book "Emotional Design" (http://www.amazon.com/o/asin/0465051367 ) has lots of studies about how people complete tasks better and have better experiences with functionally equivalent products that are well designed. The beginning has a well-cited example of how customers could use Japanese ATMs better when they were made "pretty". Suspecting it was an artifact of the Japanese cultural love of beauty and craftsmanship, they tried it in Israel, which was more culturally hard-nosed and pragmatic. The improvement in usability was even more pronounced.
It's not really a variable that can be isolated in any effective way. If you did a massive test across 1000s of sites you might be able to extract some useful data, but even then I think the psychological effect varies quite a bit depending on what kind of site it is to begin with.
Maybe this will help - Has anybody dared to do any A/B testing on professional vs "unprofessional" design?