I like this idea! And not only because I have a long-running art project called This Like That.

But I'm not sure about using Mechanical Turk. If the goal is to have unbiased opinions, is that a good place to source them?

I would assume MT participants are trying to complete tasks as fast as possible. If I'm on MT and you ask me "A or B?" my incentive is to give you an immediate answer. I have no incentive to align that answer with my actual opinion.

Also I wouldn't expect MT to give me a particularly random sample. I mean, if I'm wondering how my B2B SAAS logo is perceived, is an answer from MT going to overlap much with the opinions of my target audience?

It seems like a fun thing to try as a toy, like rolling dice only with humans. I'm just not sure I'd use it for any question I really cared about.

Make it useable by asking people to answer questions in exchange for getting to ask their own question?

Reaching critical mass may be harder, but I think it would be more random than mechanical turk.

Agreed, answering in return for ask credits is a good idea. Maybe 10 answers = 1 question or something like that. If you make the ratio too large then people will just select anything to get their credits. Another option is to only award a credit if their answer was in the majority.

You can also charge people for the ability to ask a follow up question like why they made the decision they did.

Heh, I built something similar years ago. Even got a YC interview. It was called decisioncandy. Originally targeted logos but switched to apparel...

I think the Slack idea is brilliant though, and I could totally see that taking off for a wide variety of things.

Super interesting. Any other ideas from your testing, or feedback from the YC interview on where there could be traction here? If you're willing to share, of course :)

Hmm certainly willing to, but I don't have much. Mostly just that professional graphic designers didn't seem like the right audience since they already have networks of peers who can give better feedback. Oh, and the apparel-industry-specific stuff from below.

But something like an engineer asking a colleague which web design they just hacked up looks better would be a much better fit.

I had an idea once about something similar, I called it Design Picker, where you'd basically use some combination of crowdsourcing and "algo" to help you make design decisions. I had the domain for years, but I could never come up with a version of the concept I thought actual designers would actually use, so I never built it.

Did your YC feedback yield hints as to why you were rejected (I am presuming)?

Yes, a very clear answer! The problem we were solving was that apparel firms have really long lead times: they design their lines for a season 9-12mo before the customer buys it. That means they can't iterate on their designs; DecisionCandy would let them do so in a lightweight way, by showing their sketches/previews of pieces to their customers, and doing a "which do you like more" survey to score the items.

The problem here is that it takes 9-12mo to find out whether the app does any good, and YC is a 3mo program! We were only just about to sign a pilot deal at the time, so the timing wouldn't have made sense.

And indeed, that long of a feedback loop is realistically too slow for a VC startup. We closed up and I "pivoted" to something totally different a few months later.

Hm.. I worked on a piece of software with similar needs of polling random people for their opinion. Have you considered using Amazon Mechanical Turk?

I imagine some bias would be there, but probably manageable. In return you'd get the ability to poll way more than 50 people for a very reasonable price.

It says on their site they are using Mechanical Turk:

> To do this, we take your question and use Amazon's Mechanical Turk platform to survey 50 random people.

Thanks! This is why I keep saying that all my good ideas are already taken! ;)

I participate in a poll/survey portal as a reviewer. Just to give you an idea of how it works: we the participants are free to register and get some points after every completed poll/survey (the amount of points depend on how long it took). With these points we can access to a "shop" of small prizes, like headphones, small cameras, books, etc. I assume that the companies making the polls are the ones paying and probably they select the demography of the people to survey (we're asked about this info on the polls). Maybe a model like that can work for you too.

Love this idea but are the surveys randomized or quality controlled at all? I have tried 4 surveys, all different and the results seem weirdly consistent. Option A always seems to win out with ~60% of the vote (60%, 64%, 60%, 57%). I even reversed the two images in one test and whatever was in position A won.

Thanks for flagging this! I had quality tested a month ago (nothing scientific) and tried multiple different image variants, placement (A vs. B) and they did remain consistent. That said, I refunded your orders while I take a look at quality control. Thanks again for trying out and letting me know.

Perhaps entrepreneurs wanting feedback on ideas could be good niche to start with?

If there's anything I learnt from discussing my ideas with family, friends and others, it's that finding the right people to bounce your ideas off of is crucial.

I think Will Wright (famous game designer) described this phenomenon well. To paraphrase, every time he would try to describe a new game to someone else, that person would create a game design in their head and play their game in their head, and then tell you what their thinking is. However, what you want, is someone to play _your design_ in their head. That's very hard to do without a prototype. For example, basically everyone told him that idea for The Sims is terrible. Only when they played his game did they see the light.

Interesting. How do you incentivise the opinion givers to actually consider the options instead of just efficiently (randomly) selecting an answer?

How about a reward for choosing the "correct" answer.

If the side with the most votes is the side you chose, you get 2 points. If the side with the most votes is not the side that you chose, then you get only a single point.

Have you tried removing the free trial?