You probably don't need a degree in experimental psychology to understand that forced recruitment gives you invalid/biased/low quality data. These surveys are annoying to the users; more importantly, the data provided has little if at all value to the organization behind the survey. No matter how well you design your survey.

I don't even want to start on how bad/biased some surveys are designed.

The only way the survey business is still in business is because it is so distant from the revenue. If your ad campaign doesn't boost sale, you pull your ad away. Bad survey's uselessness are not so explicit so people still trust that "something can be gained".

No, stop creating useless data from online user. If you want insights on usage, do analytics on server logs. If you want feedback on user experience, do a serious experiments and plan to spend some ten thousand bucks on it.

For instance, many girls like dolls, but if you don't like dolls that doesn't make dolls a "No." business, and it's not illuminating to hear why dolls are icky. A better discussion would be how and what makes a new doll appealing (or unappealing) and whether it can compete with Barbie.

Clearly the online survey industry is making some money. Certain parts of the web, which may not target you, rely on surveys. Combining surveys and paywalls, if not perfectly novel, is at least interesting. The combination raises a host of issues and potential problems which your comment ignores.

Your being off-base would not quite matter as much if it wasn't so rude to the team that worked on this. Did they do a good job in their implementation? I don't know, it hasn't come up yet.

Not sure how the "doll" analogy applies here. When a doll sells, even to a small proportion of population, it sells. When you do an online survey, you gain very little true insight. You definitely will gain more insight from the two alternatives I mentioned (analytics on logs, properly designed experiment).

Don't get me wrong, I am not saying the whole online survey industry is wrong. Amazon Turk is wonderful, so is SocialSci. My company actually use result from Turkers for building models. The premise for that to work is a well designed experiment with qualified turkers. A survey that replacing "paywall" basically means you do not have a clear targeted recruitment poll. You get basically anyone who pays your website a visit. Survey result from that, in my humble opinion, is worth as much as server log analytics.

p.s.: I think you meant "You are being off-base" instead of "Your being off-base" but I really want to avoid discussion on grammar on HN.

Good. Because you either have your apostrophe or your verb tense wrong here in your original post:

Bad survey's uselessness are

It's best, in an international community, to let these things slide without comment.

For most purposes, I'd say this is good enough, but this is a whole other can of worms.

Interesting point. Anecdotally, I am less honest with my selections if I'm taking a survey specifically to get a reward (i.e. contest, Kongregate points, access to a site), as I usually just want to get it done with as fast as possible.

However, if it has tangible results (i.e. will tailor my experience on the site like Netflix) then I am likely to spend more time choosing honestly.

http://www.youtube.com/watch?v=RCqT3fdAAHQ

While there are abusive uses like push polls and leading questions, there are legitimate uses too.

We certainly hope that the surveys are not annoying. We know that some people will prefer to pay, but others (like me) would rather take a few seconds to complete a survey than pay. Our hope is that the surveywalls will let users access content that otherwise would have been beyond their reach. And at the same time, quality content publishers will be able to make money off their work.

As to the researcher side, we agree with you that nothing beats revealed user behavior when optimizing web sites or apps. But it's not cheap or even possible to A/B test in other situations.

Suppose that you're a restaurant owner and want a new sign for your building. It's not practical to purchase two signs and see how alternating the signs affects business on different days. And it's also not in budget to spend $10k or more on a traditional market research survey.

Or if you're a politician, you can not wait until voting day to see which of your various ad campaigns worked in different districts. You need proxy measurements.

Companies already ask these types of questions using traditional approaches like panels and phone polling. We provide a cheaper way for them to do it online. All approaches have built in biases, and we're working to account for the biases and quality issues in online polls. As you point out, we'll have to do that to succeed.

A few suggestions: 1. Hire someone with experience on experimental psychology, I mean, by trade. Not necessarily a PhD but at least with some experience and has knowledge of the textbook experiment errors. Just one of those would help the design a long way. 2. Throw in some free analytics tools/result for the organization behind the survey. As you mentioned, restaurant owners and politicians might just want to run the survey for some straightforward result, well, not all surveys have straightforward result, even when designed well. Some analytics/visualization tool, e.g., manyeyes from IBM, doesn't really take a lot of dev time but would be quite impressive in the eyes of your customers (i.e., the politicians or restaurant owners).

Best of luck, Alex

We do plan to add polling and survey design experts to our team (I have stats but not polling background). And an analytics platform is coming.

What do you mean by our survey "might take the user by surprise"? Do you mean that it might be surprising to the user to have a survey launch when clicking one of our links? We're testing different "teaser" text for the links. And hopefully we can make it clear that a survey will be coming when you click.

If you're willing to run a test survey, fill out the contact us form on our site, and we'll give you a discount code. Appreciate the feedback.

Also, please can you take it easy on the tone? See this discussion - http://news.ycombinator.com/item?id=4396747

Edit: Analysing server logs can tell you what people did, not why they did it. At least in my experience, surveying users directly about their motivations and needs has been very enlightening and resulted in a direct increase in sales.

one comment on the survey vs. server log comparison: Survey also doesn't give you "why" on user behavior; it only give you what the user "thinks" the reason is. For certain occasions it provide false insights. Server logs are facts, you won't get why but you get concrete observations.

1. Our minimum spend is only $10 vs. $100 for Google; lower minimum brings in a group of people who want to get a "quick read" on an issue (e.g. ask 100 people an opinion on a new logo design or tagline)

2. Starting soon, we'll be offering advanced behavioral targeting. For example, you could select a target audience of active young mothers, Honda car owners, or online shoppers.

3. We're allowing multi-question surveys (up to 4 questions). Each respondent answers all questions in your survey, which enables cross-tabulation of responses (e.g. looking at how respondents who answered "Yes" to question 1 respond to question 3). Google is currently focused on 1-2 questions at a time.

In the coming months, we hope to differentiate ourselves in other areas, in particular data quality.

That said, you could probably more easily approach smaller (more local?) publishers that are more willing to experiment to get traction. Is that what you're planning on doing?

While we hope to work with big publishers too, niche blogs are an awesome place to start. As you say, they provide a self aggregated group of individuals with a common interest. They're perfect for polling.

The quality of data that will be generated from these surveys is going to be extremely poor and I would be interested in seeing how they're able to sustain advertisers (market researchers).

That being said, I'll be interested to see what, if anything, makes this different than all of the surveys running in the affiliate marketing/lead generation space.

Data quality with initial partners has been good. We'll be working hard to keep it that way.

Survata should be aware that any longer than that, or poor content hidden behind the survey-wall, means that I'm likely to just click random options on the survey. But I'm sure they are.

Good Luck to them!

Really, someone 'just' needs to fix micropayments.

It also seems like this could be useful for free versions of iOS apps, as advertisement replacements (or supplements.)

I work for a company which sells wine online, and a large number of people have strong preferences about the types of wines they like (red, white, chardonnay, half-cases, etc.)

It's useful to give people a survey - combined with an explanation of how the site works - as they are signing up. Even more useful is to use their answers to show the new customer a wine which is suited to their tastes. (If you like high-end reds, it wouldn't make sense to show you a half-case of inexpensive white wine to convince you to make you first purchase.)

And if our marketing department can manage their survey, marketing copy, and graphic design without having to ask developers every time they want a change (or if they can save me the time of writing a customizable interface) then all the better.

Onboarding. That's where it's at, yo.

The idea of surveys as a paywall mechanism (eg, New York Times paywall with a survey) is, at first glance, pretty unappealing, because paywalls are de-facto abhorrent. But we'll see; I may not mind filling out a 4-question questionnaire in order to read an article that would otherwise be blocked. Especially if I know that after filling it out once, I will never ever have to fill the survey out again.

And let's not forget about the adult industry! Instead of captchas... well, why not get some interesting marketing analytics out of the deal?

Interested to see how this evolves!

And, just to be clear, we're keeping the surveys fully anonymous; so we will not collect or tie to an e-mail address or other personally identifiable information.