The shape of this kind of graph usually isn't very meaningful by itself for showing subtle patterns -- you're right that the extremes of the graph always look like that. In the original article, if you line up the Bing-vs-Google and Bing-vs-Live graphs, the intercept near the middle of the graph is a little further left. That's all I got from it. I'm assuming there's an ANOVA table associated with this study that we're not seeing, and the probabilities there must be a little more compelling.

Maybe a two-phase Turking might be interesting, too: first have the turkers fill come up with a few parameters for each query (e.g. current events, general info, celebrities, pr0n), then compare search engine results for those queries. That would help pick out the more subtle patterns that you were looking for in the original graph.

With only 6 to 8 participants, the study definitely didn't have the power to find all but the grossest differences between query types. It's not about study design so much as the need to recruit more people.

Hey - I left a post on your blog, but I don't agree with your assessment. Generating a similar shape with a random process doesn't imply that our data is has no signal, and if you actually lay out the graphs side by side you'll notice that our graph is somewhat shifted to the right.