I didn't quite get why the article described these surveys as problematic. I skimmed the linked article in that section but this didn't provide any answers either. Assuming the participants fully understand the question, and the survey is designed well, what cultural aspect is stopping people from answering?

I grant that with a questions like "Do you often do X?", examples are necessary to specify what "often" means.

From the article:

> Some people may refuse to answer. Others prefer to answer simply yes or no. Sometimes they respond with no difficulty.

That just sounds like some people boycott the Likert questions, but we don't know why.

I don't know what reasons other people have for refusing to answer. As a bit of pedant I just wanted to explain why I personally don't like them, it has a lot to do with being vague, abstract, unclear. Confusing really. You sit there and are not sure you can even answer the question honestly.

In a normal situation I would probably refuse and ask them to clarify, or challenge their assumptions or something. In this situation I just try to get over my aversion to answer and try to interpret it as best as I can. Sometimes I got no clue what it could mean and choose neutral or whatever. It's mentally exhausting.

This doesn't invalidate the methodology. Just saying I don't like it. If people refuse to answer or answer in a way that doesn't fit the schema and they throw it out, that's obviously a problem though.

Edit: To maybe answer your question, I skimmed through the actual article and they mention things like old people not culturally accepting a young person administering questions, children thinking they should not speak in the presence of elders. But also other weird things as mentioned that seem to me as stemming from confusion. I mean, you get better at answering with practice, so the whole concept might be totally baffling to some people which haven't gone through a Western school, where you're also quite often expected to answer unclear and confusing questions on tests.

More edit: The PhD students and professors designing these questions and the college students used as test subjects are basically the most overschooled people on the planet. They are the ones that, in school, excelled at answering abstract, underspecified, questions with multiple choice answers. Many have seen these before. They might even know how they are made and how the results get processed. Of course they're going to have less trouble answering.

Agreed. They are like 'which color represents your opinion on this complex subject?' I don't know what to say - I agree with some of it, disagree with another part, and have no opinion about how it applies to others anyway.

Then there are the 'Have you quit beating your wife?' questions, posed to make it a dichotomy but I'm somewhere else entirely.

Surveyors and surveyees being in the same pool of people often is probably bad, I agree.

My conclusion is that the usability and accessibility of empiric testing could use some work. Even in regards to WEIRD people :D

The problem with these Likerts is that they are often an uncalibrated scale, so the subject is interpreting the question as well as the possible responses. They are also used in medical studies of pain, for example. You've seen them- the little faces in the doctor's office indicating differing amounts of discomfort and you pic the one that describes you). My buddy and I had a good laugh thinking up how to calibrate it :)

I'm not even sure how you test something like that for reliability.

These tests have other questions that intersect and determine how much you actually mean it by taking it from different angles. The MMPI-2 for example, may touch on the same subject from various degrees and angles to get the overall picture.

Not saying it's right, but not saying it's a singular question "do you believe X agree? slightly agree? etc." it's a bit more deep and nuanced than that. And the statistics tend to back it up.

You are right, they do this, especially in the more tedious longer ones.

I think improving these things is difficult, because of you change the questions, you also loose all the accumulated data, and maybe now you might have a better questionnaire, but you wouldn't know it, and you wouldn't know what the results mean. I mean, until you administer it to just as many people as the old one. It's a lot of work.

That's why starred-review sites have mostly reverted to thumbs-up and thumbs-down. The biggest trial of Likert items in history has spoken.

... and people have started complaining about those reversions to thumbs up or down (https://www.polygon.com/2017/4/7/15212718/netflix-rating-sys...). And in any event, why would you use the same thing for everything?

The amount of armchair criticism of this sort of thing always surprises me.

There's literature on the the number of response options and it basically says:

People respond quicker with binary options.

You lose information though. People get pissed there's no nuance available (no "maybe" for example). They refuse to answer because they can't say anything other than thumbs up or thumbs down.

As you increase the number of options, your ability to predict increases. Predict the same thing later, predict other things, etc. It stabilizes though.

It does take longer for people to respond, but then they complain about what "slightly agree" means.

Regardless, though, you can ask people to do it, and if they just do it, you can predict from it.

Even when you let people omit responses and refuse to respond, you can model that refusal to see what it predicts and then use their refusal anyway.

You can model how people use the response options in different ways, and in general it doesn't matter too much.

You can even eliminate rating scales altogether and use entirely different systems (forced-choice between different statements), but those don't actually work much better either.

Yes, you can always respond in a cheeky, subversive,or manipulative way, but then that's an entirely different issue altogether. You can always do that, and all you've shown is that you're smarter than a rating scale?

As for the original article... these criticisms have been made for decades. Decades. These are my general impressions:

1. The problem of western focus in behavioral sciences is a problem of western focus in the sciences period. Many behavioral scientists would love to do research across multiple sites but cannot afford to do so. And this doesn't stop all sorts of other biomedical research from being done on very narrowly defined groups of people (or animals).

2. Effects observed in undergrads generalize a lot more than the author is letting on, and effects observed in western populations generalize even more. I'm not saying it's not important to study things cross-culturally, only that the idea that people are fundamentally different in different settings is itself flawed. Many things have been examined across different cultural settings, and the differences are not all that dramatically different. In fact, in one recent replicability study, the effect of culture/sociogeographic population was one thing that didn't seem to matter that much. Some studies replicated and others didn't, but the sociogeographic setting didn't seem to matter very much.

I agree that being more sensitive to human variation is critical, but like a lot of things with behavior, there's a lot of grey areas, which people don't like to hear.

There are whole fields within psychology and the behavioral sciences devoted to these issues. People have put a ton of effort into studying them, considering all the issues being raised here as well as many others to numerous to count, and it's like all that work gets brushed aside like snow in the wind because of random blog posts and anecdotal experience.

What people complain about is maybe not the important thing. Its getting actionable results.

And what about the 'Have you quit beating your wife' part? Where I don't lie on any part of the spectrum they've drawn. Ok, that's the same with thumbs-up-or-down, but with the spectrum its in your face that you have no answer that's meaningful. I refuse to answer those kind far more often that thumbs-up.

Finally, mega-corporations wanting actionable results are not 'brushing aside with anecdotal experience'. That's the point I made. They're spending billions and want results, and more often use thumbs-up-or-down. That trial has five orders of magnitude more data than all the graduate students in history added together.

Or what about people actively lying?

I guess it would be a minority because telling the truth is easier and faster under most circumstances.

But yes, all these studies whose the authors authoritatively make claims but these are are based on people answering questions regarding themselves (how often do you cheat on your partner?) - I never believe these numbers.

Deception vs. honesty on certain questions in itself is a huge revealing factor in a person's personality.

Not really - these Likert question studies are so much easier to click through rapidly. You can double or triple your hourly rate depending on how bold you are. Would a college student cheat their psychology department out of 10 dollars? Maybe some.

I wonder how much shrinkage (wasted questionnaires) they have to account for? Pun semi-intended.

It's accounted for in modern psychological tests.

Edit: to clarify, there are many other questions that will seemingly be somewhat related or unrelated that will infer later on in the test if you were lying. People who aren't familiar with the tests won't notice it in an honest testing.