I don't think self-reporting is unscientific, but I do think it's not particularly reliable. Reported loneliness may correlate with actual loneliness, but they're different phenomena.
This was really driven home to me in doing product development and user testing. I've worked with some great user researchers, and they were extremely skeptical about self-reported characteristics. The classic example is, "Would you buy X?" People are terrible at answering that correctly and honestly. But there were a bunch more where we carefully rigged situations so we could measure behavior rather than self-reported internal state, because behavior ended up being a better measure of internal state for us. I could well believe the same is true for loneliness.
I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. We currently may not have the tools to usefully read that physical state. We may not have a blood test for loneliness, but we may one day have a brain scan for it. Or, perhaps sooner, a pattern in the sort of data that will be gathered by the iPhone 15m (the "m" being for medical, of course).
> I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain.
Absolutely. But in order to map those experiences to those 'physical states' you have to trust self-reporting. I agree with you that questions such as "Would you buy X?" are incredibly unreliable when it comes to self-reporting, so I understand the skepticism, but I don't think loneliness falls into that category, because it's not something that can be externally tested (we can't observe the person to see if they're "truly" lonely or not).
Hmm. I'm not following your logic. It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable.
That said, I'm not as confident as you that it's not externally testable. These researchers, for example, believe they have found EEG test results that correlate with loneliness. It would seem to me that loneliness could change both self-generated behavior and response to stimuli, which could yield reliable measures. Sociality is a biologically mediated phenomenon, so there could be metabolic measures, like how cortisol is an indicator of stress. We expect loneliness to happen in the brain, so aside from EEG measures, perhaps we could find an FMRI test that exposes it.
> It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable.
I'm not disagreeing with you on that--there's always the possibility someone's lying to you, and that has to be designed into your study as best as possible. It's difficult. My point is more that we don't have any other choice if we want to do a scientific study of things like loneliness. Or anything else that only exists within the context of one's own head for that matter.
> That said, I'm not as confident as you that it's not externally testable.
Perhaps it is. I'm only wary of these things because it's very easy to find correlations. Studies of depression are full of this. You can find a certain brain region or gene or hormone level or whatever correlated with depression, but then you find someone who also feels depressed who doesn't exhibit those symptoms. So do you tell that person they're not "really" feeling depressed? Of course not. And fMRI studies are open to abuse too; I'm sure you've heard of the "dead salmon" study, but if not: http://www.wired.com/2009/09/fmrisalmon/
I'm actually optimistic, as you are, that there will be better diagnostic tests for a whole range of mental conditions/experiences/illnesses/etc., but the core criticism still holds: if you want to trust a diagnostic test of a mental state, ultimately, at some point, you will also have needed to trust the self-reporting of that mental state.
I think we broadly agree, but I can't go as far as agreeing with this: "If you want to trust a diagnostic test of a mental state, ultimately, at some point, you will also have needed to trust the self-reporting of that mental state."
I think it's true we can generally trust people's self reports. But we all know that's not always true. There's the classic comedy staple:
A: You sound upset.
B: I AM NOT UPSET!
But there's more subtlety there, too. Anybody who deals with children has had the experience of understanding what the child is feeling better than said child does. And I remember being on the other side of that as a kid. People would ask me how I was doing, and I would self-report whatever answer I thought they wanted, not my actual state, which was often mysterious to me.
And although that has lessened as I grow older, I still can look back on key moments as an adult where I thought I was feeling one thing but, upon later reflection, realize it was something else.
So I think the best we can do is to come up with a series of external tests that correlate well with self-reporting and with each other. With that, yes, I think we can tell people, "No, you're not really depressed." For example, in the months after I lost my mom to a brain tumor, I thought that I was depressed. I went to see a therapist and she said, "No, you're mourning an important loss. It's supposed to be like this." And she was right.
This! We find focus groups nearly useless; folks are so eager to please. You get nothing but "Yes this is great. Maybe change the color to yellow". But you learn nothing about what would really appeal or sell.
I'm thinking focus groups should change to some sort of psychological test situation. "You can take only one of these home for free. Which one do you want?" I.e. perceived scarcity (cost) would elicit honest responses.
Yeah, I have never found focus groups of any value. I don't understand how one separates social effects from actual effects. And many user researchers agree. E.g.:
User focus groups may not be useful, but user testing can be. I'm thinking of studies where you give users some tasks to do, and watch how they interact with the UI to accomplish that.
As part of an ongoing study I have been conducting my whole life, I have found self-reported cleverness, self-reported attractiveness, and self-reported indications that "the guy over there wants fight me" to all be positively correlated with alcohol intake.
This was really driven home to me in doing product development and user testing. I've worked with some great user researchers, and they were extremely skeptical about self-reported characteristics. The classic example is, "Would you buy X?" People are terrible at answering that correctly and honestly. But there were a bunch more where we carefully rigged situations so we could measure behavior rather than self-reported internal state, because behavior ended up being a better measure of internal state for us. I could well believe the same is true for loneliness.
I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. We currently may not have the tools to usefully read that physical state. We may not have a blood test for loneliness, but we may one day have a brain scan for it. Or, perhaps sooner, a pattern in the sort of data that will be gathered by the iPhone 15m (the "m" being for medical, of course).