It bugs me that DK reached popular consciousness and get misinterpreted and misused more often than not. For one, the paper shows a positive correlation between confidence and skill. The paper is very clearly leading the reader, starting with the title. The biggest problem with the paper is not the methodology nor the statistics, it’s that the waxy prose comes to a conclusion that isn’t directly supported by their own data. People who are unskilled and unaware of it is not the only explanation for what they measured, nor is that even particularly likely, since they didn’t actually test anyone who’s verifiably or even suspected to be incompetent. They tested only Cornell undergrads volunteering for extra credit.
If DK is regression to the mean (a view I find convincing) that doesn't mean the effect isn't real; i.e. one would still observe that people of low ability overestimate their ability, simply because there is more "room" for overestimates than underestimates. And v.v.
Put differently, if everyone's estimate was exactly the mean, you'd still see a "DK effect".
I’m not sure I understand. If the effect shown in the paper is regression to the mean, then that does mean the paper doesn’t actually demonstrate what it claims to, right? I mean you can argue that the idea is still plausible, but this would mean that the paper doesn’t support the claim that low skill people overestimate themselves, right?
It’s also an interpretation to focus on unskilled people as the explanation. DK’s data shows the very same effect on highly skilled people. The people in the top quartile were just as bad at self-estimating as the bottom quartile, yet the paper claims only the unskilled people were unaware!
I recommend reading the DK paper. It didn’t test any people of low ability, and it did not evaluate skill in absolute terms. The sample size was tiny. The kids who participated were all earning extra credit in a class (it’s a self-selecting population that might have excluded both A students and F students.) The students were all Ivy League undergrads who might all overestimate their abilities precisely because they’re in a prestigious school and their parents told them they’re great. The paper didn’t test any actual low IQ population. The paper has methodology problems when it comes to non-native English speakers.
It absolutely blows my mind that the paper is held up as evidence for some kind of universal human trait with such miniscule and completely questionable evidence. I have no doubt that some people overestimate their abilities in some situations. Like you, I’m sure, I’ve witnessed that. But as a commentary on all of humanity, I’m becoming convinced that the so-called DK effect does not exist, that they didn’t show what they claim to show. It doesn’t help that many replication attempts have not only failed to replicate, but have ended up showing the opposite effect: that for many kinds of skilled activities, people.
It bugs me that DK reached popular consciousness and get misinterpreted and misused more often than not. For one, the paper shows a positive correlation between confidence and skill. The paper is very clearly leading the reader, starting with the title. The biggest problem with the paper is not the methodology nor the statistics, it’s that the waxy prose comes to a conclusion that isn’t directly supported by their own data. People who are unskilled and unaware of it is not the only explanation for what they measured, nor is that even particularly likely, since they didn’t actually test anyone who’s verifiably or even suspected to be incompetent. They tested only Cornell undergrads volunteering for extra credit.