Aaron Brown checked a sample of the papers Haidt relies on to build his case and found that they couldn't support his claims. They're of the sort of quality we've come to expect from the field, dominated by tiny unrepresentative samples, bad methodologies, bad statistics, weak signals and so on.
Unfortunately Haidt's initial response was to make a circular argument, saying you can't demand a high quality of evidence for the crisis because there's a crisis, and to engage in a subtle form of gish gallop by saying that surely not all 300 papers he selected can be bad even if a subsample were.
I didn't express "concerns" about specific studies; I argued that the majority of the 301 papers cited in his document are garbage. I went through each category of studies on Haidt's list, chose the first one that studied social media and depression to get a random sampling, and then showed that they were so embarrassingly bad as to be completely useless. They were guilty of coding errors, fatal defects hidden in mid-paper jargon, inappropriate statistics, longitudinal studies that weren't longitudinal, experiments in name only, and red flags for hypothesis shopping and p-hacking (that is, misusing data analysis to yield results that can be presented as statistically significant).
"A bad study is like a bad mortgage loan," I wrote in my original piece. "Packaging them up on the assumption that somehow their defects will cancel each other out is based on flawed logic, and it's a recipe for drawing fantastically wrong conclusions."
It's quite probable that there is no teen mental health crisis.
This fits what I've read. Data driven decision making is the way to go, but good data is hard to come by in these fields. We have no baseline for many parts of physical health, let alone mental health. Every time a pop culture scientist tries to justify their world perspective without taking the source of data seriously, it does the field as a whole serious harm.
Brown never denied that -- He argued that there's not much high-quality evidence it's caused by social media. This is clear from the title alone ("The Statistically Flawed Evidence That Social Media Is Causing the Teen Mental Health Crisis").
> Haidt's claims about a teen mental health epidemic have been pretty convincingly debunked [...] It's quite probable that there is no teen mental health crisis.
I don't see how you can reach this conclusion when it's very clear in the data that hospitalizations for teens due to self-harm has gone up significantly. Are you suggesting that hospitals are not now or were not previously correctly reporting this data?
Self harm is a behaviour, not a disorder. There isn't a predictable relationship between the rate of self harm and the rate of underlying psychiatric pathology. We have reasonable evidence to suggest that self harm tends to be socially contagious, occurring in socially-connected clusters rather than consistently across the population; young people whose friends engage in self-harm are dramatically more likely to engage in self-harm.
It might seem contradictory, but it is entirely plausible to argue that the internet is causing a self-harm epidemic rather than a mental illness epidemic. Obviously that would still be a serious problem, but it would lead us to very different conclusions and interventions.
A fair counterargument, but I think that the line delineating "psychiatric pathology" is not as clear as you imply. Many such pathologies are arguably socially constructed and sometimes even contagious in similar ways (see the book "Crazy Like Us").
I think "being vulnerable to social contagion to an extent that leads you to literally inflict harm on yourself" is definitely within the scope of being classified as a psychiatric pathology, even if it hasn't been yet. It seems like there's little difference between this and, say, anorexia.
Haidt presents datasets that are very short. The graphs begin in 2004. Teens have been depressed for a lot longer than 20 years. It's hard to know what the natural variance of this statistic is given just his articles.
If you look at longer term US suicide stats (stratified by gender but not age) there's quite a bit of variation over time.
The 1970s were really rough for women, but not men. 1990 was rough for men but not women. These graphs show that variance can be pretty high when taking the long view. Current overall suicide levels aren't anything special, even though they've been going up for decades (watch out for the non-linear X axis).
If we look at the 10-14 girls dataset that Haidt picks where the change seems most dramatic, it goes from 1 per 100,000 to 2 per 100,000. Very few people commit suicide at that age, fortunately. We're talking on the order of 100-200 girls in that age range per year in the entire USA. At these low initial levels, even very small changes will look big on a graph. Every death is tragic, but if "crisis" now means something that affects such an truly tiny percentage of the population then almost anything can be a crisis.
The other unrelated issue is that if you look at his graphs, self harm and suicides started going up in 2007, not 2012 as he claims. I don't see how you'd pick 2012 as the inflection point if not trying to force the smartphone narrative.
> Haidt presents datasets that are very short. The graphs begin in 2004. Teens have been depressed for a lot longer than 20 years. It's hard to know what the natural variance of this statistic is given just his article
I'm not talking about depression, but self-harm. Yes the graph starts in 2004, and then we see a 188% increase in self-harm among girls, after being basically stable for at least 6 years. That's pretty dramatic.
> Current overall suicide levels aren't anything special, even though they've been going up for decades (watch out for the non-linear X axis).
Even supposing I accept that suicide rate hasn't changed much, a stable suicide rate with increasing self-harm rate still qualifies as a mental health crisis. You keep focusing on suicide when the self-harm rate among girls was 20 times higher than their suicide rate when it was stable 2004-2010, but is now 72 times higher. That's clearly a problem.
> The other unrelated issue is that if you look at his graphs, self harm and suicides started going up in 2007, not 2012 as he claims.
That is not correct. The graph clearly shows that 2007 was at or slightly below 2004 levels. You can also see a slight increase from 2006 to 2007, and then a ~25% decline in 2008, then another ~10% decline in 2009, then a return to the 2007 levels in 2010. After 2010, you can see increases past historic levels, and 2012 you see a very sharp increase. (edit: I'm talking about Figure 2 at the link, just so we're on the same page)
That might be a coincidence, but it's not completely arbitrary as you're implying.
https://reason.com/2023/03/29/the-statistically-flawed-evide...
Aaron Brown checked a sample of the papers Haidt relies on to build his case and found that they couldn't support his claims. They're of the sort of quality we've come to expect from the field, dominated by tiny unrepresentative samples, bad methodologies, bad statistics, weak signals and so on.
Unfortunately Haidt's initial response was to make a circular argument, saying you can't demand a high quality of evidence for the crisis because there's a crisis, and to engage in a subtle form of gish gallop by saying that surely not all 300 papers he selected can be bad even if a subsample were.
https://twitter.com/JonHaidt/status/1641850836287356930
Still, to his immense credit Haidt engaged seriously with the criticism and so Brown wrote another response:
https://reason.com/2023/05/30/not-every-study-on-teen-depres...
I didn't express "concerns" about specific studies; I argued that the majority of the 301 papers cited in his document are garbage. I went through each category of studies on Haidt's list, chose the first one that studied social media and depression to get a random sampling, and then showed that they were so embarrassingly bad as to be completely useless. They were guilty of coding errors, fatal defects hidden in mid-paper jargon, inappropriate statistics, longitudinal studies that weren't longitudinal, experiments in name only, and red flags for hypothesis shopping and p-hacking (that is, misusing data analysis to yield results that can be presented as statistically significant).
"A bad study is like a bad mortgage loan," I wrote in my original piece. "Packaging them up on the assumption that somehow their defects will cancel each other out is based on flawed logic, and it's a recipe for drawing fantastically wrong conclusions."
It's quite probable that there is no teen mental health crisis.