This is a ridiculous study because it implies that the order of when cases are heard is randomly distributed. That is, a "hard" case is just as likely to be first in the morning session as it is to be fifth in an afternoon session, or vice versa.
The data is easily explained as judges taking clear case of leniency first, then doing the tough ones (that could go either way), and ruling on the clear cases of parole denials later in the session. The judges have hopefully done their homework in advance and have read the case files and petitions, so they come in with at least a rough sense for which cases are in each bucket. There's likely a significant degree of agreement on that metric among the members of the panel, so it's not surprising that cases are heard in that order: clear grants, ones that require discussion, and clear denials.
The lesson learned seems to me to be when the evidence shows a result this large, take a really hard look at your study before you publish it.
And mentioned in podcasts by people[1] who really, seriously, obviously should know much better (based on how famous and otherwise competent they seem to be).
I haven't read the specifics of the study or refutations, but knowing how court scheduling works, it's likely closer to randomly distributed than not.
Edit: just the scheduling, people. I'm not claiming the effect isn't due to that sort of thing, just that judges can't easily arrange their calendars so that "good" defendants are in the morning and "bad" ones are in the afternoon.
The study was about Israeli parole boards, not US court cases. The study was flawed from the beginning because they incorrectly categorized favorable/unfavorable decisions. However, ignoring that:
The daily court schedules were arranged to minimize the amount of time lawyers and prisoners spent waiting in court. Each lawyer would usually represent several prisoners, and would go through all of their clients consecutively. The court would schedule the lawyers and clients starting after meal breaks. After all lawyers and their clients that were scheduled for a block of time had gone through, then they would fill the remaining time until the next meal with cases by prisoners without lawyers.
Cases with lawyers had a 35% favorable outcome rate, while cases without lawyers would have a 15% favorable outcome rate. In addition, each lawyer would present their best case first and worst cases last.
So yes, the daily calendars were arranged such that the first prisoner after a meal break was extremely likely to get a favorable outcome, and the prisoner right before a meal break had only a 15% chance of a favorable outcome.
Court scheduling in US Courts isn't at all random. Even for cases on the same day, it's up to the judge to figure out in what order he or she will handle the appearances. And like this Israeli parole board, those with attorneys generally go first and it's not uncommon to have two or three matters grouped together and heard consecutively.
I did a bit of science as an undergrad and a bit more in grad school. I learned a few things, chiefly, that I suck at statistics and won't make a good formal scientist. But the other main takeaway was that when results look strikingly obvious and "beautiful" they're usually just wrong. I did GIS so the statistical data was usually in the form of big colourful maps. And when those maps told such an obvious story, they were really telling me that I messed something up in my processing.
Not that this is universally true, but it's what I found to be pretty common over the years.
>"I suck at statistics and won't make a good formal scientist"
Most really good science got done before statistics was even a thing (starting in the 1930s-1940s). Stats has very little to do with being a good scientist.
Depends on the field. Now that we all realize that you can't actually conclude that you've found something until you've thought about how likely your result was to occur randomly, only a few stragglers and the "lucky ones" where effect sizes are gigantic (you either land on the moon or you don't) can get away without it.
What does random mean? It just means you didn't include every possible influence in your mathematical model of whats going on. Instead you used some distribution of "noise" as an approximation to whatever was really going on.
Eg, if you know everything about airflow, gravitational, electrical, etc forces on a coin throughout a flip, you can predict which side it will land on with near 100% accuracy. Most of the time we don't have info about that available so use the binomial distribution approximation.
>"where effect sizes are gigantic"
Most science has nothing to do with "effect sizes", in fact I'd say concern about "effect sizes" is indicative of very rudimentary science. Advanced science is concerned with making precise and accurate predictions.
>What does random mean? It just means you didn't include every possible influence in your mathematical model of whats going on.
In physics, it is fundamentally impossible to predict the outcome of wave function collapses/pick your favorite interpretation's verb. CERN doesn't do statistics on their histograms because they don't have good models...
That aside, does the difference between a lack of knowledge and a lack of predictability really matter? Either way all you can do is calculate the odds.
>Most science has nothing to do with "effect sizes",
True, that language is not used everywhere. However if your department deals with signals that are hard to distinguish from background then you have some way of expressing that idea in your vernacular, whatever it is.
>"In physics, it is fundamentally impossible to predict the outcome of wave function collapses/pick your favorite interpretation's verb."
This is one interpretation of quantum mechanics. Plenty of physics is done without any consideration of this.
>"CERN doesn't do statistics on their histograms because they don't have good models..."
I believe the histograms you are referring to were done on the data, not on the predictions. And afaik they actually do lack a good model of the higgs boson mass, since they never predicted an exact mass.
>"However if your department deals with signals that are hard to distinguish from background then you have some way of expressing that idea in your vernacular, whatever it is."
Distinguishing signal from background is the same thing, its rudimentary stuff you do when you can't predict exactly what you are looking for. The key difference is that in advanced science you check for deviations from the model you believe is correct. No one at CERN believed the standard model without a higgs boson they used for the background reflected reality.
There are zero valid interpretations of quantum mechanics where you can tell me which particles will come out of an event, even in principle. It seems silly to suggest that describing your certainty in your results is incompatible with "advanced science," when the reason you're doing it is because of a direct law of nature. The Higgs boson might have never been observed - there is a certain chance that it was just lucky photons all along.
If you are in a field where you don't need statistics, it is not because they aren't at play, but instead because they all have -9999 in the exponent and you have implicitly decided not to worry about them.
>"There are zero valid interpretations of quantum mechanics where you can tell me which particles will come out of an event, even in principle. It seems silly to suggest that describing your certainty in your results is incompatible with "advanced science," when the reason you're doing it is because of a direct law of nature."
There is no fundamental uncertainty in GR predictions. There is none in the stefan-boltzmann law, etc. In many cases whatever QM uncertainty exists will be negligible next to measurement error and so is not considered. Acting like QM uncertainty has anything to do with 99% of physics is nonsense.
>In many cases whatever QM uncertainty exists will be negligible next to measurement error and so is not considered.
If you are in a field where
you don't need statistics, it is not because they aren't at play, but instead because they all have -9999 in the exponent and you have implicitly decided not to worry about them.
Now you seem to be saying "a field that doesn't need statistics" only doesn't need it because the error due to QM is negligible compared to other sources. How do you account for all the advances before QM and statistics?
If you have predictively modeled an error source you can subtract it, and then you're left with error sources that you have not modeled. If this is repeated indefinitely you will end up with transistor noise and quantum effects which cannot be modeled. As a result you will always have some aspects of your experiment that must be dealt with statistically. If a field is getting by without expecting its contributors to do statistics it is either a straggler before we realized this aspect of rigor or it is a case where the effects are so large that a nonrigorous detection of them works well enough.
Discoveries made before the introduction of error analysis turned out to either be lucky or wrong - a pattern repeated every time we all realize that something was being overlooked. It's worth pointing out that the astronomers of yore probably knew that their measurements weren't perfect and that averaging them was a good idea, which counts as statistics.
>"If a field is getting by without expecting its contributors to do statistics it is either a straggler before we realized this aspect of rigor or it is a case where the effects are so large that a nonrigorous detection of them works well enough."
It makes no sense to be concerned about sources of prediction error that are orders of magnitude smaller than your measurement error. Such concerns would only waste time and money for no benefit. This has nothing to do with "straggling" or expected "effect size".
It sure sounds like you're describing large effect sizes to me. The size of the effect you're measuring is large compared to signal uncertainty, and incidentally your systemics are also high.
You compare the effect sizes with the measurement uncertainty. For example if I wanted to conclude that a solenoid worked I could correlate piston position with applied voltage, and I might not need to worry about statistics because the motion of the piston is very large compared to the uncertainty in my implicit visual measurement of its movement. Admittedly that's not how anybody thinks intuitively, but it is correct.
>"For example if I wanted to conclude that a solenoid worked I could correlate piston position with applied voltage"
No one would do this, its once again a "rudimentary science" behavior. People have figured out what we need to know to get the exact relationship between voltage and position.
A great deal of older science was done by explaining what was already invented. We seem to have reached a point where rarely does a new invention throw off previous science without it being based on new science that was already in disagreement with old science. Part of it is because over the last century we had so many people with the freedom to invent that the easy inventions have already been invented, at least where the inventions don't pose significant moral/legal implications (such as someone inventing a new drug and getting people to try it without going through the proper channels).
>"We seem to have reached a point where rarely does a new invention throw off previous science without it being based on new science that was already in disagreement with old science."
Or perhaps science has become so institutionalized that any truly novel ideas are suppressed either directly or indirectly (eg the funding for that type of thing is already maxed out).
Not much of an artifact - during my BLAW101 I was told that in North America judges really don't like people intentionally dragging on cases, but can't do much about them other than give the innuendo of them raising white flag - something along lines "you babbled for 3 hours, for sure we have enough argument from you that the worst case scenario ruling is now out of the question"
Am I the only one who feels sharper and more adapt while somewhat hungry? Food makes me lethargic. I've never been sympathetic to ideas that "you should always eat breakfast!". If possible, I'd prefer to skip breakfast and lunch.
My idea: our ancestors were opportunistic savanna hunters. Why would we be unable to maintain homeostasis with respect to cognition even without food? Our ancestors would have needed their best cognition when they lacked food!
Optimal alertness/health is at a point between "paralyzed with starvation" and "paralyzed by overstuffedness". What you are calling "somewhat hungry" may be better stated as "full, but not overstuffed".
To maintain a consistent energy level, 5+ small snacks is better than 3 large meals.
"To maintain a consistent energy level, 5+ small snacks is better than 3 large meals."
Anecdotally, that's not been my experience, except in the time periods when I've been transitioning between feeding on that short schedule and feeding once a day.
I recognize that this experience is wholly anecdotal, but the only time that I've felt "paralyzed with starvation" is not when I am actually in starvation mode (that is, 48 or so hours of not eating, which I do every couple of months), but rather when I've been eating often and switch to eating less often.
That is to say, when I used to have candy and donuts and juice as part of my diet, then I would occasionally crash and lots of smaller meals prevented that crash. In that case, your advice is good.
In other cases, it doesn't fit my experience.
When I am in a period where I am eating a single large meal every 24 hours or so (mostly rice and beans and some other plants with a lot of fat), then I don't feel those crashes at all. I don't eat outside of that meal except for a short period of time before or after. I occasionally feel hungry, but I generally on-point mentally, and certainly not "paralyzed with starvation".
If I get out of that pattern (usually because I am traveling or around folks who prevent that kind of eating from happening) I will often need to go through a longer fast to get back into that daily eating schedule.
So my hypothesis (untested, of course) is that most folks have never gone more than 24 hours without eating, so if they are crashing after 3-5h then that's specific to their diet and not a universal human constant.
I'v always found it amusing that there is no word for English meaning "not hungry". There's only "full". It exists it many languages alongside "full" and there's a clear distinction between them.
Maybe you have a food sensitivity, and your body is having e.g. a mild allergic reaction. On the other hand your sympathetic and parasympathetic nervous systems are at odds, so if you try to get amped up right after eating you will get a stomach ache as your body moves blood away from your stomach and you basically stop digesting food for a while.
I often feel more clear headed and direct in my reasoning when I am slightly hungry. Once I eat, I'm more likely to entertain implausible ideas, play "what if..." and it feels easier to argue for both sides of any idea.
"The effect size is too large" seems like a weak heuristic too me. Feynmann talks about the kind of problems such a heuristic can cause:
> It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
> Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...
A more theoretical argument is as follows. Suppose we systematically remove from analysis any data point which are more than 1.5 times the IQR below the first quartile or about the third quartile. (https://en.wikipedia.org/wiki/Outlier#Tukey's_fences) Points censured by such a rule are often called "outliers," but because we do not yet know the true underlying probability distribution this terminology is misleading. Then our estimates for skew, kurtosis, and all higher moments will be biased downward, and furthermore the censured sample is more likely to "pass" normality tests such as Shapiro-Wilks which can lead to us applying inappropriate models. However, what is the interpretation of a model when applied to a new observation? If the data point is well within the Tukey fences, then we can use the model to make a valid prediction. However, if it is outside the fences, we can say nothing. What then is the population mean of the predicted value? It is the weighted mean of the values for those data points for which our model is valid and those which lie outside of it. But those that lie outside of it can be enormously influential.
An example can make this concrete. Suppose an actuary estimates the amount of an insurance payouts for house insurance. 99% of the time, these effects are small, a few hundred or a few thousand dollars. But 1% of the time they are much larger, perhaps $500,000 or more. If the actuary applies Tukey's fences to his data and fits a model, he may infer that 1% of claims payout each year, and those payouts average $1,000 dollars. So he sets the price of the insurance at $20/year and makes a 100% profit margin. Until the first house burns down, and his company goes bankrupt.
The point of this is merely to illustrate the dangers of ignoring any effect which simply seems "too large." The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds? The change in resistance of a semiconductor in the presence of an electric field seem "impossibly large" until the solid state physics is carefully analyzed - would we reject this clear and highly reproducible evidence of a qualitative change in behavior because it is too evident?
A much better approach is to scrutinize the methodology for, say, omitted variables. This is what the more detailed criticisms he cites in the third paragraph do. Or to attempt to reproduce the study, which is perhaps the strongest approach.
“The effect size is too large” is not the heuristic, and casting the argument that way does it a disservice. The article gives concrete examples of effect sizes that are impossibly large. Examples of penicillin and insurance payouts are not remotely germane to the discussion… the point of “impossibly large” is that the effect is so large that it would almost certainly have other effects that we would notice. This is the difference between “impossibly large” and “surprisingly large”, which your argument ignores.
The exhortation to look at the mechanism is also critical, and ignoring the mechanism is exactly how you end up with the worst inferences… if something is “impossible” or even just “surprising” according to your model you should probably reëvaluate your model.
The point being made isn't that large effects are impossible, or that high-d results should be dismissed outright. Rather, it's that for sufficiently large d, subtlety becomes impossible. Results will either be overwhelmingly obvious to any observer, or false. It's worth giving that list of sample effects in the article another look. In particular, the studies showing that juries usually make decisions matching what the majority of jurors started out believing is d=1.62. This study claimed that hunger was having a bigger effect on trial outcomes than whether people believed the charges were true. It was still worth looking for the confounding effect, certainly, but when people start making claims that disproportionate to comparable claims, simply refusing to accept a causative claim is an entirely reasonable decision.
The effects of penicillin are overwhelmingly obvious: for some set of diseases, penicillin produces rapid and complete recovery even in otherwise-terminal patients. I don't know the d for "penicillin treats staph", but in pharmacology d=5 isn't considered unreasonable for precisely these reasons.
The average d in psychology, though, is around 0.4 - and that was before many of the 'strongest' results fell apart in recent years. This study's effect size wasn't inherently insane, but it was claiming confidence in the realm of "antibiotics kill bacteria" for a subtle and previously unobserved force on judge's decisionmaking. It absolutely should have raised questions like "if this effect holds, why aren't trial lawyers acutely aware of it and working to manipulate it?"
What you are saying is reasonable but it is not what the author proposed. This is your rephrasing:
> The point being made isn't that large effects are impossible, or that high-d results should be dismissed outright.
But what the author actually says in the concluding paragraph is that certain results are to be dismissed outright and moreover the burden of proposing a plausible mechanism is wholly on the experimenter:
> Implausibility is not a reason to completely dismiss empirical findings, but impossibility is. It is up to authors to interpret the effect size in their study, and to show the mechanism through which an effect that is impossibly large, becomes plausible. Without such an explanation, the finding should simply be dismissed.
He does not say, "other experimenters should attempt to reproduce the result" or "theoreticians should explore the phenomena mathematically" but only "the finding should simply be dismissed." Dismissed! Note the stark contrast: while you emphasize that such results should not be "dismissed outright," that is exactly what the author calls for! No follow up studies, no novel theoretical models proposed, just blanket dismissal; furthermore, in the authors own words, the test used to divide those results which are dismissed is "implausible" vs. "impossible."
But who can so finely demarcate the implausible from the impossible? Certainly not the contemporary scientists, who in every age we find railing against newly discovered theories and inconvenient facts. And why would the experimenter who first notices a phenomenon also be required to at once to provide a satisfying theoretical explanation? The role of the experimenter is careful lab work. Was the mechanism behind penicillin understood immediately? Was the quantum physics of the depletion zone fully understood when the strange electrical properties of doped silicon were first observed? When half of the neutrinos from the sun go missing, the experimenter must simply attend to his instruments to ensure he has not made an error, and then throw up his hands and let the theoreticians get to the bottom of the thing. This is not the only way science can proceed, but it is an important way: surprising empirical observations, outside the realm of current theory or even directly in contradiction to it are announced decades, or centuries, or in a few cases (such as ancient supernovas) millennia before they are fully understood.
If the author had been content with making a specific criticism of this one study, I would not have objected. The study is almost certainly flawed, and various good reasons why it is flawed have been discussed. But the author seeks to elevates this principle to that of a general maxim (as the concluding paragraph makes clear), recommending it to one and all as as a valid and useful form of inference. But it is not (in the general case) a valid "form" of inference, as we see by swapping out the details of this one flawed psychological studies for several others, such as penicillin, the field effect, or neutrinos. In those cases, arguments of the exact same form would have resulted in the dismissal of legitimate data. It is often not a good idea to reject experimental results just because they seem "impossible." Concretely, we may analyze the authors proposed method as a three step process:
1. Before the experiment, decide what is possible and what is impossible. (We fix a hypothesis space. Anything outside of this space is given zero a priori probability.)
2. We conduct the experiment and observe the outcome.
3. If the outcome of the experiment is impossible, we "dismiss" the result. (We update no posterior probabilities.)
It is difficult to see how science could have progressed if such a rule were universally applied. For example, Rutherford was absolutely shocked when a certain percentage of alpha particles were reflected from a thin sheet of gold foil back at an angle near 180 degrees:
> "It was quite the most incredible event that has ever happened to me in my life. It was almost as incredible as if you fired a 15-inch shell at a piece of tissue paper and it came back and hit you."
Yet, because he did not follow the author's advice and dismiss these incredible observations, the nucleus was discovered.
Kuhn argues that when this situation occurs in the history of science, what actually happens is that the old theories are discarded and new ones are proposed and validated until even the "impossible" observations are now explained. (A paradigm shift.) I would suggest, and I think Kuhn would agree, that in step #3 we should not discard the observation, but rather discard the theory used in step #1 that led us to conclude that it was "impossible."
Quoting the article again: "It is up to authors to interpret the effect size in their study, and to show the mechanism through which an effect that is impossibly large, becomes plausible. Without such an explanation, the finding should simply be dismissed."
Rutherford was able to come up with such a mechanism in his work with Niels Bohr. Atomic theory was quite new when Rutherford made his discovery: in a sense this was low-hanging fruit. The article's point is that this effect size is so big that if it was real, any half-decent lawyer would aready know about it. Without an explanation for why everyone somehow missed this point, the claim is pretty much invalid.
I think the right analogy is if someone comes up with evidence for a speed of light that does not agree with the standard measurement. If the claim is to be taken seriously, not only do they have to be rigorous in their experiment - they also need to explain why all other measurements were incorrect. This stuff happens all the time in science. Experiments are sloppy, and results are difficult to replicate. The go-to reaction should be to throw out a result, unless it can be replicated time and time again.
Kuhn's description of a paradigm shift included room for 'problems' that can't be explained with the confines of normal science. You only start throwing things out when the evidence against you becomes unassailable.
I read a book where a planet would blink out of existence occasionally, but long went undiscovered because all the high tech scanners and computers kept throwing out the "impossible" readings as corrupt data.
I'm glad I read that book so I could grasp what you're saying here.
> The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds?
I think you miss the point -- the efficacy of penicillin is obvious to anyone, whether they've taken statistics or not.
The authors point, as I understand it, is that if the effects are so very large they should be obvious to a=even a casual observer. If the effect is so large it's impossible for them to be surprising.
It's not entirely just a heuristic, but does have a Bayesian interpretation in many circumstances.
Looking at the study on judge's sentences before and after meals, you wouldn't expect to see a giant effect, but seeing some kind of an effect is a priori plausible, since we know many people are irritable when hungry.
When you see an unusually large effect size, it's often the case that there is a smaller effect, but you happened to get some extreme data. You definitely shouldn't jump to the conclusion that the effect size is large if you think it is a priori unlikely to be so large (unless you have very high statistical power). As you say, the results shouldn't be discarded, though.
edit: If an effect is incredibly improbable (like a relation between country music and suicides), evidence can still sway the belief against it. However, extraordinary claims require extraordinary evidence.
Indeed it can. Why, if there were really effects as large as you claim -- where several hundred thousands of dollars of value are lost in a single incident -- then society would recognize that fact. We would have formed "Fire Departments" consisting of professionals skilled at minimizing the damage. We would have created "Homeowner's Insurance" as a way of distributing the harmful effects throughout society instead of concentrating it on a few. It is precisely because we DO observe these behaviors that we can nod and say "Yep, those 'outliers' are to be expected.
On the other hand, if you had never heard a story about an entire house being destroyed in a single incident and someone tried to tell you that the statistics showed it was possible, even likely... wouldn't you stop to ask questions? Wouldn't you feel like a well-done paper on the subject needed to address possible mechanisms by which the information might be less well-known?
«If a psychological effect is this big, we don’t need to discover it and publish it in a scientific journal—you would already know it exists.»
On the one hand, shoudn't we? There are a lot of supposed facts that need to be checked. Also, the size of the supposed effect everybody knows still needs to be measured, if, for instance, comparisons between different populations can be made at all.
On the other hand, and personally, I find it to be common sense never to have difficult conversations on an empty stomach.
This article, which boils down to "if the data seems unlikely from what we know beforehand, throw it the hell out and never look at it again", is the exact opposite of what scientists are taught, and what they are taught - not this article - is the correct approach.
This is a very poor and incorrect reading of the article here. The point is not that the data were unlikely and should be thrown out; it's that if the effect size reported was accurate, that would affect so many other things so strongly that the current state of civilization refutes the estimated effect size from the study.
It's not that the data was poor, but that it and the analysis was incomplete and ill considered.
In psychology in particular, large effects will seep into everyday life well beyond the phenomena being studied, which gives a very useful check on astronomical effects like those originally reported on judges and time of day.
GP says, and I concur, that the scientific method would not to be just say "this must be impossible, let's ignore it", but try to disprove the study (maybe by trying and failing to replicate the study, or designing better studies).
The critical part of science is the actual data and methodology. If you do X you will see Y is what constrains new theories. Conclusions on the other hand have very little weight.
The article, your comment and the argument they make, that this effect -- if it exists -- should be highly visible across all strata of society, are taking a ton of assumptions for granted. There are a number of plausible scenarios in which this argument falls apart completely:
a) The act of ruling over criminal proceedings places one in a very unique position of power over complete strangers' lives which only judges routinely experience. What subtle effects on empathy, vindictiveness, leniency do minor fluctuations in mood have that are exacerbated by the unique conditions of this profession?
b) Judges are members of an extremely privileged professional class and universally come from extremely privileged backgrounds. The mechanism of the hedonic treadmill might explain how working class folks cope with regular day-to-day stresses with almost no impact to their mood where being peckish at midday is likely the harshest reality of life that a financially secure judge making $100 an hour has to deal with and may disproportionately sour their temper to a comparable degree that someone making minimum wage worries about paying their rent.
Etc etc. There is no compelling proof of this argument.
The data is easily explained as judges taking clear case of leniency first, then doing the tough ones (that could go either way), and ruling on the clear cases of parole denials later in the session. The judges have hopefully done their homework in advance and have read the case files and petitions, so they come in with at least a rough sense for which cases are in each bucket. There's likely a significant degree of agreement on that metric among the members of the panel, so it's not surprising that cases are heard in that order: clear grants, ones that require discussion, and clear denials.
The lesson learned seems to me to be when the evidence shows a result this large, take a really hard look at your study before you publish it.