Numerous comments about the promoter's non-scientific background, the sensationalism of the original announcement and skepticism over the findings. Please note that the scientists themselves weren't sensationalizing their findings, but others were.
As this article points out: it was a "big, if true" story that had numerous red flags from the start.
Interestingly in the old comment thread, and the other ones I read at the time, there was lots of discussion about how the phosphine might have originated non-biologically, but no one suggesting a problem in the data analysis. Now it looks like that is what happened.
It also means just about no one read the real paper...a twelfth-order polynomial would certainly be a red flag for me.
Interesting moral here for interpreting science: surprising results don't usually have interesting explanations. It's usually a bug.
It might have been read by people without the proper background. Even though I can grok the math, upon reading this I would go "oh, interesting what they resort to in this field".
Polynomial fitting can be used to generate curves which fit any dataset perfectly given enough degrees. Like if you give me a data set with 150 points in it that are apparently randomly distributed throughout the sample space, I can give you a nice high-order polynomial that perfectly passes through all 150 data points.
Making any claims based on that kind of curve fitting is a huge red flag, especially if you don't discuss that.
One of the reasons we would prefer lower-order fits to higher-order fits is that there is a very real risk of overfitting in the interior of a data set but then providing completely inaccurate results anywhere but at the data points. Seeing that kind of overfitting in a scientific paper without any justification suggests that the author of the paper is making statements without any basis in science.
It's more like someone is asking you to create a regex that matches phone numbers and they give you an example number for you to work with. You have the bright idea of writing the example number verbatim as your regex and voilà. It matches the sample perfectly, job done. In reality that regex can't be applied to anything else.
It like saying that a rule with as many special cases as data points is not actually a general rule. It is just a collection of special cases that explains nothing.
More precisely, if you give a model enough parameters, you can always have it describe your data well. The question is whether it is likely to predict future data. And the answer is no.
It's more like, high-order polynomials are fundamentally wild animals. You can use a math trick to make them to go through specified points, and to early statisticians & data analysts, this seemed like a good way to model a nonlinear trend based on a set of sample points. But that trick doesn't make polynomials tame. And it gets worse the more points you need to interpolate - you have to add a degree to the polynomial every time you want to go through another point. Each new degree makes the polynomial wilder and wilder outside of the points you're interpolating.
We later discovered that the tame functions that usually work well for extrapolating from a sample are things called splines.
I don't think so. You can easily get too many returns without any overfitting at all.
If you want to make it simple, think of using a 12th order curve as being similar to picking whatever 12 points you want to match and then drawing lines from point to point.
I'm not the person you asked, but in general with polynomials, if you need to go much past the second or third order order to fit your data, you should probably be fitting with a different set of functions. Take a look at Runge's Phenomenon for an example where higher order polynomials really aren't making the approximation better in a useful way:
Draw whatever crazy line you want on a whiteboard, and I can find a 12-degree polynomial to transform it into the desired shape for whatever it is I want to prove. It proves nothing because it can prove anything.
They reanalyze the data than in the phosphine paper.
---
If you look at figure 3, they have the data that is the skyscraper-like line, and they use a polynomial of degree 3 to approximate the signal, that is that curved unhappy smooth line.
The smooth line is (a), when you subtract this smooth function, you get the other part of figure 3, that is the noise (b). There are some high and low parts, but nothing too high or low that look special. So their conclusion is that there is no interesting part (c).
---
If you look at figure 2, top left, they use a slightly smaller interval, but now they use a polynomial of degree 12 instead of a polynomial of degree 3. This is the smooth function (a).
This is a reconstruction of the process in the original paper. They fit the polynomial using the data, but excluding the central part.
The problem with the polynomial of degree 12 is that is has too much freedom, so it fits the actual smooth curve, but it also fit the noise.
The polynomial of degree 3 has to go somewhat in the middle of the data, because it can't go up and down too many times. The polynomial of degree 12 can follow the local bumps and fit the noise.
When you subtract the polynomial of degree 12 you get the the graph in the third row, with the noise that is (b). It is copied in Figure 2, and it is very similar to the graph in the original paper.
Since the polynomial of degree 12 fit the noise, the noise is too small, so you underestimate the noise level.
And since the central part was skip in the fit, in some case you get a big bump like here. It is bigger than the apparent level of noise so it looks like an unexpected peak (c).
But here the problem is that you are comparing the peak with the surrounding noise level, but the noise level is underestimated because the polynomial of degree 12 overfit.
---
They repeat the same kind of analysis in other regions, and they get a few additional fake peaks. This are the other 5 graph it the top of Figure 3.
It means that you're not just adding up some simple curves, you're taking your variable to extremely high powers, in this case all the way up to x^12. The more orders/powers you add into a fit equation, the more it's going to get artificially closer inside your data window (as you sledgehammer it into a nearly arbitrary shape). And the more it's going to immediately shoot to infinity the moment it gets out of your data window, because those extreme powers of your variable are all fighting each other and have no real connection to the underlying data; what physical process is causing x^10 and x^11 and x^12 curves or even x^5 and x^6?
Highest I know is Lighthill's (aptly called) eighth power law which says that the sound power created by a turbulent flow scales with the eight power of the characteristic turbulent velocity: https://en.wikipedia.org/wiki/Lighthill%27s_eighth_power_law, anyone knows something higher?
The difference is that in my example and in your example there are only a few coefficients to tweak to fit the data. So the shape of the curve if fixed and it is very difficult to overfit the data.
In the paper they used a full polynomial of degree 12, that has 13 coefficients to tweak and it is very easy to get weird shapes.
Nice example! But yes, I agree. My field is certainly not astronomy but... that method to remove noise seems extremely weird. XKCD-level joke weird. Even in the rebuttal arxiv paper cited here where they use a 3rd degree poly to remove the noise it seems... not to fit very well? Seems strange to use a random fit without any guesstimate of the underlying cause/model of the noise.
> they use a 3rd degree poly to remove the noise it seems... not to fit very well
They are not trying to fit the noise, they are trying to fit the hidden smooth signal that if hidden by the noise. In some cases it is difficult to make a formula for the real signal, so you can approximate it locally with a polynomial.
The idea is that after you subtract the smooth part, you get only the noise. So you can calculate the expected noise level.
And if the "noise" has a big peak, you can guess there is something strange, like a big absorption line.
Not only are the other replies correct, in that you can 'prove' anything with enough degrees in your poly fit, but any attempt to actually use a 12th-degree fit will probably suffer from math precision errors.
Often it's better to use a spline fit or other interpolation technique, once you find yourself needing to go beyond five- or six-degree polynomial fits.
For an idea of scale: 12th order is enough to fit every transcendental function in the C math library well past the point where a human could tell the difference without doing deeper analysis. Most of them will be less than one ulp of error off.
> ... scientists themselves weren't sensationalizing their findings ...
Yeah right.
They were saying things like "please prove us wrong" while doing interview rounds, and (I'm sure) celebrating, loving and looking forward to more and more limelight, so strictly speaking they were not "sensationalizing their findings".
In a related story, OJ and Casey Anthony did not kill anyone.
If you read the paper they are very very careful with their claims. Also please prove us wrong is the default position in science, thats what peer review is all about. I dont see a problem with that.
Somewhat like the scientists you are criticizing, you ascribe a lot of meaning to little data. Support for your claims that they were celebrating and reveling in fame are very thin. I would like to see more evidence before we criticize not just their work but also them as people.
> Greaves and colleagues used a twelfth-order polynomial, or an equation with twelve terms (plus a constant, the +b in the equation), to describe the noise in their ALMA data.
Hah. Wish that had been more emphasized in the original popular reports.
There's a perpetual quest in strength sports to create an equation that lets you compare the relative strength levels of athletes in different weight classes. In weightlifting the Sinclair formula uses allometric scaling to try to roughly answer the question "if we scaled this lifter to the size of the overall world record holder, how much would we expect them to lift?" That's a decent approach. However for a long time powerlifting used the Wilks formula which was, you guessed it, a 5th order polynomial. Hilariously, there are discontinuities in the formulas. If a man bulks up to a bodyweight approaching 283.034 kg, his Wilks score approaches infinity. Same for a woman at about 208.391kg. But if that's your strategy then don't overshoot as the limit from the right is negative infinity.
If you try hard enough, you can do anything computable with a single integer parameter - if your model is a Turing machine, and the parameter encodes a program.
It's perhaps less clever or cool than the hack above (by virtue of us being used to Turing machines), but serves as a friendly reminder that you can encode a lot of information in a long enough number.
The real lesson here is that "number of parameters" is not a useful measure of information content. The only useful measure of information content is information entropy, which is the logarithm of the number of distinguishable states. The base of the logarithm is arbitrary, but by modern convention is invariably taken to be 2. The resulting unit is the bit.
Number of parameters is fine when you're not trying to be sneaky with ridiculously precise constants. And it's more expressive in certain ways than a raw bit count. For general use you can impose reasonable limits, so that one parameter can't go overboard with bits. Something like a precision cap of one part per thousand and no going over a trillion.
Your example is vacuous. You cannot determine the information content of a sequence of numbers in isolation. You can only determine information content of a system with respect to a model that tells you how to use the state of the system to distinguish between a number of possible states. The information content of the system is the log of the number of possible states from which the state of the system allows you to select one.
Yes, algorithmic information theory is a thing. But neither it nor your example refute what I said (because what I said is in fact true).
On the other hand how many parameters do we need to create text?
A simple calculation of number of parameters in the human brain divided by seconds of lifetime yields: 10^14 synapses / (72.6 * 365 * 24 * 3600 seconds) =~ 43000 synapses/second, not all of them related to language and reasoning.
So we use tens of thousands of synapses for each second of our life, which is quite a contrast to the 100 bits of information per second which is estimated for conscious processing.
I recently met somebody who read and believed the initial headlines for that story but never heard any updates. I think science reporting has some serious problems and no clear way to hold anybody accountable for it.
If there was a news site that purely had follow up to old stories, I'd subscribe. The media only provides it for things like trials that people continue paying attention to.
One reason people stall investigations is they know the press will forget after a few months.
In my case I knew of the aftermath because I work on the field and followed the reports of science writers such as Ed Yong and Carl Zimmer as well as the blog of Rosie Redfield, one of the microbiologists that took part in the testing and posterior refutation of the published results.
Re the replication crisis: "Negative opinions about psychology as a science already exist. We have the opportunity to show how science works by directly investigating and addressing the problems."
https://www.psychologicalscience.org/observer/champions-of-p...
I didn't find any others, but if you doubt my thesis, that's just the magic of science at work!
> I think "show just how science works" has become a bit of a meme for "we need to roll back this mistaken/over-ambitious claim".
You say this as if it's a small feat. The willingness to rollback a claim you made, ambitious or mundane, is surprisingly difficult for people to do. Most of science is about making claims based on the best evidence that you have with some level of confidence.
Where we run into problems, just look in the news, is when claims become invariant, regardless of new data, methods, theories.
You're arguing past GP. It's one thing to be mistaken for subtle or deep reasons and say "this is how science works". It's another to be sloppy and say "this is how science works". I'm not even saying scientists shouldn't ever be sloppy, you should definitely start sloppy and there's also a sense in which if observations are reproducible across sloppy setups it's more robust - but also, maybe one should not take out a headline-grabbing presser based on sloppy or shaky work.
And it's hard to argue that the social structure of contemporary scientific discovery doesn't incentives this sort of sloppiness.
> It's one thing to be mistaken for subtle or deep reasons and say "this is how science works". It's another to be sloppy and say "this is how science works".
There is a difference, but since both kinds of errors are widespread among humans, and science addresses both problems with a common solution, and nonscientific methods of generating systems of belief about the material world are prone to error due to their failures to correct both problems, I don't think there's really that material of a difference unless you mean “you” in the narrow individual sense of the specific actor making the initial error, in which case—and only in which case—is the difference at all significant.
Unfortunately science too has its own failure modes, including the incentives to overclaim. I don't say this was true here. Source: am academic scientist, though not in a very rigorous field.
Regarding the climate data: Climate change is definitely happening. The problem is that the current model for sea ice growth is wrong because we're missing something (hypotheses need to be tested: maybe the salinity of water is dropping at the surface, maybe there's a change in water currents), but sea ice grow and melt every year. The ice shelves, however, are melting at a record pace.
No. Climate does not always change at the extreme rate it is changing now. A change in 4 degrees celsius in a century is very different from a change in 4 degrees celsius over 10 000 years.
The term "climate change" almost always refers to the current rapid changes over a century as opposed to natural rates of change; if someone misunderstands that I suspect that it is because they have decided to misunderstand it.
No it's not. When language is changed as a coordinated political effort to rebrand something it creates something akin to a streisand effect. For commercial products, whatever, no press is bad press. For science, it's a problem.
This is just being pedantic. I have never encountered anyone in-person who did not immediately understand and/or equate "climate change" as "the earth is warming". If we want to get the message across to the average person, this type of tut-tut message does no good.
That is a fair point. I do recognize the need to differentiate the two as soon as it becomes obvious that someone is trying to use the term to dispute the facts, but I find that most people just want to use the term they know and end up feeling burned if they're corrected while they're already trying to advance the cause.
> One strategy is to write an equation that describes the wiggles
Don't telescopes have some lens coatings that could dampen certain wavelengths? Or some form of giant sun glasses?
I don't understand how you would approach removing noise basically. A 12th order polynomical can fit many, many curves with really good detail already. But what signal were they trying to interpolate?
That’s actually not a good example, because the follow up analysis actually confirms the initial result. ALH84001 is actually some pretty good evidence for fossilized Martian life.
What happened though was that they went public early with too little information, because they were worried about being scooped in the media. When they got asked questions that they didn’t have answers to, the result started to look like it was on shaky grounds. The opinion was formed that it was a bad result. After that it was pretty hard to get anyone to pay attention after that. Follow-on results pretty much confirmed the original result though.
I think there's a more recent book that came out a few years ago, but a quick google search isn't finding the name, sorry.
For Wikipedia which relies on citations, it's going to reflect the majority opinion which is "ALH84001 is insufficient evidence for life." However the reasoning for saying that morphological features are insufficient signs of life (in contrast with biologists studying Earth life who routinely rely on such evidence) is actually a direct response to ALH84001. Like the Pluto-Is-A-Planet? debacle, the generally accepted requirements and definitions were drawn up after the fact specifically to exclude ALH84001, in order to end an embarrassing controversy and provide funding for more missions (why go to Mars when you can just pick up rocks in Antarctica instead?) rather than answer a scientific question.
we're still in the same place as the original publications: most scientists don't think that EM of rocks from that look like microbes on rocks on earth is a convincing argument. The morphological argument has held since before this rock, it wasn't added up afterwards.
There just isn't enough information here to conclude anything so it's easier to reject the hypothesis (I have to admit, it took me a while to accept the idea that there is that much mass exchange between planets in the first place!)
And yet, go try to find out an age for the origin of life on Earth. You'll find that ALL of the evidence is faint morphological features in rocks from Australia. It's the gold standard for one field, but circumstantial in another?
Not correct, the data on earth that is reliable is based on carbon ratios, which are a far better hallmark. The older morphological stuff, while fun, is really just speculative, still.
Not for origin of life, no. We’re talking >3gy. These fossils are rocks, not biomaterials. Isotopes of trace decay products in the rock itself are used to age it, as is done with other geologic samples.
At the time that the big announcement was made in the 90's, the scientific community immediately came up with a bunch of criticisms that weren't properly addressed in the original paper. They suggested terrestrial contamination, they suggested abiotic crystallization processes, and given the observed similarities with Earth fossils they questioned whether cross-pollination of life was possible. The scientists had no satisfactory answers and ALH84001 was dismissed by the press as a dud and the scientific community as a career-ending topic (like cold fusion is for physicists).
Still, in the years that followed further analysis was done by the original authors and their teams. Isotope ratios and captured atmosphere within and around the tracks confirmed that the fossil features were ancient and of Martian origin. Comparative characterization of terrestrial rock-worm fossils was done and it was noted that like the Earth fossils--and unlike abiotic processes--the ALH84001 features were of uniform size and were anisotropic on the sub-microscopic scale, consistent with fossilization of biological structures rather than geologic crystallization processes. Finally, testing showed that the inside of the ALH84001 meteorite itself never exceeded 40 degrees when it was blasted from the Martian surface, or when it entered Earth's atmosphere, so any biological life deep within the rocks could have survived (no one thought the fossils were live, but by implication Mars could have, and almost certainly was seeded by Earth, making it not surprising that we would see a the telltale signs of Earth microbes in a rock from Mars.
TL;DR: At the time of the press conference and the weeks after, a bunch of reasonable questions were raised about the ALH84001 work. In the years that followed a few of the scientists involved followed up with further tests which confirmed or at least did not disprove their martian-life interpretation of the fossils. But the rest of the world moved on.
This article is way too sanguine about this. Scientists putting forth a hypothesis, getting peer reviewed, getting published, and then having others fail to replicate their findings is absolutely a normal part of science.
However, touting very surprising, very unsure results using the megaphone of media, is not science. It is clickbait. It is BS. And it is intensely damaging to science.
What happens is there is a selection effect. The most surprising findings are the ones that are most likely to be touted by the media. However, the most surprising results are the most likely to be wrong. So over time, you end up with the result that much of the "science" touted by the mainstream media ends up being wrong.
And we cannot just blame this phenomenon on the media. Many times it is the institutions that are supporting the research that are contacting the media and putting out press releases and getting on twitter with very clickbaity summaries of the research.
It is great for the scientist. It makes them famous for a day. And as with most stories, the original story gets a lot more play than the correction.
However, it is very damaging to our society. The regular person sees science as a stream of exciting results that are later proven wrong. All the while people are shouting, "This is fine, because this is how science works." And it damages trust.
Imagine if you went to your doctor for a routine blood test and the doctor came back and said, "You have an elevated white blood cell count. This could mean you have cancer." Then a couple of days later, you got called back in to be told, "Well actually, the count was increased because you likely had a cold. But this is how medicine works. One test can suggest something, but different tests can give a different explanation."
If this scenario happened on a regular basis, your confidence in medicine would really be messed up.
If we present science as this ever changing, always being proven wrong entity, then how can we ask people to depend on "science" and make potentially very expensive and inconvenient adjustments to their lives such as with COVID or climate change?
Politicizing scientific results is more damaging than publicly falsifying them. I'd worry more about that. Once something is politicized, it's harder for the red/blue teamers to incorporate new data that shows it to be false.
They can't, and won't. They'll choose to believe what they want to believe. Who can blame them when horseshit is constantly being published and publicized, often by supposedly reputable journals. It was the Lancet that started anti-vax.
People often ask what HN was like in the good old days. I don't remember drivel like this in the first 5 years.
I think the underlying reason for these types of threads is that HNers want to virtue signal that they're interested in science, but too lazy to do the actual study and work to understand and contribute. Pro tip: just follow the Twitter NASA channel. You'll get pretty pictures, with no real effort.
An analogy is when the Linux kernel team started a kernel newbie project, and got nothing but white space patches. So the project had to be abandoned because of low quality and the burden of wasting time for, you know, actual kernel contributors.
Not sure if HN understands what science is, because I caught reddit saying a scientist's opinion was science.
Science is science when an author can be removed from a study and the experiment can be repeated. A 12 year old or science hater should be able to reproduce the results if it's science.
Things that aren't science-
>Authority- A PhD/medical doctor/a scientist, these are human opinions, not science. They can cite studies to be scientific.
>Art
>Tradition
There's value in non scientific disciplines, but they have their own burdens to be aware of.
There's a segment of the population that sees scientists as materialist, secular replacements for priests and mystics, instead of sophisticated monkeys trying their best to make sense of a messy, confusing world.
As a side note, I'd push back against the idea that replicability is the key ingredient of science: it's neither necessary nor sufficient to discover scientifically "true" knowledge, though it's definitely a nice to have. Plenty of important measurements have been made confirming theories we see as true today (or true enough) that turn out to have been irreproducible, plagued by experimenter bias, or even outright fraudulent. But it's a particular research programme, embedded in a social context, that generates knowledge, not a single reproducible experiment.
In a universe with spherical cows, maybe. But the issue is that theories we view as true (at least within their scope) often fail to predict observations. Which is fine: we modify them with ad hoc hypotheses to round off the rough edges. The issue is that all theories, true or false, do that. Copernican theories were initially less predictive than Ptolemaic ones, for instance; a simple heuristic of rejecting theories that generate more incorrect predictions than other theories would have left us committed to a Ptolemaic universe. And yet we moved.
I'm not saying to follow a purely incremental optimization path, leading to local maxima. Thinking outside the box is clearly important for generating new ideas to test. We're not done until we run out of things to perfectly predict.
I would say some of the ancients did science too, to the best of their knowledge.
> But the issue is that theories we view as true (at least within their scope) often fail to predict observations. Which is fine: we modify them with ad hoc hypotheses to round off the rough edges.
This passage reminds me of how neural nets learn and often fail to generalise.
> But it's a particular research programme, embedded in a social context, that generates knowledge, not a single reproducible experiment.
Somewhat disagree. That particular research programme must still have reproducible experiment as a unit. A million irreproducible experiments will generate exactly zero knowledge.
you see this sort of (religious) appeal to authority prevalently on topics like mask wearing, with phrases like “based on science” and “science tells us...” to justify largely politically-based beliefs, in many cases citing administrators or doctors (who are generally not first and foremost scientists) as ‘evidence’ (npr & nyt literally does this every day).
science has told us so far that masks might help at the margins, but hasn’t proven it, which indicates it probably shouldn’t be counted on as a primary mitigative measure, but the public signaling value is just too irresistible for the believers (transmission principally happens in private where social norms oppose mitigative measures).
Doctors' hands on experience gives them a special kind of knowledge pretty close to scientific.
Also, science doesn't really proves, but rather disproves, and the "softer" the science, the harder it is to get any certainties...
(And medicine, unlike biology, is a soft science.)
it's not that doctors don't have specialized knowledge that might be stochastically predictive in specific cases, but that their experience, especially around something like a pandemic, skews their perspecive in ways that are very difficult to self-identify and compensate for. and that's on top of doctors holding their own idiosyncratic sociopolitical views too.
in other words, expertise is narrow, and treatment experience isn't research.
Everyone has their own idiosyncratic sociopolitical views. I would not discount doctors' specialized knowledge as 'not science' (or worse, 'not evidence'). You just have to remember that medecine is a mix of hard and soft sciences (and other things too).
that's intuition, not science. that doesn't mean it's not useful, it's just not useful as science. the point of the appeal to authority fallacy is to identify and unbias us against that kind of rhetorical trap in argumentation.
Bridges are a good example. You have to remember that bridge building (as pretty much any field of engineering or even science) didn't start out as math-first, but as experience-first. And there's really no opposition there, after all whole fields of science (the 'soft' sciences) don't (can't) even use math !
Medecine is a mix of hard and soft sciences (and other things too).
Maybe that is because there is a segment of "scientists" (read: people who publish in "scientific" journals) that are just materialist, secular replacements for priests and mystics.
Can you give an example of something that isn't reproducible but was still true?
Millikan's oil drop experiment on quantization of charge and measurement of e is an infamous example. No one can reproduce his reported results and, in some ways, his reported results go beyond what's attributable to bias and cross the line into fraud. And yet his ultimate result was true enough, and he progressed science enough to win a Nobel.
Millikan's oil drop experiment is definitely reproducible.
He excluded values to reduce his error (seems like), but that doesn't make the experiment non-reproducible. You can do the same experiment today and get an approximate value for e.
Will you get the exact same results as Millikan? Of course not. But that's not what reproducible means.
The issue is that you get substantively different results from Millikan if you follow his stated experimental procedure. The only way to reproduce his results is to perform the experiment and then selectively remove datapoints that disagree with his result until you only have results remaining that are close to his. That doesn't qualify as reproducible in my book, except in the trivial sense you can perform the same experiment he did, irrespective of the actual result.
The goal of the experiment is not to get a certain arbitrary number (the one Millikan came up with), the goal is to get the approximate value of the elementary charge.
Reproducible means that if you follow the method outlined by Millikan, you should get an approximate value for e. This is indeed the case, so the experiment is reproducible.
A non-reproducible experiment would be one where you follow Millikan's method but get something that is nowhere close to being a value for e.
You're using a n̶o̶n̶s̶t̶a̶n̶d̶a̶r̶d̶ different sense of the term reproducible. Just to drive the point home, it wouldn't make sense to talk about a reproducibility crisis if having data being consistent across different experiments conducted by the same methodology with different researchers wasn't a key component of reproducibility.
Depends on the experiment. Oil drop is based around something we "know" exists (the charge of an electron) and is just trying to measure it. Measuring a fundamental value is a very different kind of experiment than trying to determine if something exists at all in a wildly multi-variate system (most of the research suffering from reproducibility issues).
Not every experiment needs to be reproducible in exactly the same way, just like not every study needs to be double-blind. You need to interrogate the reason the experiment exists in the first place.
I guess another way of saying it is this: even though Millikan's actual final result is not reproducible because he fudged his numbers, the scientific method employed in the experiment is valid, which is really all you need in the case of that experiment.
The problem with medicine / psychology / other less "hard" research is that in many instances it doesn't matter if the method is reproducible if the results are not. If the goal is to show that eating pancakes cause depression the result is actually all that matters.
This link clearly talks about reproducible results, not reproducible data.
Wherever there is some degree of randomness, we would not expect to get the same data, but we would expect to draw the same conclusion.
See also: the Mendelian paradox - it is sometimes argued that Gregor Mendel's results were too perfect, and yet he is honoured as the father of genetics.
as several of my genetics teachers put it: "wow, mendel was really so lucky that he just randomly managed to pick 7 independently associating traits on different chromosomes"
(the molecular biologists in the audience all shook their heads)
Physics classrooms around the world have had students replicating this experiment for decades. I think it's not his definition of 'reproducible' that's nonstandard.
science got the right place eventually. It was reproducible enough that the field made forward progress and even did extraordinary analysis to find out the originator showed some judicious data reduction.
So it worked despite all the bias and fraud. You could even call it luck. The reproducibility implied by the fact that at least some of it is true is not a "nice-to-have", it's the whole point of it working. We wouldn't call it true if we couldn't confirm it to be true.
To me it seems you're conflating reproducibility with insight. Of course insight is also key, and it may be useful even if it's wrong, as long as it causes others to recursively have new insights and create hypotheses and do experiments that eventually turn out to be true.
Exactly. Which renders the logic behind the experiment false. But since the insight was true, it eventually led to experiments that yielded true results. (Notice I didn't say the insight was false, I supported your idea by emphasizing that even false insights can be useful.)
I imagine the LD50 of many substances are defacto non-reproducible despite being mechanically replicable. i.e. the world will not permit us to test the LD50 of FOOF so any measure of its danger to human lives is defacto untestable. Yet we have a notion of its truth.
Of course there is an epistemological distinction here between a thing that cannot be reproduced by its nature vs. one that cannot be reproduced by societal norms.
> A 12 year old or science hater should be able to reproduce the results if it's science.
Well, that's going to be pretty limiting. A 12yo isn't going to have the budget to build their own LHC -- reproducibility is certainly an important goal, but "a 12yo or science hater" is an unreasonable place to set the bar.
Simulation is also not science. It can help narrow the search space, but you can not draw conclusions from it, because what comes out of a simulation depends on what you put it.
And if you knew what needs to be put in, you would not need the simulation in the first place.
Note that a simulation is not the same as a model. A model is tested and refined based on real world data, until the model returns the same results.
Simulation is "not science" in the same way that microscopes are "not science". These are tools that may be used by scientists, but are not themselves 'science'.
I now promise myself I will never read the comments on an HN Science article going forward. The sheer number of individuals here who presume scientists are idiots, that the decisions they make are dumb or ill- motivated demonstrates the sheer hubris and lack of humility of a large portion of HN commenters.
I guarantee you that scientists are as smart and capable as the best of the non-scientist HN population, and were the story roles reversed, that scientists on the whole would be more humble about the limits of their knowledge about programming practice and design, etc. They would not revel in the "idiocy" of your bugs, security lapses, or petty spy-based mobile apps. Well maybe the last.
Have some humility. Stop building up fragile egos by tearing others down, pretending that you wouldn't make similar "mistakes" or produce designs with similar weaknesses or limits.
Edit 1:Yes I'll defend my tribe. Each down-vote is a badge.
I spent a couple decades as a scientist. I was modestly successful, published only work that I believe was truly defensible, and never over sold any of my results. I've asked to have my name removed from a paper that was submitted when I noticed it was revised to include lies after I had written it.
I believe you are wrong in saying that scientists aren't idiots: they make colossal errors, obvious ones, all the time! Frankly, I think most successful (in terms of adulation and $$$) scientists are ones who learned how to play the games scientists play, and publish known incorrect work, or unknowingly publish overly strong estimates of their confidence, because they've mastered the incentive system, not because they're not idiots.
I see very little true humility amongst scientists- nearly every successful one I've met was aggresively egotistical.
That said, after another 10-15 years of being a SWE, I've come to the conclusion most SWEs are idiots, too- too focused on clever algorithms and microoptimizations.
We are all idiots to some degree, scientists are no exception. But I firmly disagree with your assessment of the state of science and scientists. Improvements to be made? Certainly.
They fit a TWELFTH-ORDER polynomial to the noise data. Twelfth! If you're ever fitting a twelfth-order polynomial to any data, ever, you need to seriously re-examine your underlying assumptions. I would go so far as to say that anyone fitting a 12th order polynomial to observational data is an idiot, at least when it comes to statistics.
Twelth order polynomials do seem extreme, but without being an expert in the context they used it, I cannot determine its appropriateness.
As an extreme, let us suppose that we know that there are 6 sources of noise spread out over the x-axis, and that the power of the noise will decrease x units away from each source. In that case, you might suspect that noise function needs a 12th order polynomial to fit the undulation pattern, just to get the necessary number of humps.
> The sheer number of individuals here who presume scientists are idiots
The general population are idiots. The amount of scientists studying is huge compared to 40 years ago. It has to be full of idiots.
We are not talking mistakes. We are talking fundamental structural flaws. The fact people in the scientific community defend themselves with rhetoric show they are idiots.
If scientists weren't such idiots they would attack the tech community and how useless it is. But it's scientists who support the AI lies for instance. They are a big part of the problem in tech.
Numerous comments about the promoter's non-scientific background, the sensationalism of the original announcement and skepticism over the findings. Please note that the scientists themselves weren't sensationalizing their findings, but others were.
As this article points out: it was a "big, if true" story that had numerous red flags from the start.