Everyone who cares about science has known his point for over 40 years now.
But Psychology, the subject he was discussing, continued to not take figuring out how to replicate seriously until the recent replication crisis demonstrated exactly HOW bad it was in terms that they couldn't ignore. See https://en.wikipedia.org/wiki/Replication_crisis for more on that.
And now it is a hot topic with lots of, "Look at what we've figured out about how to make our research better!" With very little discussion of the fact that this was something they should have known all along.
yes the problem is politics , or what you might call institutionalized group psychology of those with access to spending tax dollars, and the legislators they need to 'persuade' to get the money spent.
science is not an art , it is generally an institutionally funded social behavior. the art of science is such that a lot of shitty ass sciece research with poorly constructed hypothesis and protocol gets funded............
can we do a better job? maybe. but the answer lies in better policy analysts and politicians, not in better 'scientists'.
on heuristic for helping pinpoint bad and poorly constructed research is to look for where the most 'growth' in research has occured the quickest; and by that-----i mean look for the research 'bubbles'.
why? as with art in general, and clothign and every sphere of human behavior-----that which gets popular very quickly is usually by nature fleeting and without the most substance, like fast growing weeds. that which grows slowly and withstands the test of time usually resembles an oak in quality.
that does not mean that fast growth is always undesireable. you want fast growth in areas where there is great potential and low hanging fruit.
it is good that bubbles occur SOMETIMES. BUT NOT ALWAYS. some bubbles are poorly advised. you must crack a few eggs to make an ommelet but sometimes you are just breaking eggs inside the carton without ever making breakfast.
but regardless of whether or not a bubble is good or bad, all bubbles still must pop at some point to let the overgrowth and wasteful growth be cleared. all forests must eventually have a forest fire.
the longer fire and destruction is suppressed beyond a certain point........the greater the amount of destruction is going to be when it finally comes.
and in general, this is why our current economic bubble is so worrisome
Very much so. My pet theory is that most business management practices evolved from manufacturing production line environments, and the application of these management practices to, say, running a university or research facility is cargo cult in the extreme.
The parts that always strike me the most about this is that in 1974 Feynman lamented a lack of scientific rigour in deciding how we teach and how we manage crime and the criminal justice system. In both cases, despite 40 years having passed, it's hard to say there's been any large improvement in the application of rigour.
The part that boggled me was how recent (and incomplete) was the introduction of blinding to police lineups. Apparently it really was (and is?) common for the witness to pick out a suspect, and for the cop to know the 'real' suspect and say things like "Are you sure? Take your time."
From another Feynman talk: "This is not yet a scientific age."
This applies extremely well to programming and software engineering in particular. "Best practices" are often a good excuse to avoid thinking hard about the problem at hand.
I think there is a lot of cargo culting going on everywhere. The lucky ones pick the right cult. How do you know whether your model of how the world works is right?
You're looking at this the wrong way. The question is not "does reality match this model?" but rather "how well does this model match reality?".
You never know exactly to what exact extent your map matches the territory. But you can have a pretty good idea. Like, we know (to the extent that we can "know" anything) that models predicting the Sun will rise tomorrow match reality better than those which don't. We know that models which predict that turning on a switch will cause the lightbulb to emit light are more accurate that those who say that it's magic or mysterious.
To quote Feynman:
> We’ve learned from experience that the truth will out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature’s phenomena will agree or they’ll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven’t tried to be very careful in this kind of work. And it’s this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in Cargo Cult Science.
In physics or other hard sciences it's possible to validate a lot of models.
I was more thinking about other areas like politics, management or software engineering issues. There it's often not possible to validate your model. It's even hard to tell if something goes well or not. And the feedback loop from decision to result can take years in many cases. So you pretty much are forced to do cargo culting.
Ah, I see. I thought you were going more along the lines of the "how can we know anything?" sort of philosophical questions.
In the matters that you're talking about, it may just be that we can't really know. Certain things take years to figure out (take Feynman's example of the Milikan value): people still −I assume− had to use some value while the "official" value was shifting.
I'm not sure I'd call that cargo culting necessarily. If the best case to be made right now is in favour of the majority position, then I think one should go with the majority position. I'd say that one should still keep a healthy dose of skepticism and be ready to change their mind if the balance of evidence changes… but then, it may be that managers/leaders need to feel committed and that a "healthy dose" for them is very, very little.
She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to __nut__ try to repeat psychological experiments, but only to change the conditions and see __hat__ happens.
In CS (Systems, not Theory) research there's a big push now to move toward reproducibility by publishing code and dependencies; https://twitter.com/kragen/status/727220736633524228 talks a bit about some of the historical context and details.
Hackademia has a couple of synergistic efforts going on, directed mostly at securing the free-software distribution infrastructure and somewhat at simplifying system administration: on one hand you have reproducible builds being done by Tor, Mozilla, and Debian, and on the other hand you have Nix, Guix, and the like pushing purely-functional content-addressable package management.
The upshot of all of this is that, in theory, in the near future, it will be practical to publish machine-executable instructions for reproducing the bit-identical executables and other output products you used in your research, at a cost of only a few gigabytes, so they can be exactly reproduced at any time in the future, and then modified to do new experiments; and it will be straightforward to make archival copies of this entire set of dependencies so that it won't be lost because some random guy died or decided to pull his modules from npm.
The Sprite-LFS affair I mention in that tweet is an infamous case of a famous, influential systems paper by a famous, influential academic (Ousterhout) that badly failed reproduction when a different famous, influential academic (Seltzer) tried to reproduce its results by writing a new log-structured filesystem. Nobody really knows what went wrong, but our best guess at this point is that the Sprite operating system had unknown bugs in it that gave a performance advantage to log-structured filesystems.
(ChuckMcM may want to weigh in, since his company at the time (NetApp) was getting record-setting performance with WAFL, which is very similar to a LFS in some ways, but not others.)
Jan Vitek, Jean Yang, Sam Tobin-Hochstadt, and Shriram Krishnamurthi are four prominent researchers who are advocating this change; they and others have created a new institution called "Artifact Evaluation Committees", which review the artifacts submitted by authors to see if they are able to reproduce the results the authors report. They've written an overview of it here, and they seem to already be getting significant results: http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-researchers-d...
Hopefully over time this will create a bubble of real, reproducible science inside of the systems part of Computer "Science" research.
[Earlier versions of this comment did not mention Krishnamurthi or Vitek and the synergistic work in hackademia, which was a combination of cluelessness and carelessness on my part. I regret the omissions.]
There is a W3C group that is all about setting things up so that the use of big data repositories and complex scientific code can be used so that others can replicate the findings:
It's actually fairly terrible that in Environmental Science and quite a bit of Economics and other areas the code and data is meant to be around but really isn't.
I could be totally wrong, I don't read a huge number of CS research papers, but aren't most published CS papers fully self-contained? They contain the algorithms or ideas within them. Perhaps in pseudocode, but fully reproducible.
The little experience I have is in computer graphics publications and (less) distributed systems publications. Graphics papers very often have data sets and code that go along with the paper. These are nice, and it would be awesome to have a common and well-known way of publishing all that together with the papers, but they are just examples. Mike Hoye's comparison to the loss of greek fire doesn't seem applicable to the idea of reproducibility by publishing code and dependencies. The meat is in the paper!
Same with my experience in distributed systems papers. I can implement Paxos at any time using only the Paxos PDFs I've saved. Companion code is nice, and again I'd be fully supportive of a system which gave permanence to not ony the papers but companion artifacts... but the paper is the important part, isn't it?
> I could be totally wrong, I don't read a huge number of CS research papers, but aren't most published CS papers fully self-contained? They contain the algorithms or ideas within them. Perhaps in pseudocode, but fully reproducible.
They may be not reproducible if going from pseudocode to working implementation and from there to the pretty graphs published by author is nontrivial. From parent post:
The Sprite-LFS affair I mention in that tweet is an infamous case of a famous, influential systems paper by a famous, influential academic (Ousterhout) that badly failed reproduction when a different famous, influential academic (Seltzer) tried to reproduce its results by writing a new log-structured filesystem. Nobody really knows what went wrong[...]
> I can implement Paxos at any time using only the Paxos PDFs I've saved. Companion code is nice, and again I'd be fully supportive of a system which gave permanence to not ony the papers but companion artifacts... but the paper is the important part, isn't it?
Paxos is closer to the "theory" than the "systems" end of parent's spectrum. Real problem is with things that take man-months to implement and validate, not things that take man-months to invent.
The entire "agile" methodology strikes me as a prominent example of a software engineering cargo cult.
Yes, good projects have been executed successfully using agile methodology. No, this does not mean that using agile caused these projects to work out well.
Most agile projects fail, just like most non-agile projects. There is no evidence whatsoever that agile methodology causes projects to work out well -- but there is a whole mini-industry of training, books, seminars, "scrum masters" and so on built around the idea that it does.
There's very little evidence whatsoever about any software development methodology or practice working well or poorly. That said, most experienced developers I've interacted with consider most of the Agile principles to be wise -- either common sense, or shown to be true via decades of experience.
Here are the principles:
* Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
* Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
* Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
* Business people and developers must work together daily throughout the project.
* Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
* The most efficient and effective method of conveying information to and within a development
team is face-to-face conversation.
* Working software is the primary measure of progress.
* Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
* Continuous attention to technical excellence and good design enhances agility.
* Simplicity--the art of maximizing the amount of work not done--is essential.
* The best architectures, requirements, and designs emerge from self-organizing teams.
* At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Which of these would you argue aren't wise principles for all of us to use when developing software? Now, if your beef is with Scrum (or some other branded methodology) in particular, I'd be more inclined to agree.
>Which of these would you argue aren't wise principles for all of us to use when developing software?
most of these aren't principles, these are wishes. Like i can say i have 2 principles - be wealthy and be healthy - and those 2 principles look like very wise and hard to argue with, don't you think?
Except for the stuff like this :
> The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
>The best architectures, requirements, and designs emerge from self-organizing teams.
which is just some personal beliefs or dreams of somebody, and is obviously not true in general. There is a reason things are called 'manifest' and not even a 'paper/essay', less 'peer-reviewed paper'...
most of these aren't principles, these are wishes. Like i can say i have 2 principles - be wealthy and be healthy - and those 2 principles look like very wise and hard to argue with, don't you think?
Becoming wealthy takes time and luck. It's not something you can just turn on at will. I just read through each of the principles, and I find exactly 0 of them that share that quality. Most of them tell you specifically what to do, and they are things that you can start doing at any time.
Name one or more specific principles that are like "become wealthy" and we can talk about them specifically. We may be reading them differently.
> The best architectures, requirements, and designs emerge from self-organizing teams.
which is just some personal beliefs or dreams of somebody, and is obviously not true in general.
I've worked remotely for a long time and love it. However, I certainly recognize that there's a legitimate cost to the team. I've also worked for years with teams that are split geographically. Again, the distance is a cost.
Of course there are benefits too. That's why we pay the cost. But the principle says the best architectures, requirements, and design. How exactly could physical distance be a net negative to architectures, requirements, and designs, all other things being equal? The methods of communication at your disposal face-to-face are a strict superset of the methods at your disposal when physically separated.
I agree, although I don't see that there's anything uniquely terrible about "agile". You could say the same thing about almost anything in our industry.
It may well be done by cargo cult, because there's a lot of catchphrases to do with Agile that people are eager to use. I can definitely see how people can get excited, do a superficial attempt at implementing, and then wonder why it didn't work.
But IMO the main thing about Agile is short feedback cycles. If your feedback is real (not just pleasing management), you can't avoid improving your product faster than some methodology (I hesitate to use such a word; it's really just rules of thumb) that insists on taking a long time between feedbacks.
This also means that it's somewhat tautological that Agile methods will work. If you keep making product, you're getting work done. If you're not getting work done, you're not doing Agile.
I have seen teams implement agile with the tools and processes, but not the spirit. Sprints, standups, etc. but without being driven by data and without cross-functional effort. So they turn into miniature waterfalls.
I wouldn't complain if it were posted 160 times. It's brilliant, amazing, relevant stuff that hasn't dated. I've re-read it at least 16 times since first encountering it in "Surely you're joking Mr Feynman" in the 80s.
The problem can not be solved by education alone. Education alone does not prevent the minds from making mistakes.
Its about building and maintaining social structures that destroy cargo science and at the same time give freedom to groups/individuals who abstain from it.
On envisions a credit system, in which you earn credibility by proofing something you did works- or by disproofing something that was wrong (obviously you get all credits that proof held).
New credits enters the system with every entering scholar.
He probably could have been delighted to see all these "pendulums going in both directions at the same time" and other marvels of modern science we so often see on this site.
But Psychology, the subject he was discussing, continued to not take figuring out how to replicate seriously until the recent replication crisis demonstrated exactly HOW bad it was in terms that they couldn't ignore. See https://en.wikipedia.org/wiki/Replication_crisis for more on that.
And now it is a hot topic with lots of, "Look at what we've figured out about how to make our research better!" With very little discussion of the fact that this was something they should have known all along.