I think there's been a Gresham's Law in science funding in this country, as the political people who are nimble in the art of writing government grants have gradually displaced the eccentric and idiosyncratic people who typically make the best scientists. The eccentric university professor is a species that is going extinct fast.
Thinking about this, I wonder: has anyone tried to systematically measure "eccentricity" vs. scientific productivity?
Also, I wonder if there a risk that this characterization just perpetuates the pointy-egghead stereotype - that good scientists don't have social skills? The discussion of "holistic" evaluation in Steven Pinker's recent essay on Harvard admission comes to mind [1].
Perhaps the funding agencies need a separate, high-prestige "individual contributor" track similar to corporate Principal Engineer or fellowship tracks, to allow non-political but brilliant scientists to work hands-on and avoid the grant scramble?
HHMI tries to provide something like this already, giving longer grants to early-career faculty to reduce grant pressure, and I guess NSF and NIH have also been trying to do this with CAREER awards and New Innovator grants...
There is a well documented trend in the sciences away from research performed by scientists on their own or in small collaborations towards research performed by very large collaborations [1]. My understanding is that this has been going on for a long time in biology and physics so that nowadays it's very difficult to make much of an impact without participating in a large collaboration. (This is more true with regards to experiment than theory.) This trend has also been noted in my field of astronomy, though there hasn't been nearly the same crowding out of individual astronomers as in physics.
I could imagine that this trend has made it more difficult for more eccentric scientists to succeed because they might be less able to "play nicely with others" in a big collaboration.
In physics at least, isn't that because most of the low-hanging (low-energy..) fruit is gone? CMS or Atlas at the Large Hadron Collider are feats of engineering, not to mention the physics, and doing all of that takes a large organization. In theoretical physics, people seem to still be doing reasonably well with the lone-genius thing (although I suppose Peter Woit disagrees).
Biology is interesting because small, high-impact groups become arguably more sustainable every year due to cheaper, smaller instruments and the rise of contract services (e.g. DNA sequencing). On the other hand, many of the human problems biology is attempting to address are hard to inferentially study on a small-sample level, again necessitating large organizations.
It's not always about productivity. It can be about finding superstars. These two ideas contrast: in productive organizations, you typically want people closer to the mean (i.e. reduce variance among the population) which yields constant improvement. In an organization that yields superstars you want people far from the mean (i.e. a lot of variance among the population) which yields big hits but not necessarily constant improvements.
Basically, the organization (scientific community) needs to decide which type of population to select for to fulfill their strategy.
By productivity, I certainly did not intend to imply volume! Perhaps impact would have been better. Although I'll leave aside the philosophical question of historical inevitability vs. singular genius :)
Do you think that there is a widespread failure by the scientific community to identify and support superstars?
You see a similar effect in business of course. In booms like the tech boom I think we saw a crop of people with actual merit get rich, but normally it's the wheelers and dealers.
I'm not sure that's as big a problem in academia as in other circles. I would imagine that academic researchers (professors) are highly discerning among their friends based to a large extent on competence.
It's like how recruiting for engineers is best done through the networks of your engineers; they really only hang out with top people anyway (give or take).
My friend (who completed a computational biology PhD but was dismayed by the small likelyhood of being able to do meaningful science and left to do a ML based startup) has taken the route of becoming independently wealthy himself and then privately fund researchers he personally knows with his own wealth.
I feel like that would only work if many thousands of people had that attitude because the likelihood of a single person becoming independently wealthy is very small. Even if your friend was successful he would only be limited to assisting in a small number of scientific advances.
Wouldn't it make sense for your friend to try and organize a group of people with similar goals so everyone can work together to achieve them?
This also describes David E. Shaw, who was denied tenure by Columbia University. He now spends most of his time at D.E. Shaw Research, where he basically does what he would have been doing as a tenured professor anyway.
I think there's been a Gresham's Law in science funding in this country, as the political people who are nimble in the art of writing government grants have gradually displaced the eccentric and idiosyncratic people who typically make the best scientists. The eccentric university professor is a species that is going extinct fast.