This is a fascinating conversation. I've always had trouble working in teams, so I'd like to believe that superior programmers will out in the end. Or at least that they will in a few problem domains.
But I wonder if this is wishful thinking, if this isn't just another case of the prisoner's dilemma. Perhaps like how cities with mostly poor people would collaborate many times in history to conquer neighboring barbarians, even though the barbarians had more freedom and were thus richer. (See http://en.wikipedia.org/wiki/Fates_of_Nations.)
Then again, there's reason for hope. Perhaps the parallelizable sort of programming is more menial. It certainly seems that way with the way communication costs overtake large teams. It's almost like Vernor Vinge's zones of thought (http://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep, http://www.youtube.com/watch?v=xcPcpF2M27c) - as your team grows bigger you can just watch the members grow dumber in front of your eyes as more and more of their cognitive effort is eaten up by internal communication, leaving less and less for externally-useful work. If this is true, there's hope that advances in programming will automate the low-cognition tasks and allow programmers to focus on the high-cognition ones, leveling the playing field for small, high-cohesion teams.
---
Me, I've been obsessed with something raganwald said when he spawned this tendril of conversation: exercising explicit control over the space of inputs my program cares about. My current hypothesis: eliminate fixed interfaces, version numbers, and notions of backwards compatibility. All these are like petridishes of sugar syrup for code to breed more code. Replace them with with unit tests. Lots of them[1]. If I rely on some code you wrote, and I want to pull in some of your recent changes, I need to rerun my tests to ensure you didn't change an interface. Programming this way is less reassuring, but I think it empowers programmers where abstraction boundaries impose mental blocks. Great programmers take charge of their entire stack, so let's do more of that. I'm hoping this is the way to prove small teams can outdo large ones.
[1] Including tests for performance, throughput, availability. This is the hard part. But I spent a lot of time building microprocessor simulators in grad school. I think it's doable.
I'd like to believe that superior [solo] programmers will out in the end
I think you're wrong (sorry!) because it's impossible to talk about superior programmers without talking about teams. Building complex systems is a team sport. There's no way around this. But you can't have good teams without good programmers.
The phrase "scaling complexity" has at least two axes built into it: the abstraction axis -- how to get better at telling the program to the computer -- and the collaboration axis -- how to get better at telling the program to each other. Most of this thread has been about whether we suck at the former. But I say we really suck at the latter, and the reason is that we haven't fully assimilated what software is yet. Software doesn't live in the code, it lives in the minds of the people who make it. The code is just a (lossy) written representation.
We can argue about how much more productive the best individual working solo with the best tool can be- but there's no way that that model will scale arbitrarily, no matter how good the individual/tool pairing. At some point the single machine (the solo genius) hits a wall and you have to go to distributed systems (teams). One thing we know from programming is that when you shift to distributed systems, everything changes. I think that's true on the human level as well. (Just to be redundant, by "distribution" here I don't mean distributed teams, I mean knowledge of the program being distributed across multiple brains.)
Maybe you wouldn't have trouble working in teams if we'd actually figured out how to make great teams. So far, it's entirely hit and miss. But I think anyone who's had the good fortune to experience the spontaneous occurrence of a great team knows what a beautiful and powerful thing it is. Most of us who've had that experience go through the rest of our careers craving it again. Indeed, it has converted many a solo type into an ardent collaborator. Like me.
I was originally going to write about this and then decided not to go there, but you forced my hand. :) Just as long as it's clear that when I say "team" I mean nothing like how software organizations are formally built nowadays. It's not about being in an org chart. It's about being in a band.
The phrase "scaling complexity" has at least two axes built into it: the abstraction axis -- how to get better at telling the program to the computer -- and the collaboration axis -- how to get better at telling the program to each other. Most of this thread has been about whether we suck at the former. But I say we really suck at the latter, and the reason is that we haven't fully assimilated what software is yet. Software doesn't live in the code, it lives in the minds of the people who make it. The code is just a (lossy) written representation.
Ah, you're right. I was conflating the two axes.
I'd like to be part of a 'band'. I've had few opportunities, but I've caught the occasional glimpse of how good things can be.
Since that whole aspect is outside my ken, I focus on expression. Hopefully there's no destructive interference. I would argue that what you call abstraction is about 'communication with each other' more than anything (even though it breaks the awesome symmetry of your paragraph above :)
But I wonder if this is wishful thinking, if this isn't just another case of the prisoner's dilemma. Perhaps like how cities with mostly poor people would collaborate many times in history to conquer neighboring barbarians, even though the barbarians had more freedom and were thus richer. (See http://en.wikipedia.org/wiki/Fates_of_Nations.)
Then again, there's reason for hope. Perhaps the parallelizable sort of programming is more menial. It certainly seems that way with the way communication costs overtake large teams. It's almost like Vernor Vinge's zones of thought (http://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep, http://www.youtube.com/watch?v=xcPcpF2M27c) - as your team grows bigger you can just watch the members grow dumber in front of your eyes as more and more of their cognitive effort is eaten up by internal communication, leaving less and less for externally-useful work. If this is true, there's hope that advances in programming will automate the low-cognition tasks and allow programmers to focus on the high-cognition ones, leveling the playing field for small, high-cohesion teams.
---
Me, I've been obsessed with something raganwald said when he spawned this tendril of conversation: exercising explicit control over the space of inputs my program cares about. My current hypothesis: eliminate fixed interfaces, version numbers, and notions of backwards compatibility. All these are like petridishes of sugar syrup for code to breed more code. Replace them with with unit tests. Lots of them[1]. If I rely on some code you wrote, and I want to pull in some of your recent changes, I need to rerun my tests to ensure you didn't change an interface. Programming this way is less reassuring, but I think it empowers programmers where abstraction boundaries impose mental blocks. Great programmers take charge of their entire stack, so let's do more of that. I'm hoping this is the way to prove small teams can outdo large ones.
[1] Including tests for performance, throughput, availability. This is the hard part. But I spent a lot of time building microprocessor simulators in grad school. I think it's doable.