Sorry, I didn't see that this actually posted. What I intended to say is that a less verbose expression is not necessarily easier to understand. The article is really about the balance between elegance (or performance) and accessibility. An expert coder working with a powerful language like Lisp can implement a lot of functionality very quickly, but there is a point where less skilled programmers working with common understanding of a less flexible language can implement more functionality more quickly, simply because there are more of them working in parallel.
Strictly speaking yes, it's a hypothesis. But the fact that the programming ecosystem looks as it does constitutes some evidence in favor of that hypothesis. Were it otherwise, you'd expect the professional programming world to be economically dominated by Lisp and a handful of super-programmers. Yet that isn't what we see. Why not?
That's the stock objection. Here's my answer: historically speaking, we've barely started. Software is the first mass endeavor of its kind that humans have tried. It belongs to a post-industrial era that can be expected to take a long time to work itself out. Under such conditions, social proof doesn't work. Whatever the rational way of making software turns out to be, statistically speaking it hasn't been tried yet.
Will it turn out to be "Lisp and a handful of super-programmers"? I don't know. What we need is an age of experimentation. The great thing is that startup costs are now so low that we are beginning to see that happen. Emphasis on beginning.
That argument seems a little too convenient; we are after all talking about a field (and a language, Lisp) that's been around for over 50 years. I could certainly see pockets of inefficiency persisting after such a time, but I would hardly expect the exception to be the rule at this point.
Keep in mind that I'm only suggesting that a crossover point exists, I don't pretend to know where exactly it is. In order for me to be wrong, a single superior programmer would always have to be better than two slightly inferior programmers working with a slightly less expressive language. I strongly doubt that this is true. The simplest explanation for what we observe is that in fact a team of inferior programmers working in parallel can be more efficient than a single superior programmer working alone. Not always, but often enough to prevent more expressive but less comprehensible languages from becoming dominant. What constitutes "expressive" and "comprehensible" will evolve over time, as you suggest (maybe Lisp will someday become tomorrow's Java!), but the underlying scaling law will remain.
This is a fascinating conversation. I've always had trouble working in teams, so I'd like to believe that superior programmers will out in the end. Or at least that they will in a few problem domains.
But I wonder if this is wishful thinking, if this isn't just another case of the prisoner's dilemma. Perhaps like how cities with mostly poor people would collaborate many times in history to conquer neighboring barbarians, even though the barbarians had more freedom and were thus richer. (See http://en.wikipedia.org/wiki/Fates_of_Nations.)
Then again, there's reason for hope. Perhaps the parallelizable sort of programming is more menial. It certainly seems that way with the way communication costs overtake large teams. It's almost like Vernor Vinge's zones of thought (http://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep, http://www.youtube.com/watch?v=xcPcpF2M27c) - as your team grows bigger you can just watch the members grow dumber in front of your eyes as more and more of their cognitive effort is eaten up by internal communication, leaving less and less for externally-useful work. If this is true, there's hope that advances in programming will automate the low-cognition tasks and allow programmers to focus on the high-cognition ones, leveling the playing field for small, high-cohesion teams.
---
Me, I've been obsessed with something raganwald said when he spawned this tendril of conversation: exercising explicit control over the space of inputs my program cares about. My current hypothesis: eliminate fixed interfaces, version numbers, and notions of backwards compatibility. All these are like petridishes of sugar syrup for code to breed more code. Replace them with with unit tests. Lots of them[1]. If I rely on some code you wrote, and I want to pull in some of your recent changes, I need to rerun my tests to ensure you didn't change an interface. Programming this way is less reassuring, but I think it empowers programmers where abstraction boundaries impose mental blocks. Great programmers take charge of their entire stack, so let's do more of that. I'm hoping this is the way to prove small teams can outdo large ones.
[1] Including tests for performance, throughput, availability. This is the hard part. But I spent a lot of time building microprocessor simulators in grad school. I think it's doable.
I'd like to believe that superior [solo] programmers will out in the end
I think you're wrong (sorry!) because it's impossible to talk about superior programmers without talking about teams. Building complex systems is a team sport. There's no way around this. But you can't have good teams without good programmers.
The phrase "scaling complexity" has at least two axes built into it: the abstraction axis -- how to get better at telling the program to the computer -- and the collaboration axis -- how to get better at telling the program to each other. Most of this thread has been about whether we suck at the former. But I say we really suck at the latter, and the reason is that we haven't fully assimilated what software is yet. Software doesn't live in the code, it lives in the minds of the people who make it. The code is just a (lossy) written representation.
We can argue about how much more productive the best individual working solo with the best tool can be- but there's no way that that model will scale arbitrarily, no matter how good the individual/tool pairing. At some point the single machine (the solo genius) hits a wall and you have to go to distributed systems (teams). One thing we know from programming is that when you shift to distributed systems, everything changes. I think that's true on the human level as well. (Just to be redundant, by "distribution" here I don't mean distributed teams, I mean knowledge of the program being distributed across multiple brains.)
Maybe you wouldn't have trouble working in teams if we'd actually figured out how to make great teams. So far, it's entirely hit and miss. But I think anyone who's had the good fortune to experience the spontaneous occurrence of a great team knows what a beautiful and powerful thing it is. Most of us who've had that experience go through the rest of our careers craving it again. Indeed, it has converted many a solo type into an ardent collaborator. Like me.
I was originally going to write about this and then decided not to go there, but you forced my hand. :) Just as long as it's clear that when I say "team" I mean nothing like how software organizations are formally built nowadays. It's not about being in an org chart. It's about being in a band.
The phrase "scaling complexity" has at least two axes built into it: the abstraction axis -- how to get better at telling the program to the computer -- and the collaboration axis -- how to get better at telling the program to each other. Most of this thread has been about whether we suck at the former. But I say we really suck at the latter, and the reason is that we haven't fully assimilated what software is yet. Software doesn't live in the code, it lives in the minds of the people who make it. The code is just a (lossy) written representation.
Ah, you're right. I was conflating the two axes.
I'd like to be part of a 'band'. I've had few opportunities, but I've caught the occasional glimpse of how good things can be.
Since that whole aspect is outside my ken, I focus on expression. Hopefully there's no destructive interference. I would argue that what you call abstraction is about 'communication with each other' more than anything (even though it breaks the awesome symmetry of your paragraph above :)
To me the big question is how we're going to scale up complexity. The million-line programs we have today are already an absurdity. What are we going to do, have billion-line programs? Anyone who can figure out to provide equal value with 100x less code (edit: that grows, say, logarithmically in size rather than superlinearly) is going to have an edge. Brute force won't work forever. Plus it gets extremely expensive as one approaches the limits.
Which brings us back full circle to the Mythical Man Month. While I've argued here for a crossover point where more programmers = more productivity, I acknowledge that there is a similar crossover point going the other way, where more programmers = less productivity. Finding the sweet spot in-between is the art of organization, and not yet a science.
Have you read the mythical man month? More less skilled programers do not make things go more quickly. That idea is so wrong on so many levels that I dnt know where to begin.
MMM said adding more programmer to already late project doesn't make it finished faster. It didn't say having more programmer who already understand the project will make the project finish late. If that is true then one programmer must always be the optimal number to complete any project, which is clearly wrong.
Also, "less skilled programmers" are not always "incompetent programmers" so more of the former who already understands the system may be able to complete the project faster than fewer "more skilled programmers"
Yes I have. Do you remember why he said more programmers is a problem? Because it increases the amount of communication. Because of this, few higher skilled programmers get more done than more less skilled programmers, largely because less communication is required.
I've witnessed this directly many times over the years.
MMM does indeed say that too many programmers do indeed make a project late regardless of how far behind that project already is. Nine women can't have a baby in a month and all. Have you really read it? The book makes this quite clear. The Wikipedia page doesn't.
