To a degree, this common desire to know the amount of headroom available for optimization was an impetus for our web frameworks benchmark project [1].
Some readers of the results have asked why the tests are so trivial (although the multiple query test is anything but; and we are adding more computationally intensive tests for later rounds).
We chose to start with trivial request types to establish the high-water mark that the platform and framework set. Any given web application built on the framework can only expect to perform with higher latency and fewer requests per second than the high-water mark.
In turn, the higher-complexity request types provide some insight into the converge-to-zero point along the complexity axis. Prior to seeing the data, I think some of us intuitively assume the converge-to-zero point is lower than it actually is, using the all too popular refrains such as "well, the app is always waiting on the database anyway." Yes and no. Certainly database queries are generally expensive. But some frameworks can run twenty trivial queries on a gigabit-connected database and serialize the resulting list to JSON faster than others can serialize a single trivial object to JSON.
Choosing a framework that provides headroom for custom application logic--and having a rough idea of how the platform will perform with your custom logic as demonstrated by representative tests--is a good idea. And it doesn't take a whole lot of effort to do that kind of research now. It's easy to avoid premature scale pain.
The subject was 'ability to optimize' and you have changed it to 'how optimized the framework is already'. Most of one's ability to optimize is not dictated by anything about the framework.
Benchmarks which artificially exclude common optimization techniques are avoiding the interesting question of room for optimization in favor of meaningless pissing matches about 'language speed' or 'framework speed'.
Pekk, I appreciate the response, but I don't see it that way. I see an efficient framework as getting out of the way and not consuming too many CPU cycles for its plumbing business, leaving my application with room for me to code quickly and inefficiently then optimize when the time comes.
If I build on an efficient/high-performance framework, I can make sloppy code to start and get the job done fast. When the time comes to optimize, I have a great deal of headroom available. I see this as "ability to optimize."
If on the other hand, the framework (and, as importantly, the platform) is already consuming a great deal of the CPU's clock cycles for things outside of my control, my ability to optimize is greatly diminished. I will run into a brick wall unwittingly erected by the framework and its platform.
If my ability to optimize becomes a matter of replacing core components of the framework with marginally faster alternatives, the experience devolves into a frustrating guessing game wrought with arcane insider's knowledge ("Everyone knows you don't use the standard JSON serializer!") and meandering futile experimentation ("I wonder if using a while loop rather than a for loop would squeeze this request down to 200ms?")
I'd rather know that the framework is designed to give as many as possible of the CPU's cycles to me and my application. I can then be reckless at first and careful when time permits.
Which benchmarks artificially exclude common optimization techniques? If you're referring to ours, please tell us what is on your mind. You brought this complaint up in the comments on our most recent round [1], but didn't follow up to let us know what we did wrong. We are absolutely not interested in artificially excluding common optimization techniques. In fact, we want the benchmarks to be as representative of common production deployments as possible.
Some readers of the results have asked why the tests are so trivial (although the multiple query test is anything but; and we are adding more computationally intensive tests for later rounds).
We chose to start with trivial request types to establish the high-water mark that the platform and framework set. Any given web application built on the framework can only expect to perform with higher latency and fewer requests per second than the high-water mark.
In turn, the higher-complexity request types provide some insight into the converge-to-zero point along the complexity axis. Prior to seeing the data, I think some of us intuitively assume the converge-to-zero point is lower than it actually is, using the all too popular refrains such as "well, the app is always waiting on the database anyway." Yes and no. Certainly database queries are generally expensive. But some frameworks can run twenty trivial queries on a gigabit-connected database and serialize the resulting list to JSON faster than others can serialize a single trivial object to JSON.
Choosing a framework that provides headroom for custom application logic--and having a rough idea of how the platform will perform with your custom logic as demonstrated by representative tests--is a good idea. And it doesn't take a whole lot of effort to do that kind of research now. It's easy to avoid premature scale pain.
[1] http://www.techempower.com/benchmarks/#section=motivation