It's a pretty serious problem with how we benchmark though.
EDIT: when looking up what I vaguely remembered I somehow managed to come across a similar article that was published just today[1], even though I was referring to an older one[2] which was about microstuttering (basically: a high standard deviation in frame rate). The point still stands - in fact it applies to both cases in somewhat different ways.
To give an example: Crossfire and SLI graphics card setups a few years ago[1]. It turned out that while both gave a similar performance increase in average framerates. Then was it discovered that one of them had a significantly lower minimum framerate than the other. A high minimum framerate is probably more important in shooters than peak performance, but that's not what we've been testing all of these years, is it? That's exactly the problem highlighted in the article by Zed.
I know this is a gaming example, but I'm sure that in user perception of the performance this matters just as much for the responsiveness of webpages.
Indeed, it's important to look not just at the average performance and performance extremes, but also the distribution of performance.
Standard deviation helps with this. Also, often times looking at the latency at the 50, 90, and 99th percentiles is valuable as you can see events that would make your users unhappy. They're a very tangible series of metrics.
He's a fantastic software engineer but he is _very_ abrasive. I've read some of his posts claiming he would fight some other developer in person at a conference if he steps up. He would rent out a ring and he would put his yellow belt to practice.
Really, I'm not making this up. He sounds like a jerk to work with.
He's actually a really nice person. He helped me out (to him I was just some stranger on the phone) when I was trying to decide what to do with my career when I was in NYC.
Zed Shaw is probably one the best people you can know in the developer community, a very good guy.
Your sensationalism based on some of the stuff he says on Twitter and Blogs is amusing though.
I am among those appreciative of his contribution to my education. At the same time I wish he'd let the word 'fuck' regain its undoubted impact by not using it on what seems to be every conceivable occasion (excuse the pun).
He's just a bit too honest and direct for the average American.
From that perspective at least, I expect people like Zed to do great in Northern Europe. There's a reason Americans think DHH is rude. He isn't; he's just Danish.
" I mean business when I say I’ll take anyone on who wants to fight me. You think you can take me, I’ll pay to rent a boxing ring and beat your fucking ass legally. Remember that I’ve studied enough martial arts to be deadly even though I’m old, and I don’t give a fuck if I kick your mother fucking ass or you kick mine. You don’t like what I’ve said, then write something in reply but fuck you if you think you’re gonna talk to me like you can hurt me."
Oh really? Then why did Zed write such an angry rant about how you are doing it wrong?
http://zedshaw.com/essays/programmer_stats.html
Can we please see some standard deviations, at least?