“While nearly all such publi- cations detail their system’s impressive scalability, few directly evaluate their absolute performance against reasonable benchmarks. To what degree are these systems truly improving performance, as opposed to parallelizing overheads that they themselves introduce?”

That’s one of the things that stuck with me after haven taken a class in high performance computing. What matters is the absolute performance, less the speed up one achieves by parallelisation.