Actually I think many developers aren't aware of the right tools for this type of job.

JITs are easy to benchmark when tools like Oracle Studio, Java Flight Recorder, dotTrace and so forth are used.

The problem is trying to benchmark them using old style concepts, e.g. outputting time differences to a text file.