This is my concern too. Any person who's gone in for many interviews has seen that the interviewer isn't the absolute measure of performance: we've all seen problems that are just trivia questions, are designed to show off the skills of the interviewer, or (worst of all) where a second solution isn't accepted because it's not what the interviewer thought of. All of these issues can result in disagreements on performance, in both directions.

I would like to see a different analysis: for multiple interviewers of the same candidate, how similar are the ratings?