The way the system would get described to me by the people I know who used to work at Goldman was that the tech guys would all treat the system seriously and attempt to rate people while the traders would all rate everyone at 9 for everything, unless they hated someone or someone had done disasterously in that area, in which case they would be ranked at 7. The programmers would always be looking for excuses to get a trader to rate them.
It was always told as a parable about how bad at real life game theory the academic, logical, game theoretic minded programmers were vs the traders.
To poll meaningful opinion in a highly incentivized environment like that, ratings would have to be strictly relative. Not "is person p1 any good at quality q1, q2, q3?", but "quality q1, q2, q3, which one of those is p1's strongest, second strongest, third?". A decision maker could then use that data to extrapolate from qualities he can judge (even if it's as stupid as "attention to personal appearance") to those he can't. People who score badly on a visible quality they are obviously good at must be pretty stellar at the invisible ones, those who score much better than expected in a visible quality, well, there might be a reason for that.
Still, that system would be sabotaged by favor based raters. They would quickly rank qualities by perceived importance (I know I would) and blindly rate the most important category highest/lowest depending on popularity.
Enter sudoku rules: make it a square of persons/qualities and require both rows and columns to be a complete distribution. The challenge of creating a valid matrix should then take care of any remaining game-theoretic influence, leaving only honest opinion and a fair amount of noise (which is a much lesser evil than deliberate manipulation).
It doesn't say how good anyone is at game theory. It might be a proxy for how social they are. The outcome is based on how "selfish" people are, or how much retribution they expect in future iterations. I don't see how this could possibly be used to make a deduction about the understanding of game theory (or practice).
but this is _exactly_ what game theory is about, the parent probably hinted at a failed unterstanding of the theory as shown by your comment ;)
what i mean is, people may think they understand the concept of game theory but are often unable to link that to its practical consequences, as in this case.
I really enjoyed "Jane Austen, Game Theorist" which argues that Jane Austen systematically examined Game Theory years before it emerged.
The author also argues that Jane Austen has analysed in more depth game theoretic 'cluelessness' than has been done since, and interestingly, many of her 'clueless' characters show signs of being unusually interested in numbers, maths, card games, etc.
It was always told as a parable about how bad at real life game theory the academic, logical, game theoretic minded programmers were vs the traders.