Sure, but this is like insisting that every guy with Olympic gold holds a "false" medal, every time. While the stated rule is that it goes by who crosses the line first on the day, this obviously isn't a perfect measure of who's ultimately really in some other abstract sense the fastest man alive.

I guess the common way of deliberately adding noise is by rounding off scores, such as giving A/B/C instead of rank order. Which means that nobody gets a letter saying that their score (the one from the exam) has rank 1001.

> this is like insisting that every guy with Olympic gold holds a "false" medal, every time

Bad comparison. We have loads of statistics from sports showing there are highly-precise rankings. Run certain races repeatedly and you’ll find persistent rankings. Particularly at the most-competitive levels, where innate biology dominates in most sports.

Have a cohort of students re-take a standardised test a few times, on the other hand, and you’ll get a spread. Try to relate that spread to the things you’re actually trying to measure, academic potential, and it’s a hair better than a crap shoot.

Maybe. I'd hazard a guess that membership of the top 1000 places in a high-stakes national exam might actually be less noisy than a gold medal. That is, on a re-run, what percentage of people keep their top-1000 scores, and (on re-runs of the last 40 years olympics, times 100 individual events, say) what percentage of gold medalists would have kept theirs?

But I have not checked the numbers. Obviously if you restrict attention to those with exam scores near the boundary, you can get different results. And there are indeed other sports scores more precise than the one I mentioned.