> this is like insisting that every guy with Olympic gold holds a "false" medal, every time
Bad comparison. We have loads of statistics from sports showing there are highly-precise rankings. Run certain races repeatedly and you’ll find persistent rankings. Particularly at the most-competitive levels, where innate biology dominates in most sports.
Have a cohort of students re-take a standardised test a few times, on the other hand, and you’ll get a spread. Try to relate that spread to the things you’re actually trying to measure, academic potential, and it’s a hair better than a crap shoot.
Maybe. I'd hazard a guess that membership of the top 1000 places in a high-stakes national exam might actually be less noisy than a gold medal. That is, on a re-run, what percentage of people keep their top-1000 scores, and (on re-runs of the last 40 years olympics, times 100 individual events, say) what percentage of gold medalists would have kept theirs?
But I have not checked the numbers. Obviously if you restrict attention to those with exam scores near the boundary, you can get different results. And there are indeed other sports scores more precise than the one I mentioned.
Bad comparison. We have loads of statistics from sports showing there are highly-precise rankings. Run certain races repeatedly and you’ll find persistent rankings. Particularly at the most-competitive levels, where innate biology dominates in most sports.
Have a cohort of students re-take a standardised test a few times, on the other hand, and you’ll get a spread. Try to relate that spread to the things you’re actually trying to measure, academic potential, and it’s a hair better than a crap shoot.