The flip side of the fairness argument is that test scores are just one time point (often just a couple of hours, maybe several hours at most, of one day) in one format with no flexibility for unusual circumstances or anything. GPA, in contrast, is reflective of repeated things over a long time period, with lots of variables accounted for.

There's issues with GPAs too, like lack of standardization, but they both have their issues with fairness.