It is backwards compatibility, because there are a fantastic number of programs out there that simply allocate a MAX_PATH sized buffer and assume that a single file path has to fit into it. If MS simply changed things so that paths of 500 characters could be returned all those programs would fail.
Or, you could have a new set of API calls without this limitation. This new API could also receive a version argument, so that calls to its present version would return present-style data, even if the then-current implementation were much smarter. It's so obvious, in fact, I wouldn't be surprised if they already did it.
