I have the opposite experience: random HN/Reddit comments saying “this sucks” or “whoa this is a huge improvement” are the only benchmark that means anything. Standard benchmarks are all gamed and don’t capture the complexity of the real world.

Then your internal benchmarks will be in the post-training set and you’ll have to make new ones.

I may already have but I'm pseudonymous on this website.