Well, it may be that you're not ageist. You may instead be anti-natalist. If we were talking about a 20-year-old having a kid, you might go on about how they don't have the maturity necessary to take on such a responsibility. And of course it's obvious that you shouldn't have kids if you haven't achieved an upper-middle-class level of financial stability. Then there is the matter of genetics. Do you have ancestors who died of Alzheimer's disease, cancer, or heart disease? Those all have a genetic risk component. Do you want to pass on such genes to your kids?
The fact is that every kid comes into the world with some advantages and some disadvantages. Any disadvantages from having an older parent are at most on a par with the disadvantages from not having particularly wealthy parents, or not having parents with perfect genes, or having parents who were abused by their own parents, or having parents who aren't citizens of a first-world country, and so on and so on.
Would you like to assign me any more imaginary opinions? Your reply is an exercise in whatabouttery: ignore this harm because look at all those harms over there. Controlling the age at which one has children is within one’s power to control, and helps to alleviate obvious harm. Nothing you have said or can say will change that.
Your statement that "the age at which one has children is within one’s power to control" is just obviously false. Especially for men. But also for everyone, if you're trying to time things to mitigate other disadvantages.
In reality, you may be 51, and may be in a good position otherwise to have a kid, and the question is whether being 51 is a reason not to. And the answer is "no", it is not. Not unless you also think that other people with non-major reasons not to have a kid shouldn't. In other words, not unless you think nobody should have kids.
The fact is that every kid comes into the world with some advantages and some disadvantages. Any disadvantages from having an older parent are at most on a par with the disadvantages from not having particularly wealthy parents, or not having parents with perfect genes, or having parents who were abused by their own parents, or having parents who aren't citizens of a first-world country, and so on and so on.