Having a large / strong jaw and chin is correlated with higher testosterone levels and is a secondary sexual characteristic in males. Higher testosterone levels are correlated with a weakened immune response.

So yes, a "manly chin" may very well have been selected for on the basis of showing the strength of the individual by artificially handicapping them (being genetically "rich enough" as you put it).

Interestingly, beards act to further emphasize the chin. And we're one of the few animals to have those, too.