Not just that. Even if you don't make that mistake, having servers ssh into other hosts and leaving keys on them for this purpose means if one machine is compromised, others can be too. And they can use known_hosts to discover which ones.

ssh -A is a thing. A risky thing, but so much better than keeping private keys on server.