I have been threatened and people I know have been assaulted by unhomed people. I have never been threatened and nobody I know has been assaulted by homed people. So no, they aren't "ridiculously" scared of homeless, it's a reasonable response to the on-the-ground truth.
I'm not saying that it doesn't happen. But the reality is, to make one random comparison to a bigger threat, that 597 people have died in 2023 from mass shootings [0] and I can't even find any statistics on the number of people killed by homeless in 2023. When I tried various search queries, all I find is articles about the number of homeless dying and things like serial killers who target homeless in LA.
Yes, some homeless are dangerous. But the simple truth is that there are still way more crazy homed people who are dangerous. Yes, that is obviously partly because there are way more people who are homed than homeless. That is part of my point - the homeless issue is magnified by fear. I'm not saying it's made up.