If all other circumstances are equal this is true from a physical standpoint, as E = 1/2 m v^2, at least as long no high speed driver crashes into them.
It's relative velocity that counts, and in most situations it's the other cars that are more likely to cut in front of you or rear-end you than the tree on the side of the road. People who think slow (relative to the ground) is safer are oversimplifying.
The problem is that we humans throw a wrench into the mix. Slow drivers even cause irrationality in other drivers; I would like to consider myself competent, but I'm willing to admit that a bumbling old fart in his Mercedes can get on my nerves.
I realize that, "if all other circumstances are equal," was specifically mentioned.
Every fast driver thinks this about themselves.