It seems that in the late 19th century in the U.S., the annual death rate from horses was approximately 5 per 100,000 inhabitants. That surprised me, as it's ~1/3rd the current U.S. road traffic fatality rate, and as horses were used for more than just transport.
I couldn't find any figures, but obviously the distance travelled per capita has exploded, so it's likely that horse riding is at least several orders of magnitude more dangerous than driving.
In addition to just death rates there are broken bones as people fall off (possibly less of a problem with buggies as mentioned above), all kinds non-fatal injuries because of biting, kicking or just panicking and either running over someone or dragging them through something without necessarily killing them.
Ulike cows and sheep who only have front teeth in their lower jaws horses have teeth on both sides and can easily take a bite of someones arm if they snap.
Add to this the massive hygienic problems that would occur if we suddenly had these animals walking around leaving their droppings everywhere.
The blanket statement that horses have high safety ratings makes me smile :-)