Is crashes/mile the best metric or mistakes/mile that could lead to a crash? I certainly make a TON of mistakes that I can correct before they lead to problems--most of them are mild, like having to break a half second later than I'd like, but i bet you are UNDERestimating the improvement due to self driving cars.
It's definitely crashes per mile. Unexpected stuff happens all the time on real roads -- black ice, debris, animals, flat tires -- and safety depends upon the driver's or software's ability to deal with them. It's easy for self-driving not to wander into another lane because they're changing the radio, but it's hard for them to deal with situations their designers didn't anticipate.