Why do you trust it? You got into a car on an arranged deal and the driver immediately wants to back out of it--this is already a shady situation. Why do you think he'll drive you anywhere after he gets the $10? Why do you think he won't charge you for the trip on top of the $10 after alerting Uber to their own GPS records that show your phone and his car taking you to the destination you said you wanted to go to?
I would never pay for a ride in someone's car before the ride completed. Taxis don't expect that; who does this guy think he is? If he wants something up front he's out of luck. In fact I probably would reject the offer as I suggested above, but if I accepted I would immediately cancel the trip. If Uber want to charge for a cancelled trip they can bite my ass.
It is probably "privilege" as a big heavy white dude with a perpetually surly expression that I'm not alerting on "danger" in such a situation. If I were a different person I could totally imagine reporting the driver and trying to get him fired. People worry when getting in strangers' cars. Actually once in India I did get such a misdirected cab ride, but my Indian friend noticed and convinced the cab driver to stop. He chewed the driver's ass for a while, then we got another cab.
What's wrong with that? It's just an offer that you're free to reject.