Often the taxi driver has waited for a long time (sometimes hours) in the airport queue for it to be their turn to get a fare. If you take a $5 ride they’re upset because they invested all that time for no payoff. I don’t think it’s wrong for the passenger to do this, but I understand why it’s frustrating for the driver.