I grew up in a rural area with its fair share of dirt/gravel roads.
Vehicles definitely see a lot more wear from driving on dirt roads (beyond the superficial constant dust covering). I expect that the cost of pavement is less than the cost of additional maintenance for all the road's users. (This also explains why it makes sense to leave some dirt roads in very rural areas.)
This depends on amount of traffic, and conditions.
I live in Finland where the population decline in some rural areas has lead to it being economical to remove paving from roads. With not much traffic - but the road still being essential for local economy, e.g. timber transport - it's cheaper to maintain a dirt road than a paving.
Part of this is due to local climate - in the winter the frost goes a meter deep in the ground, so maintaining paving means an occasional re-build every few decades, putting under the tarmac a deep layer of material that passes water through quickly. With dirt roads, you don't need to, you just level it, which is quick and cheap.
Vehicles definitely see a lot more wear from driving on dirt roads (beyond the superficial constant dust covering). I expect that the cost of pavement is less than the cost of additional maintenance for all the road's users. (This also explains why it makes sense to leave some dirt roads in very rural areas.)