It's the perspective of roughly "maximizing ones possible achievements".

Paul Graham is rich, connected, and inluential, to what degree (in percentage terms) does he use this to make the world a better place?

Gotcha. Those who are rich, connected, and influential could all seriously consider our impacts on the world.

I don't see failing to use your wealth well and trying to push someone out of a car as equivalent "shitty things."

I do. In fact, pushing someone out of a car only hurts one person. Not utilizing your resources for good hurts thousands, maybe even, millions.

Especially when "your" wealth is fairly arbitrary, and hardly democratically decided.