I'd say there's a difference between letting someone harm you (including taking up your time) and letting them benefit from harming you.

I agree, but only in situations where that party can harm you again or repeatedly in the same way (e.g. the government, monopolies, polluters). If you can throw a chargeback on some vendor you don't have to patronize ever again (or just accept a small monetary loss), it's worth it if it saves you wasting a ton of precious time.