In addition to the "collective action problem" issue, there's also the fact that amassed wealth can often translate into significant power[1], and if "anyone who isn't a huge jerk gives what's needed to charity" that amounts to a transfer of power toward huge jerks, which is not really an ideal outcome.
[1] There is an argument that unconstrained government is responsible for this. This is not completely without merit: it is certainly true that it can exacerbate the problem, as money corrupts politics and politics wields the force politics wields. However, it can be the case with a minimalist government too; denying that just seems silly. At the extreme, consider paying directly for use of force (open or secret); but subtler things like market power also come into play.
[1] There is an argument that unconstrained government is responsible for this. This is not completely without merit: it is certainly true that it can exacerbate the problem, as money corrupts politics and politics wields the force politics wields. However, it can be the case with a minimalist government too; denying that just seems silly. At the extreme, consider paying directly for use of force (open or secret); but subtler things like market power also come into play.