It could be a large deposit, to be returned if the case is verified, or to be rewarded to the victim if it turn out to be vicious.

I like this. Large deposit with some gov agency and the money gets awarded to the victim (or refunded to the claimant) when the dispute is finally resolved. You don't submit a claim unless you're really sure that you'd win.

This is how the DMCA should have been designed in the first place.