You could try the opposite: give entirely random people the ability to vouch.

I have no idea if this would work, but.. I've been doing lots of ML lately where it is often pretty hard to beat random, and throwing noise into a system make it much more robust.

I agree about noise, but you'd need to be careful. People would try to game it by giving themselves as many random chances as possible, by making many accounts or doing whatever was necessary.

It could undermine the legitimacy of the system, as well as put more load on the server.