Adding down votes would add another input to the system. In general, elegant systems aren't improved by adding more inputs.

My plan of attack would be to think about the internals of the system and decide which inputs need to be simplified and require better feedback for the user.