Down-vote comments could be hidden by default so as to not 
  pollute threads too much.

I really like the idea of private comments for this case.

Wild tangents, pedantry, grammar/spelling corrections, personal questions, (constructive) criticism could all be optionally made private and avoid clutter in the threads