As everforward says, lenses are a way of managing that. Downranking a site doesn't mean you'll never see results from that site (unless you want it to), it means that the site will score lower in the results than it would otherwise. And you can change the rankings anytime, if you wish.

I don't use the lenses myself because I don't need different rankings for different things, but they might address your needs.