They could probably swing it in court as similar to knowingly purchasing counterfeit goods, which I'm pretty sure is illegal. Thus far they're only going after people for uploading because: 1) it's less of a legal grey area and 2) the huge monetary damages serve as a warning while going to prison would probably seem too harsh, even for people that think side with the RIAA/MPAA.
If you are using bittorrent, you are almost certainly uploading.
I'm also not sure that they never prosecuted anyone for downloading is actually a sufficient amount of evidence that it is legal. I've never heard of anyone get prosecuted for uploading just one song or one movie, that doesn't mean that it would be legal to do so.
I don't know about the U.S., but at least in The Netherlands you are allowed to freely download copyrighted movies and music for personal use. For now, anyway...
[citation needed]? I’ve never heard that before. Mind you, I’m not in the US.