Well, not really, because what if ubuntu packages rely on version X, and you need version Y.

You roll version Y yourself and install it into an alternate prefix. If the server uses debian that means make a new deb and deploy it using the standard tools. apt-get/yum/etc. are very solid deployment tools.

I wasn't saying it was impossible, but what you've described is already about 10 times harder than using virtualenv.