The drift I see is when your finger is off the stick and the cursor keeps moving. And it goes more than a few seconds.

Yeah, it'll do that because it's used to your finger being on the stick and thinks that's the neutral position. It's supposed to stop and recalibrate itself eventually, especially when it reaches the edge of the screen. I've never had that take more than a dozen seconds on my Thinkpads (X31, T60) but of course YMMV, yours might be actually buggy.