Randomness inserted like that can be filtered out statistically with a decent sample set - it's a gaussian distribution, so you'll still see the saying timing differences. This is why things have to be constant time, rather than random return time.