The problems with that approach come from trying to de-skew the data, and then from sampling the data into whatever port you're using.

Here's a document from 1997 that looks at some hardware RNGs and how they fail: http://www.robertnz.net/true_rng.html