It's not a ludicrous statement. We must assume the VC wants to make money and thinks it is an indicator, just like we assume a plumber wants to fix pipes even if he uses a divination rod to find leaks.
There is a difference between intent to be rational and actually making good decisions, and they don't have to go together.
Unless maybe the VC is in irrational human being who has biases just like everyone else.
Unless they've actually done studies on whether fidgeting is a good indication of company performance, this statement is ludicrous.