I'm not convinced it would do anything harmful, so long as you have enough NTP sources (which you should have anyway).
From their FAQ:
> We recommend that you do not mix smeared and non-smeared NTP servers. The results during a leap second may be unpredictable.
I read that as a soft SHOULD NOT, not MUST NOT. Would be a fun exercise to try doing it intentionally with common NTP implementations and see what happens.
