Well, this is because (at least as far as I've observed), the probability of having many many buffers is far greater for each time a buffer has already happened. Accordingly those who value their time will go elsewhere
I agree: in my experience on Youtube, most videos that buffer at all will do it many times. I don't mind the waiting so much as the constant interruption.