Why tolerate a delay when there is no need to? I'm sorry but there is no talk in the world where every single person is going to be completely engaged. This is such a non issue, like a teacher wasting an entire class to discuss a minor discipline problem.

Not every person in every room you're invited to talk to will absorb and care about every word that leaves your mouth. If they're minding their own business then just ignore them.

If you get distracted by people doing it or are bothered by the large number of people doing it, then you need to drastically improve your speaking skills. You distract too easily and/or your talks are boring.