I realise that the OP is about a speech, but I think universally dismissing "um"s isn't justified. A speech is not the same thing as a dialogue. Conversational fillers are useful to hold the floor (not that there aren't better ways than "um" but that doesn't mean "um" isn't useful or ineffectual).
I guess that depends on your objective. In either case (communicating a message, or arbitrary conversation) you can choose to be ok, or you can choose to be exceptional.
Ums in either case are a distraction, and to some annoying. The degree obviously varies from person to person. Being a good conversationalist is as counter-intuitive as skiing (where you have to lean forward when facing downhill) in that a good conversationalist is in fact a good listener, and only interjects to elicit more conversation from the other party. So... in a way you're right. Uming a lot when you're really just listening isn't going to affect anything ;-)
Making a speech is about communicting a message. There are many ways to structure that message, for example SCIPAB, but the structure and supporting slides, if any, can be spectacularly annihilated by a mediocre delivery.
To be exceptional you need to work on your delivery. That means slowing way down, not using ums and ahs, maintaining eye contact (I've addressed a 5000-strong crowd, so believe me when I say it's irrespective of the size of the audience), and getting your body language right.
Depends entirely on how good you want to be :-)
If you want to be exceptional you could do worse that start with The Voice Book (http://www.amazon.co.uk/dp/0571195253/ - Amazon.com is sold out). It's been derided somewhat elsewhere in this thread, but Toastmasters is another hugely useful route.
There are plenty of smooth talkers who can give a good speech, without any umms, and yet leave the audience unfulfilled. Obviously there are just as many bad speeches full of too many umms.
I think there is a middle ground where 'umms' and general fillers make the communication more authentic and seemingly less rehearsed. Most people don't like listening to a robot, not even a well-rehearsed one.
