It might not make much of a difference with a hobbyist's podcast, but filler/mouth sounds won't fly in professional productions for a variety of reasons (time constraints, professional standards, wanting to make hosts/guests sound good, etc.)