I didn’t say “defend your time, from your manager”. The people who will invite you to a meeting in most companies are myriad. Some of these are truly collaborative and for mutual benefit. Many are, as you said, just making your job harder. Hopefully for _their_ benefit but it does come at a cost to you and you can’t afford to give all of your time up for others. Defend your time.
Thanks for clarifying that you didn't mean managers. I was confused because my post was only concerned about managers, and so I thought you were replying to that.