I totally agree. I was really annoyed by the distraction until I discovered that I could just drag it into the [X More] group under my labels. No more pesky buzz reminders for me!
That's a good point, but it also makes it two clicks away. I would think the best design would be to have it one click away while not providing unread-anxiety.