I'm the same, though I don't consume a lot of caffeine lately. I do struggle with anxiety disorders but caffeine doesn't make it worse(or better).

Energy drinks do though, and they make me jittery and sweaty, which coffee doesn't. Sugar or sugar-free doesn't seem to make a difference.

My hypothesis is that it's all the niacin they put in energy drinks. 250% of RDI in a 500ml can is typical, sometimes even more than that.