I don't disagree with your overall points, but if you're using Visual Studio in such a limited way that you can possibly think of it as a "souped up text editor", you might as well just actually use a souped up text editor - it'll be a lot faster.
Good point and it's my use of Visual Studio that kind of screws me. I've got ReSharper installed, which bogs down the editor quite a bit, as well as an extension that I wrote that isn't nearly as efficient as it should be (adds measurable overhead, but nothing close to what ReSharper adds -- I'm close to having a working update that corrects my mistakes there but it filled a gap I needed filled and I originally wrote it with no intention of releasing it -- plans changed).
Visual Studio is pretty ugly in a lot of ways, though. Writing extensions for Visual Studio exposes the ugly underbelly of the beast, which is basically WPF window dressing laid on top of a whole pile of COM Interop (not all that unusual for an application of its scope and history). There's a lot of legacy there. Generally speaking, I do avoid Visual Studio for a lot of things, favoring everything from VS Code to Notepad++. If I spend too much time in Code, or Sublime, and stay out of Visual Studio, returning to it really hurts. At the same time, for all of its complexity (and probably "because of"), I do almost all of my C# development in it and a good chunk of my C++ development there (even more so after Update 2).