You can do it either way depending on what you're trying to achieve. I have a dev container set up that runs Xvnc and/or Xpra in a container so I can connect to it from anywhere.

Possibly going a bit off-topic here - but isn't Xvnc arguably a "headless" X server, in that it appears to be a X server and a VNC server - but as X client/server terminology is round they other way from VNC (and pretty much everything else) it isn't really a "graphical" application in the sense of jmnicolas's original question...