> In theory it ought to be possible to switch a single GPU to/from a VM without a reboot. In practice I have no idea how huge a refactoring to the Linux graphics stack that'd require.

I've got this working today. I do it through swapping the nvidia driver for the vfio-pci driver (and back again if required). The slight annoyance is that you may need to restart X11 (for me this is not an issue).

I wrote about this some years ago: https://me.m01.eu/blog/2016/05/pci-passthrough-vm-monitor-se...